Many infections are caused by pathogens that are similar, but not identical, to previously encountered viruses, bacteria, or vaccines. In such re-infections, pathogens introduce known antigens, which are recognized by memory T cells and new antigens that activate naive T cells. How preexisting memory T cells impact the repertoire of T cells responding to new antigens is still largely unknown. We demonstrate that even a minimum epitope overlap between infections strongly increases the activation threshold and narrows the diversity of T cells recruited in response to new antigens. Thus, minimal cross-reactivity between infections can significantly impact the outcome of a subsequent immune response. Interestingly, we found that non-transferrable memory T cells are most effective in raising the activation threshold. Our findings have implications for designing vaccines and suggest that vaccines meant to target low-affinity T cells are less effective when they contain a strong CD8 T cell epitope that has previously been encountered.