The Johnson and Johnson Ad26.COV2.S single-dose vaccine represents an attractive option for coronavirus disease 2019 (COVID-19) vaccination in countries with limited resources. We examined the effect of prior infection with different SARS-CoV-2 variants on Ad26.COV2.S immunogenicity. We compared participants who were SARS-CoV-2 naive with those either infected with the ancestral D614G virus or infected in the second wave when Beta predominated. Prior infection significantly boosts spike-binding antibodies, antibody-dependent cellular cytotoxicity, and neutralizing antibodies against D614G, Beta, and Delta; however, neutralization cross-reactivity varied by wave. Robust CD4 and CD8 T cell responses are induced after vaccination, regardless of prior infection. T cell recognition of variants is largely preserved, apart from some reduction in CD8 recognition of Delta. Thus, Ad26.COV2.S vaccination after infection could result in enhanced protection against COVID-19. The impact of the infecting variant on neutralization breadth after vaccination has implications for the design of second-generation vaccines based on variants of concern.