Sequence-based approaches rely on whole-genome sequencing (WGS) of SARS-CoV-2 cases for assigning lineage, which remains the gold standard for variant identification. Sequence-based methods were used throughout the pandemic to assess VE as new variants emerged. For example, Kirsebom et al. assessed VE of XBB.1.5 and BA.4/5 boosters given as part of the fall 2023 program in the UK against XBB, EG.5.1, and JN.1 lineages using WGS. The authors found evidence for lower VE against JN.1 lineages, which emerged in late 2023 and became predominant in January 2024 [10], mirroring findings from a multi-site cohort in the US which also used WGS to assess VE against XBB and JN lineages [11]. Unlike proxy- or period-based methods, sequence-based approaches have the advantage of full genomic resolution and could potentially also be used to assess VE by specific spike protein substitutions [12,13] or by genetic distance from the COVID-19 vaccine strain [14,15]. Methods to characterize how vaccine efficacy varies by infecting pathogen strain have been referred to as sieve analyses and have been conducted for HIV, malaria, and SARS-CoV-2 vaccines in randomized clinical trials [16], but are not yet routinely applied in observational vaccine studies.

Sequence-based lineage assignment using WGS requires active sample collection or salvaging of stored clinical samples (potentially including rapid antigen tests [17]) and efficient specimen shipping, processing, and sequencing to produce timely VE estimates. Completeness of sequencing results also depends on adequate respiratory specimen quality; specimens with high PCR cycle threshold values often contain insufficient viral material for sequencing. Sequence-based lineage assignment may introduce selection biases, as specimens that cannot be sequenced due to low viral load are excluded and viral load may correlate with variant, disease severity, or vaccination status [18,19]. Sensitivity analyses can be conducted comparing overall VE with VE among patients with sequenced specimens to assess the degree of bias [20]. Additionally, mixed infection with multiple SARS-CoV-2 strains can rarely occur, and lineage identification approaches need to distinguish between sequencing artifacts, true mixed infections, or novel variants [21]. Lastly, attaining adequate sample sizes for sufficient statistical power in VE studies is perhaps the greatest challenge for studies based on WGS compared to other assignment approaches due to cost, equipment, and personnel requirements for sequencing. This may preclude precise estimates of VE stratifying simultaneously by lineage and other covariates including age, immune status, and time since vaccination.

In the absence of WGS capacity, real-time reverse transcription PCR against select amplification targets can sometimes be used as a proxy for genetic lineage (i.e. proxy-based lineage assignment). For SARS-CoV-2, a commonly used proxy is the TaqPath COVID-19 multiplexed diagnostic test, which amplifies the S, ORF1ab, and N genes of the viral genome [8]. Failure of S gene amplification can occur due to a deletion at spike protein residues 69 and 70. This mutation was first observed with the Alpha variant [22], and has since alternated in occurrence with the emergence of some subsequent variants, including Delta, Omicron, and various Omicron descendants [9,23]. Targeted RT-PCRs for specific spike mutations have also been previously used to detect known variants of concern [24], and further work is needed to develop new scalable and accurate proxies [25].

The use of S-gene target failure (SGTF) or presence (SGTP) as a lineage marker can enable rapid estimation of lineage-specific VE. Tseng et al. used SGTP/F in a large integrated health network to estimate effectiveness of mRNA-1273 boosters against Delta and Omicron variant infection, respectively, in December 2021 [26]. Similarly, Link-Gelles et al. used SGTP/SGTF in a nationwide pharmacy testing network to conduct early estimates of VE against symptomatic XBB lineage infection in 2023 [27] and JN lineage infection in 2024 [28]. Use of SGTP/F to define lineages can be more cost-effective and higher-throughput than WGS because SGTP/F relies on existing diagnostic RT-PCR laboratory infrastructure, and results are typically available more quickly [9,23]. However, SGTP/F cannot discriminate between specific mutations or other sub-lineage variation that may confer immune escape.

One of the main challenges for proxy-based approaches is that sensitivity and positive predictive value for emerging variants of interest must be continuously assessed and validated using WGS. The sensitivity of SGTP/F for differentiating lineages has varied, and can be increased by including “partial” SGTF samples where amplification of the S-gene occurs but is reduced compared to other genomic targets [23,29]. The positive predictive value of SGTP/F can be influenced by prevalence, especially as a new variant is beginning to emerge. In December 2022, as the XBB.1.5 variant began to spread in the US, positive predictive value of SGTP was relatively low with XBB or XBB.1.5 constituting approximately 60% of SGTP specimens and the remainder being other BA.2 sublineages [9,27]. Andeweg et al. restricted to time periods when SGTP/F had >85% positive predictive value for variants of interest to address this issue [30], but this could delay VE estimation for emerging variants. Further assessments are needed to understand how this type of misclassification could influence VE estimates and to develop approaches to adjust for misclassification, such as regression calibration.

Period-based VE analyses approximate lineage assignment by assuming that all case-patients enrolled during a specified time period are infected by the predominant circulating SARS-CoV-2 variant. The period is defined using a prevalence threshold for the variants of interest, and thresholds used have varied considerably across studies: 50% was common, but higher thresholds such as 60%, 70%, 80%, and 90% have also been used (see Supplementary Methods for examples). A key advantage of this assignment approach is that it enables estimation of lineage-specific VE from platforms without individual-level WGS or PCR laboratory data, potentially enabling larger sample sizes and representativeness. For instance, Schrag et al. used a period-based approach in a large multisite electronic health record network to characterize mRNA VE against medically attended illness during Delta versus Omicron periods, with simultaneous stratifications by pregnancy status, number of doses, and days since dose receipt [31]. Period-based approaches cannot resolve variants during prolonged periods of co-circulation, such as during mid-2024 onwards, when multiple JN.1 descendants (e.g. KP.2, KP.3, LB.1, KP.3.1.1, XEC, etc.) emerged and co-circulated without establishing sustained predominance [32]. Period-based approaches may also face difficulty disentangling the effect of lineage on VE in the presence of time-varying confounders, such as waning VE from antibody level decreases following increasing time since vaccination. Extensions of period-based analyses to incorporate variant proportion as a continuous covariate may expand utility to a broader range of co-circulation scenarios [33].

Variant prevalence data for defining predominance periods can be derived using sequencing data from clinical specimens collected from the study population or from external data sources. External sources include global sequence repositories and surveillance programs such as the US CDC’s national genomic surveillance platform, which aggregated genomic data from CDC-contracted laboratories, the National Strain Surveillance program, and public sequence data repositories [32]. Sequencing data availability and capacity varies substantially globally [34], but if external data sources are available, period-based analyses likely offer cost-savings over sequence- or proxy-based approaches as it does not require collection, storage, and processing of SARS-CoV-2 clinical specimens. Robust period-based VE assessments therefore rely on sustained support for genomic surveillance programs, and decreasing counts of published sequences in some national surveillance platforms present a challenge for timely estimation of VE. Additionally, due to geographic heterogeneity in variant circulation patterns, local variant prevalence at TND enrollment sites may not match external data sources. For instance, in the US, the XBB.1.5 lineage first appeared in the Northeast in late 2022 and attained predominance several weeks later on the West Coast [35]. Defining site-specific predominance periods in multicenter analyses may better account for regional differences in variant circulation [31], provided that sufficient regional sequencing data are available.

Some investigators have used a hierarchy of data sources to classify variants [36–39]. For instance, Adams et al. and Lauring et al. conducted analyses in a multi-site TND network using periods defined by both WGS of case-patients and a 50% prevalence threshold. Cases with a sequenced discordant lineage in a given period were excluded, reducing misclassification error where possible [37,38]. Tartof et al. used a hierarchy of approaches beginning with WGS when available, SGTF, and finally period-based classification (using an 80% or higher threshold), maximizing the number of included case-patients [39].

Additionally, some studies lack test-negative controls but can still provide evidence to evaluate serologic signals of immune escape from updated COVID-19 vaccines. In these case-only study designs, the adjusted odds of receipt of updated COVID-19 vaccination are compared between two lineage groups of interest, which can provide a relative degree of vaccine immune escape. Moustsen-Helms et al. used this approach for JN.1 versus XBB lineages to provide early evidence for increased immune escape by JN.1 lineages that later TND assessments corroborated [40]. Both case-only study designs and traditional TNDs can also support early detections of changes in COVID-19 clinical severity as new variants emerge [11,40,41].

We conducted a modeling study to assess the potential impact of lineage misclassification errors using period-based versus sequence-based approaches for estimating VE. We used empirical US COVID-19 incidence and genomic surveillance data to inform model structure and parameters by estimating COVID-19 cases attributable to variants circulating in the US during July 2021 to September 2024 (Supplementary Methods). We found that most variant epidemic curves were symmetric with variable overlap between successive variants and an average standard deviation of approximately 6 weeks (Supplementary Figure 1). To mirror these dynamics, we used a simple deterministic model for two co-circulating variants assuming normally-distributed epidemic curves with the peak of variant 2 circulation occurring a range of 2 to 30 weeks after the peak of variant 1, allowing parametric control over the extent of co-circulation (Supplementary Methods, see Table 2 for a summary of parameters) [42]. We assessed other strategies for generating co-circulation patterns, such as varying the standard deviation of the variant 2 epidemic curve while holding peak timing constant (Supplementary Figure 2), but these approaches did not produce prolonged (>30 weeks) co-circulation.

Time-invariant VE against COVID-19 hospitalization was assumed to be 50% against variant 1 and 20% against variant 2, representing approximate differences in VE observed empirically against recent vaccine strain-matched and divergent variants, respectively [10,11,43,44]. These estimates of COVID-19 VE represent additional protection beyond existing levels conferred by prior vaccination or infection, which has been increasingly difficult to ascertain. We explored the impact of time-varying VE using a piecewise function where VE increases from τ=0 to a peak at τ=4 weeks. The peak time-varying VE was defined to be 50% against variant 1 and 20% against variant 2 to correspond to the time-invariant setting. We then allowed VE to wane to 0% at an identical rate for the two variants to represent natural waning of immunologic protection over time (Supplementary Methods, Supplementary Figure 3). These parameters represent one illustrative scenario and are not meant to be best estimates of current rates of waning of COVID-19 VE.

Vaccine coverage was assumed to increase linearly from the start of the season before plateauing at 20% [45]. Incidence of non-SARS-CoV-2 acute respiratory infection resulting in hospitalization was assumed to be constant. Using a TND framework, we estimated lineage-specific VE using logistic regression adjusting for calendar time (categorical week), comparing VE estimate bias and precision when assigning lineages with sequence-based and period-based approaches across different predominance thresholds.

By varying the peak of variant 2 circulation relative to variant 1, we generated a range of variant co-circulation scenarios. Setting variant 2 circulation to peak two weeks after variant 1 yielded prolonged co-circulation (37/39 weeks of circulation where both variant 1 and 2 were >10% prevalence) (Figure 1A). With increasing delay in the start of variant 2 circulation, variant epidemic curves had reduced overlap and variant 1 predominance transitioned more rapidly to variant 2 predominance (Figure 1C–F, Supplementary Figure 4). These scenarios correspond to the range of variant dynamics observed since 2021 in the US, such as prolonged co-circulation of multiple XBB descendant lineages in summer and fall 2023 followed by rapid takeover of XBB by JN.1 by January 2024 [32].

We assessed how period-based lineage-specific VE analyses using a 50% predominance threshold would perform under different variant dynamics scenarios. With prolonged co-circulation, a period-based analysis using a 50% threshold exhibited notable VE estimate bias (>10 percentage points for both variants) and underestimated the difference in VE between lineages (Figure 2A, Supplementary Table 1). As the duration of co-circulation decreased (i.e., sharper transition between variant periods), cases with misclassified lineage represented an increasingly smaller share of data and the bias in VE estimates decreased for both variants.

Because lineage misclassification during variant transition periods appeared to drive estimate bias, one solution was to increase the predominance threshold and effectively exclude weeks in which a substantial fraction of cases is likely to be misclassified. During periods in which estimation was possible, increasing the threshold to 70% (Figure 2B) or 90% (Figure 2C) reduced point estimate bias. However, by excluding samples, standard errors increased and confidence intervals widened (Supplementary Table 1). Additionally, in some scenarios with high co-circulation, predominance periods could not be defined or sample sizes were insufficient (number of COVID-19 cases for either variant <100) to estimate VE, highlighting a tradeoff between VE estimate precision and bias for period-based analyses (Figure 2B and 2C). When the modeled difference in VE between the two variants was reduced from 30% to 15% in a sensitivity analysis, the impact of this variant misclassification bias on absolute VE point estimate error was smaller, though relative trends were still similar (Supplementary Figure 5). Increasing the population-wide vaccination coverage level enhanced precision of estimates but did not substantially affect point estimates or the nature of bias observed (Supplementary Figure 6).

These findings suggested lineage misclassification error may be mitigated in scenarios in which a new variant quickly overtakes an old one. Such scenarios are often of epidemiologic and scientific interest as they suggest a fitness advantage for the new variant, but rapid estimation of VE as variant 2 is emerging is critical for early assessments of risk and public health impact. Therefore, we modified the scenario where variant takeover was rapid (Figure 1E) by estimating VE weekly following variant 2 reaching >50% prevalence, mirroring challenges of limited data availability early in emerging variant growth trajectories. Using a sequence-based approach, estimates were unbiased and initially imprecise, but increasingly gained precision as data accumulated (Figure 3A, Supplementary Table 2). In contrast, a 50% period-based analysis substantially underestimated the difference in VE between lineages early, though both precision and bias also improved with accumulation of data (Figure 3B). Use of more stringent predominance thresholds reduced bias in the point estimate, but precision decreased and estimation was delayed until the prevalence of variant 2 reached the higher thresholds of 70% and 90%, respectively (Figure 3C, Figure 3D, Supplementary Table 2). Allowing VE to wane by time since vaccination yielded similar findings: VE estimation bias was highest for the 50% period-based analysis and lowest for the 90% period-based analysis during weeks when estimation was possible (Supplementary Figure 7).