Many vaccines, including commonly used influenza and COVID-19 vaccines, provide stronger and more durable protection against severe disease than against infection or mild disease. Vaccines stimulate a complex adaptive immune response that includes cellular and humoral immunity at both systemic and mucosal levels [1]. Protection against infection may require local mucosal immunity (e.g., neutralizing lgA and IgG antibodies and resident T cells in the nasal cavity), whereas protection against severe disease may depend more on systemic immunity (e.g. memory CD8 T cells in lymphoid tissue and blood and circulating antibodies) [2,3]. Vaccine types may differ in the protection conferred against infection or mild disease. Intranasal or oral vaccines directly target the mucosal tissues and induce local antibodies, such as mucosal IgA, and T cells, which can function like a “first line of defense” for the immune system. In contrast, intramuscular vaccination primarily generates a systemic immune response, which may be slower to activate [4]. Instead of a “first line of defense,” a systemic response may function like a “second line of defense” which confers better protection against progression to severe disease after infection has been established.

In the framework of Halloran et al. [5], the direct effect of a vaccine for protecting an individual can be decomposed into (1) vaccine effectiveness to reduce susceptibility to infection, denoted VES, and (2) vaccine effectiveness to reduce progression to disease given infection, denoted VEP. These two parts of the pathway combine to determine a vaccine’s effect on disease VESP, per the equation (1-VESP=1-VES1-VEP. When VEP>0, the protection provided by a vaccine against severe disease is greater than the protection provided against infection alone (i.e., VESP>VES). Although the original framework uses infection and disease as the two steps, the logic of decomposing vaccine effects into a first endpoint and a conditional second endpoint can be generalized in many ways. For example, the first step could be disease with any symptoms, including mild symptoms, and the second endpoint could be severe disease, defined as requiring hospitalization [6]. For studies of inpatient populations, the first step could be hospitalization, and the second step could be admission to the intensive care unit or receipt of mechanical ventilation [7]. While the steps may vary across studies, the goal is to characterize the vaccine’s effectiveness in preventing progression to a more severe form of disease. Despite the presence of mild symptoms, for example, it quantifies how much a vaccine-induced second line of defense prevents severe symptoms requiring hospitalization.

The vaccine’s effect against progression to disease VEP is a less commonly reported quantity than VESP or VES, although it has some notable uses. One use is in mathematical modeling of vaccine impact, as used to model COVID-19 vaccine prioritization strategies [8]. In these models, VES and VEP must be parameterized separately as VES influences the vaccine’s ability to reduce transmission (i.e., indirect effects). Another use, as noted above, is in characterizing the vaccine-induced second line of defense, especially where this may be linked to markers of the immune system specific to preventing severe disease in an immune correlate analysis. For example, what is the role of the cellular immune response when antibody titers are insufficient to prevent SARS-CoV-2 infection [9]? Finally, VEP plays a useful role in bridging VES and VESP. Because of their mathematical relationship, knowledge of VESP and VEP can be used to infer VES, or knowledge of VES and VEP can be used to infer VESP. Studies that leverage multiple data sources to infer otherwise hard-to-estimate quantities are an interesting area of development in vaccine evaluation research.

A pattern observed for many COVID-19 vaccines is that of waning protection against infection (or any disease, even mild symptoms), but more durable protection against severe disease. Figure 1 shows an example from a 2021 study using surveillance data from North Carolina [10]. Although not directly reported, one can estimate protection against progression to severe COVID-19 requiring hospitalization conditional on having symptomatic COVID-19 using the previously noted mathematical relationship. Yet the implied pattern of VEP over time raises questions about its interpretation. For protection against hospitalization to remain roughly steady over time while protection against symptomatic COVID-19 falls, vaccine protection against progression to severe COVID-19 must appear to increase from below 0% to over 75% in 8 months. It seems unlikely for such a large increase to represent a real strengthening of the immune system, thus meriting further investigation. Using mathematical modeling, we identify a potential mechanism where vaccine effectiveness against progression to severe disease appears to increase over time despite being constant in the model. We describe the challenges this presents for the interpretation of conditional vaccine effects.

To identify settings where VEP appears to increase over time despite no true underlying increase, we structure a mathematical model defined by a limited number of subpopulations and parameters, with relative subpopulation sizes and baseline risk bounded to be realistic using COVID-19 as a model illness.

We modeled a population of 400,000 individuals, with 50% vaccinated. Our compartmental model includes four subpopulations: vulnerable (older adults, people with comorbidities, and others with reduced immune function) and immunocompetent, each divided into unvaccinated and vaccinated subgroups. Each subgroup is assigned a group number: Group 1 is vulnerable/unvaccinated, Group 2 is immunocompetent/unvaccinated, Group 3 is vulnerable/vaccinated, and Group 4 is immunocompetent/vaccinated. Group j’s daily risk of infection is governed by the baseline force-of-infection parameter γ=0.0056 times multiplier αj, as summarized in Table 1. Upon infection, a member of group j has risk of progressing to severe disease k=0.10 times multiplier βj.

We fixed the Group 1 parameters α1=1 and β1=1 and randomly sample α2,α3,α4,β2,β3,β4 from the parameter space, each bounded between 0 and 1, to identify parameter combinations where the observed VEP exhibits a large positive slope despite being modeled as constant. That preliminary analysis served as guidance for setting the default baseline parameters as shown in Table 1.

In our default setting, 7% of the population consists of vulnerable individuals, with 50% of both the vulnerable and immunocompetent individuals being vaccinated. The parameters α2=α3=1, indicating that vulnerable/unvaccinated and immunocompetent/unvaccinated individuals have the same risk of infection, and vaccination in the vulnerable group does not confer protection against infection. We relax this assumption in subsequent sensitivity analyses. In contrast, α4=0.03, which implies substantial protection against infection VES=0.97 in the immunocompetent, although we assume it wanes linearly from α4=0.03 to α4=0.27 in 6 months in corresponding Group 4, in line with estimated VES waning from data in Figure 1. We assume sterilizing immunity after infection, lasting for the 6-month study period.

In our default setting, β2=0.5, so that the immunocompetent/unvaccinated have half the risk of progression to severe disease of the vulnerable/unvaccinated. Furthermore, β3=1, implying that vaccination in the vulnerable group does not confer protection against progression to severe disease, an assumption that we subsequently relax. Finally, β4=0.03, which implies substantial protection against progression to severe disease VEP=0.94 in the immunocompetent. We assume this is constant during the study period.

One hundred simulations were run for each parameter set, and values were estimated over a 6-month period. These simulations are represented as thin grey lines in the corresponding figures (e.g., Figure 3A), with a thick line indicating the representative simulation closest to the averaged values across the six successive months. For each simulation, we summarize the count of infections in month t from group jninf,j,t and the count of severe cases in month t from group jnsev,j,t. Observed VEP in month t (denoted VE^P,t) is estimated from one minus the fraction of vaccinated infections that are severe (Groups 3 and 4) over the fraction of unvaccinated infections that are severe (Groups 1 and 2), collapsing over vulnerability subgroup: VE^P,t=1-nsev,3,t+nsev,4,tninf,3,t+ninf,4,tnsev,1,t+nsev,2,tninf,1,t+ninf,2,t

To assess the effect of within-subgroup heterogeneity in vaccine-induced protection on the observed VEP, we modeled α3,α4,β3, and β4 as drawn from beta distributions rather than fixed. We created one beta distribution for each subgroup (vulnerable and immunocompetent) to draw our α3 and α4 values, respectively. Parameters β3 and β4 were either set equal to the assigned α parameter for each individual (in the case of correlated VES and VEP) or independently drawn from the same beta distributions (for uncorrelated VES and VEP). The waning of vaccine-induced protection against infection in vulnerable Group 3 was modeled by setting the daily rate of change in individual parameter α3 as the difference between the mean of the corresponding Beta distribution, used to sample each individuals’ parameter α3, and the value of α1, divided by 730 per day (reflecting a return to the unvaccinated level over two years in the baseline setting). If, after applying the daily change, the individual parameter α3 reached a value of 1 or above, it was kept at 1. Similarly, the daily waning in the immunocompetent vaccinated group (Group 4) was modeled by adjusting each individuals’s parameter α4. Each individual parameters β3 and β4 were kept constant, indicating constant VEP. All simulations were performed using R software version 4.2.1 (R Foundation, Vienna, Austria).

Using our four-subgroup model, we identified parameter combinations where observed VEP increases over time despite true VEP remaining stable. Only a limited combination of parameters yielded results that closely resemble the experimental data in Figure 1, and these informed our default setting defined in Table 1, with results summarized in Figure 3. Observed VEP is initially around zero before increasing to above 70%. Among early vaccine breakthrough infections (Groups 3 and 4), the vulnerable group (Group 3) is the majority of the breakthroughs, despite being only 7% of the total population. We note that initial observed VEP can even be negative. This occurs because the early infections in the unvaccinated are spread across the vulnerable and immunocompetent population, while the early infections in the vaccinated are predominantly from the poorly protected vulnerable group. As VES wanes in the immunocompetent population, the immunocompetent population (Group 4) makes up a larger fraction of vaccine breakthrough infections. Given that the immunocompetent population is well-protected against progression to severe disease, observed VEP increases. During the study, depletion of the susceptible population in the vulnerable/vaccinated (Group 3) also occurs, contributing to the changing balance over time.

In the previous example, VES wanes in the immunocompetent population, leading to more vaccine breakthroughs. We assessed whether a similar phenomenon could be observed with constant VES in the immunocompetent population. Figure 4 summarizes an example otherwise identical to that in Figure 3 except that is VES is constant in the immunocompetent population (α4=0.03 at all times). With constant VES in the immunocompetent population, VEP can still appear to rise over time, yet the increase is attenuated compared to the previous waning VES scenario. We note faster depletion of the susceptible population in all subgroups except the vaccinated immunocompetent population (Group 4), so the vaccinated immunocompetent population represents a higher proportion of the vaccinated breakthrough infections as the study progresses. To further explore the impact of depletion of the susceptible population on our findings, we present variations of the default setting with two-fold higher and two-fold lower baseline force-of-infection in Supplemental Figure 1. We note that the observed value of VEP at 6 months is slightly higher for the scenarios with higher force-of-infection, indicating the depletion of susceptibles phenomenon described above.

To explore the sensitivity, we vary the parameter settings in Figure 5. In Figure 5, Panel A, we consider three additional VES waning speeds. Across all settings, observed VEP increases over time, yet the dynamic range varies across settings. In Figure 5, Panel B, we vary the fraction of the population that is vulnerable. We observe the greatest dynamic range and largest rise in observed VEP for the higher fractions studied. When a greater proportion of the population is vulnerable, the initial observed VEP is lower, leading to a greater rise over time. In Figure 5, Panel C, we changed β2, governing the relative risk of progression to severe disease given infection in the unvaccinated immunocompetent population compared to the unvaccinated vulnerable population. We find that this is another important parameter, with a negative initial observed VEP when β2=0.1 (unvaccinated immunocompetent population has one tenth the baseline risk of progression to severe disease of the unvaccinated vulnerable population). Thus, the rise in observed VEP is greater when there is a wider difference in the baseline risk of progression to severe disease given infection between the vulnerability groups.

In Figure 5, Panel D, we varied α4, which governs the vaccine protection against infection in the immunocompetent. We demonstrate the importance of high VES (i.e., low α4), at least initially, to observe a large rise in observed VEP. Where initial VES is modest, the entire trajectory is flatter because there are more infections within the vaccinated immunocompetent population at all time points, including month one. In Figure 5, Panel E, we varied β4, which governs vaccine protection against progression to severe disease (as relative to β2). We consider values of β4=0.25 and 0.50, implying immunocompetent population VEP of 0.50 and 0, respectively. This change affects both the starting and ending point of the trajectory, where the ending point is approximately the immunocompetent population VEP. Finally, in Figure 5, Panel F, we considered settings where VEP in the immunocompetent population wanes over time. This is modeled by β4 approaching β2 (risk in unvaccinated immunocompetent population) over time. We find that VEP can appear to decline if VEP wanes sufficiently fast, and then only after an initial rise. If underlying waning is slow, the entire trajectory can appear to rise.

Figure 6 summarizes results from simulations that include within-subgroup heterogeneity in VES and VEP. In general, the observed rise occurs in VEP when there is clear separation between the groups such that the vulnerable group has low VES and VEP, while the immunocompetent group has high VES and VEP. As the group means become more similar and the extent of overlap increases, the pattern lessens until the curve is nearly flat. The bimodal structure of the data with a latent subgroup induces a marginal correlation structure, but in additional simulations, we generate perfect within-subgroup correlation. We find that the added correlation slightly increases the dynamic range of the increase (see Supplemental Figure 2).

Finally, we explore the role of high initial VES in the immunocompetent group for our findings in Figure 7. In a setting where the vaccinated immunocompetent have high protection (α4 and β4 have mean 0.05) and the vaccinated vulnerable have some protection (α3 and β3 have mean 0.7) (Figure 7, Panel B), we note an increase in observed VEP over time, though starting from a higher initial value (Figure 7, Panel E). In contrast, when we reverse the pattern so VES among the immunocompetent is lower (α4 and β4 have mean 0.3) and the vaccinated vulnerable have very little protection (α3 and β3 have mean 0.95) (Figure 7, Panel C), the observed VEP curve is nearly flat (Figure 7, Panel F).

We describe a phenomenon by which vaccine effectiveness against progression to severe disease given infection appears to increase over time as an artifact of population-level heterogeneity in vaccine protection. This work was motivated by observational studies of COVID-19 vaccines, where a similar pattern is implied by waning of protection against infection (or any disease) versus more durable protection against severe disease. To reproduce this phenomenon using a mathematical modeling framework, we found fundamental features to be an immunocompetent population that is at least initially well-protected against infection and progression to severe disease, contrasted with a poorly protected vulnerable subpopulation. The existence of these two subpopulations induces a correlation between VES and VEP, such that individuals with low protection against infection also have low protection against progression to severe disease. This correlation is biologically sensible as a higher acquired level of overall immunity can translate to higher levels of both mucosal immunity (i.e., prevention of infection) and systemic immunity (i.e., progression to severe disease). As a result, the earliest vaccine breakthrough infections are predominantly those with the poorest protection, leading to a low initial estimate of VEP.VEP then appears to increase as more breakthrough infections occur in the well-protected immunocompetent population, either due to waning of protection against infection or depletion of the vulnerable vaccinated population [11].

Driving these results is the underlying challenge that VEP is based on the post-vaccination outcome of infection, and the infected vaccinated are not directly comparable to the infected unvaccinated because some infections have been averted by the vaccine [11,12]. This fundamental issue of conditioning on a post-infection outcome has been recognized for comparing HIV viral load across vaccinated and unvaccinated cases in vaccine trials [13]. A related issue also appeared in studies of SARS-CoV-2 immune imprinting, as individuals who had a prior vaccine breakthrough infection may be more vulnerable in other regards [14]. A novel feature here is in the added dimension of time since vaccination. Waning VES and the differential acquisition of immunity across subgroups change the composition of the vaccine breakthrough infections, shifting the weighting of the population VEP estimate from the vulnerable towards the immunocompetent.

Through a series of sensitivity analyses, we identified settings where this rise in observed VEP was greatest. First, there is a high level of protection against infection VES, even if short-lived (e.g., a temporary boost in neutralizing antibodies). The rise in observed VEP is also greater when the size of the vulnerable group is increased. Nonetheless, we were able to observe a large rise even with only 7% of the population in this vulnerable subgroup. In practice, the proportion of the vulnerable population could be high if the study targets a more vulnerable population for enrollment, or if the vaccine generally does not work as well (i.e., greater proportion of non-responders). We balanced the proportion of the vulnerable across the vaccinated and unvaccinated groups, akin to a randomized trial, but the effect is greater when the percentage of vulnerable is higher in the vaccinated. In fact, we can entirely remove the unvaccinated, vulnerable group (Group 1) and still replicate this phenomenon. The effect is greater when the difference in baseline risk of a severe outcome between the immunocompetent and the vulnerable group is greater.

Our analysis demonstrates the existence of this phenomenon but is not meant to be an exhaustive review of all possible scenarios. There may be other mechanisms that induce this pattern in observed VEP, although we note that with our four-subpopulation structure, only the basic parameter combinations described yield results that closely resemble the experimental data. In practice, though, real-world data sets are highly heterogeneous, with more types of subpopulations and biases, and this variability may influence outcomes in a broad range of ways. Our simulations are intentionally simplified to focus on the driving features. For example, our core results adopt the strong assumption that the vaccine is ineffective for the vulnerable. We replicate the phenomenon after relaxing this assumption, but the pattern of increasing VEP is most distinct when there is wide separation in vaccine protection between the vulnerable and the immunocompetent. Another simplifying assumption is that of sterilizing immunity after infection. For future work, an alternative approach would be to model infection-induced immunity against future infection (sometimes denoted IES) as waning over time but maintaining high infection-induced immunity against progression to severe disease (sometimes denoted IEP) [15]. Yet, altering the force-of-infection in sensitivity analysis had little impact on the results, with the pattern observed even with attack rate <25% (Supplemental Figure 1A).

These findings illustrate challenges in interpreting VEP in both randomized trials and observational studies, and they provide a reason to approach such analyses cautiously. First, estimating VEP from the earliest breakthrough infections can give a very biased picture of VEP for the broader population. Second, increases in VEP over time may reflect a change in the composition of infected individuals rather than a true strengthening of protection against progression to severe disease. To detect this potential bias in real-world studies, investigators can pay close attention to the composition of vaccinated cases over time. Where this issue may be present, earliest vaccinated cases will be highly skewed towards the vulnerable but will shift over time.

Despite the limitations of VEP, the quantity has an important role for vaccine impact modeling, as well as for comparing the performance of vaccines with different mechanisms of action (e.g., intranasal and intramuscular). For COVID-19 vaccines, there is great interest in being able to characterize the role of T cells, their durability, and their impact on protection against severe disease [16,17]. Thus, better analytic strategies for quantifying protection against progression to severe disease are needed. In our analysis, we did not adjust for subpopulation (vulnerable versus immunocompetent). With covariates on immune status, subgroup-specific estimates can be generated, as in a study of VEP for the Ad26.COV2.S (Johnson & Johnson) COVID-19 vaccine to prevent progression to highly severe disease among hospitalized adults [7]. This study found that immunocompetent adults had higher protection against hospitalization and against progression to severe disease than immunocompromised adults. In the absence of data on immune status, measurable covariates (e.g. age, pre-existing conditions) can be used to form subgroups. Standardization could be applied to adjust for differences in population composition over time. Finally, causal inference methods like principal stratification can be used [12], imposing defined assumptions on who would be infected with or without vaccination to yield an interpretable estimate.