Unmeasured confounding is a major concern in pharmacoepidemiologic studies in older adult populations. If not accounted for, unmeasured confounding may lead to profoundly misleading causal conclusions. For example, influenza typically accounts for fewer than 10% of all deaths among the elderly each winter, but observational studies have reported that influenza vaccination reduces all-cause winter mortality among the elderly by up to 50%, clearly demonstrating substantial bias^1,2. Many sources of bias may contribute to unmeasured confounding. A well-documented source is healthy-user bias: users of a preventive service are more likely than nonusers to engage in other health-promoting behaviors that lead to better health outcomes. In observational studies of hormone replacement therapy (HRT) on cardiovascular diseases for women during menopause, researchers found that HRT users tend to be higher educated, have a better risk factor profile, and do more leisure exercise than nonusers pre-menopause, suggesting at least part of the estimated benefits of HRT may have been due to unadjusted pre-existing characteristics³. Adherence bias is source of unmeasured confounding bias, which is caused by the association between adherence to the treatment and the health outcome. In a meta-analysis to study drug adherence and mortality, good adherence to placebo was found to be associated with lower mortality in eight studies (pooled odds ratio: 0.56; 95% CI: 0.50–0.63), indicating that the estimated treatment effect may generally be biased if one does not account for different adherence between treatment arms⁴. Other forms of potential unmeasured confounding bias include prevention bias, survivor bias, and so on⁵. These biases can be difficult to control, especially when using clinical health care data sources such as electronic health records (EHR)⁶.

Negative control methods have emerged as a popular way to detect unmeasured confounding, but the newer negative control methods for reducing and correcting for unmeasured confounding are less used^7,8. These methods utilize measured proxies of unmeasured confounders: 1) a negative control outcome (NCO) that is a priori known not to be causally affected by the exposure of interest, and 2) a negative control exposure (NCE) that is a priori known not to causally affect the outcome of interest. NCOs and NCEs are selected such that they are subject to the same source of unmeasured confounding as the primary outcome and exposure. As such, conditional on measured covariates, an observed association between the NCO and the exposure or between the NCE and the outcome may indicate the presence of unmeasured confounding bias, which is sometimes referred to as a “falsification test” or “bias indicator analysis”^9–15. As an example of negative control, consider a study of influenza vaccine effectiveness against mortality in older adults. To quantify potential confounding bias, researchers used all-cause mortality during the period prior to influenza season (i.e., a time period when influenza is not yet circulating and so no effect of vaccine can exist) as the NCO. The large size of the estimated risk ratio (RR=0.39, 95% CI 0.33–0.47 before the influenza season) suggested that confounding bias may have impacted the vaccine effectiveness estimates of the primary outcome¹⁵. However, the type of “bias indicator analysis” does not answer all questions about unmeasured confounding, but only whether unmeasured confounding is present. For example, if the primary analysis found a significant association, but the bias indicator analysis revealed the existence of confounding, it is unclear what the true effect size is and whether it remains the same direction once the bias is corrected. On the other hand, if the primary analysis did not find a significant association and the bias indicator analysis revealed presence of unmeasured confounding, the sign and magnitude of the true effect size also remains unclear. Therefore, it is important to not only detect the unmeasured confounding, but also account for the confounding in effect estimation.

Recombinant Zoster Vaccine (RZV) is a 2-dose vaccine for preventing herpes zoster and its complications (Shingrix^®, GlaxoSmithKline)^16,17. Post-licensure observational studies have examined RZV safety among adults aged 50 and above^18,19. However, the previous analyses did not systematically address potential unmeasured confounding.

In this article, we will showcase the double negative control approach, which leverages both NCEs and NCOs to correct the unmeasured confounding bias^7,20,21. We will re-analyze the EHR data from a prior RZV safety study¹⁸, re-calculate the relative risk estimates for the associations between RZV and selected safety outcomes and confidence intervals using the double negative control method, and compare them to a propensity score adjusted regression method used in the prior study.

Table 2 details baseline characteristics of RZV recipients, well-visit comparators, and well-visit + flu shot comparators. A total of 650,701 participants received at least one dose of RZV in the two-year study period. There were 1,869,067 well-visits indexed in the study period and 1,089,165 well-visits for which the study participant had also received an influenza vaccine in the prior year. RZV recipients were, on average, slightly older than well-visit comparators. They were more likely to seek preventive care such as vaccinations, dermatology visits, and optometry/ophthalmology visits. They were more likely to have a record of comorbid conditions such as hypertension and hyperlipidemia, within the past year. The distribution of gender was similar across groups, although a slightly higher proportion of women received RZV compared to men. As hypothesized, the prevalence of many healthy user indicators (prior Zoster vaccination, prior pneumonia vaccination, optometry/ophthalmology visits in the past year, and dermatology visits in the past year) of the well-visit + flu shot comparator group fell in between those of the RZV recipients and the well-visit comparators. The proportion of participants from each MCO site in the well-visit + flu shot comparator group was approximately the same as in the well-visit group, suggesting that influenza vaccination rates were relatively consistent across sites.

Table 3 shows RR estimates and confidence intervals for associations between RZV receipt and the four target safety outcomes for both the propensity score-adjusted approach and the double negative control method. Using the well-visit comparator group, the propensity score-adjusted analyses estimate the RR for acute MI, stroke, gout, and pneumonia to be 0.78 (95% CI: 0.70, 0.90), 0.83 (95% CI: 0.71, 0.96), 1.10 (95% CI: 1.03, 1.16) and 0.81 (95% CI: 0.71, 0.90), respectively. This suggests a potential protective effect of RZV on acute MI, stroke, and pneumonia, and a potential harmful effect on gout. The results of the double negative control method are similar, with the estimated RRs for acute MI, stroke, gout, and pneumonia to be 0.83 (95% CI: 0.72, 0.96), 0.89 (95% CI: 0.76, 1.04), 1.09 (95% CI: 1.02, 1.16) and 0.90 (95% CI: 0.80, 0.99), respectively. Notably, the negative association of RZV with stroke is not statistically significant. In general, the double negative control approach yielded comparable or slightly attenuated RR estimates, validating the conclusions of the previous analysis.

We present the results of the sensitivity analyses for the impact of different comparator groups and negative control pairs in Appendix S5 of the Web Materials. The choice of the well-visit comparators versus the well-visit + flu shot comparators did not remarkably alter our conclusion. For example, the propensity score adjusted logistic regression estimated the RR for acute MI to be 0.80 (95% CI: 0.68, 0.94) while the double negative control method estimated the RR to be 0.84 (95% CI: 0.71, 1.00). For acute MI and pneumonia, the choices of different negative control outcome variables mostly produced similar point estimates of RR but some negative control pairs may produce wider confidence intervals which covered one, likely due to increased standard errors. For example, with cataract as NCE and epistaxis as NCO, the estimated RR for acute MI is 0.84 (95% CI: 0.68, 1.06).

Unmeasured confounding is a key concern for observational studies of preventive therapies in older adult populations conducted using EHR data, where data have been gathered for the purposes of clinical care and where confounders are not proactively measured for research purposes. We demonstrate a novel method for quantifying some sources of unmeasured confounding bias and explicitly correct for this bias in their estimation efforts, including computing confidence intervals that account for the additional uncertainty that unmeasured confounding introduces. Compared with the previously proposed single negative control counterparts, which often rely on rank preservation or equi-confounding assumptions that can rarely be justified in practice^33,34, the assumptions for this newer double negative control approach are mild and can be often verified by prior domain knowledge. Using data from a prior RZV post-licensure safety study¹⁸, we found that the results of traditional covariate adjustment methods mostly align with our double negative control method for the four safety outcomes under consideration, suggesting that the presence of potential unmeasured confounding may not significantly alter the causal conclusions. Our analysis demonstrates a principled approach and exemplifies the necessary steps for detecting and correcting confounding bias for future post-market vaccine effectiveness and safety evaluation.

We hypothesized that the results of the analyses with the double negative control method versus the propensity score adjusted regression method would be more similar when using the well-visit + flu shot comparator group than when using the well-visit comparator group. This is because the well-visit + flu shot comparators may have exhibited health-seeking behavior more similar to the RZV group in the original RZV safety study, thus potentially reducing the amount of unmeasured confounding. Our results in the Appendix S5 of the Supplementary Material supported this hypothesis; estimates and confidence intervals were more similar between the double negative control analysis and the flexible propensity score regression analysis when well-visit + flu shot comparators were used than well-visit comparators.

A major advantage of the double negative control method is that it allows correction for unmeasured confounding between two independent groups, making the use of concurrent comparators a much more viable option. In vaccine safety surveillance studies, vaccinated and unvaccinated individuals are often fundamentally different, necessitating the use of self-controlled design or historical comparators, which have other important known weaknesses^35–41. Limitations of self-controlled study designs include susceptibility to bias when event rates are not constant within the study period⁴⁰. When using historical comparators or self-controlled design, temporal trends in the target outcome, or in diagnostic or coding practices may lead to false signaling or failure to identify a true safety signal^35,40. Using a concurrent comparator group avoids risk of temporal bias.

Our analysis has several limitations. The proposed approach requires correct specification of the causal diagram. A wrongly specified DAG may cause biased or numerically unstable causal effect estimates. As such, extensive input from domain experts and a careful evaluation of causal assumptions are needed to ensure the correctness of the causal DAG. We used a data-driven approach to choose appropriate double negative controls for a given outcome and exposure of interest. However, the ideal choice of negative control may not always be clear in practice. Our sensitivity analysis that used multiple different negative control pairs to estimate a particular vaccine-outcome association demonstrated that the choice of double negative control can influence the estimated comparative risk statistics from the analysis, although from our sensitivity analysis, use of different negative control variables did not lead to a reversal of an observed decrease in risk to an observed increase in risk or vice versa; the only difference was that different choices of negative control variable pairs may result in effect size estimates of varying standard errors. It is also possible that the low NCE or NCO prevalence may limit the power of detecting an association. Future simulation study research that evaluates use of the double negative control methodology under a variety of scenarios related to the choice of the negative control pair or the use of multiple negative control variables is important to gain further understanding of the method’s properties. The double negative control method requires both NCE and NCO to be associated with the unmeasured confounder and can only correct bias to the extent of the association. In other words, if there exist other unmeasured confounders which are not associated with the NCE or NCO, bias due to those unmeasured confounders may remain. In Appendix S3.2, we exemplified a case where the selected negative control variables failed to capture the bias introduced by the concerned unmeasured confounder. In such cases, the negative control method may not produce an estimator, or the resulting estimator may be numerically unstable, which further highlights the importance of domain experts’ input to ensure the correctness of the proposed causal DAG. Additional model misspecification bias may arise in our method. We used a logistic regression model for P(W = 1|Z = z, X = x, C) and compatible regression models for P(W = 1 | C), but further reducing the bias is possible through nonparametric regression. Negative control methods have been developed to incorporate more flexible estimation for the nuisance functions while maintaining valid inference, which require different statistical assumptions and extensive statistical programing⁴². As this manuscript aims to demonstrate the application of negative control methods in epidemiological studies, we do not pursue those directions. In our final analysis, we used a single NCE and NCO pair, but as demonstrated in Appendix S4, NC variables with multiple categories or multiple NC variables may be used to capture different facets of the unmeasured confounding. Selection of multiple negative control variables, however, is notably more challenging and shall be left for future research. Other limitations of the data that may also compromise the validity of our conclusions, including incomplete data capture due to delays in receipt of some insurance claims by MCO systems, the potential misclassification of safety outcomes, and the potential misclassification of vaccination status^35–41.