Vaccines are considered one of the most important public health successes of the last century. They have led to significant reductions in morbidity and mortality from many infectious diseases [1]. The success of vaccination programs depends not only on vaccines’ effectiveness, but also their safety. As vaccine-preventable diseases become increasingly unusual, the public becomes less familiar with these diseases and consequently focuses more intently on vaccine safety [2]. Widespread concerns about the occurrence of adverse events can lead to a loss of confidence in the safety of vaccines, lower vaccination rates and resurgence in vaccine-preventable diseases [2,3].

The safety of vaccines is assessed through rigorous clinical trials before they are licensed. However, clinical trials’ primary focus is on efficacy; they generally lack adequate sample size and they may also have insufficient follow-up time to identify rare adverse events or those with delayed onset [4]. Further, inclusion in prelicensure trials is typically limited to healthy individuals, and trials often specifically exclude specific vulnerable sub-populations, such as pregnant women, for whom a vaccine may be indicated. Thus, monitoring vaccines once they are used in the general population is required to detect rare adverse reactions, those that may occur long after vaccination, and those that may affect specific sub-populations.

In the U.S., programs to monitor vaccine safety began in the late 1970s. Established by CDC in 1978 and continuing until 1990, the Monitoring System for Adverse Events Following Immunizations (MSAEFI) collected reports from the parents or guardians of children who received publicly funded vaccines concerning adverse events following immunization [5]. The National Childhood Vaccine Injury Act (NCVIA) of 1986 was landmark legislation that established The National Vaccine Program Office, the Vaccine Injury Compensation Program, and the Vaccine Adverse Event Reporting System (VAERS); VAERS replaced MSAEFI in 1990 [6]. VAERS is co-managed by CDC and the Food and Drug Administration (FDA). VAERS reports are accepted from any reporter; including vaccine manufacturers, immunization programs, health care providers, patients, parents, and others [6]. This nationwide system allows for timely detection of possible vaccine safety problems. However, VAERS reporting is voluntary. Thus, VAERS has important limitations [6], including underreporting and incomplete reports. VAERS usually cannot be used to determine whether an adverse event is caused by a vaccine or is simply coincidental. Rates or relative risks of vaccine adverse events also cannot be determined because the number of people vaccinated in the population (i.e. the denominator) and reporting rates are unknown, and there are no unvaccinated comparison groups in VAERS data.

Recognizing the need for a flexible, timely and robust system to evaluate vaccine safety and supplement information provided by VAERS, CDC established the Vaccine Safety Datalink (VSD) in 1990 to conduct post-marketing vaccine safety evaluations in defined populations [3,7]. As a collaboration between CDC and several large health care organizations, VSD conducts population-based monitoring and research on important immunization safety questions. VSD provides scientific data to healthcare providers, public health officials, and others to inform national immunization policy, and helps to ensure that the public has the best available timely information regarding the safety of immunization.

Beginning as a collaboration between CDC and four health care organizations (i.e., sites), encompassing 6 million persons, to study the safety of childhood immunizations, VSD has expanded to include 9 sites that can evaluate vaccine safety for all age groups (Fig. 1). Working with CDC investigators as a multidisciplinary team, site scientific investigators provide clinical, methodological, and data expertise. The leadership and vision of this collaboration has played an essential role in the US vaccine safety monitoring enterprise and fostered innovation in the development of databases and methods to monitor and evaluate vaccine safety. VSD has systematically built the capacity to address a wide array of safety issues, including monitoring new vaccines in children and adults, and has become recognized as a model system for conducting timely vaccine safety evaluations. The aggregate population across all sites is sufficiently large so that risk can be assessed for rare adverse events. VSD has cumulative information on more than 21 million individuals who have collectively received over 134 million vaccine doses. Data from approximately 9.3 million individuals are available annually, including 2.1 million children and 7.2 million adults (Table 1).

Although VSD comprises a population of over 9 million people annually, the size of the population may not be adequate to evaluate extremely rare outcomes (e.g., Stevens Johnson Syndrome). This may be most concerning during situations of a mass vaccination campaign in which a large number of vaccinations are administered in a short period of time and there is priority in identifying potential safety problems as soon as possible. Seasonal influenza vaccination is administered in nearly a mass campaign paradigm. In VSD, approximately 3.8 million doses of influenza vaccine are administered each year, with 80–90% administered during September through November. In this situation, we have estimated that for a rare condition such as Guillain-Barré Syndrome (GBS) that has a background incidence of 1 per 100,000 people per year, VSD could detect an increased risk of 1 per million within 10 weeks of the start of vaccination.

VSD conducts studies to test hypotheses regarding vaccine-related adverse events and to identify safety signals using near real-time monitoring. To achieve these goals, VSD requires that participating sites: (1) maintain computerized data bases of healthcare encounters, including computerized immunization registries with detailed information on vaccines administered; (2) have the capability to access written or electronic medical records and other data sources to provide detailed information on specific healthcare encounters; and (3) provide integrated healthcare services to their members so that the full spectrum of healthcare from outpatient clinic and emergency department (ED) visits to hospitalizations can be captured. In addition, each site has scientists with expertise in vaccine safety, statistical analysis, and data management.

With over 20 years of data collected in a standardized format, VSD has the unique ability to conduct timely vaccine safety studies, including assessments of rare adverse events and longitudinal studies involving prolonged follow-up of individual patients. VSD has innovated and continues to innovate in linking data files to address questions, while protecting patient confidentiality. A recent priority of VSD has been the establishment of a “pregnancy platform” to effectively monitor and conduct targeted research on the safety of vaccinations given during pregnancy (e.g., influenza, Tdap) identifying potential adverse outcomes in both pregnant women and their offspring. The ability to conduct long term follow up of birth cohorts over multiple years in VSD was also effectively demonstrated in VSD’s evaluation of thimerosal exposure in early life and neuropsychological outcomes 7–10 years later [8].

A strength of VSD is the collaboration of the site investigators in the analyses: when data are combined, investigators/staff at each site contribute their expertise in understanding and interpreting their own data, which enables each study to incorporate the differences among sites in diagnosis, referral, and coding practices, among other potential differences. In addition to using automated data, VSD is able to capture data from other data sources, such as medical charts and patient interviews. Chart review is important to validate diagnostic codes in the automated data, which may be poorly predictive of specific medical conditions (e.g., positive predictive value [PPV] of narcolepsy is very low compared to intus-susception which has a high PPV), and to obtain additional clinical details and risk factors for specific health outcomes.

VSD has evolved and fostered innovation in the databases and methods used to monitor and evaluate vaccine safety. At its inception, VSD facilitated the development of the first computerized immunization tracking systems at some of the participating sites. In its early years, few VSD sites had computerized outpatient clinic data. With the increased utilization of computerized data bases in large health care organizations over the years VSD sites built electronic capacity, and all sites currently have computerized immunization databases, as well as computerized data on outpatient clinic and ED visits and hospitalizations. The sites’ electronic health records provide a rich source of information, as they are generated from patient care records and include initial diagnoses and test results as well as claims data.

Each site prepares a standardized set of files that contain individual-level data, including demographic information, health plan enrollment, birth information, vaccination records, hospitalizations, outpatient visits, ED visits, urgent care visits, and other data (Table 2). To ensure confidentiality, each site retains its own data. Each member receives a unique, randomized VSD identification number that is linked to their health plan member identification number. This link is used only to gather data for the files and is not otherwise available. The VSD identification numbers are used to link data across the various databases. A standardized data dictionary ensures consistency across sites. Data from medical records are frequently used to validate the electronic clinical diagnoses and vaccination data, and these are occasionally supplemented with surveys and in-person interviews. All studies must meet both local and CDC Institutional Review Board (1RB) and Health Insurance Portability and Accountability Act (H1PAA) requirements.

Since its inception, the VSD has led in the development of novel methods for conducting prospective vaccine safety surveillance. During its first decade, VSD used a centralized data processing model. CDC received de-identified data files to merge into a centralized database for analyses. However, in 2001, improvements in technology and heightened awareness of confidentiality and security led the VSD to implement the first vaccine safety application of the distributed data model. This approach allows each site to control, assemble and maintain its data files on its own secure server rather than sending them to CDC [7]. The creation of the distributed data model allowed VSD to reinvent the way sites create and use data files. Instead of the annually updated “cycle files” with the distributed data model, VSD created “dynamic data files” in addition to cycle files. Dynamic data files allow for the continuous capturing of information, including vaccinations, hospitalizations, clinic and ED visits, health plan enrollment, and certain demographic characteristics. Most dynamic data files are updated on a weekly basis. Consequently, this has greatly reduced the time lag until the data are available for analysis. CDC can access the dynamic data files through the distributed data model on an ongoing basis, thus allowing for near real-time analysis and extraction of data for ongoing studies, making it possible to investigate immediate safety concerns.

Increasingly, policy makers need timely information to develop recommendations and provide practical guidance to the public about new potential safety concerns or to provide rapid, early safety evaluation of recently licensed vaccines, including annual seasonal influenza vaccines. To provide timely data in such situations, VSD researchers developed “rapid cycle analysis” (RCA) as a complement to traditional retrospective studies, which could take years to complete [9,10]. The RCA process is outlined in Fig. 2. As of 2013, VSD had conducted 18 RCAs; this monitoring has included seasonal influenza vaccines [11,12], rotavirus vaccine [13,14], human papillomavirus (HPV) vaccine [15], and other recently introduced or recommended combination vaccines. The value of RCA for identifying vaccine-associated adverse events has been shown for MMRV vaccine among children aged 12–23 months [16]. During routine weekly monitoring, a preliminary signal was detected for an approximately twofold increased risk of febrile seizures occurring 7–10 days following MMRV vaccination when compared to separately administered MMR and varicella vaccination. These findings, together with nearly identical results from a manufacturer-sponsored study based on different methodology, were presented to the CDC’s Advisory Committee on Immunization Practices (AC1P), which changed its recommendations from a stated preference for MMRV to expressing no preference for MMRV or separate MMR plus varicella vaccination [17]. AC1P also recommended that healthcare providers advise parents of this increased risk of fever and seizures when using MMRV vaccine in young children.

Another important use of RCA was for monitoring the safety of the 2009 pandemic H1N1 influenza vaccine. 1n October 2009, once the H1N1 influenza vaccine became available, VSD initiated RCA monitoring of the vaccine [18]. This VSD effort occurred simultaneously and in close consultation with other DHHS-led efforts and the VSD protocol was shared with other organizations that had H1N1 influenza vaccine safety surveillance systems, including the Department of Defense, the 1ndian Health Service, FDA’s newly developed Post-Licensure Rapid 1mmunization Safety Monitoring (PR1SM) system [19], and others [20]. VSD’s important leadership role in safety surveillance and support of decision making around H1N1 vaccine was evidenced by its biweekly data reporting to the Vaccine Safety Risk Assessment Working Group which was established by the National Vaccine Advisory Committee with the charge to conduct independent, rapid reviews of available safety monitoring data for the 2009 H1N1 influenza vaccines. The initial VSD RCA findings, along with a review of VAERS reports that identified no safety signals, were published on December 4, 2009, providing the public early reassuring data on the safety of the new vaccine [21].

VSD has pioneered development of appropriate statistical methods to evaluate safety signals and minimize the chance of having false-positive signals.

With RCA, VSD researchers evaluate the safety of a vaccine on a weekly basis, performing multiple analyses over time. The use of traditional statistical methods in this situation, however, may generate false positive signals. To address this problem, VSD researchers developed the maximized sequential probability ratio test (MaxSPRT). This new signal detection method accounts for the repeated statistical testing, supports continuous or time-period analysis of data as they are collected (e.g., MaxSPRT has been applied weekly within VSD) and provides a useful and highly adaptable approach to early detection of adverse events after the introduction of new vaccines [22,23]. Using this method, for example, VSD investigators identified, earlier than would have been possible with previous methods, a new safety signal of febrile seizures associated with 2010–2011 seasonal influenza vaccination [12].

VSD investigators have also recently successfully adapted methods used in clinical trials known as group sequential analysis to prospectively monitor the safety of new vaccines. Using the group sequential monitoring approach, compared to continuous testing (e.g., MaxSPRT), less frequent testing is conducted (e.g., at time points based on the number of doses administered required to ensure power to detect a pre-specified relative risk) and this has the advantage of yielding increased overall study power for a given sample size, which may be very important for detecting rare adverse events that were not detected in pre-licensure studies. This method was used to prospectively monitor the safety of the newly introduced DTaP-IPV-Hib and PCV13 vaccines in children during their uptake within VSD [24,25]. These studies not only demonstrated the application of this novel method to observational safety data, but also provided reassurance about the safety of these new vaccines.

Another novel analytic design developed by VSD investigators is the case-centered approach for observational vaccine safety studies. This method uses a “backward” approach, where the observed odds of exposure (e.g., immunization) during a certain period of time (i.e., the risk interval) prior to the onset of an outcome (e.g., adverse event) are compared with the expected odds of exposure during the same risk interval, based on vaccination times in the population of similar vaccinees. This is similar to a matched case-control design which uses all potential controls. This method has been used in VSD studies of febrile seizures after MMRV vaccine [16], Bell’s palsy following influenza vaccine [26], and Guillain–Barré Syndrome following vaccinations [27].

MaxSPRT, group sequential analysis, and case-centered analysis have served as a model for other types of safety monitoring (e.g., for drugs and medical devices used in the U.S.), and for vaccine safety monitoring efforts in other countries.

The resources and capabilities of VSD have enabled investigators both at the sites and at CDC to conduct a large variety of studies employing traditional epidemiologic methods that require accurate data on vaccinations and other exposures, along with complete capture of health outcomes over time. In addition, VSD can be used to study both acute and chronic conditions with insidious onset. For example, VSD studies of prenatal and infant exposure to thimerosal from vaccines and autism [28], and other neurodevelopmental outcomes [8], found no evidence that thimerosal was associated with autism.

VSD also has the ability to study special populations, such as premature infants and pregnant women. With recent vaccination policies specifically targeting pregnant women for certain vaccines (e.g., influenza, Tdap), VSD has been at the forefront of studying the safety of vaccinations administered during pregnancy. The VSD collects information on an estimated 125,000 pregnant women annually and has been able to successfully utilize multi-site electronic healthcare data to identify pregnant women, to ascertain the gestational stage of their pregnancy, and to accurately link babies to their mothers. Recently, VSD has provided reassuring data on the safety of influenza vaccine in pregnant women [29], and studies are ongoing to evaluate TdaP vaccine safety in this population.

In recent years there has been an increase in the number of recommended vaccines included in the childhood immunization schedule with resultant interest in addressing the safety of multiple vaccinations and the immunization schedule in general [30]. Although policy interventions, such as immunization requirements for school entry have resulted in high overall immunization coverage in the U.S. one recent VSD study has found evidence of marked variability in patterns of adherence (i.e., 1399 distinct vaccination timing patterns among 323,247 children at VSD sites) [31]. This variability has allowed VSD to conduct initial investigations of the safety of vaccines administered according to different schedules. For example, VSD studies have shown an increased risk of seizures following measles containing vaccines when they are delayed later into the second year of life [32,33], and a substantially increased risk of pertussis in children with pertussis containing vaccines administered later than the recommended schedule [34].

VSD also has been a highly effective platform for conducting rigorous vaccine safety studies in other priority areas, including:

VSD has conducted important scientific studies (Table 3) to assess the safety of vaccines once they are available for use in the United States. Research findings have been published in leading peer-reviewed journals; have been presented at regional, national, and international scientific conferences; and have informed the deliberations of the 1nstitute of Medicine (1OM) and ACIP, that advise on immunization policy [58]. Because of the importance of public confidence in the data CDC uses, CDC, National Center for Health Statistics (NCHS), and the VSD sites have created novel, secure public-use data sets for selected completed VSD studies and established procedures for outside investigators to access VSD data under IRB-approved protocols [59,60]. The VSD further ensures transparency around its findings through sharing safety data in public sessions, particularly at the meetings of ACIP and similar meetings with a broad range of stakeholders.

VSD has been highly successful in capitalizing on the databases and the scientific resources of its sites to conduct innovative and timely assessments of vaccine safety. However, electronic healthcare databases are developed for medical care and administrative purposes and their use for research can pose challenges. Thus, VSD has developed innovative strategies to evaluate the databases to be utilized, linking and compiling them according to a standardized data dictionary, and performing rigorous quality checks of the data. Careful selection of computerized codes is needed to identify potential cases of each health outcome, to avoid misclassification bias that can lead to false positive or false negative findings. Moreover, review of individual medical records (either hardcopy or electronic) is often critical for validating potential cases identified based on computerized codes.

Health encounter data will only identify those health outcomes that come to medical attention. Since the VSD population covers insured members of healthcare organizations, lack of access to medical care is not a large concern. However, factors that can influence seeking medical care must be kept in mind and addressed in any evaluation conducted in VSD. Likelihood of seeking medical care particularly depends on the severity of a health condition. Thus, more severe conditions that lead to hospitalizations are more likely to be captured in VSD than less severe conditions for which a person may or may not seek medical care. The VSD network, however, can serve as an infrastructure for conducting special studies for conditions that are unlikely to be accurately captured in an electronic health record. For example, VSD evaluated neurodevelopmental outcomes following infant vaccinations with a follow-up study and in-person assessment of cohorts of children who had been exposed to different vaccines as infants, conducting neuropsychological tests to assess level of functioning on several neurodevelopmental domains [8].

Full ascertainment of vaccination and vaccination coverage can also pose challenges. Since vaccinations are a covered benefit of the VSD health plans, there is incentive to receive vaccinations within the health plan, and overall vaccine coverage rates are higher than the national coverage estimates for the routine infant and childhood vaccines [61]. About 20–30% of adult influenza vaccinations, however, have been found to be administered in nontraditional settings (e.g., pharmacies) outside the health plan and are not captured in the VSD immunization tracking systems [62]. VSD analyses are often restricted to vaccinated individuals, mainly to address possible biases related to missing vaccinations from outside the health plan, as well as “confounding by indication” (e.g., influenza vaccine is more likely to be administered to patients with certain underlying high risk medical conditions) [63].

Another analytical challenge arises from having a highly vaccinated population, which can lead to difficulties in identifying an unvaccinated comparison group. 1n these situations, as well as in analyses restricted to vaccinated individuals, various “risk-interval” designs are used [64]. Risk interval methods are suitable for acute conditions that tend to occur within a limited time period after vaccination. The rate of occurrence of the outcome of interest during the risk window is compared with the rate in time periods outside the risk window. Self-control methods have built upon the risk interval concept by further restricting analyses to vaccinated cases of the outcome of interest [65]. By restricting to vaccinated cases, self-control methods inherently control for any individual-level potential confounding factor (whether measured or not) that does not vary over time. Although risk-interval methods are well suited to address acute adverse events, studies of health outcomes with delayed or insidious onset can be more challenging.

Since the VSD membership population is comprised largely of employed individuals (and their families) with health insurance coverage, questions of generalizability of VSD findings often arise. Although the extremes of income distribution may be under-represented, the VSD membership populations have been found to have similar demographic characteristics to the catchment areas served by the VSD health plans [66].

Since its inception, VSD has been a major contributor to immunization decision making. VSD supports the public health mission of CDC and the vaccine safety system through optimizing the ability of policy makers to make decisions or to revisit recommendations as new data arise. As a consequence of vaccines’ success in reducing vaccine-preventable diseases, focus has shifted to the perceived risks of vaccination in some parents’ minds. Vaccine refusal has been increasing in the U.S. and has caused outbreaks in some communities [2]. Thus, the role of VSD in monitoring and ensuring vaccine safety has become critically important.

The VSD’s dynamic infrastructure has evolved and adapted to changes in medical care organization, electronic database diffusion and enhancements, and methodological innovations. Looking ahead, we anticipate new opportunities will continue to arise for VSD to improve its scope, accuracy and timeliness. Maturing of electronic health records and the capability to link records across data systems (such as insurance claims databases and immunization registries) may make additional sources of patient data available for post licensure epidemiologic evaluations of vaccine safety.

Although diffusion of electronic healthcare records and linkages of large health insurance databases may provide substantial increases in the quantity of electronic data available for vaccine safety monitoring, VSD’s experience over its first 20 years indicates that data quality is perhaps more important than quantity of data for the conduct of scientifically sound assessments of vaccine safety. Key to the quality of VSD data has been the ability to readily access individual patient records and other detailed clinical information to validate computerized diagnostic codes and obtain important information on clinical details and patient characteristics. With the universal adoption of electronic health records at all the VSD sites, future advances in electronic text mining may provide applications that could improve the speed and efficiency of reviewing individual patient records. Most importantly, VSD investigators have considerable experience using their health plan data and many of the investigators are clinicians that practice in the health plans, providing a practical perspective on the strengths and limitations of the data.

VSD is a longstanding vaccine safety research network which over the more than 20 years of its existence has been a defining force in the area of safety surveillance, not only for vaccines but also for other healthcare products, including prescription drugs. Its scientific leadership and influence have been widely acknowledged in both the U.S. and around the world. VSD has been a model for development of the FDA’s post-marketing surveillance Sentinel Network [67], in which all FDA-regulated medical products, including vaccines, drugs and medical devices are monitored. Other nations, including a European multi-national effort [68,69], have adapted systems and methodologies modeled on the VSD (e.g., common data dictionaries, the distributed data model and analytic methods such as near real-time sequential monitoring) in their vaccination safety efforts. VSD has been successful by capitalizing on the scientific, organizational and data resources provided by large integrated healthcare delivery systems and adapting to changes in medical care structure and advances in health information technology. The leadership and experience of VSD investigators and the many strengths of VSD provide compelling evidence of its importance as part of the overall U.S. vaccination program and as an influential resource providing vaccine stakeholders with the best possible information so they can make the best possible decisions. VSD can be expected to continue to evolve along with continuing changes in healthcare organization, further computerization of healthcare records and databases, and advances in analytical methods.