With rapid development of computing technology, Bayesian statistics have increasingly gained more attention in various areas of public health. However, the full potential of Bayesian sequential methods applied to vaccine safety surveillance has not yet been realized, despite acknowledged practical benefits and philosophical advantages of Bayesian statistics. In this paper, we describe how sequential analysis can be performed in a Bayesian paradigm in the field of vaccine safety. We compared the performance of the frequentist sequential method, specifically, Maximized Sequential Probability Ratio Test (MaxSPRT), and a Bayesian sequential method using simulations and a real world vaccine safety example. The performance is evaluated using three measurements: false positive rate, false negative rate, and average earliest time to signal. Depending on the background rate of adverse events, the Bayesian sequential method could significantly improve the false negative rate and decrease the earliest time to signal. We consider the proposed Bayesian sequential approach to be a promising alternative for vaccine safety surveillance.

Because vaccine adverse events (AE), especially rare AE, may not be detected during pre-licensure clinical trials due to limited sample size, it is crucial to continually monitor the safety of vaccines in the larger population after they are approved for use. To ensure that any unexpected elevated risks of AE are detected at the earliest possible time, sequential analyses are performed as data accumulate. For example, the Centers for Disease Control and Prevention (CDC) has routinely performed rapid-cycle sequential analyses to monitor the safety of newly approved vaccines, including seasonal influenza vaccines [

In a frequentist paradigm, when data are accumulated and analyzed sequentially, the issue of multiple testing is raised, and therefore efforts are made to find appropriate stopping boundaries to control the overall type I error rate below a pre-specified significance level, such as 0.05. Various stopping boundaries have been proposed in a group sequential setting, such as the constant Pocock [

The nature of sequential analysis, i.e. continuously updating analyses as data accumulate, is more in line with the Bayesian paradigm than with the frequentist approach because results from previous interim analyses can be used to form a new prior for the current and future analyses. With a Bayesian approach, we directly measure the probability of the parameter of interest greater or lesser than certain values or any other characteristics of the parameter (such as CI) from posterior distributions. Bayesian sequential methods have been recently adopted for monitoring clinical trials [

Our paper is organized as follows. In

For frequentist sequential methods, both group and continuous sequential methods have been proposed to monitor post-licensure vaccine safety, although disagreement exists in which method performs best [

For both group and continuous sequential methods, we can use the likelihood ratio as the test statistic.

Bayesian statistical inference is built upon Bayes’ rule, which can be expressed as

If

If the risk difference is the parameter of interest, we can use a Beta-Binomial model described in Tang et al. [

We simulated data under different scenarios with three background rates to evaluate the performance using both the frequentist (MaxSPRT) approach and the Bayesian sequential approach described above. Suppose there are three AEs following vaccination under surveillance with three different background rates. We set the background rate of the first event as 12.7 per 1000 person-years, which converts to 2.8e-4 in an 8-day risk window. We obtained this number based on the rate of febrile seizures following varicella vaccine [

The simulation results based on 3 different background rates are shown in

With regards to the average earliest time to signal, the Bayesian method performed better regardless of the relative risk and the background rate. The difference is most prominent (32% earlier for the Bayesian method) when the background rate is higher (25.4 per 1000 person-years) and the RR is lower (1.2), which means on average 70,400 doses and 47 cases might be avoided if we use the Bayesian approach in this scenario. For the Bayesian method with a Gamma prior, with the increase of the variance of the prior, the FPR increases while the FNR and earliest time to signal decrease (

We illustrate here the Bayesian sequential analysis method applied to the real-world vaccine safety data. We also present and compare results with those from the frequentist MaxSPRT method. The increased risk of febrile seizure following the administration of influenza vaccines is of concern for children younger than 5 years old. The CDC-sponsored Vaccine Safety Datalink established surveillance activity to monitor whether influenza vaccines are positively associated with an elevated risk of febrile seizure in young children. The sequential monitoring was conducted during the 2010–2011 influenza season for children aged 6 – 59 months who received their first dose of the trivalent inactivated influenza vaccine. A statistical signal of febrile seizure was found using a frequentist approach [

In this study, we presented a Bayesian sequential approach used for continuously monitoring vaccine safety. We demonstrated how the Bayesian approach can be applied to vaccine safety surveillance in a sequential setting through both simulations and a real-world data example. We compared Bayesian and frequentist MaxSPRT results using both simulations and an example. We found the Bayesian approach can provide better performance in terms of the FNR and average earliest time to signal. On the other hand, the FPR using the Bayesian approach was slightly higher than using the MaxSPRT approach, especially when the background rate is low. Note that there is a tradeoff between the FPR and the FNR and earliest time to signal. Any single measurement does not fully assess the performance. We also need to notice that the Bayesian method has the flexibility to balance the FNR and the FPR to achieve desirable results. For example, in the above simulations, if we choose 97.5% instead of 95% credible interval as a signal reporting criterion, with a background rate of 12.7 per 1000 person-years, the FPR would be reduced to 3%, and the FNR is still much lower than its frequentist counterpart (39% vs. 62%,

In the past, one obstacle to adopting Bayesian sequential methods in vaccine safety is that extensive computer resources are required during Bayesian MCMC optimization routines [

One of the criticism in using a Bayesian approach is that subjective information is incorporated through priors and different priors may lead to different conclusions. There is much debate in the literature on whether the Bayesian method should be subjective or more objective [

Our study had several limitations. The first was that the simulation was based on only three background rates (6.35, 12.7, and 25.4 per 1000 person-years). We used 12.7 per 1000 person-years to represent febrile seizure background rate, and we then doubled it to 25.4 and also decreased by half to 6.35. Although febrile seizure is a relatively uncommon medical outcome, there are other very rare outcomes such as GBS which might be less than 1 per 100,000 person-years. Future simulation work might be needed for rare outcomes with very low background rates. Another potential limitation was that we compared the Bayesian approach with the original flat boundary MaxSPRT method. A variant MaxSPRT with a time-varying boundary was recently introduced [

The Bayesian sequential approach is an attractive alternative to the frequentist MaxSPRT method in vaccine safety surveillance. Although the FPR using the Bayesian method may be slightly higher than 0.05 with a 95% credible interval criterion depending on the background rate, the FNR is significantly decreased (e.g. from 80% to 50% or from 60% to 30%) for low relative risks, such as 1.2. In vaccine safety, we understand that high false positives can lead to additional work in terms of checking data quality and medical chart reviews, but with rare outcomes and often low relative risks, it is also important to lower the FNR so that we are able to capture most true AE signals in our routine surveillance. We favor using Bayesian sequential methods in vaccine safety surveillance because of following three main benefits: 1) the Bayesian approach provides full posterior distribution(s) of the parameter(s) of interest instead of only point estimates and test statistics, which means more information can be obtained from the posterior distribution, including interval estimates; 2) the philosophical awkwardness or the dilemma of whether the surveillance should continue once the stopping boundary is exceeded can be avoided in a Bayesian paradigm. In our perspective, it is extremely important to continue the surveillance as long as new data continue to be accrued; 3) the Bayesian method can model any parameter (such as risk difference) or any algebraic formulation of parameters, not restricted to only relative risk. In addition, it can directly incorporate confounders in the model, while in the frequentist MaxSPRT method, confounding variables can only be stratified and included through baseline value estimation. However, since the Bayesian sequential approach does not specify a Type I error rate, it may not be suitable for clinical trial analysis wherein a strict false positive rate is required.

A stopping rule using highest density interval (HDI). The gray area shows the probability of the parameter value in the interval of

A stopping rule using highest density interval (HDI). The gray area shows the probability of the parameter value in the interval of

Impact of FPR, FNR and average earliest time to signal by variances in a Gamma prior for different scenarios with RR=1.2 and background rate (BR)=12.7, 25.4, and 6.35 per 1000 person-years. The unit for time is the number of analysis time points.

Posterior distribution curves of relative risks of febrile seizure following influenza vaccines during 2010–2011 season (the analysis signaled in week 11).

Comparing false positive rates (FPR), false negative rates (FNR) and earliest time to signal using the Bayesian sequential method with a 95% credible interval as the signal criterion and the frequentist MaxSPRT method when the background rate is 12.7 per 1000 person-years.

FPR | FNR | Earliest time to signal | ||||||
---|---|---|---|---|---|---|---|---|

| ||||||||

Methods | Variance | RR=1 | RR=1.2 | RR=1.5 | RR=2 | RR=1.2 | RR=1.5 | RR=2 |

| ||||||||

Bayesian | 0.1 | 0.055 | 0.294 | 0.001 | 0 | 5.082 | 2.525 | 1.280 |

0.5 | 0.077 | 0.262 | 0.001 | 0 | 4.614 | 2.233 | 1.177 | |

1.0 | 0.091 | 0.259 | 0.001 | 0 | 4.556 | 2.193 | 1.163 | |

2.0 | 0.093 | 0.257 | 0.001 | 0 | 4.490 | 2.165 | 1.158 | |

5.0 | 0.096 | 0.258 | 0.003 | 0 | 4.443 | 2.148 | 1.157 | |

MaxSPRT | 0.008 | 0.616 | 0.003 | 0 | 6.156 | 3.395 | 1.385 |

## Warning: package ‘xtable’ was built under R version 3.5.3

The unit for time is the number of analysis time points.

Comparing false positive rates (FPR), false negative rates (FNR) and earliest time to signal using the Bayesian sequential method with a 95% credible interval as the signal criterion and the frequentist MaxSPRT method when the background rate is 25.4 per 1000 person-years.

FPR | FNR | Earliest time to signal | ||||||
---|---|---|---|---|---|---|---|---|

| ||||||||

Methods | Variance | RR=1 | RR=1.2 | RR=1.5 | RR=2 | RR=1.2 | RR=1.5 | RR=2 |

| ||||||||

Bayesian | 0.1 | 0.087 | 0.060 | 0 | 0 | 4.122 | 1.557 | 1.010 |

0.5 | 0.095 | 0.056 | 0 | 0 | 3.847 | 1.437 | 1.007 | |

1.0 | 0.103 | 0.056 | 0 | 0 | 3.847 | 1.437 | 1.007 | |

2.0 | 0.105 | 0.056 | 0 | 0 | 3.825 | 1.427 | 1.007 | |

5.0 | 0.107 | 0.055 | 0 | 0 | 3.821 | 1.423 | 1.007 | |

MaxSPRT | 0.006 | 0.286 | 0 | 0 | 5.647 | 2.041 | 1.061 |

The unit for time is the number of analysis time points.

Comparing false positive rates (FPR), false negative rates (FNR) and earliest time to signal using the Bayesian sequential method with a 95% credible interval as the signal criterion and the frequentist MaxSPRT method when the background rate is 6.35 per 1000 person-years.

FPR | FNR | Earliest time to signal | ||||||
---|---|---|---|---|---|---|---|---|

| ||||||||

Methods | Variance | RR=1 | RR=1.2 | RR=1.5 | RR=2 | RR=1.2 | RR=1.5 | RR=2 |

| ||||||||

Bayesian | 0.1 | 0.061 | 0.564 | 0.029 | 0 | 5.688 | 4.111 | 2.038 |

0.5 | 0.088 | 0.493 | 0.024 | 0 | 4.943 | 3.450 | 1.675 | |

1.0 | 0.093 | 0.483 | 0.021 | 0 | 4.799 | 3.392 | 1.622 | |

2.0 | 0.101 | 0.479 | 0.020 | 0 | 4.747 | 3.339 | 1.595 | |

5.0 | 0.104 | 0.476 | 0.019 | 0 | 4.687 | 3.308 | 1.568 | |

MaxSPRT | 0.016 | 0.789 | 0.134 | 0 | 6.095 | 4.857 | 2.206 |

The unit for time is the number of analysis time points.

Signals generated by MaxSPRT and Bayesian methods using VSD influenza vaccine safety data during 2010–2011 influenza season.

Week | Observed | Expected | Cumulative doses | LLR | Critical value | Bayes lower CI | Bayes upper CI | Signal (MaxSPRT) | Signal (Bayesian) |
---|---|---|---|---|---|---|---|---|---|

| |||||||||

week 1 | 0 | 0.0000635 | 2 | 0.000 | 3.468 | 0.141 | 3.660 | No | No |

week 2 | 0 | 0.0004074 | 10 | 0.000 | 3.468 | 0.141 | 3.659 | No | No |

week 3 | 0 | 0.0006361 | 16 | 0.000 | 3.468 | 0.141 | 3.659 | No | No |

week 4 | 0 | 0.0015739 | 48 | 0.000 | 3.468 | 0.141 | 3.657 | No | No |

week 5 | 0 | 0.0149011 | 299 | 0.000 | 3.468 | 0.139 | 3.627 | No | No |

week 6 | 0 | 0.0530657 | 955 | 0.000 | 3.468 | 0.136 | 3.544 | No | No |

week 7 | 0 | 0.1538906 | 2808 | 0.000 | 3.468 | 0.129 | 3.342 | No | No |

week 8 | 1 | 0.3755498 | 7729 | 0.355 | 3.468 | 0.345 | 3.888 | No | No |

week 9 | 2 | 0.8393673 | 17822 | 0.576 | 3.468 | 0.308 | 3.492 | No | No |

week 10 | 3 | 1.6540956 | 35663 | 0.440 | 3.468 | 0.455 | 3.103 | No | No |

week 11 | 7 | 2.5440721 | 54904 | 2.629 | 3.468 | 1.042 | 3.834 | No | Yes |

week 12 | 9 | 3.3651772 | 72542 | 3.219 | 3.468 | 1.153 | 3.727 | No | Yes |

week 13 | 15 | 4.1043193 | 88592 | 8.544 | 3.468 | 1.791 | 4.577 | Yes | Yes |

week 14 | 17 | 4.7263061 | 101862 | 9.487 | 3.468 | 1.860 | 4.513 | Yes | Yes |

week 15 | 18 | 5.4139540 | 116501 | 9.039 | 3.468 | 1.790 | 4.245 | Yes | Yes |

week 16 | 21 | 5.9908531 | 128847 | 11.331 | 3.468 | 1.969 | 4.400 | Yes | Yes |

week 17 | 23 | 6.3116437 | 135672 | 13.053 | 3.468 | 2.094 | 4.525 | Yes | Yes |

week 18 | 24 | 6.7721527 | 145642 | 13.138 | 3.468 | 2.076 | 4.419 | Yes | Yes |

week 19 | 25 | 7.1411842 | 153628 | 13.466 | 3.468 | 2.082 | 4.368 | Yes | Yes |

week 20 | 25 | 7.4691285 | 160766 | 12.672 | 3.468 | 2.007 | 4.211 | Yes | Yes |

week 21 | 26 | 7.6449998 | 164584 | 13.470 | 3.468 | 2.057 | 4.255 | Yes | Yes |

week 22 | 26 | 7.8169554 | 168370 | 13.064 | 3.468 | 2.020 | 4.177 | Yes | Yes |

week 23 | 26 | 8.0741988 | 173899 | 12.479 | 3.468 | 1.966 | 4.066 | Yes | Yes |

week 24 | 26 | 8.3038506 | 178849 | 11.980 | 3.468 | 1.921 | 3.972 | Yes | Yes |

week 25 | 26 | 8.4930151 | 182989 | 11.583 | 3.468 | 1.885 | 3.898 | Yes | Yes |

week 26 | 26 | 8.6873295 | 187172 | 11.189 | 3.468 | 1.849 | 3.824 | Yes | Yes |

week 27 | 26 | 8.8479430 | 190705 | 10.874 | 3.468 | 1.821 | 3.766 | Yes | Yes |

Comparing false positive rates (FPR), false negative rates (FNR) and earliest time to signal using the Bayesian sequential method with a 97.5% credible interval as the signal criterion and the frequentist MaxSPRT method when the background rate is 12.7 per 1000 person-years.

FPR | FNR | Earliest time to signal | ||||||
---|---|---|---|---|---|---|---|---|

| ||||||||

Methods | Variance | RR=1 | RR=1.2 | RR=1.5 | RR=2 | RR=1.2 | RR=1.5 | RR=2 |

| ||||||||

Bayesian | 0.1 | 0.030 | 0.393 | 0.001 | 0.000 | 5.662 | 2.904 | 1.396 |

0.5 | 0.040 | 0.362 | 0.001 | 0.000 | 5.141 | 2.556 | 1.249 | |

1.0 | 0.041 | 0.360 | 0.001 | 0.000 | 5.023 | 2.503 | 1.228 | |

2.0 | 0.044 | 0.360 | 0.001 | 0.000 | 4.967 | 2.466 | 1.216 | |

5.0 | 0.045 | 0.359 | 0.001 | 0.000 | 4.941 | 2.440 | 1.208 | |

MaxSPRT | 0.008 | 0.616 | 0.003 | 0.000 | 6.156 | 3.395 | 1.385 |

The unit for time is the number of analysis time points.