Generalized Linear Mixed Model (GLMM) has been widely used in small area estimation for health indicators. Bayesian estimation is usually used to construct statistical intervals, however, its computational intensity is a big challenge for large complex surveys. Frequentist approaches, such as bootstrapping, and Monte Carlo (MC) simulation, are also applied but not evaluated in terms of the interval magnitude, width, and the computational time consumed. The 2013 Florida Behavioral Risk Factor Surveillance System data was used as a case study. County-level estimated prevalence of three health-related outcomes was obtained through a GLMM; and their 95% confidence intervals (CIs) were generated from bootstrapping and MC simulation. The intervals were compared to 95% credential intervals through a hierarchial Bayesian model. The results showed that 95% CIs for county-level estimates of each outcome by using MC simulation were similar to the 95% credible intervals generated by Bayesian estimation and were the most computationally efficient. It could be a viable option for constructing statistical intervals for small area estimation in public health practice.

A variety of model-based small area estimation (SAE) methods have been developed and applied to health survey data to generate estimates of health-related outcomes for small geographic areas in recent years [

A credible interval can be drawn directly by Hierarchical Bayesian estimation. In a hierarchical Bayesian model, the unknown parameter is treated as a random variable by giving it a probability distribution and a prior distribution. The posterior distribution of the parameter could be simulated through Markov Chain Monte Carlo (MCMC) samples and consequently a posterior distribution of small area estimate is produced. However, this approach is computationally intensive for large datasets with complex data structures, such as the nationwide Behavioral Risk Factor Surveillance System (BRFSS). In the frequentist paradigm, bootstrapping is a common approach for statistical inference purposes when the true distribution of the statistic of interest is unknown. It has been used to approximate the distribution of the small area estimates [

BRFSS is a common survey used in small area estimation for indicators of chronic diseases, health-related behaviors, and health preventive services. An appropriate approach for constructing statistical intervals using BRFSS can help health agencies or local health departments optimatize their capacity and understand how reliable the estimate is. This study is designed to compare 95% statistical intervals for small area estimates by different approaches using Florida 2013 BRFSS data, which had large sample sizes in all 67 counties. GLMMs combining unit- and area-level covariates were constructed to generate both state- and county-level estimates

The 2013 Florida BRFSS was a cross-sectional survey data and a part of the nationwide BRFSS (

Let

_{ij}: COPD that was answered as yes or no by respondent

_{ij} = 1) : the probability that the respondent has COPD.

_{i} : _{i} is the row of respondent

_{j} : the random effect for county

We used hierarchical Bayesian estimation and frequentist approaches, respectively, to estimate the parameters and simulate their distributions as below. All the analyses were implemented in SAS 9.4 (SAS Institute, Cary, NC).

Model (_{county(j)} as ~

For each iteration, parameter estimates (^{th} population category in county

_{k} is the row of population category

With _{j} is the population in county _{kj} is the population in the ^{th} category of county

By repeating (

Model (

The idea behind this approach is to use a random number process to create repeated samples of ^{th} and 97.5^{th} values) were determined.

The difference of this approach with non-parametric bootstrapping is the source of the bootstrap samples. In non-parametric bootstrapping, the bootstrap samples were drawn from the original 2013 Florida BRFSS data; while in parametric bootstrapping, bootstrap data were drawn from the model fitted to 2013 Florida BRFSS data. We adopted Zhang

Step 1. Model (

Step 2. A random sample of ^{th} and 97.5^{th} values for

Non-parametric bootstrapping is a resampling technique to estimate statistics (means, medians, SEs, and percentiles) by sampling the original dataset with replacement. The bootstrap sample usually has the same size as the original dataset. Specific steps of non-parametric bootstrapping in this study follow:

Step 1. PROC SURVEYSELECT in SAS was used to resample2013 Florida BRFSS data (

Step 2. We put all ^{th} and 97.5^{th} values.

The performance of frequentist approaches was evaluated based on how their estimates and 95% CIs were close to estimates and 95% credible intervals generated from Bayesian estimation as well as how much their computation time was consumed.

The mean estimates and 95% statistical intervals for each of the health-related outcomes at the state level are presented in

In this study, we generated point estimates for each of the selected outcomes and simulated their distributions via an SAE application in a health survey. As expected, we observed similar mean estimates for each outcome but different intervals across all the approaches at both state and county levels. The method of MC simulation and non-parametric bootstrapping yielded the closest 95% CIs to credible intervals by Bayesian estimation for all the selected outcomes, but MC simulation was much more computationally efficient than the others.

To generate a proper statistical interval, an approach needs to account for three sources of uncertainty: the residual variance, the uncertainty in the fixed effects parameter estimation, and the uncertainty in the variance parameters for the random effects. In full Bayesian analysis, one uses probability distributions (prior distribution and data likelihood) to model the credibility of possible parameter values. The outcome of Bayesian analysis, the posterior, models the probability of each possible parameter value being true given the prior and likelihood [

In the framework of model-based estimation, parametric bootstraps for linear mixed models have been introduced to estimate mean squared error (MSE) by Laird and Louis [

For example, the U.S. CDC’s PLACES Project (

This study is subject to several limitations. It should be noted that the mechanisms of Bayesian and frequent approaches are inherently different. Therefore, the intervals generated by hierarchical Bayesian models simply play a role of “benchmark value” for comparison purpose. Second, this study only focused on the sampling approaches, but there are some other techniques to create statistical intervals. Third, as we run into an “out of computational memory” issue when we attempted to use nationwide BRFSS as an illustration, we had to select a subset of BRFSS.

Statistical intervals may help local health departments identify if different counties have different sources and needs, or if different sub-population groups in the small areas are equally exposed to a particular disease. Different approaches may produce different intervals as shown in this study. Given their comparison with the Bayesian estimation and their computational performance, the MC simulation approach produced reasonable CIs for multilevel model-based small area estimates but was much simpler to implement and could be applied as a suitable option for public health practice.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention, or U.S. Bureau of Labor Statistics.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

The mean estimates and 95% credible incidence (Bayesian estimation) and 95% CIs (other approaches) for COPD in 67 counties, Florida.

The mean estimates and 95% credible incidence (Bayesian estimation) and 95% CIs (other approaches) for binge drinking in 67 counties, Florida.

The mean estimates and 95% credible incidence (Bayesian estimation) and 95% CIs (other approaches) for arthritis in 67 counties, Florida.

The means and 95% statistical intervals of the state-level estimates (%) for each outcome by different approaches for Florida, 2013.

COPD | Binge drinking | Arthritis | ||||
---|---|---|---|---|---|---|

Mean | 95% interval^{1} | Mean | 95% interval^{1} | Mean | 95% interval^{1} | |

Bayesian estimation, model-based | 7.2 | 6.9, 7.6 | 15.5 | 14.9, 16.2 | 25.7 | 25.1, 26.3 |

MC simulation, model-based | 7.5 | 7.1, 7.8 | 15.7 | 15.1, 16.4 | 25.8 | 25.2, 26.4 |

Parametric bootstrapping, model-based | 7.4 | 7.1, 7.8 | 15.6 | 14.8, 16.0 | 26.0 | 25.5, 26.6 |

Non-parametric bootstrapping, model-based | 7.4 | 6.8, 7.5 | 15.3 | 15.0, 16.3 | 25.5 | 24.9, 26.1 |

Direct survey estimate | 7.4 | 6.9, 7.9 | 15.6 | 14.6, 16.6 | 26.0 | 25.1, 26.9 |

95% intervals are credible incidence (Bayesian estimation) or confidence intervals (other approaches).

Total computational time (seconds) of different approaches for each of the outcomes.

Model construction | Post-stratification | |
---|---|---|

Bayesian estimation | 10,800 | 23,400 |

MC simulation | 12 | 960 |

Parametric bootstrapping | 1800 | 42 |

Non-parametric bootstrapping | 1800 | 42 |