The federal government allocates substantial resources each year for collection of state and national data to monitor trends and changes in the health of the U.S. population. Scientists, health care professionals, and policy makers use such data to understand current and emerging trends in public health, to provide a basis for the establishment and evaluation of health policies and programs, and to assess where best to apply limited public health resources. It is, therefore, imperative for health statistics and other population estimates obtained from these surveys to be of the highest possible quality.

The Behavioral Risk Factor Surveillance System (BRFSS), the world's largest ongoing random-digit–dialed (RDD) telephone survey, is conducted by the health departments in the 50 states as well as those in the District of Columbia, Puerto Rico, Guam, and the Virgin Islands, with assistance from the Centers for Disease Control and Prevention (CDC) (1). Estimates obtained from the BRFSS are based on sound methods for conducting surveys and performing statistical analyses; comparison of these estimates with those from other national surveys is important for measuring the validity and reliability of the estimates. Such comparisons are especially important because RDD surveys are facing an increasing number of newly emerging operational challenges.

The rate of response to telephone surveys has traditionally served as a proxy indicator of the survey's data quality. Link et al point out that telephone survey response rates have recently declined (2). BRFSS response rates declined between 1993 and 2006 (Figure). Several researchers have presented convincing arguments on the negative correlation between rate of nonresponse and survey data quality (3-5). RDD surveys face a unique and growing problem: an increasing number of people rely on cell phones as the only means of telephone communication. Most RDD surveys traditionally sample only from landline telephone numbers (6). Also of concern is the change in BRFSS sampling methods. As of 2004, all zero-listed telephone banks (sets of telephone numbers that include only unlisted or nonresidential numbers) were eliminated from the sampling frame to increase efficiency; this change has increased undercoverage of the population by up to 2% (7).

This study compares a number of key estimates obtained from BRFSS with related estimates from the National Health Interview Survey (NHIS) and the National Health and Nutrition Examination Survey (NHANES). These surveys have structural differences in design protocols, modes of data collection, and postsurvey procedures for data adjustment. NHIS and NHANES also have higher rates of participation than BRFSS. A comparison of the estimates from the 3 surveys can render valuable insights into the quality of the estimates. The estimates included in these comparisons are related to smoking prevalence, alcohol consumption, medical conditions, vaccination, health status, insurance coverage, cost barriers to medical care, testing for human immunodeficiency virus (HIV), and body measurements (height and weight).

Gentry and colleagues compared estimates of chronic drinking obtained from the National Institute on Alcoholism and Alcohol Abuse and estimates of current smoking obtained from NHIS with BRFSS estimates and concluded that their similarity was statistically significant (8). A more comprehensive study by Nelson et al examined the comparability of national estimates from the 1997 BRFSS and NHIS (9). Nelson et al concluded that the 2 surveys provided similar estimates for most of the overall measures examined. This research reexamines these findings after nearly a decade and makes new comparisons with estimates from NHANES and NHIS.

The data show varying levels of agreement between the 3 surveys (Tables 1–4). BRFSS and NHIS overall estimates were statistically similar for 5 of the 15 items examined: current smoker, diabetes, BMI, average number of alcoholic drinks per occasion, and no health insurance. BRFSS and NHANES estimates were statistically similar for 1 of the 6 items analyzed: self-reported height. BRFSS and NHIS were significantly different for 10 items; for 9 of these, BRFSS estimates were significantly higher than NHIS estimates. Only 1 BRFSS estimate (for binge drinking) was significantly lower than the NHIS estimate. In contrast, BRFSS estimates were lower than NHANES estimates for 4 of the 5 items for which differences were found; diabetes was the sole exception.

The magnitude of differences varied widely. For example, the relative difference between BRFSS and NHIS estimates of height is less than 1% (derived by dividing the difference [0.14 inches] between the BRFSS and NHIS overall point estimates by the NHIS overall point estimate [66.96 inches]). For weight, the relative difference is 1.2% (2.13%/170.90%). The relative difference in overall estimates of ever having smoked a cigarette is 4.0% (1.68%/42.35%) and of having received an influenza vaccination in the past 12 months, 4.6% (2.95%/64.63%). At the other end of the spectrum, however, the relative differences in BRFSS and NHIS estimates are 94.4% (7.35%/7.79%) for experiencing cost barriers to medical care, 35.1% (3.48%/9.91%) for having asthma, 33.9% (4.16%/12.28%) for reporting poor or fair health, 26.4% (9.15%/34.61%) for having ever had an HIV test, and 11.5% (6.54%/56.84%) for having ever had a pneumonia vaccination. In contrast, the estimate for binge drinking is 8.3% (0.39 drinks/4.70 drinks) lower for BRFSS than for NHIS.

As a relative percentage of the NHANES estimates, BRFSS estimates show less variation. For diabetes, BRFSS estimates are 31.3% (1.92%/6.14%) higher, but they are 12.2% (2.89%/23.60%) lower for current smoking, 8.3% (3.96%/47.99%) lower for ever having smoked a cigarette, 2.1% lower for BMI (0.58/27.55), and 2.0% (3.55 lb/176.58 lb) lower for self-reported weight.

BRFSS and NHIS estimates for diabetes prevalence were similar for all 12 population subgroups. The following tabulation shows the number of subgroups for which BRFSS and NHIS estimates were statistically similar:

The estimates for fair or poor health status, asthma, having been tested for HIV, and not receiving medical care because of cost were significantly different between all subgroups examined.

Similarity between BRFSS and NHANES estimates among the 12 subgroups was greatest for self-reported height (9 subgroups), followed by diabetes (8 subgroups), current smoker (6 subgroups), self-reported weight (5 subgroups), BMI (4 subgroups), and ever having smoked a cigarette (2 subgroups).

The estimates provided by the 3 surveys for the 4 demographic characteristics (age, sex, race/ethnicity, and education) are not entirely consistent despite being indexed to population totals; estimates for educational attainment differ most notably.

Some of the estimates from the 2004 BRFSS, NHIS, and the 2003–2004 NHANES vary overall and within certain subgroups. While a number of these differences are small and may not warrant more examinations, other differences — especially those that reflect a relative difference of more than 20% — may require further investigations. The BRFSS estimates tended to be higher than NHIS estimates but lower than NHANES estimates. For the 6 items examined in all 3 surveys, 1 of the BRFSS estimates (self-reported height) was identical to NHANES, 2 fell between those of NHIS and NHANES (ever smoke cigarettes and self-reported weight), and 3 were statistically identical to NHIS (current smoking status, diabetes, and BMI).

The extent of similarity between the 2004 BRFSS and NHIS estimates was less than what was reported by Nelson and colleagues (9) for the 1997 comparisons. They found statistically significant differences between overall estimates for 6 of 14 items examined; BRFSS estimates were lower for current smoking, height, and BMI and higher for pneumococcal vaccination, cost as a barrier to medical care, and reporting fair or poor health.

Several explanations may account for the inconsistencies

 BRFSS estimates and estimates from the other 2 surveys, including 1) differences in questionnaire wording, 2) mode of survey administration, 3) sampling design differences and related postsurvey adjustments such as weighting procedures, and 4) use of incentives or proxy data.

Differences in questionnaire wording represent some of the greatest challenges in comparing survey results. The wording of the BRFSS and NHIS questionnaires was identical for only 4 of the items examined (ever smoked cigarettes, ever had a pneumonia vaccination, health status, and asthma) and for 1 of the NHANES items (ever smoked cigarettes). Some questions that seemed to be measuring the same concept differed in ways that may have led to differences in their final estimates. For example, the questions used to assess binge drinking are dissimilar in wording, time frame, and response categories. BRFSS asks, "Considering all types of alcoholic beverages, how many times during the past 30 days did you have 5 or more drinks on an occasion?"; NHIS asks, "In the past year, on how many days did you have 5 or more drinks of any alcoholic beverages?" Although both surveys attempt to measure the same concept, differences in wording and time frame could explain the small but statistically significant differences in estimates that we observed.

Even with identical wording, however, different modes of survey administration can produce varying results and magnify differences in measurement (13,14). BRFSS is a telephone survey, whereas NHIS and NHANES are conducted face-to-face. For BRFSS the interaction is aural and limited to voice communication, whereas NHIS and NHANES rely on both aural and visual cues. Such differences in the patterns of interviewer–respondent communication and situational contexts may explain some of the variation in survey estimates.

Differences in weighting adjustment to compensate for design-imposed differential selection probabilities may also explain some of the observed differences in survey estimates. The comparison of weighted estimates for basic demographic characteristics, such as race/ethnicity and education, shows that even when weights are applied, differences remain between survey respondents. These differences may lead, in part, to some of the inconsistencies in health and risk estimates between the 3 surveys. In addition, the 3 surveys use different sources to obtain population control totals for weight calibrations. BRFSS uses population estimates from Claritas (Claritas Inc, San Diego, California), a private data vendor that uses census projections as part of its process for developing yearly population estimates, whereas NHIS uses census projections and NHANES relies on the Current Population Survey estimates. These sources for population control totals vary in their analysis of the U.S. population by sex, age, and race/ethnicity. In addition, the weighting procedures for both NHIS and NHANES account for differences in the race/ethnicity distribution in each state, whereas BRFSS includes this type of adjustment in selected states only. Education was the socioeconomic indicator that differed most markedly between the 3 surveys. BRFSS is revising its weighting methodology to account for educational differences; the new weight will be applied to public release data sets in 2010 (15).

Other factors, perhaps minor, may also contribute to differences in survey estimates. For instance, BRFSS and NHIS do not offer any incentives for survey participation, but NHANES relies on incentives to secure higher rates of response. The use of incentives may alter a participant's survey response or attract a different mix of participants (16). Also, differences between BRFSS and NHIS estimates may be affected by the use of proxy data in NHIS. Although BRFSS does not allow proxy respondents, some NHIS estimates (e.g., cost as a barrier to medical care, health status) can be based on proxy data. BRFSS and NHIS provide significantly different estimates for both of these measures. Hays et al report that agreement between self and proxy responses are often good for measures of function that are directly observable but poor for subjective measures (17). Moreover, Nelson and colleagues suggest that proxy-derived data are reliable for demographic and body measures but are less reliable for questions on medications and alcohol consumption (18).

Although the many differences in design, procedures, and context of these surveys may confound our ability to single out the proximal causes of differences in survey estimates noted in this research, the comparison of estimates for similar concepts is important. These periodic comparisons are particularly important because BRFSS, NHIS, and NHANES are critical components of the U.S. public health system, with each providing essential data for policy makers, researchers, and the public alike. The periodic examination of when and why measurements of similar concepts vary is equally important.

These research findings also need to be interpreted in light of tangible implications. Although many of the estimates examined here differed statistically, their programmatic implications and impact on actionable policy are subject to debate. What constitutes a "significant" difference is ultimately up to the people who use the data. Perhaps the good news for BRFSS is that despite declines in survey participation rates during the past decade, BRFSS estimates do not appear to be radically different from similar estimates produced by NHIS and NHANES. In fact, for the 6 common items in these surveys, BRFSS estimates were either similar to NHIS or NHANES estimates or were found between the estimates of these 2 surveys.