Nonresponse bias occurs when there is a systematic difference between respondents and nonparticipants to a survey or questionnaire. While reporting response rates has become common practice for researchers to demonstrate the quality of the data collected, it is a flawed measure and does not address the potential effect of nonresponse bias (Fowler et al., 2002; Groves, 2006; Johnson and Wislar, 2012; Davern, 2013; Halbesleben and Whitman, 2013). The assumption that high response rates are indicative of a high quality study or, conversely, that a low response rate suggests nonresponse bias is incorrect. Even with high response rates, respondents are not necessarily representative of the larger population. Rather, nonresponse bias is a function of both the response rate and the mean difference between respondents and nonparticipants: X¯Respondents=X¯total+PNR(X¯Respondents-X¯Nonparticipants) where X̄_Respondents is the mean value of the outcome of interest among respondents, X̄_total is the true mean value for the entire population, P_NR is the proportion of nonresponse (i.e., 1-response rate), and X̄_{Nonparticipants} is the mean value of the outcome among nonparticipants (Groves, 2006; Halbesleben and Whitman, 2013). The latter part of the equation (P_NR X̄_Respondents–X̄_{Nonparticipants})) represents the bias introduced by nonresponse. While the mean outcome value for nonparticipants can never truly be known (if it was, they would be respondents), there are multiple strategies to estimate the potential impact of nonresponse bias. Halbesleben and Whitman (2013) recently outlined and compared the strengths and limitations of multiple nonresponse bias assessment methods.

As survey response rates continue to decline in all areas of research, including birth defects, nonresponse bias assessment is becoming increasingly critical to show the potential impact of the missing data (Groves and Peytcheva, 2008; Davern, 2013). This study illustrates a nonresponse bias sensitivity assessment using a survey on the prevalence of perceived barriers to care for children with orofacial clefts (OFCs) in North Carolina (Cassell et al., 2012, 2013, 2014).

Initially, 475 eligible subjects were identified through the NCBDMP, and 160 (33.7%) responded to a mail survey. Of the remaining 315 subjects, 115 (36.5%) had valid phone numbers that could be traced; the remaining 205 could not be located and were considered lost to follow-up. Among those contacted by means of telephone, 85 (73.9%) completed the phone survey. Overall, 245 (51.6%) individuals participated in the study.

Compared with the initial mail respondents, both follow-up phone respondents and nonparticipants were more likely to be younger, less educated at the time of birth, and non-white (Table 1). Nonparticipants were also more likely to be Hispanic and unmarried at the time of birth compared with mail respondents. Phone respondents were more likely to have a child with a cleft palate, compared with mail respondents. No other significant differences in child characteristics were observed between phone respondents or nonparticipants when compared with mail respondents. When compared with follow-up phone respondents, nonparticipants were more likely to have only elementary and some high school education at the time of birth and be Hispanic (Table 1). Differences in the age distribution of the child were also observed.

Among all survey respondents, 196 (80.0%) answered the question concerning cost as a barrier to care, and 49 (25.0%) reported cost as a barrier. Reporting cost as a barrier to care was associated with maternal education at time of birth (p = 0.03) and marital status at time of birth (p = 0.05). Of the 138 mail respondents who answered the question, 39 (28.3%) reported cost as “ever” being a barrier; 58 follow-up phone respondents answered the question, and among these, 10 (17.2%) reported cost as a barrier. Twenty-two mail respondents (13.8%) and 27 phone respondents (31.8%) respondents reported “not applicable” or left the question blank. Differences in the prevalence of reporting cost as barrier to care within maternal and child characteristics were observed between mail and phone respondents (Table 2).

The mean, median, and interquartile range, as well as minimum and maximum bias-adjusted prevalences for each assumption are shown in Figure 1. As expected, no difference was observed between the observed and mean expected prevalences when nonparticipants were assumed to have the same distribution of reporting as all survey respondents (mail and phone), 25.0%. When the overall reporting distribution was stratified by maternal age, education at time of birth, race, and marital status at time of birth, the estimated mean prevalences corrected for nonresponse were slightly lower (24.7%, 23.1%, 24.7%, and 23.4%, respectively) than the observed overall estimate of 25.0%. When simulating nonparticipant responses using only phone respondents, the prevalence of cost as a perceived barrier was lower, 20.8%. Similarly, when phone respondents were stratified by maternal age, education at time of birth, race, and marital status at time of birth, the estimated mean prevalences were 20.8%, 21.1%, 20.8%, and 21.0%, respectively. Finally, when nonparticipants were assumed to be twice as likely or half as likely to report cost as a barrier to care, the estimated mean prevalences were 30.0% and 16.1%, respectively.

In general, participation in this survey appeared to be associated with maternal characteristics (age, education at time of birth, race, ethnicity, and marital status at time of birth), but not child characteristics. However, under our simulation scenarios, the selected, documented maternal characteristics had minimal impact on bias-adjusted prevalence estimates of cost as a barrier to care. The difference between respondent types (mail vs. phone) appeared more influential. Nonparticipants were more similar in documented characteristics to follow-up phone respondents than to initial mail respondents. When the underlying prevalence of cost as a perceived barrier to care among nonparticipants was assumed to equal that of phone respondents, as per a follow-up analysis, the estimated bias-adjusted prevalence was approximately 21%, compared with an observed 25% of respondents reporting cost as a perceived barrier to care.

In an analysis in which nonparticipants were assumed to be between half and twice as likely to report cost as a barrier to care as phone respondents, the estimated percentage for whom cost was a perceived barrier ranged from 16% to 30%. The maximal absolute difference in prevalence compared with the observed estimate was 9%, and the relative difference was 36%, under our simulation scenarios. Overall, the results obtained on cost as a perceived barrier to care do not appear to be substantially biased by nonresponse, even though the survey response rate was relatively low (51.6%) and there are no similar studies for comparison. That being said, while this assessment provides estimates of the potential magnitude of nonresponse bias, the “true” prevalence of cost as a barrier to care cannot be determined.

The strengths of this sensitivity analysis approach to nonresponse bias assessment are its flexibility, ability to use a reference group (phone respondents) to estimate nonparticipant results, and availability of demographic information on nonparticipants for stratification purposes. Because phone respondents did not to respond to the initial mail survey—and would have been nonparticipants without the follow-up phone survey—they are more likely to be representative of the nonparticipants. Even without the availability of demographic information on nonparticipants, this type of analysis can still be conducted using a more general approach (i.e., estimate bias-adjusted prevalences without stratifying across maternal characteristics). Perhaps most importantly, it requires researchers to critically consider and attempt to quantify the underlying mechanisms of potential bias in participation.

That being said, this type of assessment is dependent upon making reasonable assumptions about the underlying prevalence of the dependent variable among nonparticipants, and that the information available on all eligible subjects is associated with the outcome(s) of interest. Furthermore, if participation and perceiving cost as a barrier to care is associated with a variable not included in the available outside data, such as type of health insurance, then bias-adjusted estimates cannot be calculated using these methods. Follow-up respondents are also assumed to be representative of nonparticipants in this example; however, follow-up respondents are still respondents, and may be an inappropriate proxy. Additionally, due to the small number of telephone respondents and proportion of which did not respond to the question at hand, the results presented here should be interpreted with caution. The biggest limitation inherent in all nonresponse bias assessments is that, while they attempt to estimate the missing responses from nonparticipants, these responses cannot actually be measured. Regardless of these limitations, it is still of critical importance to attempt to estimate the potential impact of nonresponse bias on study results.

The novel nonresponse bias assessment strategy used in this study combines multiple methods of nonresponse bias assessment and can provide a more reliable picture of study implications, compared with the use of one method alone. While nonresponse bias analyses are facilitated by the use of record linkages of birth defect registries and vital records to identify eligible subjects, as they provide researchers information about nonparticipants, assessments like the one conducted in this study have rarely been done. We recommend that this type of assessment be routinely conducted in birth defect studies which use population-based surveillance data and have incomplete participation. Even if information on nonparticipants is limited or unavailable, nonresponse bias assessments can—and should—still be conducted.

It is critical to consider nonresponse bias during the study design phase, and not just after analysis, so that necessary information, like the method of survey response or response wave, is collected and recorded. Even if there is evidence of nonresponse bias, analytical techniques (such as imputation or weighting) exist to estimate the impact of the bias, although they should be performed with caution, as additional, stronger assumptions must be made when performing these adjustments (Groves, 2006; Johnson et al., 2006; Hoggatt et al., 2009). It is strongly advised that when these methods are used, all assumptions are explicitly described for the readers. That being said, nonresponse bias is just one potential source of bias when conducting studies involving survey data, and the methods presented here may not directly address bias introduced through other avenues or in combination with nonresponse bias (such as selection bias or recall bias). Generally speaking, researchers need to be thoughtful about the potential impact of nonresponse- and all bias- in their studies. The sensitivity analysis methods presented show the utility and ease of conducting a nonresponse bias sensitivity assessment, and demonstrates how birth defects research could benefit from nonresponse bias assessments.