Selection bias in population-based cancer research can affect the validity of the evidence used for public health practice (1). In epidemiologic case–control studies, cases and controls should arise from the same study base (2–5). However, the nature of case–control studies may lead to different sampling frames being used for cases and controls. For example, cases may arise from physician reports in a geographic region, whereas controls are selected from some type of list of the population in the same region. This is the situation when a statewide cancer registry is used to sample cases and a state-provided master file of licensed drivers is used to sample controls. Such studies make the assumption that coverage of the study base is comparable for both the case and control sampling frames. If the study base is all residents in a geographical area, then the case sampling frame needs to include all incident cases in that area, and the control sampling frame needs to include all residents in the same area. Bias will arise if sampling frame coverage of the study base is associated differently with risk factors among cases and controls. Inherently, there is no association of coverage with risk factors among cases, when a registry is nearly complete. If, however, no complete sampling frame exists for controls, bias will then arise if coverage of the control sampling frame is associated with risk factors. The potential for bias varies by exposure, sampling frame, and geographical setting requiring a fresh evaluation for most studies. While response rates and nonresponse error have been discussed extensively in the epidemiologic literature (6–10), the effects of the completeness of sampling frames have received little attention.

An official master file of drivers with valid licenses is available for epidemiologic studies in many states (11). Although coverage may vary, such files have historically been sufficiently complete to constitute a useful sampling frame (12–13). The largest coverage problems have resulted from (i) individuals who are not old enough to drive, (ii) those who have stopped renewing, or (iii) those who never obtained a driver’s license. More recently, legal constraints have compromised the coverage of these files. To comply with the Driver’s Privacy Protection Act (DPPA: 18 U.S.C. 2721–2725), the state of Wisconsin has instituted an "opt-out" program allowing drivers the option of keeping driver’s license data confidential and unavailable to research organizations. Using data from a population-based breast cancer case–control study and the Wisconsin Department of Transportation (WDOT),we had an opportunity to examine the effects of "opt out" due to privacy concerns affecting the sampling frame used to select population-based controls on the validity of study results.

The WDOT master file of licensed drivers for November 2006 included 3,018,192 records. This master file was linked with the 2,988 breast cancer cases that participated in the WWHS. A total of 2,026 (67.8%) of the WWHS cases were found to have a matching record on the WDOT master file. Of these 2,026 cases, 1,477 (72.9%) had an exact match based on gender, last name, first name, date of birth, and zip code. An additional 391 cases (19.3%) were matched on the basis of gender, last name, first name, and date of birth, and 66 (3.3%) were matched on gender, last name, and date of birth. Manual review, aided by weighting schemes based on those used by the National Death Index, matched 92 (4.5%) additional cases.

Table 1 shows the characteristics of breast cancer cases by presence versus absence on the sampling frame. Reference groups were chosen to represent commonly accepted groups with decreased risk of breast cancer. Case SPRs ranged from 0.71 to 1.12. Breast cancer cases found on the WDOT sampling frame were more likely to be older than 40 (SPR, 1.12) than breast cancer cases under the age of 40. A greater percentage of cases who were married, widowed, or single and never married were found on the WDOT master file than in cases who were divorced, separated, or living with a partner. Compared with cases with parity of 3 or more, all other cases were less likely to be found on the WDOT master file. Each one-child decrease in parity significantly predicted coverage on the WDOT sampling frame with a SPR of 0.96 [95% confidence interval (CI), 0.94–0.98; data not shown]. Compared with 71% of cases reporting earnings of less than $50,000, 66% of cases reporting incomes over $50,000 and 64% of cases who did not report income were linked with a record from the WDOT sampling frame (data not shown). In addition, cases reporting the use of hormone replacement therapy were less likely to be covered by the WDOT sampling frame. Oral contraceptive use, body mass index at time of diagnosis, smoking status, alcohol consumption, menopause status, age of menarche, and family history of cancer were not associated with coverage on the WDOT sampling frame (data not shown).

Awareness of selection bias in specific studies and study design types is important for both researchers and policy makers in public health. In case–control studies, selection bias may be compounded as cases and controls are often taken from different sampling frames but assumed to represent the same study base. Hence, selection is associated with disease outcome, and if this selection is also associated with specific risk factors, bias may result.

This study used breast cancer cases from an interview-based case–control study to examine coverage error of the WDOT sampling frame used to select controls from the population. This approach assumes that factors associated with coverage are similar in both case and control groups. In this database, 31% of cases (N = 926) found in the WDOT master file renewed their driver’s license after being diagnosed with cancer. For those cases, we were unable to determine their opt-out status before diagnosis. However, removing those 31% of cases did not change the factors determined to be associated with "opt out". In addition, to argue that the determinants of opting out of the WDOT sampling frame are the same in this population of cases as in the general population, one would have to assume that nonparticipants have the same determinants of opting out as the cases that took part in this study, and that these determinants have the same magnitude of association with opting out. While this may be a valid assumption, more research is needed on the cases that did not participate in the WWHS study to evaluate whether the opt-out determinants remain consistent.

This research indicates that breast cancer cases under 40 years of age are more likely to "opt out". Bias due to this finding in breast cancer research would be expected to be minimal with approximately 6% of all incident breast cancer cases occurring in women under the age of 40 (21). However, the finding may forecast a future deterioration of the coverage of this sampling frame if one assumes that the majority of respondents that have opted out will likely not opt back into the list sold by the state. Another option available to researchers would be to simply exclude those cases where bias is most likely. In this case that would be women under the age of 40. The potential for misinterpretation of results may be more serious than the loss of precision due to the smaller sample size. However, potentially informative data on these subgroups would be lost. Researchers could also use interaction effects to evaluate stratum specific estimates of bias related to opt out in these subgroups.

Of interest to breast cancer research were the findings that parity, income, and the use of hormone replacement therapy were associated with coverage on the sampling frame used to select controls. Women with higher parity, lower income, and never users of postmenopausal hormones are also less likely to develop breast cancer. Using a sampling frame that has fewer women with high parity, fewer women with high incomes, and fewer women who have ever used hormones to select controls may bias the results of a case–control study when those risk factors are included as primary exposures or as covariates. By assuming the same SPRs for cases and controls, we estimate control SPRs < 1.0 for greater parity, higher income, and use of hormone replacement therapy. Control SPRs < 1.0 would create observed ORs that are numerically higher than the truth (positive bias) when evaluating these factors in case–control studies that use driver’s license master files to identify controls. For example, although there is widespread agreement that postmenopausal estrogen–progesterone therapy is a risk factor for the development of breast cancer, an OR from a case–control study that uses the driver’s license master file to identify controls that enabled potential controls to opt out due to privacy concerns would likely be overestimated. For conditions other than breast cancer, these factors (income, postmenopausal hormone use, and parity) are also likely associated with socioeconomic variables.

One simple option for reducing selection bias may be to exclude cases not on the sampling frame used to select the controls. It is often assumed that this exclusion ensures that the study bases for cases and controls are comparable. However, this option requires one of the assumptions that we made in our investigation: that factors responsible for opting out are the same between cases and controls. Selection bias will remain even after excluding cases that could not be approached to serve as controls if this assumption is not met. In addition, the exclusion of potentially up to 32% of cases would be wasteful in terms of statistical power and precision. Besides using SPRs to approximate the selection bias due to inadequate sampling frame coverage, a researcher could calculate the predicted probability of coverage for each control and use inverse probability weighting or propensity scores to adjust for selection bias (17, 20, 22, 23). This approach has been used in previous research to adjust for variations in nonresponse between cases and control in population-based case–control studies on cancer etiology (20). This would require the additional assumptions that the coverage model was properly specified without omitting relevant factors or interaction terms.

Investigators should evaluate the comparability of each sampling frame’s study base when designing the study. When coverage of the study base differs between cases and controls, investigators should calculate the expected direction of bias indicated by the SPR for each exposure of interest and use one of the established correction methods to adjust results, when a new exposure of interests is evaluated. A previous study (24) investigated the association of study participation with molecular exposures and concluded that, although there was no association, further research is needed. It should be noted that for selection to create bias, association with a risk factor is a necessary, but not sufficient, condition for bias, as the SPR can still be equal to 1. For SPRs not to equal 1, participation also needs to be associated with disease outcome conditionally on the risk factor, which can occur due to different sampling frames. Bias is less likely in a case–control study nested in a cohort study as, barring selective drop out, disease outcome occurs postenrollment. A similar argument that opt out is likely unrelated to case–control status, as it mostly occurred before cancer diagnosis, is used in our estimation approach.

Also, this study assumed that the case sampling frame, the Wisconsin Cancer Reporting System, was 100% complete. This allows for a simplified calculation of SPRs based on setting α and β to 1. There appears to be regional variation in reporting due to privacy concerns of neighboring states (15). Additional research could focus on the effects of this variation.

This study and linkage procedures have some limitations. In addition to the assumptions discussed earlier, some errors in linkage between WWHS cases and the WDOT master list of licensed drivers may have occurred. Linkage procedures could partially explain the observed association between marital status and coverage. The first 3 linkage procedures used last name and address when merging the driver’s license list with WWHS case data. Last name and/or address often change when marital status changes. However, these data are normally updated when a license is renewed (every 7 years in Wisconsin). Also, linkages focusing on date of birth, aided by various weighting schemes, were evaluated to reduce misclassification. The WWHS parent study obtained a master list of licensed drivers in 2004 and 2006. However, due to cost constraints only the 2006 master file was prepared for linkage. Cases that were interviewed in 2004 may have moved away, passed away or had a disease progression that would explain absence from the 2006 master list of licensed drivers. However, the 2-year emigration and mortality rates are low for these women. Additional linkage errors probably resulted in nondifferential misclassification bias, resulting in attenuated SPRs.

Our results indicate the potential for selection bias due to differential opt out between various demographic and behavioral subgroups of controls. The potential for bias due to inadequate coverage of the study base will increase if more individuals opt out of inclusion in common sampling frames used in cancer research. All current control ascertainment schemes, including random digit dialing, have coverage issues, and as response rates and participation rates decline, understanding the effects of sampling frames that do not fully enumerate the study base will be critical to establishing the validity of study results.