We used data from the National Birth Defects Prevention Study (NBDPS), a population-based case-control study of birth defects.¹³ Cases (live births, still births, terminations of pregnancy) were identified from birth defects surveillance systems in 10 U.S. states (Arkansas, California, Georgia, Iowa, Massachusetts, New Jersey, New York, North Carolina, Texas, Utah) and controls (live births with no major birth defects) from birth certificates or hospital birth records in the same areas. Participating mothers were interviewed 6 weeks to 2 years after their child’s birth. Eligible mothers delivered on or after October 1, 1997 with an estimated due date on or before December 31, 2011. Study sites received institutional review board approval and participants provided informed consent.

The outcome of interest was nonsyndromic isolated CL/P; clinical geneticists reviewed medical records to exclude cases possibly caused by genetic or other syndromes.¹⁴ Isolated cases were those occurring in an infant without other major birth defects. The exposure of interest was prepregnancy BMI, dichotomized as obese (≥30 kg/m²) or non-obese (<30 kg/m²). BMI was calculated from prepregnancy weight and height, self-reported during the NBDPS interview.

We included study site, maternal race/ethnicity, and maternal education in the models as potential confounders. Only mothers reporting their race/ethnicity as non-Hispanic white, non-Hispanic black, or Hispanic were included, to correspond with available exposure misclassification validation data (details below).

Among 3,161 CL/P case mothers and 11,692 control mothers, we excluded 382 (12%) case mothers of infants with non-isolated CL/P, 199 (6%) case and 763 (7%) control mothers reporting race/ethnicities not meeting inclusion criteria, 1 (<1%) case and 7 (<1%) control mothers with missing race/ethnicity, and 56 (2%) case and 317 (3%) control mothers with missing data on maternal education. Following exclusions, we included 2,523 case mothers and 10,605 control mothers. Mothers with missing BMI were retained in the analysis so we could account for missing exposure data.

We used logistic regression to estimate crude and confounding-adjusted ORs and 95% CIs for associations between prepregnancy obesity and CL/P. All statistical analyses were conducted in SAS version 9.4 (Cary, NC).

In this analysis, we perform both non-probabilistic and probabilistic bias analysis. Probabilistic analyses take into account uncertainty in bias parameter estimates to be taken into account by conducting analyses using a range of values for the bias parameters.¹⁵ For readers wishing more detail on the bias analysis methods, the Supplemental Materials include: (1) a step-by-step worked example in which a non-probabilistic adjustment for multiple biases is conducted by hand, (2) sample SAS code, and (3) an algebraic proof demonstrating the general case. Probabilistic analysis is an extension of the non-probabilistic analysis in the worked example.

We used the method of Lyles and Lin to adjust for exposure misclassification.¹⁶ Predictive values (e.g., positive or negative predictive values) are ideally calculated from cross-tabulations of misclassified and correctly classified exposure categories in a validation dataset. Predictive values represent the probability that the prepregnancy obesity status reported by the participant (obese, not obese) is the true prepregnancy obesity status. If internal validation data are not available, use of external validation data, expert opinion, or educated guesses are other options.⁸ If only sensitivity (Se) and specificity (Sp) are available, the formulae included in the Supplemental Materials can be used to convert these to predictive values.

We had no internal validation data on exposure misclassification for NBDPS. We used external validation data from the 1999–2010 National Health and Nutrition Examination Surveys (NHANES), representative of the civilian noninstitutionalized population of the United States.¹⁷ NHANES participants self-report weight and height during an in-person interview. Later, height and weight are measured during a physical exam. All participants provided informed consent.

We restricted the NHANES analysis to nonpregnant females aged 16–49 years with height and weight measurements. We cross-tabulated self-reported and measured BMI categories conditional on race/ethnicity, accounting for the complex sampling design, to estimate Se and Sp. Although predictive values can be calculated from these data, we estimated Se and Sp to examine nondifferential and differential exposure misclassification (whether or not misclassification is differential depends directly on differences in Se and Sp between cases and controls, not predictive values). Reliable estimates from NHANES were available for non-Hispanic white, non-Hispanic black, and Mexican-American women, and therefore we restricted our NBDPS analysis to these racial/ethnic groups (because approximately two-thirds of Hispanics in the U.S. are of Mexican descent, we used the estimate for Mexican-Americans for all NBDPS Hispanic women).¹⁸

We assumed that the NHANES Se and Sp were accurate estimates of the Se and Sp in NBDPS. Not knowing if exposure misclassification was differential or nondifferential, we performed three analyses, assuming: (1) nondifferential misclassification, (2) “differential A” misclassification (classification is better for cases than controls), and (3) “differential B” misclassification (classification is better for controls than cases). In the first, we assigned cases and controls to have the same Se and Sp values (NHANES Se and Sp). In the second, we assigned the NHANES Se and Sp to controls and Se + 0.05 and Sp + 0.03 to cases. In the third, we assigned the NHANES Se and Sp to controls and Se – 0.05 and Sp – 0.03 to cases. Se and Sp were restricted to lie between 0.5 and 1.0, inclusive. We converted Se and Sp to predictive values (restricted to lie between 0 and 1, inclusive). Bias parameters were calculated separately for non-Hispanic white, non-Hispanic black, and Hispanic/Mexican-American women. For simplicity, we assumed they did not differ by other variables.

We also used predictive values to account for missing BMI.¹⁶ Among women with missing values for self-reported BMI but measured BMI in NHANES, we determined the probability that a woman with missing BMI was truly obese or not obese; these were the predictive values used in the analysis. Other missing data methods, such as multiple imputation, could also have been used.

For the analysis, we created a dataset with two observations for each participant (participant “copies”): one copy was assigned to have prepregnancy obesity, and the other to not have prepregnancy obesity — these represent the two possible obesity statuses the participant could have had in the absence of exposure misclassification.¹⁶ The assigned prepregnancy obesity status was used as the exposure in the logistic regression models (i.e., not the status reported by the participant). In the model, each participant copy was weighted by the predictive values corresponding to their assigned obesity status (the probability that the exposure assignment was the truth). Potential confounders were included as covariates; any confounder that would be included in a conventional analysis should also be included in the bias analysis. For non-probabilistic analyses, we calculated the OR only. The standard error from the logistic regression model should not be used to calculate a 95% CI because this does not take into account error introduced by estimating bias parameters.⁴

To conduct probabilistic bias analysis, we assigned triangular distributions to each predictive value using the values calculated above as the mode and +/− 0.10 of the mode as the upper and lower bounds (restricted to fall between 0 and 1). We sampled each parameter 5,000 times and calculated 5,000 ORs. The results were summarized as the median OR and 95% simulation interval (SI), the 2.5^th and 97.5^th percentile of the OR distribution.

We used the method of Lash et al. to account for random error, but other methods, such as bootstrapping, could also be used.⁴ For each bias-adjusted log OR, we multiplied the standard error from the conventional multivariable logistic regression model by a randomly selected value from a standard normal distribution. This value was then subtracted from the log OR and exponentiated. The result is presented as the median OR and 95% random error-added simulation interval (RESI), the 2.5^th and 97.5^th percentiles of the OR distribution following addition of random error.

We used inverse probability of selection weights (IPSW) to adjust for selection bias.¹⁹ The probability of selection into the study (or participation in the study) is ideally estimated from study records, but if unavailable, external validation data, expert opinion, or educated guesses can be used.⁸ The IPSW is the inverse of this probability.

NBDPS participation rates for cases and controls were 67% and 65%.¹³ We did not know to what extent these participation rates differed by BMI or to what degree eligible individuals were ascertained. For simplicity, we assumed complete ascertainment of eligible cases and controls. We used external validation data and educated guesses to estimate how participation rates differed by BMI. We found a study showing that mothers self-reporting normal weight were more likely to participate in a pregnancy study than other mothers.²⁰ (BMI is likely serving as a proxy for sociodemographic differences between women.) We assumed that NBDPS case mothers were motivated to participate regardless of BMI, but control mothers would be more susceptible to sociodemographic determinants of participation. We assigned all case mothers a selection probability of 0.67 (case participation rate), reflecting equal motivation to participate. We assigned selection probabilities of 0.60 for obese control mothers, 0.67 for non-obese control mothers, and 0.65 for control mothers with missing obesity status; the weighted average was 0.65 (control participation rate). For simplicity, we assumed that selection probabilities did not differ by other variables. The IPSW (analysis weights) were the inverse of these probabilities. For non-probabilistic analyses, we estimated the OR using multivariable logistic regression.

For the probabilistic analysis, we assigned triangular distributions to each selection probability, using +/− 0.10 of the mode as the upper and lower bounds, with values restricted to lie between 0 and 1. We selected 5,000 sets of selection probabilities, inverted them to calculate the IPSW, and used these to calculate 5,000 ORs. Results were summarized as median OR and 95% SI. We added random error, producing a median OR and 95% RESI.

To adjust for exposure misclassification, missing exposure data, selection bias, and confounding, we multiplied 5,000 simulated IPSW by 5,000 simulated predictive values to create 5,000 combined weights. Then, as before, to adjust for exposure misclassification we created a dataset with two observations per participant and assigned each observation the combined weight corresponding to the assigned exposure status. The multivariable model regressed assigned exposure (not reported exposure) on the outcome, adjusted for confounders (study site, maternal race/ethnicity, maternal education), and was weighted by the combined weight to estimate the OR. Probabilistic results were summarized as median OR and 95% SI. Random error was added to generate a median OR and 95% RESI.

When biases are adjusted serially in multiple bias analysis, the order of bias adjustment is important; if adjustment is done out of order, incorrect results could be obtained.^4,15 Although we did not adjust biases serially (a single model is used), there is an element of “order” needed to estimate valid parameters.

If we consider exposure misclassification and selection bias, there are four possible datasets: (1) both biases present, (2) selection bias only, (3) exposure misclassification only, and (4) no exposure misclassification or selection bias. Our goal is to move from dataset 1 (two types of bias) to dataset 4 (no bias). This can be done by removing exposure misclassification first (datasets 1 to 2 to 4) or selection bias first (datasets 1 to 3 to 4).

In NBDPS, we removed selection bias first. When estimating IPSWs, we obtained these values from a cohort study, which likely had exposure misclassification and selection bias (dataset 1). Once the IPSW were estimated and applied, this produced dataset 3 (exposure misclassification only). We estimated predictive values from a “dataset 3” (exposure misclassification, no selection bias); this was NHANES. Because NHANES-provided weights accounting for nonresponse and other selection effects, we assumed this represented what NBDPS would have been in absence of selection bias. Once predictive values were estimated and applied, this moved from dataset 3 to 4 (no exposure misclassification or selection bias). Because there was confounding in the underlying source population, we estimated bias parameters conditional on confounders.

The crude OR for the association between prepregnancy obesity and CL/P was 1.09 (95% CI: 0.97, 1.21). After adjusting for study site, maternal race/ethnicity, and maternal education, it was 1.10 (95% CI: 0.98, 1.23). Despite the confidence interval crossing the null, we continued the bias analysis because of evidence in the literature that this weak association might not be due to chance.

Using bias parameters from NHANES (Table 2) in the non-probabilistic bias analysis, the adjusted ORs ranged from 1.02 to 1.13 for the three misclassification types (Table 3). In probabilistic analyses, the adjusted median ORs ranged from 1.01 to 1.11.

In non-probabilistic and probabilistic analyses, the OR adjusted for selection bias and confounding was 0.98 (Table 3).

In the non-probabilistic multiple bias analyses for exposure misclassification, missing exposure, selection bias, and confounding, the adjusted OR ranged from 0.94 to 1.03 for the three misclassification scenarios. In the probabilistic analyses, it ranged from 0.93 to 1.02 (Table 3).