Producing timely data is a priority of National Statistics Offices (NSOs). However, some of the more timely data collections, including web-based surveys, may be subject to biases relative to large nationally representative surveys conducted by NSOs due to lower coverage and response rates. Adjusting these timelier sources with less timely but higher quality reference surveys may decrease their biases.

Selection bias has been acknowledged in different areas (Hernán 2004) and is becoming more critical in the big data era with the rapid emergence of various web surveys to address time-sensitive priority topics, referred to here as target samples. Data collected in target samples, such as web panels, can result in attrition and response rates are often found to be 10% or lower (Baker et al., 2013). Although low response is not necessarily indicative of response bias (Groves and Peytcheva, 2008; Brick and Tourangeau, 2017), selection bias has been of great concern because the composition of web panels often differs from that of the underlying population. Panel members tend to be more educated and to have higher socioeconomic status than non-panel-members (Craig et al., 2013). Epidemiologic target samples often recruit “healthy volunteers” with lower estimates of disease incidence and mortality that a general population (Pinsky et al., 2007). To reduce the selection bias of the target samples, various propensity score (PS)-based adjustment methods have been developed which use an existing high-quality probability sample (e.g., national representative surveys) as a reference, where high quality refers to probability-sampled surveys with relatively low sampling and non-sampling errors that lead to confidence in their ability to produce representative estimates (Groves, 1989). Recent PS adjustment methods include, but are not limited to, PS weighting (Valliant, 2020; Chen et al., 2019) and PS matching (Kern et al., 2020) methods.

The amount of bias reduction, however, varies depending on the outcome and variables that are collected in both the target and reference data sources. Wang et al. (2020a) studied the bias reduction of different PS adjustment methods using the non-representative U.S. National Institutes of Health–American Association of Retired Persons cohort (NIH-AARP, 2006), with the National Health Interview Survey (NHIS) as the reference survey. Among the ten selected diseases examined, they found the amount of relative bias reduction ranged from 8% to 30% using their proposed PS-based kernel weighting (KW) method. There is still a large amount of bias that is not removable by PS adjustment methods alone due to the uncollected information in the reference probability sample. The effectiveness of PS adjustment methods depends on the identification of the proper set of covariates, their availability, and their quality (Baker et al., 2013). Some references (e.g., Mercer et al. 2018) have even argued that choosing the correct variables can be more important than choosing the correct adjustment models, including PS methods.

High-quality probability samples surveyed through well-designed questionnaires are in great demand as reference surveys for at least two reasons: 1) Different PS adjustment methods, including PS-based weighting and matching methods, require a high-quality probability sample as the reference in order to create a set of pseudo-weights for the target sample to better represent the underlying target population; 2) Different target samples may use a common high-quality probability sample as the reference for cost efficiency by using the same questions with exact wordings to avoid potential reporting/measurement error. Given a high-quality population representative reference survey, we are interested in identifying the types of variables that are critical for collection in the target sample to improve its external validity in estimating population quantities. The findings can be used in turn to plan for future surveys.

The target sample motivating this research is collected through the National Center for Health Statistics’ (NCHS) Research and Development Survey (RANDS), a probability-based panel survey that has been conducted using online and phone administration (https://www.cdc.gov/nchs/rands). Although the RANDS data are more structured than nonprobability samples, RANDS is subject to potential selection bias as RANDS has lower response rates, as well as potential measurement and coverage errors compared to traditional interviewer-administered national population health surveys (Parker et al., 2020). On the other hand, probability survey panels such as RANDS have the potential to produce more timely information than national population surveys. To reduce the potential selection bias in RANDS estimates, NCHS’ NHIS has been used as a reference sample to construct pseudo-weights using PS-based weighting methods (Parker et al., 2020; Irimata et al., 2020) and raking. These adjustments have been applied to the estimation of several population health outcomes, including diagnosed asthma, diagnosed hypertension, diagnosed diabetes, health insurance, as well as for health outcomes related to the coronavirus disease 2019 (COVID-19) pandemic such as access to health care. In these studies, adjustment to the NHIS (PS-based weighting or raking) has typically been performed using the main effects for all common covariates between RANDS and the NHIS, including sociodemographic, health, and internet use variables; the adjustment for RANDS during COVID-19 used a limited subset of variables for the public release of COVID-19-related estimates (https://www.cdc.gov/nchs/covid19/rands.htm). While PS weighting and raking adjustments have been shown to improve RANDS estimates relative to those without any adjustment to the NHIS (Parker et al., 2020; Irimata et al., 2020), stability of the estimated propensity scores and how the inclusion of different variables in the propensity model or calibration affects bias and efficiency of the estimated population mean for various outcomes have been a major concern.

Propensity model variable inclusion has been widely studied in different areas, including clinical trial or medical research and survey research. In clinical trial research, participants are included for clinical and experimental purposes (mainly for treatment effect estimation) and are not necessarily representative of the U.S. population. Simulations (Brookhart et al., 2006; Leyrat et al., 2013) were performed to examine the effect of the choice of variables that are included in a propensity model has on the bias, variance, and mean squared error of estimated treatment effects. It was concluded that omitting confounding factors increases bias and the inclusion of variables that are independent of the exposure but related to the outcome in the propensity model gains efficiency without increasing bias of estimated treatment effects. However, covariate inclusion for propensity score models in clinical trial research has been limited. Ali et al. (2015) provided a systematic review of covariate inclusion in the PS model for medical studies and concluded that the quality of reporting variable inclusion is far from optimal in the medical literature. Similarly, Grose et al. (2020) found 90% out of 303 systematically reviewed studies did not provide justification for covariates included in their PS models.

In survey research, propensity analyses have been conducted to estimate response propensity (Groves, 2006; Iannacchione et al., 1991) and to adjust sampling weights in representative surveys to reduce the estimation bias due to unit nonresponse. The best auxiliary variables to be included for nonresponse adjustment are those simultaneously correlated with response propensity and the key survey outcomes (Lessler and Kalsbeek, 1992). Little and Vartivarian (2005) further suggested that most important feature of variables for inclusion is that the variables are predictive of survey outcomes; prediction of response propensity is a secondary, though useful, goal.

This paper, in contrast to the interest of estimating treatment effects in clinical research, aims to estimate population quantities such as the population mean. We are interested in identifying key auxiliary information in a reference probability survey to improve the external validity of inferences from a target dataset. This is an important obligation for survey designers because the choice and inclusion of these variables has a tremendous effect on both the bias and the precision of the estimates of population quantities. This differs from the goal of nonresponse adjustment which uses chosen covariates for predicting response propensities as, in nonresponse adjustment research, respondents are nested within the sampled units, and respondents and nonrespondents share common sampling design variables. As a result, unweighted analysis of response propensity can be performed conditional on the design and response predictive variables (Little and Vartivarian, 2003). However, this is not true for estimating the propensity of target sample inclusion because the reference survey and the target samples are often independent without sharing design variables (Wang et al., 2021). Variable choice for the propensity model used to predict the target sample inclusion propensity should be performed with additional care.

This paper aims to examine how different types of variables included in a propensity model impact the performance of population mean estimation using target samples through the directed acyclic graph (DAG), a common graphical tool in causal studies but largely under-utilized in survey research. The DAG is used to identify certain types of variables in the causal pathway to be included in the PS model which results in the lowest bias and highest precision under various scenarios. Estimated population means and their variances are evaluated analytically and numerically under various mis-specified propensity models, including with and without interactive effects. Different levels of variable correlations in the finite population are considered to mimic real data scenarios. The findings are applied to RANDS, with NHIS as the reference, to estimate the prevalence of asthma in the U.S. The RANDS evaluation demonstrates the advantage of this approach compared to the approach when the propensity model includes all available variables.

The results from this research provide insight for data analysts on propensity model construction to improve the population representativeness of target samples. It also provides insight for questionnaire designers on the critical auxiliary information to collect from the reference survey. NSOs, using the paper results, can design the questionnaires for both the target and reference surveys and release accurate estimates for priority topics from more timely data sources.

We first introduce some notation. Suppose Y is a binary outcome of interest (e.g., for estimating the prevalence of a disease or health condition: Y=1 if event and 0 otherwise). In the context of survey sampling, suppose A is the binary selection indicator variable (i.e., A=1 if a population unit participates in the target sample and 0 otherwise). Note A indicates the target sample participation with value of one representing population units who are recruited and respond to the survey.

We adapt the framework of Brookhart et al. (2006) of employing a directed acyclic graph (DAG) to study potential selection bias induced by three types of covariates (see Figure 1a):

We now present some background about PS adjustment methods. For estimation of the finite population (FP) mean of a binary outcome E(Y) = p(Y = 1), the naïve unweighted estimate using the selected target sample (A = 1) has bias relative to the FP, given by Bias = p(Y = 1∣A = 1) − p(Y = 1) = (R − 1)p(Y = 1) with R=p(A=1∣Y=1)p(A=1). In order to remove the bias, it requires the conditional distribution of A given Y is the same as the distribution of A, denoted by A ⊥ Y and adjustment methods based on PS are often employed (Valliant 2020).

More specifically in PS-based adjustment methods, the population mean μ of the outcome Y, is estimated by (1) μ^(x)=1∑j∈Scwj(x)∑j∈Scwj(x)yj, where S_c denotes the set of sample units in the target sample of size n_c; y_i for j ∈ S_c and x are, respectively, realized values of the outcome Y and X; the pseudo-weights w_j(x) for j ∈ S_c is constructed to balance the distribution in covariates between the target sample and the reference survey. Note X can be a single covariate or a vector of covariates from X₁-X₃, and are available in both the target sample and the reference survey, while the outcome variable Y is available in the target sample S_c only.

Various PS-based adjustment methods, including PS weighting and PS matching methods, have been developed under the following assumptions. First, the reference survey sample (in our real data example, the NHIS), through weighting, properly represents the target population of interest. Second, all finite population units have a positive participation rate (i.e., each individual in the population has a positive propensity to volunteer to participate in RANDS panel). Third, conditional exchangeability holds with no unmeasured confounders, that is, the probability for each individual in the FP to participate in the target sample is not related to his/her outcome, after adjusting for all measured variables. It is a common practice that the variables in the target sample are measured using same question wordings as in the reference survey to avoid potential reporting or measurement error.

While PS weighting and PS matching methods have similar assumptions, PS weighting methods construct the pseudo-weight by the inverse of the inclusion probability conditional on x, i.e., wj=1e(xj) for j ∈ S_c, with e(x) = p(A = 1∣x), the target sample inclusion probability conditional on x. It can also be verified that A ⊥ x∣e(x). In contrast, PS matching methods distribute the survey sample weights to target sample units that have similar predicted propensity scores. For example, the KW method (Wang et al. 2021) first assigns the sample weight of each survey unit, say unit i, to cohort members proportionally according to kernel distances, defined by propensity scores K(e(xi)−e(xj)h) for j ∈ S_c, where K(.) a kernel function such as the standard normal density function, and h is the bandwidth selected by Silverman’s rule of thumb method (Silverman, 1986). As such, most of the sample weight for survey unit i is assigned to those cohort members with similar propensity scores. The assigned portions from survey members to cohort member j are then summed up to form the pseudo-weight w_j.

In sections 2.1-2.2, we assume that X₁, X₂, and X₃ are mutually independent in the FP and study how the PS-based adjustment methods reduce the bias and variance through the incorporation of different types of variables in the propensity models. We further consider real situations in section 2.3 when different types of variables are correlated in the FP using DAG. Although DAG is a graphical tool developed for causal interpretation, we used it to rule out possible confounding and identify conditioning covariate set for Y ⊥ A. The actual causation is not important in this context. Section 2.4 summarizes some practical guidelines for identifying the variable types in real data and choosing between PS-based methods to construct pseudo-weights when covariates interactively affect the target sample participation and the outcome.

Simulation studies were conducted to evaluate the performance of the mean estimator from (1) with the pseudo-weights constructed by the ALP and the KW methods based on propensity models that consider different types of covariates.

RANDS, a series of web-based probability panel surveys conducted at NCHS (https://www.cdc.gov/nchs/rands), has been used for methodological research and, more recently, for providing early experimental estimates on the COVID-19 pandemic. RANDS has the capability to collect data quickly and is less costly than traditional national household surveys, but is subject to potential selection bias due to low response rates. Adjustment methods to construct pseudo-weights, including propensity-score based methods, are applied to balance the covariate distributions in the target sample and the FP, and are an important component of the RANDS program. We consider the simulation findings from this paper for selecting PS-model covariates to estimate the national prevalence of asthma compared to NCHS’ NHIS.

Data from the third round of RANDS (RANDS 3) is evaluated. RANDS 3 was collected in 2019 using NORC’s AmeriSpeak® Panel (https://amerispeak.norc.org) and included responses from 2,646 panelists aged 18 years and older. RANDS 3 panelists were surveyed via web and were asked questions related to general and mental health, medical conditions, opioid use, and pain. The RANDS 3 cumulative response rate was 18.1%. The RANDS 3 original panel weights were developed by the inverse of the probability of inclusion in the AmeriSpeak® Panel, subject to nonresponse adjustment and poststratification adjustment to external population totals of age, sex, education, race/ethnicity, housing tenure, telephone status, and Census Division (National Center for Health Statistics, 2020). The original panel-weighted estimate of diagnosed asthma (ever been told you had asthma) in RANDS 3 was 16.86% (standard error = 0.98%). For comparison, the unweighted estimate of diagnosed asthma in RANDS 3 was 16.40% (standard error = 0.72%). The 2019 NHIS (n=31,997) is evaluated as the gold standard. The NHIS (https://www.cdc.gov/nchs/nhis) is a cross-sectional household interview survey that collects information on a broad range of health topics, primarily through face-to-face interviews. The NHIS sample adult file, which is a collection of responses from one randomly selected adult per selected household, was used to evaluate the prevalence of ever having asthma among adults. The percentage of adults who ever had asthma based on the 2019 NHIS (n=31,997) was 13.46% (standard error = 0.25%).

Common covariates available in RANDS 3 and the 2019 NHIS that were potentially related to diagnosed asthma or the selection indicator were considered (see Table 3). All percent estimates in Table 3 (when expressed as proportions) meet the NCHS data presentation standards for proportions (Parker et al., 2017). As observed, demographic variables of age, sex and race/ethnicity have similar weighted distributions in the RANDS and NHIS. This result is as expected, since these variables are poststratification variables used to construct the sample (or original panel) weights in both NHIS and RANDS. As observed, persons with higher levels of education, with selected health conditions (i.e., diagnosed high cholesterol, diagnosed chronic obstructive pulmonary disease (COPD), emphysema, or chronic bronchitis, diagnosed diabetes, and diagnosed hypertension), who are current or former smokers, or who are not married (with the exception of those who are widowed) participated in RANDS at a higher rate compared to the NHIS. Since the percent of missing values across all considered variables was relatively low for both data sources, ranging from 0%-0.68% for RANDS and 0%-2.64% for NHIS (unweighted), missing values were excluded for evaluation.

To check for correlation between covariates, bivariate correlations were assessed on the weighted NHIS data. Bivariate correlations for all selected covariates were statistically significant. Prior to evaluating the propensity models, the survey weights for both data sets were normalized to their respective sample sizes (n=2,646 for RANDS, n=31,997 for NHIS) as suggested by Li et al. (2011) and Wang et al. (2021). The KW method was implemented for demonstration to construct RANDS pseudo-weights that adjust for potential selection bias due to differential non-response and under-coverage of some groups on the sample frame using the NHIS data as the reference dataset.

A full propensity model (denoted by model.all) that includes all covariates and their pairwise interactions was used to create pseudo-weights. Due to the large number of parameters in the full model, estimated propensity scores can be unstable. As a result, some form of stepwise propensity model selection methods have been conducted in different studies (Weitzen et al., 2004; Austin 2008; Wang et al., 2020a), using the combined target sample and the reference sample to identify significant terms out of the full propensity model. In the framework of our paper, the combination of the confounders and selection predictors (i.e., model.x13 which contains X₁ and X₃), which can be main effects of covariates or their nonlinear/nonadditive combinations such as pairwise interactions, are recommended as terms for inclusion. Based on the simulation results, we expect that the pseudo-weighted mean under model.x13 would be unbiased but with higher variability when compared with the estimates under model.x12 that includes the confounders and outcome predictors.

Accordingly, we conducted the outcome model selection using backward selection on the reference survey (e.g., the 2019 NHIS), to identify terms which were confounders or outcome predictors. We defined the selected model as model.x12.n (contains X₁ and X₂) with “n” indicating that the outcome predictors were identified using the NHIS. However, it is often the case that the reference probability surveys have no collected information on the outcome variable. In this case, we have only the target sample (e.g., RANDS) available for outcome model selection. With the assumption of the conditional noninformative sampling of the target sample, it is expected the unweighted regression of the outcome would produce unbiased estimates of regression coefficients (Korn and Graubard, 1999). As follows, outcome model variable selection was conducted based on the unweighted outcome regression of the RANDS data, and the selected model included both confounders and outcome predictors, denoted by model.x12.r (contains X₁ and X₂) indicating that the outcome predictors were identified using RANDS. The common terms in model.x13 and model.x12 (denoted by either x12.n or x12.r based on the information available) are confounders, and the corresponding propensity model is denoted by model.x1. The identified covariate types under each model are reported in the Appendix. Due to the correlation between x1, x2 and x3, we expect estimates under model.x1, albeit efficient, may not remove as much bias as under the model.x12 or model.x13.

The outcome models utilized the observations in the NHIS or the RANDS, whereas the propensity model utilized the observations in the combined NHIS and RANDS data, from which the estimated propensities were obtained and used for construction of the KW pseudo-weight for each individual in RANDS. Note that RANDS has panel weights, which were computed as an overall sampling weight for the selection of each panel member from the sampling frame and the selection of the panel member into RANDS. We considered two scenarios of 1) panel weights or 2) no panel weights for the propensity analysis.

Various propensity models that included different types of covariates were evaluated by the coefficient of variation (CV) of the KW pseudo-weights (CV = sd(KW)/mean(KW) with sd denoting standard deviation), relative bias (relBias=μ^RANDS−μ^NHISμ^NHIS×100%), standard error (se), and mean squared error (MSE=(μ^RANDS−μ^NHIS)2+se2(μ^RANDS)). The relative bias was calculated as the estimated asthma prevalence in RANDS relative to the NHIS estimate where the RANDS estimate, μ^RANDS, was calculated using the KW pseudo-weights produced from the various propensity models. The standard error se(μ^RANDS) considered the variability due to estimating the propensity scores, sampling, kernel weighting, as well as differential pseudo-weights by the Taylor linearization method (Wang et al., 2020b). For comparison purposes, we also report the relative bias, standard error, and MSE of the original panel-weighted and unweighted estimates of asthma prevalence in RANDS 3. Results are presented in Table 4.

Four observations can be made from Table 3. Firstly, all 12 (panel-weighted or unweighted) propensity-adjusted estimates perform better, with a smaller MSE (or relative bias), compared to the original panel-weighted RANDS estimate of asthma prevalence without PS adjustment. When the RANDS panel weights are considered in the propensity analysis, the standard errors tend to be inflated, relative to those in the lower pane, due to more variable KW pseudo-weights with their CVs ranging from 1.07-1.13 (see the upper pane) vs. 0.69-0.83 (see the lower pane). Accordingly, observations 2-4 focus on the results in the lower panel when the panel weights are not used to construct KW pseudo-weights. Secondly, consistent with our expectations, the propensity models that contain confounders and selection variables, i.e. Model.x13, produce larger estimated variances compared with Model.x12 irrespective of the outcome predictors being selected from RANDS or the NHIS (e.g., se=0.97 vs. 0.81-0.84). Thirdly, comparing the estimates under the propensity models containing the confounders and outcome predictors that are selected using NHIS data (i.e. Model.x12.n) vs. the RANDS data (i.e. Model.x12.r), similar relative bias, se and MSE are observed (relBias = 11.38 vs. 11.37; se=0.81 vs. 0.84, MSE=3.01 vs. 3.04). Lastly, the relative biases under Model.x1 are somewhat larger than that under Model.x12 (relBias = 13.67 vs. 11.38 or 13.44 vs. 11.37). This result could be due to the existing correlation between the confounders and the outcome predictors. Adjusting for confounders only may not be sufficient for maximum bias reduction.

In brief, for the evaluation of diagnosed asthma using the RANDS data, we would recommend the pseudo-weights constructed under Model.x12.n with the confounders and predictors selected from the reference survey (e.g., NHIS). In situations where outcome variables are not collected in the reference survey but available only in the target sample (e.g., RANDS), Model.x12.r can be the alternative model to construct the KW pseudo-weights, assuming conditional noninformative sampling holds for the target sample.

Identifying and collecting the best information on more timely target sample and on higher quality reference surveys can increase the ability of NSOs to produce timely estimates with lower bias from target samples. This paper examined how different types of variables that are included in a propensity model impact the performance of PS-based pseudo-weighted estimators for population mean estimation from a target sample. Means and variances of estimated population means under various mis-specified propensity models, including different types of variables with and without interactive effects, were evaluated analytically and numerically. Different levels of variable correlations in the finite population were also considered to reflect real data scenarios. We have the following findings: 1) confounders, the variables related to both the selection indicator and the outcome of interest, are important variables to include in the propensity model; 2) pseudo-weights that balance the distributions in the outcome predictor x₂ or the selection variable x₃, in addition to the confounder x₁, denoted by w(x₁, x₂) or w(x₁, x₃), should be constructed for the target sample units so that the corresponding pseudo-weighted target sample mean is approximately unbiased; 3) compared to w(x₁, x₃), the pseudo-weights w(x₁, x₂) gain more efficiency in estimating population means. In contrast, the inclusion of selection variables, compared to the outcome predictors, in the propensity model tended to inflate the estimated variances. Intuitively, the outcome predictor is related to the outcome variable; including outcome predictors in the propensity score model distinguishes differences between the outcome in the reference and target samples, which results in weights related to outcome and therefore yields estimates with smaller variance estimates. Finally, findings are applied to real target data from RANDS, a survey that uses commercial probability panels, which has potential selection bias. Under the model with confounders and outcome predictors (Model.x12) or model with confounders and selection variables (Model.x13), the RANDS estimate of U.S. asthma prevalence had the greatest bias reduction (relative bias ranging from 11.37%-13.51% compared to the NHIS) when the panel weights are not used to construct KW pseudo-weights, compared to the original panel-weighted RANDS estimates (relative bias of 25.31%).

Results from this paper have several important applications in practice for NSOs that collect data from both target surveys and high-quality reference surveys. First, this study provides a principled approach to select covariates for the PS model. Rather than including all variables or selecting certain demographic variables, covariates are assessed based on their variable type (confounder, outcome predictor, selection variable) to be included in the PS model for population mean estimation. Second, guidance on how to design the questionnaire for a target survey with specific research questions (e.g., SARS-CoV2 seropositivity web survey by Kalish et al., 2021) is provided to survey practitioners. The attributes that are most effective in reducing bias/variances of estimates can be collected and used to reduce potential selection bias for the purpose of timely data collection and minimum response burden. Third, the findings from this study can be used for future development of a high-quality probability survey, including the planning of covariates to collect through paradata or the survey questionnaire with minimized measurement/reporting error, to be used as a high-quality reference survey by various nonprobability or web-based probability surveys with selection bias.

The proposed variable inclusion strategies have limitations that can be of interest for future research. First, the strategy is developed for single-outcome studies with research questions related to one outcome of interest, e.g., SARS-CoV2 seropositivity study (Kalish et al., 2021). The target sample was collected in a web survey with questions related to COVID-19 infection only. For studies with multiple key outcome variables, it would be of interest to study how the correlation among outcome variables, the overlap for each variable type across outcomes, and their interplays affect population mean estimation by different variable inclusion strategies. Second, in our simulation, we demonstrated the use of a PS matching method (KW) and a PS weighting method numerically. It showed that KW produced approximately unbiased estimates when the analytic propensity model is slightly mis-specified (without the interaction term) while the PS weighting methods require the true propensity model to obtain unbiased estimates. In practice, the underlying selection mechanism of the target sample is often complicated, involving higher orders of nonlinearity and/or nonadditivity. For complicated propensity models, only including main or interactive effects of blocking variables in the logistic model by KW methods may not be sufficient. Nonparametric modeling such as machine learning methods may be promising (Kern et al., 2020). Third, the focus of this paper was on evaluating the bias and variance reduction of Horvitz-Thompson estimators of FP mean by the types of covariates in the propensity model and thus we did not study how the pseudo-weights, when combined with different estimators, affect the FP mean estimation. Alternative analysis methods, such as doubly robust estimators (Chen, Li and Wu, 2018) or augmented estimation equations in the missing data imputation context (Robins, Rotnitzky and Zhao, 1994) can be employed, after identifying the appropriate type of variables to include in the propensity score model. Fourth, selection bias in target samples, compared to more rigorous probability samples, can be induced by low response, different question wording/ordering, topic salience for different types of questions (for example health and health conditions can have large selection bias as shown in Table 2). It would be interesting to study how the proposed variable inclusion strategy can be adapted to reduce the selection bias in target samples with different response rates, question order/wording and salience effects. Lastly, in our data example the backward selection is employed for identifying the type of variables. It would be interesting to employ and compare alternative variable selection methods such as AIC or BIC (Lumley and Scott, 2015) that incorporate complex sample designs for the model selection.