Literature largely shows that incentives (both prepaid and postpaid) are effective in increasing response rates for probability-based, cross-sectional surveys (Brick et al. 2007; Halpern et al. 2002; Hsu et al. 2017; Mercer et al. 2015; Singer et al. 1999; Trussell and Lavrakas 2004). Although not definitive, increasing response rates can reduce the likelihood of nonresponse bias in the population estimates (Brick and Tourangeau 2017). Additionally, increasing the size of the incentive further increases response rates for probability-based, cross-sectional surveys (Fox et al. 1988; Singer et al. 1999). Despite the plethora of literature on survey incentives for cross-sectional surveys, the effect of incentives on web panels has not been extensively studied.

There are several reasons that the benefits of incentives¹ when used in web panels may be different than other modes and sampling frames. First, Callegaro et al. (2014:chap. 2) note that most of the literature on survey incentives involved respondents with no prior or limited experience with surveys. Web panelists have experience with surveys, which may build trust and feelings of reciprocity. For panelists with extensive survey experience, Callegaro et al. (2014:chap. 2) surmise that the current impact of postpaid incentives may differ from what the historic postpaid incentive literature suggests.

Second, the potential relative increase on response is often smaller for web panels than other surveys. Because panelists have previously consented to participate in the panel, research suggests that the completion rate on any given survey can be as high as 77.0%, a far departure from the 9% response rates seen on some other studies such as telephone surveys (Baker et al. 2010; Keeter et al. 2017; Vonk et al. 2006). As a result, the pool of individuals who may be converted from refusal to respondent with the use of incentives is relatively small, and the relative effect of the incentive may be diminished. For example, let us assume an incentive can convert 20% of individuals who would otherwise become nonrespondents. If the completion rate is 40%, an incentive could increase it to 52% (= 100 × [0.40 + (1 − 0.40) × 0.20]), resulting in a relative gain of 30% (= 100 × 0.52/0.40 − 100). However, if the completion rate starts at 60%, an incentive could only increase it to 68%, for a relative gain of 13.3%.

Third, web respondents may be unaware of the increased incentive, rendering them ineffective. While some panelists may receive as few as one survey invitation per year, others receive a new request every day. In a comparison study of 20 U.S. web panels, Miller (2008) found that 33.0% of respondents reported taking 10 or more online surveys in the previous 30 days. Moreover, the method of invitation within a given panel is relatively consistent (Callegaro and DiSogra 2008). With few exceptions, panelists receive an e-mail from the same source with standardized instructions and a link to the survey. Panelists may be given a similar time frame to complete each survey and a standard incentive for completion. Given the frequency of invitation and the rarity with which information within the invitation changes, participants may try to save time and effort by skimming the invitation. In the rare event that new information is included in the survey invitation (e.g., a unique incentive offer), panelists who skim may miss this information if it is not explicitly called out with different font, size, or color (Dillman et al. 2014). Moreover, social exchange theory (Dillman et al. 2014) suggests that individuals put in the same amount of effort (i.e., whether to participate) as they receive back (i.e., incentive). If panelists are unaware of an incentive that is larger than the typical amount, they will assume the standard incentive amount and decide how much effort to contribute, negating the potential effect of the larger incentive.

Even if larger incentives for web panelists do increase completion rates, their effect on data quality and nonresponse bias is debatable. Although most research suggests that incentives have no effect on data quality (Shettle and Mooney 1999; Singer and Kulka 2002; Singer et al. 1998; Singer and Ye 2013; Toepoel 2012), at least one study suggested possible data deterioration (Barge and Gehlbach 2012). The risk exists that incentives may reduce data quality by attracting respondents who normally would not have participated in the survey and whose primary goal is to collect the incentive with as little effort as possible (Goritz 2006). Research suggests that such respondents speed through the survey without giving questions much thought (Goritz 2006). Barge and Gehlbach (2012) found that 81.0% of incentivized respondents engaged in at least one form of satisficing behavior (i.e., speeding, item nonresponse, or straightlining), and 41.0% engaged in at least two forms. Satisficing increases measurement error as people are not providing accurate or thoughtful responses. Jäckle and Lynn (2008) found item nonresponse increased by 10–17% in waves of a panel survey in which incentives were used, though the differences were not statistically significant. To the extent that item nonresponse is not missing at random, it may introduce nonresponse bias that is difficult to enumerate and quantify its impact on the overall survey quality.

In this article, we examine the effect of two postpaid incentive amounts on (1) response, (2) data quality, and (3) nonresponse bias of key outcomes collected in the NIFS. We hypothesized that the response among web panel members receiving the larger 5,000-point incentive would be similar to those receiving the standard 1,000-point incentive. Given the current literature, we did not formulate hypotheses on the effect of incentives on data quality and nonresponse bias and instead only provide an evaluation.

The NIFS sample was drawn from the GfK (2012) KnowledgePanel^®, a probability-based Internet panel designed to be representative of the noninstitutionalized U.S. population 18 years and older. The sample was stratified by age-group (18–49 years, 50–64 years, and 65 years or older) and race/ethnicity (Hispanic, non-Hispanic white, non-Hispanic black, and non-Hispanic other), creating 12 strata (Table 1). The samples were selected using a single-stage stratified design meant to oversample minorities and with selection probabilities inversely proportional to the KnowledgePanel^® survey weight, a base weight adjusted for nonresponse (Lu et al. 2017).

Data collection occurred October 29–November 11, 2015, and October 27–November 9, 2016, for the 2015 and 2016 NIFS, respectively.² In 2015, sampled individuals were invited by e-mail to participate in the NIFS via a unique web link, and all were promised the standard 1,000-postpaid points (the equivalent of US$1) if they completed the survey.³ For the 2016 NIFS, we identified six strata with lower than average completion rates in years 2014 and 2015 and offered them a larger postpaid incentive of 5,000 points (the equivalent of US$5). Those offered the 5,000-point incentives included: Hispanic 18–49 years, non-Hispanic black 18–49 years, non-Hispanic other/multiple races 18–49 years, Hispanic 50–64 years, non-Hispanic black 50–64 years, and Hispanic 65 years and older. The remaining six strata received the standard 1,000-point postpaid incentive.

Key operational aspects of the NIFS remained constant across years such as the length of the questionnaire, survey topic, information provided in the introduction, and data collection field period. In both 2015 and 2016, respondents were informed that the interview would take about 10 minutes to complete, and reminder e-mails were sent approximately halfway through data collection to all nonrespondents. In 2016, additional reminders were sent to strata with lagging completion rates to meet prespecified targets regardless of their incentive level. A total of 3,301 and 4,305 interviews were completed in 2015 and 2016, respectively (Table 1).

Data for the 2015 and 2016 NIFS were evaluated in three ways: survey participation (response), quality of the respondent data, and bias. We discuss analytic methods in this section. All analyses were conducted in SAS version 7.1 and SUDAAN version 11 with base weights (inverse probability of selection) to account for different sampling rates across strata and years.

We initially hypothesized that larger incentives would be ineffective at improving response in a web panel due to historically high completion rates, the frequency with which panel members receive survey requests, and the homogeneity of those requests. However, evidence from the NIFS suggests that, like traditional cross-sectional surveys, larger incentives do increase completion rates with minimal effect on time to completion. Specifically, using the DID models, we found that the 5,000-point incentive increased the completion rate by 7-percentage points over the 1,000-point incentive. This is similar to the effect of observed in other cross-sectional, probability-based surveys of the general population (Fox et al. 1988; Singer et al. 1999) and suggests that previous literature regarding the direct correlation between postpaid incentive amount and response may be applicable to web panels.

We also investigated the effect of a larger incentive on data quality and nonresponse bias, observing minimal effects on both. In terms of data quality, respondents receiving the larger incentive took an average of 3.11 minutes more to complete the survey than their 1,000-point counterpart. One variable used to analyze nonresponse bias was significantly affected by the larger incentive—individuals receiving the larger incentive reported higher levels of income, on average, compared to the lower incentive group, making an already skewed income distribution worse. Additionally, differences were observed on the current year’s influenza vaccination rate; however, the analyses available could not deduce whether the larger incentive increased or decreased bias. We did not observe any meaningful incentive effect on the indicators of data quality and nonresponse bias. As with the research on response, these findings were relatively consistent with previous literature conducted in other modes and using other sampling frames—while larger incentives may increase the risk for lower data quality and higher nonresponse bias, it appears that the realization of such a risk and/or the magnitude of the effects are small (see, e.g., Singer and Ye 2013).

Limiting consideration to response, data quality, and nonresponse bias and given the goals of the NIFS, we concluded that the larger incentive was superior. This statement and supporting results, however, may not be applicable to all surveys. For the NIFS, the gain in the completion rate was more important than what we considered to be small negative consequences. Another researcher with another set of priorities may draw a different conclusion. We recommend that researchers use these findings to quantify the effect of larger incentives and consider those effects in the context of their unique goals.

Researchers should also consider the applicability of these findings to their own research. Web panels vary significantly (Unangst et al. 2019). Some are probability based, and some are not, and all have different guidelines on the frequency of invitations. To the extent that these features alter the effectiveness of larger incentives, our findings may have limited applicability.

Additionally, we did not have a true experimental design, potentially limiting the generalizability of these findings. All 2015 sampled members were offered a 1,000-point incentive, while in 2016 a nonrandom set of individuals in six affected strata were offered a 5,000-point incentive. The lack of random assignment coupled with the fact that all individuals selected for the larger incentive were racial/ethnic minorities, limited our ability to isolate the effect of incentives from true change or other differences across surveys. The analyses performed sought to minimize this limitation by controlling for strata in the analysis and by using DID models. The lack of large interaction effects by strata suggested that the incentives would have similar effects on all subdomains, minimizing this limitation. The DID models provided similar evidence that the 7-percentage point effect on the completion rate was also robust. However, further research using an experiment should be replicated to confirm (or dispel) our findings.

Finally, this research cannot speak to the overarching relationship between larger incentives and response, data quality, and bias. We only tested two incentive amounts. If the relationship between amount and effect is linear, then researchers may use algebra to identify the effect of any incentive amount. But, if they are not linear (e.g., if a larger incentive has diminishing returns), then additional research using varying amounts will be required.