Survey research faces two primary challenges: reaching potential subjects, and having potential subjects agree to participate. In the face of rapid societal change, survey methods must constantly be reexamined: new technology, security fears, and laws regarding communications technology may drastically change survey research [1-5]. Survey researchers must therefore adapt their techniques to changing circumstances in order to collect valid and reliable information about a representative sample of individuals.

Though once considered a major breakthrough in survey research, the utility of computer-assisted telephone interviews (CATIs) are challenged by new telephone technology and increasing use of wireless (cellular) telephones instead of (or in addition to) landlines. Caller ID and answering machines/voicemail have made it easier for people to screen their calls, making it more difficult for study staff to reach potential participants by telephone [4]. While methods for telephone sampling of landlines are well-established [6,7], sampling wireless numbers faces numerous challenges: numbers are portable, and may not be representative of a geographic region; there are no comprehensive public lists of wireless telephone numbers; federal laws prohibit use of automated dialers when calling wireless telephones; households with wireless telephones and landlines overlap; subjects generally pay for wireless minutes used; and subjects may be in public places when researchers call them, raising privacy considerations [2,4,5].

As wireless telephone usage increases, challenges to telephone-based surveys become more pronounced. Between 2003 and 2009, the number of U.S. households with wireless-only telephones increased from 6.7% to 24.5% [8,9]. Almost half of young adults, aged 25-29, live in wireless-only households [9]. Urban areas and households living in poverty also have a higher proportion of wireless-only households [8,9]. Demographic differences between people living in wireless-only versus landline households could cause bias in telephone surveys. Target populations with a low prevalence of landlines (e.g. young, urban adults) may also be hard for researchers to access in telephone surveys [2,10].

Given these increasing challenges to telephone surveys, some researchers are reevaluating the utility of mailed questionnaires (MQs). Addresses are often easier to obtain than telephone numbers and are geographically defined. MQs can be answered at a participant’s convenience. Several studies conducted from the 1970s through 1990s found that response rates for mailed questionnaires were lower than for telephone surveys [11-13]. Due to changing trends in telephone usage and declining response rates [4,14], these studies may be less relevant to current survey research. A more recent randomized study of survey instruments showed improved participation rates for those who received a MQ versus a CATI [15]. In that study, the difference was most pronounced for women and younger adults, with women in their 30s being 19% more likely to complete a MQ than a CATI [15].

In comparing instruments, however, it is important to separate the effects of the survey mode from the effects of the recruitment methods. To determine the best survey mode for participation in a follow-up study to a large, multisite population-based study, we conducted a pilot study in which families were randomized to be recruited to complete either a mailed questionnaire (MQ) or CATI using similar recruitment protocols. The purpose of this pilot was to determine whether survey instrument (MQ or CATI) impacted participation rates in a subset of respondents to a previous telephone survey. The pilot also aimed to compare the response rates of mothers and fathers using both survey instruments, and to investigate the representativeness of participants for each instrument.

A total of 392 women were eligible for the HOEPS; a current address or telephone number was not identified for 51 (13%, Figure  1). Contact information could not be identified for more mothers of controls compared to cases (16.3% vs. 9.5%). In total, 341 women were invited to participate in the HOEPS and 268 (68.4%) completed participation. Among these women, 214 (79.9%) gave permission to contact the index child’s biological father. Valid contact information could not be identified for 5 of these fathers, resulting in 209 fathers who were invited to participate in the HOEPS. Of these, 147 (54.9% of all fathers) completed participation (Table  1).

Participation rates by instrument and case/control status (Table  2) were calculated only among those who were sent an invitation to participate in HOEPS (i.e., excluding those whose contact information we could not obtain, and fathers we were not given permission to contact). Among the 341 mothers invited to participate in the HOEPS, mothers of case and control infants had similar rates of participation (80.7% vs. 76.5%; p = 0.36; data not shown).Fathers of the 209 case and control infants who were invited to participate in the HOEPS also had similar participation rates (68.1% vs. 73.3%, p = 0.45; data not shown). Mothers recruited to the MQ, however, were more likely to participate than those recruited to the CATI (86.6% vs. 70.6% respectively; p < 0.001) (Table  2). Among case mothers, participation rates were higher for those assigned to the MQ (86.9%) versus those assigned to the CATI (74.7%); there was an even larger difference in participation rates between control mothers recruited to the MQ (86.2%) versus the CATI (66.3%) (Table  2).

Cohabitating with the biological father during the six months prior to conception through the end of pregnancy was reported by 86.2% of participating mothers. Mothers who reported living with the child’s biological father during this time period were more likely to give permission to contact the father, compared to mothers who did not live with the father for all of that time period (86.2% vs. 48.65%, p < 0.001; data not shown). Mothers who completed the MQ were less likely to give permission to contact the father compared to mothers who completed the CATI (75.0% vs. 85.8% respectively; p = 0.03; Table  2). Fathers invited to complete the MQ, however, had higher participation rates than those invited to complete the CATI (85.2% vs. 54.5%, Table  2); consequently more fathers participated with the MQ overall (n = 92) compared to the CATI (n = 55, Figure  2).

The distribution of demographic and pregnancy characteristics (maternal age, race, education, smoking status, parity, family income, and infant birthweight) was similar between mothers who participated in HOEPS and eligible mothers from the NBDPS (Table  3). Within every demographic strata we examined, participation rates were higher for mothers assigned to the MQ versus mothers assigned to the CATI (Table  3).

After receiving their introductory packets, the median time for mothers completing and returning the mailed questionnaire was 17 days; this was significantly (p < 0.05) shorter than the median time for completing the CATI (29 days, Table  4). Mothers assigned to the MQ also required significantly fewer reminders (median of 3 reminders versus 6, p < 0.05). Fathers participating in the MQ also completed their surveys in less time compared to fathers participating in the CATI (median 28 days vs. 17 days; p = 0.09), and required fewer reminders (median 3 vs. 6; p < 0.001) (Table  4). There was no significant difference between survey modes in time between birth of the index child and completion of the HOEPS (Table  3).

Overall, participation among recruited mothers was good in the HOEPS. Fathers invited to participate in the HOEPS also had high participation rates, unlike a previous study that found women were 10% more likely to complete an initial survey than men [19]. Fathers were lost to participation, however, when the mother could not be located, refused participation in HOEPS, or did not give permission to contact the father. Information provided from fathers directly may be more accurate than those reported by proxy, however [20-24]. Our results show that obtaining information from fathers, in addition to mothers, is feasible though participation rates may be lower when mothers are ‘gatekeepers’ to study participation. In studies of reproductive outcomes, mothers are often contacted first: mothers are the sources of information (or permission to medical records) about pregnancy and delivery characteristics, fathers may not be specified on birth records, and biological fathers may not be aware of the birth of the child. Since mothers who resided with the index child’s biological father six months before and during the index pregnancy were more likely to give permission for researchers to contact the father, this could be an important source of selection bias in evaluating paternal exposures in studies of birth outcomes where mothers act as gatekeepers.

Fewer mothers completing the MQ gave permission to contact fathers compared to mothers completing the CATI; this was not explained by underlying differences in the percentages of mothers who reported living with their child’s biological father during the entirety of B6-T3. Mothers who received the MQ had the opportunity to ask fathers if they wanted to be contacted before giving us permission; this may explain why mothers assigned to the MQ tended to give permission less often to contact the father, but the fathers were more likely to participate when contacted. During the CATI, mothers also had an opportunity to clarify what was expected of the father (including what information would be solicited and how long the interview would take). Sending both the maternal and paternal MQs to the mother at the same time might increase paternal participation; mothers could then choose whether or not to pass the paternal MQ to the father immediately if they share a residence.

Because the HOEPS was a follow-up among previous study participants, our population is probably more motivated and the overall participation rates we observed are likely higher than would be expected in a survey within the general population. This structure, however, created an excellent opportunity for comparing participation between instruments: we had good contact information (both telephone and mailing), allowing our recruitment protocols to be similar between both instruments; all families were drawn from the same population of NBDPS participants and randomly assigned to an instrument, removing most potential confounding/bias in a comparison of the instruments; and we had demographic information on eligible non-participants, allowing us to evaluate differences in participation rates between specific subgroups. Our design may have been somewhat biased towards high participation rates among those assigned to the CATI, because participation in the NBDPS requires completing a telephone interview. All those eligible for the HOEPS therefore had telephone access and a demonstrated willingness to provide information over the telephone.

We found that participation rates were higher for both mothers and fathers assigned to the MQ versus the CATI. Participants assigned to the MQ also required fewer reminder call cycles or letters before completing the study versus participants assigned to the CATI. As we expected, the largest impact of survey method on participation rates occurred among control subjects. Case subjects are generally more highly motivated than control subjects, thus we would expect the attractiveness of a survey method to have less weight in motivating participation among cases than controls. In all demographic strata we examined, participation rates were higher for mothers assigned to the MQ versus the CATI. The distribution of participating subjects across demographic groups, however, was similar to the distribution of eligible subjects. This suggests that mothers recruited to the HOEPS are likely representative of NBDPS participants, and that the increased response to the MQ was not attributable to a specific demographic subgroup.

Future studies that evaluate the use of a MQ should consider whether the increased cost of postage and printing for a questionnaire will be offset by savings in recruitment costs and increased participation rates. We found that the MQ saved personnel time (primarily due to fewer reminders) compared to the CATI, while postage and printing costs of the MQ were only slightly increased compared to those for the CATI. Overall, the mailed questionnaire was more cost-effective for this study, though following up on missing or confusing responses to the MQ required considerable time and effort whereas interviewers could immediately probe vague answers during the CATI. Complex surveys soliciting detailed answers may therefore be better suited to a CATI than a MQ.

There are several important limitations to this study. The increased participation rates we observed among those invited to the MQ versus the CATI may not be generalizeable to all groups; our population consisted primarily of the parents of young children who might have difficulty finding quiet time to complete a telephone interview or predicting when they will have free time. A paper questionnaire might be easier for these busy parents to fill out a little at a time when opportunities arise. In this study, we were also unable to compare the impact of survey type on the validity of respondents’ answers. Research from the 1970s and 1980s found that responses to MQs were generally equivalent to [25] or more valid than responses to CATIs [24]. A 1979 study of item omission between mailed and telephone surveys found that participants were more likely to answer sensitive questions on a mailed survey [13]. These studies, however, were undertaken in decades past and it may not be accurate to apply these results to current survey research; few similar studies have been conducted recently. We also did not evaluate the impact of offering more than one mode of survey instrument, which has been shown to improve participation rates [26,27]. As internet coverage increases, web-based surveys may also provide a practical and economical alternative.

Despite these limitations, this study provides a valuable comparison of survey instruments. Since subjects were randomized to the MQ or CATI, neither self-selection effects nor confounding is likely to affect the comparison of the two instruments. This is supported by the similar distribution of maternal demographic characteristics between mothers assigned to each instrument, as well as the similarities we observed between all those eligible for HOEPS and those who completed participation. We obtained a higher response rate with less recruitment effort using the MQ.

In a follow-up survey, we randomized families to be recruited to a MQ or a CATI. Both instruments contained the same information and similar recruitment protocols and materials. We obtained higher participation among mothers recruited to complete a MQ versus a CATI. This higher participation rate for the MQ held true in every demographic subgroup we examined. Although mothers who completed the MQ were less likely to give us permission to contact fathers, those fathers we contacted were more likely to participate compared to fathers recruited with the CATI; consequently we had higher overall participation from fathers using the MQ. While we found that it was very time-consuming to follow up on missing or confusing responses to the MQ, we also did not have to devote as much time to sending reminders for the MQ compared to the CATI. The cost-benefit ratio of one instrument versus another may vary depending on the target population and complexity of information being sought. Such information may guide other epidemiological studies, particularly those targeting the parents of young children, in deciding which mode of survey administration to implement.

The authors declare that they have no competing interests.

PAR conceived the study. All authors assisted in designing the survey instruments and/or recruitment protocols for the HOEPS. CMR prepared all IRB documents, assisted in participant recruitment/ data management, and performed data analysis. SHS oversaw recruitment and data collection. All authors contributed to writing and revising this manuscript. All authors read and approved the final manuscript.

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2458/12/579/prepub