Current enrollees received a baseline survey via text on October 16 or 17, 2012, that asked, Are you planning to get a flu shot this year? There were three response options: Yes, No, and I already got it. Enrollees responding Yes or No were eligible to participate in the evaluation.

Participants responding to the baseline survey that they were planning to be vaccinated (“planners”) were randomly assigned to two groups (simple randomization scheme using the sum of the digits of their phone number [even/odd]). The “usual message” group received one encouragement message advising them to put a reminder on their calendar. The “enhanced messages” group received one encouragement message plus the opportunity to set up a general reminder (sent 2 weeks after receiving their response) or a specific reminder (sent 1 day before their chosen date) (Figure 1).

Participants responding at baseline that they were not planning to be vaccinated (“non-planners”) were randomly assigned using the same process. The “usual message group” received a general message stressing the importance of influenza vaccination. The “enhanced messages” group was asked why they were not planning to be vaccinated with five options: I think it may give me flu, Cost, Don’t think it’s safe, Don’t need it, and Other. An educational message tailored to the identified concern was sent to each participant who responded (Figures 1 and 2).

Planners who received specific reminders were sent the follow-up survey 1 week after receiving their reminder. All other participants were sent the follow-up survey in late November. The survey asked if participants received the vaccine, with three response options: Yes, No, and No, but planning to.

Information captured at enrollment in Text4baby includes

language preference;

Enrollees can update their due date and baby’s birthdate at any time. Enrollees not entering a birthdate by the expected end of their 41st week of pregnancy are auto-transitioned to receive postpartum messages and assigned a birthdate. Text4baby enrollees who reported their user type and did not identify as “pregnant” or “mother” were excluded from the study (4%).

In addition to baseline and follow-up survey data, the following data were available for participants. Type of participant (pregnant, mother of infant aged <1 year—hereafter referred to as “mother”) was calculated based on most recent due date or birthdate captured in the Text4baby database. Trimester (first, second, third) was calculated using date of last menstrual period based on due date at baseline. Age of infant (<6 months, 6–12 months) was calculated using birthdate at baseline. Poverty group (<20% poverty, ≥20% poverty) was assigned by matching participant’s ZIP code with ZIP code–level poverty data available through the American Community Survey (2011). Language (Spanish, English) was based on the language chosen at enrollment. Length of most recent enrollment was calculated using date of most recent registration relative to date of the baseline survey.

Univariate and multivariable regression analyses were conducted to determine the effect of:

enhanced reminder messages on “planners”; and

All analyses were stratified by type of participant at time of receipt of the enhanced or usual message because of differences in the sent messages. Predetermined subgroup analyses were performed according to stated reason for not planning to be vaccinated, and according to type of reminder (general or specific). Multinomial logistic regression models were used to assess the impact of enhanced messages on participants’ vaccination status at follow-up (in three categories: received vaccine, did not receive vaccine, planning to get vaccine). AORs and CIs were computed. Participant characteristics entered as covariates were length of most recent enrollment, trimester, age of infant, and language. Poverty group was considered only for the reminder intervention, given its relationship with the cost component of the tailored education intervention. Data analysis was performed in 2013–2014 using SAS/STAT, version 9.3. Values of p<0.05 were considered statistically significant.

Analysis of de-identified data was approved by Johns Hopkins IRB for this program evaluation. Because this evaluation is not an RCT, the CONSORT Electronic and Mobile Health Applications and online TeleHealth (CONSORT-EHEALTH) were not strictly followed.²⁷

Among planners, more than half (53%) of women who received the enhanced reminder messages (n=1,139) or the usual message (n=4,597) completed the follow-up survey (3,021/5,736). A total of 504 (9%) canceled their Text4baby subscription and were not eligible for the follow-up survey, and 2,211 (39%) did not respond (Figure 1). The enhanced reminder and usual message groups were similar across all characteristics for pregnant women and mothers, indicating that randomization balanced characteristics across groups and that loss to follow-up did not affect the balance (Table 1).

Four multivariable logistic regression models assessed the impact of the enhanced reminders compared with the usual message on self-reported receipt of influenza vaccination and continued intent to be vaccinated among women who planned, at baseline, to receive vaccination (Table 2). Two models were used to assess the impact of any enhanced reminder (both general and specific) on pregnant women and mothers. Among pregnant women, receipt of any enhanced reminder increased the odds of influenza vaccination at follow-up (AOR=1.5, 95% CI=1.0, 2.1) and continued intent to be vaccinated later in the influenza season (AOR=2.1, 95% CI=1.4, 3.1). Among mothers, receipt of any enhanced reminder had significant effects on influenza vaccination (AOR=2.0, 95% CI=1.4, 2.9) and continued intent to be vaccinated (AOR=1.7, 95% CI=1.1, 2.5). For pregnant women and mothers, there was no significant difference between the effect of reminders on receipt of vaccination or continued intent.

The other two models assessed the impact of enhanced reminder type (general or specific) compared with the usual message on receipt of influenza vaccination and continued intent to receive vaccination. Mothers who received a general reminder had higher odds of vaccination at follow-up (AOR=1.9, 95% CI=1.3, 2.8), and both pregnant women and mothers who received a general reminder had higher odds of continued intent (pregnant, AOR=1.8, 95% CI=1.1, 2.9; mother, AOR=1.8, 95% CI=1.2, 2.7). Pregnant women who received a specific reminder had increased odds of continued intent (AOR=3.1, 95% CI=1.4, 6.8); they also had higher odds of continued intent than actual vaccination (AOR=2.0, 95% CI=1.1, 3.3; data not shown).

Among non-planners, slightly less than half (46%) of women who received enhanced tailored education messages (n=3,725) or the usual message (n=4,591) completed the follow-up survey (3,820/8,316). A total of 968 (12%) canceled their Text4baby subscription and were not eligible to receive the follow-up survey, and 3,528 (43%) did not respond (Figure 1). The enhanced tailored education and usual message groups were similar across all characteristics for pregnant women and mothers (Table 1).

For women who did not plan to be vaccinated at baseline, receipt of the enhanced tailored education messages compared to the usual message was not associated overall with self-reported receipt of influenza vaccination or intent to receive vaccination (Table 3). There was also no significant difference between the effect of tailored education on receipt of vaccination or intent to be vaccinated. However, among mothers who reported they were not planning to be vaccinated because of cost, those who received a single tailored message about where to obtain low-cost or free vaccination had nearly two times higher odds of reporting vaccination at follow-up (AOR=1.9, 95% CI=1.1, 3.5). No other significant effects were observed at the individual message level for tailored messages that addressed other concerns.

This evaluation also has limitations. Given the non-response that occurred throughout the evaluation, these findings cannot be generalized to the overall Text4baby population or to other mHealth programs. Additionally, because the amount and type of information that can be collected by text are limited, and collecting too much data may deter enrollment, only very basic information is collected by Text4baby. Some standard demographic data such as education level and race/ethnicity are not collected. Further, all data are either self-report (e.g., vaccination status) or proxy data (poverty group, language, and birthdate for participants who were auto-transitioned) and were not independently verified. Other characteristics not assessed as part of this evaluation may have differed between the enhanced and usual message groups and between participants and those lost to follow-up, potentially impacting the findings. However, there were no notable differences in participant characteristics at baseline and for those who were lost to follow-up. Thus, the findings are internally consistent. Participants had to respond to one or more additional questions to receive enhanced messaging; thus, it is possible that these participants were those who were more favorably inclined toward vaccination. Additionally, a slightly higher proportion of Spanish language participants responded to the follow-up survey in the enhanced reminder group compared to the usual message group. However, the number of Spanish language participants is very small; future research should explore if there are differential effects for those who receive Spanish (versus English) messages. Finally, the design made it impossible to evaluate the end-of-season effects of enhanced messaging by collecting additional data after November 2012.

Given that mHealth is still in its infancy, rigorous research and evaluations are needed to understand factors contributing to successful, effective programs. As mHealth programs grow in popularity given their potential to improve healthcare access, behaviors, and outcomes, future considerations include

choosing research designs and comparison groups to answer specific questions (i.e., an external comparison group is needed to assess the net effect of Text4baby separately from the effectiveness of the messaging assessed here);

These evaluation results contribute to growing evidence that text-based reminders and tailored education can encourage positive health behavior, such as influenza vaccination, among vulnerable populations.