pmc971567221566Matern Child Health JMatern Child Health JMaternal and child health journal1092-78751573-6628349888631024972510.1007/s10995-021-03319-9HHSPA1902374ArticleDevelopmental Monitoring and Referral for Low-Income Children Served by WIC: Program Development and Implementation Outcomeshttp://orcid.org/0000-0001-7012-6344FarmerJanet E.15FalkLee Walker26ClarkMary J.16MayfieldWayne A.37GreenKatie K.4Department of Health Psychology, University of Missouri, Columbia, MO 65211, USAThompson Center for Autism and Neurodevelopmental Disorders, University of Missouri, Columbia, MO 65211, USAOffice of Social and Economic Data Analysis, University of Missouri, Columbia, MO 65211, USANational Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, 4770 Buford Hwy, Mail Stop S106-4, Atlanta, GA 30341, USAPresent Address: School of Health Professions, University of Missouri, 4323 E. Ravens Ridge Drive, Columbia, MO 65211, USAPresent Address: School of Health Professions, University of Missouri, 401 Lewis Hall, Columbia, MO 65211, USAPresent Address: Institute for Public Policy, University of Missouri, 137 Middlebush, Columbia, MO, USAJanet E. Farmer, farmerje@health.missouri.edu

Author Contributions JF provided conceptualization and oversight for program development/evaluation, and wrote the manuscript with input from all authors. LF, MC, and KG contributed to program design. LF coordinated on-site program implementation and evaluation. MC and WM contributed to data collection and data analysis.

2620232202206120220862023262230241Objective

To develop, implement, and assess implementation outcomes for a developmental monitoring and referral program for children in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).

Methods

Based on Centers for Disease Control and Prevention’s Learn the Signs. Act Early. campaign, the program was developed and replicated in two phases at 20 demographically diverse WIC clinics in eastern Missouri. Parents were asked to complete developmental milestone checklists for their children, ages 2 months to 4 years, during WIC eligibility recertification visits; WIC staff referred children with potential concerns to their healthcare providers for developmental screening. WIC staff surveys and focus groups were used to assess initial implementation outcomes.

Results

In both phases, all surveyed staff (n = 46) agreed the program was easy to use. Most (≥ 80%) agreed that checklists fit easily into clinic workflow and required ≤ 5 min to complete. Staff (≥ 55%) indicated using checklists with ≥ 75% of their clients. 92% or more reported referring one or more children with potential developmental concerns. According to 80% of staff, parents indicated checklists helped them learn about development and planned to share them with healthcare providers. During the second phase, 18 of 20 staff surveyed indicated the program helped them learn when to refer children and how to support parents, and 19 felt the program promoted healthy development. Focus groups supported survey findings, and all clinics planned to sustain the program.

Conclusions

Initial implementation outcomes supported this approach to developmental monitoring and referral in WIC. The program has potential to help low-income parents identify possible concerns and access support.

Developmental monitoringEarly identificationDevelopmental delay or disabilityWICHealth disparitiesIntroduction

Early detection is an essential first step in helping children with developmental delays and disabilities gain access to intervention services that promote their health and well-being. Timely developmental monitoring and screening substantially increases the likelihood of entry into early intervention services (Bethell et al., 2011; Lipkin, Macias, et al., 2020). Children who access early intervention services are more likely to show improvements than those children who receive services when they are older (Bruder, 2010; Rogers et al., 2012). Unfortunately, a significant gap exists between the number of young children with suspected developmental concerns and those children who are identified for early intervention services (Boh & Johnson, 2017; Jimenez et al., 2014; Rice et al., 2014). This gap is especially wide among children living in poverty and among children from racially and ethnically diverse groups. These sociodemographic factors have been associated with higher risk of developmental delay, lower screening rates, and less access to needed services (Durkin et al., 2017; Houtrow et al., 2014).

One potential strategy to improve access is to standardize approaches to developmental monitoring and referral in existing programs for low-income children. Centers for Disease Control and Prevention (CDC) took a step in this direction when it launched a national public health campaign in 2004 called Learn the Signs. Act Early. (LTSAE; Centers for Disease Control & Prevention, 2017). The campaign encourages parents, healthcare providers, and early childhood service professionals to engage in regular developmental monitoring using checklists that track whether children have achieved typical milestones at specific ages. Such monitoring complements formal developmental screening, which is conducted with standardized assessment tools administered by qualified professionals, and has been shown to support entry into early intervention services (Barger et al., 2018; Daniel et al., 2009).

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) is a program with potential to use LTSAE materials for parent education and developmental monitoring of young children from low-income families. This large-scale, federally-funded nutrition program serves pregnant, postpartum and breast-feeding women, infants, and children under the age of 5 years who are at nutritional risk and living in households with income less than 185% of the Federal Poverty Level (United States Department of Agriculture and Food and Nutrition Services, 2017). Participants return to WIC approximately four times each year to receive funds for specific nutritional foods, nutrition education, child monitoring, and referrals for health and social services. Although WIC focuses on physical health and growth, adequate nutrition is also essential for early brain development and the emergence of cognitive, linguistic, motor, and social milestones (Black et al., 2015).

The research literature includes only a few published reports about programs designed to promote early child development in WIC settings. In one study, child development services were delivered onsite during WIC visits in Los Angeles County, CA (Pinto-Martin et al., 2005). Program evaluation was not conducted due to the challenges of collecting data in busy WIC clinics, and program sustainability was uncertain due to its dependence on outside funding. Other researchers developed a curriculum for WIC staff to teach participants ways to communicate with health care providers about child development (Guerrero et al., 2013). Following training in curriculum use, WIC staff reported learning new information about child development and felt competent to conduct the class, potentially enhancing parent–provider communication about development. Another study found WIC staff were ready to interact directly with parents about typical child development and to make referrals to community-based services, but the staff had no systematic approach for doing so (Zuckerman et al., 2017). These researchers recommended that the WIC referral process for children with suspected developmental concerns be strengthened, but noted that previous attempts to forge these links had limited success due to lack of staff time and funding.

The purpose of the current project, called the WIC Developmental Milestones Program, was to overcome these barriers by partnering with WIC staff to create a developmental monitoring and referral program using LTSAE materials in WIC settings. The project was based on tenets of Implementation Science, an emerging field that investigates ways to promote successful uptake of empirically-supported public health and health care practices in community settings (Proctor et al., 2011). The first step in this research is to examine early-stage implementation outcomes in real world settings through measures of program acceptability (satisfaction with content/process), appropriateness (perceived fit, relevance), feasibility (actual fit, practicability), adoption (initial use), and fidelity (adherence). These outcomes are interrelated, influencing each other, but considered distinct from improvements in service outcomes (e.g., efficiency, effectiveness) and client outcomes (e.g., increased function, reduced symptoms). This paper describes the development and implementation (Phase I), replication (Phase II) and assessment of implementation outcomes in both phases of the WIC Developmental Milestones Program. The primary research question was, “Can LTSAE messages and tools be integrated into WIC clinics to support parent-engaged developmental monitoring and referral?”

MethodsOverview and Partnerships

The WIC Developmental Milestones Program was developed at WIC clinic sites in eastern Missouri in two phases, as shown in the Fig. 1 timeline. During Phase I, project staff at the University of Missouri partnered with WIC staff to shape and implement the program at 11 WIC clinics in urban St. Louis City, Missouri (2010–2012). Together they reviewed the LTSAE messages and materials, and identified ways to adapt and integrate them into existing WIC clinic workflow. State and district WIC leaders also provided input. The resulting program focused on communicating three key LTSAE messages to parents: (1) learn and track the signs of healthy development during early childhood, (2) act early if there is a concern, and (3) talk to your child’s doctor or primary care provider (PCP). Online Resource 1 summarizes implementation strategies and describes partnership-building activities.

During Phase II, university staff replicated the program at nine different WIC clinics in four eastern Missouri counties that included rural, suburban, and urban communities (2014–2016). These clinics were selected for their diverse community and clinic characteristics to help determine program generalizability to sites that were not involved in program development (Table 1).

Materials

During both phases, WIC staff used nine LTSAE developmental milestone checklists (ages 2 months through 4 years, in English and Spanish) to serve as monitoring tools and to help parents learn about their children’s development. Each LTSAE checklist included developmental milestones in the areas of language/communication, cognition, social/emotional functioning, and motor/physical development, with “red flags” indicating areas of potential concern. Referral envelopes were developed and used if a possible concern was identified, reminding parents about the need for follow up and providing language that parents could use to request an appointment. WIC clinics also installed colorful, family-friendly environmental graphics designed for this project (i.e., wall and floor decals) and distributed LTSAE handouts to reinforce key messages. These materials included contact information for other community agencies that offered developmental screening and assessment. Online Resource 2 provides a sample checklist and supplemental information.

Procedures

During Phase I program implementation, WIC staff invited parents of children ages two months through four years to complete age appropriate LTSAE checklists every six months during eligibility recertification and mid-certification visits (hereafter referenced only as recertification visits). Depending on WIC clinic preferences, checklists were distributed by nutrition staff (i.e., nutritionists, registered dieticians, registered nurses), clerical staff (e.g., eligibility certifiers), or both. Some clinics allowed parents to take the checklists home and return them at their next visit, but WIC staff reported low rates of return, so most dropped this approach. Once the checklist was completed, nutrition staff briefly reviewed it with the parent. If the checklist indicated that the child was reaching his/her developmental milestones, WIC staff encouraged the parent to share the checklist with the child’s PCP and talk about development during the next routine visit. If the checklist indicated potential developmental concerns (three or more missing checks on the milestones list or at least one red flag checked, for purposes of this project), nutrition staff placed the checklist in a referral envelope and asked the parent to follow up with the child’s PCP for screening and further assessment as soon as possible. LTSAE checklists are not standardized measures. The referral guideline was based on research-informed, expert opinion to give staff a way to decide when to refer. Online Resource 3 provides information about procedural decisions (e.g., determining referral guidelines).

Because program participation was voluntary, parents could choose not to participate and WIC staff could choose not to administer the program (e.g., due to high participant volume or practical challenges such as fussy children). Parents of children with developmental delays or disabilities previously diagnosed by health or educational professionals were not asked to complete checklists since these children did not need WIC developmental monitoring or referral.

Prior to the launch of the 12-month implementation, university staff presented a 30 to 45-min training session to nutrition and clerical staff during regularly scheduled WIC agency meetings. Training consisted of an overview of the project timeline and expectations, checklist protocol, and program evaluation. Two key program materials were reviewed in detail with nutrition staff: (a) Talking Points Guide, which provided simple step-by-step instructions for program administration and answers to common questions, and (b) Materials Toolkit, which was given to nutrition providers to organize and store LTSAE checklists and handouts for easy access. Role-playing exercises demonstrated how to introduce the project to parents, encourage parental involvement, complete and review the checklists, and provide feedback. Because workflow varied across clinics, WIC staff had flexibility to determine the best method to integrate the checklist protocol into their routine procedures.

The university coordinator installed environmental graphics immediately prior to program launch. The coordinator conducted informal clinic visits monthly and attended WIC agency meetings every two months during program implementation. These activities allowed the coordinator to monitor the integrity and fidelity of the program, track materials use, seek staff feedback, respond to questions, engage in problem solving about challenges, and share innovative program integration practices across WIC clinics. University staff also shared educational materials about the program with community health care, childcare, and early education providers to promote awareness and collaboration (Online Resource 1).

During Phase II, the 12-month program was generally the same as Phase I. However, implementation strategies were standardized to promote consistency in program administration, and several improvements were made based on lessons learned from WIC staff throughout Phase I. Checklist administration always occurred onsite during recertification appointments to encourage regular opportunities for checklist completion, though staff had the flexibility to administer the checklist any time a parent or staff member noted a possible concern, or at a later visit if time was not available. Only WIC nutrition staff reviewed checklists with parents since they typically facilitated referrals to children’s PCPs or other agencies and preferred to have oversight of this process. Referral criteria continued to include one or more checks in the “red flags” box, but for this study, the referral guideline increased from three to five or more missing milestones checks (Online Resource 3). All parents received the completed checklists, but referral envelopes were used only when parents were asked to follow up with their children’s PCP for developmental screening. Nutrition staff tracked referred children in their electronic records and followed up with parents regarding referral outcomes at subsequent visits. They could record the referral as a parent goal under Growth and Development as long as it was linked to nutrition goals.

Training procedures used in Phase I were supplemented with a written Implementation Manual that was distributed to all WIC staff.1 During follow up visits to WIC clinics, the university coordinator was often accompanied by the WIC Eastern District Nutrition Coordinator, a program champion who supported staff efforts to integrate developmental monitoring and referral into WIC clinics. University project staff sent program information packets to 188 PCPs who served children participating in Phase II WIC clinics to support successful referrals.

Program Evaluation

The evaluation assessed use of program materials and staff perceptions about the program using quantitative (survey) and qualitative (focus group) measures of implementation outcomes. Parent and child demographics collected during other Phase I and II studies provided a comparison of participant characteristics across phases. All aspects of program evaluation were reviewed and approved by University’s Health Sciences Institutional Review Board. Informed consent was waived because all assessments were voluntary and anonymous.

Participant Demographics

A consecutive sample of parents of children ages 12–47 months at each WIC clinic completed a brief written survey prior to Phase I (n = 160) and Phase II (n = 367). Six items measured parent demographic characteristics and child age in both phases, and group differences were compared using χ2 or t-test analyses (Table 2). One new Phase II item asked for the respondent’s relationship to the child.

Materials Use

During Phase I, the university project coordinator tracked the number of clinics with environmental graphics and the number of LTSAE checklists shared with parents at each clinic. During Phase II, the coordinator tracked graphics and checklists during implementation, and counted referral envelopes used during the first 6 months. Parents may have completed more than one checklist for their children and they could decline referrals, so checklist and envelope use does not correspond directly with number of children participating or referred. Materials use served as one estimate of program adoption.

Staff Surveys

During Phase I, a 15-item survey was used to assess WIC staff perceptions about program acceptability, appropriateness, feasibility for staff and families, along with program adoption and fidelity and respondent’s position/title. Survey items included four-point Likert scale (“Strongly agree” to “Strongly disagree”) and multiple-choice response options. All WIC staff members were invited to complete the paper-and-pencil survey (n = 35), which was administered at each of the 11 clinics after nine months of program implementation. Nineteen nutrition and seven clerical staff from 11 clinics completed the survey (n = 26; 74% participation). University staff conducted data entry and compiled descriptive statistics for WIC survey items. Likert scale responses were dichotomized into Agree/Disagree. Table 3 presents survey items and staff ratings, and indicates which of the implementation outcomes were most closely associated with each item based on Proctor et al. (2011) definitions.

After 12 months of Phase II implementation, WIC agency coordinators sent all nutrition staff an email inviting them to participate in a 21-item online survey (n = 21). Fifteen survey items were repeated from the Phase I survey with minor changes to improve clarity. Five new items were added to further assess fidelity and impact of the program on staff (Table 3), and one new item for agency affiliation. WIC nutrition staff (n = 20; 95% participation) from nine clinics completed the online survey. Only nutrition staff participated because they were primarily responsible for program administration during Phase II. Responses were collected and compiled using Research Electronic Data Capture (REDCap) tools hosted at the university (Harris et al., 2009). Phase I and II staff sample sizes were too small to detect statistically significant differences in survey responses, so comparisons were descriptive.

To monitor program fidelity during the Phase II implementation, the coordinator met with WIC nutrition staff during agency meetings every three months and asked each to complete a two-item “self-check” survey to assess: (1) how often the staff member handed out LTSAE checklists during certification appointments, and (2) how often the staff member reviewed the completed checklist with parents. These questions were the same as two fidelity items used on the staff survey at the end of Phase II.

Focus Groups

After the Phase II online survey closed, all WIC nutrition staff (n = 21) were invited to participate in one of five focus groups held at WIC clinics. The 1-h groups (n = 19; 90% participation) were led by a university evaluator who briefly summarized survey findings and then asked 10 open-ended questions about program strengths and barriers, impact on staff and perceived impact on parents, sustainability, and potential for statewide expansion. Each focus group was recorded with WIC staff permission, and data were analyzed using constant comparison analysis (Leech & Onwuegbuzie, 2007). Three university staff independently reviewed anonymous focus group recordings, took detailed written notes, and coded their notes into themes. Two of the staff then compared notes and codings to identify convergent themes, and finally all three met as a group to obtain consensus on key themes and sample quotes.

ResultsParticipant Demographics

In Phases I and II, most parents were younger than 35 years, and most Phase II respondents were mothers or stepmothers (Table 2). Compared to Phase II, Phase I inner city parents were significantly more likely to be non-white, identify as Hispanic/Latino, and report lower levels of education and household income.

Materials Use

During Phase I, WIC staff distributed more than 8600 LTSAE checklists to parents of young children at the 11 WIC clinics. Nine clinics installed the environmental graphics depicting developmental milestones, and two did not due to site restrictions on adhesive materials. During Phase II, staff distributed approximately 6000 checklists at the 9 WIC clinics, and nutritionists used 414 referral envelopes during the first 6 months of implementation. All clinics installed environmental graphics.

Staff Surveys

Most staff responses were similar in both phases (Table 3), even though the types of staff completing the surveys varied between Phases I and II. All staff reported that the checklists were easy to use. Most indicated that checklists fit easily into their workflow (≥ 80%), adding 2 to 5 min or less to WIC appointments. More than half of staff estimated they incorporated the checklists into WIC appointments 75% or more of the time. Nearly all staff (≥ 92%) indicated that they referred one or more children with a potential developmental delay. 12% of staff reported referring six or more children during Phase I compared to 60% of staff during Phase II.

When asked about parent response to the program, all staff who had environmental graphics installed at their clinics indicated families spent time looking at them in both phases. Furthermore, nearly all staff (≥ 95%) reported that parents were willing to complete the checklists and that the checklists were easy for parents to use. According to more than 80% of staff in both phases, parents indicated the checklists helped them learn about their children’s development and planned to share the checklist with their children’s healthcare providers. On items included only at Phase II, nearly all staff (≥ 90%) agreed that the program helped them learn how to support concerned parents and when to refer children, and also promoted children’s healthy development.

During Phase II implementation, 75% to 80% of nutrition staff participated at each of three fidelity self-check assessments. Approximately two-thirds of respondents (68% on average; 76%, 69%, 57% at each assessment) reported administering the checklists 75% or more of the time (data not shown). If a checklist was administered, 90% of staff reported reviewing it with the parent 75% or more of the time (data not shown). These findings were consistent with responses on fidelity survey items at the end of Phases I and II (Table 3).

Focus Groups

Table 4 depicts five themes derived from the Phase II focus groups. Nutritionists underscored the ease of program use, since the program was integrated into clinic flow.

They said that LTSAE checklists were key program components because they gave parents concrete, individualized information and empowered parents to talk with healthcare and other community service providers about their children’s development. Nutrition staff felt empowered by a decision tool to help them know when to refer children for screening and by improved collaboration with health care providers to identify and treat suspected concerns. They considered environmental graphics important because they promoted universal engagement of WIC participants regardless of age or language/literacy skills and made it easier to initiate a conversation with parents about milestones. Noting that seven of the nine clinics had staff turnover during the year, nutrition staff also reported the ease of training new staff in program administration using materials provided. Barriers to implementation included family characteristics (e.g., parent language/literacy differences, distractions, stress) and clinic characteristics (e.g., limits on time due to staff shortages and workload, space limitations). These difficulties were consistent with barriers reported in survey data across phases. Staff noted these issues also disrupt WIC nutrition services at times. Despite such challenges, all 20 clinics indicated a plan to continue the program, and nutrition staff participants in Phase II focus groups recommended that state WIC leaders explore opportunities to expand the program statewide.

Discussion

This project developed and implemented innovative strategies to support developmental monitoring and referral for a diverse group of low-income children receiving WIC services in eastern Missouri. The two-phase staff evaluation of implementation outcomes supported staff satisfaction with the program’s content and process, utility in helping staff know when to refer children with possible delays and how to support concerned parents, acceptance and ease of use of materials by families, and relevance to WIC’s goal of promoting healthy growth and development. The program also resulted in an acceptable level of protocol adherence and initial adoption, based on staff reports and materials use. Procedures were designed in partnership with staff and built on strategies already employed in WIC nutrition programs—parent education, child monitoring, and health referrals. This approach created a good fit for the WIC Developmental Milestones Program that contributed to the intention of all 20 WIC clinics to sustain the program, along with the recommendation for statewide expansion.

The ease and time efficiency of this voluntary program was critically important, given the main objective of WIC staff–parent interactions was to provide nutritional supports. The program used brief LTSAE checklists for milestone monitoring during biannual recertification visits, which limited the frequency of monitoring while providing regular opportunities to track child progress, celebrate healthy development, or refer if there were potential concerns. Other structural/organizational tools were designed to support effective time use and promote fidelity (e.g., Materials Toolkit, Talking Points Guide). Flexibility was built into the program so staff could adjust the core elements of the program to their own clinic flow and adapt the program to meet daily schedule demands and family needs. Not surprisingly, time constraints remained the biggest barrier to checklist administration in both phases. Given most staff were able to administer the checklists most of the time, the time barrier was usually overcome.

This early-stage program uptake was encouraging, given the previous lack of a systematic approach to developmental monitoring and referral reported by WIC staff and other researchers (Zuckerman et al., 2017). Previous WIC programs offered parents general education about child development and communication with health care providers (Guerrero et al., 2013). In contrast, this WIC program was specifically designed to engage parents in developmental monitoring and offered parents concrete, individualized education about their children’s milestones. The referral envelopes gave concerned parents a simple script to help them initiate a conversation about development with children’s PCPs. At the same time, LTSAE checklists gave WIC staff immediate access to evidence-based information about milestones for specific ages, helping them decide when to refer a child for screening. Ongoing contact with families allowed staff to assess parent and service provider response to the referral and offer additional LTSAE and community resources when needed.

Notably, many healthcare, education, and early childhood professionals are explicitly charged with responsibility for developmental monitoring and/or screening of young children (Bright Futures Periodicity Schedule Workgroup, Committee on Practice, and Ambulatory Medicine, 2014). A federal initiative called Birth to 5: Watch Me Thrive (U.S. Department of Health and Human Services and Administration for Children and Families, 2017) has been designed to coordinate these efforts, encouraging universal screening and timely supports for young children with potential delays. Yet gaps in early identification persist in the population of children served by WIC. The WIC Developmental Milestones Program has potential to complement and strengthen existing community-based systems of care for early detection in this vulnerable group of young children.

Limitations

Service system administrative datasets sometimes contribute to understanding implementation outcomes like program adoption, helping to keep data collection brief and non-intrusive in real world settings (Proctor et al., 2011). WIC administrative data only provided the caseload of participants receiving benefits, which overestimated the number of individual children seen and could not be used to determine the percentage of eligible children participating in developmental monitoring and referral (Online Resource 3). Future studies should explore methods to determine program adoption using measures such as actual number of children seen, those with completed checklists and those referred, number already diagnosed with developmental conditions, and number excluded due to language/literacy or other barriers.

The project evaluation was also limited by other challenges common to early-stage implementation research, including small sample size, reliance on self-report surveys, and lack of control groups. For instance, staff estimates of children referred in Phase II appeared higher than those referred in Phase I, but these findings were not analyzed statistically due to small sample size. Such results may be due to program standardization in Phase II, but also may be related to other factors such as differences in Phase I and II communities (urban vs. rural), participant characteristics (race/ethnicity, education), length of implementation prior to survey (9 vs. 12 months), or types of staff that administered the program and were surveyed across phases (all staff vs. nutritionists only). Rates of staff survey participation were relatively high, but they were not 100% and non-respondent characteristics were undetermined.

Despite study limitations, program implementation in two phases across WIC clinics in geographically and demographically diverse communities strengthens the conclusion that this approach may be feasible across Missouri and potentially in other states. In fact, since project completion, the program has been expanded statewide in Missouri, and the CDC has adapted the model for use in other states (https://www.cdc.gov/ncbddd/actearly/wic-providers.html). Such expansion will provide opportunities to clarify factors that affect program delivery and to investigate the program’s impact on children and parents served by WIC.

Supplementary MaterialAcknowledgements

Special thanks to Katie Dunne for contributions to initial program design and implementation, and to Missouri WIC leadership and Eastern District staff for their program contributions and dedication to children’s healthy development.

Funding

This study was supported by Health Resources and Services Administration (HRSA) Grant H6MMC1059-03-01 with supplement from the Centers for Disease Control and Prevention (CDC), 2010–2012; Disability Research and Dissemination Center (DRDC) through Cooperative Agreement Number 5U01DD001007 from CDC, 2014–2016. Contents of the publication are solely the responsibility of the authors and do not necessarily represent the official views of HRSA, DRDC or CDC.

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s10995-021-03319-9.

Implementation manual available upon request from the first author. An updated online implementation manual is available at https://www.cdc.gov/ncbddd/actearly/wic-providers.html.

Conflict of interest The authors have no conflicts of interest.

Ethical Approval All aspects of program evaluation in this project were reviewed and approved by the University of Missouri Health Sciences Institutional Review Board.

Informed Consent Informed consent was waived since all assessments were voluntary and anonymous.

Data Availability

Data and links to materials are available upon request.

ReferencesBargerB, RiceC, WolfR, & RoachA (2018). Better together: Developmental screening and monitoring best identify children who need early intervention. Disability and Health Journal, 11(3), 420–426. 10.1016/j.dhjo.2018.01.00229459217BethellC, ReulandC, SchorE, AbrahmsM, & HalfonN (2011). Rates of parent-centered developmental screening: Disparities and links to services access. Pediatrics, 128(1), 146–155. 10.1542/peds.2010-042421646266BlackMM, Perez-EscamillaR, & RaoSF (2015). Integrating nutrition and child development interventions: Scientific basis, evidence of impact, and implementation considerations. Advances in Nutrition, 6(6), 852–859. 10.3945/an.115.01034826875208BohA, & JohnsonL (2017). Universal screening to promote early identification of developmental delays: Exploring childcare providers’ beliefs and practices. Early Child Development and Care, 188(12), 1–15.Bright Futures Periodicity Schedule Workgroup, Committee on Practice, and Ambulatory Medicine. (2014). 2014 Recommendations for pediatric preventive health care. Pediatrics, 133(3), 568–570.24567012BruderMB (2010). Early childhood intervention: A promise to children and families for their future. Exceptional Children, 76(3), 339–355.Centers for Disease Control and Prevention. (2017, August 7, 2017). Learn the Signs. Act Early. https://www.cdc.gov/ncbddd/actearly/index.html.DanielKL, PrueC, TaylorMK, ThomasJ, & ScalesM (2009). ‘Learn the signs. Act early’: A campaign to help every child reach his or her full potential. Public Health, 123(Suppl 1), e11–16. 10.1016/j.puhe.2009.06.002DurkinMS, MaennerMJ, BaioJ, ChristensenD, DanielsJ, FitzgeraldR, …, & Yeargin-AllsoppM (2017). Autism spectrum disorder among US children (2002–2010): Socioeconomic, racial, and ethnic disparities. American Journal of Public Health, 107(11), 1818–1826. 10.2105/AJPH.2017.30403228933930GuerreroAD, InkelasM, WhaleySE, & KuoAA (2013). A WIC-based curriculum to enhance parent communication with healthcare providers. Journal of Community Health, 38(5), 958–964. 10.1007/s10900-013-9706-923760769HarrisPA, TaylorR, ThielkeR, PayneJ, GonzalezN, & CondeJG (2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381.18929686HoutrowAJ, LarsonK, OlsonLM, NewacheckPW, & HalfonN (2014). Changing trends of childhood disability, 2001–2011. Pediatrics, 134(3), 530–538. 10.1542/peds.2014-059425136051JimenezME, FiksAG, ShahLR, GerdesM, NiAY, PatiS, & GuevaraJP (2014). Factors associated with early intervention referral and evaluation: A mixed methods analysis. Academic Pediatrics, 14(3), 315–323. 10.1016/j.acap.2014.01.00724767785LeechNL, & OnwuegbuzieAJ (2007). An array of qualitative data analysis tools: A call for data analysis triangulation. School Psychology Quarterly, 22(4), 557–584. 10.1037/1045-3830.22.4.557LipkinPH, MaciasMM, Council on Children with Disabilities, Section on Developmental and Behavioral Pediatrics. (2020). Promoting optimal development: Identifying infants and young children with developmental disorders through developmental surveillance and screening. Pediatrics, 145(1), e20193449. 10.1542/peds.2019-344931843861Missouri WIC Database. (2010). Fiscal year report of monthly participation. Missouri Department of Health and Senior Services.Missouri WIC Database. (2013). Fiscal year report of monthly participation. Missouri Department of Health and Senior Services.Pinto-MartinJA, DunkleM, EarlsM, FliednerD, & LandesC (2005). Developmental stages of developmental screening: Steps to implementation of a successful program. American Journal of Public Health, 95(11), 1928–1932. 10.2105/AJPH.2004.05216716195523ProctorE, SilmereH, RaghavanR, HovmandP, AaronsG, BungerA, …, HensleyM (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.20957426RiceCE, Van Naarden BraunK, KoganMD, SmithC, KavanaghL, StricklandB, & BlumbergS (2014). Screening for developmental delays among young children—National Survey of Children’s Health, United States, 2007. MMWR: Surveillance Summaries, 63(Suppl 2), 27–35.RogersSJ, EstesA, LordC, VismaraL, WinterJ, FitzpatrickA, …, DawsonG (2012). Effects of a brief Early Start Denver model (ESDM)-based parent intervention on toddlers at risk for autism spectrum disorders: A randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 51(10), 1052–1065.10.1016/j.jaac.2012.08.00323021480U.S. Department of Health and Human Services, and Administration for Children and Families. (2017, January 26, 2017). Birth to 5: Watch Me Thrive! https://www.acf.hhs.gov/ecd/child-health-development/watch-me-thriveUnited States Census Bureau. (2010). Quick Facts—Missouri-population estimates [Tables: Population, race, income and poverty]. https://www.census.gov/quickfacts/fact/table/stlouiscitymoUnited States Department of Agriculture, and Food and Nutrition Services. (2017, 05/04/17). Frequently asked questions about WIC. Women, Infants and Children (WIC). https://www.fns.usda.gov/wic/frequently-asked-questions-about-wic#3.ZuckermanKE, ChavezAE, & ReederJA (2017). Decreasing disparities in child development assessment: Identifying and discussing possible delays in the special supplemental nutrition program for Women, Infants, and Children (WIC). Journal of Developmental and Behavioral Pediatrics, 38(5), 301–309.28520635

Timelines for Phase I (development and implementation) and Phase II (replication)

Community and clinic characteristics of WIC programs participating in Phases I and II

Phase IPhase II
Community characteristics
  Geographic locationSt. Louis City, MOFour counties in eastern MO
  Population sizea319,3714-County mean 419,917Range 101,492–998,954
  Racial minorities (non-white)a54.1%4-County mean 12.1%Range 3.4–31.4%
  Percent living in povertya25%4-County mean 9.1%Range 5.3–10.7%
Clinic characteristics
  Number of participating WIC clinics11 of 11 clinics in St. Louis City (100%)b9 of 9 clinics in four county area (100%)
  Administrative oversight3 Private, not-for-profit agencies4 County public health and community service agencies
  WIC clinic staff35 Total19 Nutritionists7 Clerical, 8 unspecified38 Total21 Nutritionists17 Clerical, 0 unspecified
  Average Monthly Caseload for each WIC clinicc,e806 Children (1091 Total) receiving WIC benefits1281 Children (1731 Total) receiving WIC benefits
  Range 443–1466 Children (595–2003 Total)Range 749–1838 Children (1111–2387 Total)
  Average Monthly Caseload for all WIC clinicsd,e8861 Children (11,996 Total) receiving WIC benefits11,527 Children (15,576 Total) receiving WIC benefits

Children infants and children < 5 years, Total women, infants, and children < 5 years

Characteristics of geographic location (United States Census Bureau, 2010). Poverty is defined as < 100% of the federal poverty level

Initially there were four agencies with 16 sites in St. Louis City. One agency shut down shortly after the program began, and WIC services consolidated into three agencies with 11 sites that participated in Phase I

Average number of WIC participants receiving benefits per month by participating clinics during Fiscal Year (FY) 2010 (Phase I) and FY 2013 (Phase II). (Missouri WIC Database, 2010, 2013)

Average number of WIC participants receiving benefits per month summed across participating clinics during FY 2010 (Phase I) and FY 2013 (Phase II). (Missouri WIC Database, 2010, 2013)

WIC did not track actual number of participants seen each month in state administrative datasets. Average monthly caseload overestimates the number of participant visits to clinic, but is an indicator of WIC benefits recipients each month and range across clinics

Comparison of participant demographics for Phases I and II of the WIC Developmental Milestones Program

CharacteristicPhase I (n = 160)Phase II (n = 367)Test results
Parent age (< 35 years)83%84%χ2(1) = .0003, p = .987, ns
Mother or Stepmother–95%–
Racial minorities (non-white)71% (Black = 68%, Other = 3%)41% (Black = 35%, Other = 6%)χ2(1) = 12.01, p < .001
Hispanic/Latino12%2%χ2(1) = 22.53, p < .001
Education (high school or less)75%40%χ2(1) = 16.15, p < .001
Household income (< $15,080)70%48%χ2(1) = 6.14, p = .013
Mean child age in months (SD)21.4 (9.46)21.7 (9.94)t(525) = 0.39, p = .69, ns

Parents of children ages 12–47 months provided demographic data at the beginning of Phases I and II

WIC staff survey responses for Phases I and II

Survey items (implementation outcomes assessed)Phase I (n = 26)a
Phase II (n = 20)a
Agree (SA, A)Disagree (D, SD)Agree (SA, A)Disagree (D, SD)
The developmental milestones checklists are easy to use. (AC, FE)100% (54, 46)0% (0, 0)100% (50, 50)0% (0, 0)
The toolkit (black crate filled with program materials) is easy to use. (AC, FE)100% (58, 42)0% (0, 0)100% (55, 45)0% (0, 0)
The checklists fit easily into the flow of certification appointment. (AC, FE)89% (23, 65)12% (8, 4)80% (20, 60)20% (20, 0)
The referral process fits easily into the flow of certification appointments. (AC, FE)––100% (15, 85)0% (0, 0)
How much extra time does the program add to certification appointments? (AC, FE)
  Less than 2 min35%10%
  2–5 min50%70%
  More than 5 min15%20%
How often have you been able to ask parents to complete the checklist during certification appointments? (AD, FI)
  Almost always35%35%
  About 75% of the time31%20%
  About 50% of the time23%35%
  About 25% of the time12%10%
  Almost never0%0%
How often are you able to review the checklist after it is completed? (AD, FI)
  Almost always–80%
  About 75% of the time10%
  About 50% of the time5%
  About 25% of the time5%
  Almost never0%
Which materials helped families understand the importance of monitoring their children’s developmental milestones the most? (AP)
  Checklists58%60%
  Referral envelopes4%20%
  Wall and/or floor blocks27%15%
  Parent handouts12%5%
About how many children did you refer to the doctor using the referral envelop provided? (AD, FI)
  0 Children8%5%
  1–5 Children81%35%
  6–10 Children4%30%
  More than 10 children8%30%
What has been the biggest barrier to implementing the program? (AC, AP)
  Lack of time during WIC appointments42%60%
  Lack of interest from families27%20%
  Materials not available in languages other than English and Spanish19%15%
  Materials too difficult for some English-speaking parents4%–
  No barriers4%–
  Other: Need for more training about checklist items–5%
Families are willing to complete the developmental milestones checklists. (AC, FE)96% (35, 62)4% (4, 0)95% (40, 55)5% (5, 0)
The developmental milestones checklists are easy for families to understand. (AC, FE)96% (23, 73)4% (4, 0)95% (25, 70)5% (5, 0)
Families look at the environmental graphics (floor/wall blocks). (AC, AP)100%b (67, 33)0% (0, 0)100% (65, 35)0% (0, 0)
Families indicate that checklists help them learn about their children’s development. (AC, AP)81% (27, 54)19% (15, 4)80% (20, 60)20% (20, 0)
Families indicate that they will share the checklists with their children’s healthcare providers. (AC, AP)81% (15, 65)19% (15, 4)85% (15, 70)15% (15, 0)
Program helped me learn about when to refer children with possible developmental delays. (AC, AP)––90% (35, 55)10% (10, 0)
Program helped me learn about how to support parents when they are concerned about their child’s development. (AC, AP)––90% (40, 50)10% (10, 0)
Program promotes children’s healthy development. (AP)––100% (65, 35)0% (0, 0)

All values in the table are percentage of respondents. Percentages may not sum to 100 due to rounding within item responses

Dash (−) indicates item not asked during this Phase. Data with Agree/Disagree responses were dichotomized for ease of reporting across items, but the non-dichotomized data are also presented in parentheses

Survey response abbreviations: SA Strongly Agree, A Agree, D Disagree, SD Strongly Disagree

Implementation outcomes abbreviations: AC Acceptability, AD Adoption, AP Appropriateness, FE Feasibility, FI Fidelity

In Phase I, respondents included 19 nutrition and 7 clerical staff (26/35 = 74% participation). In Phase II, all 20 respondents were nutrition staff (20/21 = 95% participation)

During Phase I, two of the 11 clinics did not install graphics due to building restrictions on adhesive materials, so only 24 staff responded to this question

Qualitative data from staff focus groups following Phase II of the WIC Developmental Milestones Program (n = 19)

ThemeSample quotes
Program is easy to implementThe checklists are self-explanatory. The directions are on there [Talking Points document] and it tells you. Once you get going, then you’re rollingMost parents had no problems completing the checklist…and many were enthusiastic about receiving it because of questions about their child’s development [Graphics] helped to engage families in conversation about children’s development
Parents are empoweredIt’s good [for parents] to catch something early in their children’s developmental process, to get them to services earlier, instead of waiting till they are in kindergarten and they are already really behindChecklist gives the mom something concrete to refer to when they go to the doctor
Doctor referrals are strengthened[Checklist] gave us a list of what to look for and provided an approach with parents. It didn’t put it all on us. It gave us some specifics for parents and doctors, tooSome children who needed help, [parents] initiated conversation with paediatricians and are now receiving services
Staff are empowered[Checklist] helped WIC staff to know when the child needs a referral. When I was in school, we didn’t get much about child development or it was so minimal, and it’s been years ago that you forget. It’s actually a good way to be more aware of where they should be at different agesThe more we work with [parents] and meet them where they’re at, they do come to us with questions they may not bring up to their doctors. It [milestones program] builds our credibility with them
Barriers to administrationTimewise, we have a pretty full clinic flowParents under stress don’t think very far aheadParents who speak limited English had more issues with completing the checklist, but this factor impacts all parent–WIC staff interactions
SignificanceWhat is already known?

Children from low-income families may be at increased risk for developmental concerns and face disparities in access to early identification and intervention services.

What this study adds?

This study describes the acceptability, appropriateness, feasibility, adoption, and fidelity of a promising approach to the integration of developmental monitoring and referral in WIC clinics, potentially enhancing the likelihood of early detection and intervention among young children from low-income families served by WIC.