EBFIs are effective at preventing and treating youth problems (Van Ryzin et al., 2016). Most of these programs were designed for specialty settings (e.g., schools, juvenile justice) that have limited reach to families in need (Kolko & Perrin, 2014). Delivering EBFIs in primary care could improve reach. In the United States, 92.9% of children have visited a primary care provider in the past year (Bloom, Jones, & Freeman, 2013), and about 15% of children seen in primary care have psychopathology (Williams, Klinepeter, Palmes, Pulley, & Foy, 2004). Parents trust their child’s pediatrician above all other sources for questions about psychosocial concerns (Polaha, Dalton, & Allen, 2011); however, when pediatricians refer them to mental health services, their attendance rate is low (Garland et al., 2005). Although primary care may increase access, most EBFIs have not yet been integrated in a realistic way. Leslie et al. (2016) and Rubio-Valera et al. (2014) indicate the primary care setting presents unique barriers to implementation, including physical space, referral pathways, and personnel to deliver the intervention. The translation of EBFIs therefore requires consideration of the interplay between system characteristics (e.g., organizational culture, physical structure, economics) and the core components of the intervention (Hoagwood & Kolko, 2009).

The Family Check-Up (FCU) is an EBFI for the prevention of problem behaviors and the improvement of child adaptation. It has been rigorously tested in diverse service systems, including public schools (Stormshak, Connell, & Dishion, 2009), home visitation (Dishion et al., 2008), and community mental health (Smith, Stormshak, & Kavanagh, 2015). Because of its brevity and flexible delivery format, the FCU is well suited for, but has not yet been tested, in primary care.¹

In this paper, two pilot studies of the FCU in primary care are used to illustrate the process of translating an EBFI to a novel setting using an IS framework and outcome measurement. The first author (Smith), in collaboration with Thomas Dishion (the FCU developer) and colleagues at Arizona State University (ASU), conducted a pilot feasibility study in two clinics (Montaño, Smith, Chiapa, Miloh, & Dishion, 2014): An outpatient pediatrics clinic serving adolescents ages 13 to 18 and a specialty clinic treating youth ages 6 to 18 diagnosed with non-alcoholic fatty liver disease. Both clinics were situated within a large urban children’s hospital serving predominantly low-income (55% Medicaid) and ethnic minority (≈65% non-white) patients. The second author (Polaha) and colleagues at East Tennessee State University (ETSU), in consultation with Thomas Dishion, conducted a pilot feasibility trial in a medium-sized, outpatient pediatrics clinic serving children ages 0 to 12 (Polaha, Smith, Smith, & Schetzina, 2015). This clinic, located in Southern Appalachia serves predominantly low-income (70.1% Medicaid) families. Both projects aimed to adapt the FCU for implementation in primary care, which required: (1) devising and installing a screening procedure to identify appropriate families; (2) shortening the time and number of contacts required to complete the FCU; and (3) developing a feasible plan to maximize access to FCU while limiting clinic disruption.

To direct project planning and guide the assessment of its effectiveness, the implementation team should prospectively employ an implementation framework. From the existing frameworks in implementation research (Tabak, Khoong, Chambers, & Brownson, 2012), we chose the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework (Aarons, Hurlburt, & Horwitz, 2011) because it efficiently yet comprehensively expresses the critical aspects of implementation through four distinct phases.

Within each phase, EPIS considers the influences of outer (service environment, interorganizational environment, consumer support) and inner context factors (intraorganizational and adopter characteristics) on the process. For example, during Exploration, agency leaders evaluate applicable policies and funding options (outer context). In the inner context, the organization’s capacity to implement depends upon culture and leadership, as well as the characteristics of the individuals delivering the program (e.g., values, goals). In practice, ongoing assessment allows for the EPIS phases to operate as a continuous feedback loop for corrective action (Gallo et al., 2016). For example, during the implementation phase, there may be a need to address low reach, poor fidelity, or emerging barriers (e.g., staff turnover, policy changes), requiring a return to earlier phases. Thus, EPIS can simultaneously guide the implementation process and inform the ongoing evaluation of critical milestones.

We now illustrate how the first three phases of the EPIS framework informed our FCU pilots and can be applied to EBFI implementation in primary care. Additionally, we describe what the Sustainment phase would look like had our pilot projects entered this phase. Table 1 provides more detailed information on the activities of each phase of EPIS and the process and outcomes assessed in each pilot project. The complete results of these projects are not presented in this article but can be found in Montaño et al. (2014) and Polaha et al. (2015).

One advantage of the EPIS model is that it focuses on factors that can be measured. Local contexts that apply a theory-guided implementation framework and outcome assessment strategy enables can produce generalizable knowledge. This is particularly true when combined with a sufficiently rigorous research design (Brown et al., 2017). Just as effect sizes from clinical trials can be compared when similar outcomes are targeted, so too can implementation outcomes be compared across projects with similar characteristics, as shown in our two examples.

We evaluated the success of our pilot trials using the Proctor et al. (2011) taxonomy of implementation outcomes, which are defined as the effects of deliberate and purposive actions to embed new interventions into real-world systems of care. Implementation strategy is the term given to the array of available actions (Waltz et al., 2015). The outcomes in the taxonomy are acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration, and sustainability. Figure 1 is a conceptual illustration that synthesizes the way the implementation framework, strategies, and outcomes are related. Their relation is integral for selecting the outcomes that are applicable to the aims of the implementation at different stages (Proctor et al., 2011). In Figure 1, EPIS guides the identification of barriers and the corresponding implementation strategies to address them. The next step is to select how measure the effect of the strategies on salient implementation outcomes. This varies within and across the four phases as depicted by the funnel arising out of EPIS. The Proctor et al. taxonomy also includes service and clinical outcomes to show the downstream effects of improving implementation outcomes. Implementation research can also include evaluation of service and clinical outcomes alongside the focus on the effects of the implementation (Curran, Bauer, Mittman, Pyne, & Stetler, 2012). Table 1 contains a description of what was measured and how during the first three phases of EPIS in the two pilot projects. For example, rigorously assessing acceptability was a higher priority for ASU compared to ETSU. This was because the ETSU pilot was conducted in a clinic that was already familiar with and accepting of providing behavioral interventions to families whereas the clinics at ASU were not.

Selecting implementation outcomes occurred during the Preparation phase in collaboration with key stakeholders. In doing so, we aimed to reduce reporting burden on the system and families by using electronic health record data and keeping surveys brief. We now discuss how these data were collected in our pilot projects. The outcomes are presented in the order in which they aligned with EPIS in these projects.

The pilot studies described in this paper had the luxury of grant support. Primary care practices without this funding will need to consider the upfront cost to get the program up and running, the ongoing costs of delivering it to families, and sustainable financing mechanisms (e.g., managed care reimbursement). The Affordable Care Act could expand reimbursement for EBFIs in primary care but this is not yet the case (Rawal & McCabe, 2016).

Using IS methods can increase the rigor of real-world practice change initiatives. In our case, the application of the EPIS framework provided a guide that allowed us to anticipate, measure, and preempt some barriers to implementation. At ETSU, the Preparation phase initiated an important dialogue among stakeholders regarding the impact of the FCU on space and existing behavioral health service delivery. We narrowed the scope of our pilot to address these concerns. The dialogue about how to implement the FCU was transparent and providers felt ownership going forward, which is reflected in our data showing adoption.

EPIS also gave us a structure onto which we are able to map our process in a replicable manner. Many projects aimed at integrating care lack this kind of framework and are therefore less generalizable and more difficult to publish. A “story” about how a program was implemented in one setting has only coarse meaning for implementation in another; however, organizing the story around a tested framework and assessing salient implementation outcomes allows for comparison. We measured essential indicators for understanding the success of embedding the FCU into primary care by engaging Proctor et al.’s taxonomy of implementation outcomes in synchrony with EPIS. For example, in the ASU trial, the team found that when the pediatrician simply referred families to the FCU, the likelihood of the family contacting the FCU therapist was low. So, the implementation plan was changed to incorporate a warm handoff between the pediatrician and FCU clinician to increase program enrollment. At ETSU, the outcomes indicated adoption of the screener and referral to FCU but lower penetration rates within the FCU. This led us to include social workers in home visits and to schedule longer well-visits for high risk families. We hypothesized that these strategies would address the barrier of attending a second clinic visit needed to complete the FCU. As translation of the FCU continues in primary care, the EPIS framework can be used to design implementation strategies in other “real-world” settings, such as clinics in health departments, family medicine or general internal medicine clinics. These settings will present unique challenges for implementation, such as a lower volume of pediatric patients.

The implementation science methods we describe are useful for those “in the trenches” who wish to pursue program development or who wish to scale up internal projects so that they contribute to the corpus of research in integrated care. Some key implementation data can be collected without undue burden making it within reach for providers with a desire to conduct research in their practice. Using IS methods increases the prospect of successful integration of evidence-based care by building investment, addressing barriers, and measuring the impact.