The development of vaccines has been credited with the prevention of more than 100 million cases of contagious diseases since 1924 (van Panhuis et al., 2013). In spite of this success, new cases of vaccine-preventable disease occur each year. There were more than 28,000 confirmed cases of pertussis reported in the United States in the year 2014 alone (“2014 Provisional Pertussis Surveillance Report,” 2015), and from September 28, 2014–May 23, 2015, more than 125,000 confirmed cases of seasonal influenza (Appiah et al., 2015). Approximately 28,000 adults are hospitalized for influenza-related critical illness annually (“Estimates of deaths associated with seasonal influenza --- United States, 1976–2007,” 2010). In addition to the human impact, the persistence of vaccine-preventable disease has a significant financial toll. One recent study estimated that for the population of adults age 50 and older in the U.S., the annual economic burden attributed to four vaccine-preventable diseases (influenza, pneumococcal disease, shingles, and pertussis) was $26.5 billion. These estimates included medical costs per case related to diagnoses and treatment as well as indirect costs, such as work loss and lost productivity (McLaughlin, McGinnis, Tan, Mercatante, & Fortuna, 2015).

In spite of these known impacts, immunization rates remain low. The rate of the combination tetanus, diphtheria, and acellular pertussis vaccine (Tdap) vaccination among adults aged 19 or older is just 17.2% (Williams et al., 2015). Rates for influenza immunization are only marginally better but increase with age: 32.3% for those aged 18–49, 45.3% in those 50–64 years, and 65.0% for adults aged 65 and older (CDC, 2013). Numerous barriers to immunization have been well documented. Consumer barriers include concern about side effects (Johnson, Nichol, & Lipczynski, 2008; McIntyre, Zecevic, & Diachun, 2014), low perceived risk of contracting disease (Johnson et al., 2008; McIntyre et al., 2014; Miller, Kretsinger, Euler, Lu, & Ahmed, 2011), low collective awareness of the availability of the vaccine (Johnson et al., 2008), and limited recommendations by medical providers (Johnson et al., 2008; McIntyre et al., 2014; Miller et al., 2011). Healthcare professionals perceive that patient barriers to immunization include inadequate healthcare coverage, patients’ fear of needles, and patients’ failure to attend well-care visits (Johnson et al., 2008). Provider and practice-level characteristics are also associated with low immunization rates and include the lack of adequate reminder systems (Johnson et al., 2008) as well as financial barriers and inadequate reimbursement, which specifically limits the abilities of small practices to stock and supply vaccines to their patients (Hurley et al., 2014).

A number of evidence-based interventions have been shown to increase immunization rates in primary care settings. The Community Preventive Services Task Force organizes these strategies into three key areas: (1) enhancing access to vaccination services, which includes providing immunizations in convenient settings and reducing out-of-pocket costs; (2) increasing demand among patients, via patient reminders and community education; and (3) provider- or system-based interventions such as reminders, modified work flow, standing order programs, or electronic immunization tracking (“ Guide to Community Preventive Services,”). Although these strategies are known to be effective, they are not consistently implemented (Stinchfield, 2008). The process of implementing change within primary care settings is often complex and requires a range of facilitative supports (Crabtree et al., 2011).

The 4 Pillars^™ Practice Transformation Program (formerly known as the 4 Pillars^™ Immunization Toolkit) was designed to support practices in implementing recommended immunization strategies. The 4 Pillars^™ Program is a web-based practice improvement program and transformation dashboard based on the implementation science framework of Fixsen et. al. (D. L. Fixsen, Naoom, Blase, & Friedman, 2005), which emphasizes staff selection, pre-service training, coaching, evaluation, and development of facilitative supports. The 4 Pillars^™ Program provides background information about adult immunizations emphasizing their importance, adverse reactions, and healthcare coverage of immunizations, as well as evidence-based solutions framed within the following four pillars: (1) convenient vaccine services, including extending the season for influenza vaccination; (2) patient communication; (3) enhanced office systems, including standing order programs; and (4) motivating staff through an “Immunization Champion.”

An early version of the 4 Pillars^™ Program was shown to improve immunization rates, especially among high-risk adults and in practices that successfully implemented strategies across all four pillars (Nowalk et al., 2016; Nowalk et al., 2014)¹. The 4 Pillars^™ Practice Transformation Program has since been improved and digitized, and tested in a randomized controlled cluster trial. The intervention was based on the RE-AIM evaluation framework, which suggests that the impact of an intervention is a function of its Reach, Efficacy, Adoption, Implementation, and Maintenance (Belza, Toobert, & Glasgow, 2007). Thus, RE-AIM provides an evaluation framework that improves interpretation of findings in applied settings, extending evaluation domains beyond a priori primary outcomes, such as changes in immunization rates, to include broader evaluation domains such as practice setting and adoption (Kessler & Glasgow, 2011).

Results from the randomized cluster trial are not included herein; however, changes in rates varied across sites. This variability was not fully explained by intervention efforts and is consistent with previous research, which documents the fact that even when evidence-based strategies explain what must occur to improve public health outcomes, there is a significant gap in the practice of how change occurs (D. Fixsen, Scott, Blase, Naoom, & Wagar, 2011). The context in which change occurs is a critical factor when implementing evidence-based approaches to care (Tomoaia-Cotisel et al., 2013).

Herein we share findings from a mixed methods evaluation of the intervention’s implementation, including the use of the 4 Pillars^™ Program and its impact on immunization rates within primary care practices, and describe characteristics of a practice environment that are conducive to effective practice change. The mixed-methods approach provides valuable detail about the setting and context in which implementation takes place and therefore improves interpretation of results (Klassen, Creswell, Plano Clark, Smith, & Meissner, 2012).

The study was approved by the Institutional Review Boards of the two universities and one Texas health system that collaborated on this research. This randomized cluster trial was conducted in 2013–2015 in twenty-four primary care practices that had a majority of adult patients, baseline immunization rates for at least one adult vaccine <50%, and a willingness to participate in the study to improve vaccination rates. Nineteen of the practices were located in Southwestern Pennsylvania and affiliated with a large, private integrated healthcare delivery system. The remaining 6 practices were located in Houston, Texas and were part of a county health system that contracted with a nearby health sciences university to provide physician care within these safety net clinics.

Twelve practices received the intervention in Year 1 (2013–14) (one practice dropped out in Year 1) and twelve practices serving as control sites during Year 1 received the intervention in Year 2 (2014–15). Four practices which took part in the active intervention in Year 1 elected to maintain active engagement in the study during Year 2. This study focuses on the eleven practices that completed the intervention in Year 2 and three of the Year 2 re-intervention practices. One Year 2 re-intervention practice was not able to schedule an interview within the dedicated study period.

The intervention included an initial visit to each site by one of the study investigators to introduce the study and the 4 Pillars^™ Program and to identify site-specific strategies for implementation. Practices identified an Immunization Champion to be responsible for interacting with the web-based 4 Pillars^™ Program to guide strategy implementation. Other roles for the Immunization Champion included biweekly telephone-call coaching with the research liaison to ensure that chosen strategies were being implemented. The practices were given bi-weekly graphs delineating their progress toward immunization goals.

Influenza and Tdap immunization data were collected using de-identified information from practices’ electronic medical records from January 2012 through January 2015. The population consisted of all patients ≥18 years of age with at least one visit to the practice during each year of the study. Immunization rates were calculated using the patient census as the denominator and the number of patients who received influenza vaccine as the numerator. Tdap vaccination was calculated as a cumulative value for each year. Because three of the practices were held over to receive a second year of the intervention, the differences in rates were calculated for each practice based on the first year it received the intervention versus the year immediately preceding the intervention.

Paired samples t-tests (alpha <.05) were conducted in SPSS 22 to examine changes in immunization rates by type of practice.

Qualitative data were collected with the primary goal of understanding implementation success, specifically, how implementation took place and why differential rates of implementation occurred. RE-AIM was used as an organizing framework. In Year 1 of the study, qualitative interviews had been conducted with each of the 13 practices that received the intervention that year, in order to assess the extent to which 4 Pillars^™ strategies were implemented, or the degree to which implementation took place. Findings from Year 1 observations also informed the interview protocol for Year 2 of the qualitative evaluation.

In Year 2, the aim of the qualitative evaluation was to understand how implementation took place and why differential rates of implementation occurred. During the winter of 2014–2015, a qualitative researcher visited all the Year 2 intervention sites for onsite observation and individual or small-group interviews with practice staff members such as ICs, physicians, medical assistants, practice managers, and other team members identified by each practice as being integral to immunization efforts. A semi-structured protocol (Table 1) was used to assess barriers to and facilitators of implementation within the context of the reach, effectiveness, adoption, implementation, and maintenance of the 4 Pillars^™ Program strategies. Interviews were audio-recorded and subsequently reviewed for thematic analysis by two qualitative researchers. These themes were then used to develop systematic classification into groups to describe practice characteristics and explain the level of implementation of the intervention.

Four practice characteristics were identified from the thematic analysis of the interviews as being important to intervention implementation, namely, degree of quality improvement history, communication and practice leadership, Immunization Champion leadership effectiveness, and organizational flexibility. A scoring system was developed in which each practice was ranked by the researcher who conducted the interviews as being low (score = 1), medium (2), or high (3) in relation to each of these four practice characteristics, and the scores were summed across characteristics. Thus, the lowest possible score a practice could receive was 4, and the highest possible score was 12, with high scores indicating high readiness for success in implementing practice change for quality improvement. A fifth characteristic that was included was system affiliation; the practices in Pennsylvania and Texas differed significantly in governance, culture, and size. Members of the intervention team who had provided 4 Pillars^™ Program orientation and support and were familiar with practice characteristics also assigned scores to each practice in order to triangulate results, i.e., provide multiple perspectives.

Each of the practices was then examined for implementation success, stratified by the five-level practice classification system. Implementation success was assessed by the qualitative researcher, who conducted the site observations and documented the number of strategies that were implemented as well as degree of implementation. Implementation success was also scored by the intervention team. High Implementer practices, with the highest practice characteristic scores, implemented the most 4 Pillars^™ Program strategies, with the fullest degree of uptake. Staff members at these practices could readily describe the strategies that were implemented as well as the impact that they had on immunization rates. Low Implementer practices implemented the fewest 4 Pillars^™ strategies, and strategies were only superficially implemented. For example, Low Implementer practices that implemented standing order programs had staff members who could not accurately describe how standing orders worked, or which staff members were able to give immunizations. Mid-Implementer and Public/University practices demonstrated mid-range implementation success.

The four practice characteristics that were scored for grouping practices into types were as follows.

Four of the practices were assigned as Low Implementers, four of the practices were assigned as Moderate Implementers and four of the practices were assigned as High Implementers. The split loyalty of the University/Public practices made implementing changes in office systems difficult to manage, because physicians expressed that they had limited freedom to implement staffing or practice-wide policies. Even though these practices frequently demonstrated high degrees of quality improvement experience, two-way communication patterns, motivated Immunization Champion leadership, and organizational flexibility, these characteristics were dominated by the health system structure thus did not always effectuate practice change.

Some 4 Pillars^™ Program strategies occurred more frequently within specific practice types. For example, High Implementer and Public/University practices were more likely than Moderate- and Low Implementer practices to have fully implemented standing order protocols, as well as to have Immunization Champions with clinical responsibility and authority within the practice setting. Moderate- and Low Implementer practices were less likely to have fully adopted standing order protocols or to have Immunization Champions with authority, who were less able to motivate the team to improve immunization rates. Also, staff members from High Implementer and Public/University practices more frequently reported that the progress graphs they received were motivational and inspired a sense of “healthy competition,” because they wanted to outperform other practices. Moderate- and Low Implementer practices were more likely to express doubts about the accuracy of the data in the progress graphs, or to not use them at all. Table 3 shows examples of 4 Pillars^™ Program strategies that were evaluated using the RE-AIM framework with examples of each of the practice types.

Practice type was related to changes in immunization rates for influenza and Tdap. At baseline, mean vaccination rates for both influenza and Tdap were lowest among Low Implementers and highest among High Implementers (Table 4). At the end of the study period, High Implementer practices significantly increased average influenza uptake (3.0 percentage point difference; p = .038) and average Tdap vaccination rate (9.3 percentage point difference; p=.006) and Public/University practices significantly increased average Tdap vaccination rate (6.5 percentage point difference; p=.012), while Moderate and Low Implementer practices did not significantly improve rates for either vaccine.

Because of its value in predicting implementation success and public health impact, the use of RE-AIM in translational research has grown consistently since its development, with at least 144 published studies citing use of this framework in the past 14 years (Shoup, Gaglio, Varda, & Glasgow, 2015). In keeping with the RE-AIM model, primary care practices in this study were supported through the intervention using the 4 Pillars^™ Practice Transformation Program, which included approaches such as improving patient notification and accessibility of immunizations (Reach); implementing evidence-based interventions including standing order protocols (Effectiveness); increasing the number of staff members who were skilled at delivering 4 Pillars^™ Program strategies and creating an Immunization Champion role (Adoption); supporting the use of site-specific immunization strategies via conference calls and an online dashboard to track progress (Implementation); and motivating staff by sharing progress towards goals via monthly progress charts (Maintenance). The 4 Pillars^™ Program is designed to be adaptable to a wide range of practices because strategies can be chosen and modified to fit the unique culture and structure of each. Given the complexities associated with practice-level change (Crabtree et al., 2011) and the fact that practices do not uniformly achieve desired increases in adult vaccination rates, an in-depth examination of the process and degree of implementation of the intervention was warranted.

This qualitative evaluation provided a critical view of the participating practices’ barriers and facilitators to implementation of toolkit strategies. This approach is supported by previous implementation studies that demonstrate the need to qualitatively evaluate the process for change in conjunction with a priori quantitative target outcomes (Balasubramanian et al., 2015). By using the RE-AIM framework to examine implementation success and the resulting changes in immunization rates by practice type, it is apparent that pre-existing practice characteristics may help to explain implementation success. Practices with histories of outcomes-based care and that demonstrated open communication patterns, organizational flexibility, and motivational and involved Immunization Champion leadership implemented a greater number of 4 Pillars^™ Program strategies than other practices, and these strategies were more likely to be fully implemented. Although practices affiliated with the University/Public setting scored similarly to the High Implementers, their success with implementing the Toolkit and changes in vaccination rates were less pronounced. System characteristics appeared to outweigh practice characteristics.

This study has several limitations. First, generalizability of results is limited by the fact that only 14 primary care practices were studied, and parsing these practices into four different types limits power for the quantitative analysis. In addition, on-site observations and qualitative interviews were conducted by one researcher. This limitation is partially mitigated by the fact that the scoring of practices was also conducted by members of the implementation team who were also familiar with the participating practices in order to triangulate the results. Conducting qualitative interviews with practice leadership and staff members gave practices the opportunity to articulate challenges and strategies for overcoming them. When paired with quantitative findings, these qualitative results provide a richer dataset, contributing to the development of best practices in improving immunization rates at the practice level.

The value of this study is two-fold in that the findings have implications for clinical providers who are seeking to undertake quality improvement projects, as well as researchers who are designing practice change interventions. From a clinical perspective, these findings may be useful in helping other practices to implement practice change strategies. A practice assessment based on the four-level practice typology may help providers better prepare for implementation of the 4 Pillars^™ Practice Transformation Program and other practice change approaches and may be an enhancement to the 4 Pillars^™ Program. Previous research shows that a one-size-fits all approach is not successful in primary care practices (Crabtree et al., 2011). Recognizing characteristics that prepare for positive change can enable investigators to adjust intervention methods to align more closely with baseline practice characteristics. Consequently, they may experience greater rates of success in improving immunization and other patient care outcomes.

These findings also have several implications for researchers. When designing future randomized cluster trials in primary care settings it may be useful to stratify practices not just by patient population and location, but also by practice type to gauge a practice’s readiness for change. Doing so may present a clearer picture of barriers that need to be addressed before or as an intervention is executed. Additional research is needed to test and refine the practice typology as suggested here, and to assess the degree to which modifying the intervention in response to practice characteristics will improve adult immunization rates.

The ability to implement change within primary care settings requires more than intent to participate, especially when considering methods to improve adult immunizations rates. Practice characteristics such as experience with quality improvement or practice change research projects, leadership and communication style, organizational flexibility, as well as the potential effectiveness of the selected immunization champion may be critical factors in predicting success of the intervention.