In Table 1, we show patient demographics in the 8 continuity clinics. Most (7 out of 8) practices were hospital-based, 5 were in urban settings, and all participated in the Vaccines for Children program; most patients were publicly insured.

As shown in the P-chart of monthly chart reviews (Fig 1), captured opportunities for HPV vaccination increased by 16.4 percentage points (from 46.9% to 63.3%) overall. Special cause was demonstrated by centerline shift after the intervention.

Comparing the intervention period to the baseline period, we found that adolescents were more likely to receive a vaccine at an eligible visit during the intervention than before the intervention (odds ratio: 1.87; 95% confidence interval: 1.54–2.28). HPV captured vaccination rates improved at both well child (from 65% to 77%) and other visits (from 27% to 40%), with P < .001. Improvement in captured opportunities varied by practice, from 2.5 to 30 percentage points (Fig 2). Practices with the lowest initial rates made the least improvement; practices with the middle range at baseline made the most improvement. Documentation of refusals and deferring vaccination increased from 9.3% to 17.1% and from 5% to 12.8%, respectively, whereas no documentation for an MO declined from 80.2% to 65.5%, with P < .001 overall.

Monthly diaries were used to describe the changes in the prompt intervention, barriers encountered, parent responses, and lessons learned (Table 2). Barriers included nursing shortages, staff changes, and decreased response to the prompts at busy periods. A minority of parents continued to be resistant to vaccination, and safety was a concern that was raised by several parents. Lessons learned from successful practices included sending frequent communications about the project and meeting frequently about practice change.

Regarding a strong provider recommendation, providers noted that staff recommendations were not as strong as expected at baseline but appeared to improve because of the intervention. Practice comments toward the end of the study were as follows:

The staff continues to get resistance from some parents. Overall seems less over the past 6 months. The 2 areas for this are: (1) my child is not sexually active yet, “I’ll wait.” and (2) The thought that the HPV is only for girls to prevent cervical cancer only; there is a hesitancy to give it to their sons. Some staff have stepped up their education and overall approach to the vaccine. I believe this change in staff has led to overall less resistance from the parents.

[We have] much less hesitancy or resistance than in the earlier part of the year. It now feels like the HPV is automatic, not just an add-on.

Practices with little improvement (<5%) lacked consensus for the QI activity from staff and providers. In addition, more complicated interventions (such as having the nurse place an LED light on the computer) were difficult to sustain. The simplest interventions, such as putting the VIS sheet on fluorescent paper, led to greater improvement (>25%). In addition, successful practices made it a priority to update new staff and to review monthly data with the QI team.

In this QI study in pediatric resident continuity clinics, we found that a bundled intervention consisting of a combination of provider prompts from nurses and EHRs, provider and staff training in giving a strong recommendation for the vaccine, and performance feedback to providers increased rates of captured opportunities for HPV vaccination by 16 percentage points. However, there was great variation in improvement by practice, ranging from 2.5 to 30 percentage points. In addition, absence of documentation for MOs declined from 80% to 66%.

The range of improvement for practices was broad and provides lessons for practice improvement. Practices with the greatest increases in captured opportunities used the following approaches: frequent communications for staff and residents, changing EHR note templates to improve documentation, using multimodal prompts (paper, VIS, EHR), and having a nurse champion who reviewed medical records ahead of time. By improving documentation, practices have a better understanding of causes of under-vaccination. A previous study that was focused solely on EHR prompts without other interventions was not effective.¹² This may be because of “prompt fatigue,”^22,23 indicating the necessity to use multimodal cues. However, prompts need to be simple to be sustainable. For example, a fluorescent VIS sheet worked well as a prompt, whereas stamping the immunization registry printout had an intermediate effect. We also believe that the addition of performance feedback to prompts contributed to the large effects. In addition, practice champions can improve adherence to guidelines, including hand hygiene,²⁴ vaccine delivery,²⁵ and infection prevention.²⁶ Finally, the success of our intervention supports other studies in which researchers have found that multicomponent interventions are more effective^27,28 than single-component ones.

Two practices with the lowest baseline rates of captured opportunities experienced the least improvement. In 1 case, the barriers were potentially identifiable a priori, whereas in the other practice they were not. In 1 practice, the project champions had trouble convincing all providers and nurses in the practice to support vaccinating at illness visits. Practice engagement is a critical component of successful QI initiatives.²⁹ It is important to involve the entire office staff in the intervention³⁰ and to engage leadership in practice change for it to be successful.³¹ A second practice experienced major staff turnover; hence, the intervention became a low priority. Such real-life factors can dramatically diminish practices’ abilities to improve the delivery of preventive services.³² Because primary care practices face a multitude of barriers to practice improvement,³³ a potential target for future research is to identify factors that predict the likelihood of improvement from modifiable practice features.

At the other extreme, the highest performing practice at the baseline period also experienced a low rate of improvement in captured opportunities. At the high range of performance, there is minimal room for improvement, and alternative strategies may be needed to reach the remaining few MOs. This is consistent with other studies that have revealed that high-performing practices make fewer advances than midrange performers in QI interventions.^34,35

The highest rate of captured opportunities in any intervention practice was 84%, with an average of 63%. We believe rates of captured opportunities are unlikely to reach 100% for HPV vaccination in most practices because of several factors. First, parent delay and refusal are likely to contribute to ∼20% of HPV under-vaccination.³⁶ Although a strong recommendation can impact vaccine refusals, as shown by a 12% increase in captured opportunities at well visits, other strategies are needed to reach the most vaccine-hesitant parents. Second, a few patients will be too sick to be vaccinated. Lastly, some days are likely to be too busy to allow time for nurses to assess vaccination status or to draw up and administer vaccines during acute visits. At busy times in the practice, such as during influenza season, the staff had particular difficulty reviewing vaccination records from multiple sources, and, consequently, some practices noted a drop in nurse prompts. During times of stress, practices may need to adjust the intervention.²⁹

One strength of this study was the multidisciplinary nature of the practice teams, which included attending physicians, residents, nurses, and staff. Collaborating with staff from different areas in a practice has been shown to enhance teamwork and communication and counters the “silo effect.”³⁷ In particular, training residents in both QI and in providing a strong recommendation is important because residents tend to use the skills learned during residency during their later practice.^38,39 A second strength was having a multisite network of practice-based teams that allowed providers to learn from one another.⁴⁰ Such LC models are now increasingly used for a variety of health topics. A third strength is potential sustainability, which was enhanced by providers earning Maintenance of Certification Part IV credit for participation; this has been shown to motivate physicians to participate in QI for HPV vaccination.⁴¹ Finally, the 8 practices involved in the intervention care for a large number of adolescents, so this intervention has the potential to impact a sizeable population.

There are several characteristics of the setting and patient population that may limit generalizability. The intervention was performed in pediatric resident continuity clinics serving mostly low-income populations; findings may not be generalizable to other settings. Academic practices may be particularly motivated to implement newer guidelines and are expected to teach current recommendations. On the other hand, these large practices with rotating residents are complex environments in which to implement practice change. There may be selection bias in practices volunteering to participate. The practices received a modest incentive for performing the randomly selected chart reviews. Also, the research and LC faculty leads were supported by grant funding. To scale up this intervention, we would need to integrate such core funding into local, state, or national QI efforts.

In this QI study, we demonstrate that a bundled intervention consisting of multimodal provider prompts combined with training on the delivery of a strong provider recommendation for HPV vaccination and performance feedback to providers can substantially increase captured opportunities for HPV vaccination in pediatric resident continuity clinics. For best results, practices should have an engaged champion, communicate frequently with the practice team, ensure that all providers understand and support the initiative, and incorporate ongoing performance feedback. If scaled up, such a QI intervention might help raise HPV vaccination rates across the United States.