Alcohol and other drug use among the youth exact a high toll in communities as they are linked to increased violence, accidents and crime. Evidence-based community-oriented substance abuse prevention and treatment programs can improve these outcomes and recoup treatment costs (NIDA, 1997). They need to be comprehensive and implemented well to reap these benefits, however (Backer, 2001). One way to improve program quality is to use evaluation data. While substance abuse program practitioners may have process and/or outcome evaluation data, they often lack capacity to use them to improve programming. Putting evaluation results to work in this manner is a Continuous quality improvement (CQI) core tenet. While practices similar to CQI have been associated with better outcomes in community-based substance abuse program researcher-led evaluations (Durlak and DuPre, 2008), communities remain challenged by evaluation tasks and how to apply data to systematically improve programs. Given CQI’s positive impact in industry and healthcare, it would appear beneficial to build capacity among community-based organizations to use similar practices and improve programming quality. In this article, we assess quality improvement process acceptability and feasibility in ten community-based, substance abuse prevention and treatment programs.

Continuous quality improvement can be defined as a planned approach to transform organizations by evaluating and improving systems to achieve better outcomes (Colton, 2000). Historically, US CQI began in manufacturing, using Shewhart’s ideas (Colton, 2000). In the 1980s and 1990s, CQI was adapted for healthcare, stimulated by multiple factors such as Medicaid cost containment (Burda, 1988); the National Demonstration Project in Quality Improvement in Healthcare (McLaughlin and Simpson, 1994) and the Joint Commission (1996) requiring hospital managers to have CQI programs in place.

Many CQI models have been used in industry and healthcare to provide systematic and ongoing feedback. For example, Balanced Scorecards (BSC - Kaplan and Norton, 1996) were developed to assess company performance in financial, customer, internal business processes, learning and business expansion domains. Whole System Measures (Martin et al., 2007) are similar metrics developed in healthcare to measure quality and performance using relatively few high-level indicators. Dashboards are increasingly popular, which update performance data in real time and aggregate them at individual, program and organization levels. Another common CQI model, developed by Shewhart and refined by Deming (2000), recently made popular by the Institute for Healthcare Improvement (IHI), is the Plan-Do-Study-Act (PDSA) cycle - a method in which individuals, drawn from the organization, make small, repeated and rapid changes to organizational function, test their impact and then decide whether to incorporate the change permanently. Recently, PDSA cycles were tested in the Network for the Improvement of Addiction Treatment (NIATx), small, community-based organizations providing substance abuse treatment. In the NIAtx effort, tools and technical support were provided to help improve access and retention in substance abuse treatment settings. The NIATx agency published several case studies demonstrating PDSA’s feasibility when ongoing technical assistance and support are provided (McCarty et al., 2007; 2009). The CQI effort we describe differs from NIATx, which only involved treatment programs, dictated CQI work focus and only included programs that met stringent criteria. In this CQI project, most programs were prevention, the staff chose the CQI focus and all program staff wanting to participate were included. All the QI approaches share three common principles (Miller et al., (2008): determining baseline effectiveness; engaging in deliberate practice such as setting objectives; and providing direct feedback on performance to make adjustments and set new objectives.

We demonstrate a certain CQI approach; i.e., PDSA cycles, in substance abuse prevention and treatment programs typical to community-based organizations. Manufacturing industry-based CQI techniques were adapted to make them appropriate for traditional healthcare settings (Colton, 2000). The same effort is needed to adapt CQI for community-based settings, which typically lack resources to collect and manage quantitative outcome data. We highlight CQI development for community settings and present evaluative data from CQI workshops with program staff and three waves of qualitative interviews with program participants to assess feasibility, acceptability and resources required in community-based settings.

The CQI process described below resulted from study investigators collaborating with Council staff and incorporating concepts and tools popularized by the Institute for Healthcare Improvement (i.e., PDSA); RAND’s work in improving practice guideline implementation in the US Army’s medical systems (Nicolas et al., 2001), and RAND’s collaborative quality improvement projects with Phoenix House, a national substance abuse treatment organization. However, these concepts and tools needed to be adapted because in community-based organizations, measuring quality - a task central to CQI success - was challenging. Process and outcome indicators were more difficult to identify and track. In community-based organizations delivering substance abuse prevention services, outputs are services delivered rather than tangible manufactured products or a specific medical procedure. Despite these challenges, many techniques and tools developed in business and healthcare fields were applicable (with some adaptation). Although many Council programs had developed process and outcome measures using the GTO model in a previous study (Chinman et al., 2008), most still faced challenges when using evaluation data to make systematic improvements. With leaders, we developed a broad CQI process: developed and planned CQI activities during semi-annual CQI Workshops and used the time between Workshops (called Implementation Periods) to implement the strategies (Figure 1).

Starting in November 2006, a CQI Workshop Planning Committee met eight times before the first CQI Workshop (Spring, 2007). The Planning Committee included representation from over half the participating programs staff and collaboratively worked on developing the workshop agenda, activities and CQI tools - worksheets developed by modifying previously created instruments (Phoenix House materials, Ishikawa or fishbone diagrams) and creating new ones. All were designed to help program staff progress the PDSA cycle. A key Planning Committee decision was to ask program staff to summarize their process and outcome evaluations for the CQI Workshops. Some tools were developed to assist program staff to summarize evaluation information. The Planning Committee reported progress and got input at regularly scheduled monthly meetings with all ten participating program directors.

Study investigators served as CQI coaches, helping to refine processes and tools. At the first CQI Workshop, study investigators, Planning Committee members and the Executive Director for the organization all played key roles explaining CQI’s value. Then staff, in small break-out groups, answered questions about CQI and how it could be implemented and sustained. Using their evaluation summaries, staff from each program developed specific CQI actions to implement using the PDSA cycle with input from the study investigators and executive director. After this workshop, the first implementation period got underway when program staff started their CQI actions. The executive director asked participants for CQI updates at monthly staff meetings.

Council staff asked study investigators to facilitate a second half-day CQI workshop (Fall, 2007) to educate seventeen new staff. Previous workshop participants were not required to attend. This workshop focused on developing or refining CQI actions rather than didactic GTO or CQI presentations. After, program staff continued their revised CQI actions (i.e., the second implementation period). In May 2008, a third, full-day CQI workshop was conducted for all nineteen participating program staff. Participant numbers varied over time as the budgeting and staffing changed year-to-year. After a CQI refresher, cross-program groups were formed and developed an organization-wide CQI action to improve referrals between programs, based on staff requests to work on an issue that cut across programs. Staff also refined old or developed new, program-specific CQI actions. The study ended after the third workshop. Additionally, two study investigators attended monthly staff meetings to provide further coaching and to answer questions. Following the first workshop, study investigators contacted program staff quarterly to administer an interview protocol and provide additional technical assistance (e.g., tips to improve program recruitment, easy to use techniques for coding data from open-ended survey questions). These calls lasted 45 to 90 minutes.

There are key differences between our CQI process and those used in other studies. For example, many screened and then selected participants (or CQI proposals) with higher capacity; provided participants with significant additional funding to carry out CQI activities; and/or stated a priori what organization staff would work on (e.g., McCarty et al., 2007; Rubenstein et al., 2006). In our study, all participating organization programs were included, no additional funding was provided and research staff (as coaches) worked collaboratively with program staff to focus their CQI efforts. We took this approach because we were interested in assessing the degree to which CQI could be undertaken in real world situations (i.e., no additional funding) and we believed that consistent with organizational change theories (Green et al., 1980), extensive collaboration and tailoring to meet local staff needs would promote greater CQI process uptake.

We assessed the feasibility of using traditional CQI techniques in community-based substance abuse prevention and treatment settings. While there were challenges, we found that it is feasible for staff to use CQI techniques when supported. Staff in ten participating programs developed several CQI actions, involved most program staff, secured additional resources in half the CQI actions, forged collaborations with other Council programs or outside entities in about a quarter of the CQI actions and spent on average between 2.33 to 3.75 hours a week on CQI activities over the nine-month period. Two-thirds completed their CQI actions; i.e., progressed through the entire PDSA cycle. Although progression is not equivalent to program impact, we argue that it is an important proxy (necessary, but not sufficient) for program impact to occur. Program staff were generally positive about using CQI; enthusiasm, while declining over nine months, was high overall. Staff ratings of opportunities to develop new ideas declined in the second and third workshops. This finding is consistent with how the workshops were run. All time in the first workshop was spent on generating new ideas, whereas the subsequent workshops split time between generating new and refining old ideas. Qualitative data showed that program staff felt the CQI process helped them become more organized and accountable. As such, results indicate that these CQI methods can be a bridge between conducting self-evaluation and making concrete changes to improve programming.

We learned several lessons that have implications for using CQI in community-based settings. Much progress getting CQI off the ground followed significant support from Council leaders, which resulted from having a multi-year relationship with Council staff in which data value and improvement were discussed often through using the GTO model. From there, CQI organization champions emerged, playing a key role shepherding the process. Having local organizational support (Colton, 2000; Heller and Arozullah, 2001) and champions present (Shortell et al., 2004) have been cited as key factors for adopting quality improvement activities. Outside facilitators (GTO staff) were critical to engaging and supporting Council staff and helping to launch the CQI effort. At the outset, Council staff asked for assistance in using their evaluation data to improve programming, signaling a readiness to begin a CQI process, but needed help getting started. In this case, GTO staff played an important role engaging program staff to participate in CQI development and planning for their organization, training Council staff in CQI fundamentals, establishing CQI basic structure (i.e., repeated workshops with intervening implementation periods) and providing ongoing support. This technical assistance or facilitation is becoming widely accepted and necessary for translating research findings into practice (Stetler et al., 2006). Several successful CQI studies in various domains have included this support (McCarty et al., 2007; McBride et al., 2000; Ockene et al., 1996; Carney et al., 1992).

Continuous quality improvement activity may impact its likelihood for success in these settings. The CQI actions, larger in scope (i.e., changing a program significantly), were completed at a lower rate than smaller strategies. After reviewing 55 CQI studies across several domains, Shortell et al., (1998) also found that negative results were more common when applied to more complex care processes. The choice to significantly alter programs may undercut a PDSA cycle’s key component: making small, low-risk changes in the clinical setting (Powell et al., 2008). Choosing large-scale changes put staff in situations where they needed significantly more resources beyond their own ideas and ingenuity to implement those changes successfully. Establishing CQI activities involves making decisions about issues on which program staff often have strong opinions. Many implementation theories suggest that failing to incorporate local stakeholder input can undermine new practice implementation (Fixsen et al., 2005) and we submit that starting CQI is no different. Therefore, we believed it was critical to collaborate with Council staff and leaders throughout the CQI implementation. Although study investigators broadly outlined the CQI process, Council staff made important contributions to many CQI tools and CQI workshops. This facilitated staff ‘buy-in’ from most and ensured the resulting process and tools were tailored to fit their organization.