<!DOCTYPE article
PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD with MathML3 v1.3 20210610//EN" "JATS-archivearticle1-3-mathml3.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" dtd-version="1.3" xml:lang="en" article-type="research-article"><?properties manuscript?><processing-meta base-tagset="archiving" mathml-version="3.0" table-model="xhtml" tagset-family="jats"><restricted-by>pmc</restricted-by></processing-meta><front><journal-meta><journal-id journal-id-type="nlm-journal-id">8106454</journal-id><journal-id journal-id-type="pubmed-jr-id">4522</journal-id><journal-id journal-id-type="nlm-ta">J Asthma</journal-id><journal-id journal-id-type="iso-abbrev">J Asthma</journal-id><journal-title-group><journal-title>The Journal of asthma : official journal of the Association for the Care of Asthma</journal-title></journal-title-group><issn pub-type="ppub">0277-0903</issn><issn pub-type="epub">1532-4303</issn></journal-meta><article-meta><article-id pub-id-type="pmid">33287598</article-id><article-id pub-id-type="pmc">8950084</article-id><article-id pub-id-type="doi">10.1080/02770903.2020.1861625</article-id><article-id pub-id-type="manuscript">HHSPA1789613</article-id><article-categories><subj-group subj-group-type="heading"><subject>Article</subject></subj-group></article-categories><title-group><article-title>Case analyses of state-sponsored asthma quality improvement interventions &#x02013; benefits and technical assistance efforts</article-title></title-group><contrib-group><contrib contrib-type="author"><name><surname>Perkins</surname><given-names>Ayana</given-names></name><degrees>PhD</degrees><contrib-id contrib-id-type="orcid">http://orcid.org/0000-0003-0822-356X</contrib-id><xref rid="A1" ref-type="aff">a</xref></contrib><contrib contrib-type="author"><name><surname>Bradley</surname><given-names>Anna</given-names></name><degrees>MS</degrees><xref rid="A2" ref-type="aff">b</xref></contrib><contrib contrib-type="author"><name><surname>Magaldi</surname><given-names>Judith</given-names></name><degrees>BS</degrees><xref rid="A3" ref-type="aff">c</xref></contrib></contrib-group><aff id="A1"><label>a</label>Asthma and Community Health Branch, Division of Environmental Health Science and Practice, National Center for Environmental Health, Centers for Disease Control and Prevention, Atlanta, GA, USA;</aff><aff id="A2"><label>b</label>Montana Department of Public Health and Human Services, Public Health and Safety Division, Helena, MT, USA;</aff><aff id="A3"><label>c</label>Cancer Control Section, Division of Chronic Disease, Primary Care and Rural Health, Indiana State Department of Health, Indianapolis, IN, USA</aff><author-notes><fn fn-type="con" id="FN1"><p id="P1">This work was authored as part of the Contributor&#x02019;s official duties as an Employee of the United States Government and is therefore a work of the United States Government.</p></fn><corresp id="CR1"><label>&#x02709;</label><bold>CONTACT</bold> Ayana Perkins <email>vzt8@cdc.gov</email> Asthma and Community Health Branch, Division of Environmental Health Science and Practice, National Center for Environmental Health Centers for Disease Control and Prevention, 4770 Buford Hwy, Atlanta, GA 30341, USA.</corresp></author-notes><pub-date pub-type="nihms-submitted"><day>17</day><month>3</month><year>2022</year></pub-date><pub-date pub-type="ppub"><month>3</month><year>2022</year></pub-date><pub-date pub-type="epub"><day>04</day><month>1</month><year>2021</year></pub-date><pub-date pub-type="pmc-release"><day>25</day><month>3</month><year>2022</year></pub-date><volume>59</volume><issue>3</issue><fpage>616</fpage><lpage>627</lpage><abstract id="ABS1"><sec id="S1"><title>Objective:</title><p id="P2">A cross-case analysis was used to discover how two states benefited from expanded use of evaluation for asthma quality improvement initiatives. If an asthma quality improvement (QI) initiative is successfully evaluated, data can inform how to effectively integrate clinical practice guidelines and circumvent non-clinical reasons that interrupt QI projects such as low staff interest. This article addresses a gap in the literature on quality of evaluation support needed to improve and sustain asthma QI at local health care organizations by describing the similar discoveries observed at two independent QI statewide initiatives in Indiana and Montana.</p></sec><sec id="S2"><title>Methods:</title><p id="P3">As part of a larger review, two states funded through the National Asthma Control Program at the Centers for Disease Control and Prevention were identified based on similarities in evaluation approaches. Each state used an iterative stakeholder-driven evaluation approach, mixed methods, process evaluation indicators, and active use of evaluation findings. The asthma QI initiatives and evaluations in Indiana and Montana were coordinated independent of each other.</p></sec><sec id="S3"><title>Results:</title><p id="P4">Although both states found that asthma QI initiatives improved health outcomes, evaluation data were able to further pinpoint areas that would improve quality of technical support to health care organizations and identify markers of sustainability, such as nontraditional benefits to staff, and intervention sites.</p></sec><sec id="S4"><title>Conclusion:</title><p id="P5">Findings suggest that when evaluation is used to guide implementation, data are available to develop site-specific assistance and identify sustainability markers to prevent interruption of positive health outcomes associated with an asthma QI initiative.</p></sec></abstract><kwd-group><kwd>Asthma</kwd><kwd>clinical guidelines</kwd><kwd>quality improvement</kwd><kwd>evaluation</kwd><kwd>program improvement</kwd><kwd>technical assistance</kwd></kwd-group></article-meta></front><body><sec id="S5"><title>Introduction</title><p id="P6">An estimated 25 million US children and adults have asthma (<xref rid="R1" ref-type="bibr">1</xref>). However, access to optimal care for managing symptoms is not always guaranteed (<xref rid="R2" ref-type="bibr">2</xref>&#x02013;<xref rid="R9" ref-type="bibr">9</xref>). Educating health care professionals on best practices for treating high-risk populations can increase access to optimal care for asthma management (<xref rid="R2" ref-type="bibr">2</xref>,<xref rid="R5" ref-type="bibr">5</xref>,<xref rid="R10" ref-type="bibr">10</xref>&#x02013;<xref rid="R12" ref-type="bibr">12</xref>).</p><p id="P7">The quality improvement (QI) initiative is a recommended strategy for educating health care professionals on asthma best practices for treating high-risk populations. QI can involve reviewing and correcting the clinical environment to improve patient experiences and disease outcomes, and to encourage safety and efficiency of structures and resources. The US Department of Health Resources and Services Administration describes health care QI as a progressive and methodical process that contributes to measurable advances in patient treatment, care, and health outcomes (<xref rid="R13" ref-type="bibr">13</xref>). Typical components of an asthma QI activity include patient and provider education, clinical assessment improvement, and asthma action plans and other information systems (<xref rid="R14" ref-type="bibr">14</xref>).</p><p id="P8">Asthma QI initiatives can help health care organizations (HCOs) by suggesting recommended clinical practice guidelines such as the National Asthma Education and Prevention Program Expert Panel Report 3 (EPR-3) guidelines for diagnosis and treatment of asthma (<xref rid="R15" ref-type="bibr">15</xref>,<xref rid="R16" ref-type="bibr">16</xref>). HCOs can have greater success with inclusion of multiple QI components (<xref rid="R14" ref-type="bibr">14</xref>,<xref rid="R17" ref-type="bibr">17</xref>) and support from their state agencies (<xref rid="R18" ref-type="bibr">18</xref>). The Agency for Healthcare Research and Quality&#x02019;s 2009 <italic toggle="yes">Resource Guide for State Action</italic> (AHRQ) suggests state agencies have an important role in fostering success in an asthma QI effort by offering evaluation and other types of supports to promote successful translation of clinical guidelines to HCOs (<xref rid="R18" ref-type="bibr">18</xref>). A state&#x02019;s provision of a comprehensive systematic inquiry could help build capacity of a local health care organization (HCO) to sustain the promise of multifaceted QI initiatives than simply the planned reflection that is the centerpiece for some health care QI models (<xref rid="R13" ref-type="bibr">13</xref>).</p><p id="P9">State health departments are ideal partners for HCOs because state health departments are often not equipped with the evaluation capacity to assess how their partners would benefit from such an asthma QI investment. The 2009 AHRQ report also suggests that potential for improving patient health and increasing return on investment is strengthened by state-level asthma QI efforts (<xref rid="R18" ref-type="bibr">18</xref>). This assertion is further strengthened by the evidence Rojanasarot et al. (<xref rid="R19" ref-type="bibr">19</xref>) found indicating effectiveness of a state-level asthma QI efforts in their cross-state study.</p><p id="P10">In 2014, the Centers for Disease Control and Prevention (CDC) funded 23 state health departments through the National Asthma Control Program (NACP) to use a multiprong approach to asthma control (<xref rid="R20" ref-type="bibr">20</xref>). One recommended strategy was the use of asthma QI with health care systems that served at-risk populations (<xref rid="R20" ref-type="bibr">20</xref>). The QI strategy was designed to encourage better adherence to the EPR-3 guidelines for diagnosis and treatment of asthma (<xref rid="R15" ref-type="bibr">15</xref>). Many of the NACP-funded states used evidence-based QI models for asthma, such as the Chronic Care Model employed by one of the described cases in this article (<xref rid="R21" ref-type="bibr">21</xref>). Evaluation was not specifically required for these asthma QI initiatives. Instead, the NACP recommended evaluation as a systematic inquiry to improve efficiency and effectiveness of QI programs. NACP also provided support to their grantees through pairing each state with an evaluation advisor and the dissemination of the <italic toggle="yes">Learning and Growing</italic> evaluation series (<xref rid="R22" ref-type="bibr">22</xref>).</p><p id="P11">This article provides evidence on the utility of AHRQ recommendations for state-level asthma QI efforts by describing the successful use of asthma QI evaluation by two state asthma programs, Indiana and Montana. With QI initiatives often abandoned for non-clinical reasons, such as lack of interest or lack of capacity to fully implement a QI model in the health care setting, HCOs might benefit from access to evaluation data to help resolve setting-specific challenges that are outside of the design of the QI model or the control of the HCO (<xref rid="R23" ref-type="bibr">23</xref>). Many nonacademic HCOs lack the scientific capacity for rigorous examination of their QI initiatives (<xref rid="R24" ref-type="bibr">24</xref>). Building evaluation capacity through state collaborations would help HCOs more clearly determine the influences and the effectiveness of asthma QI for their particular setting. When HCOs, such as federally qualified health centers, pair with a state asthma program, they can capitalize on resources states possess, such as evaluation support.</p><sec id="S6"><title>Technical assistance for site-specific challenges</title><p id="P12">Developing evaluation capacity in HCOs can assist with the assessment of technical assistance and other processes that are normally overlooked in asthma QI initiatives. State health departments and other funders often offer technical assistance through training and material support to their partners and grantees to navigate emerging challenges of health interventions (<xref rid="R25" ref-type="bibr">25</xref>&#x02013;<xref rid="R28" ref-type="bibr">28</xref>). Technical assistance is a valuable resource for providers interested in sustaining an asthma QI initiative, but it needs to be tailored based on the unique factors of each HCO. Data gained from evaluating technical assistance help validate the quality of support each HCO needs to effectively integrate clinical practice guidelines. These data also reveal critical information about staff and organizational capacity to sustain an asthma QI initiative in an HCO.</p></sec><sec id="S7"><title>Asthma QI benefits as sustainability markers</title><p id="P13">Beyond staff and organizational capacity, QI projects are often susceptible to challenges that are outside of clinical practice, such as staff interest and support (<xref rid="R23" ref-type="bibr">23</xref>,<xref rid="R29" ref-type="bibr">29</xref>,<xref rid="R30" ref-type="bibr">30</xref>). Even if asthma QI implementation is initially a success, an HCO can fail to sustain a QI initiative long enough to achieve health improvements. Evaluation can be used to discover contextual data that serve as markers for sustainability such as benefits to providers and implementation sites. More often, it is the funder who defines what is beneficial (<xref rid="R31" ref-type="bibr">31</xref>,<xref rid="R32" ref-type="bibr">32</xref>). Expanding perspectives beyond the funder could help predict local interest in continuing an asthma QI initiative after state-level support is no longer available.</p></sec><sec id="S8"><title>Staff benefits</title><p id="P14">Benefits to staff are essential data to discover, but in many outcome assessments, much of the staff&#x02019;s work is ignored in favor of the expected changes for an intervention. Frontline staff are often tasked with new responsibilities without the relief of reducing pre-existing work requirements. Asthma QI initiatives are promising interventions for HCOs but may also introduce to HCO staff new complications of translating clinical practice guidelines in a dynamic setting of an HCO (<xref rid="R33" ref-type="bibr">33</xref>&#x02013;<xref rid="R35" ref-type="bibr">35</xref>). In Mansour&#x02019;s 2009 study of an asthma QI initiative in a school-based health center, staff partially attributed project success to finding that the &#x0201c;perceived benefits of doing the work ultimately outweighed the perceived &#x02018;costs&#x02019; of participation&#x0201d; (p. 725) (<xref rid="R29" ref-type="bibr">29</xref>). Staff benefits included gradual adoption of the asthma QI initiative and observable health improvements of patients. Learning staff benefits is another compelling way that HCOs can use evaluation to sustain their asthma QI work.</p></sec><sec id="S9"><title>HCO benefits</title><p id="P15">Evaluation of site-specific benefits may offer more local buy in than anticipated health improvements. HCOs can lower costs when they successfully implement clinical processes that integrate EPR3 and other clinical practice guidelines, especially when an HCO is serving disproportionately affected populations. HCOs demonstrating QI project success may become eligible for national credentialing programs that would allow them to expand to alternative payment models. Exploring site-specific benefits is another way to determine if an asthma QI initiative is in alignment with the priorities of that local HCO.</p><p id="P16">This study utilized two case studies to explain the benefits of the broadened use of evaluation and the consideration of indicators that are often overlooked in more traditional studies of asthma QI initiatives. Collecting evaluation data on nontraditional benefits for staff and local HCOs may introduce new standards of success as well as long-term insight on whether or not these initiatives can continue after funder&#x02019;s initial investment. The study also aims to illuminate how evaluation can be helpful in determining the quality of technical assistance needed for QI at local HCOs.</p></sec></sec><sec id="S10"><title>Methods</title><sec id="S11"><title>Purpose</title><p id="P17">This cross-case analysis uses a case-oriented approach to identify similarities and differences in how evaluation was used for asthma QI in the individual cases studies being examined (<xref rid="R28" ref-type="bibr">28</xref>,<xref rid="R36" ref-type="bibr">36</xref>). Since evaluations were only intended for program planning, the Indiana and Montana health departments are not required to seek review from their Institutional Review Boards. We also focused on the unique achievements found within each setting (<xref rid="R36" ref-type="bibr">36</xref>,<xref rid="R37" ref-type="bibr">37</xref>). Each state independently designed its own evaluation of its QI initiatives based on the needs of local stakeholders. This case analysis examined the similarities in evaluation use, not all aspects of the asthma QI in each state.</p></sec><sec id="S12"><title>Case descriptions</title><p id="P18">Indiana and Montana receive funding from the NACP to implement asthma QI and evaluation as best practices for improving asthma control. Each state evaluator (and co-author) independently designed an evaluation of their state-coordinated asthma QI efforts to improve program planning. As part of its ongoing technical assistant effort, the NACP launched a preliminary investigation on national cross-state evaluation findings of asthma QI to better under understand how successfully asthma QI initiatives were being implemented among their funded states. After NACP preliminary review across states, the focus on Indiana and Montana was initiated after finding adequate similarities in evaluation approaches including the following: use of CDC&#x02019;s Evaluation Framework (<xref rid="R22" ref-type="bibr">22</xref>), mixed methods, process evaluation indicators, and active use of evaluation findings.</p><p id="P19">The first case in this report is the evaluation of asthma QI initiatives in Indiana in select federal, community health, and rural health centers (<xref rid="F1" ref-type="fig">Figure 1</xref>). The second case is the evaluation of a multi-phase asthma QI initiative in select critical access hospitals (CAHs) in Montana (<xref rid="F1" ref-type="fig">Figure 1</xref>).</p></sec><sec id="S13"><title>Data collection</title><p id="P20">Lead evaluators (and co-authors) of asthma QI initiatives in Montana and Indiana health departments independently collected, maintained, and secured the project data. Both evaluation projects used multiple data sources to triangulate findings, which helps to establish credibility of findings (<xref rid="T1" ref-type="table">Table 1</xref>) (<xref rid="R38" ref-type="bibr">38</xref>,<xref rid="R39" ref-type="bibr">39</xref>).</p><p id="P21">Indiana began evaluation data collection in 2015. Data included interviews with staff, surveys completed by HCO staff, and de-identified data collected monthly and quarterly from the electronic medical records of HCOs. Quarterly data included the following frequencies on patients 5&#x02013;50 years of age: a) percentages with poorly controlled asthma in the previous 12 months, b) percentages with poorly controlled asthma who were followed up by the HCO in the previous 12 months, c) percentages of patients with a diagnosis of asthma immediately preceding the reporting, and d) percentage of patients with a diagnosis of poorly controlled asthma as of the last day of the month preceding the reporting month. Monthly data were collected only from HCOs participating in the Learning Collaborative that chose asthma as the target area for QI. Monthly data included the following frequencies on HCO patients 5&#x02013;64 years of age: a) percentage of patients identified with persistent asthma and were appropriately ordered medication, b) percentage of adult patients diagnosed with asthma that had a documented asthma action plan<sup><xref rid="FN5" ref-type="fn">1</xref></sup> in the previous 12 months, and c) percentage of patients identified as having persistent asthma who received intensive self-management education or referred to community-based services for asthma self-management training and services. These data were used only for future programming planning efforts by Indiana Asthma Program (IAP) and were not intended to be interpreted as the exact cause for improved health outcomes.</p><p id="P22">Montana data collection methods initially included pre- and post-project chart audits on health outcomes of asthma patients such as frequency of emergency department (ED) visits, literature reviews, and interviews with project leaders at each site. Because of low participation in post-project chart audits, MACP started a smaller planning evaluation in 2014 to discover the most effective strategies to increase satisfaction among sites that volunteered to apply the asthma QI. Since 2014, MACP funded QI projects for seven hospitals, with three hospitals participating in multiple years. MACP included interviews with QI coordinators from the Montana Diabetes Program and Montana Cardiovascular Health Program in the planning evaluation to learn from their long histories of successfully implementing QI in Montana.</p></sec><sec id="S14"><title>Data analysis</title><p id="P23">Lead Evaluators from Indiana and Montana (study co-authors) were invited to participate in the cross-site analyses. Our analyses began with an in-depth exploration of how each evaluation project unfolded in Indiana and Montana. Data analysis was guided by the standard of verification, an important principle for qualitative methods which ensures that data are comprehensive, trustworthy, and confirmable (<xref rid="R39" ref-type="bibr">39</xref>,<xref rid="R40" ref-type="bibr">40</xref>).</p><p id="P24">Each state evaluator examined data from all available sources and identified meaningful changes associated with the asthma QI at their respective sites (<xref rid="T1" ref-type="table">Table 1</xref>). The triangulation of data helped to establish how comprehensive and confirmable the findings were. Next, a detailed summary of evaluation processes for each state was created to document how evaluation was used and what new discoveries were revealed. The detailed description of the within-case analyses was developed to increase transparency of how each state evaluation unfolded (<xref rid="R40" ref-type="bibr">40</xref>).</p><p id="P25">During the within-case analyses, we examined Indiana and Montana evaluation summaries separately. After we became familiar with the data and preliminary coding for each state, the team began identifying similar themes across states through discussions. The team used inductive analyses where the selection of codes are unplanned and are guided by what patterns emerge within the data (<xref rid="R41" ref-type="bibr">41</xref>). The novel use of evaluation surfaced as an overarching theme across Indiana and Montana that was later further reduced into three thematic subcategories.<sup><xref rid="FN6" ref-type="fn">2</xref></sup> Our team collaborated on verifying the boundaries of the thematic categories and confirmed use were consistent across sites (<xref rid="T2" ref-type="table">Table 2</xref>).</p></sec></sec><sec id="S15"><title>Results</title><p id="P26">In this section, we will describe state-specific insights drawn from their evaluations and two similar themes that emerged in our cross-site analysis. The exercise provided interesting insights on where evaluation findings might overlap for asthma QI initiatives in different settings.</p><sec id="S16"><title>Indiana case study findings</title><p id="P27">Housed under the Indiana State Department of Health (IDOH), the Indiana IAP has been funded under the NACP since 2002. Approximately 509, 651 people in Indiana have asthma, which represents 10% of the population (<xref rid="R42" ref-type="bibr">42</xref>). Among people with asthma in Indiana, a large proportion reported that their disease is not in control, with 39.0% reporting asthma symptoms in the last week, 24.2% reporting asthma symptoms every day in the last month, and 54.0% indicating that they limited their activities because of asthma in the last month (<xref rid="R43" ref-type="bibr">43</xref>). Indiana has approximately 27 000 asthma discharges from the emergency department (ED) each year (<xref rid="R44" ref-type="bibr">44</xref>).</p><p id="P28">The IDOH sponsored three one-year asthma QI activities during their 2014&#x02013;2019 cooperative agreement with NACP: Indiana Primary Care Learning Collaborative, American Lung Association (ALA) Enhancing Asthma Care in the Primary Care Setting, and IDOH Spread and Sustainability Grant. During this time frame, the Indiana Primary Care Learning Collaborative engaged 72 HCOs, and the ALA Enhancing Asthma Care in the Primary Setting project engaged 21 HCOs. Some ALA QI initiative participants also completed the Indiana Primary Care Learning Collaborative. The IDOH Spread and Sustainability Grant was initiated in 2015 with five HCOs participants during this cooperative agreement. Four of the five IDOH Spread and Sustainability grantees also completed an asthma QI project under the Indiana Primary Care Learning Collaborative and the ALA Enhancing Asthma Care in the Primary Care Setting. A more detailed description of each of the IDOH-sponsored asthma QI initiatives can be found in the following text.</p><p id="P29">The Indiana Primary Care Learning Collaborative began in 2012. In addition to brief action planning, it uses the Chronic Care Model with the Model for Improvement&#x02019;s Plan-Do-Study-Act cycles (<xref rid="R21" ref-type="bibr">21</xref>).Participants include state-funded community health centers (CHCs). The purpose of the project is to improve patient health using QI tools. IDOH practice coaches provide technical assistance through coaching, training, health communication materials, and data collection tools (<xref rid="R21" ref-type="bibr">21</xref>).</p></sec><sec id="S17"><title>ALA QI in Indiana</title><p id="P30">Indiana&#x02019;s asthma program entered a partnership with the ALA to recruit Indiana HCOs for participation in their Enhancing Asthma Care in the Primary Care Setting. ALA is an important partner in IAP&#x02019;s work. ALA collaborates with Indiana CHCs, which primarily serve disproportionately affected subpopulations. Through this ALA partnership, clinics receive technical assistance and coaching from ALA staff. Additional benefits to participating clinic staff include sponsorship to the Asthma Educator Institute and spirometry trainings. According to the ALA, this asthma QI model has proven to be effective in reducing hospitalizations and emergency room visits for patients from participating clinics (<xref rid="R19" ref-type="bibr">19</xref>).</p><p id="P31">In 2016, IAP expanded its asthma QI efforts, through partnership with the IDOH Office of Primary Care and Cardiovascular Health and Diabetes Section. IAP could then support eligible CHCs that had participated in either the Indiana Primary Care Learning Collaborative or the ALA Enhancing Asthma Care in the Primary Care Setting. This effort was known as the Spread and Sustainability grant. It was intended to provide self-management education, expand the reach of QI efforts to include encouragement of a bidirectional referral system, and ensure components of the EPR-3 Guidelines were addressed.</p><p id="P32">Evaluation data collection methods began in 2015. Data included staff interviews, existing Indiana Primary Care Learning Collaborative (PCLC) monthly and quarterly data, and survey responses provided by the staff at HCOs. To measure initial effectiveness of the asthma QI initiative, IAP used measures of poorly controlled asthma from the participating health centers that were collected monthly and quarterly data through electronic medical records by clinic staff. State-funded CHCs who provided asthma-specific data showed a 5.9% decrease in the number of patients with poorly controlled asthma in 2017. CHCs participating in the ALA initiative and PCLC Spread and Sustain initiative showed a 34% and 26% decrease respectively in measures of poorly controlled asthma. The data were used only for future programming planning efforts by IAP and were not intended to be interpreted as the exact cause for improved health outcomes.</p></sec><sec id="S18"><title>Benefits of asthma QI initiatives</title><sec id="S19"><title>Benefits to staff.</title><p id="P33">Throughout all of Indiana&#x02019;s asthma QI initiatives, staff saw benefits in the peer-to-peer learning opportunity to discuss best practice, identify similar barriers to care, and provide opportunities to brainstorm solutions. However, when discussing specific content, interviews revealed that they consistently found ALA trainings to be helpful for their work. ALA trainings included the Implementation and Interpretation of Spirometry in the Primary Care Practice course. During interviews, clinic staff indicated that spirometry use increased the ability to change rooming practices. Their training also allowed the QI coordinator to encourage clinicians to provide a more accurate diagnosis on level of severity for patients. A further benefit from ALA trainings was increased opportunities for shared learning among staff across clinic sites.</p></sec><sec id="S20"><title>Benefits to implementation sites.</title><p id="P34">Participating HCOs increased use of asthma action plans and use of electronic medical record systems. These two improvements are common outputs for asthma QI initiatives. Interviews revealed that participation in multiple asthma QI initiatives encouraged some clinics to bridge asthma QI efforts with treatment for other chronic diseases by incorporating shared medical appointments and streamlining rooming practices. These improvements increased efficiency in patient care and helped control costs for the HCO.</p></sec></sec><sec id="S21"><title>Exploring the quality of technical assistance</title><p id="P35">Assessing technical support quality also offered IAP, as a funder, the opportunity to refine its technical assistance strategies for later cohorts. In particular, clinic staff were able to use IAP print resources, providing patients with information they could understand and use at home. Clinic staff also highlighted areas in need of additional technical support. They indicated they would continue to benefit from supplementary resources, such as print materials and online trainings.<sup><xref rid="FN7" ref-type="fn">3</xref></sup></p><p id="P36">These findings suggest that specified support from a funder such as IAP could help continue asthma QI initiatives and provide direction for expanded partnerships and policy examination. When evaluation data revealed opportunities to improve support for asthma QI initiatives, IAP developed a post-evaluation action plan and is working to address evaluation findings that are within IAP&#x02019;s control. These action items are the basis for IAP&#x02019;s continued QI efforts to help advance the asthma QI projects it sponsors.</p></sec><sec id="S22"><title>Montana case study findings</title><p id="P37">Located in the northwestern part of the USA and over one thousand miles away from Indiana, the Montana Asthma Control Program (MACP) has worked alongside IDOH and other NACP asthma programs for more than nine years. Considerably larger in size but less populated than Indiana, Montana has approximately 82 660 persons with asthma statewide representing 10% of the population (<xref rid="R42" ref-type="bibr">42</xref>). A large portion of persons with asthma in Montana report that their disease is not in control. That includes 56% reporting asthma symptoms in the last week, 25% reporting asthma symptoms every day in the last month, and 50% saying that they limited activities because of asthma in the last year.<sup><xref rid="FN8" ref-type="fn">4</xref></sup> Montana has about 2,570 asthma discharges from the ED each year.<sup><xref rid="FN9" ref-type="fn">5</xref></sup></p></sec><sec id="S23"><title>Montana asthma QI efforts</title><p id="P38">The goal of the Asthma Health Care Quality Improvement (AHCQI) initiative is to improve health outcomes for people living with asthma in Montana through partnerships with CAHs.<sup><xref rid="FN10" ref-type="fn">6</xref></sup> Participating health care facilities receive an initial educational presentation and teaching tools and undergo a pre- and post-implementation chart abstraction to monitor progress. They also receive regular technical assistance through scheduled teleconferences and site visits with MACP staff to support their efforts.</p><p id="P39">The AHCQI program is composed of Asthma Hospital and Emergency Department Patient Education, Action Plan, and Discharge (AHEAD) Protocol; Diagnose and Manage Asthma (DMA) Protocol; and an Emergency Department (ED) Recognition Award.</p><sec id="S233"><title>The AHEAD protocol and ED recognition.</title><p id="P40">The Montana asthma QI started in 2011 with AHEAD Protocol activity, part of a multifaceted workplan designed to meet requirements for a grant from NACP (<xref rid="R20" ref-type="bibr">20</xref>). The AHEAD Protocol trains Montana health care providers on EPR-3 guidelines for asthma. The training helps them apply recommendations from guidelines for patient treatment, education, and follow-up with a primary care provider after discharge from an ED. As incentives, MACP provides grants of $2,000 for the AHEAD Protocol and $1000 per year for three years for the ED recognition program.</p></sec><sec id="S234"><title>The DMA protocol.</title><p id="P41">Using evaluation findings, MACP expanded its asthma QI activity to include the DMA protocol. Similar in design to the AHEAD Protocol, the DMA Protocol was designed to provide support to people interested in conducting asthma QI initiatives in primary care setting. The DMA protocol was implemented in 2015, largely in response to sites participating in the AHEAD protocol that wanted more information for their primary care patients.</p><p id="P42">MACP started its evaluation efforts in 2013, two years before Indiana asthma QI efforts started. This activity underwent evaluation in 2013, 2014, and 2018, and has had slight adjustments made to its design each time as a result. One asthma QI adjustment was to offer local EDs a chance to participate in the ED recognition program. As of 2015, to maintain contact and site&#x02019;s momentum, EDs can opt into the ED recognition program, which provides continued support for an additional three years.</p><p id="P43">Montana data collection methods included pre- and post-implementation chart abstractions, literature reviews, and interviews with project leaders at each site. Among sites that received follow-up chart abstraction, MACP found the rate of repeat ED visits decreased 41%. Decreased ED visits are a marker of improved asthma control (<xref rid="R45" ref-type="bibr">45</xref>). However, only five of the original 11 sites completed their post-implementation chart abstraction due to staff turnover and low interest. Because of low participation in post-project chart abstraction, MACP started a smaller planning evaluation in 2014 to discover the most effective strategies to increase satisfaction among sites that volunteered to apply the asthma QI. Since 2014, MACP funded QI projects for seven hospitals, with three hospitals participating in multiple years. MACP included interviews with QI coordinators from the Montana Diabetes Program and Montana Cardiovascular Health Program in the planning evaluation to learn from their long histories of successfully implementing QI in Montana.</p></sec></sec><sec id="S24"><title>Benefits of asthma QI initiatives</title><sec id="S25"><title>Benefits to clinic staff and other stakeholders.</title><p id="P44">The AHEAD protocol appeared to have led to positive gains for staff members and other stakeholders. Through interviews, MACP learned that CAH staff felt they had more knowledge about asthma, had increased awareness of the importance of documentation, and saw improved quality of care among facilities that were able to participate in chart abstractions.</p><p id="P45">Professional development was an unintended benefit for staff. Training resulted in staff members becoming more confident in promoting the importance of the asthma QI. In another example, one site leader who enhanced her proficiency in writing policies through her asthma QI now develops standardized policies for respiratory therapy department issues at her facility.</p></sec><sec id="S26"><title>Benefits to intervention sites.</title><p id="P46">MACP discovered two unintended benefits for their intervention sites. One unintended benefit for Montana&#x02019;s intervention sites was the spread of QI model to other types of programs within CAHs, such as tobacco cessation counseling. Another unintended benefit was a revenue increase associated with increased use of spirometry at one of the CAHs.</p></sec><sec id="S27"><title>Exploring the quality of technical assistance.</title><p id="P47">For Montana, evaluation was instrumental in pinpointing types of technical assistance that could help overcome barriers to successful implementation. Evaluation data were collected after each initial education presentation to participating facilities. HCO staff rated quality of materials and the instructor after each training presentation. MACP continues to collect data on technical assistance for its asthma QI grantees.</p><p id="P48">Planning evaluation was initially used to assess how to increase satisfaction among implementation sites. The 2014 planning evaluation was later used to discover what future technical assistance would be beneficial. MACP used the socioecological model to provide support for change at each level (<xref rid="R46" ref-type="bibr">46</xref>).</p><p id="P49">One finding from the planning evaluation data indicated that a memorandum of understanding (MOU) should be established to clarify expectations for providing technical assistance and ongoing communication between MACP and the HCO. Under the MOU, used to make expectations clear, participants would engage in quarterly calls for ongoing technical assistance from MACP. Final grant payments were to be issued after submission of post-project chart abstractions, thereby motivating participants to submit evaluation data on completion of the project.</p><p id="P50">The relationship between MACP&#x02019;s QI coordinator and project coordinators at participating sites was fostered by required quarterly calls, which allowed site coordinators to provide feedback to MACP&#x02019;s QI coordinator and offer site-specific technical assistance. The call schedule also allowed MACP&#x02019;s QI coordinator to make connections between HCOs so that participants could support each other, depending on issues they were facing at that time. Lastly, by allowing sites to opt into the ED recognition program, technical assistance for CAHs could continue past the original project timeline. The initial one-year timeline generally was insufficient to initiate and maintain change related to a QI activity.</p></sec></sec><sec id="S28"><title>Cross-state comparison of findings</title><p id="P51">Evaluation data revealed similarities in exploration of nontraditional benefits and quality of technical assistance across states. Indiana and Montana QI initiatives yielded benefits for staff and intervention sites (<xref rid="T2" ref-type="table">Table 2</xref>). Evaluation of technical assistance was offered in both states to further improve implementation of their asthma QI initiatives. An amplification of how Montana and Indiana varied in their benefits and use of technical assistance follows.</p><p id="P52">Staff in Indiana appreciated the discoveries that result from shared learning opportunities (e.g. comparing barriers across sites). One Montana staff member was able to apply the acquired knowledge on developing policies for asthma to the entire respiratory therapy department. Indiana found immediate benefits for its implementation sites, which included increases in the use of asthma action plans and electronic medical records. Montana observed that one staff indicated that they believed there was a revenue increase from improved quality of care and the asthma QI model, later spread to other programs within CAHs (e.g. tobacco cessation).</p><p id="P53">Although types of intervention benefits differed, the spillover from asthma QI initiatives to other health programs appeared to be similar. In Indiana, HCOs began to coordinate their services between asthma and other chronic diseases through scheduling appointments and streamlining rooming practices. In Montana, the QI approach was adopted for other diseases.</p><p id="P54">Exploring the quality of technical assistance appeared to be a fruitful endeavor. Montana and Indiana were able to use evaluation findings iteratively to determine how to guide their technical assistance. Across Indiana&#x02019;s three QI initiatives, clinic staff found technical assistance materials suitable in format and content for the needs of their patients. Indiana also was able to use earlier findings to improve the utility of technical assistance for later cohorts. Montana began a separate planning evaluation later in its project to correct some of the implementation challenges experienced by some implementation sites. It eventually developed a technical assistance mechanism to foster shared decision making between MACP&#x02019;s QI coordinator and local HCO coordinators.</p></sec></sec><sec id="S29"><title>Discussion</title><p id="P55">IAP and MACP were able to identify different levels of benefits from participation in asthma QI initiatives and apply evaluation data to improve quality of technical assistance. For asthma QI to be recognized as a best practice and applied successfully, compelling evidence from well-designed evaluation studies is needed. State health department investment of evaluation resources and active use of findings can increase understanding of contextual factors that influence local asthma QI initiatives and support evaluation capacity building (<xref rid="R27" ref-type="bibr">27</xref>,<xref rid="R47" ref-type="bibr">47</xref>,<xref rid="R48" ref-type="bibr">48</xref>).</p><sec id="S30"><title>Asthma QI benefits</title><p id="P56">Examining benefits (and even consequences) throughout program implementation can reveal adjustments needed to prevent later challenges in operations or outcomes. Evaluation can offer funders and other asthma QI end users new understanding of the benefits of the initiative. Frequently, it is the funder who defines what is beneficial (<xref rid="R31" ref-type="bibr">31</xref>,<xref rid="R32" ref-type="bibr">32</xref>). Based on the extant literature, asthma QI initiatives would logically offer many advantages through association with health improvements. However, in community systems, interventions rarely achieve widespread support, thereby increasing risk of aborting the asthma QI before outcomes are achieved. If these interventions are to be sustained long enough to achieve their expected health outcomes, programs must explain to stakeholders in the outset what the benefits are. Recognizing this need to identify markers of sustainability, IAP and MACP evaluation teams examined different levels of benefits: patients, staff, and implementation sites.</p><p id="P57">The evaluation teams found early evidence that asthma QI initiatives can help improve health outcomes, as seen in improved asthma control and reduced ED visits. Improved health outcomes are important accomplishments for these states because it shows that the asthma QI worked as intended. While improved health is considered an important outcome, evaluations should also examine what contextual factors may influence implementation and sustainability. IAP and MACP helped show why evaluations should expand beyond a focus on health outcomes. Evaluation targets might include benefits to staff and general improvements observed at an HCO (<xref rid="R20" ref-type="bibr">20</xref>,<xref rid="R49" ref-type="bibr">49</xref>).</p><p id="P58">Staff impressions are valuable for understanding how to expand and sustain an intervention. Including their perspectives could reveal importance of tasks that are not specifically included in an asthma QI but are very much essential to health outcomes (e.g. establishing trust with patients) (<xref rid="R35" ref-type="bibr">35</xref>). Staff lack of interest has been linked with QI project delays (<xref rid="R23" ref-type="bibr">23</xref>). IAP and MACP understood the value of staff perspectives and gathered these data in their evaluations. The positive impressions described by staff in both states suggest that these stakeholders would support continuing these asthma QI initiatives.</p><p id="P59">Also related to staff benefits are the improvements observed at participating HCOs. Despite both states demonstrating improved QI outcomes, there were initial differences in how sites fared in implementing the asthma QI, a finding that clarifies the importance of using evaluation to detect setting specific factors. IAP saw improvements in best practices, as documented in increased use of asthma action plans and electronic medical records. Indiana also saw that QI practices used for asthma were being adopted to other chronic diseases as did Montana.</p><p id="P60">Earlier in its project, MACP had difficulties achieving expected improvements because many sites found this project challenging to implement. That led MACP to collect more process evaluation data on how to make this experience more beneficial for intervention sites.</p></sec><sec id="S31"><title>Evaluation of quality of technical assistance</title><p id="P61">IAP and MACP examined the quality of technical assistance delivered to participating sites. When possible, evaluation should be used early in technical assistance development process. That could help create more appropriate content and identify goals for technical assistance (<xref rid="R3" ref-type="bibr">3</xref>,<xref rid="R26" ref-type="bibr">26</xref>).</p><p id="P62">IAP and MACP plan to use or have used evaluation repeatedly for their technical assistance. While evaluating asthma QI use, IAP and MACP were able to clarify whether or not training and material support were appropriate for asthma QI implementation sites. IAP learned that earlier technical assistance activities had enhanced intervention work, but recipients also listed other areas in which work could be improved further. MACP shifted to technical assistance strategies that were co-developed with grantees. Montana Asthma Control Program also set up quarterly calls with QI coordinators to actively collaborate on emerging technical assistance needs and discuss quality of technical assistance delivered.</p><p id="P63">Evaluation offers asthma QI stakeholders more precise information on needs of a particular site. Selecting technical assistance as an evaluation target offers funders and other end users&#x02019; information on how well sites were adopting QI models and what level of capacity was needed to sustain these interventions. The materials and training support are to be delivered as needed to ensure that asthma QI is applied as designed.</p></sec></sec><sec id="S32"><title>Conclusion</title><p id="P64">This case study highlights how state asthma programs can bolster success of local asthma QI projects by partnering with HCOs. QI initiatives are a promising prospect for the control of asthma. Indiana and Montana used evaluation to discover new benefits and emerging technical assistance needs of local HCOs. Such evaluations can offer insights on factors that influence how well a project might succeed. They can reveal who would benefit from such an initiative and what types of technical support are needed to strengthen HCOs&#x02019; impact. This project has implications for other heath initiatives that integrate public health and HCOs.</p></sec><sec id="S33"><title>Limitations</title><p id="P65">The cross-case analyse design was used to explore similarities in evaluation use in two independent evaluations of asthma QI initiatives. The findings revealed the promise of learning more about staff and HCO benefits and quality of technical assistance, but their generalizability is limited because study design, variability in access to data sources across states, and the intended use of these program evaluations were based on state-specific needs.</p></sec></body><back><ack id="S34"><title>Acknowledgment</title><p id="P66">We thank Jessie Fernandes (Montana Asthma Control Program), IDOH Office of Primary Care, and Maureen Wilce (CDC Asthma and Community Health Branch) for their scientific consultation and technical assistance in the development of evaluation, case analyses, and article.</p></ack><fn-group><fn fn-type="COI-statement" id="FN2"><p id="P67">Declaration of interest</p><p id="P68">The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.</p></fn><fn id="FN3"><p id="P69" content-type="publisher-disclaimer">Disclaimer</p></fn><fn id="FN4"><p id="P70" content-type="publisher-disclaimer">The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.</p></fn><fn id="FN5"><label>1.</label><p id="P71">Asthma action plan should have instruction on the appropriate medications for patient to use, how to recognize escalating symptoms, and recommended strategies for asthma-related emergencies.</p></fn><fn id="FN6"><label>2.</label><p id="P72">Using evaluation to improve evaluation was a third theme identified for both states. This theme is not discussed in this article.</p></fn><fn id="FN7"><label>3.</label><p id="P73">Other requests were beyond IAP&#x02019;s scope and included requests for spacers and other direct patient care supplies.</p></fn><fn id="FN8"><label>4.</label><p id="P74">
<ext-link xlink:href="https://dphhs.mt.gov/publichealth/brfss" ext-link-type="uri">https://dphhs.mt.gov/publichealth/brfss</ext-link>
</p></fn><fn id="FN9"><label>5.</label><p id="P75">
<ext-link xlink:href="https://dphhs.mt.gov/publichealth/brfss" ext-link-type="uri">https://dphhs.mt.gov/publichealth/brfss</ext-link>
</p></fn><fn id="FN10"><label>6.</label><p id="P76">Most hospitals in Montana are designated critical access hospitals. Often, they are the only emergency medical facility for hundreds of miles, providing primary care and emergency care.</p></fn><fn id="FN11"><label>7.</label><p id="P77">A comprehensive description of these health outcome data are reported in Sirikan, Heins, Nesvold, Pinar, et al (2019) report on &#x0201c;Enhancing guidelines-based asthma care processes through a multi-state, multi-center quality improvement program.&#x0201d; <italic toggle="yes">Journal of Asthma, 56(4), 440&#x02013;450</italic>.</p></fn></fn-group><ref-list><title>References</title><ref id="R1"><label>1.</label><mixed-citation publication-type="webpage"><collab>Prevention CfDCa</collab>. <source>Asthma</source>. <year>2020</year> [<comment>accessed</comment>
<date-in-citation>2020 Jan 30</date-in-citation>]. <comment><ext-link xlink:href="https://www.cdc.gov/asthma/default.htm" ext-link-type="uri">https://www.cdc.gov/asthma/default.htm</ext-link>.</comment></mixed-citation></ref><ref id="R2"><label>2.</label><mixed-citation publication-type="journal"><name><surname>Volerman</surname><given-names>A</given-names></name>, <name><surname>Chin</surname><given-names>MH</given-names></name>, <name><surname>Press</surname><given-names>VG</given-names></name>. <article-title>Solutions for asthma disparities</article-title>. <source>Pediatrics</source>. <year>2017</year>;<volume>139</volume>(<issue>3</issue>):<fpage>e20162546</fpage>. doi:<pub-id pub-id-type="doi">10.1542/peds.2016-2546</pub-id>.<pub-id pub-id-type="pmid">28228500</pub-id></mixed-citation></ref><ref id="R3"><label>3.</label><mixed-citation publication-type="journal"><name><surname>Le</surname><given-names>LT</given-names></name>, <name><surname>Anthony</surname><given-names>BJ</given-names></name>, <name><surname>Bronheim</surname><given-names>SM</given-names></name>, <name><surname>Holland</surname><given-names>CM</given-names></name>, <name><surname>Perry</surname><given-names>D</given-names></name>. <article-title>Research. A technical assistance model for guiding service and systems change</article-title>. <source>J Behav Health Serv Res</source>. <year>2016</year>;<volume>43</volume>(<issue>3</issue>):<fpage>380</fpage>&#x02013;<lpage>395</lpage>. doi:<pub-id pub-id-type="doi">10.1007/s11414-014-9439-2</pub-id>.<pub-id pub-id-type="pmid">25239308</pub-id></mixed-citation></ref><ref id="R4"><label>4.</label><mixed-citation publication-type="book"><collab>Medicine Io</collab>. <source>Unequal treatment: confronting racial and ethnic disparities in health care</source>. <publisher-loc>Washington (DC)</publisher-loc>: <publisher-name>The National Academies Press</publisher-name>; <year>2003</year>.</mixed-citation></ref><ref id="R5"><label>5.</label><mixed-citation publication-type="webpage"><collab>National Heart, Lung, and Blood Institute</collab>. <source>National Asthma Control Initiative: Reducing asthma disparities</source>. <year>2019</year>. <comment><ext-link xlink:href="http://www.nhlbi.nih.gov/health-pro/resources/lung/naci/discover/disparities.htm" ext-link-type="uri">http://www.nhlbi.nih.gov/health-pro/resources/lung/naci/discover/disparities.htm</ext-link></comment></mixed-citation></ref><ref id="R6"><label>6.</label><mixed-citation publication-type="journal"><name><surname>Nelson</surname><given-names>A</given-names></name>
<article-title>Unequal treatment: confronting racial and ethnic disparities in health care</article-title>. <source>J Natl Med Assoc</source>. <year>2002</year>;<volume>94</volume>(<issue>8</issue>):<fpage>666</fpage>&#x02013;<lpage>668</lpage>.<pub-id pub-id-type="pmid">12152921</pub-id></mixed-citation></ref><ref id="R7"><label>7.</label><mixed-citation publication-type="journal"><name><surname>Slejko</surname><given-names>JF</given-names></name>, <name><surname>Ghushchyan</surname><given-names>VH</given-names></name>, <name><surname>Sucher</surname><given-names>B</given-names></name>, <name><surname>Globe</surname><given-names>DR</given-names></name>, <name><surname>Lin</surname><given-names>S-L</given-names></name>, <name><surname>Globe</surname><given-names>G</given-names></name>, <name><surname>Sullivan</surname><given-names>PW</given-names></name>. <article-title>Asthma control in the United States, 2008&#x02013;2010: indicators of poor asthma control</article-title>. <source>J Allergy Clin Immunol</source>. <year>2014</year>;<volume>133</volume>(<issue>6</issue>): <fpage>1579</fpage>&#x02013;<lpage>1587</lpage>. doi:<pub-id pub-id-type="doi">10.1016/j.jaci.2013.10.028</pub-id>.<pub-id pub-id-type="pmid">24331376</pub-id></mixed-citation></ref><ref id="R8"><label>8.</label><mixed-citation publication-type="journal"><name><surname>Tschudy</surname><given-names>MM</given-names></name>, <name><surname>Cheng</surname><given-names>TL</given-names></name>. <article-title>The &#x0201c;black box&#x0201d; of racial disparities in asthma</article-title>. <source>JAMA Pediatr</source>. <year>2016</year>;<volume>170</volume>(<issue>7</issue>): <fpage>644</fpage>&#x02013;<lpage>645</lpage>. doi:<pub-id pub-id-type="doi">10.1001/jamapediatrics.2016.0564</pub-id>.<pub-id pub-id-type="pmid">27183533</pub-id></mixed-citation></ref><ref id="R9"><label>9.</label><mixed-citation publication-type="journal"><name><surname>Cremer</surname><given-names>NM</given-names></name>, <name><surname>Baptist</surname><given-names>AP</given-names></name>. <article-title>Race and asthma outcomes in older adults: results from the National Asthma Survey</article-title>. <source>J Allergy Clin Immunol Pract</source>. <year>2020</year>;<volume>8</volume>(<issue>4</issue>): <fpage>1294</fpage>&#x02013;<lpage>1301.e7</lpage>. doi:<pub-id pub-id-type="doi">10.1016/j.jaip.2019.12.014</pub-id>.<pub-id pub-id-type="pmid">32035849</pub-id></mixed-citation></ref><ref id="R10"><label>10.</label><mixed-citation publication-type="journal"><name><surname>Akinbami</surname><given-names>LJ</given-names></name>, <name><surname>Simon</surname><given-names>AE</given-names></name>, <name><surname>Rossen</surname><given-names>LM</given-names></name>. <article-title>Changing trends in asthma prevalence among children</article-title>. <source>Pediatrics</source>. <year>2016</year>;<volume>137</volume>(<issue>1</issue>):<fpage>e20152354</fpage>. doi:<pub-id pub-id-type="doi">10.1542/peds.2015-2354</pub-id>.</mixed-citation></ref><ref id="R11"><label>11.</label><mixed-citation publication-type="book"><collab>Council AaAFoAaTNP</collab>. <source>Ethnic disparities in the burden and treatment of asthma</source>. <publisher-loc>Reston (VA)</publisher-loc>: <publisher-name>Asthma and Allergy Foundation of America</publisher-name>; <year>2005</year>.</mixed-citation></ref><ref id="R12"><label>12.</label><mixed-citation publication-type="journal"><name><surname>Canino</surname><given-names>G</given-names></name>, <name><surname>McQuaid</surname><given-names>EL</given-names></name>, <name><surname>Rand</surname><given-names>CS</given-names></name>. <article-title>Addressing asthma health disparities: a multilevel challenge</article-title>. <source>J Allergy Clin Immunol</source>. <year>2009</year>;<volume>123</volume>(<issue>6</issue>):<fpage>1209</fpage>&#x02013;<lpage>1217</lpage>. doi:<pub-id pub-id-type="doi">10.1016/j.jaci.2009.02.043</pub-id>.<pub-id pub-id-type="pmid">19447484</pub-id></mixed-citation></ref><ref id="R13"><label>13.</label><mixed-citation publication-type="book"><collab>Administration HRaS</collab>. <source>Quality improvement</source>. <publisher-loc>Washington (DC)</publisher-loc>: <publisher-name>United States Department of Health and Human Services</publisher-name>; <year>2011</year>.</mixed-citation></ref><ref id="R14"><label>14.</label><mixed-citation publication-type="journal"><name><surname>Bravata</surname><given-names>DM</given-names></name>, <name><surname>Gienger</surname><given-names>AL</given-names></name>, <name><surname>Holty</surname><given-names>J-EC</given-names></name>, <name><surname>Sundaram</surname><given-names>V</given-names></name>, <name><surname>Khazeni</surname><given-names>N</given-names></name>, <name><surname>Wise</surname><given-names>PH</given-names></name>, <name><surname>McDonald</surname><given-names>KM</given-names></name>, <name><surname>Owens</surname><given-names>DK</given-names></name>. <article-title>Quality improvement strategies for children with asthma: a systematic review</article-title>. <source>Arch Pediatr Adolesc Med</source>. <year>2009</year>;<volume>163</volume>(<issue>6</issue>):<fpage>572</fpage>&#x02013;<lpage>581</lpage>. doi:<pub-id pub-id-type="doi">10.1001/archpediatrics.2009.63</pub-id>.<pub-id pub-id-type="pmid">19487615</pub-id></mixed-citation></ref><ref id="R15"><label>15.</label><mixed-citation publication-type="book"><collab>National Asthma Education and Prevention Program TEPotDaMoA</collab>. <source>Export Panel Report 3: guidelines for the diagnosis and management of asthma</source>. <publisher-loc>Bethesda (MD)</publisher-loc>: <publisher-name>National Heart Lung, and Blood Institute</publisher-name>; <year>2007</year>.</mixed-citation></ref><ref id="R16"><label>16.</label><mixed-citation publication-type="journal"><name><surname>Carlton</surname><given-names>BG</given-names></name>, <name><surname>Lucas</surname><given-names>DO</given-names></name>, <name><surname>Ellis</surname><given-names>EF</given-names></name>, <name><surname>Conboy-Ellis</surname><given-names>K</given-names></name>, <name><surname>Shoheiber</surname><given-names>O</given-names></name>, <name><surname>Stempel</surname><given-names>DA</given-names></name>. <article-title>The status of asthma control and asthma prescribing practices in the United States: results of a large prospective asthma control survey of primary care practices</article-title>. <source>J Asthma</source>. <year>2005</year>; <volume>42</volume>(<issue>7</issue>):<fpage>529</fpage>&#x02013;<lpage>535</lpage>. doi:<pub-id pub-id-type="doi">10.1081/JAS-67000</pub-id>.<pub-id pub-id-type="pmid">16169784</pub-id></mixed-citation></ref><ref id="R17"><label>17.</label><mixed-citation publication-type="journal"><name><surname>Parikh</surname><given-names>K</given-names></name>, <name><surname>Keller</surname><given-names>S</given-names></name>, <name><surname>Ralston</surname><given-names>S</given-names></name>. <article-title>Inpatient quality improvement interventions for asthma: a meta-analysis</article-title>. <source>Pediatrics</source>. <year>2018</year>;<volume>141</volume>(<issue>5</issue>):<fpage>e20173334</fpage>. doi:<pub-id pub-id-type="doi">10.1542/peds.2017-3334</pub-id>.<pub-id pub-id-type="pmid">29622722</pub-id></mixed-citation></ref><ref id="R18"><label>18.</label><mixed-citation publication-type="book"><name><surname>Coffey</surname><given-names>RH</given-names></name>, <name><surname>Adamson</surname><given-names>DM</given-names></name>, <name><surname>Matthews</surname><given-names>TL</given-names></name>, <name><surname>Sewell</surname><given-names>J</given-names></name>, <name><surname>Kassed</surname><given-names>CA</given-names></name>. <source>Asthma care quality improvement: a resource guide for state action</source>. <publisher-loc>Rockville (MD)</publisher-loc>: <publisher-name>Agency for Healthcare Research and Quality</publisher-name>; <year>2006</year>.</mixed-citation></ref><ref id="R19"><label>19.</label><mixed-citation publication-type="journal"><name><surname>Rojanasarot</surname><given-names>S</given-names></name>, <name><surname>Heins Nesvold</surname><given-names>J</given-names></name>, <name><surname>Karaca-Mandic</surname><given-names>P</given-names></name>, <name><surname>St Peter</surname><given-names>WL</given-names></name>, <name><surname>Wolfson</surname><given-names>J</given-names></name>, <name><surname>Schommer</surname><given-names>JC</given-names></name>, <name><surname>Carlson</surname><given-names>AM</given-names></name>. <article-title>Enhancing guideline-based asthma care processes through a multi-state, multi-center quality improvement program</article-title>. <source>J Asthma</source>. <year>2019</year>;<volume>56</volume>(<issue>4</issue>):<fpage>440</fpage>&#x02013;<lpage>450</lpage>. doi:<pub-id pub-id-type="doi">10.1080/02770903.2018.1463378</pub-id>.<pub-id pub-id-type="pmid">29641271</pub-id></mixed-citation></ref><ref id="R20"><label>20.</label><mixed-citation publication-type="book"><collab>Prevention CfDCa</collab>. <source>Comprehensive asthma control through evidence-based strategies and public health&#x02014;health care collaboration CDC-RFA-EH14&#x02013;1404</source>. <publisher-loc>Atlanta (GA)</publisher-loc>: <publisher-name>Centers for Disease Control and Prevention</publisher-name>; <year>2014</year>.</mixed-citation></ref><ref id="R21"><label>21.</label><mixed-citation publication-type="journal"><name><surname>Wagner</surname><given-names>EH</given-names></name>. <article-title>Chronic disease management: what will it take to improve care for chronic illness?</article-title>
<source>Eff Clin Pract</source>. <year>1998</year>;<volume>1</volume>(<issue>1</issue>):<fpage>2</fpage>&#x02013;<lpage>4</lpage>.<pub-id pub-id-type="pmid">10345255</pub-id></mixed-citation></ref><ref id="R22"><label>22.</label><mixed-citation publication-type="book"><collab>Prevention CfDCa</collab>. <source>Learning and growing through evaluation: state asthma program evaluation guide</source>. <publisher-loc>Atlanta (GA)</publisher-loc>: <publisher-name>Centers for Disease Control and Prevention</publisher-name>; <year>2010</year>.</mixed-citation></ref><ref id="R23"><label>23.</label><mixed-citation publication-type="book"><collab>Improvemernt IfH</collab>. <source>How to beat the boring aspects of QI [Internet]: Institute for Healthcare Improvement</source>. <publisher-name>Podcast</publisher-name>. <comment><ext-link xlink:href="http://www.ihi.org/education/AudioVideo/WIHI/how-to-beat-the-boring-aspects-of-qi/Pages/default.aspx" ext-link-type="uri">http://www.ihi.org/education/AudioVideo/WIHI/how-to-beat-the-boring-aspects-of-qi/Pages/default.aspx</ext-link></comment>; <year>2019</year>.</mixed-citation></ref><ref id="R24"><label>24.</label><mixed-citation publication-type="book"><collab>Medicine Io</collab>. <source>The learning healthcare system: workshop summary</source>. <publisher-loc>Washington (DC)</publisher-loc>: <publisher-name>The National Academies Press</publisher-name>; <year>2007</year>.</mixed-citation></ref><ref id="R25"><label>25.</label><mixed-citation publication-type="book"><name><surname>Matter</surname><given-names>R</given-names></name>
<source>Technical report: evaluation of technical assistance services provided by the ADA National Network</source>. <publisher-loc>Seattle (WA)</publisher-loc>: <publisher-name>ADA Knowledge Translation Center</publisher-name>; <year>2017</year>.</mixed-citation></ref><ref id="R26"><label>26.</label><mixed-citation publication-type="book"><collab>Childrens&#x02019;s Bureau</collab>. <source>Supporting change in child welfare: an evaluation of training and technical assistance</source>. <publisher-loc>Washington, DC</publisher-loc>: <publisher-name>U.S. Department of Health and Human Services</publisher-name>; <year>2015</year>.</mixed-citation></ref><ref id="R27"><label>27.</label><mixed-citation publication-type="book"><name><surname>Perkins</surname><given-names>AN</given-names></name>. <source>A case study on the utility of sustained evaluation practice in public health partnership</source>. <publisher-name>INQUIRY</publisher-name>. <year>2018</year>;<volume>55</volume>:<fpage>1</fpage>&#x02013;<lpage>9</lpage>.</mixed-citation></ref><ref id="R28"><label>28.</label><mixed-citation publication-type="book"><name><surname>Miles</surname><given-names>MB</given-names></name>, <name><surname>Huberman</surname><given-names>AM</given-names></name>, <name><surname>Salda&#x000f1;a</surname><given-names>J</given-names></name>. <source>Qualitative data analysis: a methods sourcebook</source>. <edition>4th ed</edition> . <publisher-loc>Los Angeles (CA)</publisher-loc>: <publisher-name>SAGE</publisher-name>; <year>2020</year>.</mixed-citation></ref><ref id="R29"><label>29.</label><mixed-citation publication-type="journal"><name><surname>Mansour</surname><given-names>ME</given-names></name>, <name><surname>Rose</surname><given-names>B</given-names></name>, <name><surname>Toole</surname><given-names>K</given-names></name>, <name><surname>Luzader</surname><given-names>CP</given-names></name>, <name><surname>Atherton</surname><given-names>HD</given-names></name>. <article-title>Pursuing perfection: an asthma quality improvement initiative in school-based health centers with community partners</article-title>. <source>Public Health Rep</source>. <year>2008</year>;<volume>123</volume>(<issue>6</issue>): <fpage>717</fpage>&#x02013;<lpage>730</lpage>. doi:<pub-id pub-id-type="doi">10.1177/003335490812300608</pub-id>.<pub-id pub-id-type="pmid">19711653</pub-id></mixed-citation></ref><ref id="R30"><label>30.</label><mixed-citation publication-type="book"><name><surname>Hughes</surname><given-names>RG</given-names></name>. <part-title>Tools and strategies for quality improvement and patient safety</part-title>. In: <name><surname>Hughes</surname><given-names>RG</given-names></name>, ed. <source>Patient safety and quality: an evidence-based handbook for nurses</source>. <publisher-loc>Rockville, MD</publisher-loc>: <publisher-name>Agency for Healthcare Research and Quality</publisher-name>; <year>2008</year>.</mixed-citation></ref><ref id="R31"><label>31.</label><mixed-citation publication-type="journal"><name><surname>Chelimsky</surname><given-names>E</given-names></name>
<article-title>Public-interest values and program sustainability: some implications for evaluation practice</article-title>. <source>Am J Eval</source>. <year>2014</year>;<volume>35</volume>(<issue>4</issue>):<fpage>527</fpage>&#x02013;<lpage>542</lpage>. doi:<pub-id pub-id-type="doi">10.1177/1098214014549068</pub-id>.</mixed-citation></ref><ref id="R32"><label>32.</label><mixed-citation publication-type="journal"><name><surname>Owczarzak</surname><given-names>J</given-names></name>, <name><surname>Broaddus</surname><given-names>M</given-names></name>, <name><surname>Pinkerton</surname><given-names>S</given-names></name>. <article-title>Audit culture: unintended consequences of accountability practices in evidence-based programs</article-title>. <source>Am J Eval</source>. <year>2016</year>;<volume>37</volume>(<issue>3</issue>): <fpage>326</fpage>&#x02013;<lpage>343</lpage>. doi:<pub-id pub-id-type="doi">10.1177/1098214015603502</pub-id>.<pub-id pub-id-type="pmid">28989275</pub-id></mixed-citation></ref><ref id="R33"><label>33.</label><mixed-citation publication-type="journal"><name><surname>Benjamin</surname><given-names>LM</given-names></name>. <article-title>Nonprofit organizations and outcome measurement: from tracking program activities to focusing on frontline work</article-title>. <source>Am J Eval</source>. <year>2012</year>;<volume>33</volume>(<issue>3</issue>): <fpage>431</fpage>&#x02013;<lpage>447</lpage>. doi:<pub-id pub-id-type="doi">10.1177/1098214012440496</pub-id>.</mixed-citation></ref><ref id="R34"><label>34.</label><mixed-citation publication-type="journal"><name><surname>Carman</surname><given-names>JG</given-names></name>, <name><surname>Fredericks</surname><given-names>KA</given-names></name>. <article-title>Evaluation capacity and nonprofit organizations: is the glass half-empty or half-full?</article-title>
<source>Am J Eval</source>. <year>2010</year>;<volume>31</volume>(<issue>1</issue>):<fpage>84</fpage>&#x02013;<lpage>104</lpage>. doi:<pub-id pub-id-type="doi">10.1177/1098214009352361</pub-id>.</mixed-citation></ref><ref id="R35"><label>35.</label><mixed-citation publication-type="journal"><name><surname>Hwang</surname><given-names>H</given-names></name>, <name><surname>Powell</surname><given-names>WW</given-names></name>. <article-title>The rationalization of charity: the influences of professionalism in the nonprofit sector</article-title>. <source>Adm Sci Q</source>. <year>2009</year>;<volume>54</volume>(<issue>2</issue>):<fpage>268</fpage>&#x02013;<lpage>298</lpage>. doi:<pub-id pub-id-type="doi">10.2189/asqu.2009.54.2.268</pub-id>.</mixed-citation></ref><ref id="R36"><label>36.</label><mixed-citation publication-type="journal"><name><surname>Ragin</surname><given-names>CC</given-names></name>. <article-title>Turning the tables: how case-oriented methods challenge variable-oriented methods</article-title>. <source>Comp Soc Res</source>. <year>1997</year>;<volume>16</volume>:<fpage>27</fpage>&#x02013;<lpage>42</lpage>.</mixed-citation></ref><ref id="R37"><label>37.</label><mixed-citation publication-type="book"><name><surname>Miles</surname><given-names>MBH</given-names></name>, <name><surname>Michael</surname><given-names>A</given-names></name>. <source>Qualitative data analysis: an expanded sourcebook</source>. <publisher-loc>Thousand Oaks (CA)</publisher-loc>: <publisher-name>Sage Publications</publisher-name>; <year>1994</year>.</mixed-citation></ref><ref id="R38"><label>38.</label><mixed-citation publication-type="book"><name><surname>Creswell</surname><given-names>JW</given-names></name>. <source>Research design: qualitative, quantitative, and mixed methods approaches</source>. <edition>2nd ed</edition>. <publisher-loc>Thousand Oaks (CA)</publisher-loc>: <publisher-name>Sage</publisher-name>; <year>2007</year>.</mixed-citation></ref><ref id="R39"><label>39.</label><mixed-citation publication-type="book"><name><surname>Lincoln</surname><given-names>YS</given-names></name>, <name><surname>Guba</surname><given-names>EG</given-names></name>. <source>Naturalistic inquiry</source>. <publisher-loc>Newbury Park (CA)</publisher-loc>: <publisher-name>Sage Publications</publisher-name>; <year>1985</year>.</mixed-citation></ref><ref id="R40"><label>40.</label><mixed-citation publication-type="book"><name><surname>Creswell</surname><given-names>JW</given-names></name>. <source>Qualitative inquiry and research design: choosing among five approaches</source>. <edition>3rd ed</edition>. <publisher-loc>Thousand Oaks (CA)</publisher-loc>: <publisher-name>Sage</publisher-name>; <year>2007</year>.</mixed-citation></ref><ref id="R41"><label>41.</label><mixed-citation publication-type="journal"><name><surname>Elliott</surname><given-names>V</given-names></name>
<article-title>Thinking about the coding process in qualitative data analysis</article-title>. <source>Qual Report</source>. <year>2018</year>;<volume>23</volume>(<issue>11</issue>):<fpage>11</fpage>.</mixed-citation></ref><ref id="R42"><label>42.</label><mixed-citation publication-type="book"><collab>Indiana BRFSS Office</collab>. <source>Behavioral risk factor surveillance system survey data Indianapolis</source>. <publisher-name>Indiana Department of Public Health</publisher-name>; <year>2017</year>.</mixed-citation></ref><ref id="R43"><label>43.</label><mixed-citation publication-type="other"><collab>Centers for Disease Control and Prevention, National Center for Environmental Health</collab>. <source>Data, statistics, and surveillance: asthma surveillance data</source>. <year>2016</year>.</mixed-citation></ref><ref id="R44"><label>44.</label><mixed-citation publication-type="book"><source>Indiana Hospital Discharge Data Indianapolis</source>. <publisher-name>Indiana Department of Public Health</publisher-name>; <year>2017</year>.</mixed-citation></ref><ref id="R45"><label>45.</label><mixed-citation publication-type="journal"><name><surname>Guilbert</surname><given-names>TW</given-names></name>, <name><surname>Garris</surname><given-names>C</given-names></name>, <name><surname>Jhingran</surname><given-names>P</given-names></name>, <name><surname>Bonafede</surname><given-names>M</given-names></name>, <name><surname>Tomaszewski</surname><given-names>KJ</given-names></name>, <name><surname>Bonus</surname><given-names>T</given-names></name>, <name><surname>Hahn</surname><given-names>RM</given-names></name>, <name><surname>Schatz</surname><given-names>M</given-names></name>. <article-title>Asthma that is not well-controlled is associated with increased healthcare utilization and decreased quality of life</article-title>. <source>J Asthma</source>. <year>2011</year>;<volume>48</volume>(<issue>2</issue>):<fpage>126</fpage>&#x02013;<lpage>132</lpage>. doi:<pub-id pub-id-type="doi">10.3109/02770903.2010.535879</pub-id>.<pub-id pub-id-type="pmid">21128880</pub-id></mixed-citation></ref><ref id="R46"><label>46.</label><mixed-citation publication-type="journal"><name><surname>McLeroy</surname><given-names>KR</given-names></name>, <name><surname>Bibeau</surname><given-names>D</given-names></name>, <name><surname>Steckler</surname><given-names>A</given-names></name>, <name><surname>Glanz</surname><given-names>K</given-names></name>. <article-title>An ecological perspective on health promotion programs</article-title>. <source>Health Educ Q</source>. <year>1988</year>;<volume>15</volume>(<issue>4</issue>):<fpage>351</fpage>&#x02013;<lpage>377</lpage>. doi:<pub-id pub-id-type="doi">10.1177/109019818801500401</pub-id>.<pub-id pub-id-type="pmid">3068205</pub-id></mixed-citation></ref><ref id="R47"><label>47.</label><mixed-citation publication-type="book"><name><surname>Fierro</surname><given-names>LA</given-names></name>. <source>Clarifying the connections: evaluation capacity and intended outcomes [dissertation]</source>. <publisher-loc>Ann Arbor (MI)</publisher-loc>: <publisher-name>The Clairmont Graduate University</publisher-name>; <year>2012</year>.</mixed-citation></ref><ref id="R48"><label>48.</label><mixed-citation publication-type="journal"><name><surname>Preskill</surname><given-names>H</given-names></name>, <name><surname>Boyle</surname><given-names>S</given-names></name>. <article-title>A multidisciplinary model of evaluation capacity building</article-title>. <source>Am J Eval</source>. <year>2008</year>;<volume>29</volume>(<issue>4</issue>): <fpage>443</fpage>&#x02013;<lpage>459</lpage>. doi:<pub-id pub-id-type="doi">10.1177/1098214008324182</pub-id>.</mixed-citation></ref><ref id="R49"><label>49.</label><mixed-citation publication-type="book"><collab>World Health Organization</collab>. <source>The case for investing in public health: a public health summary report for EPHO 8</source>. <publisher-loc>Copenhagen</publisher-loc>: <publisher-name>World Health Organization</publisher-name>; <year>2014</year>.</mixed-citation></ref></ref-list></back><floats-group><fig position="float" id="F1"><label>Figure 1.</label><caption><p id="P78">Comparison of Indiana and State Asthma QIs.</p></caption><graphic xlink:href="nihms-1789613-f0001" position="float"/></fig><table-wrap position="float" id="T1" orientation="landscape"><label>Table 1.</label><caption><p id="P79">Comparison of evaluation methods for Indiana and Montana.</p></caption><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" valign="middle" span="1"/><col align="left" valign="middle" span="1"/><col align="left" valign="middle" span="1"/></colgroup><thead><tr><th align="left" valign="middle" rowspan="1" colspan="1">Evaluation findings</th><th align="center" valign="middle" rowspan="1" colspan="1">Indiana</th><th align="center" valign="middle" rowspan="1" colspan="1">Montana</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Single and Group Interviews</td><td align="left" valign="top" rowspan="1" colspan="1">Interviews with clinic staff</td><td align="left" valign="top" rowspan="1" colspan="1">Single and group interviews with project leaders</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Surveys</td><td align="left" valign="top" rowspan="1" colspan="1">Provider surveys</td><td align="left" valign="top" rowspan="1" colspan="1">N/A</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Chart Abstraction</td><td align="left" valign="top" rowspan="1" colspan="1">N/A</td><td align="left" valign="top" rowspan="1" colspan="1">Pre and post chart abstractions</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Literature Review</td><td align="left" valign="top" rowspan="1" colspan="1">N/A</td><td align="left" valign="top" rowspan="1" colspan="1">Literature included trends on quality<break/>improvement trends, asthma, and burnout experienced by health care workers</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Performance Measurement Data</td><td align="left" valign="top" rowspan="1" colspan="1">Monthly and quarterly data health outcome data from participating sites<sup><xref rid="FN11" ref-type="fn">7</xref></sup></td><td align="left" valign="top" rowspan="1" colspan="1">N/A</td></tr></tbody></table></table-wrap><table-wrap position="float" id="T2" orientation="landscape"><label>Table 2.</label><caption><p id="P80">Comparison of evaluation findings for Indiana and Montana.</p></caption><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" valign="middle" span="1"/><col align="left" valign="middle" span="1"/><col align="left" valign="middle" span="1"/></colgroup><thead><tr><th align="left" valign="middle" rowspan="1" colspan="1">Evaluation findings</th><th align="center" valign="middle" rowspan="1" colspan="1">Indiana</th><th align="center" valign="middle" rowspan="1" colspan="1">Montana</th></tr></thead><tbody><tr><td align="left" valign="top" rowspan="1" colspan="1">Benefits to clinic staff and other stakeholders</td><td align="left" valign="top" rowspan="1" colspan="1">Increased ability to create changes to rooming practices, increased opportunity for shared learning, provided support in encouraging physicians</td><td align="left" valign="top" rowspan="1" colspan="1">Increased knowledge about asthma, increased awareness of the importance of documentation, and improved continuity of care</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Benefits to intervention site</td><td align="left" valign="top" rowspan="1" colspan="1">Diffusion of the QI practices for other diseases across HCOs, increased use of asthma action plans and use of electronic medical record systems</td><td align="left" valign="top" rowspan="1" colspan="1">Diffusion of the QI model to other diseases across HCOs, anecdotal observation of revenue increase for one site</td></tr><tr><td align="left" valign="top" rowspan="1" colspan="1">Quality of technical assistance</td><td align="left" valign="top" rowspan="1" colspan="1">Identified need for continued training and materials</td><td align="left" valign="top" rowspan="1" colspan="1">Identified needs for changes to MOU and addition of a quarterly call</td></tr></tbody></table></table-wrap></floats-group></article>