A two-phased literature search was conducted through PubMed in October and November of 2019. The literature search was not intended to be systematic, but rather to provide background information for a working group of subject matter experts who would develop the new model. In light of this modest objective and a short time frame, the reviewer searched only one database. In the first phase, we reviewed articles that included the term “maturity model” in the title or abstract and were indexed with the MeSH term “public health” or any of its sub-terms. We reviewed articles detailing the development of maturity models in public health settings, with the goal of identifying elements to include in a NITAG maturity model. Previous maturity models developed for public health settings include between four and six levels of maturity. [9,12] Most public health maturity models are structured like academic scoring rubrics, describing increasingly advanced standards for specific features and functions. Alternatively, some models are structured like checklists, identifying binary indicators that are expected at increasing levels of maturity. A variety of scoring schemas are used in existing models. Some models generate a maturity score for each indicator or function. Other models produce only an overall score that summarizes maturity across all indicators or functions.

The second phase of the literature search included results of a PubMed search for articles whose abstracts included the terms “NITAG” and at least one of the following: establish/establishment, strengthen/strengthening, evaluate/evaluation, and sustainability. This phase also included reports from the Global NITAG Network (GNN) annual meetings and existing evaluation tools including the NITAG functionality indicators, the SIVAC Evaluating NITAG Performance Practical Tool, and the WHO/CDC NITAG Simplified Assessment Tool. A total of 21 publications were reviewed in phase 2 (Table 1).

Published guidance for strengthening and evaluating NITAGs was compiled and grouped by thematic areas, e.g., conflict of interest policy, meeting frequency, and secured funding. These thematic areas were then further grouped into one of seven broader categories called indicators: establishment (regarding the NITAG’s legislative or administrative basis, terms of reference (ToR), and membership); independence/bias (regarding the conflict of interest process, transparency, and independence from government, industry, and interest groups); resources (regarding access to material, informational, and administrative support), standard operating procedures (SOP) (regarding meeting logistics, communication and reports, and the annual work plan); developing recommendations (regarding the quality of the process for collecting and evaluating evidence); integration into the policy-making process (regarding governmental solicitation, consideration, and implementation of recommendations); and stakeholder recognition (regarding the NITAG’s relationship with scientific and professional organizations, the Global NITAG Network, and the public).

Conflicting guidance was highlighted for focused discussion by the working group. For example, Duclos (2010) recommended public dissemination of meeting minutes, while Blau et al. (2013) argued against including this practice in NITAG evaluations because of limited understandability, ease of collection, and/or perceived usefulness. [1,13].

The U.S. Centers for Disease Control and Prevention (CDC) convened a working group (WG) with representatives from several organizations supporting immunization policymaking and programs. The group included members from CDC, the Chinese Center for Disease Control and Prevention (China CDC), WHO (including the Headquarters, the Regional Office for Africa, and the Pan American Health Organization), the Global NITAG Network, and the Wellcome Trust. Each member had worked with national and/or international NITAGs, and their collective experience covered a geographically and economically diverse range of NITAG settings. The WG therefore understood the standards and practices for a wide variety of NITAGs to fulfill their mandates.

The WG met monthly via teleconference between November 2019 and June 2021. The WG’s first priority was to choose a structure and scoring scheme for the model. After reviewing key differences among the example models identified through the literature search, the WG agreed to depict five levels of maturity through a rubric, i.e., by detailing increasingly advanced approaches to key functions rather than through a cumulative sum of binary indicators. WG members endorsed this approach as most reflective of the process by which NITAGs develop. Finally, the WG decided that scoring for each indicator should be based on the highest level of maturity at which all criteria for that indicator are met. For example, if a NITAG meets all of the criteria for Developing and three of the four criteria for Intermediate, it should receive a score of “Developing” for the given indicator. This would provide a conservative estimate of the maturity status of the NITAG. The rubric structure and scoring process are described in depth in the “Tool Development” section.

After determining the structure and scoring scheme, the WG populated the model with criteria for each indicator. Two members who performed the literature review (ED and EK) proposed rubrics for the first three indicators–Establishment, Independence/Bias, and Resources–for review and input from the other members. The WG collectively reviewed the compiled written feedback and arrived at consensus about rubric contents and phrasing through discussion. For the remaining indicators, the WG made suggestions about how to populate the rubric and through an iterative discussion process, reached consensus on the draft model. The criteria for each indicator were developed to align as closely as possible with best practices based on the literature and WG members’ expert opinion. While populating the rubric, WG members also recorded definitions of key terms and articulated distinctions between potentially subjective indicators (such as the difference between basic and robust funding). Throughout the process of developing the rubric, WG members kept GNN members apprised of their objectives and progress, offering opportunities for input at conferences and partner meetings. Following WG member completion of the initial draft of the maturity model, the model was reviewed for completeness and consistency.

Between January and February 2022, we received input on a beta version of the NMAT from the GNN Steering Committee, WHO colleagues, the Strategic Advisory Group of Experts on Immunization (SAGE), and other NITAG partners. Most of this input related to overall clarity of the tool and the suitability of criteria for each level of maturity.

The NMAT was subsequently pilot tested with a convenience sample of five NITAGs from low, middle-, and high-income countries; this was done to ensure its usability and acceptability among countries representative of the intended audience. The first pilot, an in-country assessment of Uganda’s National Immunization Technical Advisory Group (UNITAG), was conducted by a joint team from WHO AFRO and U.S. CDC in February 2022. [14] For the remaining pilot tests (July 2022 to January 2023), country NITAGs and their Secretariats used the NMAT to conduct their own assessments. For Germany’s Standing Committee on Vaccination (STIKO) and China’s National Immunization Advisory Committee (NIAC), the NITAG used the tool and gave feedback on its usability and areas for improvement. For the Australian Technical Advisory Group on Immunization (ATAGI) and the U.S. Advisory Committee on Immunization Practices (ACIP), the NITAGs were provided the tool but the assessments were completed by several NITAG members and/or the Secretariat during virtual or face-to-face sessions where CDC and WHO provided support as needed; these sessions were an opportunity to collect feedback on the tool through observation as well as discussion.

The feedback collected from pilot testing led to refinements and improvements to almost all aspects of the NMAT. Key changes included: adding steps to instructions, clarifying terms and definitions in the glossary, and providing additional language for interpreting outputs based on country context. In response to questions and comments received from the pilot testers, we developed an online tutorial including an overview of the assessment process, a step-by-step video walkthrough of entering data into the Excel-based tool, and a list of frequently asked questions and answers.

Each of the seven indicators is assessed on its own Excel sheet. Moving one sub-indicator (row) at a time, assessors are instructed to mark with an “x” the maturity levels (columns) for which the NITAG meets all criteria (Fig. 1). The tool is coded with an algorithm that auto-populates results from the inputted information. Results are displayed in a column for each sub-indicator, and an overall maturity level for the indicator is assigned based on the highest level at which the criteria for all sub-indicators are met. The tool’s code also calculates the percentage of criteria met for each indicator and auto-populates a summary worksheet with these percentages.

Key terms developed by the WG, which appear in bold within the rubric, are further elaborated in a separate definition section of the tool. This structure allows the tool to contain comprehensive explanations and examples of maturity criteria while keeping wording in the tables concise. An instructions section includes suggested activities to do before, during, and after using the tool. For example, assessment team members are instructed to read through the tool in its entirety and make notes before meeting as a group.

The beta version introduced an optional data collection tool and a summary worksheet. The optional data collection tool is a fillable Microsoft Word document embedded within the Excel-based NMAT; it provides prompting questions, following the layout of the indicator tables, to help assess the maturity levels of the seven indicators and provide results in a way that is easy to review and enter into the Excel tool. The summary worksheet contains auto-populated results from the assessment and blank fields for assessors to identify planned actions for each indicator based on the lowest unmet maturity levels. Assessors are instructed to select the party responsible for leading on priority actions and set a deadline for their implementation. The summary worksheet was created so that it could be shared as a separate report (printed or electronic) in user-friendly format to participants.

The tool is flexible and can be used to conduct either an external or self assessment. Self assessments are wholly conducted by members of the NITAG and its Secretariat; members from the country’s MoH may also be involved. External evaluations are conducted by outside assessors in close collaboration with the NITAG Chair and Secretariat. Outside assessors often include people experienced with the NMAT (e.g., NITAG stakeholders from neighboring countries, regional representatives of global health partners, or assessors trained to implement the tool). Outside assessors can also include experts within the country who are not involved in NITAG operations. There are benefits for each option and the decision whether to conduct an external or self assessment depends on such things as availability of resources, convenience in scheduling, interest of NITAG members to become familiar with the tool, and if the type of assessment will impact its use for advocacy purposes.

Once the assessment team is established, key NITAG-related documents should be collected and reviewed (e.g., legal or administrative documents establishing the NITAG, Standard Operating Procedures, Terms of Reference, Conflict of Interest policy, meeting minutes, etc.). While not required, in-depth individual interviews of key stakeholders (e.g. select NITAG members, professional association, scientific societies) can help assessors gain an understanding of different perspectives and identify topics for further discussion; the optional data collection tool can assist with this. The last phase of conducting an assessment using the tool is a group session where at a minimum all NITAG members and the Secretariat meet to complete the tool; if an external assessment is being conducted, this session would be led by the external review group. During this session, the group should come to consensus on whether and to what extent the NITAG meets criteria for the different indicators assessed by the tool, with findings from the desk review and interviews (if conducted) contributing to the discussion.

The tool and a tutorial guide to using the NMAT are posted to the GNN NITAG Resource Center site, where it is available for download. [15].

The key strength of the NMAT is that it delivers on the objective of evaluating NITAGs and providing clear recommendations for their strengthening. A major additional strength of the tool is the iterative, collaborative approach used to develop it. At each stage of the tool’s development, perspectives from immunization policymakers operating in low, middle, and high-income settings in a variety of global regions were considered and incorporated. Another of the NMAT’s strengths is its flexibility to be implemented through either self or external evaluation. There are advantages of each option. If sufficient resources are available, outside evaluators can lend objectivity, transparency and ensure balanced discussions. Potential benefits of self-assessment include that it can be conducted with limited resources, can be done at a pace convenient to the NITAG (e.g., in one session or through several weeks), and allows the Secretariat and NITAG to become familiar with the tool and maturity levels.

There were limitations to the process of developing the tool, and some challenges remain in using the tool. We sought to make the tool relevant to a variety of country contexts; however, fundamental differences among countries that do not actually reflect NITAG maturity could make rating according to standardized criteria challenging. For example, indicator 6, which assesses a NITAG’s integration into the policymaking process, will look very different in a centralized government than a federated state government structure. Additionally, there are ways NITAGs can be successful that are not captured in the tool; for example, certain NITAGs may perform auxiliary functions such as advising governments on national immunization laws or off-label vaccine use. While these functions certainly add value, they are not universal enough to include in the tool and therefore do not factor into a NITAG’s maturity. The tool is currently available only in English, French, Spanish and Portuguese. As the need arises, further translations will be considered.

Past assessments have identified that the likelihood of conducting a successful assessment is impacted by NITAG members’ availability and willingness to participate, as well as by support from the country (e.g., MoH or national program) to conduct the assessment.[17,18] During the preparatory phase, it is important to clarify the objectives of the assessment to participants and make clear that it is not an audit. NITAG-related documentation must be accessible and available during the desk review phase. If stakeholder interviews are conducted, evaluators should be able to adapt their interview style (e.g., face to face or virtual) and type (e.g., use of open-ended questions may be preferable for heads of professional societies). For the group session, it is helpful to send the tool before meeting with participants, set rules for visiting participants, and focus the discussion on unclear aspects of whether the NITAG meets certain criteria.

Early usage of the NMAT has suggested that the tool can be used to provide insights and strategic direction for not only individual countries, but also regionally. During the first half of 2023, all of the WHO’s Eastern Mediterranean Region countries conducted NMAT self-assessments [19]. The results were used by individual NITAGs but also to generate an aggregate description of NITAG capabilities and to identify cross-cutting regional needs. A summary of these needs can be helpful to regional and global partners interested in strengthening NITAGs.

In developing the tool, the WG confronted the challenges of formulating a tool that was clear and specific, while not overly prescriptive. We also observed this challenge during the piloting of the tool. It was necessary to communicate to evaluation teams that reaching the leading edge level is not necessary or even desired for every NITAG. Rather, assessors should adapt the assessment findings to fit their country context. Also, some areas that the NMAT identifies for possible improvement will not be entirely under control of the NITAG to address (e.g., resources, integration into policy process).