Re-emergence of human cases of avian H5N1 in 2003 spurred global efforts to assist countries to develop or increase their capacity to detect and contain novel influenza viruses. In September 2004, the United States Centers for Disease Control and Prevention (CDC) announced a program that would assist countries to develop their surveillance capacity. CDC entered into bilateral cooperative agreements with nine countries that identified cases of H5N1 or were close to such a country, adding an additional three countries in 2005. The purpose of the agreements was to improve surveillance, epidemiology, and laboratory infrastructure related to influenza detection. In addition to funding, the cooperative agreements provided countries with training and technical assistance to address important gaps in infrastructure and surveillance. Countries were eligible to enter into this CDC bilateral agreement only if they had an established national influenza center (NIC) with terms of reference agreeing to share relevant data and samples with the WHO Global Influenza Surveillance and Response System (GISRS).

When H5N1 spread to a peak of 115 human cases in nine countries in 20061, the US government released the National Strategy for Pandemic Influenza (national strategy)2, identifying three pillars of activity and investment: 1) preparedness and communication, 2) surveillance and detection, and 3) response and containment. In accordance with the national strategy, CDC expanded its bilateral program for pandemic preparedness and response to include 29 additional countries globally. Specifically, the bilateral agreements provided CDC funding and technical support in developing infrastructure and capacity relevant to laboratory diagnostics, epidemiology, and surveillance; rapid response teams; national preparedness plans; communication strategies for healthcare workers; and related activities. Although expanding the WHO GISRS and assisting countries to meet the terms for designation as a NIC remained a priority, countries were no longer required to have a NIC. Countries without a NIC used the funds awarded to work toward fulfilling requirements for NIC designation.3 In addition to supporting individual countries, CDC established cooperative agreements with the WHO and all six of its regional offices to support improvements in surveillance and pandemic preparedness. CDC project officers were assigned to each recipient country to guide and support activities. In 2006, CDC began developing a tool to assess and document countries' capabilities in preparing for and responding to pandemic influenza. The objectives were to measure changes in each country's capacity over time and to assess how the US government and other global partners' financial and technical assistance were improving preparedness.4 At the time, few tools existed that could measure progress in building capacity for pandemic preparedness. The WHO checklist, which recorded the absence or presence of essential minimum elements of preparedness in a national plan to determine completeness of pandemic plans5, was frequently used. Rather than document planning elements such as the WHO checklist, CDC recognized the need for a tool that could assess discrete progress over time, with an increase in score linked to tangible evidence of improved systems. The CDC-produced assessment tool is the National Inventory of Core Capabilities for Pandemic Influenza Preparedness and Response (national inventory tool). Tool design took two years, and the design methodology is published concurrently with this study.6 The national inventory tool was designed to be administered periodically to gauge a country's progress toward having capacities important for dealing with an influenza pandemic. This article looks at scores of 36 countries that conducted the assessment twice: once to establish a baseline in 2008 and again approximately 2 years later to measure improvements.

In 2008, CDC used the national inventory tool7 to document the status of 40 partner countries' preparedness for an influenza pandemic. CDC instructed each country in use of the tool, including recommendations on participants in the assessment and the value of including all sectors involved in pandemic preparedness and response within the country. CDC provided the assessment tool in advance to ensure that individuals and teams with the requisite content expertise or knowledge of the status of preparedness would be included in implementing the assessment. CDC worked with the host country to plan and conduct facilitated assessments using the national inventory tool. Two people from CDC participated in each assessment in 2008; one served as co-facilitator with the host country's representative, and the other documented the discussion and results. In some cases, additional CDC staff from the host country participated in discussions.

To ensure consistency in data collection across sites, CDC relied on three facilitators for most 2008 assessments. All CDC staff participating in assessments received training on use of the tool. In each country, facilitators guided participants through the tool, focusing on one indicator at a time and discussing the question that each indicator was meant to address.6 If there were difficulties in understanding the indicator, the facilitators used measurement notes included in the tool for clarification. Measurement notes provide definitions for terms in the indicator and examples of documentation and references. Assessment participants discussed each of 50 indicators in 12 capabilities. Facilitators asked a series of questions to reach score consensus and recorded a score of 0, 1, 2, or 3 for each indicator, documenting the discussions with notes to show how the final score was reached. The scores correspond to specific milestones that must be reached on the path to improved preparedness and response capability. A score of 0 denotes little or no demonstration of the function represented by the indicator, and a score of 3 denotes advanced accomplishment relevant to the indicator. Upon completion, the facilitator shared a draft report of scores and comments with the country's participants for validation. In 2010, 36 countries completed a second assessment; the average number of days between assessments was 694, with a range of 601 to 820 days. In 2010, usually one CDC staff member participated in facilitation and data collection along with a host national co-facilitator and relevant in-country staff. Many in-country assessors were the same in 2008 and 2010. The 2008 assessments established baseline scores at the indicator and capability levels and an estimate of each country's overall level of preparedness. Of the 36 countries repeating the assessment in 2010, 7 are low income, 28 are mid-income, and 1 is high income according to the World Bank designations.8

Each country's 2010 score was compared with its 2008 score. We recorded an increase in score from 2008 to 2010 as positive movement, a decrease as negative movement, and the same score in both years as no change. In addition to comparing scores using a simple percentage analysis, we made a statistical comparison at the capability and indicator levels. We used the Wilcoxon's signed-rank test to determine statistically significant changes in scores from 2008 to 2010. We performed the test on scores for each country and in aggregate by capability and indicator. A P value of <0·05 indicated statistical significance. To rank the relative performance of each capability or indicator, we first compared the P values. The smaller the P value, the higher the rank (1 being the highest rank). As so many P values were <0·0001, we compared the statistics from the Wilcoxon's signed-rank test: The higher the statistic obtained, the higher the rank order. We used SAS 9·1·3 for all analyses.9 Finally, we looked for any patterns of positive or negative movement by specific indicator, country, and income status.

On the basis of a nonparametric signed-rank test, when all 36 countries are considered in aggregate, all 12 capabilities showed statistically significant improvement from 2008 to 2010 (P value<0·0001 for 11 of 12 capabilities) (Table 1). Epidemiologic capacity had a P value of 0·0098. Table 2 shows the capabilities and indicators with a short description of the content of the indicator. For the 36 countries that participated in assessments in 2008 and 2010, Table 2 compares the aggregate average scores for each year and indicator. The number of countries with positive, no change, or negative movement in score is shown. A nonparametric signed-rank test of indicators showed statistically significant improvements for 47 of 50 indicators from 2008 to 2010, with P values <0·0001 for 28 of the 47 indicators. The three indicators lacking statistical significance were number of practicing epidemiologists in the population (4B), quality of practicing epidemiologists (4C), and availability of infection control materials (11C). As 11 capabilities and so many indicators had a P value of <0·0001, a statistical method to examine (or order) the indicators was applied to determine the relative performance of each capability or indicator. The rank order of the capabilities (Table 1) and indicators (Table 2) is shown in the last column. From 2008 to 2010, indicator scores showed positive movement 53·97% of the time, showed no change 38·70% of the time, and negative movement 7·33% of the time. Thirty-three of the 36 countries showed a decrease in scores for one or more indicators, with three countries (8%) reporting a decrease for more than 10 indicators. Only three countries recorded either positive movement or no change for all indicators.

In addition to analyzing the data in aggregate, we performed a signed-rank test on scores for individual countries. Of 36 countries that participated in both assessments, 30 made statistically significant progress, with a P value of 0·001 in 14 of 30 countries.

For data analyzed in aggregate, the overall trend was an increase in scores. Analysis of data at the country level for trends in specific indicators showed seven indicators with decreased scoring for five or more countries from 2008 to 2010 (Table 2, Movement). Conversely, five or more countries showed large increases in score (from 0 to 3) for 9 indicators from 2008 to 2010 (Table 3). Only one country reported a large decrease (from 3 to 0) for one indicator relevant to communications. Lastly, at 0·75 points, the increase in overall average score from 2008 to 2010 for 7 low-income countries (1·08 to 1·83) and 28 mid-income countries (1·30 to 2·05) was the same although mid-income countries started with a higher average baseline.8

The number of raw scores possible for 2008 and 2010 was 3600. We have a total of 3588 scores, with 12 scores missing (8 for 2008 and 4 for 2010). Two scores for indicator 12B1 and six for the indicator 8D are missing for 2008. For 2010, one for 8D, two for 1D1, and one for 9A are missing.

The 2009 A (H1N1) pdm pandemic accelerated the development of some core capabilities. Although all capabilities showed statistically significant progress from 2008 to 2010, the capabilities with the greatest increase deserve further examination. Table 1 shows that the largest increases were, in order, in capabilities 9 (resources for containment), 3 (communications), 6 (routine influenza surveillance), and 10 (community-based interventions). It is important to look at the indicators relevant to these capabilities. Using the calculated statistic for P values <0·0001, we ranked indicators from most significant to least significant performance (Table 2). Many indicators with the greatest increase from 2008 to 2010 represent activities or functions that lend themselves to rapid improvements with concerted effort by the country or with donated goods from the international community. For example, as needs became evident during the pandemic, countries quickly addressed issues related to voluntary isolation and quarantine (10C), exercising or practicing for containment (9C), using data to inform decisions for pandemic preparedness (2D), and awareness of the need to report A (H1N1) pdm (7A). Other indicators with the greatest increases in scores represent activities or functions that improved rapidly because the country developed capacity or put infrastructure in place to prepare for and respond prior to the A (H1N1) pandemic. Those indicators are 12C: implementation of clinical guidelines; 3C: testing of formal and informal channels of communication; 3B: development and testing of communication materials; 6A: sentinel sites collecting virologic and epidemiologic data; and 6C: timeliness of reporting influenza surveillance data.

The 10 indicators that improved most from 2008 to 2010 (Table 2) proved their value under pandemic conditions. For example, surveillance systems and laboratory methods established to handle an H5N1 outbreak allowed countries to respond to the 2009 A (H1N1) pdm pandemic effectively. Likewise, because countries developed teams and practiced response to H5N1 outbreaks, they were able to respond quickly to the A (H1N1) pdm outbreak. Similarly, because countries had set up communications systems in preparation for H5N1, they were able to produce timely reports on the status of the outbreak. Only three indicators did not show statistical significance from 2008 to 2010: 4B and 4C (epidemiologic capacity) and 11C (infection control). The epidemiologic indicators relate to the presence and competency of field or practicing epidemiologists within a country. In general, the scores for these indicators remained the same from 2008 to 2010, highlighting how slowly factors related to human resources improve. Little or no change in the indicator of infection control shows that infection control materials at hospitals did not improve from 2008 to 2010.

Improvements for several of the indicators with the least progress from 2008 to 2010 (Table 2) would have required substantial investment of resources and time (e.g., availability of practicing public health epidemiologists). Two of the indicators were related to improving facilities (i.e., increasing bio-safety levels for public health laboratories and creating climate-controlled storage for materials). Improving these indicators requires considerable time and dedicated resources that may be cost-prohibitive for some low-resource countries. Collaboration between human and agricultural health authorities showed only modest improvements in scores from 2008 to 2010. Similarly, the use of standard case definitions for surveillance did not improve substantially, largely because most countries scored a 2 or 3 in 2008, leaving little room for improvement. That some indicators showed little progress from 2008 to 2010 was not unexpected given the long time needed to develop human resources or to improve buildings or facilities. Lack of improvement in human resources highlights the need to establish or sustain activities such as field epidemiology training programs to ensure that countries have appropriate human resources to handle disease outbreaks.

We examined changes in scores from 2008 to 2010 for each country, looking for trends in specific indicators. The movement columns of Table 2 show that scores for seven indicators decreased for five or more countries from 2008 to 2010. Three of these (4B, 4C, and 11C) are indicators that did not have statistically significant changes from 2008 to 2010. Two of the seven (2A: collaboration between human and animal health authorities; and 7B: monitoring rumors) are among the 10 indicators with the lowest improvement in score. The other two indicators with a negative trend in five or more countries were 1D1: mechanisms for sustainability of financial resources for pandemic planning; and 7A: awareness of the need to report by mounting public awareness campaigns. Although dedicated resources for preparedness and response activities improved greatly in some countries, in other countries the lack of resources for these activities persisted from 2008 to 2010. The decline in scores relevant to public awareness of the need to report pandemic H1N1 was likely related to the countries' health authorities placing less emphasis on the urgent need to report H1N1 than on the need to report H5N1.

Five or more countries reported improvements for 9 indicators. Of those 9 indicators, 7 are ranked among the top 10 in Table 2. Most of these indicators represent activities or functions for which substantial improvements can be made quickly, if the proper foundation is already in place. Additionally, the H1N1 pandemic forced many countries with little or no activity in some areas to rapid capacity advancement. For example, they were forced to set up systems to disseminate plans, use research findings to develop recommendations, create communication materials, train spokespersons, improve the timeliness of surveillance activities, and activate teams for outbreak investigations. Many countries' efforts to control a pandemic, in combination with systems established through preparedness efforts, resulted in large positive changes in indicator scores.

In addition to analyzing scores, we also analyzed the notes taken during the 2008 and 2010 assessments. We learned through this analysis that although channels of communication were established and functioning in many countries, the actual messages pertaining to A (H1N1) pdm were sometimes slow to be developed. Participants highlighted the importance of developing draft materials in advance of a pandemic and the value of other countries or partner organizations sharing materials.

Although the 2009 pandemic resulted in improvements in certain capabilities, some scores seem to have decreased because of the pandemic. Many countries with scores lower in 2010 than in 2008 said that before the pandemic they were confident of their capabilities because of their planning and preparation. However, their perceptions about their level of capability changed when faced with the realities of a pandemic. Therefore, some countries reported a lower score on an indicator after the opportunity to test their function in pandemic conditions. In particular, their efforts to put into effect community-based interventions to prevent the spread of influenza (capability 10) gave them an opportunity to test the effectiveness of their plans.

The six countries that did not have statistically significant improvements from 2008 to 2010 were distributed globally, with no more than two countries in any WHO region. Of the six countries, two had average overall scores higher than 2 (2·01 and 2·13) in 2008 and two had scores in the upper range of overall average scores (1·81 and 1·85). So, although their improvements were not statistically significant, four of these six countries had relatively high average baseline scores.

For the 12 missing responses for 2008 and 2010 combined, 8D (activation and deployment of trained, equipped teams in response to a public health emergency of international concern[PHEIC]) accounted for 58% of missing scores: six countries had no score in 2008 and one in 2010. This finding could relate to the newness of the International Health Regulations (IHR) language and the lack of a need to respond to a PHEIC before the 2008 assessment. Of the other missing scores, three involved absolute counts of beds or antivirals. We found when an indicator was assessed through absolute numbers, it was harder for countries to determine status. Two of the non-scored indicators in 2010 dealt with whether a country had a clearly defined decision-making structure in place, something identified by at least two of the countries as an issue during the 2009 pandemic.

The assessment tool has some limitations. CDC facilitators made every effort to standardize data collection in 2008 and 2010 and maintain consistency in conducting assessments between countries and over time. However, the final score was always determined by the country. Scores for some indicators may be inflated. However, according to CDC facilitators working with countries on these assessments, this did not seem to be the case.

CDC's national inventory tool shows that rigorous assessment of progress toward dealing with complex public health issues such as pandemic preparedness is possible. CDC created and used this tool for two main purposes. The first was to measure the effect on pandemic preparedness of providing technical assistance and support to partner countries. Figure 1 shows the aggregate capacity scores for 36 countries through the CDC assessment tool in both 2008 and 2010 where red represents an average score of 0–0·99, yellow 1–<2, light green 2–<3, and dark green an average of 3. This figure indicates that partner countries are making progress and that CDC and other international agencies providing support are having a positive effect on preparedness and response to pandemics. The second purpose was to learn how CDC could assist countries to fill gaps in their preparedness for pandemics by offering technical expertise in laboratory, epidemiology, and outbreak response. Countries have shown that they can use the results of the assessments to spur improvement in areas of need.10 Moreover, the capacity for pandemic response built through technical assistance from CDC, other agencies, and the countries' own work and that documented through these assessments contributes to increased worldwide capacity to follow IHR and serves countries in their work beyond influenza preparedness and response.