The data improvement team (DIT) strategy was developed and managed by a national DIT strategy management group, which included UNEPI, the Resource Center (the responsible entity for managing health information) of the Uganda Ministry of Health, World Health Organization (WHO) Uganda, the US Centers for Disease Control and Prevention (CDC), the African Field Epidemiology Network (AFENET), UNICEF, and Gavi. Implementation was funded jointly by Gavi Health Systems Strengthening Grant 1, WHO, UNICEF, and CDC and led by a national coordinator from AFENET, with technical assistance from CDC.

The strategy aimed to strengthen the immunization information system and quality of the resultant data at the district and health facility levels through practical classroom training, deployments involving rapid data quality and organizational assessments, and on-the-job training (Figure 1). The number of DIT members required for each district was determined on the basis of ability to reach all health facilities that provided immunization services in that district (range 6–117) within 5 to 6 working days, spending 2 to 3 hours at each. A district-level DIT included an average of 4 district staff members (with additional members in high-population areas) and 1 Makerere University School of Public Health (MakSPH) student. Districts were asked to identify staff to form a DIT, which included the district biostatistician, district Expanded Programme on Immunization (EPI) and surveillance focal persons, and a health records assistant. MakSPH staff and the national DIT coordinator led recruitment of students.

The DIT strategy was designed to be implemented in a phased approach by region (Figure 2); several district-level DITs were trained together, then deployed in their respective districts. All official government districts in Uganda as of November 2014 were divided into 17 DIT operational regions to ensure that the number of attendees at regional training was logistically manageable and there was close geographic proximity between districts in each region.

Before implementation, a 5-day orientation to the strategy and Uganda’s immunization information systems was provided to national staff, who self-selected to support delivery of the regional-level training and to conduct supportive supervision of DIT activities. The 3-day regional training aimed to build selected DIT members’ knowledge and skills in data management and quality, which were applicable during the DIT deployment and their regular duties thereafter (Figure 1).

In the week after each regional training, DIT members were deployed to their home districts for 5 to 6 days to work at the district office and visit health facilities (Figure 1). Working in pairs, DIT members identified problems, proposed solutions, developed recommendations, and enhanced staff capacity through on-the-job training on locally identified problems (e.g., how to create an immunization monitoring chart) (Figure 2). DITs initially prioritized health facilities with outlying (high or low) coverage for the third dose of the diphtheria/tetanus/pertussis/Haemophilus influenzae type B/hepatitis B vaccine (Penta3), negative dropout rates, or inadequate completeness or timeliness of Health Management Information System (HMIS) monthly reports (18). Staff from the Ministry of Health and national EPI program partner organizations provided supportive supervision to DIT activities in some districts, assisting coordination and implementation of activities, conveying national-level support for the DIT strategy to district leadership, and enhancing their own awareness of ground-level operations.

A participatory and utilization-focused (19,20) approach was taken to routine monitoring and evaluation of processes, outputs, and short-term outcomes. Training was evaluated through a self-administered survey focused on quality of the training experience; a pretest and posttest measured participants’ acquisition of knowledge and level of preparedness to implement DIT activities. DITs conducted an organizational assessment at the district and health facility levels to inform their work and to gather baseline information on key indicators (21). Organizational assessments contained a mix of closed and open questions covering dimensions of, and factors affecting, vaccination data quality. Results of organizational assessments were reported to the DIT national coordinator either through a reporting template in Excel (Microsoft, Redmond, WA, USA) (106 districts, 1 Kampala division) or by using an open data kit–based mobile application linked to a cloud-based database (5 districts, 4 Kampala divisions).

At the health facility, DITs also used a data quality improvement (DQI) questionnaire to review practices for data management, collection, accuracy, analysis, and use (Table 1). The primary purpose of this questionnaire was to identify gaps that would inform recommendations and on-the-job training. For purposes of analysis for monitoring and evaluation, DQI questionnaires from health facilities were sampled from 107 of the 116 districts (92%) for which these data were not reported through the mobile application. The sample included all hospitals and every second health facility selected from an alphabetized list, until the sample size reached 50% of all health facilities in the district. In an additional 7 districts, DQI reports from all visited health facilities were entered in the mobile application. Descriptions of the DIT activities, outputs, and recommendations were presented in a written report for each district health management team. Line-listed results from organizational assessments and DQI questionnaires were aggregated nationally and quantitative data were descriptively analyzed in SAS version 9.3 (22) and Tableau version 9.3.1 (23). The sign test was used to assess the statistical significance of the direction of difference between sources of vaccine dose administration data and was performed in R version 1.5.1 (24).

After DITs had been deployed to all districts, a review of DIT implementation was undertaken to gather feedback about the approach and understand extent of action on recommendations through a rapid organizational-level survey in a sample of districts and health facilities. Four regions were selected from the 17 DIT operational regions; 2 or 3 districts were selected from each region, and within each of these, 4 health facilities were selected, totaling 11 districts and 44 health facilities. If a selected site could not be visited, it was replaced with the next one of the same type on an alphabetized list of health facilities in the district. Selection was purposeful to gain insights across a range of characteristics, including geographic location, implementation of national supervision, Reaching Every District categories (25), and level (type) of health facility (26). Eight data collectors (4 AFENET/CDC staff and 4 MakSPH students) completed a 1-day training, then worked in pairs to visit the selected sites to conduct the survey through group interviews with district and health facility staff. Resultant data were descriptively analyzed in Epi Info software (27).

The proportion of health facilities in a district submitting monthly HMIS reports on time to the district (timeliness) and the proportion of expected reports received by the district (completeness) are routinely calculated in the national electronic HMIS (12,18). In districts for which these data were available for the 3 months and after the DIT visit (n = 104) and for the second 3-month period after the DIT visit (n = 95), timeliness and completeness, by month and district, were extracted from the electronic HMIS. Median timeliness and completeness were calculated per district across each 3-month period, then compared between periods to identify change.

The national DIT strategy management group held periodic meetings (Figure 2) to review results from monitoring and evaluation and the budget, as well as to solicit feedback from all stakeholders. These meetings, in conjunction with national priorities, informed any adjustment of DIT activities and implementation plan.

During November 2014–September 2016, all 112 districts and 5 divisions of Kampala (total 116 DIT operational districts) in Uganda sent staff to DIT regional training and deployed district-level DITs. Seventeen regional trainings, covering 2–14 districts per training, attended by 451 district and health subdistrict staff and 35 MakSPH students (some attended multiple trainings [range 1–9]). In response to participant and stakeholder feedback, the training format was altered to enhance the balance between the practical and didactic sessions (Figure 2). After training, 83% (355/429) of district staff demonstrated improved knowledge on posttest compared with pretest scores, and more participants felt “fully prepared” to conduct DIT activities (14% pretest, 82% posttest).

In total, 476 DIT members (including 35 MakSPH students) were deployed and reached 89% of health facilities that provided immunization services (Table 1). Health facilities not visited (n = 413) were predominantly health center IIs (HCIIs; n = 332, 80%), which offer a limited number of services, serve smaller catchment areas, and are often geographically remote. Initially, DITs reviewed paper copies of monthly HMIS reports from health facilities submitted to the district office and compared doses reported for all antigens with those recorded in the electronic HMIS for the 12 months before the DIT visit (Table 1). Time spent on this activity reduced the time available to reach all priority health facilities by an average of 8 hours per district. Because early results showed high congruence between these 2 data sources (Figure 3, panel D), this activity ceased after the midterm review meeting, enabling DITs additional time to conduct organizational and DQI assessments and develop recommendations for improvement (average 1.2 hours per health facility) and to implement on-the-job training (average 1.5 hours per health facility).

Through the organizational assessment, DQI questionnaire, and discussions with staff, DITs identified a combination of external factors, often specific to the site visited, that affected vaccination data collection, management, analysis, and use. Commonly identified challenges included poorly motivated, new, untrained, or absent staff; unavailability of materials for recording and reporting data; competing priorities on staff time due to integration of services; inadequate supportive supervision for data quality; limited transport, technological, and financial resources; variable understanding and commitment by political or organizational leaders; and competition with other public health initiatives for human, financial, and material resources.

In Uganda, doses of vaccines administered are recorded on 4 tools: tally sheet, child register, and monthly report (at health facility) and the electronic HMIS (entered at district level using data from health facilities’ monthly report). We found variable congruence between monthly totals of vaccine doses across these 4 sources for any given month (Figure 3). On average, the number of administered doses aggregated on the monthly report was higher than that recorded individually on the tally sheet (Figure 3, panel C), which was higher than that recorded on the child register (Figure 3, panel A). This finding suggests that vaccine administration is overreported by the health facility and that use of the child register is low compared with other sources of vaccine dose administration data (Figure 3, panels A, B). We found stronger agreement between the number of doses on the paper HMIS monthly report and those in the electronic HMIS (Figure 3, panel D), highlighting infrequent transcription error or loss of data from district to national level. There was individual variation in the discordance by health facility, with no clear pattern by district or health facility type. Similar patterns in data congruence were also seen for single-dose measles vaccine offered to older children (data not shown).

The postimplementation follow-up survey found that DITs had visited all sampled districts (n = 11) and 77% (34/44) of sampled health facilities. Recommendations provided by DITs addressed all dimensions of data quality; however, the extent of implementation varied (Table 2). Recommendations for each district most frequently related to improving systems for archiving, checking data on monthly HMIS reports, and charting coverage data. At the district level, recommendations relating to data management and collection were more fully implemented than those related to analysis and use (Table 2). Recommendations for health facilities most commonly focused on improving recording and reporting of data, analysis, and archiving. Recommendations related to management and collection were more completely implemented than those related to analysis. No health facility reported taking action on recommendations to improve data use (Table 2). Reasons for inaction across all recommendations included insufficient availability of required materials (standard data collection/reporting tools, archiving space); inadequate human resource capacity (new staff, untrained staff, low motivation); and a management structure that limited staff awareness of, and roles in, immunization data collection, management, analysis, and use.

During the follow-up survey, district staff frequently reported that participation in the DIT activities catalyzed improvements in existing, or development of new, systems and processes, such as supportive supervision about vaccination data quality. Health facility staff felt that the visit by the DITs was a catalyst for provision of updated recording and reporting tools and helped them develop systems to enhance completeness and accuracy of data reported on the monthly HMIS report (Table 3).

Timeliness and completeness of HMIS monthly reporting (from health facility to district) averages >90% nationally (28). This high performance limits the opportunity for and measurement of change; however, there was some improvement. Comparing 3 months before and after DIT implementation, 15% (15/104) of districts showed improvement in completeness, 6% (10/104) decreased completeness, and the remainder no change. From the initial 3 months to the second 3 months post-DIT implementation, completeness improved in 25% (24/95) of districts, decreased in 10% (9/95), and showed no change in the remainder. More districts showed improvement in timeliness of monthly HMIS reporting. Comparing 3 months before DIT implementation to 3 months after, 38% (40/104) improved, 20% (21/104) decreased, and the remainder showed no change in timeliness. From the first to second 3-month periods after implementation, 27% (26/95) of districts showed improvement, 50% (47/95) decreased, and the remainder showed no change.