Many statistical methods can be used to analyze ITS designs (e.g., ARIMA modeling or time series regression). However, the design is often analyzed inappropriately, which makes interpretation of individual studies difficult (12). Methods of analyzing ITS data were examined critically (12). The preferred method for short time series is segmented time series regression analysis, which is a statistical comparison of time trends before and after the intervention to identify either an immediate change in the level of the regression line or a sustained change in the slope of the line (12,13). In this report, we have distinguished 2 intervention effects: immediate (a sudden change in the level of the regression line at the point of intervention) and sustained (a sustained change in the slope of the regression line from the start of the intervention phase). If the original report did not include an appropriate analysis, data were reanalyzed by using segmented time series regression.

The following model was specified: Y_t = B₀ + B₁ × preslope + B₂ × postslope + B₃ × intervention + e_t, where Y_t is the outcome (e.g., CDAD incidence) in month t, preslope is a continuous variable indicating time from the start of the study up to the last point in the preintervention phase and coded constant thereafter, postslope is coded 0 to and including the first point postintervention and coded sequentially from 1 thereafter, and intervention is coded 0 for preintervention time points and 1 for postintervention time points. In this model, B₁ estimates the slope of the preintervention data, B₂ estimates the slope of the postintervention data, and B₃ estimates the change in level of outcome as the difference between the estimated first point postintervention and the extrapolated first point postintervention if the preintervention line was continued into the postintervention phase. The difference in slope is calculated by B₂ – B₁. The error term e_t was assumed to be first-order autoregressive. Confidence intervals (95%) were calculated for all effect measures.

Formal metaanalysis of results was not attempted given the differences in context, setting, and type of outcomes. However, to gain an overall summary picture of the heterogeneity of effect sizes we standardized all measures so that they were all on the same scale. To do this, we divided the change in level and the change in slope by the preintervention standard deviation (SD) in each study. The resulting metric has no unit, it is known in standard metaanalysis as the standardized mean difference. Standardized effect sizes of 2 to 3 SD were considered large, whereas an effect size <0.5 SD was considered of questionable clinical significance even if statistically significant (14). To visually display the heterogeneity of the standardized effect sizes, graphic plots of level effects versus slope effects for each study (with associated 95% confidence intervals) were generated.

The statistical analysis assessed how likely it was that study results could simply have happened by chance, and the Cochrane quality criteria assessed common threats to the validity of interventions to change practice or organization of care. To assess other threats to the validity of infection control interventions, we used the format for reporting the results of included studies recommended by guidelines derived from a recent systematic review of isolation measures to control methicillin-resistant Staphylococcus aureus (MRSA) (15). We required studies to provide reliable data about the effect of interventions on both microbial and drug outcomes with clear case definition, description of infection control measures, and other variables such as bed occupancy or staffing levels that could provide plausible alternative explanations for changes in microbial outcomes. We have provided a summary of detailed information from the included studies (Table A1). Additional information is available from the British Society for Antimicrobial Chemotherapy (http://www.bsac.org.uk). We classified case definitions into colonization, infection or clinical isolates, or a combination of >2 with the following definitions.

Colonization was defined as a microorganism, usually detected by screening, at a host site (normally nonsterile, although the urine of a catheterized patient may be an exception) without causing systemic signs of infection or a specific immune response. Colonization by case note review was established by excluding infection diagnosed according to criteria adopted by the authors or defined by appropriate bodies, e.g., the Centers for Disease Control and Prevention criteria for diagnosing nosocomial infections. Infection was established by case note review according to criteria adopted by the authors or defined by appropriate bodies or by recording specific symptoms and/or signs, such as diarrhea in patients with CDAD. Clinical isolates were defined as the recovery of a microorganism after culture of a clinical (not screening) specimen without specifying whether it represents colonization or infection.

All 3 clinical trials reported appropriate statistical analysis (30–32), whereas only 2 of the 13 ITS studies reported appropriate statistical analysis (17,27). Of the remaining 11 ITS studies, 5 did not report statistical analysis; 6 reported inappropriate statistical analysis by using tests such as χ² or t tests that assume independence between observations and do not account for time trends. Data from these 11 studies were reanalyzed.

Overall, 4 studies provided strong evidence of control of the microbial outcome by change in prescribing (17,27,30,31). All of these studies provided reliable data about antimicrobial drug prescribing, with significant changes in both microbial and drug outcomes after planned interventions. In addition, 2 studies provided further protection against regression to the mean by using a crossover design (27,30). Three of these studies have rigorous case definitions based on prospective screening cultures plus full description of infection control measures.

Eight studies provided less convincing evidence. Two studies showed significant changes in prescribing that were associated with nonsignficant changes in CDAD (20,26). An additional 6 studies reported statistically significant improvement in microbial outcome but without reliable data about the effect of the intervention on prescribing (18,19,23,24,28,29). The importance of this omission is confirmed by the 6 studies that included reliable data about prescribing because all showed that there was some prescription of restricted drugs during the intervention phase (17,20,26,27,30,31).

Four studies had negative results (21,22,25,32). One study provided strong evidence of failure to control microbial outcomes despite a successful change in prescribing (32). One study reported an intervention that failed to change use of vancomycin (22). The remaining 2 studies showed no change in microbial outcome but did not provide reliable data about the effect of the intervention on prescribing (21,25).

The most consistent evidence was for the 5 interventions designed to reduce the incidence of CDAD. Four were implemented throughout the hospital (17,18,20,24) and 1 was implemented in the elderly care service (26); all 5 targeted prescribing of cephalosporin or clindamycin. All of the interventions were associated with a change in the expected direction (Figure part A), which was a change in the incidence of CDAD in the same direction to a change in use of cephalosporin or clindamycin. For 1 intervention, the expected direction was an increase in CDAD incidence after the introduction of ceftriaxone (20); for all other interventions a decrease in CDAD incidence was expected to accompany a decrease in use of cephalosporin or clindamycin. These 5 studies reported 7 interventions. The immediate effect after 6 of the 7 interventions was at least 0.5 SDs; 5 of these 7 immediate effects were statistically significant (Figure, part A). Sustained changes after the intervention were more modest, but all were in the expected directions and 4 of 7 were statistically significant (Figure, part A). The 5 CDAD studies had results expressed in different units: cases per month (24,26); cases per quarter (18,20); or cases per 1,000 admissions per year (17). Consequently, we were only able to compare effect sizes in numbers of CDAD cases per quarter by recalculating results from 2 studies (24,26). The antimicrobial drug intervention was associated with a mean immediate reduction of 15.0 CDAD cases per quarter (range 6–26) and a median sustained reduction of 3.2 CDAD cases per quarter (range 1–6).

The results of the 10 interventions designed to reduce the incidences of GNRB were less consistent. Three were implemented throughout the hospital (19,21,23), 1 was implemented in the neurology and neurosurgery service (25), and 5 were implemented in a single intensive care unit (ICU), which included 4 with pediatric patients (28–30,32) and 1 with adult patients (31). One intervention was designed to reduce the duration of treatment with any antimicrobial drug for ICU patients at low risk for pneumonia; this was associated with a significant reduction in the incidence of colonization by any GNRB and exposure to antimicrobial drugs (31). The remaining 9 interventions involved changes in antimicrobial drug treatment, mainly aminoglycosides or cephalosporins. One RCT provided no evidence that antimicrobial drug cycling reduced the incidence of GNRB in a neonatal ICU (32). The 8 ITS studies reported 9 outcomes (Figure, part B). The expected direction of effect from a change in aminoglycoside or cephalosporin prescribing was usually a reduction in GNRB. For 1 intervention, the expected direction of effect was an increase in the incidence of GNRB after gentamicin was reintroduced (19). The expected direction for all 9 outcomes changed, but the effect size was <0.5 SD in 2 studies and not statistically significant in 5 studies (Figure, part B). In 3 studies the changes in slope were in the expected direction and in 1 the changes were both statistically significant and >0.5 SD, which is likely clinically important. Unlike with CDAD data, effects cannot be expressed in a common unit. Some studies measured colonization and others examined infection. Units of measurement were also variable (e.g., number of isolates, percentage of isolates, number of cases, and number of cases per time period).

Data for gram-positive bacteria were very limited. One study provided strong evidence that restricting ceftazidime in a hematology unit was associated with significant reduction in risk for colonization by VRE (27). However, reduction of cephalosporin use in a hospital was not associated with any change in the prevalence of VRE isolates (17). A third study targeted at VRE showed that implementation of a vancomycin order form had no significant impact on vancomycin prescribing, with a trend in the unintended direction (22). Two studies report effects on MRSA prevalence (17,21). Our segmented regression analysis showed no significant change in response to a reduction in use of third-generation cephalosporins (Table A1), although 1 of the reports claimed that a change did occur (21).

The evidence supports the theory that limiting the use of specific antimicrobial drugs will reduce the prevalences of resistant gram-negative bacteria and CDAD. For gram-positive bacteria, there is a lack of evidence rather than evidence of no effect. Hospitals would like to know how much they should limit their antimicrobial drug prescriptions and what is the minimum that will show a real effect. Unfortunately, the available evidence is too limited to provide definitive answers to these issues. Thus, hospitals must estimate the effect of their own interventions. The good news is that the data required for ITS analysis of the incidences of drug-resistant bacteria or CDAD should be readily available in most hospitals. Healthcare providers need to invest in data analysis so that evaluation of antimicrobial drug control in hospitals becomes a routine measure of the quality of care rather than a research project.

Standardized reporting of outbreaks and interventions to control the incidence of antimicrobial drug resistance or hospital-acquired infection would greatly enhance the ability to combine results from hospitals in metaanalyses. Key issues include full description of other infection control measures, consistent and reproducible case definitions, the length of preintervention and postintervention phases, and the intervals between data points (15). Ideally, data should be made available in a way that allows reanalysis and, where appropriate, metaanalysis. Metaanalysis of single hospital studies is no substitute for good multicenter studies, but it could be used to provide some evidence of reproducibility and thus to prioritize targets for definitive trials.

The research agenda needs to move to multicenter studies with randomized allocation to interventions. This will provide better evidence of external validity as well as the power to measure cost-effectiveness and exclude important unintended adverse clinical outcomes. Development and pilot testing of the effectiveness of clinical decisions for reducing unnecessary exposure to antimicrobial drugs should be a priority for research in hospitals.