88040994791Infect Control Hosp EpidemiolInfect Control Hosp EpidemiolInfection control and hospital epidemiology : the official journal of the Society of Hospital Epidemiologists of America0899-823X1559-683421515978367898010.1086/659760NIHMS461412ArticleDeriving Measures of Intensive Care Unit Antimicrobial Use from Computerized Pharmacy Data: Methods, Validation and Overcoming BarriersSchwartzDavid N.MD1EvansR. ScottMS, PhD23CaminsBernard C.MD, MSCR4KhanYosef M.MD, MPH5LloydJames F.BS2ShehabNadinePharmD, MPH6StevensonKurtMD, MPH5for the Centers for Disease Control and Prevention Epicenter ProgramJohn H. Stroger, Jr Hospital of Cook County and Rush Medical College, Chicago, ILLDS Hospital/Intermountain HealthcareUniversity of Utah School of Medicine, Salt Lake City, UTBarnes-Jewish Hospital and Washington University School of Medicine, St. Louis, MOOhio State University Medical Center and College of Medicine, The Ohio State University, Columbus, OHDivision of Healthcare Quality Promotion, Centers for Disease Control and Prevention, Atlanta, GACorresponding author and request for reprints: David N. Schwartz, MD, Division of Infectious Diseases, John H. Stroger, Jr. Hospital of Cook County, 1901 W. Harrison St., Chicago, IL 60612 david.schwartz@hektoen.org252013520111162013325472480Objective
To outline methods for deriving and validating intensive care unit (ICU) antimicrobial utilization (AU) measures from computerized data and to describe programming problems that emerged.
Design
Retrospective evaluation of computerized pharmacy and administrative data.
Setting
ICUs from four academic medical centers over 36 months.
Interventions
Investigators separately developed and validated programming code to report AU measures in selected ICUs. Antibacterial and antifungal drugs for systemic administration were categorized and expressed as antimicrobial days (each day that each antimicrobial drug was given to each patient) and patient-days on antimicrobials (each day that any antimicrobial drug was given to each patient). Monthly rates were compiled and analyzed centrally with ICU patient-days as the denominator. Results were validated against data collected from manual medical record review. Frequent discussion among investigators aided identification and correction of programming problems.
Results
AU data were successfully programmed though a reiterative process of computer code revision. After identifying and resolving major programming errors, comparison of computerized patient-level data with data collected by manual medical record review revealed discrepancies in antimicrobial days and patient-days on antimicrobials ranging from <1% to 17.7%. The hospital for which numerator data were derived from electronic medication administration records had the least discrepant results.
Conclusions
Computerized AU measures can be derived feasibly, but threats to validity must be sought and corrected. The magnitude of discrepancies between computerized AU data and a gold standard based on manual chart review varies, with electronic medication administration records providing maximal accuracy.
National Center for Advancing Translational Sciences : NCATSUL1 TR000448 || TRNational Center for Advancing Translational Sciences : NCATSKL2 TR000450 || TR
Antimicrobial resistance rendering previously treatable infections unresponsive to most drugs is a significant and growing public health concern.1,2 This threat has been recognized in the most recent national action plan for the prevention of healthcare-associated infections outlined by the Department of Health and Human Services,3 and calls for a coordinated national effort to monitor resistance and implement prevention and control efforts have been longstanding.1,4 The Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America recently published guidelines promoting the implementation of antimicrobial stewardship interventions in hospitals.5 An integral component of evaluating the impact of any of these strategies is the accurate and continuous measurement of antimicrobial utilization over time.
The increasingly computerized processes of healthcare delivery have made the automated acquisition of antimicrobial utilization data possible. Indeed, most modern hospitals have universal computerization of laboratory, pharmacy, admission-discharge-transfer (ADT), and patient demographic and financial data.
However, just as these data sources have been incorporated into fully functional electronic health records for only a small minority of hospitals,6 the derivation of reliable and accurate reports based on computerized hospital data has generally been difficult to achieve.7 Because these data are stored at each hospital or health system in disparate information systems, procedures required for their collection, extraction, cleaning, validation, and computation are often complex and error-prone.8 With regard to antimicrobial use measures, specifically, published data that have used electronic data sources to derive these estimates have almost exclusively relied on propriety measurement systems and the methods underlying the acquisition, validation, and consolidation of such data have not been well described.
We describe our efforts to demonstrate the feasibility and validity of obtaining uniform measures of antimicrobial utilization in selected intensive care units (ICUs) in four academic medical centers by accessing pharmacy and administrative data contained in computerized data warehouses. Despite considerable expertise and relevant experience by healthcare informatics specialists and investigators in accessing such data, we encountered a number of problems that were largely unforeseen and, therefore, may be informative to the development of a standardized approach to deriving antimicrobial measures from electronic data.
Methods
Four tertiary-care, academic medical centers were recruited from institutions participating in the current Prevention Epicenter Program,9 funded by the Centers for Disease Control and Prevention (CDC), to participate in this study. The Institutional Review Boards of each hospital approved the study protocols, and waived requirements for patient and physician consent.
Hospital Intensive Care Unit Types and Information Technology Resources
The characteristics of the participating facilities, respective data warehouses, and medication ordering, dispensing, and administration systems are outlined in Table 1. Descriptions of these data warehouses have been provided elsewhere.8,10–12 Antimicrobial use data for selected ICUs in each hospital over a 36-month time period (July 2004 – June 2007) were acquired, analyzed, and validated separately at each institution before being sent to one central Epicenter [Chicago Epicenter (DNS)] for collation and final analyses.
Antimicrobial use numerators (described below) were computed from varying electronic data sources (Table 1). For Hospital A, whose pharmacy information system vendor changed during the study period, the numerator data source changed from pharmacy dispensing to computerized physician order entry (CPOE) data. Hospitals B, C and D obtained numerator data from the electronic medication administration record (eMAR), from pharmacy dispensing data and from CPOE data, respectively. Although eMARs were used to document medication administration in three of the four hospitals, eMAR data were available in a format amenable to analysis only at hospital B, which was able to distinguish antimicrobial doses that were administered from those that were ordered but not administered for any reason (e.g., because of doses held or refused). Denominator data (ICU patient-days) were derived from the same data sources as those from which numerator data were computed in all hospitals, with the exception of Hospital A where ADT data sources had to be queried separately after the change in pharmacy information system vendors.
Antimicrobial Use Measures
The primary numerator measure for antimicrobial use was antimicrobial days, defined as calendar days on which patients received each antibacterial or antifungal agent given by intravenous (IV) or oral administration. For example, one patient given two drugs for 5 days accrued 10 antimicrobial days. Drugs available in injectable, oral, or other systemically-administered form were counted just once per day irrespective of route of administration. Secondary numerator measures of antimicrobial use included patient-days on antimicrobials, antimicrobial starts, antimicrobial courses, and defined daily doses (Table 2). Our “antimicrobial days” have the same meaning as the “days of therapy” reported by Polk and colleagues13; however, we’ve avoided this latter term because of its resemblance to “patient-days on antimicrobials.” The programming logic used to compute numerator events from the different data sources is summarized in Table 3. Antimicrobial use measures were summed in each ICU for each calendar month over the study period and for each of the antimicrobial agents and predefined drug classes (Table 4).
Antimicrobial use rates were calculated using ICU patient-days as the denominator. An ICU patient-day was attributed to each patient occupying an ICU bed at midnight of each day, as previously recommended14, so that events occurring on ICU admission days were counted while those occurring on the day of ICU discharge were not.
Data validation
The investigators participated in regular teleconference calls to discuss problems with programming and data collection, providing a forum for shared learning as problems encountered at one institution were evaluated in the context of experience at the others. Inspection of data derived from draft program code for face validity sometimes identified the presence of programming errors before more detailed validation efforts were begun, prompting detailed examination of programming code to pinpoint and correct programming flaws, and subsequent validation studies to confirm data validity.
Each hospital used two methods to systematically validate their data. First, to measure how accurately medication administration records reflected actual medication administration to patients, convenience samples of at least 100 intravenous antimicrobial doses scheduled for administration in study ICUs were prospectively audited and observed at the bedside for timeliness of administration using the method of Itokazu, et al.15 The results (timely dose administration or not) were compared with the disposition of the dose as registered in the medication administration records. Second, retrospective validation studies were conducted by assembling cohorts of randomly selected antimicrobial recipients from each of the ICUs during the study period and comparing counts of numerator events compiled by applying draft computer queries to these cohorts with manual review of the same patients’ medication administration records, our gold standard. After identifying and correcting programming errors, numerator counts derived via revised program code and manual chart review were applied to new cohorts of antimicrobial recipients and results compared until no new programming errors could be identified.
Results
Investigators at the four participating institutions were able to generate antimicrobial utilization data for each of the selected ICUs over the first 24 months of the 36-month study period and reported preliminary intra- and inter-ICU antimicrobial use rate comparisons.16 However, preliminary retrospective and prospective validation studies revealed major discrepancies between numerator counts that substantially biased the preliminary results and prompted careful review of the code used in the computer queries for systematic error by programmers and investigators. A summary of the programming errors is provided in Table 5. These errors were detected after inspection of the resulting reports suggested a lack of face validity or after detection of inconsistencies between computerized reports of patient-level data and charted medication administration record entries during retrospective validation.
After identifying and correcting all identifiable programming errors, numerator counts derived via revised program code and manual chart review were applied to new cohorts of antimicrobial recipients and results compared until no new programming errors could be identified. The discrepancies between computer-derived and manual counts of antimicrobial days and patient-days on antimicrobials presented in Table 6 reflect these final comparisons. The retrospective validation studies revealed variable levels of discrepancy by institution between numerator counts derived from application of final computer queries and manual review of medication administration records. Overestimation of counts of antimicrobial days and patient-days on antimicrobials generated by computer code were < 1% at hospital B, where antimicrobial utilization was computed from eMAR data. By contrast, programming of computerized pharmacy dispensing data at hospital C, where delayed delivery of paper medication orders from ICUs to pharmacy may have led to antimicrobial dispensing after drug discontinuation or patient discharge orders were written,15 counted 17.7% more and 14.5% more antimicrobial days and patient-days on antimicrobials, respectively, than manual record review did. Use of CPOE data to derive numerator antimicrobial measures at hospitals A and D generated intermediate levels of discrepancy (Table 6).
Prospective bedside observations of the intravenous administration of routinely scheduled antimicrobial doses revealed > 95% concordance between the observed outcomes of dose administration events and the corresponding dose administration status recorded in the MARs in all study ICUs (data not shown).
Discussion
Our findings show that derivation of standardized patient-level measures of antimicrobial utilization from a sample of hospitals with disparate computerized pharmacy systems is feasible. However, our experience highlights a few important issues related to the use of computerized data sources to derive and report hospital antimicrobial utilization rates.
First, inter-institutional differences in pharmacy computer systems and available data sources (Table 1) necessitated the use of institution-specific computing strategies (Table 3), contributing to varying levels of fidelity between antimicrobial utilization results obtained by application of computer code versus manual review of patient records (Table 6). Until greater uniformity of hospital data systems is achieved or until valid antimicrobial measurement programs are included within commercial and governmental hospital pharmacy computer systems, institution-specific strategies for data programming, validation and interpretation will have to be developed to ensure that accurate and comparable measures of antimicrobial utilization data are derived and reported across hospitals.
Second, we found that programming of antimicrobial utilization measures based on computerized pharmacy and administrative data was complex and error-prone. Despite considerable experience in querying and analyzing computerized data from our respective institutions,8,10–12 we made important errors in our initial attempts at deriving these antimicrobial utilization measures (Table 5). Our need to adopt separate, institution-specific computing strategies (Tables 1 and 3) and the complexities of computerized medical records, in general,17 and of pharmacy data, in particular,8 likely contributed to these problems. However, many of these mistakes stemmed from conceptual misunderstandings and inadequacies in communication between clinician- investigators and informaticists whose mitigation requires careful application of basic tenets of multidisciplinary collaboration and data review and validation (Table 7). Complementary methods – assessment of face validity, review of procedures for developing computer code and retrospective validation procedures – were necessary to fully identify and correct these errors, highlighting the importance of adopting a careful, systematic approach to collecting and validating data from electronic health records.
Third, after maximally validating program code in our respective institutions, we measured variable levels of over-estimation of computed numerator counts compared to a gold standard based on retrospective medical record review (Table 6). These discrepancies likely reflect inter-institutional variation in efficiency and coordination of medication ordering, distribution and administration procedures.15,18 In particular, in hospital C which had the highest level of discrepancies, delays in transport of paper medication orders to a centralized pharmacy may have contributed to delayed pharmacy response to ordered changes in patient antimicrobial regimens, leading to pharmacy dispensing of antimicrobial doses to the ICU that were not administered and therefore omitted from the MAR (Tables 1, 3 and 5). By contrast, use of bedside charting of eMAR data for both computer-derived and manually collected numerator data in hospital B doubtlessly accounted for the high affinity between data obtained through these different sources. Our results suggest that eMAR is the most accurate source for pharmacy numerator data in hospitals where it is in use; however, this finding requires confirmation from other institutions. Measures based on pharmacy dispensing or physician orders, being further removed from the antimicrobial administration event, are more likely to overestimate actual utilization.
The optimal metric for reporting hospital antimicrobial utilization is unclear.19–23 We chose antimicrobial days and patient-days on antimicrobials (Table 2) as primary numerator measures because they provide complementary information on the intensity and breadth of ICU antimicrobial use,23 they are minimally affected by variation in antimicrobial dosing regimens and they should be readily extractable, given the current widespread availability of detailed, patient-specific computerized pharmacy data within U.S. hospitals. Previous studies have used “days of therapy,”13 analogous to our antimicrobial days, to rank and assess secular trends in antimicrobial utilization in an alliance of 22 U.S. academic health centers and in 130 U.S. hospitals.24,25 However, those analyses were based on charge and billing data rather than the pharmacy dispensing, physician order or eMAR data that we employed. Also, the specific methods – proprietary in one instance – were not detailed and validation efforts were limited.24,25
Older U.S. and European antimicrobial utilization surveys, performed where detailed, patient-specific data may not have been available, have used conversion factors such as the defined daily dose (DDD) or recommended daily dose 26,27 to estimate patient-level antimicrobial use from aggregate antimicrobial use and census data to make inter-institutional or international comparisons,27–29 analyze secular trends,29,30 and correlate antimicrobial use with antimicrobial resistance.31–33 However, pharmacy reports of aggregate antimicrobial utilization may substantially overestimate actual use in hospitals with poorly coordinated mechanisms of medication ordering, distribution and administration.15 Moreover, DDDs underestimate patient-level exposure to drugs requiring renal dose adjustment,34 overestimate the use of antimicrobials for which the conversion factor is lower than doses that are commonly prescribed,13,35,36 and do not apply to most pediatric patients.13
Adaptation of programming approaches to the disparate information systems and data sources available in each institution will be a formidable challenge that will require understanding and avoiding potential errors that can impede the valid and efficient measurement of antimicrobial use. The CDC is currently revising the data submission requirements for the Antimicrobial Utilization component of the National Healthcare Safety Network (NHSN) to receive standardized summary measures from eMAR systems directly. This may represent the most effective approach to achieving accurate centralized collection, analysis and reporting of antimicrobial utilization measures from multiple institutions.37 Application of methods similar to ours by these and other surveillance efforts will be instrumental in achieving the widespread availability of valid and efficient measurements of antimicrobial utilization.
Acknowledgements
We thank Rick Carlson, PharmD for his assistance in the manual validation of data at LDS Hospital/Intermountain Healthcare.
Financial support. This work is supported by the Centers for Disease Control and Prevention cooperative agreement 1 U01 CI000327. Bernard Camins received salary support through a NIH/NCRR grant TL1RR024995.
Potential conflicts of interest. None to report (all authors).
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
ReferencesShlaesDMGerdingDNJohnJFJrSociety for Healthcare Epidemiology and Infectious Diseases Society of America joint committee on the prevention of antimicrobial resistance: Guidelines for the prevention of antimicrobial resistance in hospitalsClin Infect Dis1997255845999314444RubinMASamoreMHAntimicrobial use and resistanceCurr Infect Dis Rep2002449149712433323Interagency Task Force on Antimicrobial ResistanceA Public Health Action Plan to Combat Antimicrobial Resistance. Part 1. Domestic IssuesInteragency Task Force on Antimicrobial ResistanceAccessed June 23, 2009Available at: http://www.cdc.gov/drugresistance/actionplan/Centers for Disease Control and PreventionCampaign to Prevent Antimicrobial Resistance in Healthcare SettingsAccessed June 23, 2009Available at: http://www.cdc.gov/drugresistance/healthcare/default.htmDellitTHOwensRCMcGowanJEJrInfectious Diseases Society of America and the Society for Healthcare Epidemiology of America guidelines for developing an institutional program to enhance antimicrobial stewardshipClin Infect Dis20074415917717173212JhaAKDesRochesCMCampbellEGUse of electronic health records in U.S. hospitalsNew Engl J Med20093601628163819321858TokarsJIRichardsCAndrusMThe changing face of surveillance for health careassociated infectionsClin Infect Dis2004391347135215494912WisniewskiMFKieszkowskiPZagorskiBMTrickWESommersMWeinsteinRADevelopment of a clinical data warehouse for hospital infection controlJ Am Med Inform Assoc20031045446212807807Centers for Disease Control and PreventionPrevention Epicenters ProgramAccessed June 23, 2009Available at: http://www.cdc.gov/ncidod/dhqp/epicenter.htmEvansRSLarsenRABurkeJPComputer surveillance of hospital-acquired infections and antibiotic useJAMA1986256100710113735626McMullinSTReichleyRMKahnMGDunaganWCBaileyTCAutomated system for identifying potential dosage problems at a large university hospitalAm J Health Syst Pharm1997545455499066863KamalJRogersPSaltzJMekhjianHInformation warehouse as a tool to analyze Computerized Physician Order Entry order set utilization: opportunities for improvementAMIA Annu Symp Proc200333634014728190PolkREFoxCMahoneyALetcavageJMacDougallCMeasurement of adult antibacterial drug use in 130 US hospitals: comparison of defined daily dose and days of therapyClin Infect Dis20074466467017278056CohenALCalfeeDFridkinSKRecommendations for metrics for multidrugresistant organisms in healthcare settings: SHEA/HICPAC position paperInfect Control Hosp Epidemiol20082990191318808340ItokazuGSGlowackiRCSchwartzDNWisniewskiMFRydmanRJWeinsteinRAAntimicrobial consumption data from pharmacy and nursing records: how good are they?Infect Control Hosp Epidemiol20052639540015865276SchwartzDNEvansRSCaminsBCElectronic measures of hospital antimicrobial utilization: a multi-center pilot assessment of feasibility and variability in intensive care units17th Annual Meeting of the Society for Healthcare Epidemiology of AmericaApril 14–17, 2007Baltimore, MDabstract 36HoganWRWagnerMMAccuracy of data in computer-based patient recordsJ Amer Med Inform Assoc199753423559292840BarkerKNFlynnEAPepperGABatesDWMikealRLMedication errors observed in 36 health care facilitiesArch Intern Med20021621897190312196090FiliusPMGLiemTBYvan der LindenPDJanknegtRNatschSVultoAGVerbrughHAAn additional measure for quantifying antibiotic use in hospitalsJ Antimicrob Chemother20055580580815814603de WithKMaierLSteib-BauertMKernPKernWVDefined or prescribed daily doses? Patient days or admissions as denominatorInfection200634919416703299FitzpatrickRWEdwardsCMCEvaluation of a tool to benchmark hospital antibiotic prescribing in the United KingdomPharm World Sci200830737817682836KusterSPRuefCLedergerberBHintermannADeplazesCNeuberLWeberRQuantitative antibiotic use in hospitals: comparison of measurements, literature review, and recommendations for a standard of reportingInfection20083654955919011740BerringtonAAntimicrobial prescribing in hospitals: be careful what you measureJ Antimicrob Chemother20106516316819884120PakyzALMacDougallCOinonenMPolkRETrends in antibacterial use in US academic health centers: 2002 to 2006Arch Intern Med20081682254226019001203MacDougallCPolkREVariability in rates of use of antibacterials among 130 US hospitals and risk-adjustment models for interhospital comparisonInfect Control Hosp Epidemiol20082920321118257689HutchinsonJMPatrickDMMarraFMeasurement of antibiotic consumption: a practical guide to the use of the Anatomical Therapeutic Chemical classification and Defined Daily Dose system methodology in CanadaCan J Infect Dis200415293518159441de WithKMeyerESteib-BauertMSchwabFDaschnerFDKernWVAntibiotic use in two cohorts of German intensive care unitsJ Hosp Infect20066423123716891039CarlingPCFungTColdironJSParenteral antibiotic use if acute-care hospitals: a standardized analysis of fourteen institutionsClin Infect Dis1999291189119610524962Vander SticheleRHElseviersMMFerechMBlotSGoosensHHospital consumption of antibiotics in 15 European countries: results of the ESAC Retrospective Data Collection (1997–2002)J Antimicrob Chemother20065815916716698845Müller-PebodyBMuscatMPelleBKleinBMBrandtCTMonnetDLIncrease and change in pattern of hospital antimicrobial use, Denmark, 1997–2001J Antimicrob Chemother2004541122112615537692MonnetDLLennoxKAPhillipsLTenoverFCMcGowanJEGaynesRPAntimicrobial use and resistance in eight US hospitals: complexities of analysis and modelingInfect Control Hosp Epidemiol1998193883949669619FridkinSFEdwardsJRCourvalJMThe effect of vancomycin and third-generation cephalosporins on prevalence of vancomycin-resistant enterococci in 126 U.S. adult intensive care unitsAnn Intern Med200113517518311487484LeschCAItokazuGSDanzigerLHWeinsteinRAMulti-hospital analysis of antimicrobial usage and resistance trendsDiagn Microbiol Infect Dis20014114915411750169ZagorskiBMTrickWESchwartzDNWisniewskiMFHershowRCFridkinSKWeinsteinRAThe effect of renal dysfunction on antimicrobial utilization measurementsClin Infect Dis2002351491149712471568MandyBKoutnyECornetteWoronoff-LemsiM-CTalonDMethodological validation of monitoring indicators of antibiotics use in hospitalsPharm World Sci200426909515085943de WithKBestehornHSteib-BauertMKernWVComparison of defined versus recommended versus prescribed daily doses for measuring hospital antibiotic consumptionInfection20093734935219277464EdwardsJRPollockDAKupronisBAMaking use of electronic data: the National Healthcare Safety Network eSurveillance initiativeAm J Infect Control200836supplS21S2618374208
Intensive Care Units and Information Technology Resources of Participating Hospitals
HospitalIdentifier
ICUs
MedicationOrderingSystem(PhysicianOrders)
MedicationDispensingSystem(PharmacyOrder Entry)
MedicationAdministrationSystem
ComputerizedDataWarehouse(CDW)
ProgrammingLanguage
NumeratorData Source(s)
DenominatorData Source(s)
Hospital Aa
MICU, SICU, CCU
Paper, CPOE
SMS, Cerner
Paper, eMAR
CARP
SQL
Pharmacy dispensing, Physician orders
Pharmacy dispensing, ADT
Hospital B
MICU, CCU, STRI, RSCU, TICU
Paper
HELP Pharmacy
eMAR
HELP 1 and 2 Enterprise
Oracle SQL, C++
Medication administration
Medication administration
Hospital C
MICU, SICU, CCU
Paper
PDM Medication Control System
eMAR
Washington University and BJC Healthcare
Sybase SQL
Pharmacy dispensing
Pharmacy dispensing
Medical Informatics Database
Hospital D
MICU, SICU, BMT
CPOE
Centricity, Siemens, Essentris OSUMC IW
Paper
OSUMC Information Warehouse (IW)
Oracle SQL
Physician orders
Physician orders
Note. ADT, admission-discharge-transfer; BMT, bone marrow transplant; CCU, coronary care unit; CPOE, computerized provider order entry; eMAR, electronic medication administration record; ICU, intensive care unit; MICU, medical ICU; RSCU, respiratory special care unit; SICU, surgical ICU; SQL, Structured Query Language; STRI, shock-trauma-respiratory intensive care unit; TICU, thoracic ICU.
A change in the pharmacy system vendor during the study time period necessitated use of two data sources.
Numerators Used in Antimicrobial Utilization Measures and Their Definitions
Measure
Definition
Example
Antimicrobial days
Sum of the calendar days on which each antimicrobial was administered
2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 15 antimicrobial days
Patient-days on antimicrobials
Sum of the calendar days on which one or more antimicrobial drugs was administered
2 drugs given for 5 days followed by a different patient-days on antimicrobials
Antimicrobial starts
Sum of the calendar days on which each new antimicrobial drug was started, following 3 or more days without exposure to that drug
2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 3 antimicrobial starts
Antimicrobial courses
Sum of the calendar days on which any antimicrobial drug was started, following 3 or more days without exposure to any antimicrobial drug
2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 1 antimicrobial course
Defined daily doses (DDDs)
World Health Organization-standardized conversion of aggregate drug dosing data into number of doses26
200 grams of vancomycin dispensed divided by 2 grams per vancomycin DDD = 100 DDDs of vancomycin
Logic Used in Computing Numerators Used in Antimicrobial Utilization Measures from Different Computerized Data Sources
Data Source
Events Measureda
Logic Applied
Pharmacy dispensing
Antimicrobial doses dispensed from pharmacy
One or more doses of each antimicrobial dispensed during an ICU day constitutes an antimicrobial day; one or more doses of any antimicrobial dispensed during an ICU day constitutes a patient-day on antimicrobials.
Physician orders (CPOE)
Antimicrobial start and stop orders; ICU admission and discharge days
ICU days on which each antimicrobial is ordered for continuous scheduled administration and subsequent ICU days are counted as antimicrobial days until either that drug’s iscontinuation d is ordered or until ICU discharge. ICU days on which any antimicrobial is ordered and subsequent ICU days are counted as patient-days on antimicrobials until either the discontinuation of all antimicrobials has been ordered or until ICU discharge.
Medication administration (eMAR
Antimicrobial doses administered by nurse
One or more doses of each antimicrobial administered during an ICU day constitutes an antimicrobial day; one or more doses of any antimicrobial administered during an ICU day constitutes a patient-day on antimicrobials.
ICU day constitutes a patient-day on antimicrobials.
Note. CPOE, computerized provider order entry; eMAR, electronic medication administration record.
Numerator events are counted only through the calendar day before ICU discharge.
Errors Encountered During Validation of Computed Antimicrobial Utilization Rates from Computerized Data Sources
Data Source
Error
Cause
Solution
Numerator data
1. Physician orders (CPOE)
Spuriously high cefazolin utilization rates in an ICU in Hospital A after change to new pharmacy computer system.
Unknown to study personnel, an automated testing procedure added 233 cefazolin days over 7 months to non-existent patients.
Test entries were removed, with revised cefazolin utilization rates more comparable to rates ascertained from data from older pharmacy system.
2. Physician orders (CPOE)
Spurious increase in antimicrobial utilization rates in Hospital A after change to new pharmacy computer system.
Errant programming of ADT data in new system led to inappropriate attribution to the ICU stay of antimicrobial use that occurred following transfer from the ICU.
Programming was revised to limit numerator events to ICU patient- days as defined by the ADT tables and new results were validated.
3. Pharmacy dispensing
Patient-days on antimicrobials were calculated incorrectly, with rates exceeding the maximum 1,000 patient-days on antimicrobials per 1,000 ICU patient-days in Hospital C.
The programmer misunderstood the definition of patient-days on antimicrobials.
Rates were corrected and validated after the definition of patient-days on antimicrobials was clarified.
Denominator data
1. Medication administration (eMAR)
Spuriously high antimicrobial utilization rates in an ICU in Hospital B.
Communication error led to programming of ICU antimicrobial recipients only, rather than all ICU patients, in computing denominator data.
Rates were reprogrammed to include all ICU patients and new results were validated.
2. Pharmacy dispensing
Spuriously low antimicrobial utilization rates in one ICU in Hospital C.
Summed denominator days from two parallel nursing units comprising an ICU were used to calculate antimicrobial use rates for each nursing unit instead of calculating denominator ICU days from each nursing unit separately.
Rates were re-computed and validated after the programmer was made familiar with physical layout of the ICU.
A change in the pharmacy system vendor during the study time period necessitated use of two data sources.
Potential Sources of Error in Computing Antimicrobial Use Rates from Computerized Pharmacy Data and Suggestions for Avoiding Them
Potential Sources of Error
Suggested Solutions
Data request/acquisition
Programmer’s understanding of clinical concepts and goals underlying investigator’s request for data may be incomplete.
Written request for data must clearly outline clinical concepts and goals.
Programmer should have frequent access to investigator to clarify conceptual issues
Programmer’s understanding of the physical context and clinical processes and procedures relevant to the request for data may be incomplete.
The programmer should tour site(s) of care being studied.
The programmer should be oriented to the processes of care relevant to the request for data (e.g., hospital processes of medication ordering, distribution and administration) emphasizing the point(s) at which the computerized data being studied are generated.
Investigator’s understanding of data structure and programming procedures may be limited.
Programmer should describe proposed approach to request for data, including anticipated shortcomings in data structure or availability, to investigator before writing program code.
Programmer should review program code with investigator before executing programming procedures.
Data analysis
Computerized data may require careful review for outlier entries and may require re-formatting (e.g., converting free text to categorical tabular entries) before programming is possible.
Programmer should involve investigator in review of data tables and obtain investigator’s guidance in interpreting outlier entries.
Magnitude of data cleaning and conversion efforts must be estimated and the necessary resources - primarily personnel time - allocated.
Investigator must assist in designing data restructuring plan and provide necessary nomenclature and definitions.
Data validation
Investigator’s predisposition to trust integrity of programming processes and data derived thereby (“if it’s on a computer screen, it must be right”) may be misguided.
Reports based on queries of computerized data must be carefully reviewed for face validity:
Are all expected data elements (e.g., antimicrobial names) represented?
Are results comparable to previously validated data, if available?
Do the results reflect anticipated variation?
Reports based on queries of computerized data require validation, ideally via manual comparison of samples of computerized data with an acceptable gold standard.
Systematic sources of error must be vigorously sought to explain recurring or substantial discrepancies in the results of these comparisons.