Central line–associated bloodstream infections (CLABSIs) are an infection outcome for many infection control studies. Despite the utilization of standard definitions and training, there may be variability in the performance of CLABSI surveillance [11]. In a study of identical patient records reviewed by multiple infection preventionists, up to 40% of the cases were described as “uncertain,” and the overall agreement among multiple raters was only moderate (κ, 0.45) [12]. Poor-to-moderate interrater reliability has been demonstrated in other studies [13, 14]. Such variability likely reflects the underlying clinical uncertainty in diagnosing many catheter-related bloodstream infections. Although common CLABSI surveillance definitions contain objective elements (“patient has a recognized pathogen cultured from 1 or more blood cultures”), there are subjective elements that rely on the assessor’s judgment (“organism cultured from blood is not related to an infection at another site”) [15]. In practice, incomplete clinical data or subjective assessments of infections at non-blood body sites can complicate judgment of whether a bloodstream pathogen originated from the bloodstream; for example, Escherichia coli in the blood, recovered from a patient with a central venous catheter who had recent abdominal surgery, could reasonably represent either a CLABSI or a surgical site infection. Anecdotal disagreement in CLABSI determinations among expert reviewers underscores the surveillance definition’s uncertainty in some clinical contexts [16].

Ventilator-associated pneumonias (VAPs) are even more clinically challenging than catheter-related bloodstream infections to diagnose because many common conditions can mimic VAP, such as acute respiratory distress syndrome, thromboembolic disease, pulmonary hemorrhage, congestive heart failure, and atelectasis [17]. The clinical challenges in diagnosing VAP are reflected in many standardized definitions of VAP. Commonly used definitions contain multiple subjective elements, requiring the assessor to judge “new or progressive and persistent infiltrate” or “new onset of purulent sputum, or change in character of sputum, or increased respiratory secretions, or increased suctioning requirements” [18]. In a study of 50 ICU patients on mechanical ventilation, 2 experienced assessors used NHSN criteria to assess for VAP; 1 assessor identified nearly twice the number of VAPs (20) as the other (11), for only moderate agreement (κ, 0.50) [19]. Including microbiologic criteria such as respiratory cultures (not uniformly available and thus optional in most surveillance definitions) increases specificity, but cultures lack sensitivity [17] and, more importantly, do not eliminate the subjective criteria.

A standardized pneumonia measure, the clinical pulmonary infection score, combines clinical, radiologic, and microbiological criteria for diagnosis of VAP. Yet when the clinical pulmonary infection score was compared with quantitative bronchoalveolar lavage cultures as a reference standard, specificity was low and interrater agreement between 2 intensivist assessors was imperfect [20].

Other infection outcomes that are defined by surveillance definitions contain elements requiring subjective interpretation (Table 1) [7]. Common definitions for catheter-associated urinary tract infections require at least 1 of the following signs or symptoms—fever, suprapubic tenderness, or costovertebral angle tenderness—with “no other recognized cause” [21]. In actuality, symptoms are often subtle or nonexistent among bladder-catheterized patients [22], and determining the source of fever requires substantial judgment. Similarly, Clostridium difficile infection surveillance requires the assessor to judge whether patient has “liquid stool” [23, 24].

Thus, infection outcomes used for research, even those that are driven by protocolized surveillance definitions, can be highly subjective. This is not a criticism of the surveillance definitions per se, because the subjectivity was originally designed to allow trained assessors in routine surveillance settings to use clinical judgment to potentially improve the specificity of their determination. Nevertheless, in research settings in which an infection control intervention is being evaluated, unblinded assessment of infection outcomes raises the danger of assessment bias.

If a subjective outcome is used for a study, such as NHSN surveillance-defined infections, then the ideal defense against assessment bias is to blind the assessor to the allocation of the intervention. This method can be feasibly performed in retrospective studies of historical infection rates, if the assessor (infection preventionist) is unaware of a study or intervention occurring. For example, in a study comparing prospectively determined infection preventionist CLABSI rates with computer algorithm CLABSI rates determined retrospectively, the infection preventionists could be blinded to the study protocol [11]. Blinding becomes more difficult for prospective infection control interventions that are designed to alter infection control practice, because infection preventionists who perform outcome assessment are usually closely involved with instituting infection control interventions as part of their duties. Blinded assessment by infection preventionists is feasible in situations where there is randomization of the intervention and masking of allocation. For example, in a study of the efficacy of stop orders to reduce catheter-associated urinary tract infection as one of several endpoints, patients were randomized to stop orders or usual care. The assessors in the study were explicitly blinded to the intervention assignment and found that there was no difference in infection outcome between the 2 groups [27].

Another general approach to limit assessment bias is to use objective outcomes that are less susceptible to bias. Commonly available objective outcomes include mortality, length of stay, antimicrobial use, or incidence/prevalence of pathogen-specific colonization. For example, investigators of decontamination of the digestive tract and oropharynx in ICU patients chose 28-day mortality as their primary endpoint instead of VAP [28], because they recognized the subjectivity of the pneumonia outcome. Further, the investigators recognized that even in-hospital mortality could be biased by physicians who, knowing treatment allocation, could influence discharge decisions among patients in one intervention group versus another; thus, a 28-day mortality outcome that included out-of-hospital events was selected.

For investigators who are interested in objective measures of infection, several options are available. For example, NHSN has developed objective surveillance definitions that only rely on clinical culture results obtained from the laboratory (“laboratory-identified events”) [29]. Laboratory-identified events can be reported to NHSN for organisms such as methicillin-resistant and methicillin-susceptible Staphylococcus aureus, VRE, carbapenem-resistant Klebsiella spp., and C. difficile, and the surveillance rules can be adapted to any bacteria of interest. The laboratory-identified event is designed to be determined either by computers or by humans.

A simpler and practical form of a laboratory-identified event would be to use the presence of nosocomial clinical cultures identified by the microbiology laboratory for pathogens and culture sites of interest. Such laboratory-identified culture-based methods are potentially scalable; a single hospital, or a large number of hospitals, can obtain such an outcome as long as microbiology data are accessible. For example, a cluster randomized trial of 3 different interventions to reduce MRSA disease in hospital ICUs among 45 hospitals utilized a primary outcome of “number of ICU patients who have MRSA-positive clinical cultures occurring at least 2 days after ICU admission through 2 days after ICU discharge” [30].

Other objective measures of infection include the use of automated computer systems to perform infection surveillance [31]. Automated surrogate measures of infection have been developed for CLABSI [14, 32], catheter-associated urinary tract infection [33, 34], surgical site infection [35, 36], and VAP [37]. Commercially available automated measures of infection have also been developed [34, 38]. These infection surveillance measures can be used either as primary or secondary outcomes; in particular, the automated measures can be used to confirm findings of another infection measure. Investigators studying the use of chlorhexidine bathing among medical ICU patients used CLABSI as a primary outcome; because of the possibility of incomplete blinding, they used a computer algorithm for CLABSI determination to validate their primary outcome, demonstrating that there was no significant bias [39].

Other miscellaneous methods can be used to combat assessment bias. A study can use multiple observers for infection determination (eg, use 2 different assessors plus an adjudicator in cases of disagreement). This consensus strategy is resource intensive and may lead to more conservative rates compared with single-observer surveillance, but it has been used to increase confidence in infection-related outcomes such as CLABSI [39] and VAP [40]. Lastly, with remote computer access available for electronic medical records, it is feasible for external assessors from a nonstudy location who are not intellectually invested in the study to perform chart reviews for infection assessment.

Measures designed to minimize assessment bias still require study designs that protect against other types of bias. Any measure that relies on microbiological cultures can be susceptible to surveillance bias (ie, prior knowledge of an intervention can affect how intensively clinicians search for infection) [41]. For example, physicians who are unblinded to an intervention to decrease urinary tract infections may order urine cultures less frequently among one group of patients compared with another in response to fever, leading to a potentially false difference in culture-based infection rate. If masking of the intervention is not feasible, then standardizing microbiologic culturing practice may be advisable. Additionally, misclassification bias can occur if clinical cultures are obtained after (rather than before) the initiation of new antibiotics at the time that an infection is suspected, increasing the rate of falsely negative cultures. Investigators should be aware of secular changes in antimicrobial prescribing practice (such as initiatives for early antimicrobial therapy in response to suspected sepsis) that may differentially bias the ability of cultures to detect true infection.