Surveys were distributed to 143 SHEA Research Network members from 126 hospitals. In total, 113 responses were obtained, representing 61 unique hospitals (30 respondents did not name their hospital). Assuming that the number of respondents not identifying a hospital had a percentage of unique hospitals similar to that among those who did identify their hospital, an additional 22 hospitals would be obtained, giving a hospital response rate of 65.9%.

Of the respondents, 79.2% (84 of 106 nonmissing responses) were IPs, with a mean length of employment in infection prevention of 11.9 years (Table 1). Eighty-one participants (75.7% of 107 nonmissing responses) had a clinical background. Most respondents worked at large acute care academic hospitals. Eighty-nine respondents (78.8% of 108 nonmissing responses) worked in states mandating HAI reporting.

Regarding the 6 vignettes, 97.2% (106 of 109 nonmissing responses) of respondents correctly answered the negative control question, and 97.3% (108 of 111 nonmissing responses) correctly answered the positive control question. Percentage of correct answers for the other items ranged from 87.5% (98 of 112 nonmissing responses for an item involving a CLABSI) to 27.4% (29 of 106 nonmissing responses for an item involving C. difficile). The mean percentage of correct responses (out of 6) was 61.1% (standard deviation, 0.96%; Table 2). The free-marginal κ score over all 6 items was 0.51. If the positive and negative controls were removed, the free-marginal κ score decreased to 0.32.

In unadjusted analyses, number of correct responses was not associated with being an IP, clinical background, hospital size, infection prevention and control department size, or length of time in infection prevention (Table 3). In multivariate analyses, when adjusted for individual and hospital characteristics, percentage of correct responses was not associated with being an IP, years in infection prevention, presence of a CIC credential, presence of state-mandated reporting, hospital size, or infection prevention and control department size (Table 3). However, having a clinical background was associated with answering more questions correctly (adjusted P = .044). There was no association between state-mandated reporting of a particular HAI and the likelihood of correctly characterizing a related clinical vignette (Table 4).

Of those who gave the name of their hospital, most (47 [74.6%]) were the only respondents from their hospital. Other hospitals had between 2 and 5 respondents. Percentage of correct responses per hospital did not differ significantly from the study population overall (data not shown). Within individual hospitals, percentage of correct responses varied among respondents.

Analyses were performed to determine whether participants who did not name their hospital differed from those who did. Those who named their hospital had a higher percentage of correct responses (P = .022). The mean number of correct responses for those who named their hospital was 3.8 (95% confidence interval [CI], 3.6–4.0) of 6, or 63.3%. The mean number of correct responses among those who did not name their hospital was 3.3 (95% CI, 2.8–3.8) of 6, or 55.0%. There were no significant differences in infection prevention and control department characteristics, hospital characteristics, or professional background between those who did and did not name their hospitals.

Analyses also determined the effect of removing the tracheobronchitis item (which is uncommonly targeted in hospital surveillance) from the results. Without this item, the mean number of correct responses was 2.91 (95% CI, 2.76–3.07) of 5, or 58.2%, which was not significantly different. The free-marginal κ score in the absence of the tracheobronchitis item over the 5 remaining items was not significantly different (0.50). When the tracheobronchitis item and the positive and negative controls were removed, the free-marginal κ score decreased to 0.24.

By studying how those performing HAI surveillance in hospitals across the United States apply NHSN HAI definitions to clinical vignettes, we sought to model whether variation in interpretations of definitions may exist. While the majority of the respondents correctly identified a positive control as an HAI and a negative control as a non-HAI, rates of respondents correctly identifying other HAIs ranged from a high of 87.5% to a low of 27.4%. The mean percentage of correct responses on the survey was low (61.1%), and interrater reliability was also low between respondents (κ = 0.32, after excluding the control items). The overall percentage of correct responses and overall interrater reliability may be low enough to raise doubts about agreement between those performing HAI surveillance. Presence of a clinical background was the only characteristic identified as associated with more correct responses to survey questions. Research should focus on improving accuracy and reliability in applying HAI surveillance criteria.

Prior studies looking at the reproducibility of NHSN HAI criteria have focused on just one type of HAI, typically CLABSI or VAP.^{10–12,14–16} We included items about not just CLABSI and VAP but also C. difficile, tracheobronchitis, and organ space/surgical site infection. Some have focused on the experiences of reporting HAIs in just one state^10,11,13,20 or in a few academic medical centers.^15,16 We looked at reporting from 28 different states. No studies before ours have included information about respondent workplaces or their training. In addition, 17.9% of our respondents were hospital epidemiologists, a group in which HAI interpretation concordance has not been well studied. While our study was not powered to explore differences between hospital epidemiologist and IP responses to HAI items, we did not see a significant difference in the number of correct responses. Understanding hospital epidemiologist HAI reporting is important and should be an area of future research, as in many hospitals both hospital epidemiologists and IPs discuss reporting of particular HAIs.²¹

The mean percentage of correct responses on the survey was 61.1%, similar to the 57.1% concordance with NHSN experts seen in a survey investigating CLABSI cases in 18 Australian hospitals.¹¹ Our findings are also similar to a survey of IPs in 10 pediatric ICUs, in which all would have named at least one CLABSI in a manner inconsistent with NHSN definitions.¹⁴

In this study, interrater reliability was lower than that seen between CLABSIs reported by 44 Oregon hospitals versus those identified by state health officials (κ = 0.32 vs κ = 0.77)¹³ but was similar to that seen between CLABSIs reported by Australian hospitals versus those identified by central health officials (κ = 0.31)¹⁰ and between CLABSIs categorized by Veterans Affairs IPs (κ = 0.42).¹² While precise comparisons are difficult given that these prior studies focused just on CLABSIs, involved actual patient data, and used different scenarios, the data suggest only low to moderate agreement among those performing HAI surveillance. Prior studies have suggested that some may overreport infections,²² while others may feel pressure to underreport HAIs by their place of employment.¹⁹ Outside experts may find more HAIs than were reported.^20,23 In this study, however, with no consequence to their answers, participants likely felt little external pressure to report or not report certain HAIs. Our study may present a more accurate depiction of respondent understanding of NHSN definitions.

Two items had particularly low percentages of correct responses. First, only 27.4% of participants correctly reported that a clinical vignette did not meet criteria for a healthcare-associated C. difficile infection. Respondents may have struggled with the multiple infection sources in the scenario. Second, only 57.1% of respondents correctly identified that a potential CLABSI, in reality a secondary bloodstream infection from a surgical site infection, should not be reported. Others have commented on the difficulty in identifying NHSN-defined CLABSIs in the presence of other potential infections.^14,21 HAI definitions in the presence of more than one potential source of infection may require further clarification.

We did not identify differences in HAI reporting characteristics based on state-mandated reporting requirements, hospital characteristics, or characteristics of infection prevention and control departments, although those with a clinical background were more likely to achieve higher scores. Unfortunately, it is unclear how to improve the interobserver reliability of HAI surveillance. Further studies are first needed to understand what affects the application of HAI criteria.

Problems with standardization of HAI definitions are becoming more apparent. For example, members of the CDC’s Prevention Epicenters Program are considering criteria for ventilator-associated events as a replacement for VAP.^15,16,24 Others are trialing electronic surveillance systems for HAIs.^25–33 A recent study comparing interrater reliability between IPs at Veterans Affairs hospitals and through a laboratory-based algorithm actually found a higher concordance between IPs and the algorithm than among IPs.¹² Efforts such as these should continue.

Our study also offers an example of the use of a powerful new research tool for healthcare epidemiology, the SHEA Research Network. Prior studies using this resource have included a survey of influenza epidemic preparedness³⁴ and a retrospective multicenter cohort study of member hospitals on the effect of CMS catheter-associated urinary tract infection reimbursement rule changes on antimicrobial prescribing practices.³⁵ Future studies in healthcare epidemiology could continue to take advantage of this research consortium.

Our study has some potential limitations. The study population may not represent all those performing surveillance. To ensure that we were capturing those who may be affected by changes in CMS reimbursements,^4,5,17 we focused on US hospitals that used NHSN criteria to report HAIs. Our results may not be applicable to those who work outside the United States or do not use NHSN criteria. Furthermore, we excluded those working at pediatric-only hospitals, so results may not apply to those performing surveillance in pediatric hospitals. In addition, some of our respondents came from the same hospital. However, the bias this would have created would have been toward the null; that is, the actual variation between those performing surveillance may have been even higher.

We used hospitals enrolled in the SHEA Research Network for our survey so as to reach hospitals in different states. However, those performing HAI surveillance in the SHEA Research Network may not represent all those performing HAI surveillance. Hospitals in the SHEA Research Network are more likely than other hospitals to be larger, to be academic, and to have more members in their infection prevention and control departments.¹⁸

Finally, decision making in this survey may not apply to real-world situations. Reading a short paragraph to determine the presence of an HAI is not the same as a thorough chart review. Responding anonymously to a clinical vignette is not the same as making a decision with potential consequences to the hospital, clinician, and patient. Responses to our survey may not precisely match behavior, although we did seek feedback to make vignettes as realistic as possible. In addition, to decrease the length of the survey we included only a small number of vignettes, which may not be completely representative of all HAIs.

We showed low interrater reliability between those performing HAI surveillance in our study. Rates this discordant in practice could dramatically affect not only hospital reputations but also hospital reimbursement in the Value-Based Purchasing Program.^4,5,17 Prior to the implementation of CMS HAI reimbursement changes, more reproducible definitions of HAIs—or even new approaches to HAI surveillance, such as electronic surveillance—are urgently needed.