We assembled a 10-member panel of clinical experts in hospital medicine, pediatrics, critical care, healthcare epidemiology, and infectious diseases - with additional expertise in infections in immunocompromised and community hospital-based practice; six were affiliated with CDC Prevention Epicenters sites participating in the multi-center study of HOB and four others identified based on complementary clinical expertise.

Hypothetical HOB case scenarios were constructed by 4 infectious diseases physicians and healthcare epidemiologists (GMS, ASS, SB, SL) based on cases at their institutions, aiming for observed distributions of HOB sources and causative organisms in prior research and clinical experience.^{12 13} Scenarios were outlined using a standard template (Supplemental Figure 1). A total of eighty-two HOB case scenarios were created, representing a spectrum of sources of HOB attribution in adult and pediatric patients (Table 1).

First, each expert panelist was asked to independently review and rate each HOB case scenario using a 6-point Likert scale of preventability ranging from “Definitively or Almost Certainly Preventable” (1) to “Definitively or Almost Certainly Not Preventable” (6).¹⁴ This scale is modeled on preventability rating of adverse events in healthcare ¹⁵ and has previously been used in healthcare epidemiology for preventability evaluation of sepsis-related mortality, healthcare-associated bloodstream infections at a French university hospital, and in our pilot study of HOB preventability in the United States.^{8 16 17} Experts were instructed to assess preventability as: a) unintended bacteremia or fungemia related to medical management, b) as a failure to meet expected standard of care while accounting for a patient’s risk of infection, or c) as a deficiency of known preventive measures and standards of care including diagnosis and management.

We used guidance outlined in the RAND/UCLA Appropriateness Method User’s Manual to define agreement, disagreement, and development of expert consensus, using three clinically meaningful categories of preventability ratings based on the 6-point Likert scale: Preventable (1–2), Uncertain (3–4), or Not Preventable (5–6).¹⁸ We defined the level of agreement amongst the ten-member expert panel as follows: disagreement occurred when ≥20% of experts rated a scenario 1–2 (Preventable) and ≥20% of experts rated the same scenario 5–6 (Not Preventable). In the absence of disagreement, consensus was concurrence on the same preventability category among ≥70%, and majority agreement was concurrence on the same preventability category among >50%. Case scenario review and rating were done independently by each expert panelist using an online survey and data collection tool (Qualtrics, Provo, UT). We created summaries of ratings for each scenario including median, 75^th percentile, and whether criteria for consensus, majority agreement, or disagreement were met. A distribution of ratings by the panel and the individual’s rating in relation to others was provided to each expert (Supplemental Figure 2).

Using Delphi methodology, case scenarios that did not meet consensus criteria were deliberated by the expert panel via two teleconferences and a web-based discussion board (Slack Technologies, Inc., San Francisco, CA) moderated by the principal investigator (SL).^{7 8 19} During discussions, panel members were asked to voluntarily offer rationale for their choices. Individual ratings were not disclosed to the group by the moderator unless explicitly offered by the individual themselves. Teleconferences were recorded with permission from all participants. Following deliberation, a second round of independent review and rating was performed of the scenarios that did not meet consensus criteria during the first round (Figure 1).

We reviewed the content of teleconference and web-based discussions to inform the development and refinement of the structured rating guide and to provide insight into the aspects of the case that were deemed to be preventable.

In a previous pilot study of HOB preventability assessment, we conceptualized risk of developing HOB as a function of “intrinsic” or patient-related factors and “extrinsic” or healthcare-related factors. To explicitly account for both components of HOB risk, we developed a matrix to separately assign a patient’s intrinsic risk of HOB and preventability related to extrinsic healthcare factors. The overall preventability of HOB was then derived as the combined effect of the intrinsic and extrinsic risk rating from this matrix and translated to the 6-point Likert scale (Figure 2). During the pilot study, limited guidance around classification of intrinsic risk and extrinsic preventability was provided and variability in reviewer assessment of risks was recognized as a limitation.⁸ To overcome that limitation and minimize subjectivity of ratings, we created a more comprehensive list of clinical conditions and developed the present HOB preventability rating guide (Supplemental Appendix).

In this guide, intrinsic risk guidance was based upon the acute illness and underlying comorbidities present upon index admission and prior to the HOB event, informed by both literature review and known pathogenesis of bloodstream infections related to these conditions. Extrinsic risk accounted for preventability relative to the suspected source of infection. The guide was not made available to the reference standard expert panel.

Two additional healthcare epidemiology experts (ADH and JJ) were asked to independently review and rate the same eighty-two hypothetical case scenarios utilizing the structured guide. For each scenario, reviewers assigned an intrinsic and extrinsic health-care related risk of HOB using the guide, which informed the overall HOB preventability rating. The two reviewers were asked to provide justification for their decision making for each rating. Summaries of each scenario, including the difference between reviewers’ overall preventability rating, intrinsic risk, and extrinsic health-care related risk assignments, were created.

Following the initial round of scenario rating with the structured guide, two investigators (GMS and SL) performed review of all scenario ratings where agreement between the two reviewers was not achieved and provided feedback to reviewers on any notable deviations from intended use of the guide. Reviewers’ ratings were maintained as independent and not shared with the other. Reviewer feedback on the structured guide was also incorporated and the guide was refined to improve usability. This was followed by repeat independent review and rating of scenarios by the same two reviewers (Figure 1).

For the purposes of analysis, HOB scenarios were categorized by pre-defined sources of infection, and Likert scale ratings were assigned to the appropriate category of preventability: 1–2 (preventable), 3–4 (uncertain), or 5–6 (not preventable) as previously described. For expert consensus development, descriptive analyses of expert panel and reviewer ratings were performed and summarized as consensus, majority agreement, and disagreement after each round of review. For guide-based rating by the two independent reviewers, results were compared to those of the expert panel as a reference standard. The proportion of cases (and 95% confidence intervals) where guide-based preventability category (i.e., 1–2, 3–4, or 5–6) matched expert consensus was used to calculate overall agreement, sensitivity, and specificity by preventability category. These calculations were only done for cases where expert consensus was achieved, given the lack of an available reference standard in the absence of that consensus. Inter-rater reliability of ratings by the two reviewers using the structured rating guide was evaluated using the Kappa statistic using the same preventability categories.^{20 21} Analyses were performed using SAS 9.4 (SAS Institute, Cary, NC). This study was approved by the Institutional Review Board of the University of Maryland, Baltimore.

After the first round of expert panel preventability rating, 29 scenarios (35%) met criteria for consensus; an additional 17 (21%) had majority agreement. Nineteen scenarios (23%) met criteria for disagreement. During panel discussion of scenarios that did not meet consensus criteria, a number of themes emerged regarding the assessment of HOB preventability. These are summarized with illustrative quotations in Table 2.

Following the second round of expert panel rating, a total of 52 scenarios (63%) met criteria for consensus (27% [n=14] preventable, 44% [n=23] uncertain, 29% [n=15] not preventable). Eighty-five percent of scenarios had at least majority agreement and two (3%) met criteria for disagreement. Expert consensus was achieved more frequently for HOB attributed to contaminants, respiratory, and surgical site infection sources than for urinary tract and central lines (Figure 3).

Following an initial round of adjudication using the rating guide, there was agreement between the two reviewers in 45 (55%) scenarios (Kappa 0.31 [95% CI 0.12, 0.49]). Reviewers acknowledged challenges in separating the concept of intrinsic from extrinsic risk in several cases. Key issues included prolonged hospitalization, patients with numerous comorbidities with dependence on multiple indwelling devices and for whom the intrinsic risk was so great that HOB seemed “inevitable”, and scenarios in which there were deviations from the expected standard of safe, quality care delivery such that considering the patient’s intrinsic risk seemed inappropriate when determining HAI preventability. Following the second round of reviewer adjudication, agreement between the two reached 84% overall (Kappa 0.76 [95% CI 0.64, 0.88]). Comparison of preventability ratings between the first and second rounds of review is demonstrated in Supplemental Figure 3.

After two rounds of rating, comparing reviewer adjudication of preventability ratings with those of the expert panel, among cases with expert consensus (n=52), guide-based rating agreed with the expert panel consensus in 40 (77% [95% CI 65%, 88%]) scenarios for Reviewer 1 and 39 (75% [95% CI 63%, 87%]) for Reviewer 2. For both reviewers, the sensitivity and specificity were higher for cases rated as preventable or non-preventable, and lower for expert panel ratings in the uncertain range (Table 3). Inter-reviewer agreement was 87% (Kappa 0.79 [95% CI 0.65, 0.94]).