Inpatient hypoglycemia can be life-threatening, and is associated with longer hospital stays and increased medical costs.^1–5 Severe hypoglycemia (<40 mg/dL) occurs in 2–5% of hospitalized patients with diabetes mellitus,^5–8 and medication-related hypoglycemic events are common causes of adverse drug events occurring in inpatient settings.⁹ Up to half of inpatient adverse drug events may be preventable,¹⁰ and recent studies show that rates of severe hypoglycemia vary across hospitals, suggesting opportunities for improved care.¹¹

In 2014, the National Quality Forum (NQF) endorsed an electronic clinical quality measure (eCQM) for hospital-wide inpatient hypoglycemia reporting (NQF 2363) and recommended use of the measure in Centers for Medicare & Medicaid Services (CMS) inpatient quality reporting programs.¹² While NQF 2363 was intended to support reliable and timely measurement of hypoglycemia event rates, its implementation has been limited by costs associated with paying third parties to extract electronic health record (EHR) data, inconsistent adoption of standard nomenclatures and mapping terms across hospital systems, and complexity of the query necessary to generate the measure. Moreover, disagreement persists regarding threshold values that should be used for hypoglycemia quality measures. A blood glucose threshold of <40 mg/dL is recommended by NQF for adverse event reporting in clinical trials, whereas a threshold of <54 mg/dL is recommended by the American Diabetes Association (ADA).¹³

One approach to facilitating implementation of eCQMs involves adoption of a standardized format or common data model, which uses normalized databases and standard vocabularies to represent clinical conditions, laboratory results, medications and other information, in addition to shared query tools among facilities to generate summary data. In this report, we describe the development and validation of electronic queries to measure inpatient hypoglycemia with the NQF 2363-defined blood glucose threshold (<40 mg/dL) and an alternative threshold (<54 mg/dL) using the Patient Centered Outcomes Research Network (PCORnet) common data model.^14,15 PCORnet is comprised of clinical data research networks representing over 300 healthcare-delivery systems that participate in multi-network studies using shared informatics and governance structures.¹⁶ Electronic queries in this study were tested in three medical centers with PCORnet normalized databases and standard vocabularies, validated through manual chart review, and refined based on feedback from hospital sites. The PCORnet data tables from all three hospitals were used to identify electronic medication administration record (eMAR) data to capture anti-diabetic drug (ADD) information, and laboratory test data to capture blood glucose results. For this project, we augmented the common data model with a bed information table to track patient movement in the hospital.

Results from this study in three tertiary care hospitals demonstrate that fully electronic queries based on a clinical quality measure can accurately identify inpatient hypoglycemia events and rates. The queries demonstrated very good performance in identifying the quality measure numerator (hypoglycemia events) and the quality measure denominator (ADD days). Facility-wide and unit-specific hypoglycemia rates showed statistically significant variation across hospitals and comparable intensive care units.

While highly accurate, instances of misclassification occasionally occurred with the queries. Misclassification primarily resulted from failure of the queries in detecting all instances where insulin was administered by continuous infusion (insulin drips). Further investigation at Hospital A showed that this related to an underlying database issue that flagged insulin administration only on infusion initiation or a change in infusion rate, and intermittent and irregular documentation of insulin infusions in the EHR, which translated to less than up-to-the-minute capture of insulin administration in the PCORnet data warehouse. Insulin infusion is a recommended strategy for glycemic management in critically ill patients.²³ Ensuring accurate capture of these types of medication administration events will be important for quality measurement and improvement efforts, especially for intensive care units. Query misclassification from failed detection of insulin drips could bias hospitals with poor capture of insulin infusion information to falsely low hypoglycemia rates relative to hospitals where ADD administration and days are captured accurately.

There were 15 instances (11 eCQM-/MCR+ and 4 eCQM+/MCR-) in which misclassification was not due to failed detection of insulin drips; all in Hospital C. Reasons for these remaining discrepancies likely included incomplete capture of EHR data in the research database or chart abstraction errors. Database errors may have been due to inaccurate information in the hospital clinical data warehouse (e.g., time stamp error) or inaccurate or incomplete information in the PCORnet data warehouse due to errors during the extract-transform-load (ETL) process (e.g., date-time shifting).

Although we undertook extensive review of manually-entered data to address potential chart abstraction errors, errors on chart review (e.g. date-time errors) or REDCap data entry may still have been possible. Indeed, we found that the manual chart review process was prone to error, with more than 60 instances of chart abstraction or data entry inconsistencies that were identified and subsequently corrected during adjudication with query results. The chart abstraction errors we identified underscore the fallibility of manual methods for surveillance and the importance of validated electronic approaches for hospital surveillance data.²⁴

A blood glucose threshold of <54 mg/dL recommended by the ADA¹³ produced three to four-fold higher measurements of inpatient hypoglycemia compared to the quality measure-defined blood glucose threshold of <40 mg/dL in the three participating hospitals. Given that even blood glucose <70 mg/dL can indicate clinically important hypoglycemia,²⁵ quality improvement at higher thresholds of hypoglycemia should be considered. Additionally, recent data suggest that blood glucose values between 45 mg/dL and 93 mg/dL on the last day of hospitalization are associated with higher rates of 30-day readmission (<93 mg/dL) and post-discharge mortality (<45 mg/dL and <67 mg/dL),²⁶ underscoring the importance of surveillance and quality measurement at more than one blood glucose threshold.

Substantial variation in inpatient hypoglycemia rates was found among the three participating sites, and among comparable intensive care units between sites. Such variation indicates potential opportunities for improvement through inter-institutional rate comparisons (benchmarking). It is unclear whether this variability is due to differences in quality of care, or differences in patient case-mix (e.g., patient age or renal dysfunction) and severity of illness. Risk adjustment, similar to what is utilized for reporting and benchmarking healthcare-associated infections in NHSN, could be considered to adjust for variability in patient populations across hospitals.²⁷

After this study was initiated, NQF 2363 hypoglycemia measure underwent minor re-specifications by CMS.²⁸ The newer proposed measure is an incidence proportion of patients with hypoglycemia <40 mg/dL who received an ADD within 24 hours not followed by a blood glucose result >80 mg/dL within 5 minutes among all hospital inpatient admissions. In contrast, the NQF-measure is an incidence rate of hypoglycemic events over ADD days, rather than patient-admissions, and allows for repeat events >20 hours apart. The newly proposed measure also does not distinguish between short- and long-acting ADD preparations. We successfully developed fully electronic queries that accurately counted hypoglycemic events and ADD days according to current NQF criteria, indicating that implementation challenges can be overcome from the technical perspective. Identification of repeat hypoglycemic events is especially important since prior episodes can portend future episodes,²⁹ and should trigger programmatic review.³⁰ Consideration should be given to measurement of repeat hypoglycemic events since these events appear fully amenable to electronic capture. Additionally, our findings demonstrated variability in hypoglycemia event rates among different patient care units. Similar to other hospital-acquired adverse events, such as healthcare-associated infections,³¹ unit-level measurements and comparisons will be important for benchmarking medication-related hypoglycemia.

Our approach to electronic capture of hypoglycemic events has some limitations. First, the quality measure on which the electronic query was based may not perfectly distinguish between iatrogenic and non-iatrogenic hypoglycemia. Other conditions (e.g., shock, liver failure) likely contributed to hypoglycemia in some patients, particularly during the last 48 hours of life; however, ADDs were administered before these hypoglycemic events and likely were contributory. Second, two of three facilities in our study did not have LOINC terms for laboratory tests, and not all local test names definitively identified specimen source. A manual process was required to verify which glucose results were blood glucose results. This is largely a reflection of lack of standardization in laboratory test terminology across EHRs; however even with use of standard vocabularies, like LOINC, validation to ensure that LOINC mappings are inclusive of specimen source, such as urine or blood, may still be needed. Third, we chose not to have more than one independent manual chart review at each site to compute inter-annotator agreement; however, we did perform a third round of manual chart review for circumstances of disagreement between data entry based on chart review and electronic query results.^19,24 Fourth, despite multiple efforts to resolve discordant data between manual chart abstraction and the electronic query, there remained a subset of discrepancies for which the origin could not be fully identified. However, the absolute number of these events was low, representing only 1.1% of sampled inpatient days and all occurring at Hospital C. Discrepant data at Hospital C included uncommon episodes of ADD administration present in the EHR but not captured in the PCORnet data warehouse; the converse was also present, but rare. It may have been possible to reload Hospital C data to the research warehouse, but this would not have represented a real-world application of the electronic query. While we hoped for complete data capture from all three research data warehouses, having a small percentage of missing values was consistent with our expectations. We believe these findings are representative of real-world conditions and reinforce the need for validation of data capture in future work. Lastly, our approach is not intended to generate real-time data, which are helpful for acute clinical care interventions by diabetes programs (e.g., identification of patients for whom expert input regarding ADD dosing is necessary). However, electronic surveillance of hypoglycemia quality measures does not preclude simultaneous use of other methods (e.g., EHR-generated alerts to the endocrine service) to support immediate clinical interventions. Despite these limitations, we demonstrated that fully electronic queries for measuring NQF 2363 were highly accurate in capturing inpatient hypoglycemia events and rates across hospitals that mapped their data to an augmented PCORnet common data model.

Use of common data models has the potential to foster standardized and efficient capture of electronic quality measures across the United States and elsewhere. Further work on electronic hypoglycemic measurement could involve reuse of our query in additional facilities that either share common data models or vendor systems, and testing in facilities outside of PCORnet. This would allow for more opportunities to improve the current query (e.g., better capture of insulin drips), and greater understanding of the variability of hypoglycemia rates across hospitals and unit types. Future work could also further address the epidemiology of hypoglycemic events at the level of the individual event or through an ecologic analysis. For example, factors associated with hypoglycemia can be elucidated, which could inform predictive models to determine expected event rates; such work could be facilitated through a common data model. Additionally, a quality measure for inpatient hyperglycemia (NQF 2362), has also recently undergone re-specification by CMS.¹² Future work could similarly address electronic capture and validity of this hyperglycemia measure. Moving forward, quality improvement efforts should include both measures so that initiatives to reduce hypoglycemic events do not result in increased hyperglycemic events.

In summary, fully electronic queries were highly accurate in capturing inpatient hypoglycemia events and rates across hospitals with different EHR systems that mapped their data to an augmented PCORnet common data model. Such interoperability has the potential to extend hypoglycemia measurement to additional hospitals. More widespread implementation of the NQF hypoglycemia measure can inform patient safety programs within institutions, provide opportunities for benchmarking across institutions, and contribute towards decreasing harmful and potentially fatal hypoglycemic events in hospitals.