The advent of universal health care coverage in the United States and the use of electronic health records can make the medical record a disease surveillance tool. The objective of our study was to identify criteria that accurately categorize acute coronary and heart failure events by using electronic health record data exclusively so that the medical record can be used for surveillance without manual record review.
We serially compared 3 computer algorithms to manual record review. The first 2 algorithms relied on ICD-9-CM (
Cohen’s κ for the initial algorithm was 0.47 (95% confidence interval [CI], 0.41–0.54). Cohen’s κ was 0.61 (95% CI, 0.55–0.68) for the second algorithm. Cohen’s κ for the third algorithm was 0.99 (95% CI, 0.98–1.00).
Electronic medical record data are sufficient to categorize coronary heart disease and heart failure events without manual record review. However, only moderate agreement with medical record review can be achieved when the classification is based on 4-digit ICD-9-CM codes because ICD-9-CM 410.9 includes myocardial infarction with elevation of the ST segment on ECG (STEMI) and myocardial infarction without elevation of the ST segment on ECG (nSTEMI). Nearly perfect agreement can be achieved using IMO interface terminology, a more detailed coding system that tracks to ICD9, ICD10 (
Medscape, LLC is pleased to provide online continuing medical education (CME) for this journal article, allowing clinicians the opportunity to earn CME credit.
This activity has been planned and implemented in accordance with the Essential Areas and policies of the Accreditation Council for Continuing Medical Education through the joint sponsorship of Medscape, LLC and Preventing Chronic Disease. Medscape, LLC is accredited by the ACCME to provide continuing medical education for physicians.
Medscape, LLC designates this Journal-based CME activity for a maximum of 1
All other clinicians completing this activity will be issued a certificate of participation. To participate in this journal CME activity: (1) review the learning objectives and author disclosures; (2) study the education content; (3) take the post-test with a 70% minimum passing score and complete the evaluation at
Upon completion of this activity, participants will be able to:
Analyze recommendations regarding surveillance systems for cardiovascular disease
Compare electronic with manual health records in identifying cases of cardiovascular disease
Assess the accuracy of electronic electrocardiogram data in identifying cases of myocardial infarction
Distinguish strategies to improve the accuracy of electronic records in the surveillance of cardiovascular disease
Rosemarie Perrin, editor; Caran Wilbanks, editor,
Charles P. Vega, MD, Health Sciences Clinical Professor; Residency Director, Department of Family Medicine, University of California, Irvine. Disclosure: Charles P. Vega, MD, has disclosed no relevant financial relationships.
Disclosures: Thomas E. Kottke, MD, MSPH; and Courtney Jordan Baechler, MD, MCE, have disclosed no relevant financial relationships.
Affiliations: Thomas E. Kottke, HealthPartners Institute for Education and Research, Minneapolis, Minnesota; Courtney Jordan Baechler, Department of Medicine, School of Public Health, University of Minnesota, Minneapolis, Minnesota.
Surveillance to track the incidence, prevalence, and treatment of disease is a basic function of public health. Medical records can be used as surveillance tools, and the expense of manual record review could be avoided if medical records were representative of populations and if data in the medical record were standardized. Because the Patient Protection and Affordable Care Act will provide universal health care coverage for Americans (
We are developing a decision-support tool that will identify opportunities for preventing heart disease and for improving outcomes for patients with heart disease (
The first iteration of the computer algorithm was approved on November 20, 2008, by the HealthPartners Research Foundation Institutional Review Board as protocol no. 08–093, and the next 2 iterations were approved on September 21, 2009, as part of protocol no. 09–023. We completed the analysis on February 29, 2012.
The algorithm classifies coronary heart disease and heart failure events into 6 types: 1) acute myocardial infarction with ST segment elevation (STEMI) on electrocardiogram (ECG), 2) acute heart failure with a left ventricular ejection fraction at or below 35% (systolic heart failure), 3) acute myocardial infarction without ST segment elevation (nSTEMI) on ECG, 4) unstable angina or other acute ischemic event (unstable angina), 5) incident diagnosis made in the ambulatory setting (ambulatory presentation), and 6) chronic prevalent disease without an acute event during the period of interest (chronic disease). The classification system is hierarchical: a patient with a STEMI complicated by acute heart failure would be classified as having a STEMI. Likewise, a patient with acute heart failure, elevated levels of troponin (a protein that is present in blood during acute myocardial infarction), and an ECG that did not show a STEMI would be classified as having acute heart failure. The diagnoses are presented in hierarchical order in the first column of the
| Event Type | Manual Review Criteria | First Iteration of Computer Algorithm (N = 254) | Second Iteration of Computer Algorithm (N = 245) | Third Iteration of Computer Algorithm (N = 184) |
|---|---|---|---|---|
| 1. STEMI | Hospital discharge code | Hospital discharge code 410.0–410.9 plus ≥1 troponin level > ULN; statements on ≥1 ECG indicate acute ST elevation consistent with acute MI; n = 64 when combined with nSTEMI category. | Hospital discharge code 410.0–410.6 or 410.8 OR code 410.7, 410.9, and any MUSE | Any hospital discharge code starting with 410.0–410.6, 410.8, or IMO term 410.90CP, 410.90CS, 410.90CU, 410.90CX, 410.90DV, 410.90FZ, 410.90GB, 410.90GD, 410.91D, 410.92F, or 410.92S; n = 50. |
| 2. Acute heart failure with symptomatic depressed left ventricular contractility | Hospital discharge code 425 or 428 as primary diagnosis and EF is ≤35%, BNP > ULN if measured, and dyspnea, pleural effusions, or other symptoms and signs are reason for hospitalization. | Hospital discharge code 425 or 428 as primary diagnosis plus BNP > ULN. Troponin may be positive; n = 56. | Hospital discharge code 425 or 428 as primary diagnosis plus BNP > ULN and EF ≤35%. Troponin may be positive; n = 21. | Hospital discharge code 425 or 428 as primary diagnosis plus BNP > ULN and EF ≤35%. Troponin may be positive; n = 34. |
| 3. nSTEMI | 410.7 or 410.9 or 420–429 and ≥1 troponin level >3 × ULN; and clinical history consistent with acute coronary syndrome. Serial ECGs do not show evolving ST elevation by visual inspection OR diagnosis of nSTEMI on discharge summary. | 410.7–410.9 or 420–429 plus ≥1 troponin level > ULN and no ECG codes with evolving ST elevation; n = 64 when combined with STEMI category. | Hospital discharge code 410.7 or 410.9 without any MUSE | Any hospital discharge code starting with 410.7, or IMO |
| 4. Unstable angina | Hospitalized with code 411.x or 786.5 and clinical history consistent with heart disease problem, and no troponin level >3 × ULN. | Code 411.x and no troponin level > ULN; n = 70. | Hospital discharge code 411.x or 786.5x and no troponin level >3 × ULN. No procedure codes for insertion of pacemaker or defibrillator (37.80–37.99; CPT-4 33216); n = 50. | Category deleted by authors |
| 5. Ambulatory presentation | Clinical history documents first episode of heart disease | No nonelective hospitalization at time of diagnosis. ≥2 years of observation before first diagnosis; n = 39. | First claim for heart disease | Patient has heart disease diagnosis at any time. No nonelective hospitalization for heart disease |
| 6. Chronic prevalent disease | Heart disease | ≥2 years of observation with codes 412–414 or 420–429 in first year of enrollment; n = 25. | ≥2 years of observation with heart disease | |
| Cohen’s κ (95% confidence interval) | NA | 0.47 (0.41–0.54) | 0.61 (0.55–0.68) | 0.99 (0.98–1.00) |
Abbreviations: STEMI, myocardial infarction with ST elevation; ULN, upper limits of normal; ECG, electrocardiogram; MI, myocardial infarction; nSTEMI, myocardial infarction without ST elevation; EF, ejection fraction; BNP, B-type naturetic peptide; CPT, common procedural terminology; NA, not applicable.
Categories are hierarchical with 1 being the highest. The number of cases identified by the computer algorithm is provided in each cell.
All codes are from
MUSE Cardiology Information System (
Intelligent Medical Objects, Inc (
Codes 410–414, or code 425 or code 428.
For the first iteration of the computer algorithm we used medical record data from 254 HealthPartners Medical Group (HPMG) patients who were also insured by HealthPartners. These patients were aged 40 to 74 years and had been treated for coronary heart disease or systolic heart failure from August 8, 2007, through July 31, 2008. August 8 was chosen as the start date because that was the day that the hospital ECG data file and the outpatient ECG data file were combined into a single file. To be included, patients needed to be HPMG patients so that we had access to their clinical records, and they needed to be insured by HealthPartners so that we had access to the ICD-9-CM diagnostic codes associated with their claims data. We used the computer algorithm to select patient records of coronary heart disease and heart failure events from each of the 6 categories. Each of the records was randomly assigned to a cardiologist (T.E.K. or C.J.B.) for manual record review, and 48 records were reviewed by both cardiologists. The cardiologists agreed on the diagnosis for 47 of the 48 patients. We sampled patient records until we had at least 30 verified cases of coronary heart disease and heart failure in each diagnostic category or had exhausted potential records for analysis. This process resulted in more than 30 cases identified as fitting some of the 6 diagnostic categories. The initial computer algorithm is presented in column 3 of the
After an ECG is interpreted by a physician, the text interpretation is converted to numerical codes and stored in an electronic database. During the first iteration, it was not possible to search the database for ECGs that had codes consistent with a STEMI. Software that supported search for codes became available when we were working on the second iteration, so we added ECG computer codes to the algorithms for STEMIs and nSTEMIs. As described in column 4 of the Table, we added ICD-9-CM 786.5 to the screening codes for unstable angina and excluded patients who had been admitted for pacemaker or defibrillator placement or revision. For the diagnosis of ambulatory presentation we required that the first insurance claim for heart disease was from an office visit, that there were no prior insurance claims for heart disease, and that there were 2 or more years of observations in the medical record. For chronic prevalent disease we required 2 or more years of observation with heart disease diagnosed before the period of interest and no hospitalization for heart disease or chest pain during the period of interest.
We analyzed 245 cases to test the second iteration of the computer algorithm. We based the second iteration on data from HPMG patients who were also insured by HealthPartners and were treated by HPMG from January 1 through December 31, 2010. We used the second iteration to draw random samples for each diagnostic category, and 2 nurses and a physician reviewed the medical records. As with the first iteration, we calculated Cohen’s κ and revised the computer algorithm in response to post hoc analysis of the discordant cases.
We analyzed 184 records for the third algorithm, 50 each for STEMI, nSTEMI, and chronic prevalent disease and 34 for acute heart failure with a depressed ejection fraction. We did not require HPMG patients to be insured by HealthPartners for the third iteration, because we did not need claims data to identify ICD-9-CM codes. However, 87% of the patients in the sampling frame had been insured by HealthPartners at some time, and 57% were insured by HealthPartners in 2010. As with the second iteration, the sampling time frame was from January 1 through December 31, 2010. We modified the computer algorithm for the third iteration (
We made the second modification, eliminating the category of unstable angina, because we found that most patients in this category already have a diagnosis of coronary heart disease. Because there is no compelling evidence that a patient hospitalized for chest pain with normal troponin levels benefits from an angiogram or therapies other than risk factor control, all gaps in their care would be detected during the analysis of patients with chronic disease.
We made the third modification, combining ambulatory presentation and chronic disease into a single category, because pursuit of therapies other than symptom control, daily aspirin, and risk factor control for a patient who has coronary heart disease diagnosed in the ambulatory setting is optional and based on the results of further diagnostic testing.
We compared the classification by the computer algorithm to manual record review in 254 patients in the first iteration. After we developed the computer algorithm for the first iteration, we learned that ECG analysis software used by HPMG at that time did not permit searching for abnormalities across patients. Thus, we could not distinguish between STEMIs and nSTEMIs without manual review of the ECGs. However, when we grouped STEMIs and nSTEMIs together, we achieved 94% (60/64) agreement between the computer algorithm and manual review of the cases that were identified as myocardial infarction by the computer algorithm. Agreement in the other categories ranged from 38% for ambulatory presentation to 52% for systolic heart failure; 46 of the cases selected by the computer algorithm did not fit any of the 6 categories by manual record review. Overall agreement on the first iteration was 57% (Cohen’s κ, 0.47; 95% confidence interval [CI], 0.41–0.54) when STEMIs and nSTEMIs were considered as a single category (
We compared classification with the computer algorithm to manual review in 245 patient records for the second iteration. Agreement was 67% (16/24) for STEMI diagnoses and 100% (21/21) for heart failure diagnoses. Agreement for nSTEMI diagnoses was 78% (39/50) and 38% (19/50) for unstable angina. Agreement was 46% (23/50) for ambulatory presentation and 90% (45/50) for chronic disease diagnoses. Overall agreement was 67% (163/245), and Cohen’s κ was 0.61 (95% CI, 0.55–0.68) (
We compared classification with the computer algorithm to manual review in 184 cases for the third iteration. The only disagreement was a single patient diagnosis identified as an nSTEMI by the computer algorithm and systolic heart failure by the reviewers. Overall agreement was greater than 99%, and Cohen’s κ was 0.99 (95% CI, 0.98–1.00).
By using an algorithm that processes only electronic data from the medical records of patients treated for coronary heart disease or heart failure, we were able to replicate the clinician’s diagnosis with a high degree of accuracy. To distinguish between STEMI and nSTEMI, the algorithm uses highly specific diagnostic statements (
Our study has limitations. The analysis is based on the records of only 1 medical group; other groups may have a different experience, so algorithm performance should be tested before adoption for routine use. The purpose of this study is to determine the best algorithm with which to estimate the number and type of acute events in a population. If the purpose were to identify patients for case management or to possibly change treatment strategies, a final step of manual record review might be indicated.
It appears unlikely that electronic data can be used to distinguish unstable angina pectoris from symptoms of chest pain and dyspnea that are not due to ischemic heart disease. However, we believe that this is not an important task. Given the sensitivity of troponin assays to detect ischemia and the fact that many of the patients who have normal troponin levels while hospitalized with chest pain or dyspnea are already known to have heart disease, it is probably not necessary to identify this class of patients. There is no compelling evidence that angioplasty reduces mortality in patients who have normal troponin levels in association with their chest pain. Proceeding to angiography would still be an option, but in this case angioplasty would be for symptom control, not mortality reduction.
The use of the ECG database to classify patients may deserve further attention. Only 3 of the 16 diagnoses that manual record review confirmed as STEMIs in the second iteration had codes in the ECG database that indicated ST elevation. Conversely, most of the ECGs with ST elevation codes were not associated with acute myocardial infarction. Using a combination of codes that indicates ST elevation followed by the appearance of Q-wave codes may be more fruitful, but if rescue angioplasty prevents the development of Q-waves, even this strategy would not detect STEMIs.
Our effort to classify events without manual record review is not unique. Natural language processing has been used to assess quality of care for acute myocardial infarction (
It may not seem appropriate to accept the clinical diagnosis as the gold standard for acute myocardial infarction. Ideally, criteria would be based entirely on objective evidence. However, several points must be acknowledged. First, our goal with the project is to develop a method by which the clinician’s diagnosis can be identified from electronic media with the granularity that is necessary to assess whether the patient received all of the evidence-based care that was indicated. We have shown that this is possible. Whether the clinician made the correct diagnosis is a different question. Second, all criteria depend to some extent on diagnostic coding, interpretation of laboratory data collected, and subjective interpretation of patient symptoms. Third, the development of a universal definition of myocardial infarction (
In 2007 the American Heart Association issued a scientific statement outlining the essential features of a surveillance system that would support the prevention and management of heart disease and stroke (
The following provided funding for this study: HealthPartners Research Foundation (a partnership grant to T.E.K.), the Heart Disease and Stroke Prevention Unit at the Minnesota Department of Health from a Capacity Building — Cooperative Agreement grant from CDC no. 5U50DP000721-04, and National Institutes of Health training grant no. T32 HL69764 (supporting C.J.B.).
The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.
To obtain credit, you should first read the journal article. After reading the article, you should be able to answer the following, related, multiple-choice questions. To complete the questions (with a minimum 70% passing score) and earn continuing medical education (CME) credit, please go to
You are part of a task force trying to implement the 2007 American Heart Association recommendations for implementation of a surveillance system for cardiovascular disease. Which of the following statements regarding these recommendations is most accurate?
The focus of the surveillance should be limited to local populations only
All counties and ethnic subgroups should be represented equally in the sample
Systems should be in place to link healthcare data systems with the surveillance program
Cost should not be a factor in the implementation of a successful surveillance program
You want to set up the cardiovascular disease surveillance program using electronic health records. Which of the following statements regarding the accuracy of electronic vs manual records in the current study is most accurate?
Even basic electronic health records correlated well with manual records
Only electrocardiogram (ECG) data helped to correlate electronic health records with manual records
Agreement between electronic and manual health records was largely dependent upon modifications to the original iteration
There was no substantial agreement between manual and electronic health records in any study analysis
Your electronic health record contains an ECG analysis system. What should you consider regarding ECG data and myocardial infarctions in the current study?
The electronic ECG analysis was only effective in differentiating ST-segment elevation myocardial infarctions (STEMIs) from non-STEMIs (NSTEMIs)
The ECG system was effective in achieving agreement in the overall diagnosis of myocardial infarction
The ECG system effectively achieved agreement regarding myocardial infarction and the presence of ST elevation
The first iteration of the current system failed to identify myocardial infarction in all analyses
What else should you consider regarding electronic data in the current study?
Adding a more specific ICD-9 code with a fifth digit improved the accuracy of the electronic health record
The electronic health record outperformed the manual record in differentiating causes of chest pain
Most ECGs in the database with ST elevation were associated with diagnosis of acute myocardial infarction
STEMIs were identified to similar degrees in the manual and electronic health records
|
| ||||
|
|
| |||
| 1 | 2 | 3 | 4 | 5 |
|
| ||||
|
|
| |||
| 1 | 2 | 3 | 4 | 5 |
|
| ||||
|
|
| |||
| 1 | 2 | 3 | 4 | 5 |
|
| ||||
|
|
| |||
| 1 | 2 | 3 | 4 | 5 |