Non-small cell lung cancer (NSCLC) is the second most frequently-occurring cancer in the United States (240,000 new cases/year) and is the nation’s leading cause of cancer deaths (127,000 deaths/year).^1,2,3 Immune checkpoint inhibitors (ICIs) received Food and Drug Administration (FDA) approval as second-line therapy in 2014 and were approved for first-line use in 2017.⁴ ICIs facilitate an anti-tumor immune response by neutralizing T-cell disactivating “immune checkpoint” proteins, specifically cytotoxic T lymphocyte antigen 4 (CTLA-4)–CD and programmed cell death 1 (PD-1)/programmed cell death 1 ligand 1 (PD-L1), that are commonly activated by tumor cells, impeding endogenous immune detection.^5,6

Beyond intended anti-tumor effects, the immune dysregulation caused by ICIs diminishes self-tolerance and leads to immune-related adverse events (irAE). More than 70% of patients experience these adverse events, and 10% develop severe irAEs.⁷ IrAE can affect every tissue and organ system, although the site, timing, and severity of events differ by the specific ICI molecule and cancer type. Severe irAEs among treated individuals with NSCLC include pneumonitis, colitis, myocarditis, and hepatitis, and often lead to temporary or permanent withdrawal of the ICI therapy.⁸ While irAEs are typically recorded as adverse outcomes in clinical trials of ICI therapies, trials are often short, enroll small number of individuals, and conventionally exclude individuals with underlying autoimmune disorders and individuals with multiple comorbidities who may be at greater risk of severe irAE. Real-world data (RWD), such as billing data or electronic health record data, can complement clinical trial data by including broader populations, more observations, and longer follow-up, which enable more robust measurement of irAE, including among subgroups excluded from clinical trials.

However, because there are no International Classification of Disease (ICD)-10 diagnosis codes for irAE, the occurrence of irAE must be deduced based on other diagnosis and procedure codes, and perhaps pharmacy records and clinician notes, temporally associated with exposure to treatment.^9,10,11 Despite this challenge, RWD sources have been used to assess the benefits and side effects of ICIs, particularly to evaluate patient populations not well represented in randomized controlled trials.^{9,10,12,13,14,15,16} Researchers have primarily used three algorithms to identify irAEs with RWD.^9,12,15 However, no study has assessed the accuracy of these irAE case definitions. We aimed to validate case definitions for irAE in a large health system’s irAE monitoring registry. We validated one existing and four modified irAE case definitions. A valid case definition will allow more reliable future studies of the safety of these therapies.

We conducted a retrospective study using chart review as the reference standard to validate algorithms for identifying irAE. Institutional review board (IRB) approval was received for this study. Informed consent was waived due to minimal risk of harm to subjects, and data were de-identified.

In this validation study, we identified EHR-based algorithms that are sufficiently valid for identifying individuals who had severe irAE (CTCAE Grade 3–5). The algorithms generally performed poorly in individuals with mild/moderate irAE. Differential algorithm validity by irAE severity likely stems from the fact that four of the five algorithms required medication codes, which align with guideline-recommended treatment of severe, but not necessarily moderate or mild irAEs. For this reason, too, the Dx code-only algorithm had a much poorer C-statistic for assessing severe irAE than other algorithm versions, though it appeared sufficient for identifying common, “high-confidence” severe irAE.

For all algorithm versions and irAE severity levels, algorithm PPV was low, indicating that a majority of irAE toxicities identified using the algorithms were not identified as true cases by clinician chart reviewers. This surprisingly low proportion arises from the relatively low incidence of irAE in the validation population and the imperfect specificity of the algorithms (maximum: 90.8% for the most stringent algorithm in one sub-analysis). In the context of real-world clinical management of irAE, the low PPV also reflects the tendency to treat suspected irAE early using guideline-recommended treatments before a definitive diagnosis is made. Many of these may have been later deemed to be due to another cause. This practice was evident in the irAE registry’s irAE ascertainment protocol, whereby evidence of improvement after pharmacologic irAE treatment was taken as indicative that the symptoms had been caused by irAE. It is plausible, then, that the low PPV is an artifact of this clinical uncertainty in the early stages of irAE.

Across the main and sub-analyses, the Rx only algorithm outperformed other algorithm versions as assessed using the C-statistic, including being the only algorithm to exceed 0.70 for the overall analysis. This case definition is relatively simple to implement, and appears sufficient to identify patients with severe irAE, relative to other, more complex case definitions. However, given the range of sensitivity and specificities, the algorithms may have different use cases. First, in a clinical research context, where there is interest in identifying “true” irAE to assess etiology and risk factors, the highest specificity algorithm (Dx/Rx Discontinue) might prove most useful. Second, in health care planning, the algorithm with the highest sensitivity will detect most completely all affected patients. Therefore, Dx only or Rx only depending on irAE severity may be more useful. Third, the Dx-only case definition, having a high sensitivity and high NPV, could be useful to identify patients potentially in need of chart review and outcome adjudication in order to inform potential follow-up care for irAE.²³ Fourth, the irAE case definition algorithms could also prove useful in real-world studies comparing the risks across ICI-containing regimens, across patient subgroups, or to measure changes in risk over time, assuming that outcome misclassification is nondifferential across those settings. An imperfectly calibrated measurement tool, while potentially unsuitable to estimate true incidence rates, may be used to contrast irAE, for example, for calculating risk ratios and other measures of association.

Validated irAE case definition algorithms could also be implemented in predictive models aimed at prognosticating the risk of irAE toxicity in ICI-eligible patients. Such predictive models would be valuable to aid clinical decision-making, such as by informing knowledge of individualized risk-benefit balance of ICI²⁴ or prospectively identifying risk factors for severe irAE.¹⁵ In this context, the specific goal of the prediction study would inform the choice of algorithm; but a more sensitive algorithm may result in patients erroneously being classified as high-risk (disadvantageous when follow-up is invasive and/or expensive), while a more specific algorithm might correctly identify irAE-prone patients at the cost of missing some cases (costly when preventing irAE is of predominating concern).

Our study has several strengths. This is the first study to validate EHR-derived case definitions for irAE, which is important given the growing number of studies using EHR and other RWD to assess this important clinical endpoint. The protocol for ascertaining true cases was rigorous, guideline-based, and implemented by an experienced clinical team.²³ We relied on data fields that are available in administrative claims data, enabling the implementation of these irAE algorithms across other data sources. The study also has limitations. Our validation population included research patients in clinical trials, which may not be generalizable to the broader real-world use population. We also excluded patients with a diagnosis of COVID-19, limiting generalizability in settings where this diagnosis is likely to be prevalent. We used only one health system’s data, which may not generalize to all patients and settings of care. We used diagnosis and medication codes only, as opposed to other EHR-available fields such as laboratory and radiology results for rule-out diagnoses, which may have improved algorithm performance. Unstructured clinical notes, in addition, contain important data that can be parsed by natural language processing models in settings such as this one, where diagnostic uncertainty is high.²⁵ Diagnosis codes alone can be unreliable to ascertain specific conditions, as signs and symptoms are possibly coded differentially depending on the presentation and billable medical care received (e.g., “hypoxic respiratory failure” instead of “pneumonitis”). Diagnosis codes also may not be consistently entered into EHR, leading to under-ascertainment of cases using this field alone. We used EHR medication administration records, which reflect medications administered by a nurse or other provider in an inpatient setting, as opposed to outpatient medications taken at home. This choice of data source may have caused us to miss mild cases that could be treated outside the hospital. Finally, although our EHR-based case definition is inter-operable with claims data, algorithm performance may differ when implemented in claims due to differential granularity of inpatient care records when relying on claims (e.g., due to diagnosis-related group (DRG)-based coding of inpatient admissions).

IrAE case definition algorithms using both diagnosis and medication codes perform best at identifying severe irAEs (CTCAE Grade 3–5), and commonly encountered irAEs such as pneumonitis, colitis, and hepatitis. IrAE thyroiditis, which is common and most often low-risk, appears to be unreliably measured using these algorithms. Ultimately, analyses can be adjusted according to known measurement error in irAE case definitions,^26,27 and algorithms have utility for comparing relative risk of irAE between ICI regimens or between patient subgroups.