Every year in the United States nearly 50,000 people die due to suicide and nearly 500,000 present in emergency departments (EDs) for nonfatal self-directed violence (SDV) (also referred to as self-harm or self-inflicted injuries; Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). This public health problem is worsening, as the age-adjusted rate of suicides and nonfatal SDV increased by 33% and 40%, respectively, between 2001 and 2018 (Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). In 2018, suicide was the tenth leading cause of death in the United States (Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). The direct and indirect costs for suicides and suicide attempts in the United States was estimated at $93.5 billion in 2013 (Shepard et al., 2016).

National estimates for nonfatal SDV presenting at EDs are calculated from the National Electronic Injury Surveillance System – All Injury Program (NEISS–AIP). NEISS–AIP is a collaboration between the U.S. Consumer Product Safety Commission (CPSC) and the U.S. Centers for Disease Control and Prevention (CDC) with the purpose of tracking first-time, nonfatal injury-related ED visits on all types and causes of injuries; deaths are excluded from the surveillance system. NEISS–AIP is a nationally-representative sample of 24-hour EDs with at least six beds. NEISS–AIP included 66 EDs when it started in 2000 and has decreased over time as more hospitals have dropped out than have been replaced; the 2018 sample included 59 hospitals.

CPSC and CDC train hospital coders to review ED medical records and abstract the necessary information on all injuries. CPSC manages and cleans the database with support from CDC. Select NEISS–AIP variables are available for querying through the WISQARS^™ website (https://www.cdc.gov/injury/wisqars/index.html) within one year. Within 3–4 years the public version of the NEISS–AIP dataset can be accessed free through the Inter-University Consortium for Political and Social Research website (https://www.icpsr.umich.edu/web/ICPSR/series/198/studies) (Fig. 1).

In 2005, CDC and CPSC added several questions on patient characteristics and event circumstances for all intentional, nonfatal SDV cases captured in NEISS–AIP. The additional questions along with the parent NEISS–AIP constitute the NEISS–AIP SDV surveillance system for purposes of this study.

While aspects of NEISS–AIP and its other special studies have been evaluated in the past (Davis, Annest, Powell, & Mercy, 1996; Jhung et al., 2007; Thompson, Wheeler, Shi, Smith, & Xiang, 2014), NEISS–AIP SDV data have not been evaluated, so little is known about their usefulness. In this project, we evaluated NEISS–AIP SDV in terms of overall quality and utility.

CDC defines a surveillance system as “the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event.” These data are then used to inform prevention efforts “to reduce morbidity and mortality and to improve health” (Buehler, 1998; German, Horan, Lee, Milstein, & Pertowski, 2001; Teutsch & Thacker, 1995; Thacker, 2000). We used a mixed methods design to evaluate the NEISS–AIP SDV as a surveillance system through an assessment of 10 system attributes: usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, predictive value positive, representativeness, timeliness, and stability (Table 1).

We reviewed data entry forms, the coding manual, and training materials to understand how the system functions. To identify strengths and weaknesses of the surveillance system attributes, we interviewed multiple key informants, including CDC users, CPSC managers, and hospital and quality assurance coders of the data. Finally, we analyzed the NEISS–AIP SDV data from 2018—the most recent data year available—to assess data quality by examining the completeness of variables; specific variables included time of arrival at the ED, patient self-reported SDV intent (e.g., intent to die, intent to harm oneself, intent to escape), staff description/diagnosis, patient SDV risk factors (e.g., previous episodes of self-harm, depression, bipolar disorder, anxiety), use of alcohol at time of injury, use of recreational drugs at time of injury, substances used (if poisoning), and final disposition (if admitted or transferred).

NEISS–AIP SDV provides the opportunity to describe systematically collected risk factors and characteristics associated with nonfatal SDV that are not regularly available through other data sources, which, in turn, would be useful for prevention purposes. While this surveillance system has the potential to be useful, this evaluation suggests that there are challenges with many of its system attributes.

The NEISS–AIP SDV surveillance system attributes of simplicity and stability benefit from being a part of the larger NEISS–AIP surveillance system. Another system strength is the focus on medical record review to capture cases, which is likely more sensitive than if the system relied only on administrative codes. Despite these strengths, NEISS–AIP has its limitations, particularly because it is currently reliant on human resources to manually abstract information from ED medical records and then enter the data into the NEISS–AIP data collection system. CPSC is exploring machine learning to help automate data abstraction from electronic medical records, but currently NEISS–AIP is not integrated with other data systems like electronic medical records.

NEISS–AIP SDV has aspects that are timely, including data reporting from the hospitals to CPSC (within a week of the ED visit) and feedback from CPSC and CDC to hospitals flagging certain errors (within about a week). However, it takes nearly a year after the end of the calendar year for the weighted data to be available for internal use at CDC and for select variables to be available to the general public through WISQARS^™. Historically, the publicly available data set for NEISS–AIP was only available after 3–4 years. CDC is in the process of expediting the release of the public dataset that will allow for more timely analysis of the NEISS–AIP data by public health partners. The SDV data captured from the additional questions added in 2005 have not been and currently are not available to the public as we continue to evaluate their utility.

The usefulness of the NEISS–AIP SDV data requires further consideration due to a couple of system challenges. First, medical records that are incomplete or that do not require the same fields as NEISS–AIP SDV leads to unknown values being entered into NEISS–AIP SDV. For example, race is not always included in hospital ED records, which results in this variable frequently being missing and thus limiting the ability to look at associations between SDV and race.

In addition, many of the SDV-specific variables (e.g., patient self-reported intent, staff diagnosis, patient risk factors, BAC, poisoning substances and their respective quantities) use open text fields for large proportions of responses, making data entry time-consuming and data analysis difficult and inefficient. These variables should be examined to determine which questions and responses can be modified to reduce the amount of open-ended responses. For example, the poisoning substances variables could be displayed in a drop-down list by drug/substance class.

In addition to reducing the number of open-ended responses, there could be a review of which NEISS-AIP SDV variables should be maintained—either as is or with modifications—and which could be dropped because they are no longer relevant or because the data can be obtained elsewhere. For example, the NEISS–AIP SDV variable that captures staff description or diagnosis of the SDV injury is important to have a clinical assessment of self-harm intent but may need to be updated to avoid outdated terminology (e.g., suicide gesture) that has been in place since 2005. In addition, in 2019, NEISS–AIP added new variables to its core and modified the medical narrative field to collect more information regarding alcohol use, perhaps allowing for elimination of other alcohol use-related variables.

This evaluation was subject to at least two limitations. First, data were collected primarily through review of manuals and interviews with CPSC and CDC stakeholders. A standardized survey of all members of the NEISS–AIP surveillance team and a systematic review of medical records to validate the NEISS–AIP SDV data would have made this evaluation more robust. However, this was not possible due to time, financial, and planning constraints. Second, sensitivity and predictive value positive were difficult to assess due to a lack of a gold standard.

In summary, NEISS–AIP SDV is a unique surveillance system based on medical record review that collects SDV-related risk factors and characteristics that are not collected in other data sources. With some modifications to data fields and yearly analysis of the additional SDV questions, NEISS–AIP SDV can be a valuable tool for informing suicide prevention.