Because we linked our site’s NEISS cases to data from the hospital’s electronic medical record (EMR), Institutional Review Board (IRB) approval was obtained through a full human subject protection review by the Nationwide Children’s Hospital IRB. Written consent was not obtained, and it was waived by the approving IRB.

The NEISS database is maintained by the Consumer Product Safety Commission (CPSC) and collects data from a stratified, national probability sample of approximately 100 US emergency departments (EDs) [10]. NEISS coders review electronic medical records (EMRs) from the ED and enter visits meeting the inclusion criteria into the NEISS database [22]. The NEISS has its own unique coding system, and injuries are assigned both a diagnosis code and body part code [22]. The NEISS has monitored product-related injuries since 1971, but was expanded in 2000 to monitor all injuries, not just those related to specific consumer products [10], [23]._ENREF_22 The NEISS All Injury Program (NEISS-AIP) uses a subsample of the NEISS participating emergency departments for its data collection. We evaluated data collected at Nationwide Children’s Hospital, one of 66 NEISS hospital EDs that participates in the NEISS-AIP [23]. As a consequence of its role as a NEISS-AIP site, this study includes data on all injuries, not just consumer product-related injuries.

The NEISS-AIP is a collaborative effort between the CPSC and the National Center for Injury Prevention and Control, the Centers for Disease Control (CDC) [23]. The CDC excludes non-injury cases in its NEISS-AIP public release data files and Web-based Injury Statistics Query and Reporting System (WISQARS). The site-specific NEISS data used in this study was obtained directly from the CPSC and contained additional data not available in the NEISS-AIP public use data files and WISQARS estimates because CDC excludes non-injury cases in their public release of the data (personal communication, July 2013) [24]. For this reason, the research team did its own review to identify non-injury cases. Specifically, the NEISS-AIP public use data files exclude cases which are included in our NEISS dataset obtained from the CPSC including “illness,” “psychological harm only,” “contact dermatitis…associated with exposure to consumer products… or plants,” “pain symptoms…[without an injury-related diagnosis specified],” “visit [for] adverse effects of therapeutic drugs or of surgical and medical care,” unknown diagnoses, and deaths occurring upon arrival or in the ED [23].

A random sample of 10% of each month’s NEISS cases seen at the study hospital between January 1, 2008 and December 31, 2008 was selected (n = 3,000). These NEISS records were then linked with their corresponding EMR using the date of treatment (+/– 1 day) and the patient medical record number (matched n = 2,992). Because the treatment date was loosened to +/– 1 day during the match, some EMR entries were inappropriately matched to a NEISS record if a patient visited the ED on two consecutive days. These inappropriately matched cases were removed after manually reviewing the EMR. Additionally, off-site urgent care records (which are not entered in the NEISS but were in the EMR data) and follow-up visits were eliminated. Children were defined as ≤18 years old; consequently, 48 records were eliminated because the patients were ≥19 years. The final database contained 2,890 matched cases.

Drawing upon the review methodology used in a previous NEISS and ICD-9-CM comparison TBI study, [11] the EMR for each case was reviewed separately by two research team members trained in ICD-9-CM coding. Each researcher assigned one final ICD-9-CM diagnosis code for an individual case. Multiple injuries, burns, and poisonings required additional consideration to select the most serious injury and/or appropriate final ICD-9-CM code.

The benchmark ICD-9-CM standard definition of injury used in this study drew upon the

ICD-9-CM injury codes used in the Barell matrix, a current standard for injury data collection, analysis and presentation [16], [17]. Following the Barell matrix, this study considered all diagnoses within the 800-999 range as injuries, with the exception of 909(.3,.5): “Late effect of complications of surgical and medical care, late effect of adverse effect of drug, medicinal or biological substance”; 995(.0-.4,.6,.7,.86,.89): “Certain adverse effects, not elsewhere classified”; 995.9 “Systemic inflammatory response syndrome” and 996-999: “Complications of surgical and medical care, not elsewhere classified” [25]. Cases not assigned a final code falling within this injury definition were considered non-injuries. Table 1, column 3 presents the ICD-9-CM codes included within the benchmark injury definition used in this study.

If the case was determined to be a “non-injury” by ICD-9-CM standards, no code was assigned. The codes assigned by the two research team members were compared, and codes that did not agree within three digits were re-reviewed together by the research team before assigning a final ICD-9-CM code or classifying the case as a non-injury. An ICD-9-CM coding manual and software were used [25], [26].

Statistical analyses were conducted using SAS 9.3 (SAS Institute, Cary, NC). The percentage of non-injury cases in the NEISS sample was calculated by comparing the final ICD-9-CM code assigned by the research team against this study’s ICD-9-CM benchmark injury definition. Additionally, the percentage agreement and kappa coefficient for agreement were calculated to compare the injury versus non-injury classification of each case according to the research team-assigned ICD-9-CM code and the hospital-assigned ICD-9-CM code. Codes with the same first three digits “agreed.”

The proposed matrix was evaluated by calculating a cross-tabulation of the injury type classification under the ICD-9-CM system by the injury type classification under the NEISS (non-injuries excluded). Based on this cross-tabulation, a kappa coefficient was calculated. Using the entire sample (injury and non-injury cases), percentage differences between the injury type distributions of the two systems were calculated as the NEISS proportion minus the ICD-9-CM proportion.

Of the 3,000 NEISS cases originally selected for this study, 2,890 cases were non-duplicate pediatric cases that were able to be linked to their original medical record. Of the 2,890 cases reviewed, 2,505 (86.68%) were injuries based upon the research team-assigned ICD-9-CM code. The remaining 385 cases (13.32%) were determined to be non-injuries because they did not meet this study’s ICD-9-CM benchmark injury definition. An additional analysis of all of the cases classified as non-injuries by the ICD-9-CM standard revealed that 93.5% (n = 360 out of 385) of all non-injuries were coded with at least one NEISS product code. Sports or recreation-related codes (n = 72), diapers (n = 66), medical equipment (general, n = 61), liquid drugs (excluding aspirin, aspirin substitutes, iron preparations and antihistamines, n = 46), motor vehicles or parts (licensed, four or more wheels, n = 22), and other drugs or medications (n = 20) were the products most frequently related to ICD-9-CM non-injuries. (For specific codes used, see Table S2.)

To gain an understanding of case ascertainment under different coding systems, we examined the sensitivity of injury identification using three different diagnosis codes: the NEISS diagnosis code, the research team-assigned ICD-9-CM codes, and the hospital-assigned ICD-9-CM codes. Because the sample was gathered from the NEISS, all 2,890 cases were injuries according to the NEISS. The research team identified the next greatest number of injuries (n = 2,505), and the hospital coders identified the least number of injuries (n = 2,121). Of 2,890 NEISS cases, 769 (26.6%) did not have a hospital-assigned ICD-9-CM code meeting the benchmark injury definition.

Given the variations in injury identification, the relationship between the research team-assigned and hospital-assigned ICD-9-CM codes was further explored. All 2,121 cases that were injuries according to the hospital code were also identified as injuries by the research team; however, the research team found an additional 384 injuries. The kappa coefficient between the hospital injury classification (i.e., injury or non-injury) and the research team classification was 0.60 (95% CI: 0.56-0.63). More specifically, the percentage agreement between the research team-assigned ICD-9-CM code and any of the ICD-9-CM codes assigned by the hospital coders was 77.41% (1,939 of 2,505 cases).

The main objective of this study was to develop and evaluate a bridging matrix between the ICD-9-CM and NEISS coding systems. Using the matrix and injury type classifications described in the methods (see Table 1), we calculated a cross-tabulation of the injury type according to the ICD-9-CM classification by the injury type according to the NEISS classification. Only injury cases were used in this analysis (n = 2,505) (Table 2). Among the injury cases, overall agreement for injury type between the two classification systems was high (κ = 0.87, 95% CI: 0.85–0.88), although some discrepancies existed among certain injury type categories. Overall totals for soft tissue injuries were similar for both the ICD-9-CM (n = 292) and NEISS systems (n = 295), but only 231 cases were soft tissue injuries under both (Table 2). Many injuries coded as soft tissue injuries by the NEISS were classified as other or unspecified injuries under the ICD-9-CM system (n = 42), while some cases coded as soft tissue injuries under ICD-9-CM were also coded as other or unspecified (n = 26) injuries under the NEISS. Similarly, totals for other or unspecified injuries were similar under the ICD-9-CM (n = 450) and NEISS systems (n = 456), but only 360 overlapped. The ICD-9-CM code classifies many NEISS “other” injuries as TBI (n = 18), soft tissue (n = 26), and open wound or amputation (n = 28) injuries. The NEISS classifies many ICD-9-CM “other” injuries as soft tissue (n = 42) and sprain or strain (n = 20) injuries (Table 2).

Because the cross-tabulation and agreement analysis was limited to injury cases, the percentage differences between the type of injury distributions of the two classification systems were calculated to provide a comparability measure for the total sample (injury and non-injury cases, n = 2,890) (Table 3). Percentage differences (the NEISS minus the ICD-9-CM percentages) were minimal for most injury types, with the exception of open wound or amputation (–1.08%), poisoning (2.63%), and other or unspecified (9.27%) cases. Under the NEISS categories of poisoning and other or unspecified, there were significant numbers of ICD-9-CM non-injuries (e.g. adverse reactions to pharmaceuticals and medical devices, dermatitis, and conjunctivitis). The matrix analysis presented in Table 2 shows that, among injury cases, open wound or amputation injuries are coded more frequently under the ICD-9-CM system, and that the NEISS categorizes many of these injuries as other or unspecified (Table 2, n = 28).

Our study has some limitations. Like all chart reviews, [6] assignment of an accurate ICD-9-CM code depended upon details available in the EMR. Because code assignment from medical records is sometimes subjective, [30] this study attempted to mitigate subjectivity by having two researchers review each case separately and assign a diagnosis code. A certain level of inter-rater agreement (a recommended practice for chart reviews) was required, [30] and questionable cases were discussed as a team. Also, using the research team’s ICD-9-CM codes instead of the hospital-assigned ICD-9-CM codes provided an additional quality control. Additionally, although ICD-9-CM codes are still used to code billing data in the US, the US is scheduled to switch to ICD-10-CM coding in October 2014 [31]. _ENREF_20

Although quality reviewers at the CPSC code cause of injury for NEISS data in a fashion that is “consistent” with ICD-9-CM external cause of injury codes (E-codes), [32] this study only examined comparability based upon non-supplementary ICD-9-CM diagnosis codes and NEISS diagnosis codes. Also, while previously used maps provided corresponding combinations of both NEISS diagnosis and body part codes to ICD-9-CM codes, [18]–[21] this study used only the NEISS diagnosis code (except for traumatic brain injuries, which used only the body part equal to the head).

Data limitations may affect the generalizability of results. The data used in this study comes from a single institution. If variations in coding exist across hospitals, then the external validity of these results is limited; however, no current indications suggest that the NEISS coding or design issues identified in this study are particular to our hospital. Additionally, it is unknown if these results are generalizable to adults or non-pediatric EDs.

By proposing and evaluating a relatively simple matrix that bridges NEISS diagnosis codes with ICD-9-CM codes, this study provides a basic framework for conducting standardized injury research. Although comparability was imperfect, the generally favorable matrix evaluation results suggest that the NEISS has good comparability potential. Strategies such as manually reviewing the selected NEISS cases may further improve comparability. Additionally, the identified differences in injury and injury type definitions between NEISS and ICD-9-CM coded data will allow researchers to more accurately interpret NEISS results and pay attention to specific criteria of the NEISS when conducting research.