The present study used results from three groups of readings. As part of an earlier research investigation, 1401 miners participating in the NIOSH Enhanced Coal Workers’ Health Surveillance Program completed both digital and film chest radiography on the same day (4). For the purposes of that earlier investigation, all the radiographs were independently classified by at least two of eight B Readers, and 172 miners were found to have a profusion of small pneumoconiotic opacities >0/0 by at least one Reader. To investigate a potential effect of image modality on within- and between-reader variability, a follow-up study obtained additional readings for each of the 172 digital and film chest radiographs, using seven of the eight B Readers from the original study (6). Detailed information on study subjects, image acquisition and processing, and image interpretation is available in those previous reports (4,6). The present investigation used the group of 1204 ILO classifications of traditional film-screen radiographs (FSR) obtained during the earlier studies (seven B Readers and 172 miners) and the group of 1204 classifications of digitally acquired chest images performed using the “research” digitized standards (DR_RES image set) described previously. For the purposes of the present study, a third group of classifications was obtained for the 172 digital chest radiographs by the same seven B Readers (1204 interpretations), but now using the ILO Standard Digital Images (2011-D) (DR_ILO2011-D image set). Each B Reader performed the additional classification of the digital images in a manner blinded from their own and other readers’ previous interpretations. In summary, the present study used these three groups of 1204 B Reader classifications (total 3612 observations) to investigate any potential differences in pneumoconiosis outcomes related to the specific set of standard images used. The results of traditional classification, in which both radiographs and ILO standard images were film-based, were considered the “gold standard” and compared to the classifications of the digital images performed using either the DR_ILO2011-D set or the DR_RES standard image set. The central tendency of small opacity profusion, the prevalence of ILO Category 1/0 or greater, and measures of reader variation were assessed using previously described analytic methods (6). Within-reader variability of small opacity profusion category was compared by the standard image set used for classification (FSR vs. DR_ILO2011-D and DR_RES vs. DR_ILO2011-D), using Cicchetti–Allison weighted κ values and 95% confidence intervals (CIs). In addition, small opacity profusion categories classified using the DR_ILO2011-D set were compared to the DR_RES set and FSR by calculating a global interset weighted κ value and Spearman’s correlation coefficient, using all readings in aggregate. Finally, small opacity profusion classifications using DR_ILO2011-D, DR_RES, and FSR were examined using Bowker’s test of symmetry. The SAS statistical software version 9.3 (SAS Institute, Cary, NC) was used for all analyses. Data collection for this study was approved by the NIOSH institutional review board (HSRB 12-DRDS-NR02).

In November 2011, the ILO issued a new edition of the International Classification of Radiographs of Pneumoconioses including a set of digitized standard images (2011-D), which, for the first time, sanctioned the application of the ILO pneumoconiosis classification to digitally acquired chest radiographs. We compared the results of classifications of coal miner digital chest radiographs, interpreted using these new ILO 2011-D standard images, to a “gold standard” traditional classification of the miner’s film-screen radiograph taken on the same day. To reduce reader effects, all seven of the B Readers performed all classifications. The results showed a high degree of concurrence in the recognition of small pneumoconiotic opacities between digitally acquired radiographs interpreted using the current ILO Standard Digital Images (2011-D) and traditional film-screen radiographs interpreted using the ILO film-based standards (Spearman’s correlation coefficient r_s = 0.68). Intrareader variability for the seven readers, measured by the κ statistic, was relatively similar to previously reported values and did not appear to be affected by the use of the new 2011-D standard set. These findings help to validate the use of the newly released ILO digitized standard images (2) for classification of small pneumoconiotic opacities.

Both similarities and differences between FSR and DR imaging modalities reported in earlier comparisons, which used the DR_RES standards, were largely observed in the present study using theDR_ILO2011-D referent standards. As mentioned previously, we observed no major difference between FSR and DR_ILO2011-D. The present findings appear to confirm previous reports that small pneumoconiotic opacities are more often classified as irregularly shaped on digital radiographs compared to FSR. The available research data suggest that the visualization of small linear shadows may be enhanced through the use of DR, although, even for digital images, reader agreement on the shape of small opacities remains low. Clinical experience does seem to confirm a modestly enhanced visualization of linear/irregular small opacities as reported here and elsewhere. Adding credence to this view is our observation that the smallest opacity types (s and p) were recorded significantly more frequently using DR, compared to FSR. However, these are subjective interpretations of the study findings and we are unable to make definitive inferences based on our results regarding whether DR provides enhanced visualization of pneumoconiotic shadows.

We did observe some minor differences between classifications of coal miner digital radiographs using the DR_ILO2011-D referent standards compared to readings using the DR_RES set of standards developed for previous research studies. In general, when the results of the two digital classifications differed, the readings using the DR_ILO2011-D more closely reflected the FSR result than the readings using DR_RES (Tables 2 and 3 and Fig 1).

This study has a number of advantages. It was based on repeat interpretations of a large number of radiographs by seven experienced B Readers who were blinded to their previous readings and the readings of others. The methods in this article are well established and reported in previous peer-reviewed publications. A number of limitations also exist for this study. Figure 3 demonstrates that the seven B Readers have many different patterns of reading. Most of the radiograph interpretations in this study indicated that no disease was visibly present, (Fig 1) which may affect calculations of agreement both within and between readers. Pneumoconiosis classification using digital chest radiographs is a new procedure, and the readers who performed the earlier research readings may have become progressively more familiar with it. Given the time difference between the two groups of digital readings used in this study (at least 2.5 years), some readers may have evolved or modified their natural reading patterns during that time. It is possible that the differences observed between the two digital readings (DR_ILO2011-D and DR_RES) may in part be related to alterations in reading behavior over time.

In conclusion, our results help to validate the classification of digital chest radiographs of pneumoconiosis using the new ILO digitized standard images. The body of scientific research conducted to date supports the use of digital radiography in the recognition of small interstitial lung opacities. The results of the present study demonstrate that, when chest radiographs are obtained and displayed using appropriate equipment, methods, and conditions, the results of classification of digital chest radiographs performed using 2011-D standards should be quite comparable to results based on the previous digitized research standards (DR_RES) and also to the film-based ILO 2000 classification. The current challenge for the ILO is to complete development and testing of a set of reference standards for the classification of digital chest images that are digitally acquired.