The Healthy Directions-Small Business study was a randomized controlled trial that assessed the effectiveness of an integrated cancer prevention program. The worksite was the unit of randomization and intervention with 26 worksites recruited and pair-matched on unionization status (i.e. whether or not they were unionized). One worksite in each pair was randomly assigned to the intervention; the other to the minimal-intervention control arm. [7-9] One worksite dropped out soon after recruitment, leaving 25 worksites at the time of baseline assessments. This study was reviewed and approved by the institutional review boards of both the Dana-Farber Cancer Institute and the Harvard School of Public Health (protocol #98-333 and P10400-109 respectively), and complies with the Helsinki Declaration.

Worksites that were eligible to participate in the study had: 1) between 50 to 150 employees, 2) at least 25% of workers who were first or second generation immigrants, or people of colour, 3) less than 20% turnover rate in the past year, and 4) the ability to decide if they wanted to participate in the study (if part of a national or international company). One hundred and thirty one companies met the inclusion criteria and were invited to participate, and 26 agreed to participate. The 26 participating worksites were largely manufacturing businesses (e.g., medical equipment, dog food, specialty pumps, textiles for the automobile industry, and electronics); 3 provided laundry and printing services to other businesses. There were 96 workers per worksite, on average. More than half of the participants were white, non-Hispanic (60%), and there were significantly fewer women than men; additional demographic and description of worksites is available elsewhere [7-9].

The intervention was 18 months in duration and focused on improving nutrition, physical activity, smoking cessation, and occupational health and safety. The intervention targeted behaviour change by intervening at multiple levels. These levels of intervention were: 1) individual workers (e.g., health education about diet, physical activity, smoking cessation, occupational health and safety), 2) the organization (e.g., worksite food options, lunchtime walking groups, occupational health and safety policies), and 3) the physical environment (e.g., reduction of hazardous exposures) [7,11]. The control sites received a minimal intervention of only smoking cessation programs.

The theoretical basis of this approach has been outlined in detail previously [5]. In the "hierarchy of controls," upstream or source-focussed prevention is the most effective at exposure prevention, and downstream the least [12]. We applied a simplified hierarchy of controls to express a gradient of upstream (materials correspond with source of the hazard) versus midstream (process corresponds with path between source and worker) versus downstream (human interface corresponds with the level of the worker as the receiver of exposure) preventive efforts. This was combined with an examination of the balance between exposure potential and exposure protection at each of these three levels. The resulting Potential and Protection matrix, expressed as a 2 × 3 table, allows both a horizontal (balance of Potential and Protection at each level) and a vertical (degree to which those efforts are focused upstream) assessment of exposure prevention. Previous studies documenting upstream shifts in hazardous substance control efforts, for example in response to toxics use reduction legislation, demonstrate the feasibility of this approach as well as increasing receptivity by employers [15]. Valuing of an upstream focus is further reinforced by the precautionary principle [13,14] as well as analogous principles in other aspects of public health [16].

Six sets of indicator variables (yes/no) were developed to assess exposure Potential and Protection at the Material, Process, and Human Interface levels (detailed in Additional Files 1 and Additional file 2: the SBEI Guide and SBEI Checklist Form). Three potential routes of exposure (inhalation, dermal, ingestion) and a wide range of prevention and control–or protection–methods were assessed. Material indicators include material properties, hazard monitoring, and hazard inventory-keeping. Process level indicators include specific process types, equipment, physical conditions, and engineering and other controls. Human interface indicators include work tasks, work practices, and personal protective equipment (PPE) requirements and use.

The SBEI checklist (Additional File 2) was initially written and pilot-tested in a previous study [5], and was refined for application in a small business manufacturing context for Healthy Directions-Small Business. All baseline walk-through assessments as well as pre-visit contacts, and site visits, were conducted by the same certified industrial hygienist (CIH), with a subset of 36 production areas also assessed by a second CIH to assess inter-rater reliability (Table 1).

Data was collected in hard copy on SBEI Checklist Forms (Appendix B) by the project industrial hygienists during walk-through inspections guided by worksite contacts. One checklist form was completed for each identifiable manufacturing process, area, or group. Groups may include maintenance departments that work in various processes or areas. In the judgment of the inspecting hygienist, processes or groups assessed on each SBEI form constituted similar exposure groups.

The first page of the SBEI checklist (Additional file 2) records general information about each process, such as numbers of workers, general air quality, housekeeping, obvious safety issues, odors, evidence of spills of potentially hazardous substances, and visible evidence of hazardous contaminants. Brief impressions of physical, safety, and ergonomic stressors are also recorded, though not incorporated into the hazardous substance exposure prevention rating process. This is followed by more specific assessments of materials used (Additional file 2, page 2), the process (Additional file 2, page 3), and the human interface (Additional file 2, page 4). Variable numbers of processes were assessed at each site, yielding a comprehensive and systematic assessment of potential for, and protection from, hazardous substance exposures for each worksite.

Our goal in developing the following set of measures and ratings was to provide the basis for an SBEI summary score that preferentially values or rewards material-focused prevention and control, gives medium weight to process-focussed control, and values worker-focused control the least. Accordingly, materials are considered first, followed by process, and finally by human interface. Similarly, at each level (materials, process, human interface), low exposure potential was judged as more desirable than high protection from exposure.

Figure 1 outlines the generation of measures from the walk-through checklist. For each set of checklist indicators (six cells of 2 × 3 matrix), a simple weighting scheme was applied wherein each indicator was designated as a Major, Moderate, or Minor contributor to potential for or protection from exposure (revised from two categories of Major/Minor in the previous version). Indicator information for each cell in the 2 × 3 matrix was then combined to give a rating of High, Moderate, or Low.

A second industrial hygienist also administered the walk-through checklist in 36 production areas across seven study sites for the purpose of evaluating inter-rater reliability (detailed in Table 1, far right column). Both hygienists were involved in the development and the writing of the SBEI Guide (Additional File 1). Assessments were conducted on the same day, within a short time of each other, such that the conditions evaluated were as close to identical as possible. The two hygienists assessed each area independently and did not communicate during assessments.

The individual indicator variables and ratings were tabulated over the departments assessed, with percentages reported. Although production areas were clustered within worksites, we treated the assessment of each production area as an independent measurement for these descriptive analyses.

To assess inter-rater reliability, we first computed the percent agreement for the five Potential/Protection matrix ratings (low/moderate/high), as well as for the overall summary ratings (6-point scale) for each area assessed. Weighted Kappa statistics were then calculated for each of these seven ratings. Standard arithmetic weighting was used for the six 3-point scales and the one 6-point scale evaluated. The Kappa statistic ranges from negative when the raters disagree more than would be expected by chance, to 0 when the amount of agreement is what would be expected by chance, and up to 1 when there is perfect agreement. Landis and Koch suggest the following interpretations for kappa values: k > 0.75, Excellent; 0.40 <= k <= 0.75, Good; 0 <= k < 0.40, Marginal.[17]

The total number of production areas or processes assessed at each of the 25 worksites ranged from 3 to 10, with a median of 6 per site (Table 1). A wide variety of types of processes were assessed, including for examples: packing and promotion (food products), which involved the bulk transfer of material with potential airborne dust or liquid exposures; machining (metal product fabrication) that involved crushing, grinding, and sanding of metal parts with open tanks of fluid and the potential for airborne exposures to dusts and liquids. Some processes assessed had no observable or documented hazardous substance exposures, including for examples: shipping and receiving in a warehouse (plastic products), maintenance (printing services), and quality control (food products).

A wide variety of hazardous substances were captured in rated processes, including carcinogens (e.g., methylene chloride, silica, metal-working fluids), irritants (e.g., acids, nickel compounds), asphyxiants (e.g., carbon monoxide), asthmagens (e.g., epoxies), neurotoxins (e.g., methyl ethyl ketone), and reproductive hazards (e.g., lead, various chlorinated solvents such as trichloroethylene). As noted above, there were also some processes assessed where there were no observable hazardous substances.

Material Potential ratings were generated on the basis of up to 8 hazardous substances observed in a given process. In five areas, however, there were no observable hazardous materials in use (for examples, areas described as 'Inspection' and 'Quality Control'). These areas were automatically assigned a rating of low Material Potential. Thus, one or more hazardous materials were being used in 144/149 areas assessed. In 108 areas, two hazardous materials were being used; three hazardous materials were being used in 68 areas; four hazardous materials were being used in 29 areas; five hazardous materials were being used in 9 areas; six hazardous materials were being used in 4 areas; seven hazardous materials were being used in 3 areas; and eight hazardous materials were being used in 2 areas. As detailed in Table 2, ratings of the indicators for all materials in the area were considered collectively, with the highest rating observed dictating the area rating.

For Material Potential, most areas were rated either low or moderate (80% combined) (Table 7, first row). At the Process level, the majority of areas were rated high for Potential (60%), with smaller proportions rated moderate and low (~40%) (Table 7, second row). This was offset, however, by the majority of areas being rated as having high Process Protection or engineering controls (58%). Human Interface Potential was typically high (46%) or low (38%), with relatively few areas rated moderate (15%). Human Interface Protection, or reliance on personal protective equipment, was almost always rated high (94%).

The definitions and frequencies of SBEI Summary scores are presented in Table 8. In summary, these results suggest that there was a fairly urgent need for improvements in roughly 10% of the areas assessed (scores of 5 and 6). An example in this category was the mixing oven area at a food products site that involved bulk transfer and mechanical mixing of materials (cultured whey, maltodextrin, food starches, and flavoured additives) with high potential exposures to dusts and liquid aerosols, and a reliance on personal protective equipment rather than engineering controls. There was some need for improvements in another 5% (score of 4), that there is still room for improvement–though not urgent–in roughly one on four areas (27% with score of 3). An example in this category was the soldering assembly area at a metal product fabrication site where there was potential exposure to airborne particulate not otherwise classified (NOC), inadequate local exhaust ventilation, and no respiratory protection (though general protective clothing was noted). The exposures were well controlled in the majority of areas assessed (58% of areas with scores of 2 or 1). An example in this category was the quality control (QC) process at a food products site, where there were no observable hazardous substances in use. The third column in Table 8 presents generic intervention recommendations in order of preference. These recommendations reflect the rationale of the rating scheme and encourage upstream over downstream intervention efforts, first emphasizing material factors, then process, with human interface intervention recommended only as a temporary stopgap measure.

The percent agreement and inter-rater reliability of computed ratings are presented in Table 9. In the Potential/Protection matrix, percent agreement in subscale ratings (high/moderate/low) was high (83–89%). Summary score percent agreement was also high (78%). Correspondingly, weighted kappa statistics for subscale ratings were all in good to excellent range (0.67–0.89). Summary score inter-rater reliability was also good to excellent (0.73). The 95% confidence limits for all point estimates all excluded zero (level of agreement that would be expected by chance).

The distributions of ratings showed reasonable discriminatory power of the SBEI exposure prevention rating method, with a general pattern towards low Potential and high Protection ratings, and a distribution of overall SBEI scores that was strongly skewed towards the favourable end. A similar pattern was observed previously in large manufacturing worksites in the Wellworks-2 trial [5]. The frequency of favourable ratings in our sample may be artefactually elevated relative to the full population of manufacturing worksites due to the selection biases inherent in this study. Participating companies had to voluntarily agree to occupational health intervention together with health promotion if they were randomized to the integrated intervention group [7]. Thus companies that have exposure concerns or that do not place a high priority on occupational health would have been less likely to participate. We would also note, however, that many companies that expressed willingness to participate noted the OHS consultations as an important incentive.

Despite the likely overestimate of favourable ratings in comparison to the full population of small manufacturing businesses, a gradient of intervention needs was identified in our sample. Significant fractions of the sample received the very poor (~10% with score of 5 or 6) or intermediate (~32% with score of 3 or 4) SBEI scores. A strength of these scores is that each has corresponding intervention recommendations to guide the user in shifting prevention and control efforts upstream. In this regard, the detailed Potential/Protection matrix and SBEI scores perform a detailed needs assessment and prioritization function as well as providing baseline measures for effectiveness evaluation. As noted in the Introduction section above and elsewhere [18], occupational disease constitutes a substantial and inequitably distributed burden. Thus prioritising of intervention using the SBEI approach, which in this sample identified 10% in need or urgent attention, represents an expeditious and targeted means to reduce this burden.

Inter-rater reliability of the 5 Potential and Protection ratings used to compute SBEI ratings was good to excellent, and the overall SBEI exposure prevention summary scores demonstrated the best reliability of all. Because the two observers were both involved in instrument and protocol development, however, this may overestimate the inter-rater reliability that would be observed with two completely independent reviewers working solely from the written protocol. This limitation notwithstanding, field performance of the SBEI scoring method is good to excellent.

The basis of the SBEI scoring method on the hierarchy of controls supports its face validity. Furthermore, when used as pre- and post- intervention effectiveness measures as intended in this study, the baseline assessment of each area serves as its own reference or control, with the final evaluation metric being a measure of change [6]. To the extent that a given area or process does not change fundamentally over the course of the intervention (e.g., gets replaced with an unrelated process or gets phased out), this strategy overcomes limitations inherent in comparing area ratings and scores cross-sectionally as well as longitudinally (as an intervention effectiveness measure).

We would hypothesize that cross-comparison of ratings and scores across production areas assessed and study sites would show corresponding relative levels of hazardous substance exposures. This has not been assessed in the current study because of the developmental stage of the instrument, technical and economic feasibility issues, and concerns about decreasing participation. With respect to feasibility, numerous agents would have to be sampled many times in each area assessed, which would involve considerable expense. In addition, requests to conduct such extensive sampling in the recruitment phase would be likely to further bias the sample of participating companies towards those with relatively good exposure control programs.

One approach to validation would be to obtain summary measures from multiple quantitative exposure measurements for each hazardous material in each area assessed. Measurements for each agent could then be transformed to a percent of a chosen set of Occupational Exposure Limits (OEL) (e.g., ACGIH, NIOSH, or OSHA). These summary percent OEL's could be averaged into an overall percent OEL across the range of agents present in each given area, paired with SBEI scores for each area, and analysed using standard correlational methods. We explored this possibility in the Wellworks-2 study, but found that routine monitoring was reported for only 14% of areas assessed [5]. This showed that there was not enough company-collected quantitative exposure data available for validation studies even in large manufacturing worksites. We expected even less routine exposure monitoring to occur in small manufacturing settings, and thus did not explore this possibility in the Healthy Directions-Small Business study. Additionally, this demonstrates a gap in workplace exposure assessment practice that might be addressed in part through the application of more economical alternative strategies such as the approach described in this report.

Comparable assessment approaches to other hazardous exposures may also be feasible, such as ergonomic, safety, or other hazards. The development of a similar health and safety rating system has been reported for farm operations, wherein 'positive aspects' are balanced against 'negative aspects' for four different farm characteristics (operator attitude, operator characteristics, status of facility, and status of equipment) [19]. A Site Rank Score is generated as the average ranking of the four characteristics. In this example, a very similar conceptual approach to the SBEI was generated independently for a different work context. Such rating schemes have broad applicability beyond manufacturing work settings.

SBEI also shares fundamental characteristics with control banding. The control banding approach has evolved from its origins in the pharmaceutical industry [20] to the sophisticated yet accessible web-based "COSHH Essentials" program designed by the U.K.'s Health and Safety Executive to assist small businesses in their efforts to understand and control risk http://www.coshh-essentials.org.uk[21]. Control banding allows managers to use process knowledge, walkthroughs, toxicological data and other information sources to assign a job task to a "control band" based on risk potential. The band dictates the level of control needed for a particular operation. SBEI similarly promotes the integration of multiple hazard and exposure information sources, but also incorporates into its assessment the current control strategies for each operation. SBEI extends the control banding approach to create an integrated framework of risk and control assessment. The SBEI measures also extend control banding approaches in providing ratings and scores that are usable as intervention effectiveness evaluation measures.