The test development process began with NIOSH creating a comprehensive list of learning objectives included within each of the six main lessons of the Talking Safety curriculum (Table II), to be used as the initial “blueprint” for building the test. The learning objectives are specific statements that describe what a student should know or be able to do upon completion of the Talking Safety program.

A key component of assessment development is the participation of qualified Subject Matter Experts (SMEs) to assure content quality of an assessment [Linn et al., 1991]. NOCTI criteria for SMEs requires that they be business and industry representatives and teachers (secondary and post-secondary) who have at least three years of technical experience. NIOSH recruited six SMEs in young worker safety and health (from the Massachusetts Department of Public Health, the University of California, Berkeley, Labor Occupational Health Program, Oregon State University, and University of Washington) who met the criteria. (The NIOSH authors also participated as SMEs). In addition, three SMEs who were not involved in the development of the Talking Safety test served as reviewers. A small honorarium ($1,000) was provided to each expert. The SMEs received an email from NOCTI explaining their roles, tasks, and time commitment (approximately 30–40 hr over 30 weeks).

The first task for SMEs was to rate the learning objectives based on the importance of: (i) students being competent in that task upon learning the curriculum; and (ii) the task being suitable for inclusion on the multiple-choice assessment. SMEs were given written instructions on how to use a Likert Scale (1–5) for rating the objectives, with five being the most important and one being the least important. The 33 student learning objectives delineated within the six, main Talking Safety lessons were synthesized into 24 “Core Components,” or essential learning objectives to be covered on the test. NOCTI compiled the ratings and determined a weighting for the content coverage on the assessment. SME consensus was established on the content for the assessment and a final blueprint was created aligning the NIOSH 8 Core Competencies and the 24 Core Components (Table III).

Multiple-choice tests are used in diverse settings and often for high-stakes purposes. Achieving proper construction of such tests may be challenging [Little et al., 2012]. In the first of three consecutive, 3-hour webinars, the NOCTI facilitators trained SMEs on constructing effective multiple choice questions. During the webinars, the SMEs also viewed existing questions from the NOCTI question bank and selected items that had potential to be revised for use on the Talking Safety assessment.

Once the SMEs selected questions from NOCTI’s existing item bank, the number of new questions that needed to be drafted was determined and the task was divided among the group members.

Multiple rounds of question writing and editing among NOCTI, NIOSH, and the SMEs occurred, resulting in the development of a total of 160 potential test questions, which were then entered into the NOCTI online system. Three independent reviewers who were not involved in the test development, reviewed the questions online in the actual test format/environment and select items were revised based on their feedback.

The SME team conducted a second round of online review through a series of three consecutive, 2–3 hr webinars. The review entailed sorting the 160 questions into the 24 Core Components (Table III), deleting poorly worded or redundant items and drafting additional questions closing any gaps in the blueprint. The final assessment for pilot testing included 118 items. These questions underwent a formal “plain language” review [Plain Writing Act of 2010] to place the assessment at an 8th grade reading level (as measured on the Fleisher–Kincaid scale). The revised questions were then entered into the NOCTI online system. Two optional demographic questions (requesting the test-taker’s age and grade) were also included.

The next phase of the assessment development involved administration of the 118 items with a pilot test group. NOCTI determined that the pilot sample for the NIOSH Talking Safety assessment should include a minimum of 150 participants and that the population should be representative of those who would take the assessment in a regular testing environment.

Through its partner network, NIOSH identified a school district in rural Oklahoma to participate in the pilot. The district is approximately 2 hr southeast of Oklahoma City and has a student population that is approximately 60% Caucasian and 30% American Indian, mainly from the Chickasaw, Cherokee, and Choctaw Nations. Race, ethnicity, and sex were not factors in the school’s selection for the pilot test. The junior high school, comprised of grades 7 through 9, had approximately 160–170 students and the high school, with grades 10 through 12, had a similar number of students.

Prior to conducting the pilot test, NIOSH obtained an Institutional Review Board research exempt determination from the NIOSH Human Subjects Review Board (HSRB). The project occurred within a regularly established educational setting, during normal school hours and as part of established and ongoing classroom activities. NIOSH researchers were not in contact with students involved in the pilot testing, the project presented no risk of harm to participants, and therefore involved no procedures for which written consent is normally required outside of the research context. The pilot test was carried out with 8th through 12th grade students in the school system who were trained the Talking Safety curriculum (over one school day) and then completed the online test.

NIOSH provided an experienced instructor to lead the train-the-trainer workshop (in August at the start of the school year) for 14 teachers and assistants from the pilot school in Oklahoma. NIOSH project coordinators were also in attendance to assist with the training. The workshop provided an overview of the Talking Safety curriculum as well as hands-on experience with the activities.

The teachers and assistants who participated in the train- the trainer workshop taught the six, 45-min main lessons of the Oklahoma Talking Safety curriculum the following day, during regularly scheduled school hours. Students were put into groups of 25–30 by grade, and assigned to separate classrooms. Approximately 200 students participated in the training. Students were informed by their teachers of the purpose of the pilot testing, and they received written and verbal instructions for completing the proctored, online assessment.

Each student was assigned a test code number and the school coordinator maintained a log of the codes. Due to the large number of items to be tested (118 questions), the assessment was divided into two 59-question tests which could be completed in one 120-min session or two 60-min sessions. Students utilized the same test code for each session so that the results could be paired. No identifying information, such as student name, was collected. All test-takers completed the pilot assessment within a designated, 2-week period subsequent to receiving instruction on Talking Safety.

The examination of the Talking Safety assessment for test reliability revealed a high level of internal consistency (Alpha/KR20 statistic of 0.93, with a standard error 3.05%). The average proportion of test takers choosing the correct answers on the Talking Safety assessment was 0.81. Item-level statistics revealed the proportion of correct scores for a single item ranged from a low of 5% to a high of 98% correct. The average discrimination index (DI) for the 118 items on the test was 0.25. The range of discrimination indices for the 118 questions was -0.11 to 0.73. A negative DI occurs when more of the lower scoring students answer the item correctly than did the higher scoring students.

Detailed, item-level statistics were also analyzed for each of the 118 items to identify questions that did not perform well during pilot testing. Through the data analyses, potentially problematic topics—items or clusters of items on which students had lower scores—were identified. Thirteen items had proportions correct scores of less than 60%. These items were examined to determine whether they were poorly constructed or whether they were simply more difficult. As noted previously, students underperformed on items related to Competency 4 (Recognize how to prevent injury and illness. Describe the best ways to address workplace hazards).

Based on the pilot data, and after several rounds of discussion between NOCTI and NIOSH as to the scoring and importance of various items, two parallel forms of 50 multiple choice questions each were drafted, measuring similar material and that were of comparable difficulty. Of the items piloted, 34 of the 118 were not included on either form. Low-performing and redundant questions (as determined by the statistical analyses) were eliminated from the test. Approximately a quarter of items appear on both forms. The remaining 68 items appear on only one form. The mean score for Form 1 was 83.5% with a standard deviation of 12.89%, an Alpha/ KR20 reliability statistic of 0.87, and a standard error of 4.65%. Form 2 had a mean score of 84.3% with a standard deviation of 12.41%, an Alpha/KR20 reliability statistic of 0.87, and a standard error of 4.47%. Once the two parallel forms were constructed, the next step was to establish a suitable cut score for the test.

A criterion-referenced cut score is established to set the point at which a student is considered minimally competent on a multiple-choice assessment [Cizek, 1996; National Occupational Competency Testing Institute, 2015] and numerous methods exist for determining cut scores [Kaftandjieva, 2010]. The Jaeger method [Jaeger, 1982], one of the most widely used procedures for setting cut scores [Kaftandjieva, 2010], was utilized because of the method’s involvement of stakeholders [Cizek, 1996]. This method requires the SME panel to establish a criterion question. This question is used to judge individual items on an assessment to determine whether an examinee must be able to answer an item correctly in order to be considered minimally competent in the subject matter. To implement this procedure, participants answer the following question for each item on the assessment: “Should every examinee.. .be able to answer the test item correctly?” [Jaeger, 1989, p. 494]. Jaeger’s procedure involves many iterations of data collection. The SMEs are provided an opportunity to reconsider their initial opinions based on the judgments of other subject matter experts and on information related to actual test-taker performance (e.g., anticipated pass fail rates).

For the Talking Safety assessment, a group of 6 SMEs, including two new members (from the Oklahoma Department of Labor and the New York Committee for Occupational Safety and Health) not included in the original test development, were recruited and trained by NOCTI. The cut score workshop, facilitated by a NOCTI psychometrician (and an author on this manuscript), was conducted for the two forms of the assessment over two, 1.5 hr webinars a week apart. The panel of SMEs was required to make judgments on each individual item on the two assessments, determining which questions were critical for an examinee to answer correctly to be minimally competent in the subject matter. The panel also received the pilot testing data which provided information on the difficulty of each item or group of items. The pilot data were also used to calculate the percentage of students who would have passed on each test at the established cut score point. After determining the critical items for each test, a cut score of 74.0% was established for each form. The estimated percent of pilot test takers to achieve the cut score on Form 1 was 84.4%, and on Form 2 was 82.8%.