The previously published TWH core competencies and the proposed KSAs in this article were informed by the Council on Linkages Between Academia and Public Health Practice (Council on Linkages) competency framework^32,33 and the Competency-to-Curriculum Toolkit (Toolkit).³⁴ The Council of Linkages framework describes core competencies as foundational skills necessary for professionals engaging in the practice, education, and research of public health. The competencies are organized into domains that represent the skill areas within public health and the different types of responsibilities within an organization or level of expertise for public health professionals. Similar to this, the TWH core competencies are divided into six broad domains, for which KSAs are proposed to train professionals and practitioners across the spectrum of OSH and allied professions. The first part of the Toolkit focuses on assumptions, definitions, and the development of competencies, while the second part focuses on the process to develop curriculum and training models from proposed competencies. Moving from selected competencies to the development of a competency-based curriculum involves selecting the competency, describing key words and sub-competencies, developing learning objectives, resources for theory and practice, methods for learning experiences and develop evaluation criteria and indicators. The TWH KSAs proposed in this paper integrate TWH core competencies, learning objectives, and resources for OSH and allied professions, including OSH and workplace health promotion (WHP) practitioners, health educators, human resource (HR) specialists, and business and labor relations professionals.

Corporate and academic spending on education and training has steadily increased, making the development and evaluation of training and its impact on the US economy even more important. From 2020–2021, US companies spent over $92 billion on training, including on the job and remote training, to improve employees’ skills and productivity.³⁵ In 2019, U.S. educational expenditures for post-secondary education increased to $671 billion; total expenses for public ($430 billion), private nonprofit ($228 billion) and private for profit ($13 billion) institutions.³⁶ Given the increased expenditure and the number of training and educational modalities currently available, developing and evaluating training programs is critical to ensure participants gain and maintain the necessary competencies and KSAs to effectively improve work performance, productivity, and behaviors that support organizational values and results.³⁷ Some TWH Centers are already conducting evaluations of their seminars, symposia events, summer institutes, and training and certificate programs, but information on TWH evaluation approaches used to evaluate education and training programs is not readily available or standardized, making it difficult for others to learn from or replicate.

According to James D. Kirkpatrick, “Evaluation is not an afterthought to training, but rather is meant to be integrated into the entire learning and development process.”³⁸ The selection of the appropriate evaluation method and the data obtained at each stage of the process are critical, from identification of training needs and key stakeholders through the development of the training program, including modifications along the way, learning progress and barriers, improved performance and behaviors, improved skills and applications, and ensuring training objectives are met. The CDC Framework for Evaluation in Public Health (CDC Evaluation Framework) is one approach that can be used to plan for and conduct evaluations, including those for education and training programs.³⁹ The CDC Evaluation Framework is a non-prescriptive tool designed to summarize essential elements of program evaluation, presenting framework users with six connected steps in the evaluation practice and four standards for effective evaluations. The CDC Evaluation Framework emphasizes the utility of logic models to describe programs, which can result in effective program evaluations. Additionally, numerous models have been created and adapted over time to support the development and evaluation of education and training programs. These training evaluation models focus on the continuum toward impact; from the development of learning, to individual knowledge gain, to organizational or societal outcomes and impact. They include the CIPP Evaluation Model (Context, Input, Process, and Product), the CIRO Model (Context, Input, Reaction, Output), the Phillips Return on Investment (ROI) Model, Kaufman’s Model of Learning Evaluation, and the Kirkpatrick Model, to name a few.^40–43 However, according to Tamkin et al, these models’ limitations include a significant focus on changes resulting from learning, and do not address the learning process. Essentially, these models center on outcome evaluations and don’t include formative or process evaluations which assess development and implementation.

We recognize that numerous models may be used to successfully plan for and evaluate TWH education and training programs. Organizations developing, implementing, and evaluating their education and training programs are encouraged to seek out and use models that fit their needs best. When selecting an evaluation model to assess education and training programs, Tamkin et al., recommend considering: 1) the purpose of the evaluation, 2) the intended evaluation audience, taking into account the culture and needs of the organization where the evaluation is occurring, 3) the purpose of the training and its expected outcomes, and 4) the learning process model underlying the training and possible responses one wants to capture via evaluation during the various stages in the process. Researchers and evaluators should consider needs and constraints while maximizing the informativeness of the evaluation.⁴² In addition to evaluation models, various programs and organizations often seek models that help guide the creation and implementation of the training and education program itself. When developing training or education programs, or conducting an evaluation, the development and use of a logic model can serve as guiding steps through the process, including indicator development. For this reason, we introduce a logic model for TWH education and training in this article and provide examples of indicators.

A logic model may be useful to help plan for and focus the training and education program and process, to determine what to measure, and to identify areas of the program most in need of evaluating.⁴⁴ Previously, NIOSH collaborated with the TWH Centers of Excellence to develop the Total Worker Health Communication and Outreach Logic Model which was included in the “NIOSH Centers of Excellence for Total Worker Health^® (U19)” Notice of Funding Opportunity posted in 2020.⁴⁵ The TWH Education and Training Logic Model presented in Table 1 builds on the education and training component of that logic model, and was developed using the TWH competencies³¹ and proposed KSAs required to successfully perform TWH-related job functions such as the design, implementation, and evaluation of TWH interventions to prevent, improve and enhance the working lives of the US workforce.

The TWH Education and Training Logic Model is a graphic depiction of the relationship, process, and impact of providing TWH educational/training opportunities both at the academic and continuing education levels for professionals and organizations (target audiences). This logic model is divided into seven sections: Inputs, Activities, Outputs, Short-term Outcomes, Intermediate Outcomes, Long-term Outcomes, and Future Impact. It is intended to provide a framework for identifying the numerous resources and expertise available to inform TWH knowledge and principles (Inputs), and the actions, behaviors, and processes (Activities) needed to accomplish the ultimate goal. The Outputs section includes identification of target audiences, and captures the quantity of various TWH educational/trainings conducted and persons trained. Outcomes are the desired results of implementing the educational/training program and the goals achieved. Outcomes are displayed in a sequence and may be divided into short-term, intermediate, and long-term, depending on the objective and length of the program. Short-term outcomes represent the immediate effects of the educational/training program and focus on the knowledge and attitudes of the intended audience. For example: Increased awareness and knowledge of the principles of TWH. Intermediate outcomes can be changes in behavior, normative, and policy. For example: Increase in TWH concepts integrated into workplace practice through trained TWH professionals. Long-term outcomes often take years to accomplish and represent the desired results of an educational/training program. For example: Increased capacity for TWH in practice. Future Impact refers to the ultimate impact of the program, which could be achieved within several years or longer. For example: Reduced workplace injury and illness and improved safety, health, and well-being among organizations and communities. The logic model is broad and may be modified as needed by those seeking to develop and implement TWH education and training programs and program evaluations. Developing or modifying the proposed logic model may help program staff determine at which step in the process the ADDIE and Kirkpatrick models may be applied, as discussed below.

In this article we propose the combined use of the ADDIE model with the Kirkpatrick model as a framework to both design and evaluate education and training of current and future TWH OSH and Allied Public Health professionals. The ADDIE model has been widely used in developing and evaluating education and training programs across various disciplines, including OSH.^46–50 The ADDIE model goes beyond evaluation; it is comprehensive in nature and provides a framework for creating and implementing education and training programs, while also including an evaluation component. Its cyclical, ongoing process allows for continuous improvement. The products of each phase may be evaluated against the goals and learning objectives of the education/training, with results informing which phase of the model to enter next, and where changes may be needed to improve the education/training. One criticism of the ADDIE model, however, is the difficulty in verifying the degree of participants’ expertise at the conclusion of the education/training.⁴³ Therefore, we also tie in and describe the utility of the Kirkpatrick model and how it can be applied in the context of the ADDIE model to surpass this limitation.

The ADDIE Model was developed in the 1970s as a field-effective and efficient process to prepare trainees to meet work-performance requirements.⁴³ The model has evolved over time to become simpler and more flexible, enabling instructional system developers with all levels of expertise to apply its use in developing effective and efficient instructional systems.⁵¹ Below is a brief description of each ADDIE phase, where we provided some examples relevant to TWH.

The Kirkpatrick Model has been used extensively to evaluate education and training programs.^{48, 53–54} The model posits that programs can be evaluated on four levels: reaction, learning, behavior, and results. This worldwide standard can be used to evaluate the effectiveness of an education or training program and can be applied before, during, and after to maximize and demonstrate an education or training program’s value to the organization. The four-levels of the Kirkpatrick Model are briefly described below, with an example evaluation question at each level.

Researchers and practitioners in the discipline of TWH are already implementing and evaluating education and training programs using this model.^55–56 One example is an article which applied the four levels of Kirkpatrick’s model to evaluate the effectiveness of a 14-week TWH intervention designed for improved safety, health, and well-being measures of construction workers. The study assessed reactions and post-test scores to trainings and group education sessions, self-reported behaviors pre- and post-intervention, and objective measures of health outcomes.⁵⁵

The ADDIE model has been successfully used to develop and implement education and training programs, and to evaluate them in conjunction with the Kirkpatrick model.^57–60 Using the combined approach addresses some of the limitations of using each model separately. For example, any one or all four of the Kirkpatrick model’s levels can be applied to conduct an outcome evaluation as part of the ADDIE model’s evaluation phase. Several examples that apply both the ADDIE and Kirkpatrick models showcase that the use of both models strengthens the overall training development and implementation process, from the analysis to the evaluation phase.^57–60 Subsequent examples highlight how the ADDIE and Kirkpatrick models were both applied to develop, implement, and evaluate various education and training programs. As part of the ADDIE evaluation phase, all these programs employed Kirkpatrick levels 1 and 2 to gauge participant reaction and satisfaction with the education or training program, and to assess gain in knowledge. All programs noted that using Kirkpatrick levels 3 and 4, if feasible by the program, would strengthen the case for program effectiveness and assess improvement in health indicators long-term. In one example, as part of the Public Health Learning Network (PHLN), the nation’s most comprehensive system of public health educators, experts, and thought leaders to advance public health practice and improve population health in the U.S., the Midwestern Public Health Training Center Regional Coordinating Center (MPHTC-RCC) applied the ADDIE model to develop, implement, and evaluate high-quality adult eLearning in public health.⁵⁷ They also employed Kirkpatrick’s level 1 to ascertain participant satisfaction with the modules, and participant pre- and post-test scores as part of level 2 to assess overall gain in knowledge and skills. MPHTC-RCC developed an eLearning checklist for using the ADDIE model, which they shared in the article; although not used as part of their evaluation, they noted that using Kirkpatrick level 3 would strengthen the case for its program effectiveness in the long-term. In another example, Fernandes et al., used the ADDIE model to plan, develop, implement, and evaluate a 10-month management specialization course in oncology using in-person and virtual learning components (i.e., blended learning) in Brazil.⁵⁹ In the evaluation stage of the ADDIE model, the first three levels of the Kirkpatrick model were applied two months after completion of the course. Participants evaluated the course using a 27-item online form that assessed their reaction to the course, how much they learned based on course objectives, and changes in behavior and professional conduct. Although the study did not evaluate Kirkpatrick’s level 4 due to time constraints, the authors noted that ideally, an assessment of improvements in health indicators and overall results would be measured two years after the intervention. Another example of the application of the ADDIE and Kirkpatrick models was the development and evaluation of e-learning modules as an approach for the dissemination and implementation of an evidence-based individual placement and support (IPS) model to support employment in 86 community health treatment programs in New York State.⁶⁰ Through iterative development, the ADDIE model was successfully applied to develop a series of e-learning modules for IPS. The authors focused on the first two levels of the Kirkpatrick model in the report, using a survey to ascertain learner reactions to the training, self-reported knowledge acquisition and self-reported practice change, and resulting knowledge via post-module quizzes. These are just some examples that highlight how both the ADDIE and Kirkpatrick models were used to develop and evaluate education and training programs. Institutions seeking to develop and evaluate TWH education and training programs are encouraged to apply components of the ADDIE and Kirkpatrick models to fit their needs, resources, and timing constraints.

Selecting appropriate indicators to answer evaluation questions is important to effectively evaluate education and training programs. Indicators can provide information on individual-level and organizational-level effects. The proposed KSAs and TWH Education and Training Logic Model can guide the development of indicators. Additionally, although not included in the Kirkpatrick model, if your organization finds it important to assess factors that lead up to training implementation (e.g., was training developed based on need, were appropriate stakeholders involved, etc.), we provide an example indicator which may align with the ADDIE model analysis and development phases. Likewise, an organization may want to assess factors beyond Kirkpatrick’s Level 4: Results, such as their organization’s capacity to provide ongoing and relevant training, ROI using a cost-benefit analysis to determine the value of the education/training program, and/or societal outcomes resulting from the education/training. In such a case, organizations may opt to apply evaluation models in addition to, or in place of, the Kirkpatrick Model to evaluate their education/training efforts.⁶¹ For additional ideas on indicators that may be used to collect data on the four Kirkpatrick levels, see: Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Association for Talent Development; 2016³⁸.

As the TWH discipline evolves toward new and innovative evidence-based research and practical applications within workplaces, there will be a continued demand to identify and apply appropriate training and education for OSH and allied professionals in the discipline. These professionals will require knowledge, skills, and a multidisciplinary expertise not formally provided in the current educational structure. Previously, six broad core competencies for TWH, key priority audiences, and several training methods were proposed, continuing the discussion with key stakeholders on the training and education needs of the public. As a result, some academic institutions now offer training and certificate programs or degrees in TWH¹⁸, several TWH Affiliate professional societies are adopting TWH approaches in their educational programs¹⁹, and examples of the application of TWH concepts and findings continue to be illustrated in research publications, interventions, and practical applications. In this article, we proposed examples of specific KSAs that expand and enhance each TWH core competency. We also developed a TWH Education and Training Logic Model and proposed the combined use of the ADDIE and Kirkpatrick models as a framework for developing and evaluating TWH education and training programs. The logic model may provide prospective institutions examples of measurable inputs, activities, outputs, outcomes, and future impact for developing and evaluating TWH education and training programs, and the ADDIE and Kirkpatrick models may aid institutions in planning for and evaluating such efforts and creating a standardized process for TWH education and training planning and evaluation. All these proposed components can be integrated into public health practice, guide and enhance TWH workforce development planning and TWH education and training, be used to develop a TWH curriculum, evaluate such efforts, and prepare for future accreditation. As Black pointed out, “for developing, implementing and measuring the impact of integrated TWH programs, practices and policies in the workplace, health and safety professionals with the TWH competency-based-education knowledge and skills is a requisite to working collaboratively and effectively with allied professionals, employers, employees, labor, and government organizations.”⁶² For example, TWH trained professionals will be able to effectively communicate the relationship of work and non-work health and well-being, translate and apply TWH research into practice-based solutions to advance the discipline, and illustrate the value of the adoption of an integrated approach to key stakeholders including industry, academia, labor, and health care.⁶²

Through the evolution of the competency-based-educational framework, several examples of a blended approach have been successful, incorporating TWH competencies and courses into existing programs. This strategy may be appropriate for transitioning from the traditional educational model to provide time and resources, modification of the curriculum, and the development of TWH educators. Initially, TWH topics and principles can be incorporated into the NIOSH funded TWH Centers of Excellence, ERCs and other Centers of Excellence to train graduates and provide continuing professional development that will build a pool of qualified professional practitioners and researchers to conduct research, outreach and research-to-practice activities. Additionally, other educational and training programs and products outside of academia exist or can be developed and made accessible to OSH and other health care professionals, HR and other business professionals during meetings, workshops, seminars, panel discussions, e-newsletters, listservs, and journals to expand capacity, such as, e-learning modules, case studies, toolkits, micro-training, TED talks and other training resources. Support of existing educational programs and curricula where TWH education is currently offered and where TWH principles can be incorporated is critical. This paper can offer guidance on the development of KSAs, and an education and training development and evaluation framework that can be further tested through other educators, academic professional societies, and organizations for incorporating TWH training into existing and new degree, certificate, and continuing education programs. It can also help identify potential gaps in TWH education and training and the evaluation of such efforts, given the lack of TWH literature that describes which evaluation frameworks were used in program evaluation efforts. Academic and educational/training institutions are encouraged to pilot test the proposed KSAs and development and evaluation framework for TWH education and training, as feasible, and publish their findings. With the recent establishment of a TWH Professional Society, numerous partnership and educational opportunities are now available or planned to further expand TWH research, interventions and outreach approaches, and provide effective business strategies that will build TWH into leadership and management practices within the workplace and community.

As work and the workplace changes and the TWH approach evolves through new research, trained TWH professionals, effective interventions and outreach, and the integration of health protection to eliminate/reduce injury and illness with the promotion of health and well-being, it will have profound and far-reaching effects within and outside the workplace and throughout the nation.