Nearly 170 million adults in the United States spend much of their day engaged in work,¹ and work-related factors such as stress, scheduling, and organizational climate contribute to the development of unhealthy physical activity and eating behaviors that lead to chronic conditions such as obesity.^2,3 Healthy eating and physical activity programs tailored for low socioeconomic status (SES) populations (i.e., those with lower incomes, occupational status, and educational attainment) are essential to address the increasing prevalence of obesity-related chronic diseases such as diabetes and cardiovascular disease.^4,5 There is a lack of effective and scalable programs tailored for this population,^6-8 and low SES workforces face barriers to engagement with health behavior change programs,^6,9 such as lack of time, resources, or management support. The workplace can be an ideal place to engage low SES workers in health behavior change programs to address eating and physical activity behaviors,^7,10,11 though limitations in implementing and evaluating such programs hinders their use in addressing the growing burden of chronic disease among blue-collar working adults in particular.³

The National Institute for Occupational Safety and Health’s Total Worker Health^® (TWH^®) program seeks to understand how work environmental factors (e.g., perceived support and organizational health climate in the workplace) influence worker health and productivity.¹² To customize interventions for diverse settings, TWH^® practitioners and researchers increasingly apply participatory approaches to engage workers in health behavior change.^13,14 Participatory approaches may enhance worker buy-in and perceived autonomy, improving engagement and sustainability of health programs.^13,15,16 Evaluating such participatory approaches in lower SES workers at risk for obesity-related chronic diseases may improve understanding of how to effectively and pragmatically engage workers.^7,11,14 Integrative interventions that engage end users may also be more effective than those that impose top-down changes,^2,17 especially among workers of lower SES.¹⁸ However, integrative participatory interventions thus far have mixed evidence of substantial benefit for workers,¹⁹ likely due to insufficient use of comprehensive implementation models, frameworks, and evaluation tools as well as inadequate engagement of managers and workers.^14,20 Thus, researchers are focusing on improving the quality of tools used in process evaluation.

Quality of implementation and effectiveness of health behavior change programs are associated,²¹ and multilevel factors (e.g., coworker, supervisory, and organizational) influence these associations. Understanding these associations requires evaluation approaches capable of capturing implementation and clinical measures across individuals, groups, and organizations. Participatory approaches which engage study participants in planning, implementing, and evaluating programs, such as the TWH^® Healthy Workplace Participatory Program, have been applied to evaluate complex interventions in frontline retail and corrections workers.^14,15 However, few studies of participatory worker programs offer systematic guidance on concurrently evaluating participatory program implementation and efficacy.^21-23 In addition, better specifying the process of tailoring implementation strategies to different settings is important for informing other such interventions.^24-26

Using tools such as the Process Evaluation Rating Sheet (PERS) may enhance process evaluation frameworks used to identify which implementation factors are associated with the effectiveness of participatory workplace programs and thus facilitate tailoring of program delivery.²⁷ The PERS provides a brief assessment of five aspects of program implementation: 1) organizational support and engagement, 2) steering committee engagement, 3) design team engagement, 4) program facilitation effectiveness, and 5) intervention planning and implementation. Evaluators rate four questions within each aspect on a five-point Likert type scale. As described below, we modified the PERS for use in this study. The PERS is increasingly used in participatory worker health promotion studies,^14,28 but few of these studies have both applied and adapted the PERS for the purposes of evaluating program implementation in lower SES workers.

In this study, we examined the utility of the modified PERS as part of an evaluation approach for the Working For You (WFY) program. WFY was a cluster-randomized trial testing a participatory worker program that addressed individual health behaviors and work environment factors contributing to obesity in workers of lower SES.²⁹ Design teams of workers representing their workgroups collaborated with researchers to propose, implement, and evaluate healthy eating and physical activity solutions for their workgroups. We sought to evaluate implementation of the WFY program with short-term efficacy outcomes including perceived organizational health climate in the Multifaceted Organizational Health Climate Assessment (MOHCA) and perceived coworker and supervisor supported in the Job Content Questionnaire 6.^30,31

Our specific objectives were to: 1) assess associations between our process measures and the implementation factors in the modified PERS in our logic model and 2) assess associations between the process measures and modified PERS, and short-term efficacy outcomes over time in the program logic model. We hypothesized that 1) higher modified PERS scores would be associated with higher scores on process measures and 2) better quality program implementation as indicated by higher modified PERS scores and process measures would be associated with larger changes in short-term efficacy scores over time.

Study data came from a cluster-randomized trial of a participatory weight loss program conducted in 2015-2020 among employee workgroups in a large American healthcare system and affiliated university.²⁹ The Washington University School of Medicine Institutional Review Board approved this study (ID: 201505131), and all participants gave written informed consent.

The intervention arm of the WFY study comprised of 11 workgroups of between 25 to 137 workers, and trial implementation occurred in 2018-2020. Four to six workers made up the design team for each group, and a program facilitator worked closely with each group. The design teams and program facilitators met weekly during the first three months of the intervention and then bi-weekly for 21 months. The meeting goals were to propose and implement healthy eating and physical activity initiatives within their work units.²⁹

To propose workplace changes, each workgroup’s design team followed a seven-step participatory process described by Center for the Promotion of Health in the New England Workplace’s Healthy Workplace Participatory Program:^13,32 1) evaluating the workplace; 2) identifying problems; 3) generating solutions; 4) creating solution prototypes; 5) gathering feedback about the prototypes; 6) modifying the prototypes; and 7) implementing the solution prototypes. Program facilitators from the research team worked closely with design teams during the first 12 months of the intervention; design teams ran independently during the second 12 months. Throughout the intervention, the design team and program facilitators vetted proposals with managers when feasible.

We demonstrated the utility of the modified PERS in participatory worker program process evaluation frameworks. The modified PERS scores were associated with multiple program process measures made during the intervention, indicating that the PERS provides a summary measure of implementation quality. The modified PERS was also associated with changes over time in workers’ perceptions of organizational health climate, and perceptions of social support from supervisors and co-workers. These findings indicate that the modified PERS may be useful for specifying associations between implementation quality and short-term efficacy outcomes. Our findings partially supported our hypothesis that higher PERS scores would be associated with better process measures, MOHCA, and JCQ scores. Associations between modified PERS and process measures also demonstrated that the PERS measures implementation quality as expected. Work groups with higher modified PERS scores showed higher scores on process measures taken during the intervention. Moreover, work groups with higher modified PERS scores experienced greater increases in MOHCA and JCQ scores from baseline during the WFY intervention, although these groups did not sustain these changes after the first 12 months of the intervention, and not all subcategories and items demonstrated expected associations.

Our findings also demonstrated associations between the modified PERS and short-term program efficacy measures including improved scores on the MOHCA and the JCQ. In addition, the MOHCA and JCQ scores over time demonstrated stronger associations with changes in modified PERS scores than with other process measures. Taken together, these findings suggest the modified PERS may be useful in holistically assessing program implementation quality in participatory workforce interventions. Our findings also suggest future directions below to develop better tools for assessing participatory processes in worker health programs.

First, using a breadth of process and efficacy measures within participatory workforce programs may better elucidate connections between implementation quality and short-term efficacy outcomes. Although measures within our framework captured a change in perceived organizational health climate among groups with high levels of program quality at 12 months, these associations weakened by 24 months. Facilitator notes indicated that the design teams’ engagement with the program tapered off over time. This highlights the importance of maintaining high levels of implementation quality throughout a long-term workforce intervention. Although our findings showed positive associations between the implementation measures and short-term efficacy measures at 12-months, these positive associations diminished by 24-months, even among groups with high overall implementation quality. One potential explanation for this effect is that sustaining design team engagement may be challenging over time, and the research team noted challenges with sustaining engagement in a time-intensive program.

Second, other studies corroborate that sustaining a participatory workforce intervention requires sustained high-level engagement from a variety of stakeholders to obtain changes in desired outcomes.^14,15,20,28 Better understanding sustainability challenges that continually impact participatory workforce programs requires the development of measures, tools, and evaluation frameworks.^28,35 For example, the Program Sustainability Assessment Tool is a tool that specifically measures the sustainability of a complex community-based program.^38,39 To our knowledge, this tool has not been used in workplaces for the purpose of assessing participatory worker health promotion programs, so occupational safety and health practitioners may seek to use the Program Sustainability Assessment Tool in future participatory worker health promotion programs.

In our evaluation of the modified PERS’ association with perceived supervisor support, all participants, including the highest scoring modified PERS groups, experienced a larger drop in perceived supervisor support than in perceived coworker support or organizational health climate throughout WFY implementation. This indicated that future studies of participatory workforce intervention may consider more closely assessing quality of worker-supervisor relationships, sustainment of supervisor support, and organization-level leadership changes that influence programs over time.^14,16 Furthermore, better distinguishing between the quantity and quality of administrative support for programs and workers’ perceptions of administrators and leaders may inform our understanding of the relative influence of observed and perceived support of participatory worker programs.

Finally, although the efficacy of the program diminished over time, our study offered evidence of the modified PERS’ usefulness to capture associations between implementation factors and program efficacy. Because implementation quality and desired outcomes are often associated,²¹ observing implementation via process measures and rating implementation via tools like the modified PERS offers further options to evaluate the quality of program implementation. Individual observations of a process such as counts of meetings attended may ignore perceived features influencing an intervention’s success, such as perceptions of stakeholder engagement and intervention implementation, which may influence implementation outcomes such as acceptability or feasibility.^16,25,40 Further, the modified PERS enables researchers and participants to measure holistic information quickly and efficiently about the quality of design teams that implement programs in an intervention such as WFY. The modified PERS enables research teams to quickly assess over- or under-performing groups in a participatory intervention at any timepoint of implementation.

Our study has several limitations. First, a majority of participants were female and non-Hispanic Black, so future studies should seek to replicate findings in other demographic groups to generalize findings. Other factors such as institution-level policy changes and changes in management and work unit staff over the course of the WFY program may have influenced assessments within or among the groups. For example, some of the work groups were completing participation in the parent trial during the early months of the COVID-19 pandemic in 2020. We were unable to evaluate the extent to which changes to workplace policies and worker health and well-being confounded overall results. Our study protocol also only measured staff turnover during recruitment, so the attrition of individual workers due to turnover within participating work units may also have influenced findings.

Second, our team only rated modified PERS scores at one time point, after the conclusion of the study. Thus, we may have overlooked details from earlier in the study implementation during our rating process. The associations between process measures collected during the study and the modified PERS scores are not surprising, as the modified PERS raters were sometimes the program facilitators who had recorded some of these process measures from team meetings. Future studies may seek to expand assessment time points or incorporate rigorous interrater reliability processes, and collect additional data on contextual factors and implementation factors affecting participatory worker approaches.^40,41

Third, although low-, medium-, and high-PERS groups had similar MOCHA and JCQ scores at baseline, sociodemographic characteristics varied among the three groups. Thus, additional unmeasured sociodemographic factors outside of our process evaluation framework may have influenced our limited study outcomes,⁴² such as perceived organizational health climate and work support over time. Moreover, work units consenting to participate in a worker weight loss program likely had stronger health cultures and climates compared with non-participating units. Future study protocols should measure more factors related to these outcomes and include more than 11 work groups to adequately sample a range of worker, work unit, and organizational features which exist in complex workplaces. Assessing changes to weight and differential treatment effects in participants with and without obesity was beyond the scope of this exploratory study. However, adjusting classifications of obesity to include ethnicity-specific cut-offs, rather than a generic BMI ≥ 30 kg/m² cut-off, may facilitate more precise evaluations of BMI changes.

In summary, our findings indicated the utility of the modified PERS in our process evaluation framework as well as the modified PERS’ associations with short-term efficacy measures of organizational health climate and perceived supervisor support. As clinicians and researchers develop participatory workplace health programs, embedding rigorously developed tools and evaluation frameworks within programs may facilitate higher quality targeted action towards improving worker health. In addition, our findings underscore the importance of including a breadth of measures to fully capture the implementation quality and efficacy in participatory worker programs.