This experiment was approved by the Purdue University Institutional Review Board (IRB). Thirty-one participants were recruited from the university population to perform the study. All participants had no professional experience in workplace lifting tasks, and no participant reported any musculoskeletal pain or discomfort at the time of the experiment. The sample contained 19 males and 12 females, and 80% of the participants were right-handed.

Tactile gloves (PPS, Inc., Hawthorne, CA) were utilized in this study (figure 1). Sixty-five pressure sensors were embedded on the palmar side of each glove. Per the glove manufacturer, each sensor measured the pressure that was exerted on the palmar side of the hand with a scanning rate of 25-40 Hz, and pressure data was streamed continuously to a computer via Bluetooth connection. The gloves were calibrated using the manufacturer’s software before each data recording session while participants were in a relaxed, non-working state. All lifting tasks were recorded by a GoPro camera (30hz) positioned on the left side of the participant when the participant faced the platform. The camera position was fixed across all task conditions.

Six different lifting tasks (Figure 3) were performed in a controlled laboratory environment. Participants performed six tasks that involved moving a box with different assigned weights between four specific locations (Figure 2). The platform’s height level (location 2) was adjusted to three different levels (high [1.1m], middle [0.9m], low [0.7m]). Each task (1-6) was performed at every height level. The weights for each lifting task were determined using the Revised NIOSH Lifting Equation (RNLE). To manage participant risk levels, the Recommended Weight Limits (RWL) were estimated independently using the task conditions of the 6 tasks with variations in platform height (Location 2). For each platform height, the estimated RWLs were used to propose a range of lifting weights, roughly categorized the risk into the three LI ranges [LI < 1, 1 ≤ LI ≤ 2, and LI > 2]. The exact distribution of weights was selected based on the availability of weight plate combinations and grouped for each weight/height combination to improve the robustness of the lifting weight prediction results, resulting in a total of nine combinations (e.g., a light weight at low/middle/high heights, a medium weight at low/middle/high heights, and a heavy weight at low/middle/high heights). The weight was constant throughout the six tasks for a specific LI category. A Frequency-Independent Lifting Index approach (FM=1) was used, and each lifting task was treated as an independent action, and a good coupling factor (CM = 1) was assumed for the boxes used in this study because they all had handles that ensured good hand-to-object contact. Specifically, the range of weights (Table 1) for light, medium, and heavy conditions corresponded to LI < 1, 1<= LI <= 2, and LI > 2, respectively (Table 1).

Participants completed one action set (tasks 1-6) twice for each weight/height combination. After completing the action set twice for a specific weight, the platform’s height was adjusted for the next set of tasks. After completing all action sets on the three height levels for a given weight, a new weight level (light, medium, or heavy) would be assigned for the participants to repeat the same process. The height order was controlled, and the order of task 1-6 within an action set was not randomized since the start/end location of each task depended on the previous task (the box ending location for Task 2 is the start location for Task 3). We prioritized randomizing weight assignment and order across participants in the study design to focus on the impact of how different weights at same task designs influenced model performance. A 10-second rest was given between each lifting action, a 2-minute rest was given after the participants completed an action set (tasks 1-6), and a 10-minute rest was given after the participants completed all required tasks at each weight level. Each participant completed 108 lifting/lowering actions in total. For safety purposes, no lifting task with a LI higher than three was given.

A total of 3348 lifts were observed, but 576 lifts were removed due to missing video files, and 31 lifts were removed due to missing glove recordings. The final dataset encompassed 2747 lifting tasks. For each task, the ground truth LI (based on task variables) and algorithm-predicted LI (based on sensor measurements) were calculated at both the origin and destination of the lift, thus resulting in a final dataset with 5494 entries. For the classification task, there were 1784 lifting instances in the low-risk category, 2694 lifting instances in the moderate-risk category, and 1016 instances in the high-risk category.

We compared the task variables estimated from computer vision to the values defined from the designed task conditions (Table 4). The result yielded an average difference of 8.86 cm by averaging the mean absolute differences (excluding asymmetric angle) and 18.71° (asymmetric angle) compared to the expected experimental values for each task condition.

The results of the machine learning models are presented in Table 5. The CNN was the best-performing model and achieved a 0.89 overall accuracy score and a 0.88 precision score. The detailed classification performance of all tested models is presented in the appendix section (Appendix A3–5).

Using the best model (CNN), we further explored the contribution of sensor modalities to prediction performance. Comparing the performance of the three models (glove+pose, pose only, glove only) constructed with different modalities, the model with only the pose-related variables as inputs yielded performance with less than 50% overall accuracy. A significant improvement in overall performance was achieved by using only the glove features as inputs. The multimodal pose+glove classification yielded an additional 0.08 in F1 score than the glove-only model. The primary improvements between the multimodal classification model and the glove-only model were observed in the middle and high-risk classification tasks, with the corresponding F1 scores increased from 0.83 to 0.89 and 0.60 to 0.77, respectively (Appendix A3–5). The confusion matrix of the multimodal classification model shows a decreasing trend in prediction performance from low risk to high risk (Figure 5).

The SHAP values for each feature of our classification model were generated to explain the feature contributions to the model performance (Figure 6). The mean absolute SHAP value of the right hand was 9% higher than that of the left hand, indicating the right hand was more important in the classification model. Fingers were more important than palms as the mean absolute SHAP value of the fingers was 14% higher than the palms. Both the middle and index fingers from the left and right hand were the more significant contributors to our model.

D and V both had mean SHAP values close to 4, as shown in Figure 6B, indicating that they were the most significant contributors to overall model performance among all the input features. Among all pose-related task variables, both the asymmetric angle and horizontal distance had a weaker contribution to the model compared to the vertical distance (V) and the lifting distance (D).

Figure 7 illustrates the contribution of glove and pose-related variables based on the cumulative distribution of SHAP values across all three classes. In lower-risk tasks, glove features contributed significantly more to the model than pose-related variables. However, as the risk level increased, the importance of pose-related variables became more prominent in the model’s predictions.

For this study, the participants were instructed to perform symmetric liftings (asymmetric angle = 0) with fixed horizontal distance. However, relatively inconsistent estimation of foot positions was observed due to camera angles and occlusion of feet. The optimal camera placement for this study should be at a height approximately level with the subject’s upper torso, perpendicular to the sagittal plane, with minimized occlusion and distortion for accurate body segment detection. Another issue arose from the fact that the sagittal line for asymmetric angle calculation is difficult to estimate perfectly from the joint-based skeleton, which is also a common challenge faced by the industrial practitioners during the implementation of the NIOSH lifting equation because the sagittal line is well-defined only when the body is in symmetrical postures [19]. We used unconventional body landmarks (wrist joints, center of pelvis) as practical adaptations to the constraints of the pose estimation algorithm. We used the asymmetry line connecting the mid-point of the hands (wrist joints) and the center of the pelvis (the root joint) instead of the mid-point of the inner ankle bones (as given by the standard NIOSH definition) to estimate the asymmetric angle, and we used foot joints as a proxy for the ground level because the ground plane was not defined by the algorithm. This approach can introduce variability in vertical (~10 cm) and horizontal distance measurements (~17 cm) with a mean absolute error of ~8.86cm, and it provides only an approximation of the true sagittal line and limits the model’s ability to capture subtle variations in foot placement during lifting actions. This approach was established to accommodate limitations in real-world application of computer vision-based pose estimation where precise anatomical landmarks may not always be detectable and ensure efficiency in real-world adaptation. This trade-off is balanced by the system’s ability to automate risk classification efficiently based on the estimated pose information without manually measuring and inputting the task conditions.

Using non-standard body landmarks could introduce slight systematic shifts in vertical and horizontal distance estimations. Variation in estimated pose variables could lead to an under- or over-estimation of LI and potentially misclassify tasks near risk category thresholds into incorrect risk categories. Compared to the fixed task conditions, the estimated asymmetric angle (A) and horizontal location (H) exhibited considerable standard deviations, reflecting higher variability in those pose estimation outputs. In real-world applications, individuals rarely perform tasks in the exact manner specified by the workstation design [60]. This can be attributed to multiple factors, including personal habits, comfort, or adaption to specific task demands, which often result in random movements and body twists [61]. While the task design specified a zero-asymmetry angle, subtle twists occurred as participants anticipated moving toward a subsequent workstation positioned 90 degrees away. These slight movements, which are not captured in the standard NIOSH asymmetry angle definition, introduce inaccuracies in the estimation. In practice, field data suggest variability in how the asymmetry angle is measured by different practitioners. Both the sagittal and asymmetry lines are imaginary and cannot be physically marked or drawn in the workplace, and feet movement during lifting tasks adds to the difficulty of capturing such static measurements [19]. Moreover, participants wouldn’t have identical lifting actions for every repetition, potentially due to fatigue or learning from the lifting tasks [62]. Consequently, sensor-based systems provide a more dynamic and precise assessment of body postures during lifting tasks, enhancing the NIOSH definitions by accounting for the variability caused by body movements.

A key challenge in biomechanic assessments is the requirement of complex sensor or camera calibration/setup to account for varying distances, angles, and environments. Despite known limitations of a single-camera setup, it remains more feasible than multiple cameras [46], [63] and wearable motion capture systems [28], [64] in an industry setting due to its adaptability and ease of use. The use of 3D pose estimation from a single camera coupled with height-based scaling offers versatility; standardizing measurements across individuals ensures that joint coordinates are represented consistently regardless of the relative position of the person to the camera. Height-based scaling also offers ease of implementation, height is a readily available and easily measurable parameter, making it convenient for scaling purposes. A limitation of our proposed method is the reliance on manual height measurement, which introduces an additional step that can lead to potential error, especially since the estimated skeleton only roughly reflects the person’s actual height. Additionally, the use of a 3D pose estimation algorithm with uncalibrated cameras can affect the precision of joint coordinate mapping, especially in complex environments with occlusions and inconsistent camera angles. To obtain the most accurate pose information, the gold standard would still be to calibrate the camera or use a multi-camera setup to enhance accuracy. Furthermore, we showed that a single camera was accurate in extracting the proposed pose-related features for our model in our study environment, recognizing that laboratory-level precision is not always essential for all occupational safety applications [47]. Although it demonstrates the potential of our camera and glove approach to risk assessment, we acknowledge the next step is to test the robustness of this approach against the known limitations of single camera studies.

We achieved the best classification result using the CNN model in comparison to other statistical ML models (Table 5). The CNN model had the best performance at correctly predicting the low-risk lifts, and the performance progressively dropped as the risk level got higher and decreased when predicting high risks. Several reasons could contribute to this behavior. First, approximately 25% of the high-risk lifts were mistakenly classified as moderate risks. Specifically, the average LI difference between low and moderate was 1.22, while the average LI difference between moderate and high was 0.70. The fact that tactile gloves are the sole predictors for lifting weights might also be a contributing factor to reduced performance. The earlier regression model of weight prediction based on the same dataset but using the time-series glove sensor data reported significantly lower accuracy when estimating heavier weights [53], and lifting weight was the most significant contributor to increasing lifting risks in this study. The full CNN model with both glove and pose only showed marginal improvement over the glove-only model, likely due to the experiment’s design, which focused on weight as the main risk factor (LI: 1, 2, 3), with limited variation in other variables like vertical and horizontal distance. This also explains why the pose-only model had reduced performance.

Current methods typically rely on the indirect estimation of lifting weight factors, utilizing tools like sEMG for muscle activity, IMUs for dynamic movement data, or facial expressions [33], [34], [43]. Our multimodal approach provided a direct assessment of lifting risk variables (both posture and lifting weight information) with precision scores for each class (Low, Moderate, High) as 0.97, 0.88, and 0.80. While the majority of previous efforts strived to achieve accurate estimation of lifting-risk-related factors such as joint kinematics, range of motions, or human postures using CV-based or IMU-based methods [41], [48], [65]; methods that provided complete lifting risks assessment were still relying on human posture only and limited to risk prediction based on ACGIH lifting zones [29] or Rapid Upper Limb Assessment (RULA) scores [40]. From efforts that tried to include lifting weight as a significant contributor to risk assessment in lifting, previous studies have included kinematic pose-related variables such as acceleration/jerk or facial expressions to account for lifting weight and perceived effort in their lifting risk prediction models; and each has achieved either a 0.65 accuracy in binary weight classification task [66] or 0.83, 0.63, and 0.67 precision score in a three-class risk prediction task [43]. Using only the static pose-related variables as the predictors in our model to predict lifting risks has yielded a very poor performance, showing that pose-related variables alone were not enough to distinguish differences in lifting index without considering the associated load weight.

A possible explanation for the observation that asymmetric angle and horizontal contributed significantly less than the rest of the pose-related variables is that they were designed (i.e., inputs into ground truth LI) to be constant values throughout the lifting experiment. The variation in asymmetric and horizontal measurements of how participants performed the task vs. what was expected through design. Since the ground truth was calculated on designed variables, this is a potential explanation for why asymmetric angle and horizontal distance were not important predictor features in our model.

Results suggest that glove variables tend to have a greater impact on prediction compared to pose-related variables, particularly for tasks classified as low-risk, as illustrated by the cumulative SHAP values in Figure 7. For the classification of low LI categories, pose-related variables on average contribute significantly less to model performance compared to glove features. However, as the lifting risk rises, the increasing importance of pose-related variables may be attributed to changes in posture in response to varying weights. Although poses alone may not be strong indicators of lifting risk, their relevance grows when heavier weight and higher risk levels are involved. For example, lighter weight generally places less strain on the body, making it easier for workers to choose different poses without significantly increasing the risk of injury. However, for heavy lifting tasks, postures that minimize extended reach and movements for workers may be needed to mitigate strains on various body joints and reduce overall risk.

Similar to the earlier study using tactile gloves to predict lifting weight [51], our CNN prediction model gave more weight to the right hand than the left when assessing lifting risks, likely due to stronger grip strength in the dominant hand (80% of all participants are right-handed) [67], [68]. Moreover, the model relied slightly more on the other four fingers (index, middle, ring, and pinky) than on the thumb or palm. Since the boxes used in the experiment had either fully cut-out or squared-edge handles, this observation can be explained as most load from the boxes was applied to the participants’ index, middle, ring, and pinky fingers during lifting. However, it has been shown that hand and wrist positions could impact gripping strength [69]. Thus, the inclusion of different types of handles and hand postures could make the model more robust toward the lifting weight component under different task conditions, bringing this initial risk prediction model closer to workplace applications.

Our work demonstrated the feasibility of a non-intrusive method for predicting lifting risks using computer vision and tactile gloves, achieving high accuracy and addressing the limitations of single-modality approaches. Wearing gloves is common [52] and recommended [70] for workplace lifting tasks. Compared to methods that require the placement of multiple IMU or sEMG sensors on the workers [23], [32], [71], the non-intrusive nature of our method enhances its potential applicability and simplifies implementation in workplace settings.

In practical application, the proposed algorithm provides a potential tool for ergonomists and safety professionals in the field of manual material handling (MMH). The integration of computer vision with sensor gloves offers a novel approach to risk assessment. To implement this system, practitioners would equip workers with sensor gloves and place cameras to capture workers’ movements without the need for specialized calibration except for noting worker height, thereby reducing setup complexity. Upon capturing the data, the algorithm processes the information and provides feedback on lifting risk levels that can be used for proactive interventions to prevent musculoskeletal injuries. While using standard cameras is low-cost and easy to implement, the current tactile gloves used in this study are an emerging technology with a higher cost, primarily designed for research purposes. The current research-grade gloves feature 65 sensors per glove, but reduction of sensors (e.g., our feature extraction reduced the 65 sensors into 7 hand regions) and scaling manufacturing may make the technology accessible to more workers once future research establishes the robustness of this approach in industry settings.