In this paper, we explore the use of a tree-based classifier known as XGBoost to classify wear/non-wear epochs¹⁵. Wear is labeled as 1, and non-wear has been labeled as 0 from the Spectrum data. Mimicking the data of the closely related and commonly used Actiwatch 2 model (which has no non-wear sensor), the two key inputs taken from the actiwatch data are the white light and the activity levels. A correlation study between the different light signals confirmed that the white light correlated at least 0.9 with all other light signals; hence, only the white light level was taken as input. To leverage information concerning short- and long-term time dependencies, dynamical features were extracted using the tsfeatures package in R^16,17. Dynamical features created using the tsfeature package are spectral entropy, maximum level shift, maximum variance shift, maximum Kulback-Leiber divergence, hurst, spike, heterogeneity, the standard deviation of the first derivative of the time series, and activity lags. Spectral entropy, hurst, heterogeneity, and standard deviation of the first derivative of the time series capture if the time series is random/noisy or if it is stable and has a constant trend. Maximum level shifts and spikes keep track of any sudden changes in the time series. Maximum Kulback-Leiber divergence and maximum variance level shift detect any changes in data distribution in consecutive time windows. Dynamical features were created to provide statistical information to the XGBoost model and improve its ability to discern trends or patterns in the time series. The features have been made on the activity levels with a window size of 20 minutes and 1 hour. The 20-minute window captures shorter naps, and the 1-hour window size captures the average nap duration¹⁸. Furthermore, activity lags were created over 5 minutes, 10 minutes, 15 minutes, and 20 minutes. Next, we processed the time series to generate several temporal features such as minute, hour, quarter, day of the week, and time of the day. This information helps to interpret the timestamp in a useful representation, such as if the time of the day is night or the hour is early morning, then the person is more likely to be asleep. In addition, for each participant, the temporal features were converted into sinusoidal curves to capture prominent periodicities in the data. For each participant, employer information was added as a binary variable (1: extended care and 0: information technology). After processing each participant’s data, the data were combined to form a larger dataset. This dataset was split into a 70:30 ratio to create the training and test sets. The random split was formed such that the training set has 597 unique participants and the test set has 256 unique participants.

The parameters for the XGBoost model were determined by using hyperparameter optimization and cross-validation. The Hyperopt Python library¹⁹ was used to efficiently explore a range of hyperparameter values for the XGBoost model (see Fig. 1). The parameters were optimized on the average balanced accuracy across five folds of cross-validation training data. The initial ranges of hyperparameter values were expanded and re-evaluated when any hyperparameter value was observed near the search space’s boundaries. The range of the search space of the parameters is given in Table 1. The final optimized hyperparameter values and parameter descriptions are summarized in Table 2. We used the histogram tree method, which buckets continuous features into bins to yield approximate tree-splitting solutions to enhance computational speed²⁰. To further speed up the algorithm gbtree booster and GPU Quadro K4000 were used. Here we tuned the scale_pos_weight as part of the hyperparameter values to be optimized via Hyperopt to allow for heavier weighing of the rare instances of non-wear under imbalanced data (i.e., with the inflated presence of wear data) in the loss function²¹. The two regularization parameters helped reduce overfitting. From Fig. 1. We can see that after fitting the XGBoost model to the training set, we used Youden’s J statistic²² to select the probability threshold for classifying data as non-wear under imbalanced data optimal threshold was found and computed the final average balanced accuracy across the validation folds. The entire modeling process is shown in Fig.1. Once we got the classification results from the XGBoost model based on the optimum parameters, we manually changed the probability threshold to get the final performance metrics on the training dataset. We manually changed the threshold instead of relying on Youden’s J statistic because Yoden’s J statistic increased the specificity at the expense of Negative Predictive Value ( NPV). The effects of using different probability thresholds for non-wear classification on the performance metrics are shown in Fig 2. We observe that at probability threshold 0.55, there is a balance between the specificity and NPV value. Increasing or decreasing the probability threshold would lead to unequal tradeoffs in the performance metrics; hence, 0.55 was set as the optimum probability threshold for our dataset.

To further improve our model, we employed a temporal smoothing function to address abrupt (and rare/unlikely 1-epoch) changes in the output at an epoch level. This smoothing function checks neighborhood labels and assigns the label that has the majority in that time window. For example, in a window size of 3 epochs, if there are two non-wear labels and one wear label, all epochs are labeled as non-wear. A range of window sizes from 3 to 20 epochs were used to find the optimal window size. The optimal window size was found to be 3 for the training dataset. This window size gave the maximum specificity; as we increased the window size, specificity decreased.

The main aim of this paper was to distinguish non-wear epochs in actigraphy data. The XGBoost model provided probabilities at an epoch level and we determined the optimal probability threshold level based on different classification metrics. The classification metrics studied in the paper are sensitivity, specificity, balanced accuracy, F1 score, positive predictive value (PPV), and negative predictive value (NPV). Area under the receiver operating characteristic curve (AUC) was computed for the initial set of results (Table 3). All other results are based on the optimal probability threshold.

To further evaluate our methodology, we compared our result with a published methodology addressing the same general problem⁵, which showed that 95% of the inactive episodes are of length shorter than 20 minutes in an Actiwatch 2 with a resolution of data of 60s. To create these non-wear /inactive episodes automatically, they used R studio to label any consecutive epochs with zero activity counts as an episode of non-wear. The episodes were then visually labeled as wear and non-wear. They then generated predictions on the episodes by calculating the length of each episode and classifying it as non-wear if the episode had a length of more than 20 minutes. The optimal time threshold was computed from the receiver operating curve. Following their methodology, we derived an optimal threshold of 1 minute, which gave the highest accuracy in detecting non-wear episodes in our dataset. They used activity levels to classify wear/non-wear, and we followed similar approaches, adding dynamic features and white light to predict non-wear and wear epochs.

In this paper, using data from 853 participants, 597 in the training dataset, and 256 in the test dataset. Each participant contributed ~ 20,000 epochs. Sensitivity refers to correctly predicting wear epochs whereas specificity gives the performance of the model on the non-wear epochs. The results from the optimum XGBoost model (Table 3) show that the balanced accuracy in the training is 0.881 and 0.876 in the test set. Due to the class imbalance in the data, we have a high sensitivity of 0.991 as opposed to a specificity of 0.771 in the training set. To balance this, we reclassify the probabilities generated by the XGBoost model using 0.55 as the new threshold and add the temporal smoothing function. The updated results are shown in Table 4, here, we can see that our specificity rises to 0.822 in the training data. The specificity is lower in the test at 0.791 but is still comparable to the training accuracy. Thus, we do not overfit or underfit our model. The sensitivity decreases from 0.991 to 0.985 in the training set but is still high enough to be determinantal in classifying wear time points. The NPV comes down when we use the probability threshold, but that is the best tradeoff we found in the model against increasing the specificity. The results of the comparison study can be seen in Table 4. We can see that our proposed model outperforms the temporal threshold method on every classification metric. This evaluation supports the claim that machine learning algorithms such as XGBoost can be an automation tool to classify wear and non-wear.

To validate the model further, we matched the invalid days in the dataset against the invalid days predicted by our algorithm. A day is defined as invalid if the person does not wear the actiwatch for more than 4 hours. We label invalid days as 1 and valid days as 0. From Table 5, we can calculate that we are predicting valid days with an accuracy of 0.997 and invalid days with 0.921 accuracy in the training dataset. We get comparable accuracy in the test set, as can be seen in Table 5. Table 6 shows a comprehensive list of classification metrics for predicting invalid and valid days. Most of the metrics are above 0.9; hence, our proposed methodology does not lead to a mismatch between valid and invalid days.