Smartphone telematics applications are increasingly used to measure risky driving behaviors like speeding and phone use while driving. Many usage-based insurance (UBI) programs rely on these apps to create driver risk profiles and price policy premiums accordingly (Kasperowicz, 2022; Reimers & Shiller, 2019). Safety researchers have also deployed telematics apps in observational studies of risky driving behaviors and in controlled field experiments to improve driver safety (Choudhary et al., 2022; Ebert et al., 2022; Stevenson et al., 2021). Telematics apps, which run in the background of a user’s smartphone and require minimal user effort once installed, have cost and implementation advantages over traditional plug-in and on-board telematics devices—enabling driver safety programs to scale effectively.

A potential drawback is that telematics apps measure trip activity regardless of whether the smartphone user was driving. To correctly attribute risky driving behaviors to the user, an app must accurately predict whether they were driving on a given trip. This prediction problem is typically solved with deep-learning algorithms that recognize patterns in smartphone sensor (e.g., accelerometer) data or extracted features (e.g., braking) when the user is driving (Park, 2018). A recent review of studies examining the accuracy of driver identification algorithms found that the best ones achieve accuracy of 85% or greater (Zhao et al., 2022). However, these studies looked at algorithms trained on rich on-board sensor data and tasked with identifying which driver out of a small sample (2 to 25 drivers) was driving (Abdennour et al., 2021; Cai et al., 2018; Choi et al., 2022; Girma et al., 2019; Jeong et al., 2018; Ravi et al., 2022; Van Ly et al., 2013). Research examining the accuracy of algorithms that rely solely on smartphone sensor data to predict whether a given user was driving would advance the field of traffic safety.

The present study set out to test the accuracy of a driver/non-driver classification algorithm widely used in smartphone telematics apps. The primary outcome was overall accuracy. Secondary outcomes were sensitivity (proportion of actual driver trips classified as such) and specificity (non-driver trips classified as such). We also investigated the impact of inaccurate classifications on drivers’ risk profiles, as well as smartphone and environmental characteristics that could plausibly affect accuracy.

This was a prospective, preregistered field study (NCT05422586) recruiting from a sample who had previously participated in a national trial of behavioral interventions to encourage safer driving in a UBI program (NCT04587609) and indicated openness to future research. Recruitment was conducted by email the week of July 11, 2022.

Participants consented and completed an intake survey about their smartphone, vehicle features, and driving habits in Qualtrics. To reduce participant burden, demographic characteristics from the parent trial were used to describe this sample. Participants were invited by text and email to install the University of Pennsylvania’s Way to Drive app, which measured vehicle trips for 4 weeks. Way to Drive is a white-label version of a telematics app developed by TrueMotion, Inc. (now Cambridge Mobile Telematics). It uses the same driver classification and risky behavior detection algorithms (v2.3.0) as leading UBI apps. Driver classification is accomplished with a deep-learning algorithm that recognizes driving patterns characteristic of the user (Park, 2018).

On the Sunday morning after each week, participants were sent a survey to review trips within the app. Tapping on a given trip displayed its start time, a map of the route taken, and the driver/non-driver classification. For each day of the week (reverse-chronological order), the survey asked about:

Number of trips the app detected

Of the 400 solicited, we arrived at an analytic sample of 57 participants (Fig. 1). Demographic characteristics are shown in Table 2.

Self-report data about whether there was a misclassification were available for a mean of 24.9 (SD = 3.5) of the 28 days. On these days, the app classified a mean of 120.9 (SD = 69.8) trips as driver and 26.5 (SD = 19.5) as non-driver. Participants reported a small number of false positives (M = 2.5, SD = 5.6) and false negatives (M = 2.3, SD = 4.4). Subtracting false positives from trips classified as driver yielded a mean of 118.4 (SD = 69.8) true positives. Subtracting false negatives from trips classified as non-driver yielded a mean of 24.2 (SD = 18.1) true negatives. The primary and secondary outcomes are shown in Fig. 2. Overall accuracy was high (M = 96.5%, SD = 5.1%). Sensitivity (M = 97.5%, SD = 4.6%) was greater and less variable than specificity (M = 91.2%, SD = 14.8%).

Participants as a group were high on our reliability index (M Pearson’s r = 0.95). When app accuracy was regressed onto r², the coefficient for r² was small (−0.034) and nonsignificant (P = 0.292). Adding this coefficient to the intercept (0.996) projected app accuracy to be 0.962 for a hypothetical participant with perfect reliability, in line with mean accuracy for the sample.

Unadjusted (M = 74.5, SD = 5.1) and adjusted (M = 74.4, SD = 5.2) driver risk scores were very similar (P = 0.166). Across participants the correlation between unadjusted and adjusted risk scores was essentially perfect (r = 1.00). These results suggest that occasional trip misclassifications do not invalidate drivers’ overall risk profiles.

No vehicle features or driving habits significantly predicted accuracy (all Ps > .26). Having a phone in “low power” mode while driving 1–2 days in the prior 2 weeks was significantly associated with lower app accuracy (5.3 percentage points, P < .005), and having a phone at least 3 years old was marginally associated with lower accuracy (5.2 percentage points, P = .081). No other smartphone characteristics predicted accuracy (all Ps > .12).

The driver/non-driver classification algorithm used by Way to Drive and other smartphone telematics apps achieved accuracy comparable to the best-performing driver identification algorithms that take on-board sensor data as input (Zhao et al., 2022). Accuracy was lower for phones that occasionally went into low power mode while driving; otherwise, the algorithm was robust to varying phone characteristics, vehicle features, and driving habits. The relatively small number of misclassifications did not significantly affect drivers’ overall risk profiles, which should give insurers, researchers, and users greater confidence when using apps that rely on this classification algorithm to attribute risky behaviors to the user.

Sensitivity was greater than specificity, indicating a bias toward classifying trips as driver. When the algorithm is used with populations that have a smaller proportion of driver trips, we can expect overall accuracy to be lower. Based on the present study’s sensitivity and specificity, a user with a 50/50 driver/non-driver split may see accuracy of 94.4%; a user with a 25/75 split may see accuracy of 92.8%. This decreased accuracy may not substantially alter a user’s overall risk profile, but it could lead to lower user acceptance. For apps that let users correct misclassifications, encouraging infrequent drivers to make corrections could improve acceptance and classification accuracy.

A limitation of this study is its reliance on participants’ reports of misclassifications instead of “ground truth” data (e.g., coded video footage from participants’ vehicles). We chose this method because it was less invasive, costly, and effortful. Allaying some concern about inaccurate or incomplete self-reports, participants reliably reported the number of trips their apps detected, and the most reliable participants reported a high level of app accuracy.

Smartphone telematics apps hold great promise for measuring and improving driving behavior at scale. To be trusted, these apps must attribute behaviors to a user when the user is driving, and not attribute behaviors when the user is not driving. Correct attribution depends on the accuracy of driver/non-driver classification algorithms. Our results confirm that this classification can be done with minimal error using smartphone telematics data, and without user input or additional in-vehicle measurement devices.