The seven health systems participating in this research (HealthPartners; Henry Ford Health System; and the Colorado, Hawaii, Northwest, Southern California and Washington regions of Kaiser Permanente) serve a combined population of approximately eight million members in nine states. Each system provides insurance coverage and comprehensive health care (including general medical and specialty mental health care) to a defined population enrolled through employer-sponsored insurance, individual insurance, capitated Medicaid or Medicare, and subsidized low-income programs. Members are representative of each system’s service area in age, race/ethnicity, and socioeconomic status. All systems recommend using the PHQ-9 at mental health visits and primary care visits for depression, but implementation varied across systems during the study period.

As members of the Mental Health Research Network, each health system maintains a research data warehouse following the Health Care Systems Research Network Virtual Data Warehouse model¹⁴. This resource combines data from insurance enrollment records, EHRs, insurance claims, pharmacy dispensings, state mortality records, and census-derived neighborhood characteristics. Responsible institutional review boards for each health system approved use of these de-identified data for this research.

The study sample included any outpatient visit by a member aged 13 or older either to a specialty mental health clinic or to a primary care clinic when a mental health diagnosis was recorded. Sampling was limited to visits to health system clinics (to ensure availability of EHR data) and people insured by the health system’s insurance plan (to ensure availability of insurance claims data). All qualifying visits from 1/1/2009 through 6/30/2015 were included, except at Henry Ford where only visits after implementation of a new EHR system (12/1/2012) were included.

Potential predictors extracted from health system records for up to five years prior to each visit included: demographic characteristics (age, sex, race, ethnicity, source of insurance, and neighborhood income and educational attainment), current and past mental health and substance use diagnoses (organized in 12 categories), past suicide attempts, other past injury or poisoning diagnoses, dispensed prescriptions for mental health medication (organized in four categories), past inpatient or emergency department mental health care, general medical diagnoses (by Charlson Comorbidity Index¹⁵ categories), and recorded scores on the PHQ-9 questionnaire¹⁶ (including total scores and item 9 scores).

Potential predictors were represented as dichotomous indicators. Each diagnosis category was represented by three overlapping indicators (recorded at or within 90 days prior to the visit, recorded within one year prior, and recorded within five years prior). Each category of medication or emergency/inpatient utilization was represented by three overlapping indicators (occurred within 90 days prior to the visit, one year prior, or any time prior). To represent temporal patterns of prior PHQ-9 item 9 scores, 24 indicators were calculated for each encounter to represent number of observations, maximum value, and modal value (including value of missing) during three overlapping time periods (previous 90 days, previous 183 days, and previous 365 days). The final set of potential predictors for each encounter included 149 indicators and 164 possible interactions (see Appendix 9a for complete list).

Diagnoses of self-harm or probable suicide attempt were ascertained from all injury or poisoning diagnoses recorded in EHRs and insurance claims accompanied by an ICD-9 cause of injury code indicating intentional self-harm (E950-E958) or undetermined intent (E980-E989). Data from these health systems during the study period indicate that inclusion of injuries and poisonings with undetermined intent increases ascertainment of probable suicide attempts by approximately 25%⁷ (see also Appendix 4). While use of E-codes varied across the US during the study period¹⁷, participating health systems were selected for high and consistent rates of E-code use (Appendix 1). Records review⁷ also supports the positive predictive value of this definition for identification of true self-harm in these health systems (see also Appendix 2). Furthermore, observation of coding changes across the transition from ICD-9 to the more specific ICD-10 coding scheme indicate that most “undetermined” ICD-9 diagnoses actually reflect self-harm¹⁸ (see also Appendix 3). Ascertainment of suicide attempts was censored at health system disenrollment, after which insurance claims data regarding self-harm diagnoses at external facilities would not be available.

Suicide deaths were ascertained from state mortality records. Following common recommendations^19,20 all deaths with an ICD-10 diagnosis of self-inflicted injury (X60-X84) or injury/poisoning with undetermined intent (Y10-Y34) were considered probable suicide deaths. Inclusion of injury and poisoning deaths with undetermined intent increases ascertainment of probably suicide deaths by 5–10%⁷ (see also Appendix 4).

All predictor and outcome variables were completely specified and calculated prior to model training.

Prediction models were developed separately for mental health specialty and primary care visits, with a 65% random sample of each used for model training and 35% set aside for validation. Models included multiple visits per person in order to accurately represent changes in risk within patients over time. For each visit, analyses considered any outcome in the following 90 days, regardless of a subsequent visit in between. This approach uses all data available at the time of the index visit, but avoids informative or biased censoring related to timing of visits following the index date. In the initial variable selection step, separate models predicting risk of suicide attempt and suicide death were estimated using logistic regression with penalized LASSO variable selection²¹. The LASSO penalization factor selects important predictors by shrinking coefficients for weaker predictors toward zero, excluding predictors with estimated zero coefficients from the final sparse prediction model. To avoid over-fitting models to idiosyncratic relationships in the training samples, variable selection used 10-fold cross-validation²² to select the optimal level of tuning or penalization, measured by the Bayesian Information Criterion²³. In the second calibration step, generalized estimating equations with a logistic link re-estimated coefficients in the training sample, accounting for both clustering of visits under patients and bias toward the null in LASSO coefficients. In the final validation step, logistic models derived from the above two-step process were applied in the 35% validation sample to calculate predicted probabilities for each visit. Results are reported as receiver operating characteristic (ROC) curves²⁴ with c-statistics^25,26 along with predicted and observed rates in pre-specified strata of predicted probability. Over-fitting was evaluated by comparing classification performance and in training and validation samples and by comparing predicted risk to observed risk in the validation sample. Variable selection analyses were conducted using the GLMNET²⁷ and Foreach²⁸ packages for R statistical software, version 3.4.0. Confidence intervals for c-statistics were calculated via bootstrap with 10,000 replications.

A public repository (www.github.com/MHResearchNetwork) includes: specifications and code for defining predictor and outcome variables, a data dictionary and descriptive statistics for analytic datasets, code for variable selection and calibration steps, coefficients and confidence limits from all final models, and comparison of model performance in training and validation samples.

We identified 19,961,059 eligible visits by 2,960,929 patients during the study period, including 10,275,853 mental health specialty visits and 9,685,206 primary care visits with mental health diagnoses (Table 1). Following the specifications above, health system records identified 24,133 unique probable suicide attempts within 90 days of an eligible visit, and state mortality records identified 1240 unique suicide deaths within 90 days.

Models predicting probable suicide attempt over 90 days were developed and validated for both mental health and primary care visits, excluding 0.3% of visits because of disenrollment within 90 days. Clinical variables with the largest positive prediction coefficients are shown in the left portion of Table 2 (see Appendices 9b and 9c for all selected predictors and coefficients). Strongest predictors of suicide attempt were similar in mental health specialty and primary care patients: prior suicide attempt, mental health and substance use diagnoses, responses to PHQ-9 item 9, and prior inpatient or emergency mental health care.

The left portion of Figure 1 shows ROC curves illustrating sensitivity and specificity of suicide attempt predictions in training and validation samples. C-statistics (equivalent to AUC or area under the ROC curve) for prediction of suicide attempt in the validation samples were 0.851 (95% CI 0.848 to 0.853) for mental health specialty visits and 0.853 (95% CI 0.849 to 0.857) for primary care. In each graph, comparison of ROC curves shows no appreciable difference in prediction accuracy between the training and validation samples (i.e. no evidence for model over-fitting). Table 3 compares predicted to observed risk for specific strata selected a priori. Among mental health specialty visits, the lowest two strata included 75% of all visits and 21% of all suicide attempts, while the highest three strata included 5% of visits and 43% of suicide attempts. Among primary care visits, the 75% of visits with lowest risk scores accounted for 21% of suicide attempts, while the 5% of visits with highest scores accounted for 48%. Comparison of predicted risk levels in the training sample and observed risk levels in the validation sample again shows no appreciable decline in model performance or evidence for model over-fitting. Sensitivity analyses limited to diagnoses of definite self-harm slightly improved prediction accuracy (especially among primary care patients) but excluded approximately 25% of probable suicide attempts (Appendix 4). Sensitivity analyses limited to visits preceded by at least 5 years of complete data yielded essentially identical prediction accuracy (Appendix 5). Model fit was consistent across the seven participating health systems and across age and sex subgroups (Appendix 8)

The same process was implemented for prediction of suicide deaths over 90 days, with separate models for mental health specialty and primary care visits. Clinical variables most strongly associated with suicide death in each group are shown in Table 2 (see Appendices 9d and 9e for complete list). Predictors of suicide death were similar in mental health specialty and primary care patients, and were similar to predictors of suicide attempt.

The right portion of Figure 1 shows ROC curves for prediction of suicide death in training and validation samples. C-statistics for prediction of suicide death in the validation samples were 0.861 (95% CI 0.848 to 0.875) for mental health specialty visits and 0.833 (95% CI 0.813 to 0.853) for primary care. Comparison of ROC curves for the training and validation samples shows no evidence of over-fitting in the mental health specialty sample and a minimal separation of training and validation curves in the primary care sample. The right portion of Table 3 compares predicted to observed risk for risk strata selected a priori. Among mental health specialty visits, the lowest two risk strata included 75% of visits and 19% of suicide deaths, while the highest three risk strata included 5% of visits and 48% of suicide deaths. Among primary care visits, the 75% of visits with lowest risk scores accounted for 25% of suicide deaths, while 5% of visits with highest scores accounted for 43%. Comparison of predicted risk levels in the training sample and observed risk levels in the validation sample shows no evidence for over-fitting in the primary care sample and a minimal fall-off between training and validation samples in the primary care sample. Sensitivity analyses limited to deaths coded as due to definite self-inflicted injury or poisoning found no meaningful difference in model fit (Appendix 4).

Table 4 displays sensitivity, specificity, positive predictive value (PPV), and negative predictive value for all four models at cut-points defined by percentiles of the risk score distribution.

In a sample of 20 million visits by 3 million patients in seven health systems, data from EHRs accurately stratified mental health specialty and primary care visits according to short-term risk of suicide attempt or suicide death. Observed rates of probable suicide attempt and suicide death were over 200 times as high following visits in the highest 1% compared to visits in the bottom half of predicted risk (Table 3). Strongest predictors included mental health diagnoses, substance use diagnoses, use of mental health emergency and inpatient care, and history of self-harm. Absolute risk was lower in primary care, but predictors selected and accuracy of prediction were similar across care settings. Responses to PHQ-9 questionnaires were selected as important predictors, even though such data were available for only 15% of visits.