For evaluation of each risk equation, we used the CHD event definition used in the study that provided the equations. To evaluate the UKPDS risk equations, we defined a CHD event as: a fatal or nonfatal MI (ICD-9-CM code of 410.xx administrative data; ICD-10 of I21-I22 NDI data); to evaluate the Framingham risk equations, we defined a CHD event as: angina pectoris, MI, coronary insufficiency, sudden death, or CHD death (ICD-9-CM code of 410.xx, 413, 411.89, 414.8 administrative data; ICD-10 of I20-I22 and I46.1 NDI data). For each CHD event, we calculated the “CHD event time” as the time from the TRIAD baseline survey to the first CHD event. Observations were censored at the end of 2003, date of death from another cause, or the date of the first health plan enrollment gap of more than two months, whichever came first.

We evaluated various versions of UKPDS, Framingham initial and Framingham secondary CHD risk equations (Table 1). Specifically, two UKPDS risk equations were evaluated: The first equation predicts the risk of an initial CHD event for a patient with newly diagnosed diabetes and we referred to it as the incident UKPDS risk equation. The second equation takes into account diabetes duration when predicting the risk of an initial CHD event and we thus referred to it as the duration UKPDS risk equation. Framingham risk equations are gender-specific and were thus evaluated separately for men and women. We evaluated the risk equations for predicting an initial CHD event using total cholesterol categories and referred to it as Framingham-initial. We also evaluated the performance of the equations for predicting a subsequent CHD event and referred to it as the Framingham-secondary.

For each eligible participant, we calculated the absolute risk of a CHD event using each equation. Because the racial/ethnic composition of the TRIAD cohort differed from that in the UKPDS cohort, we used the “Afro-Caribbean” risk adjustment for African American patients and the “Caucasian or Asian-Indian” calculation adjustment for the remaining participants. Framingham risk equations were not adjusted for race/ethnicity. Because the Framingham-initial equations were published with the 10-year baseline survival rates, we obtained the 1–5 year baseline survival rates directly from the Framingham investigators.

We evaluated the risk equations for 1) how well they separate individuals who develop a CHD event from those who do not (discrimination) and 2) how close predicted risks are to observed risks [6,31] (calibration, or goodness-of-fit (GOF)). When we examined the performance of the UKPDS and Framingham-initial CHD equations, we only included patients without a CHD history; when we examined the performance of the Framingham-secondary equations, we only included patients with a CHD history.

Discrimination was evaluated using the Harrell’s c-index for censored data (R package HMISC available on CRAN at http://cran.r-project.org), a statistic similar to the area under a receiver operating characteristic curve [32]. In general, a c-index greater than 0.7 indicates good discrimination while a value of 0.5 indicates discrimination equivalent to chance. Intermediate values indicate limited discriminating utility. Calibration plots were generated and Hosmer-Lemeshow-type chi-square statistics [6,33] were calculated to compare differences between predicted and observed risks based on deciles of risk scores. We conservatively defined lack of calibration as chi-square values greater than 23.2 (the 99^th percentile of chi-square distribution with 10 degrees of freedom). We also recalibrated the UKPDS and Framingham risk equations by replacing the average values of predictors and event rates in the original populations by those in the TRIAD population. Specifically, we used the method of D’Agostino et al.[6] to recalibrate the Framingham-initial equations and the method of van Houwelingen [34] to recalibrate the UKPDS and Framingham-secondary equations because the latter were parametric models.

To investigate the difference between study populations with regard to the effect of risk predictors, we fitted each of these equations on TRIAD data and compared the estimates of relative risk (hazard ratio) using the method described in D’Agostino et al. [6]. Specifically, we fitted the Cox regression models and used the same CHD event definition as well as the risk predictors in the original equations. For simplicity, the models using the TRIAD data were all referred to as the TRIAD models. Regression coefficients, hazard ratio (HR) estimates, Harrell’s c-index and GOF statistics [35] were calculated.

Missing data ranged from 1.3% (smoking) to 20.7% (HDL), and was handled in the data analysis using multiple imputation. Imputations were generated using a sequential regression imputation method via the software package IVEware, and results were combined using Rubin’s rule implemented in SAS v9.2 MIANALYZE procedure [36-38].

The follow-up time ranged from 0–3.5 years with a median of 2.7 years (inter-quartile range = 0.9). In the overall TRIAD sample (n = 8303 subjects), there were 319 UKPDS-defined and 592 Framingham-defined CHD events. Among those without a history of CHD (n = 5914), there were 120 UKPDS-defined and 218 Framingham-defined CHD events over the analysis period, with a corresponding 3.5-year CHD event rate (Kaplan-Meier estimate) of 3.0% (95%CI: 2.3%, 3.8%) and 5.1% (95%CI: 4.2%, 6.0%), respectively. Mean age at baseline was 59.8 (SD = 12.3) years with a mean diabetes duration of 10.6 (SD = 9.1) years; 56.7% were female, 39.7% were non-Hispanic White, and nearly 18% were smokers. Of those for whom we had information on medication use and co-morbid burden (n = 4602), 78.4% took oral medication, 26.4% took insulin, 68.7% took hypertension medication, 28% took statins, and 50% had a Charlson score > =2. For those with a prior CHD history (n = 2389), there were 199 UKPDS-defined and 374 Framingham-defined CHD events, with a corresponding 3.5-year CHD event rate of 11.9% (95%CI: 9.6%, 14.2%) and 20.9% (95%CI: 17.9%, 23.9%), respectively. Compared to those without a CHD history, they were generally older with longer diabetes duration and better cholesterol control; they also took more hypertension medication, statins, and suffered from a greater number of comorbid conditions (Table 2).

Evaluation of discrimination and calibration of the UKPDS and Framingham risk equations is summarized in Table 3. The Harrell’s c-index of discrimination for the UKPDS risk equations was generally low: 0.63 (95%CI: 0.58, 0.68) for the incident equation and 0.64 (95%CI: 0.59, 0.69) for the duration equation. The (unadjusted) GOF chi-square value for both UKPDS equations exceeded the established cutoff, indicating a lack of calibration. Specifically, these equations tended to over-estimate CHD risk (Figure1). Similar results were found in our evaluation of Framingham-initial and Framingham-secondary risk equations. Specifically, we found that discrimination was generally low, and so was the calibration except the Framingham-initial equation for women and the Framingham-secondary equation for men. When using these equations, risk tended to be over-estimated for men without a CHD history, and under-estimated for women with a CHD history. After recalibration, the goodness-of-fit of all of the risk equations greatly improved as indicated by the adjusted GOF chi-square statistics that are below the cutoff values (Table 3 and Figure1).

For the TRIAD version of the UKPDS equations (Table 4), only age and systolic blood pressure remained significantly associated with CHD (p<0.05). HRs associated with age, gender, racial difference, log_e(total cholesterol/HDL) and diabetes duration were significantly different, mostly smaller, than those in the original equations (p<0.05). For the TRIAD versions of Framingham initial and secondary equations, almost none of the risk predictors were significantly associated with CHD. Most HR estimates did not significantly differ from those from the original estimates, except for age and HDL for women in the initial equation. However, because the standard errors of regression coefficient estimates for the original Framingham-initial equations were not provided in Wilson et al. [4], thus they were not included in these calculations and the actual p-value might be slightly bigger. The goodness-of-fit of these TRIAD models is generally good (p<0.05), but the discriminating power was low (all the c-indexes were < 0.65).

Our study showed that the UKPDS and Framingham risk equations may be inappropriate for predicting short-term risk of CHD events for adults with long-standing type-2 diabetes. All of these equations exhibited low discriminating power. All except the Framingham-initial equation for women and Framingham-secondary equation for men had low levels of calibration. Our findings were similar to those found in other studies, including van Dieren et al.[9] that evaluated the performance of the UKPDS risk equations by looking at the CHD event risk at 4, 5, 6 and 8 years, and the ADVANCE trial that evaluated the 4-year CHD risk [11]. Separate analyses also showed that the UKPDS and Framingham-initial equations tended to greatly underestimate the risk of a CHD event for patients with a CHD history and discriminating power was consistently low (data not shown).

Several factors may explain our findings of low discrimination and calibration of these equations. First, our study participants had an average of diabetes duration longer than ten years and were community-based health plan enrollees. In contrast, the UKPDS cohort was derived from a clinical trial that only included individuals newly diagnosed with diabetes, while the Framingham cohort only included a small proportion of individuals with diabetes. The general health status, patterns of medication use, and presence of other CHD risk factors in the TRIAD cohort (Table 1) likely differ from those in the previous studies. For instance, at least 68% of TRIAD participants received anti-hypertensive therapy, while fewer than 30% of UKPDS participants and fewer than 10% of Framingham participants had received anti-hypertensive therapy [4,39]. Moreover, the participants in TRIAD were on average 10 years older than those in the UKPDS and Framingham cohorts [3,4]. Risk estimates of these equations may need adjustment when applied to older patients because older patients generally are more susceptible to cardiovascular risks, such as higher blood pressure and declining levels of physical activity. They also tend to have more co-morbid conditions, both physically and mentally; some co-morbid conditions may even lead to non-cardiovascular deaths as competing risks [40-42]. In addition, the racial/ethnic composition of our cohort included greater numbers of non-white patients than the UKPDS or Framingham cohorts. Risk profiles of minority groups not specified by the algorithms (e.g., Hispanics and Asians other than South Asians) may be significantly different. Finally, UKPDS and Framingham risk equations were developed from cohorts formed between the 1970s and the 1990s [3,4]. Treatment of type 2 diabetes and management of cardiovascular risk among diabetic adults have improved substantially since then [43-46].

In investigating the relations of these risk equations with the risk of CHD events in the TRIAD cohort, we found that most predictors were not statistically significant in the TRIAD models. This may be caused by patterns of medication use (e.g., hypertensive drugs, diabetes treatment and statins; Table 1), comorbid conditions, or other factors unaccounted for in the risk equations. Some recent algorithms do include medication as a predictor, such as the prediction model of Donnan et al. [10]. To improve the discriminating power of these models, additional variables, such as medications, family history, life-style related risk factors, socioeconomic status, co-morbid conditions, and novel biomarkers [20,26,45,47,48], may need to be incorporated in the risk equations.

Recent studies suggest that using a “blanket” approach or aggressive risk factor modification (e.g., lowering LDL and/or blood pressure), based on the public notion that diabetes is a CHD risk equivalent, may lead to an overly aggressive treatment and thus offset a patient’s net benefit from treatment [13,14,49]. These authors instead recommend that patients be treated based on appropriate personalized CHD risk estimates. Our analysis showed that the UKPDS and the Framingham-initial equation for men tend to overestimate the initial CHD risk for diabetic patients in a contemporary cohort. Refinement of these equations to reflect the CHD prognostics in a modern diabetic cohort are needed to provide reliable risk estimates to inform effective treatment.

The strength of our study is the large sample size. However, our study has some limitations. The longest follow-up time for CHD events in our study was 3.5 years, thus limiting our ability to evaluate the use of these equations to predict longer-term CHD event risk. With longer follow-up (e.g., 10 years), it is possible that these equations may provide better predictions of CHD risks. However, our study population is more susceptible to CHD than the general population, and the average age of our study population tends to be older (> = 60 years). Evaluating short-term CHD risk in this population can provide useful insights for disease management and treatment. Since CHD events were identified mostly through health plan administrative data, identification of these events may not be complete [50], particularly for patients with “silent” infarction who do not seek health care and thus are not represented in claims data. As a consequence, while the numbers of CHD events represent what a health plan should expect from a typical diabetic patient population, it will miss events that are not clinically recognized.