Within the past two decades, growing interest in breast cancer risk prediction has stimulated the development of several risk models. These models are used to identify women at high risk of cancer who might benefit from targeted screening or chemoprevention, to estimate population burden, and to assist physicians and patients in clinical decision-making. Yet validation studies of these models have shown variability in calibration and discrimination when applied to differing populations. Many of the models have significantly underestimated risk in women who are nulliparous or whose first live birth was over the age of 30 as well as in other high-risk populations.[1–4] Risk models are best calibrated to the population on which they were developed, and are typically developed in general populations to enhance the applicability of the model to outside populations. Thus, the models may have lower performance in populations with high numbers of either high or low risk women.

The County of Marin, just north of San Francisco, has the highest rates of nulliparity and age at first birth over 30 of all California counties, and has for many years had among the highest breast cancer rates of any county in the United States.[5–8] It has been postulated that at least part of the explanation for these increased rates is the delayed childbirth seen in Marin women.[5,8] Of respondents in the Marin Women’s Study, a mammography-based study of women in Marin County, 57.6% have not had a child by the age of 30. According to a report by the CDC in 2009, the average age of first birth has increased from 21.4 to 24.0 in the United States during the period of 1970 to 2006, emphasizing the importance of accuracy of existing models in this population.[9] This trend is not just limited to the United States, and was seen in all developed countries studied, with the United States actually having the lowest age of first birth of all developed countries. According to a report from Pew Research, from 1990 to 2008 the percentage of children born to mothers 35+ has increased from 9 to 14%, and has tripled in women 40+.[10]

This study was done as a retrospective cohort within the Marin Women’s Study (MWS), and all women without breast cancer as of January 1, 2003 were selected to compare performance of three different risk prediction models in a 5-year follow-up period. The performance of the risk prediction models was assessed using two criteria: calibration and discrimination. Calibration is a measure of the ability of a model to accurately predict the number of events in a population. Discrimination measures the model’s ability to discriminate at the individual level between women who will and will not develop breast cancer, and is measured by calculating the area under the receiver operating characteristic curve, or AUC. An AUC of 0.5 identifies a model whose discriminatory accuracy is no better than the toss of a coin, whereas an AUC of 1.0 identifies a model with perfect discriminatory accuracy.

Included in the current analysis are three of the most widely used models, the Breast Cancer Risk Assessment Tool (BCRAT, or Gail model),[11] BRCAPRO[12] and IBIS Breast Cancer Risk Evaluation Tool (Tyrer-Cuzick model).[13] These models are tested for calibration and discrimination in participants of the Marin Women’s Study to assess the performance of these models in women known to have high rates of nulliparity and delayed childbirth.

This study was approved by the Marin General Hospital Institutional Review Board, as well as the Kaiser Permanente Northern California Institutional Review Board, and all participants provided informed consent to fully participate in the study. MWS questionnaire data was collected between years 2006 and 2009, the reference baseline was set at the start of 2003, and the population of interest was restricted to women who were breast cancer free at this baseline. The outcome was defined as occurrence of any invasive breast cancer between 2003 and 2007 either reported by women on the questionnaire or included in the cancer registry data obtained from the SFMR.

Analysis of the study population was performed on data from 12,843 women who were breast cancer free at study baseline (beginning of 2003). During the course of the 5-year follow up period, 203 breast cancers were diagnosed. Table 2 shows the distribution of risk factors in participants.

Calibration results are displayed in Table 3, which includes data on expected and observed number of cases in the MWS population for the BCRAT, BRCAPRO and IBIS models. The BCRAT model predicted 165 breast cancer cases while 203 were observed. With an E/O ratio of 0.81 (95%CI 0.71– 0.93), this model underestimated absolute risk in our population. The BRCAPRO model predicted only 121 breast cancers compared to the 203 observed, generating the lowest E/O ratio of 0.59 (95%CI 0.52 – 0.68). The IBIS model fared the best, predicting 221 cases, with an E/O ratio of 1.09, which was not significantly different from 1 (E/O=1.09, 95% CI=0.95–1.25), while the BCRAT and BRCAPRO models had ratios significantly less than one, indicating that on average these models underestimated absolute risk in the MWS population.

We then compared the expected and observed number of cases within deciles of predicted risk for each model, using a two-sided χ²-goodness of fit test with 9 degrees of freedom (Table 3). As with the overall test of calibration reported above, we fail to reject the null hypothesis that there is a significant difference between the expected and observed number of cases within deciles of predicted risk for the IBIS model (χ²=8.64, p-value=0.47), while this is not the case for the BCRAT and BRCAPRO models.

Figure 1 further illustrates the observed and the expected proportion of cases for each decile of predicted risk in the three models. From Figure 1 it can be seen that BCRAT appears to under-predict in women with slightly higher than median risk. On the other hand, BRCAPRO can be seen to most significantly under-predict in women in the highest deciles of risk. The IBIS model shows a relatively even prediction pattern across all deciles.

Further analysis was conducted to calculate expected to observed ratios for subgroup categories defined by parity and age of first birth (Table 4). The results show that the BCRAT and BRCAPRO models underestimate risk to a greater degree for nulliparous women, with BCRAT having an E/O of 0.64 (95% CI 0.50–0.84) and BRCAPRO 0.48 (95% CI 0.37 −0.63), while the E/O for the IBIS model in nulliparous women was 0.98 (95% CI 0.75 – 1.27). Among parous women, all models performed worse in terms of calibration for women with an age at first birth over 30, although these differences were not statistically significant.

In terms of the discriminatory accuracy of the BCRAT, BRCAPRO and IBIS models in the overall MWS population, the highest AUC achieved was 0.65 (95% CI 0.61 – 0.68) by the IBIS model. The BCRAT (0.62, 95% CI 0.59 – 0.66) and BRCAPRO (0.60, 95% CI 0.56 – 0.63) models had slightly lower discrimination, but all showed an area under the curve significantly greater than 0.50, and overall the confidence intervals overlap for the three models. These results are displayed graphically in the ROC curves shown in Figure 2 (supplementary online figure).

When looking at the discriminatory performance of these models in nulliparous women, the IBIS model performed best with an AUC of 0.65 (95% CI 0.58 – 0.70), with the BCRAT having a slightly lower AUC of 0.65 (95% CI 0.62 – 0.72), and the BRCAPRO model an AUC of 0.60 (95% CI 0.54 – 0.65). Among parous women with age first birth over 30, the IBIS model had the highest AUC of 0.69 (95% CI 0.62 – 0.75), compared to BCRAT 0.63 (95% CI 0.56 – 0.70), and BRCAPRO 0.62 (95% CI 0.56 – 0.68).

We used data from the Marin County mammography population to examine the calibration and discrimination of three risk prediction models for invasive breast cancer: BCRAT, BRCAPRO, and IBIS. Until recently, only the BCRAT model had been independently validated in populations in the United States, and it has shown inconsistent performance in terms of calibration and discrimination. A 2012 study in the New York Breast Cancer Family Registry compared Gail and Tyrer-Cuzick models and found Tyrer-Cuzick demonstrated both better calibration and discrimination (25+). Gail was very poorly calibrated in their population in women who were nulliparous or who had age of first live birth >25. Furthermore, there is limited evidence that any of the existing risk assessment models have been adequately tested in high risk populations.

Although the IBIS model over-predicted risk by 9 percent (E/O = 1.09), it demonstrated better overall calibration than the other two models (BCRAT= 0.81 and BRCAPRO = 0.59). One reason for this may be that the IBIS model utilizes information on hormone replacement therapy, which is not utilized by the other two models. Combination hormone replacement therapy use in Marin County was shown to rank fourth highest of 41 California counties in 2001 and thus may be an important risk factor in the Marin population.[7] Based on our analysis, the BCRAT and BRCAPRO models were not well calibrated for the Marin County mammography population and underestimated risk for the given time period. Our results for calibration of the BCRAT model are contrary to some validation studies which have revealed good calibration of the model [26], yet poor calibration has been shown in many other studies.[2,19,27] Perhaps the largest study, done in the Women’s Health Initiative cohort, demonstrated a similar overall calibration of 0.79 (Chlebowski et al.). The BRCAPRO model may not have performed well in this population because we were lacking much of the family history data that serve as inputs for this model. Breast cancer history in second degree relatives, number of unaffected relatives, family history of ovarian cancer, history of bilateral breast cancer, or history of breast cancer in males could not be included for this model. In addition, the high prevalence of delayed childbirth in the Marin population is not reflected in the scores from this model.

Calibration of the models within strata of parity and age of first birth revealed a similar pattern to the overall calibration of the models. The BCRAT model was not well calibrated for nulliparous women (E/O = 0.64) and women with age of first birth after 30 (E/O = 0.81). The BRCAPRO model also consistently underestimated risk in the subgroups defined by parity and age of first birth. On the other hand, the IBIS model had the best calibration within these subgroups with none of the E/O ratios significantly different from 1. The results of the subgroup analysis suggest that the poor performance of these models may be that the populations used to construct the BCRAT and the BRCAPRO models differed significantly in terms of parity and age of first birth. Our results are consistent with a study in the United Kingdom which also found that the BCRAT and BRCAPRO models underestimated risk in women who are either nulliparous or whose first live birth occurred after the age of 30.[3] Breast cancer risk models are calibrated to the populations they were designed on, and both the BCRAT and BRCAPRO models were calibrated on US populations, while the IBIS model was calibrated to populations in England and Wales. According to the Office for National Statistics in the United Kingdom, age of first birth in England and Wales is much later than that in the United States, and in the year 2008 47.3% of parous women had an age of first birth over 30, which better approximates the Marin County population in which 44.2% of parous women in 2008 had an age of first birth over 30.

Some applications of risk assessment models such as deciding appropriate screening procedures or interventions require good discriminatory power. In term of discrimination, the IBIS model achieved the highest estimated AUC of 0.65, compared to BCRAT (0.62) and BRCAPRO (0.60). The apparently higher AUC estimate for the IBIS model is in agreement with the previously noted United Kingdom study evaluating the performance of breast cancer risk models.[3] However, the 95% confidence intervals obtained from bootstrap are wide and overlapping, and a larger sample is required for more precise conclusions. It should be pointed out that all of these models would be considered to have only fair discriminatory power, and further continuing efforts to improve the performance of these models are clearly needed.

It is surprising that the BRCAPRO model performed as well as it did with an AUC of 0.60 in this population, considering that it is based solely on basic family history data. This could be because of the high rate of positive family history in first degree relatives seen in the MWS cases (unpublished MWS analysis). In regards to discrimination, all models performed about the same in nulliparous and parous women, but among parous women the IBIS model performed best in women with age first birth over 30.

A potential limitation of the study is the possibility of selection bias in the mammography cohort. Women with a history of breast cancer may have been more likely to have chosen to participate in the study. The yearly incidence rate of breast cancer in this study was 316.1 per 100,000. Applying rates seen in Marin County as a whole in White non-Hispanic women obtained from SEER in the years 2003–07 to the age distribution in this study yields an expected rate of 280.6 per 100,000. Thus the rate seen is in our study is 12.6% higher than that expected in Marin overall for the age distribution of the study. This could have resulted in a lower calibration of the models, but would not be expected to have a differential effect.

An additional limitation of the study was that assumptions had to be made about baseline risk factors. Number of first-degree relatives with breast cancer and BMI are assumed to be the same as they were at the time of the questionnaire. It is unlikely that the number of first-degree relatives with breast cancer increased for many women during the period between the baseline (2003) and the time of the questionnaire (2007–8). And substituting a higher number of family relatives with breast cancer in 2007 for the number at baseline in 2003 into the models would result in an increase in estimation of 5-year risk by each of the models (assuming family history of breast cancer increases risk) and does not explain the underestimation of risk by the BCRAT and the BRCAPRO models. BMI was incorporated only into the IBIS model, and using BMI at the time of the questionnaire can result in a similar bias. However, if the average weight of the women in our population stayed the same between 2003 and the time of the questionnaire, the calibration of the IBIS model would not be affected. Data from the California Health Information Survey from 2005 reveals that Marin County has the lowest incidence of obesity of any county in California, as well as demonstrating only a minimal increase in BMI in its women over a 5-year time period, suggesting a minimal impact of this limitation. Women who are diagnosed with breast cancer are known to be more likely to gain than lose weight in the subsequent few year. However, the amount is relatively low and would not be expected to have a major effect on the risk scores assigned by the IBIS model, but certainly could have resulted in a slight positive bias in discrimination for this model.

Based on our comparison of the BCRAT, BRCAPRO, and IBIS models, it appears that the BCRAT and the BRCAPRO models significantly underestimate risk for the Marin County population, which has a high rate of nulliparity and late age of first live birth. The two models did better in terms of discriminatory ability, but also had a lower discriminatory ability than the IBIS model. With the trend in society towards delayed childbirth, further assessment of existing breast cancer risk models in populations similar to that of Marin County should be performed.

Since the IBIS model appeared to be best calibrated and had the best discriminatory capability in our population, it may be the preferred model for estimation of breast cancer risk in women with risk factor profiles similar to those of Marin County.

Health professionals depend on accurate assessment of breast cancer risk to make decisions about potential interventions for women at very high risk, as well as determining optimum screening for individual women such as mammogram frequency and use of other tests such as Magnetic Resonance Imaging (MRI). Unfortunately existing models have limitations in their ability to accurately identify women at high risk, and have even less accuracy in populations that are different from those on which the model was initially developed.

Many women are delaying childbirth for personal, career, and financial reasons placing them at higher risk for breast cancer, and existing models have demonstrated uncertain accuracy in these women. Our study applies three existing models to a population with high rates of nulliparity and late age at first birth. The most commonly used model, the Gail model (BCRAT), underperformed the Tyrer-Cuzick model (IBIS) in the overall Marin County mammography population, and specifically performed more poorly in women with delayed childbirth, where it significantly underestimated breast cancer risk.

Clinicians need to be aware of the limitations of existing breast cancer risk models in women with delayed childbirth, and results from this retrospective cohort study support the consideration of using the Tyrer-Cuzick (IBIS) model if the model’s variables are available. Further advances in breast cancer risk modeling should address the current trends and changes in risk factor profiles.