Preference-based measures of health-related quality of life (HRQL) are needed for monitoring population health and for program evaluation for comparative effectiveness research. Most importantly, these measures are required for estimating quality-adjusted life years (QALYs). A number of widely used generic preference-based measures are available such as the EQ-5D (1), Health Utilities Index Mark 2 (HUI2) and Mark 3 (HUI3)(2), the Quality of Well-Being - Self-Administered scale (QWB-SA) (3),and Short-Form 6D (SF-6D) (4;5). Although these measures share a common core (6;7) and all include items on mobility, mental health, and pain, there are also important differences with respect to which attributes (dimensions or domains of health status) are included. HUI and the QWB-SA include vision, hearing, speech, and dexterity; the EQ-5D and SF-6D do not. The QWB-SA is unique in that it includes 58 symptoms or health problems, only some of which are included in the other measures. These measures also differ in the range of function or symptom severity covered in each attribute. The QWB-SA asks respondents if they have or do not have a problem such as pain and stiffness; in contrast HUI and SF-6D have gradients such as the categories mild, moderate, and severe pain.

These measures also differ with respect to the methods that were used to elicit preference scores with which to estimate their respective multi-attribute scoring functions, the methods for estimating those functions, and their functional forms (8). For instance, the QWB-SA scoring function is based on valuations using the visual analog scale (VAS) and a linear additive scoring function. SF-6D is based on the standard gamble (SG) and an ad hoc modified linear additive functional form. EQ-5D is based on the time trade-off (TTO) and an ad hoc modified linear additive functional form. HUI is based on transformed VAS and SG scores and a multiplicative functional form.

It is therefore not surprising that several investigators that have used two or more measures have concluded that the scores from these measures are not interchangeable (9–14). Further there is evidence from prospective studies that the estimates of absolute and/or relative change (responsiveness, including effect size (ES) and the standardized response mean (SRM)) (15) often do not agree (12;16–19).

The objective of this paper is to examine agreement among the above measures in classifying patients into the same categories of change: We want to know if the measures agree on which patients get better, remain stable, or get worse. Data from two prospective cohort studies that employed all five of the above measures as well as disease-targeted measures are used to assess agreement among these measures: one study of patients undergoing cataract surgery, the other of patients referred for treatment for congestive heart failure by a specialty clinic.

This paper builds on an earlier paper (19) based on the data from the same study. That paper provided cohort-level estimates of responsiveness (SRM) for each of the five preference-based measures in each of the two cohorts. Responsiveness varied among measures and across cohorts. Results from that paper underscore the lack of interchangeability of scores among these measures.

This paper asks an important follow-up question. Even if overall responsiveness differs among measures, do they agree on who gets better, who gets worse, and who was stable?

Previous work (19) indicated that patients undergoing cataract surgery changed substantially between baseline and the one-month follow-up survey (after surgery) and were typically then stable in the period between the one- and six-month follow-ups. Analyses for the cataract cohort therefore focus on change between the baseline to one-month follow-up. Improvement was more gradual in the heart failure cohort (19). Analyses focus on the change between baseline and the six-month follow-up.

A total of 376 cataract patients and 160 heart failure patients were recruited to the study. The majority of patients were white, cataract patients tended to be female, heart failure patients tended to be male, most cataract patients were 65+, and the heart failure patients tended to be younger with the majority in the 45–64 age group (Table 1).

For the cataract cohort data for baseline and one-month follow up assessments were available for 315 of the 376 cases. Complete data for all pairs for all measures were available for 210 cases. The distribution of demographic variables for those with and without complete data was similar and there were no statistically significant differences between the two groups.

For the heart failure cohort data for baseline and the six-month follow-up assessments were available for 110 of the 160 cases. Complete data for all pairs for all measures were available for 86 cases. Those with missing data were older than those without missing data and the difference between the two groups was statistically significant.

The overwhelming majority of respondents, 93%, reported that no one helped them to complete the questionnaires; 7% reported receiving help. Among those who received any help, 90% reported that someone read the questions to them; 55% reported that someone wrote the answers on the questionnaire for them; 6% reported that someone answered the questions for them; 4% reported that someone translated the questions into their language for them; and 9% reported some other kind of help. Therefore the overwhelming majority of responses were based on self-completion and self-assessment.

Scores for each of the measures at baseline and one month and the change scores for the cataract cohort are displayed in Table 2. Note that the mean change in the total score for the NEI-VFQ-25 (VFQ_t) of 9.96 exceeds the guideline for a clinically important difference of 5.00. Similarly, the mean change in overall HUI3 scores exceeds the 0.03 clinically important difference guideline. The mean change in HUI3 vision score and HUI2 sensation scores exceed the 0.05 clinically important difference guideline. The mean changes in scores for EQ-5D, QWB-SA, SF-6D, and SRH are less than the guidelines for a clinically important difference. The distribution of change scores for the VFQ_t is displayed in Figure 1. Using the change of 5 or more in VFQ_t score as the criterion, 43% of patients improved, 52% were stable, and 4% got worse.

Scores for each of the measures at baseline and six months and the mean change scores for the heart failure cohort are displayed in Table 3. Note the mean change of 8.72 in score for the MLHF exceeds the 5.00 guideline for a clinically important difference. The mean change in QWB-SA, SF-6D, and HUI3 scores exceed the guideline while the mean changes in scores for the EQ-5D, HUI2, and SRH do not. The distribution of change scores for the MLHF is displayed in Figure 2. Using the change of 5 or more in MLHF score as the criterion, 47% of patients improved, 35% were stable, and 19% got worse.

There is very little pair-wise agreement between the disease-targeted measures and the five preference-based measures about which patients improved, were stable, or deteriorated. In general, agreement for the cataract cohort was poor and for the heart failure cohort negligible. For the cataract cohort, the agreement between the relevant HUI single-attribute (“disease-targeted”) scores and the NEI-VFQ-25, those for HUI2 sensation and HUI3 vision, were the exceptions; agreement was fair. Agreement was also fair between the utility scored and conventional versions of the NEI-VFQ-25. Given that both of these measures are based on the same questionnaire, it is perhaps surprising that the agreement is only fair and is not clearly higher than agreement between the NEI-VFQ-25 and HUI2 sensation and HUI3 vision. In the heart failure cohort, fair agreement was observed only for the SRH. On the basis of the ROC analyses, the results reported here appear to be robust to the choice of cut points for a clinically important change.

The agreement analyses treat the disease-targeted measure as the gold standard. Yet even though there is evidence of cross-sectional and longitudinal construct validity for the two disease-targeted measures, neither can be regarded as a true gold standard. Furthermore, that vision-related or heart-related HRQL improved does not necessarily imply that overall HRQL improved. It is possible that the side effects of interventions could more than offset the gains and therefore overall HRQL might not improve. It is also possible that even though vision- or heart-related HRQL improved, overall HRQL did not due to the burdens associated with comorbidities. The NEI-VFQ-25 asks subjects about a wide variety of difficulties that they might experience due to limited vision, including reading, hobbies, navigating, driving, going up and down stairs, interacting with others, dressing, and the amount of assistance the subject needs from others. Similarly, the MLHF asks about limitations in/problems with mobility, sexual activity, interacting with others, fatigue, hobbies, worry, concentration, memory, and depression that the subject experiences due to the subject’s heart condition. Although the breadth of coverage of these disease-targeted measures probably reduces the scope for a discrepancy between trends in vision- or heart-related HRQL and overall HRQL, it does not eliminate the possibility for such discrepancies.

The results on overall change in measures underscore that scores from these five preference-based measures are not interchangeable (Table 2). In the cataract cohort, using published guidelines (2;57) on clinically important differences/changes, clearly clinically important change was detected by the NEI-VFQ-25 and HUI3. Given that vision is included in HUI3 and that the NEI-VFQ-25 is a vision-targeted measure, this results is not surprising. But vision is included in the QWB-SA and the overall score did not reflect the gain in HRQL that was measured by the NEI-VFQ-25 and HUI3. Of course, it should be noted that only the “worst” symptom for that subject in the QWB-SA is used to compute the overall score and further that in a relatively elderly cohort, it is likely that many subjects were experiencing symptoms that are more burdensome than impaired vision and thus the vision item frequently did not affect the calculation of the QWB-SA score. Others have noted the lack of responsiveness of the EQ-5D to the effects of cataract surgery(60).

For the heart failure cohort, using published guidelines (23–25;46;47;51) on clinically important differences/changes, the MLHF, HUI3, QWB-SA, and SF-6D recorded clinically important change. Fatigue and shortness of breath symptoms on the QWB-SA as well as the mobility, physical, and social activity scales may have captured some of the effects of heart failure on HRQL. Similarly, the physical functioning, vitality, and role attributes on SF-6D may have registered some of the effects. HUI3 ambulation may have performed similarly(61)

It should be noted that reliability is less than perfect for each of the measures used in the study (62), so disagreement between change scores is also influenced by measurement error and short-term fluctuations in health that are unrelated to the conditions of primary interest. Change over time is measured with even less precision than absolute scores at a point in time. Jones and Feeny (63) and Pickard et al. (64) found lower levels of agreement between proxy and self-report for change scores than was evident for cross-sectional comparisons of baseline and follow-up scores. Other investigators have pointed out that due to the size of measurement error typically found in HRQL measures, change must be often be quite substantial for measures to agree (62). Our results were not sensitive to the magnitude of change that was considered clinically important, but the magnitude of true underlying changes does influence the agreement that can be expected. Hence, some measures of change may have better agreement than found in our studies, when reflecting interventions with larger overall effects on HRQL.

Some limitations of the study should be noted. Because the analyses are based on subjects for whom both baseline and the designated follow-up assessments were available, the results are not necessarily representative of the experience of the entire inception cohorts. In the cataract cohort those with and without complete data were similar. In the heart failure cohort, those with missing data were older than those without missing data. If older patients experienced less improvement in HRQL than younger patients, it is possible that the estimate of change based on subjects for whom we had complete data is biased upwards. As noted in the Results, 7% of subjects had help in completing questionnaires so responses could have been influenced by others. Another limitation is that while the scoring functions for the QWB-SA and EQ-5D are based on preference scores from random samples of community-dwelling adults in the US, the scoring functions for HUI2 and HUI3 are based on preferences from random samples of the Canadian population and the function for the SF-6D is based on UK preferences. There is evidence of the generalizability of the QWB scoring function, (30;65;66) the HUI2 scoring function, (28;67) and the HUI3 scoring function.(29;68–70) In contrast there is considerable variability across “national” EQ-5D scoring functions. Nonetheless having not relied exclusively on US-based scoring functions is unlikely to be an important factor influencing the results. Finally, we classified cataract and heart failure patients as changed if the absolute value of their change score was ≥5.0 whether or not the difference was statistically significant. Hays et al. (71) note that changes that are statistically significant at the level of an individual subject will typically exceed the guideline for a clinically important difference(72).

The results underscore the lack of interchangeability of scores among these five widely used preference-based measures. Not only are the absolute scores not necessarily interchangeable; in these results the change scores were also not interchangeable (12). The results also point to a lack of precision in estimating the magnitude of change in HRQL.

In making choices about which preference-based measure(s) to use in a study, investigators need to consider carefully the coverage of the health-status classification systems and the relevance of those systems to their clinical or population health application, evidence on the cross-sectional construct validity of the measures in that application, and evidence of the responsiveness (longitudinal construct validity) of the measures in that context. Further, users of the results of studies that have employed preference-based measures to assess HRQL need to interpret those results carefully.