84068997945VaccineVaccineVaccine0264-410X1873-251830236631645243410.1016/j.vaccine.2018.08.049HHSPA1009927ArticlePREFERENCES FOR HEALTH ECONOMICS PRESENTATIONS AMONG VACCINE POLICYMAKERS AND RESEARCHERSRichardsonJohn S.M.P.H.aMessonnierMarkPh.D.bProsserLisa A.Ph.D.cUniversity of Michigan, Department of Health Management and Policy, 1415 Washington Heights, Ann Arbor, MI 48109 jsrich@umich.eduCenters for Disease Control and Prevention, National Center for Immunization and Respiratory Disease, 1600 Clifton Road, N.E., MS E-52, Atlanta, GA 30029 qzm3@cdc.govUniversity of Michigan, Department of Health Management and Policy; and the Child Health Evaluation and Research (CHEAR) Center in the Division of General Pediatrics, 300 North Ingalls Building 6A14, Ann Arbor, MI 48109lisapros@umich.edu (Corresponding author)5220191792018151020180842019364364166423Purpose:

Measure the preferences of decision makers and researchers associated with the Advisory Committee on Immunization Practices (ACIP) regarding the recommended format for presenting health economics studies to the ACIP.

Methods:

We conducted key informant interviews and an online survey of current ACIP work group members, and current and previous ACIP voting members, liaison representatives, and ex-officio members to understand preferences for health economics presentations. These preferences included the presentation of results and sensitivity analyses, the role of health economics studies in decision making, and strategies to improve guidelines for presenting health economics studies. Best-worst scaling was used to measure the relative value of seven attributes of health economics presentations in vaccine decision making.

Results:

The best-worst scaling survey had a response rate of 51% (n=93). Results showed that summary results were the most important attribute for decision making (mean importance score: 0.69) and intermediate outcomes and disaggregated results were least important (mean importance score: −0.71). Respondents without previous health economics experience assigned sensitivity analysis lower importance and relationship of the results to other studies higher importance than the experienced group (sensitivity analysis scores: −0.15 vs. 0.15 respectively; relationship of the results: 0.13 vs. −0.12 respectively). Key informant interviews identified areas for improvement to include additional information on the quality of the analysis and increased role for liaisons familiar with health economics.

Conclusion:

Additional specificity in health economics presentations could allow for more effective presentations of evidence for vaccine decision making.

Vaccine policybest-worst scalinghealth economicsAdvisory Committee on Immunization Practices (ACIP)Introduction

Since 1964, the United States Advisory Committee on Immunization Practices (ACIP) has been making policy recommendations regarding vaccines for recommended age of vaccination, number of vaccine doses, time between doses, precautions and contraindications with each vaccine, and target populations.(1–3) These policy recommendations are used by the Centers for Disease Control and Prevention (CDC) to set the U.S. childhood and adult immunization schedules.(1–4) Under provisions of the Affordable Care Act, starting in September 2009, vaccination recommendations by the ACIP that are adopted by the CDC must be covered by health plans with no copayment.(1) ACIP recommendations also guide the purchase, delivery, and administration of pediatric vaccines in the Vaccines for Children Program.(1)

In making vaccine policy recommendations, the ACIP considers information on prevalence of the disease and disease severity; safety; efficacy and effectiveness; cost-effectiveness; and feasibility of different vaccines and vaccine schedules.(1, 5) The presentation and discussion of cost-effectiveness studies, or more broadly health economics studies, in the ACIP review process has evolved over the past several decades.

In order to improve the standardization and comparability of health economics presentations to the ACIP, the CDC developed guidelines and a presentation template for health economics studies in 2007. (6–8). The purpose of the current study is to understand preferences regarding content and approach for health economics studies presented to the ACIP in order to identify ways to enhance the value and interpretability of these presentations.

Methods

The ACIP consists of 15 voting members, an executive secretary, eight ex-officio members from government agencies other than the CDC, and liaison representatives from 30 health-related professional organizations and foundations. There are also ACIP work groups that include researchers who meet regularly under the direction of CDC Leads to review relevant vaccine information and prepare draft policy recommendations for the full ACIP. (2) A quantitative online survey was developed to understand preferences among those involved with the ACIP. This study was reviewed and given exempt status under the Medical Institutional Review Board at the University of Michigan (IRB# HUM00087889). CDC’s determination of this study was non-engaged.

Survey Sample

The survey sample included all current ACIP work group members and current and previous ACIP voting members, CDC Leads, liaison representatives, and ex-officio members dating back to 2007 (n=181).

Survey Development and Design

We conducted 13 key informant interviews to understand qualitatively the context and any potential issues around health economics presentations to the ACIP and to guide the development of the quantitative survey (see Supplemental Materials for more details).

To evaluate which aspects of health economics presentations were most and least valuable to respondents, we used an object case best-worst scaling approach for the quantitative survey. (9, 10) We defined seven primary attributes of health economics presentations (Table 1) and refined these using results from the key informant interviews. A balanced incomplete block design was used to create seven blocks of questions. Each question asked participants to compare three of the seven attributes and decide which was the most valuable and which was the least valuable to them for decision making.

Additional questions assessed more detailed preferences for the presentation of results (text descriptions, figures, or tables) and sensitivity analyses (credible intervals or bar charts with error bars, tornado diagrams, cost-effectiveness plane scatter-plots, and cost-effectiveness acceptability curve plots). We also asked questions on the following topics both in key informant interviews and the online survey: appropriateness of presenting health economics studies, influence of health economics studies, aspects requiring improvement, barriers to interpretation, ways to improve collaboration between the work groups and voting group, value of a quality measure, and use of supplemental materials.

The survey was pre-tested with one ACIP work group and then fielded to the full sample in August and September 2015. The final survey instrument is provided in the Supplemental Materials.

Analysis Plan

We analyzed the best-worst scaling questions by calculating mean importance scores, also known as sample-level best-minus-worst choice frequencies. (11) These were calculated as the percentage of times the attribute was chosen as the most valuable for decision making minus the percentage of times the attribute was chosen as least valuable for decision making. Stratified subgroup analyses were also conducted by calculating the importance scores among previous or current voting members, those without previous involvement in a health economics study, and those with previous involvement in a health economics study. We also conducted a regression analysis using a sequential best-worst conditional logit model. (11) For the reference group, we used the middle-ranked attribute (i.e. fourth) which we identified using the mean importance scores ranking. Stata® 14 was used to conduct the regression analysis.

ResultsSurvey Response

Of the potential 181 respondents, 93 responded yielding a 51% response rate. Among previous or current ACIP voting members, the response rate was 83%. 2% of the respondents were economists, however, 26% had participated in a health economics study and 49% reported they had read the ACIP guidance on health economics studies. (Table 2)

Attribute Mean Importance Scores

Using mean importance scores, the most valuable attribute of health economics presentations for decision making was the attribute of summary results or cost-effectiveness ratio. When this attribute was present in a best-worst scaling question, 72% of the time it was chosen as the most valuable attribute and only 10% of the time it was chosen as the least valuable attribute, with an overall mean importance score of 0.63 (Table 3a). The least valuable attribute for decision making was the attribute of intermediate outcomes and disaggregated results, which had an overall mean importance score of −0.69 (the negative value meaning that more people voted this as least valuable rather than most valuable). All other variables were considered least valuable nearly as often as they were considered most valuable. (Table 3a)

Subgroup Analysis

Previous or current voting ACIP members identified the same attributes as most and least important compared with the full sample. (Table 3a) We did find some differences for those with previous experience in health economics studies (Table 3b). The relationship of the results to other relevant studies had a lower mean importance score (−0.15) among respondents that were experienced with health economics studies compared with others (0.13). Conversely, sensitivity analysis results and methods had a higher mean importance score of 0.15 for the experienced group compared with −0.12 for the inexperienced group. Despite these differences, both groups consistently identified summary results or cost-effectiveness ratio as the most important for decision making and intermediate outcomes and disaggregated results as least important. (Table 3b)

Regression Analysis

Regression results confirmed summary results or cost-effectiveness ratio and the intermediate outcomes and disaggregated results as the most and least important attributes, consistent with the mean importance scores. There were no significant differences among the middle ranked attributes. Results were also similar for subgroup analyses (Tables 3a and 3b).

Additional questions on presentation format

For comparisons of study results, the most commonly preferred comparison was between studies of the same vaccine. When multiple studies have been conducted on the same vaccine or vaccine policy, respondents preferred either a full presentation when there were large differences or if one of the studies was industry funded (40% of respondents) or they said a slide or two would be sufficient regardless of the magnitude of the differences (38% of respondents). (Table 4) Most respondents (62% of all; 64% of current or previous voting members) ranked tables or bar graphs with credible intervals as the most valuable for decision making, with cost-effectiveness acceptability curves as the least preferred. (Table 4)

Suggestions for improving health economics presentations and guidelines

67% of all respondents and 59% of voting members reported the most common barrier to understanding the content of health economics presentations was because the technical language was not well defined (Table 5). In addition to this, the most common element of a presentation that was commonly inadequate was the relationship of the results to other relevant studies (39% of all respondents, 47% of previous or current ACIP voting members). 72% of all respondents desired that work group review comments of the health economics studies be presented to the ACIP voting group (80% of previous or current voting members). Respondents on average said that having an overall score of the quality of the health economics study would be quite helpful.

Role of health economics studies in the ACIP

Nearly all survey respondents (92%) said that health economics studies should be presented to and discussed by the ACIP. However, some responded that health economics studies should not be discussed (5% of all respondents and 8% of previous or current voting members), and a few that had no opinion (3% all respondents).

Among previous or current voting members, 19% said that health economics studies had not influenced any vaccine recommendation decision of which they were a part. The situations where voting members were most frequently influenced by health economics studies were with the meningococcal vaccine recommendations (57% reported being influenced) and the human papillomavirus vaccine recommendations (48% reported being influenced). Other vaccine recommendations that only a few voting members reported as having some kind of influence on their vote were pneumococcal conjugate vaccine (14%); Tdap vaccine (10%); zoster vaccine (10%); hepatitis A vaccine (10%); hepatitis B vaccine (5%); rotavirus vaccine (5%); Japanese encephalitis and yellow fever vaccine (5%); measles, mumps, rubella, and varicella (MMRV) vaccine (5%); and influenza vaccine (5%).

Discussion

Key informant interviews and survey results demonstrated that all elements of health economics studies discussed in the current guidelines for the ACIP were considered necessary and important to the presentation; the summary statistic or cost-effectiveness ratio was identified the most valuable element for decision making. Sensitivity analyses had the second highest importance score among those that had experience conducting health economic studies. For those who were not experienced with conducting health economic studies, which consisted of the majority of respondents, the relationship of the results to other relevant studies was also highly valued in decision making. The majority of ACIP members stated ill-defined technical language as a significant barrier to understanding health economics presentations in general. Ongoing training was suggested to help overcome this barrier. Additional preferences and suggestions from the ACIP members were to provide more comparisons of the results to other studies, have the work groups describe to the voting group their assessments of the economic models, and establish a measure of overall quality that could be used by reviewers and reported with each health economics study.

Only one previous study has evaluated the role of health economics evidence in the ACIP process. Dempsey and colleagues found some similar results regarding overall use of health economics evidence in decision making when interviewing ACIP voting members in 2006. (6) ACIP members desired a standardized process for presenting health economics studies. The majority thought that health economic information was important but should not outweigh the important issues of vaccine efficacy, disease burden, and safety. This study provides additional detail on the presentation of the economic study results.

Recent opinion articles have emphasized that cost-effectiveness should not be a determining factor for vaccine decision making. Luyten and Beutels argue that economic evaluations do not capture the full value of vaccination programs particularly in the areas of health equity, sustaining the public good of herd immunity, and social integration of minority groups. (12) Schwartz and Mahmoud make a similar point and emphasize that more research should be done to understand how evidence is used in vaccine decision making. (13) From this study we found that decision makers associated with the ACIP do not solely use cost-effectiveness information in making decisions; however, in the case of meningococcal and HPV vaccine-related policies, health economics information did have an impact on the decisions of some ACIP voting members who participated in the key informant interviews. It was also clear that almost all respondents agreed that the economic information for new vaccination recommendations continue to be presented to the ACIP.

Many suggestions for improving the presentation and use of health economics studies were made that align with the current literature. Currently, the ACIP guidelines do not include any recommendations on intermediate outcomes and disaggregated results. Even though these types of results were found to have little relative value in the ACIP decision making, the qualitative interviews suggested that they may be valuable in understanding the overall health impact of the vaccine recommendation. Many guidelines for health economics studies in other countries request the presentation of intermediate outcomes and disaggregated results to help understand the impact of the vaccine in greater detail and to assess the validity of both the modeling assumptions and the summary results. (14–23)

Some respondents suggested a more active role for work group members or CDC economists in interpreting the economic evidence. From assessing the quality of the modeling approach to the validity of the model to the strength of the evidence, there could be a potential role for a trusted expert to provide additional interpretation and comparison, especially across multiple analyses. While decision makers in the US do not use a formal cost-effectiveness threshold, there was discussion that some additional context could be provided to assist interpretation of the ratios.

A related area for future research is the development of quality scores for health economic studies. Survey participants reported that having an overall score of the quality of each health economics study would be quite helpful, either as part of an overall interpretation of the analysis or as a separate piece of information. Using GRADE criteria (24) or something similar to assess the quality of model inputs was one suggestion made by interviewees; however, this scoring system is not directly applicable to economic models and available scoring systems have not been widely used.(25)(26, 27) Future research could focus on development of a measure of quality that is applicable to various types of health economics analyses, includes an assessment of both the quality of the overall model as well as the inputs to the model, and can be generated quickly so as not to delay the use of the cost-effectiveness evidence in decision making.

Data visualization is also an area that was suggested for future development, specifically in how sensitivity analyses are presented. Recent literature has encouraged the use of infographics or other meaningful yet simple ways to display complex information. (28) Within the complex information of health economics studies, survey participants preferred simple formats, such as tables and bar graphs, over the more complex formats of cost-effectiveness scatter plots and cost-effectiveness acceptability curves. These more complex forms of sensitivity analyses, however, can be useful tools for decision making. More training may be required to help ACIP members understand them and use them as decision tools.

Additional training may also help to overcome some of the other barriers to understanding health economics studies that were identified through this survey. Due to the changing membership of the ACIP and its work groups, training would need to be an ongoing effort in order to ensure that newer members are informed and that continuing members remain informed regarding the various elements of health economic analyses. Since poorly defined technical language was the most common barrier to understanding health economics presentations, a glossary of technical terms similar to those found in guidance documents of other guidance documents on health economic studies may be helpful. (16, 22, 29–31)

A common theme among respondents was that while training would address interpretation of the economic evidence, the ACIP charter provides no specific guidance regarding how much weight should be given to health economics studies relative to other types of evidence. The ACIP charter does not provide weights for other types of evidence either. One approach that could potentially be used in providing transparency in the weights in the decision making process is multi-criteria decision analysis. (32) This approach has been used in a few applications, primarily outside the US. (33) Some examples include the Analytic Hierarchy Process (34) , the EVIDEM framework (35), and the SMART Vaccine tool developed by the Institutes of Medicine in the US. (36) The SMART Vaccine tool specifically allows individuals to specify the weight given to cost-effectiveness evidence in addition to other forms of evidence that is relevant to vaccine policy (e.g., health considerations, public concerns, programmatic considerations, etc.). (36)

Even though detailed reviews of health economic evidence happen within the ACIP work groups, there are still some groups that have never examined health economic studies. Health economists have varying levels of involvement with the different work groups. As suggested by one of the interviewees in this study, a protocol for identifying and conducting health economics studies at the work group level could help establish uniformity in how and when health economic evidence is considered in the ACIP decision making process.

Some limitations to the study are that the survey only asked participants to consider the value of different elements of health economics presentations in terms of decision making and not in terms of quality assessment. This study also had lower response rate for non-ACIP members or former members. One of the major strengths of this study, however, was that among ACIP voting members we had a high response rate and therefore we assume that our estimates for this subgroup are fairly representative.

In conclusion, there are various guidelines for health economics studies (7, 25, 31, 37–40) and this study provides information that can be used to revise and refine and tailor one set of guidelines for use by a decision making body in the US. Having a clearer understanding of the value of different aspects of health economics presentations allows for the development of more efficient and effective presentations. Also, allowing for open feedback and suggestions for improvement by the decision making body has provided insights that may not have been known otherwise. These results can inform the improvement of guidance and training for the ACIP, and can guide researchers in presenting high quality health economics studies that are tailored to the decision making needs of the ACIP.

Supplementary MaterialAcknowledgments

We acknowledge the efforts of Norma-Jean Simon in the development and pretesting of the survey, Brittany D’Cruz in survey administration, and Acham Gebremariam in the sequential best-worst conditional logit regression model. Financial support for this study was provided through the Joint Initiative for Vaccine Economics, Phase 4, a cooperative agreement between the University of Michigan and the Centers for Disease Control and Prevention (5U01IP000499-03).

The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report. Mark Messonnier is the only author employed by the sponsor. The findings and conclusions in this report, however, are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Declarations of interest: none

ACIP

Advisory Committee on Immunization Practices

CDC

Centers for Disease Control and Prevention

ReferencesAdvisory Committee on Immunization Practices. ACIP Charter. USA 2014.SmithJC. The structure, role, and procedures of the U.S. Advisory Committee on Immunization Practices (ACIP). Vaccine 2010;28 Suppl 1:A68–75. doi: 10.1016/j.vaccine.2010.02.037.20413002WaltonLR, OrensteinWA, PickeringLK. The history of the United States Advisory Committee on Immunization Practices (ACIP). Vaccine 2015;33(3):405–14. doi: 10.1016/j.vaccine.2014.09.043.25446820SmithJC, SniderDE, PickeringLK. Immunization Policy Development in the United States: The Role of the Advisory Committee on Immunization Practices. Annals of Internal Medicine 2009;150(1):45–9.19124820Advisory Committee on Immunization Practices. The Advisory Committee on Immunization Practices (ACIP) Flyer 2013.DempseyAF, CowanAE, StokleyS, MessonnierM, ClarkSJ, DavisMM. The role of economic information in decision-making by the Advisory Committee on Immunization Practices. Vaccine 2008;26(42):5389–92. doi: 10.1016/j.vaccine.2008.07.085.18708111LieuTM, MartinI, Messonier, MarkL. Guidance for Health Economics Studies Presented to the Advisory Committee on Immunization Practices (ACIP) 2007.Advisory Committee on Immunization Practices. ACIP guidance for presentation slides for health economics studies 2008 Available from: http://www.cdc.gov/vaccines/acip/committee/guidance/economic-studies.html.FlynnTN. Valuing citizen and patient preferences in health: recent developments in three types of best–worst scaling. Expert review of pharmacoeconomics & outcomes research 2010; 10(3):259–67. doi: 10.1586/ERP.10.29.20545591FlynnTN, LouviereJJ, PetersTJ, CoastJ. Best--worst scaling: What it can do for health care research and how to do it. Journal of Health Economics 2007;26(1):171–89. doi: 10.1016/j.jhealeco.2006.04.002.16707175LouviereJJ, FlynnTN, MarleyAAJ. Best-worst scaling: theory, methods and applications Cambridge, United Kingdom: Cambridge University Press; 2015.LuytenJ, BeutelsP. The Social Value Of Vaccination Programs: Beyond Cost-Effectiveness. Health Aff (Millwood) 2016;35(2):212–8. doi: 10.1377/hlthaff.2015.1088.26858372SchwartzJL, MahmoudA. When Not All That Counts Can Be Counted: Economic Evaluations And The Value Of Vaccination. Health Aff (Millwood) 2016;35(2):208–11. doi: 10.1377/hlthaff.2015.1438.26858371ChaikledkaewU, TeerawattananonY. Presentation of economic evaluation results. Journal of the Medical Association of Thailand 2008;91(2):S66–S73.19253489Department of Health Republic of South Africa. Guidelines for Pharmacoeconomic Submissions In: Department of Health Republic of South Africa, editor. South Africa 2012.National Institute for Health and Care Excellence. Process and Methods Guide: Guide to the Methods of Technology Appraisal 2013 In: National Institute for Health and Care Excellence, editor. England 2013.Health Information and Quality Authority. Guidelines for the Economic Evaluation of Health Technologies in Ireland In: Health Information and Quality Authority, editor. Ireland 2010.WalterE, ZehetmayrS. Guidelines on Health Economic Evaluation In: Research IfP, editor. Vienna, Austria 2006.College voor zorgverzekeringen. Guidelines for pharmacoeconomic research, updated version In: The Health Care Insurance Board (CVZ), editor. The Netherlands 2006.Scottish Medicines Consortium. Guidance to Manufacturers for Completion of New Product Assessment Form (NPAF) Scottland 2007.Academy of Managed Care Pharmacy. A format for Submission of Clinical and Economic Evidence of Pharmaceuticals in Support of Formulary Consideration In: Academy of Managed Care Pharmacy, editor. Alexandria, USA 2012.Canadian Agency for Drugs and Technologies in Health. Guidelines for the Economic Evaluation of Health Technologies: Canada In: Canadian Agency for Drugs and Technologies in Health, editor. Canada2006.Taiwan Society for Pharmacoeconomics and Outcomes Research. Guidelines of Methodological Standards for Pharmacoeconomic Evaluations in Taiwan Taiwan 2006.GRADE Working Group. Grading the quality of evidence and the strength of recommendations 2014 Available from: http://www.gradeworkinggroup.org/intro.htm. Accessed February 15, 2015ChiouC-F, HayJW, WallaceJF, BloomBS, NeumannPJ, SullivanSD, Development and validation of a grading system for the quality of cost-effectiveness studies. Medical Care 2003;41(1):32–44.12544542Center for the Evaluation of Value and Risk in Health. Cost-Effectiveness Analysis Registry 2016 Available from: https://research.tufts-nemc.org/cear4/. Accessed February 15, 2016.ThoratT Correspondence with the Center for the Evaluation of Value and Risk in Health on the quality rating used in the Tufts Medical Center Cost-Effectiveness Analysis Registry In: RichardsonJ, editor. Not published. 2015.OttenJJ, ChengK, DrewnowskiA. Infographics And Public Policy: Using Data Visualization To Convey Complex Information. Health Aff (Millwood) 2015;34(11):1901–7. doi: 10.1377/hlthaff.2015.0642.26526248Consejo de Salubridad General, Instituto Nacional de Salud Publica. Guía para la conducción de estudios de evaluación económica para la actualización del Cuadro Básico de Insumos del Sector Salud en México Mexico 2008.AvksentievaMV, AntonovaNV, ArutyunovGP, VlasovVV, DerkachEV, ZorinNA, Procedure for clinical and economic evaluation of drug lists that are submitted for reimbursement coverage from public health care budget: decision-making criteria In: Research Center for Clinical and Economic Evaluation and Pharmacoeconomics of the Russian State Medical University named after N. I. Pirogov, editor. Moscow, Russia 2010.World Health Organization. WHO guide for standardization of economic evaliations of immunization programmes In: World Health Organization, editor. Geneva, Switzerland 2008.MuhlbacherAC, KaczynskiA. Making Good Decisions in Healthcare with Multi-Criteria Decision Analysis: The Use, Current Research and Future Development of MCDA. Applied health economics and health policy 2016;14(1):29–40. Epub 2015/11/01. doi: 10.1007/s40258-015-0203-4.26519081MarshK, LanitisT, NeashamD, OrfanosP, CaroJ. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. PharmacoEconomics 2014;32(4):345–65. Epub 2014/02/08. doi: 10.1007/s40273-014-0135-0.24504851ReddyBP, KellyMP, ThokalaP, WaltersSJ, DuenasA. Prioritising public health guidance topics in the National Institute for Health and Care Excellence using the Analytic Hierarchy Process. Public health 2014;128(10):896–903. Epub 2014/11/05. doi: 10.1016/j.puhe.2014.07.003.25369354WagnerM, KhouryH, WilletJ, RindressD, GoetghebeurM. Can the EVIDEM Framework Tackle Issues Raised by Evaluating Treatments for Rare Diseases: Analysis of Issues and Policies, and Context-Specific Adaptation. PharmacoEconomics 2016;34(3):285–301. Epub 2015/11/09. doi: 10.1007/s40273-015-0340-5.26547306Institute of Medicine [IOM]. Ranking vaccines: a prioritization software tool: phase II: prototype of a decision-support system MadhavanG, SanghaK, PhelpsC, FrybackD, RappuoliR, MartinezRM, , editors. Washington, D.C.: The National Academic Press; 2013.GoldMR, SiegelJE, RussellLB, WeinsteinMC. Cost-effectiveness in health and medicine New York, USA: Oxford University Press; 1996 xxiii, p. 425.International Society for Pharmacoeconomics and Outcomes Research [ISPOR]. Pharmacoeconomic guidelines around the world 2015 Available from: http://www.ispor.org/peguidelines/index.asp. Accessed November 16, 2015.HusereauD, DrummondM, PetrouS, CarswellC, MoherD, GreenbergD, Consolidated Health Economic Evaluation Reporting Standards (CHEERS)--explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health 2013;16(2):231–50. doi: 10.1016/j.jval.2013.02.002.23538175DrummondMF, SculpherMJ, TorranceGW, O’BrienBJ, StoddartGL. Methods for the Economic Evaluation of Health Care Programmes New York: Oxford University Press Inc.; 2005.

Attributes of health economics studies used in best-worst scaling questions.

Attributes of Health Economics StudiesDefinition
1. Model overview and structural assumptionsThe model and structural assumptions include a description of: (1) the health states included, (2) the progression of illness recovery and immunity, (3) how individuals enter, exit, or remain in the model, (4) how individuals in the model interact.
2. Description of cost and health valuation inputsCosts may include direct medical, direct non-medical, and changes in productivity (i.e., time costs). Health utility valuations are assigned to each health state. Utilities are assigned on a scale from 0 (representing dead) and 1 (representing perfect health) and are used to determine the quality adjusted life years (QALYs) gained by the intervention.
3. Intermediate outcomes and disaggregated resultsIntermediate outcomes are those that precede the key outcomes. Disaggregated results show the separate contributions of costs and the separate components of QALYs by source or health state.
4. Summary results or cost-effectiveness ratioThese are the results that answer the study question, typically the ratio of incremental costs divided by the incremental gain in QALYs.
5. Sensitivity analysis results and methodsSensitivity analyses explore how the results change when model inputs are varied across a predefined range.
6. Discussion of limitations to the analysisStudy limitations describe the potential biases of the model due to missing evidence or characteristics of the studies used to develop model inputs.
7. Relationship of the results to other relevant studiesThe results and sensitivity analysis of the study are compared to other studies that examine the same vaccine or alternative interventions for the same condition.

Characteristics of survey respondents.

Respondent CharacteristicsAll respondentsPrevious or Current ACIP Voting Membersa
N%N%
Current role with ACIP9325
 ACIP voting member78%728%
 ACIP liaison representative1516%416%
 CDC employee regularly working with the ACIP1011%00%
 ACIP ex officio member33%14%
 ACIP work group member (only)4852%832%
 Not currently affiliated with the ACIP or regularly working on ACIP projectsb1011%520%
Primary role outside of ACIP9125
 Clinician or medical provider2730%728%
 Biomedical or clinical researcher2325%728%
 Epidemiologist2426%624%
 Economist22%00%
 Other1516%520%
Number of health economics presentations seen at ACIP or Work Group meetings9125
 None1112%28%
 1 to 22426%28%
 3 to 52629%624%
 6 or more3033%1560%
Prior participation in a health economics study9125
 Yes2426%312%
 No6774%2288%
Familiarity with ACIP’s guidance for conducting and reporting health economics studies9025
 Read through it and am very aware of it1517%832%
 Read through it but do not remember much2932%728%
 Never read through the guidance4651%1040%

Notes:

7 people (28%) were current voting members and 18 people (72%) had previously been a voting member.

This may include CDC employees that are not currently working on ACIP projects but have done so in the past, or former work group or voting members that are no longer working with the ACIP.

Attributes ranked by mean importance score—all respondents.

Attributes of health economics presentationsMost Valuable FrequencyaLeast Valuable FrequencybMean Importance ScorecCLogitd CoefficientCLogitd P-value
All respondents
Summary results or cost-effectiveness ratio0.720.100.631.09<0.0001
Model overview and structural assumptions0.390.320.07−0.150.7913
Relationship of the results to other relevant studies0.370.320.06−0.170.9007
Discussion of limitations to the analysis0.290.270.01−0.19Reference
Description of cost and health valuation inputs0.240.27−0.03−0.250.6514
Sensitivity analysis results and methods0.260.32−0.05−0.290.4409
Intermediate outcomes and disaggregated results0.060.74−0.69−1.59<0.0001
Previous or current ACIP voting members
Summary results or cost-effectiveness ratio0.780.040.741.65<0.0001
Relationship of the results to other relevant studies0.360.260.10−0.040.6351
Model overview and structural assumptions0.370.350.02−0.210.8662
Sensitivity analysis results and methods0.310.300.00−0.190.9924
Discussion of limitations to the analysis0.250.250.00−0.16Reference
Description of cost and health valuation inputs0.200.35−0.15−0.450.2921
Intermediate outcomes and disaggregated results0.070.78−0.71−1.63<0.0001

Note:

Sample-level frequency or percent, ranging from 0 to 1, of being voted as most valuable.

Sample-level frequency or percent, ranging from 0 to 1, of being voted as least valuable.

Score ranges from −1 to 1 and is the difference between the most valuable frequency and the least valuable frequency.

Coefficients and P-values were calculated using sequential best-worst conditional logit regression.

Attributes ranked by mean importance score—subgroup analysis.

Attributes of health economics presentationsMost Valuable FrequencyaLeast Valuable FrequencybMean Importance ScorecCLogitd CoefficientCLogitd P-value
Experienced with health economics studies
Summary results or cost-effectiveness ratio0.750.070.691.36<0.0001
Sensitivity analysis results and methods0.380.230.150.050.3143
Model overview and structural assumptions0.390.360.03−0.210.987
Description of cost and health valuation inputs0.250.250.00−0.160.8468
Discussion of limitations to the analysis0.310.310.00−0.21Reference
Relationship of the results to other relevant studies0.230.38−0.15−0.450.3699
Intermediate outcomes and disaggregated results0.030.75−0.71−1.55<0.0001
Inexperienced with health economics studies
Summary results or cost-effectiveness ratio0.710.100.611.01<0.0001
Relationship of the results to other relevant studies0.430.300.13−0.060.4613
Model overview and structural assumptions0.390.300.08−0.130.7648
Discussion of limitations to the analysis0.280.260.02−0.18Reference
Description of cost and health valuation inputs0.240.28−0.04−0.280.5227
Sensitivity analysis results and methods0.220.35−0.12−0.400.1442
Intermediate outcomes and disaggregated results0.060.74−0.68−1.60<0.0001

Note:

Sample-level frequency or percent, ranging from 0 to 1, of being voted as most valuable.

Sample-level frequency or percent, ranging from 0 to 1, of being voted as least valuable.

Score ranges from −1 to 1 and is the difference between the most valuable frequency and the least valuable frequency.

Coefficients and P-values were calculated using sequential best-worst conditional logit regression.

Preferences for the presentation of results and sensitivity analyses

Preferences for resultsAll RespondentsPrevious or Current ACIP Voting Members
n%n%
Preferred results comparisons41100%10100%
 To other studies of the same vaccine3278%990%
 To other vaccines for the same illness2971%770%
 To vaccines for other illnesses2459%770%
 To pre-specified ranges of low, medium, and high incremental cost-effectiveness values2356%770%
 To non-vaccine interventions for the same illness2049%110%
 To non-vaccine interventions for other illnesses615%00%
 None - the results of the study should not be compared to anything else25%00%
Preference for an entire presentation comparing results from multiple studies on the same vaccine40100%10100%
 Yes - always25%00%
 Yes - but only if there are large differences718%110%
 Yes - but only if there are larger differences OR if one was industry funded1640%440%
 No - a slide or two would be sufficient1538%550%
Preferences for sensitivity analysesn%n%
Table or bar graph with credible intervals37100%10100%
 Ranked first2362%764%
 Ranked second616%218%
 Ranked third616%19%
 Ranked fourth25%19%
One-way sensitivity analysis tornado diagram37100%10100%
 Ranked first514%327%
 Ranked second1438%436%
 Ranked third1027%218%
 Ranked fourth822%218%
Cost-effectiveness acceptability curve37100%10100%
 Ranked first719%19%
 Ranked second822%327%
 Ranked third1130%436%
 Ranked fourth1130%327%
Cost-effectiveness scatter plot of simulations37100%10100%
 Ranked first25%00%
 Ranked second924%218%
 Ranked third1027%436%
 Ranked fourth1643%545%

Note: Survey respondents were randomly assigned to either the preferences for results or the preferences for sensitivity analysis questions. See the Supplemental Materials for the specific questions asked.

Most common barriers to understanding health economics presentations and the elements of the presentation that are most commonly not presented adequately

All RespondentsPrevious or Current ACIP Voting Members
Preferencesn%n%
Most common barriers to understanding health economics presentations58100%17100%
 Technical language was not well defined or explained3967%1059%
 Adequate justification for analytic decisions or assumptions was not provided2340%1059%
 Not enough time was spent on the details2340%953%
 Interpretation of figures and tables was not provided1424%741%
 Other (please specify)916%212%
Elements of presentations that are most commonly inadequate77100%19100%
 Relationship of the results to other relevant studies3039%947%
 Discussion of limitations to the analysis2229%632%
 None - these elements are always presented adequately to the ACIP2026%316%
 Intermediate outcomes and disaggregated results1621%632%
 Description of cost and health valuation inputs1216%316%
 Sensitivity analysis results and methods1216%316%
 Model overview and structural assumptions810%211%
 Summary results or cost-effectiveness ratio34%15%