Since the Human Genome Project concluded a decade ago, the landscape of genomic medicine has become increasingly complex, burgeoning with applications for clinical or personal use.^1–3 The state of evidence surrounding implementation of genetic tests, however, has been described by some as a dilemma.⁴ Indeed, oftentimes no guidelines exist that aid in decision making on the implementation of genomics-based technologies. When guidelines do exist, they often frustrate clinical and other audiences due to findings of insufficient evidence on their use.⁵ Furthermore, not all guideline development groups use the same methods to arrive at recommendations; therefore, if different groups were to address the same topic, they might come to dissimilar conclusions. As a result of this variability in methods, conclusions on the same topic may differ. Potential discrepancies may be resolved by careful comparison of guidelines; however, it would be useful to have a systematic approach that allows decision makers a horizon scanning of synthesized evidence sources relevant to the use of a particular genetic test.

The Office of Public Health Genomics (OPHG) at the Centers for Disease Control and Prevention (CDC), in collaboration with the National Cancer Institute Epidemiology and Genomics Research Program, recently modified an existing classification system⁶ and created an evolving table of genomic tests sorted by level of evidence.⁷ The table presents an evidence-ordered classification of “genomic applications” (defined here as the use of gene-based tests in specific clinical scenarios, or use of genome-related information, such as knowledge about a specific variant, to aid in clinical decision making in specific clinical scenarios), rather than a classification of laboratory assays themselves. We have previously described classification of cancer genomic tests (http://blogs.cdc.gov/genomics/2012/08/23/evidence-matters-in-genomic-medicine-round-2/) and family history applications (http://blogs.cdc.gov/genomics/2012/09/27/evidence-matters-in-genomic-medicine-round-3/) into the three-tiered system in blogs that are available on the OPHG website. Here, we describe the methodological underpinnings of this system and use the example of pharmacogenomics tests to illustrate how this system can be used to classify genomic applications.

We supplement our three-tier classification system with a color scheme that emulates the workings of a traffic signal to emphasize that the evidence under consideration relates to whether or not the application is likely to be considered for action. Our classification scheme stratifies applications into three categories:

We have developed a simple classification method to aid in the tier assignment process. When coupled with the aforementioned online table, the method may be viewed as the basis of an interim, working process intended to inform critical evaluation of genomic applications by end users until the evidence base becomes more robust and more comprehensive resources (e.g., ClinGen, funded by the National Institutes of Health, which aims to catalog medically relevant human gene variants) become available. It is intended to serve as a launching point for health researchers and as a guideline for developers, policy makers, and others looking for a basic overview of the amount of synthesized evidence relevant to particular genes or tests. It offers a first, quick look at how much synthesized evidence is available for use in decision making, as well as a subjective overview of what that evidence has to say about implementation. We have designed the approach primarily to benefit an audience of clinicians and public health policy makers interested in implementing specific genomic applications. This horizon-scanning method can also point quickly to existing evidence gaps and/or lack of guidelines and recommendations. Our approach is not, however, intended to supplant clinical decision making. The approach does not prescribe how evidence synthesis or guidelines can be arrived at. The following sections describe the method and illustrate its application through the lens of pharmacogenomic applications. In tandem, we highlight salient characteristics of the processes currently used for guideline development and evidence synthesis by the many groups working to address the field of genomic medicine.

Synthesized evidence on genomic applications can be considered the input for our method, and the output is an evidence-based classification of applications. Thus, the process will not yield conclusions such as “testing for gene X is a Tier 2/Yellow application.” Instead, conclusions are of the form “testing for gene X to predict susceptibility to disease Y, in people with characteristics Z_1,2,…n is a Tier 2/Yellow application.” The classification method (Figure 1) incorporates evidence derived from US Food and Drug Administration (FDA) labeling information, Centers for Medicare & Medicaid Services (CMS) coverage decisions, clinical practice guidelines (CPGs), and systematic reviews (SRs). Classification centers on (i) intent of guidance (e.g., to inform decisions on use of a test vs. decisions on use of the results of a test) and (ii) applicability of sources, rather than methodological rigor in review or guideline development processes.

Guidelines reporting their basis to be an SR are categorized by our process as evidence based, regardless of whether the SR has been published or is publicly available. If a review process appears to have involved prespecified search terms regarding key questions, we consider it to be an SR, even if the authors did not label it as such. Examples of some major sources of evidence, and how they could be classified for use in our method, are provided in Table 1. We have not included guidelines and reviews that are primarily aimed at facilitating laboratory practice.

The genomic application to be classified must be clearly described before relevant evidence can be identified. Our definition process is based on methods for topic development used by the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Working Group, and we echo its view-point that “. . . clear definition of the clinical scenario is of major importance, as the performance characteristics of a given test may vary depending on the intended use of the test, including the clinical setting (e.g., primary care, specialty settings), how the test will be applied (e.g., diagnosis or screening), and who will be tested (e.g., general population or selected high risk individuals).”⁸ Thus, inspired by the EGAPP and the precedent ACCE (analytic validity, clinical utility, ethical legal and social implications) model process,⁹ we consider the disorder, the test, and the context in which the test is used (the clinical scenario) in describing specific genomic applications.⁸

Once the genomic application has been specified, the first step toward classification is to identify the synthesized evidence to be considered. The CDC OPHG has previously created a broad-based horizon-scanning process to track the introduction of genomic applications into clinical research and practice,¹⁰ and we assess the findings for sources of synthesized evidence on genomic applications. We use regular, weekly PubMed searches to identify relevant literature, such as published SRs, and Google Alerts to probe key areas of the gray literature. We also regularly check the websites of major guideline development groups, such as the US Preventive Services Task Force,¹¹ the American College of Medical Genetics and Genomics (ACMG),¹² and many others, for new or updated guidelines. General resources such as the National Guideline Clearinghouse,¹³ National Institutes of Health Genetic Testing Registry,¹⁴ PharmGKB,¹⁵ and PubMed¹⁶ are also highly effective tools in searching for evidence. PharmGKB also includes criteria for levels of evidence for variant-drug combinations, and although these criteria differ from those presented here, the results are nonetheless useful in determining what type of evidence is available to support use of pharmacogenomics tests. The information we find on newly introduced tests and guidelines is searchable online at the Genomic Applications in Practice and Prevention Knowledge Base (GAPP KB) (http://www.hugenavigator.net/GAPPKB/home.do), and any information identified from synthesized evidence sources is used to update the table of genomic tests by level of evidence⁷ on an as-needed basis.

Identifying relevant FDA-approved drug label information is relatively simple using the FDA Table of Pharmacogenomic Biomarkers in Drug Labels,¹⁷ which points directly to the appropriate label sections. However, this table may not necessarily be a complete listing, e.g., cytochrome P450 (CYP) 2D6 inclusion in the label for sertraline hydrochloride is not found in the current FDA table; therefore, additional sources may increase yield. The FDA 510(k) premarket notification database is a good source to search for approved tests that may be required in FDA labeling¹⁸; however, search fields require some previous knowledge of the device, applicant, or other specialized information. CMS coverage decisions are also relatively easy to locate in the Medicare Coverage Database¹⁹; however, the user must devise appropriate search terms to retrieve relevant information.

After synthesized evidence for a genomic application has been identified and collected, the list of criteria (Figure 1) is simply read from top to bottom. Four sources of evidence are considered: (i) FDA drug label information, including companion diagnostic designation; (ii) CMS national coverage determination, local coverage determination, and coverage with evidence development (CED) determination; (iii) CPGs; and (iv) SRs. Examples of some major sources of evidence, and how they could be considered using our method, are provided in Table 1. If any of the evidence for the test conforms to one or more of the bulleted descriptions in the Tier 1/Green category, then the process is complete, and the assignment is Tier 1/Green. If none of the Tier 1/Green descriptions apply, then the process is repeated for the Tier 2/Yellow category and then for the Tier 3/Red category, if necessary. In general, if more than one bulleted description can be applied, then the higher tier/category is assigned. An exception occurs when the Tier 2/Yellow description “FDA label mentions biomarker” applies; in this case, reassignment to the lower category remains possible if any Tier 3/Red descriptions also apply.

This section illustrates the use of our horizon-scanning method with several specific examples. The examples have been chosen to (i) demonstrate how the method works and (ii) present cases for which classification is difficult and involves greater use of subjective judgment. Pharmacogenomic examples are used wherever possible because they are often the most challenging in terms of classification. Readers wishing to examine results for all the genomic applications that we have classified to date are encouraged to look at the evolving online table of Genomic Tests and Family History by Levels of Evidence that is maintained and updated periodically on the CDC OPHG website.⁷

In beginning to address questions of clinicians, healthcare payers, and patients about whether or not to offer a genomic application, cover it, or use it, simple heuristic devices can help in gathering, organizing, and understanding existing relevant evidence. Our classification method and associated table are primarily designed to answer two questions: (i) what are the sources of synthesized evidence on a particular genomic application and (ii) what do these sources have to say, in general terms, about whether or not the genomic application is ready for implementation. They are not intended to answer the ultimate question of whether or not to implement an application; rather, we hope they will help to direct decision makers toward genomic applications for which there is some supporting synthesized evidence so that more formal decision-making techniques can be focused on the most promising areas. For those who wish to base implementation on current sources of synthesized evidence, Tier 1/Green and Tier 2/Yellow listings should be of immediate assistance in obtaining an overview of which genomic applications are associated with existing supportive evidence from CPGs and SRs, as well as with information from the FDA and the CMS.

Our method classifies genomic applications into one of three categories, with implications for what available evidence has to say about implementation. In addition, this scheme points to key evidence sources that can be useful in informed decision-making processes. The approach is flexible so that modifications in evidence sources considered, or in how they are weighted, can be tailored to meet the needs and address the values of various decision makers. In many instances, the classification results convey a general idea of the types of research needed to strengthen an existing evidence base or to determine whether a genomic application may be productively implemented in the future.

We are using the classification method presented here to populate an evolving online table of Genomic Tests and Family History by Levels of Evidence.⁷ Because the authors are involved in public health–driven approaches to genomic implementation, being associated with CDC OPHG and/or the National Cancer Institute’s Epidemiology and Genomics Research Program, our approach may favor higher evidence thresholds than some would propose. Arguments could be made that applying common thresholds to all genomic applications is not warranted, or that implementation might be reasonable in some situations where no synthesized evidence is likely to be available, such as with some rare disorders. Therefore, in deciding which source material to consider “evidence-based,” we have taken a some-what liberal view. Ultimately, we acknowledge that this is an area of widespread disagreement,⁴⁶ and others may choose to use our classification method with more or less restrictive criteria to account for different priorities and values. Ideally, as synthesized evidence continues to accumulate, we might expect to see movement of genomic applications away from the Tier 2/Yellow category as clear recommendations for or against use become increasingly possible. We are committed to updating the online table on a regular basis in order to test this hypothesis. We are also interested in engaging in active dialogue with researchers, policy makers, providers, and test developers on the best approaches to collecting and synthesizing evidence for genomic applications in specific scenarios. Ultimately, we plan to keep the online table as current as possible, as an ongoing service to decision makers in the field. For those whose philosophy on thresholds diverges from our approach, the method may still be helpful as a basis for modifications.

The incorporation of evidence derived from US federal sources (e.g., the FDA and the CMS) make our classification method and table a somewhat US-centered tool; however, the system could easily be modified to meet the specialized needs of other regions, e.g., by substituting European Medicines Agency information for FDA data. Our other sources, such as CPGs and SRs, have not been limited to those produced in the United States or those based on studies of the US population. Our system involves classification based on synthesized evidence in the form of SRs and CPGs rather than individual studies. We recognize that individual studies (whether clinical, cost focused, modeling based, or others) are crucial for the successful implementation of genomic tests; nevertheless, such studies must be synthesized into SRs, CPGs, and FDA and CMS decision processes if we are to thoroughly understand how they fit into the larger evidence base. Some may object to Tier 1/ Green assignment based solely on FDA labeling or CMS coverage decisions because these processes do not exactly match the model proposed in the Institute of Medicine report titled “Clinical Practice Guidelines We Can Trust.”⁴⁷ Our rationale for their inclusion is that FDA drug labeling and CMS coverage decisions are constructed through defined, deliberative processes that use the best available evidence. These sources warrant a similar level of consideration as high-quality CPGs (based on SRs) when considering implementation strategies. For example, the CMS convenes the Medicare Evidence Development & Coverage Advisory Committee as an independent consensus advisory panel that “judges the strength of the available evidence and makes recommendations to CMS based on that evidence.”⁴⁸ Moreover, we included CMS and FDA into our tiering scheme where others have not as we felt that their regulatory and/or cleavage decisions affect the use of these applications.

It might be argued that the Tier 3/Red category contains two qualitatively different subsets that should not be combined: (i) applications with recommendations against use and (ii) applications for which no synthesized evidence in the form of a CPG, an SR, or CMS or FDA information is available. For those who may be interested in conducting additional research and incorporating the latest primary study results into decision making, combining these subsets in the Tier 3/Red category may not be ideal. We have designed the system primarily to benefit an audience of clinicians and public health policy makers interested in implementing genomic applications, or not, on the basis of existing synthesized evidence. For this audience, Tier 3/ Red applications are expected to be of little interest until some supporting synthesized evidence is available.

An acknowledged weakness of our approach is that our classification system is not designed to finely differentiate among guidelines on the basis of quality criteria or rigor because not all guidelines are created equal.⁴⁷ In addition, the use of different conventions in evaluating evidence and reporting of results among various groups creating SRs and CPGs cannot be corrected for in rough classification systems such as ours. More indepth evaluation is possible using instruments such as AGREE II (Appraisal of Guidelines, Research, and Evaluation)^49,50 and has been done quite effectively to compare genomic guideline products⁵¹ and methods.⁵² Instead, we have focused on the intent and general conclusions of evidence sources (e.g., informing use, choice, and dose) while providing a rough classification involving a few source types (SR, CPG, FDAapproved drug label, and Medicare coverage) that are critical for successful implementation. Because of this, our classification process can generally be completed and updated more quickly than is possible with more detailed instruments.

Another weakness of this and any other summary of evidence is that clinical scenarios may not always be specified or well defined in source documents. Even when clinical scenarios and recommendations are clear, some degree of interpretation, condensation, and/or simplification is necessary when presenting results in tabular form. Critical nuances included in FDA-approved drug labels, CMS determinations, CPGs, and SRs may be missed during the classification process. Our classification method and evolving table are designed to make rough interpretations of the evidence base and point users to the relevant sources of evidence, thus saving time spent searching. Nevertheless, decision makers must actually read the indicated evidence sources in order to effectively apply their findings.

No horizon-scanning system will be perfectly sensitive, and it is a certainty that we have not collected all of the synthesized evidence. There are also enormous challenges associated with maintaining a horizon-scanning system and organizing results, due to the rapidly evolving evidence base in public health genomics. Even as final revisions to this article are being completed, our horizon scans have identified an SR that may have the potential to elevate the Tier 3/Red example used in this article to Tier 2/Yellow.⁵³ We will continue horizon scanning and classification of results on a continuous basis, acknowledging that classifications must be expected to change as evidence develops and evolves. Our system has led to an expanding and evolving table of Genomic Tests and Family History by Levels of Evidence,⁷ which we hope will prove useful as a rough guide to stakeholders in the field of genomic medicine.

The original impetus for our classification system came from a 3 × 3 risk–benefit policy matrix, risk–benefit profile by level of uncertainty, developed by Veenstra et al.⁵⁴ Although an enormously useful tool, we sought to develop an approach that did not require formal decision-analytic modeling and resulted in fewer and simpler categories. We hope that our simplified approach will prove useful in helping to shape future studies. Ongoing translational research can be tailored specifically to each of the tier/color classification levels. For applications in Tier 1/Green, research is needed on effective integration of tests into practice, whereas evidence on when to implement assumes increased importance in Tier 2/Yellow scenarios, and accumulating relevant evidence on clinical utility will be needed. A key question for Tier 3/Red scenarios is whether, and in which cases, additional research on validity and utility might be justified. Although our classification system cannot provide detailed answers to these questions, we hope that it will help to identify, organize, and prioritize future research while serving decision makers as a heuristic to guide the beginnings of more detailed assessments of evidence.

Note: While this article was in press, new data were published suggesting that no improvement in anticoagulation control with warfarin resulted from adding genotype to clinical variables in a dosing algorithm (Kimmel et al., N. Engl. J. Med. 19 November 2013, Online First). The results of this original study do not immediately affect placement in our classification scheme but could be integrated into future guidelines or systematic reviews. Rapid progress in the field underscores the need for continuous horizon scanning of the evidence.