Assessment includes the evaluation, integration, and interpretation of several sources of information (e.g., outcomes of different measures, tests, or scans) to derive a valid and reliable decision (e.g., a best-estimate diagnosis) [8]. Accurate assessments are crucial. In clinical practice, under- or over-estimation of illnesses can have severe negative impacts on people's lives. In research, inaccurate assessments threaten the validity of scientific results. For policy and implementation development, assessments are the basis for guideline development and the economic and societal evaluations of interventions. Furthermore, obtaining more accurate assessments has become increasingly important considering that high-accuracy assessments are needed in diverse fields such as Biological Psychiatry (e.g., to find reliable biomarkers linked to reference standard assessments [9–11]) and Artificial Intelligence (e.g., to train models to reference standard assessments [12–14]).

Here, we connect three bodies of literature that have proposed similar assessment methods: The Expert Panel method in medicine [7,15,16] – as well as the Best-Estimate Diagnosis [6,17] and the Longitudinal Expert All Data (LEAD [5]) methods in clinical psychology and psychiatry. The three methods share the same goal of attaining best-estimate assessments through similar methodological approaches: All three methods use expert panels or consensus teams (e.g., clinical psychologists or medical doctors) to review several sources of information (e.g., clinical questionnaires and medical tests) to establish a more accurate assessment (e.g., a best-estimate diagnosis).

The Best-Estimate Diagnosis method was introduced by Leckman et al. [6] as a strategy to set accurate lifetime psychiatric diagnoses. The method focuses on two components, namely 1) using all data (e.g., information from medical records and relatives in addition to interviews) that is 2) evaluated by expert clinicians (who review all data and then reach a consensus [17]). Consequently, Spitzer [5] proposed the Longitudinal Expert All Data (LEAD) method to obtain a criterion or reference standard to validate the Diagnostic Interview Schedule [18] for setting psychiatric diagnoses. LEAD extends the Best-Estimate Diagnosis method and involves three essential components, namely Longitudinal data collection (i.e., not limited to a single examination at one point in time), Expert evaluation (i.e., the diagnoses are set by expert clinicians), and All Data (i.e., the experts have access to multiple data sources). A similar approach was proposed in medicine, where the Expert Panel method was developed as a solution for diagnostic accuracy studies with an imperfect or missing reference standard [16,19,20]. Here, a panel of experts decides on a medical condition based on all relevant information. A review of expert panel studies [7] identified four critical components of the expert panel design, namely 1) the panel constitution, 2) the information presented to the panel, 3) the decision process of the panel, and 4) the validity of the panel diagnosis.

So, all three methods employ a similar approach to obtain best-estimate assessments (e.g., for diagnostic purposes or as a reference standard) while accentuating parts of it: The Best-Estimate Diagnosis method accentuates the use of informants and objective tests next to self-reported data [6,17]; the Expert Panel method focuses on the characteristics, constitution, and procedure of the panel [7,15] and only the LEAD method requires a longitudinal design [5], although longitudinal data are also used in some Expert Panel designs (≈27 % of studies [7]). Herein, we collectively refer to these three approaches as the assessment methods.

The result of the assessment methods is a consensually derived criterion (e.g., a best-estimate assessment) that has been used for many different applications where there is no single error-free measure. It has, for example, been used to i) evaluate the accuracy of a measurement tool or marker through comparison to a best-estimate assessment [21–26]; ii) establish the prevalence of symptoms and disorders [27–29]; iii) establish the temporal stability or development of symptoms and disorders [30–32]; iv) improve (earlier) detection or screening of symptoms or disorders [33–35]; v) study genetics and family history [36–38]; and vi) examine classification systems or diagnostic criteria [39–41]. The applications span diverse fields, including medicine, psychiatry, clinical psychology, public health/epidemiology, and artificial intelligence. Box 1 provides more examples of how the best-estimate assessments have been applied in different types of studies across fields.

The assessment methods possess a high potential for achieving best-estimate reference standards in many situations. However, the quality of such proclaimed best-estimate assessments varies substantially and is typically very difficult to evaluate due to poor reporting of the method how they were achieved (e.g., see reviews of expert panels [7,15]). A systematic review of assessment methods and reporting of expert panels [7] has demonstrated that the methods used for panel or consensus diagnoses vary substantially across studies and that many aspects of the procedure are often unclear or not reported at all (i.e., in 83 % of the reviewed studies). Many recent studies fail to report central aspects of the assessment procedures, including the quality, structure, or presentation of the data [43], the training and qualifications of the experts [44], the method for avoiding biases and achieving consensus [45], and the time span of the longitudinal design-component [46]. The poor operationalization of the assessment methods jeopardizes the goal of achieving best-estimate assessments – where a vaguely described method makes it difficult to evaluate the research. Referring to an assessment as a best-estimate (and sometimes even as a gold standard) while vaguely describing or poorly operationalizing the method for achieving the assessment is alarming [47,48].

These assessment methods aim to achieve high validity (i.e., the degree to which the assessment captures what it aims to measure). Typically, the assessment methods aim to achieve as high validity as possible (i.e., a “leading” assessment) or, depending on resources, at least more accurate than a single error-prone measure. Despite this central aim, research often fails to clearly describe the degree of validity of the attained assessment. Using these assessment methods does not automatically guarantee high validity – it depends on how well the method is executed.

In addition, the derived assessments are often described with different terms: reference standard is often used in medicine, and criterion standard or best-estimate diagnosis is often used in psychology. We propose that the reporting of these assessment methods benefit from more explicitly describing what was measured and how well it measures up to different standards – whether and how they relate to a state-of-the-art assessment. Whereas reference and criterion standards fail to convey an intention of “nearing” a state-of-the-art assessment, the best-estimate diagnosis narrowly focuses on the classification of a diagnosis and not on symptom severity. Therefore, we here use the term best-estimate assessment in the context of describing a “leading”, state-of-the-art assessment.

Previous well-established guidelines have focused on the complete reporting of specific study designs, such as the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE [49]) for observational studies; the Statement for Reporting for Diagnostic Accuracy (STARD [50]) for diagnostic accuracy studies; the Consolidated Standards of Reporting Trials (CONSORT [51]) for randomized trials, and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD+AI [52]) for prediction model studies (see the supplementary material [SM] for other relevant guidelines). The STARD guidance is most closely related to the reporting of the assessment methods since best-estimate assessments are often used to evaluate a measure's (diagnostic) accuracy. However, none of the guidelines are sufficient for complete reporting of the assessment methods, where (multiple) experts review several sources of (longitudinal) information to form a best-estimate assessment. Although an earlier systematic review identified and structured the various choices involved in Expert Panel procedures [7], no attempt was made to develop a formal guideline for the reporting of Expert Panel assessments.

Our aim is to develop reporting standards for comprehensive reporting of Expert Panel, Best-Estimate Diagnosis, and LEAD methods – which can help researchers plan and report studies employing these assessment methods, as well as help readers evaluate them. We call the reporting guideline the LEADING guideline, emphasizing the methodological components and the importance of describing what is assessed and how well (i.e., how it relates to a “leading” assessment). The individual reporting standards are divided into four groups according to the components of LEAD (Longitudinal, Expert, All Data [5]), from which we revised the original meanings to Longitudinal, Evaluation – experts, materials and procedures, Appropriate Data, and Validity. In short, the LEADING guideline aims to guide the reporting of assessment method to improve evaluations of the assessment standard.

The working group, with the support of the steering group members, identified relevant research using or describing the assessment methods, including the three bodies of literature: Expert Panel [7], Best-Estimate Diagnosis [6], and LEAD [5]. In addition, articles using any of the three assessment methods were identified through a literature search using Google Scholar with the following search terms: [“expert panel diagnosis” OR “expert panel assessment” OR “expert panel consensus” OR “expert panel methodology” OR “expert panel standard” OR “expert panel reference”] for Expert Panel studies; [“best-estimate diagnosis” OR “best-estimate diagnostic” OR “best-estimate standard” OR “best-estimate assessment” OR “best-estimate methodology” OR “best-estimate reference”] for Best-Estimate Diagnosis studies; and [“longitudinal expert all data” OR “longitudinal evaluation all data”] for LEAD studies. Articles that clearly stated using one of the three assessment methods were selected. Articles stating another purpose than assessment (e.g. when an expert panel was used to reach a consensus about a treatment strategy) were excluded.

Furthermore, relevant reporting guidelines and systematic reviews were identified, including a review of expert panel applications [7], the STROBE statement [54], and the STARD guidance [50]. Other complementary reporting guidelines and systematic reviews are presented in the SM. The aim was for the reporting standards in the LEADING guideline to complement rather than repeat them (i.e., new standards should extend or complement existing standards rather than repeat them [53]). For example, when reporting a randomized trial that includes best-estimate assessments, one may use CONSORT [55] to report the trial design and main results, the LEADING guideline for describing the specifics for reaching the best-estimate assessments, and the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) [56] for reporting the economic evaluations and comparisons.

Potential standards were drafted by the working group with the objective of encompassing a comprehensive reporting of the assessment methods. The reporting standards were grouped into four groups: Longitudinal design, Appropriate data, Evaluation – experts, materials and procedures, and Validity. Empirical and theoretical inclusion rationales were stated for the groups and the individual standards (i.e., explanations and elaborations). Lastly, the standards with inclusion rationales were further developed through a workshop with a patient organization for depression, followed by feedback from the steering group members to receive a wide range of perspectives early in the process.

To systematically collect expert feedback from different perspectives, we used a consensus-building procedure called the Delphi technique [57]. We used an iterative process based on two rounds of questionnaires (i.e., Delphi surveys), enabling feedback from round 1 to feed into round 2. Delphi participants received relevant background research, the reporting guideline aims, and the reporting standards with their inclusion rationales. They provided feedback through open- and closed-ended response formats. Through open-ended responses in Round 1, experts could propose new standards and provide feedback on the formulations of existing standards and their inclusion rationales. In addition, two closed-ended questions [58] about standard inclusion (This item should be included in the reporting checklist) and perception of study quality (Whether this information is present or not would influence my perceptions of the quality of a study) were answered with rating scales ranging from 1 = Strongly disagree to 7 = Strongly agree. In Round 2, the experts were asked to rate the updated reporting standards using the same two closed-ended questions as in Round 1 and to provide feedback on the clarifications and reformulations through open-ended responses.

To recruit Delphi participants, the first and/or last authors of articles since 2013 using any of the three assessment methods were identified using the search terms described above (n = 87 articles, n = 124 authors; see the SM for more details). These authors and the seven steering group members were invited via email to participate in the Delphi Round 1 (n = 131 participants emailed). In total, 27 participants completed the survey (response rate 21 %). Only participants from Round 1 who provided their contact details were invited to Round 2 (n = 25). In total, 20 participants completed the survey (response rate 80 %). All participants provided their informed consent. Fig. 1 presents the research experiences and demographics of the Delphi participants. Participants reported a wide range of academic backgrounds (e.g., Clinical Psychology, Psychiatry, Medicine, Artificial Intelligence, Journal Editors) and an extensive variety of relevant methodological experiences (e.g., Ecological Momentary Assessments, Biological Markers, and Expert Panels; Fig. 1), with an age range of 30–70 years (M = 51.54, SD = 12.40).

The authors finalized the guidelines in an expert consensus meeting. The meeting was held online with nine working and steering group members. The content and structure of the consensus meeting were prepared by the working group, and the meeting was led by the last author (O.K.). Participants had access to the guidelines, inclusion rationales (i.e., elaboration and explanation), and the drafted paper before the meeting, where they also had the option to provide comments and feedback in writing. The meeting included reviewing the Delphi Rounds 1 and 2 findings and discussing the paper draft, including the individual reporting standards and groups. We decided not to carry out another Delphi round since i) the median agreement for each reporting standard in both Delphi Rounds 1 and 2 ranged from 6 = Agree to 7 = Strongly Agree, ii) no new standards were suggested, and iii) only minor changes were needed after Round 2, which taken together suggest consensus.

To test the applicability of the guideline, the guideline was tested i) by independent researchers with experience of each method piloting the reporting of each standard and ii) by the authors (V.E., K.K.) applying it to published articles. The two test procedures resulted in adding minor clarifications to three standards (2.4 The access to the index measure, 3.3 Blindness and conflict of interest, and 3.4 Instructions and training). Also, a concrete example of how to report the items was added to the guideline instructions.

Applying the guideline to a random selection of 30 articles indicated severe heterogeneity in what of the methods is reported and how (Table 2; see the SM for the search strategy). Across the 30 studies, 10 to 63 % (Mean = 33 %) of the standards were not reported. Regarding the reporting standards, the type and quality of the data (2.1), the access to the index measure (2.4), the expert and panel characteristics (3.1), the number of experts and panels (3.2), the assessment procedure (3.5), and the assessment description (4.1) were mostly reported (i.e., green in more than 50 % of the studies). However, the data presentation (2.3), the instructions and training (3.4), the data combination method (3.7), the inter-rater and inter-panel reliability (3.9), and the validity and standard (4.2) were not reported at all in the majority of the studies (i.e., red in more than 50 % of the studies). Considering that most changes that resulted from discussing disagreements between the raters (V.E. and K.K.) were from green to orange, and that green refers to a (minimally) sufficiently reported and orange to insufficiently reported, this suggests that the table is conservative in regards to the severity of the current state of poor reporting (i.e., potentially showing a more positive picture; for more information see the SM).

We have connected three assessment methods with similar approaches from related fields and drafted applicable reporting standards. We presented the rationale for these three methods and each reporting standard with supporting evidence in the Delphi survey for review, which did not bring up additional methods or reporting standards. As we did not carry out a systematic literature review, we cannot exclude the existence of other assessment methods with similar approaches. We welcome any suggestions about similar methods to which the guideline is applicable.

The Delphi survey participants and the author group had a wide range of experiences and backgrounds; however, geographically, Europe and North America were the most common, whereas several areas were not represented. The Delphi participants were the first or last authors of studies employing the assessment methods. The quality of the articles and the education or experience of the authors were not taken into account as selection criteria (although it was self-reported, as presented in Fig. 1). The number of Delphi participants (27 in Round 1, 20 in Round 2) is relatively small compared to some other standard developments (e. g., 73 in the development of STARD [88]) but comparable to others (e.g., 24 for the development of the TRIPOD statement [89]). Even though the response rate in Round 1 (21 %) can be considered low, the number of participants was sufficient to cover a broad range of academic backgrounds, methodological experiences, and demographics (Fig. 1). The same limitation is applicable to the size of the steering group (n = 7) and the current test-user group (n = 2).

The LEADING guideline emphasizes the transparent reporting of the methodological components of Expert Panel, Best-Estimate Diagnosis, and LEAD designs and the importance of reporting what was assessed and how well. Considering the increasing need for high-accuracy assessments in diverse fields, we hope that the LEADING guideline will be useful in assisting researchers in planning, carrying out, reporting, and evaluating research that aims to achieve accurate assessments.