Autism spectrum disorder (ASD) is a neurodevelopmental disorder defined by a profile of persistent challenges in social communication and interaction and the presence of restrictive, repetitive patterns of behavior, interests, or activities that are present early in childhood and cause clinically significant functional impairments (APA, 2013). At the individual level, being diagnosed with ASD can inform intervention, supports, and understanding of a person’s strengths and challenges. At a community or population-level, diagnoses of autism spectrum conditions can inform policies, services, and supports available and may impact ultimate functioning. Understanding why and how people are diagnosed with behavioral conditions such as ASD can provide insight into the meaning of ascribing such labels for individuals and communities. This paper describes an iterative process of specifying and reliably applying exemplars to endorse the presence/absence of each of the core ASD DSM-5 criteria and to, ultimately, classify children as meeting an ASD diagnostic classification. This goal is to increase conceptual and practical consistency in the way clinicians consider behavioral criteria for diagnosis ASD.

Autism was first described as a condition by Kanner (1943) and “Infantile Autism” was first included in the Diagnostic and Statistical Manual (DSM) in 1980 as one subtype of several “Pervasive Developmental Disorders” (PDDs) (APA, 1980). ASD is now considered in the DSM-5 as a single categorical spectrum of behaviors with heterogenous core and co-occurring features (APA, 2013). Across taxonomic changes, diagnosis has relied on behaviorally-defined diagnostic criteria to meet a categorial diagnosis (Volkmar & McPartland, 2014). Guided by current diagnostic criteria, professionals with appropriate training, credentials, and experience (Shulman et al., 2020a) assess a person’s profile of behavior to determine if the evidence supports endorsing persistent deficits in the domains of Social Communication and Interaction (SCI) and Restricted and Repetitive Behaviors or Interests (RRBI) that are present early, cause impairment, and are not better explained by other conditions, specifically intellectual disability or global developmental delay (APA, 2013).

Given the behaviorally-defined and heterogenous nature of an ASD diagnosis, clinician judgment is an important factor to consider in arriving at an ASD diagnosis. Studies comparing community-based diagnoses of ASD with an independent assessment battery and best-estimate diagnosis found a lack of agreement on ASD diagnoses for 1/4 of the children (Hausman-Kedem et al., 2018; Wiggins et al., 2020). In another study, only 60% of differential diagnoses for ASD were made with high clinician certainty (McDonnell et al., 2018). An analysis of clinical team diagnostic discussions indicated that subjective “feelings of ASD” and professional evaluation of the quality of informant reports were prominent among clinicians when making the diagnosis (Hayes et al, 2020). Given the identification of ASD by affirming the presence of a profile of relevant behaviors, understanding the way different professionals ascribe specific behaviors as evidence of the diagnosis can add specificity to clinical and research efforts to link symptoms with treatment or with risk factors or etiology in research.

Research has focused on developing and evaluating tools to increase the reliability of autism spectrum diagnoses (including “autistic disorder” and other PDD historical subtypes) for both research and clinical purposes compared to “best-estimate clinical judgment” of experienced research clinicians (Bishop et al., 2017; Hausman-Kedem et al., 2018; Kamp-Becker, 2018; Klin et al., 2000; Lord et al., 2012a, 2012b). Specific behaviors that constitute the way these criteria are identified, measured, and applied individually and together are often guided by the specific questions asked in diagnostic instruments (Aiello et al., 2017; Rutherford et al., 2016; Shulman et al., 2020b; Wiggins et al., 2015). Instruments, such as the Autism Diagnostic Observation Schedule (ADOS-2) (Lord et al., 2000) and Autism Diagnostic Interview-Revised (ADI-R) (Lord et al., 1994), have organized specific autism-relevant behaviors rated from person observation and caregiver interview and provide research-based algorithms that determine whether the person satisfies the instrument criteria for ASD with validation comparing to clinical diagnosis. However, meeting the threshold on a single instrument is not sufficient for diagnosis. Ultimately, ASD diagnosis depends on clinician confirmation of the DSM-5 criteria from multiple sources of information (Shulman et al., 2020a; Volkmar & McPartland, 2014), and little attention has been given to how clinicians identify and endorse behavioral exemplars that constitute the evidence needed for an ASD diagnosis. Despite robust research on the overall diagnosis and supporting subdomains (Bishop et al., 2016; Frazier et al., 2012; Lord et al., 2012a, 2012b; McPartland et al., 2012; Thurm et al., 2019; Uljarevic et al., 2017; Zheng et al., 2020), research on behavioral exemplars that clinicians consider in endorsing the social and behavioral diagnostic criteria is lacking.

The DSM-5 ASD criteria were described broadly to encompass a range of symptom presentations across age and developmental level. Beyond the initial categorical diagnosis (ASD/non-ASD), further indicators of functioning such as language and cognitive levels, adaptive behavior, and co-occurring symptoms and features also must be specified (APA, 2013; Bolte & Diehl, 2013; DiRezze et al., 2016; Gardner et al., 2018). Moreover, some investigators recommend shifting from dependence on overall (categorical) diagnosis to recognition of specific phenotypic features, behaviors, and/or biologic pathways and to intervention research that cuts across behaviorally-defined psychiatric conditions (Constantino & Charman, 2016; Cuthbert & Insel, 2013). Clinical and research trends have emphasized a dimensional approach in overall symptom distribution across the population (Constantino & Todd, 2005; Riglin et al., 2020), subdimensions of primary social and behavioral domains of ASD (Bishop et al., 2016; Uljarevic et al., 2017), and stipulation of specifiers such as cognitive level, language capacities, and the presence of other co-occurring conditions (APA, 2013; Gardner et al., 2018). With the move to “lump” the overall diagnostic category into a single ASD diagnosis without subtypes, and “split” the dimensional aspects across the profile of symptoms and into domains and by co-occurring features, the challenge has become establishing valid and reliable categorical and dimensional ways of characterizing behaviors to inform etiologic and treatment research and meaningful supports (Gardner et al., 2018; Lord et al., 2012a, 2012b). Careful specification of the behaviors representing each of the ASD diagnostic criteria may be informative in clarifying a dimensional approach to the spectrum of presentation representing each criterion. For example, the diagnostic criterion A3 “deficits in developing, maintaining, and understanding” relationships is likely to be influenced by multiple dimensions such as age, verbal and nonverbal communication skills, cognitive profile, intensity of other autism spectrum behaviors, etc. Evidence of the deficits in relationships applied to a toddler with minimal expressive verbal language and limited attention to other people manifests differently than for an adult with above average expressive language who lacks flexible understanding of unstated social rules; however, the end result may still be sufficient criteria to endorse criterion A3. Documenting behaviors across age and developmental levels that represent the expression of each of the ASD social and behavioral criteria is important in evaluating the reliability, validity, and utility of further defining each diagnostic criterion.

Specification of the behaviors within each criterion may also be useful for reliably classifying ASD over time in the population (Arvidsson et al., 2018; Durkin et al., 2015; Miller et al., 2013). According to the CDC’s Autism and Developmental Disabilities (ADDM) Network, ASD prevalence increased from an estimated 1 in 150 in 2000 to 1 in 59 among 8-year-old children in 2014 based on DSM-IV-TR (CDC, 2007; Baio et al., 2018) and 1 in 54 children in 2016 based on DSM-5 criteria (Maenner et al, 2020). ADDM prevalence estimates from surveillance years 2000–2016 were derived from multiple sites across the US using consistent procedures previously described (Bertrand et al., 2001; CDC, 2007; Rice et al., 2007; Yeargin-Allsopp et al., 2003). Across these years, ADDM ASD case confirmation required clinician reviewers to reliably apply a detailed assessment of behavioral exemplars documented in existing evaluation records as meeting evidence of the DSM-IV-TR autism/PDD criteria. Establishing and maintaining reliability across multiple sites and clinicians in endorsing each criterion has been dependent on operationalizing the diagnostic criteria.

With the advent of the DSM-5 criteria, a process was undertaken to update operationalized criteria for ASD in a manner that could be replicated in an effort to specify key behaviors associated with the autism spectrum behind the overall and domain levels in clinical and research diagnoses of ASD. The objective of this paper is to describe an iterative process used to define the spectrum of exemplars for each of the seven core DSM-5 ASD criteria and associated features to determine ASD case status, and to evaluate interrater reliability at the criterion and case classification levels.

Identifying and coding DSM-5 exemplars for each of the diagnostic criteria followed procedures previously advanced by the ADDM Network. ADDM has published population-based estimates for ASD every two years since surveillance year 2000 (CDC, 2007) through 2016 (Maenner et al, 2020). In these reports, ADDM procedures involved a two phase approach with phase 1: abstracting diagnostic and behavioral descriptions from developmental evaluation reports contained in health and education service records (including professional completing the evaluation, reason for referral, all text descriptions of development and behavior, type and results of ASD assessment tools, adaptive and intellectual assessment results, and diagnostic summary); and phase 2: reviewing the information to identify and code DSM-IV-TR (APA, 2000) behaviors and associated characteristics; and using a standardized coding procedure to classify a child as ASD or non-ASD based on clinician review of behaviors described in developmental evaluation reports and by consensus of clinicians when certainty of classification was in question. The standardized coding procedures were outlined in an ASD clinician review manual created to establish and maintain consistency in classification of ASD across a network of clinicians with varied of clinical and research backgrounds.

The standardized ADDM coding procedure used involved line-by-line review of information abstracted from health and education records by a study clinician to determine if text descriptors exemplified the DSM criteria, associated features (AFs), or other coding items such as timing and type of early developmental concerns. Based on the exemplars identified and the DSM criteria endorsed across all available information in the record, the child was classified as an ASD Case (meeting DSM-5 ASD criteria), Suspected Case (meeting some, but not all criteria and not counted in final case counts), or Not Case (no evidence of social criteria or previous ASD diagnosis or special education eligibility). Clinicians also provided ratings based on their overall assessment of the quality and consistency of the information to indicate their degree of certainty of the final classification, and levels of impairment. Based on all information in the composite record, including IQ and Adaptive test scores, when available, the clinicians’ overall certainty included their evaluation of the ASD diagnostic criteria of C. D. and E. (APA, 2013), indicating early manifestations of symptoms, impairment, and not being better accounted for by intellectual disability, global delay, or other potential conditions. If the clinician classified the child as an ASD Case based on the coding of exemplars, but was not certain they met criteria for ASD, a second independent clinician review was conducted, and a final consensus case classification was made by the two clinicians.

An initial criterion-level DSM-IV-TR to DSM-5 mapping was completed and reported in a study estimating impact of the change in criteria on ASD prevalence (Maenner et al., 2014). More detailed specification of criterion-based exemplars and assessment of reliability in using these exemplars to endorse DSM-5 ASD criteria was conducted in two phases.

The phase 1 SC SUCCESS study represented a population of 8780 8-year-old children from which 240 children had records abstracted for possible ASD based on a documented classification or suggestion of autism spectrum behaviors in source records. For the pilot reliability sample, composite records detailing developmental evaluations (n = 79) of 10 children were independently reviewed by two clinicians. Percent and weighted average agreement met or exceeded acceptable thresholds with 90% (ķ = 0.78) agreement on final ASD case status. For full phase 1 reliability sample, 2 clinicians completed blinded reviews for 155 developmental evaluations compiled for 20 children (Table 3) with 90% agreement (ķappa = 0.70) on the DSM-5 criteria (range 85%–95%; ķ = 0.57–0.85); 92% (ķ = 0.83) on early developmental concerns; 85% (ķ = 0.69) on AFs; 90% (ķ = 0.69) on the presence of an earlier PDD/ASD diagnosis; and 90% (ķ = 0.62) on overall ASD Case Classification (n = 17 ASD Case; n = 3 not ASD Case) (Table 3). Manual refinements were undertaken to increase clarity, particularly for any item with less-than-optimal reliability (B2, AF Cognitive Scatter) prior to the next phase including all ADDM sites.

In phase 2, 11 ADDM sites for the 2014 surveillance year with a base population of 263,775 8-year-old children had 9304 records abstracted for possible ASD. For the blinded 10% reliability review, 34 clinicians from 11 sites completed review of 4319 evaluations for 924 children. Agreement standards were at or well above prescribed levels with the exception of “good” agreement on criterion A1. The percent agreement was between 82% (B3) to 89% (A2 and B4) on DSM-5 criteria (range: ķ. = 0.58–0.78) and 92% (ķ = 0.85) on overall ASD Case Classification (n = 449 ASD Case; n = 475 not ASD Case) (Table 3). There was consistently high interrater agreement on ASD Case Status across sites (MN, 87% to AR, 97%) and by sex (male, 92% and female, 95%), race/ethnicity (Asian or Pacific Islander, 90% to Hispanic, 100%), cognitive level (Average or Above Average, 91% to No cognitive data, 95%), and previous classification of ASD (ASD Suspected, 85% to ASD classification on record, 97%) (Table 4). Further, interrater agreement was consistently high for endorsement of all seven ASD diagnostic criteria across sex, race/ethnicity, and cognitive level (Fig. 1).

Clinician review procedures were repeated for the ADDM Surveillance Year 2016 in 11 sites (n = 952 reliability pairs) with excellent overall agreement on Case/Non-Case Classifications (0.95; K = 0.89) [range 89%, GA to 99%, AR], and a range of overall percent agreement on the DSM-5 criteria from a low of 87% (A3 and B3) to a high of 92% (B2 and B4).

Meaningful clinical diagnosis of ASD involves an appreciation of the complex presentation and functional impact and clinical support of the caregivers and identified individual (Craddock & Mynors-Wallis, 2014). Better specification of the clinical decision process may increase the reliability and validity of the ASD diagnosis for clinical and research purposes. While there is evidence on the stability of early research-based autism spectrum diagnoses, there is conflicting evidence of community clinician agreement on these diagnoses (Daniels et al., 2011; Hausman-Kedem et al., 2018; van Daalen et al., 2009; Wiggins et al., 2020). At the same time, little is known about how clinicians apply the specific diagnostic criteria to arrive at those diagnoses. In addition, although practice guidance emphasizes the importance of using evidence-based tools in making a diagnosis, these tools rarely provide explicit instruction on how the overall score or the information obtained relates to the ultimate endorsement of DSM criteria, and diagnosis of behaviorally-defined conditions such as ASD cannot be reduced to a clinically-agnostic formula of applying criteria as has been attempted in some machine learning efforts (Bone et al., 2015). This paper reports an iterative process to operationalize the DSM-5 criteria for ASD with specific behavioral exemplars and guidance to classify children as an “ASD Case” for multi-site surveillance. These results indicated that a diverse group of clinicians can agree upon more nuanced definitions of each of the ASD DSM-5 criteria and reliably evaluate the presence/absence of these criteria based on the same set of information.

Overall, our findings indicate that a diverse group of clinicians from multiple sites across the US can reliably categorize specific behavioral exemplars documented in evaluation records to classify presence/absence of DSM-5 criteria, associated features, ASD case status, and other overall diagnostic specifiers. The process and product of operationalizing the DSM-5 ASD criteria resulted in a detailed guide which can facilitate a transparent and consistent process for evaluating both the specific behavioral descriptions and the DSM criteria, as well as the overall clinical decision on the presence of ASD. These findings can have implications for research and clinical practice.

We found high reliability across the initial study in SC and across ADDM for 2 surveillance years in application of exemplars to criteria from records-based descriptions and in the overall case classification. In addition, we found no significant variation based on site location, the child’s sex, or race/ethnicity. Robust agreement on the presence of the DSM criteria and final classification was also found across cognitive levels, despite the varied functional and clinical presentations across intellectual ability (Thurm et al., 2019). Given concerns about differential identification of ASD in community practice, it is encouraging that a basic level of consistency in evaluating behavioral descriptions was achieved across the demographic categories, thereby providing evidence in support of a process to more consistently determine if ASD characteristics are present. There is evidence that improved identification across different sociodemographic groups have contributed to the increases in ASD prevalence (Durkin et al., 2017), and that overall autism spectrum diagnoses reflect lower symptom intensity over time (Arvidsson et al., 2018). For the first time, the most recent ADDM report indicated similar ASD prevalence among children identified as non-Hispanic white, non-Hispanic black, and Asian/Pacific Islander (Maenner et al., 2020). ASD prevalence among Hispanic children has increased from earlier reports but was still below other groups. Continued periodic monitoring of ASD prevalence using replicable and consistent methods for clinician review in a subset of the population could help inform trends in prevalence and to inform targeted community identification efforts (Rice et al., 2012).

While the ADDM method to date is dependent on a child coming to the attention of a health or educational professional and receiving a developmental evaluation, that evaluation does not have to be for an “autism spectrum disorder” per se to be identified for clinician review. The presence of the evaluation is an indication that the child is experiencing some functional impairment; however, the characteristics of children who receive evaluations and the ways behaviors are documented in evaluations may change over time (Arvidsson et al., 2018). Not surprisingly, agreement on case status was highest when a child was already classified as having an ASD by a community clinician or for special education, but there was also high agreement when children were not previously classified with ASD as well, indicating the robustness of the methodology described here.

These ADDM clinician review guidelines were developed for identifying children with ASD for the purpose of monitoring population prevalence; however, the operationalization of the DSM-5 criteria also has additional utility for research and clinical practice. The standardized methods employed resulted in high inter-rater agreement, and if utilized in clinical settings, could also improve the consistency and the precision of clinical diagnoses. This consistency may be particularly useful for efforts to disentangle racial/ethnic- or sex-associated bias from differential service and diagnostic patterns (Imm et al., 2019). Consistency in applying the diagnostic criteria across these demographic factors was an encouraging finding in the present studies.

While it is assumed that clinicians interpret diagnostic criteria in consistent ways, the multiple rounds of clinician discussions of the concepts behind each criterion indicated that this is not necessarily the case. Most challenging was developing guidance to distinguish (A1) Deficits in social-emotional reciprocity, and (A3) Deficits in developing, maintaining, and understanding relationships. For example, the DSM-5 text indicates that (A1) includes “little or no initiation of social interaction” and (A3) includes “reduced social interest… preference for solitary activities.” When clinicians consider descriptions of behaviors written in diagnostic evaluations such as “child rarely played with other children” or “child stood quietly in the circle as the other students engaged in a game”, additional guidance was needed to increase consistency in deciding if these statements better represent a deficit in social emotional reciprocity (A1) or in relationships (A3). As such, the authors established guidelines to specify that (A1) “involves the exchange of social behaviors” such as initiating, responding, conversing, sharing, reciprocating and is represented by observable behaviors, while (A3) represents the “awareness, interest, or insight into other people” and is more about less observable concepts.

In addition, clarifying the concepts represented in the diagnostic criteria (e.g., A3 further detailed into “awareness”, “interest”, and “insight” of other people) provides a framework to further delineation of the phenotypic spectrum represented by these criteria. For example, when engaging in communication with other professionals, parents, or an individual diagnosed, it is useful to have a schema for mapping the person’s profile to specific criteria in a way that specifies current needs on different dimensions relevant to the spectrum (Di Rezze et al., 2016). Additional information on developmental history; age, place, and professional for evaluations; language level, rating of degree of impairment etc. was also classified but not reported in this paper. However, the framework used for characterizing exemplars within each criterion reflects an iterative process taking research from existing diagnostic classifications, tools, and clinical wisdom into account. This framework could be utilized to develop a more structured clinical and research tools and training to quantify and qualify “best estimate expert diagnosis” that considers both categorial and dimensional aspects of ASD. Future research could further evaluate the validity and utility of the behavioral exemplars and concepts representing the expression of the full spectrum within each domain and may serve as a platform for establishing research domain criteria (rdoc) within criteria and across the ASD phenotype.

This paper reports a large-scale effort to detail exemplars and define a process for characterizing for endorsing ASD DSM-5 criteria. Consistency in the way those criteria are defined and endorsed is an important research and clinical need. The data presented here indicate that a diverse group of clinicians across multiple sites can reliably apply an operationalized guide to endorsing the DSM-5 ASD criteria regardless of child sex, race/ethnicity, or cognitive level. The described methods and procedures can be used to inform the development and evaluation of tools and training for clinical, research, and surveillance efforts.