Autism spectrum disorder (ASD) is a developmental disorder that affects social, communication, and behavioral development and is sometimes associated with co-occurring features such as intellectual disability (ID), attention and activity deficits, and abnormal sensory response [American Psychiatric Association (APA) 2006, 2013; Levy et al. 2010], among others. The number of children identified with ASD has increased substantially over the past decade, rising from about 6.4 per 1,000 children in 2002 to about 11.3 per 1,000 children in 2008 [Centers for Disease Control and Prevention (CDC) 2007, 2012]. This increase in the number of children identified with ASD has fueled interest in understanding the complex genetic and environmental factors that may affect ASD phenotypes. Yet defining ASD phenotypes has been challenging given the heterogeneity in symptom presentation and severity and the reliance on behavioral criteria to define the disorder. As such, developing standardized criteria that can both classify and characterize children with ASD is important for clinical practice and epidemiologic research.

It is now well established that best estimate clinical judgment by experienced and reliable clinicians informed by standardized diagnostic instruments is the best predictor of stable ASD diagnoses (Lord et al. 2006). For instance, Lord et al. (2006) found that clinical judgment at 2 years of age predicted ASD diagnoses at 9 years of age more than results of standardized diagnostic instruments—the Autism Diagnostic Interview Revised (ADI-R) and Autism Diagnostic Observation Schedule (ADOS)—when these instruments were used alone (Gotham et al. 2007; Lord et al. 1994, 1999, 2000, 2012; Rutter et al. 2003b). However, scores from the ADI-R and ADOS are both sensitive and specific in detecting children with ASD when used in combination (Risi et al. 2006), and offer several advantages to classify children with ASD in clinical practice and research studies. First, ADI-R and ADOS scores are assigned by experienced and reliable clinicians and offer a uniform method of characterizing ASD symptoms in large cohorts of children that can be replicated in other studies. Second, symptom profiles gleaned from the ADI-R and ADOS allow the opportunity to create ASD sub-groups based on observed and/or reported symptoms that could represent a range of behavioral trajectories and phenotypes. Consequently, using the ADI-R and ADOS to classify children with ASD may be advantageous when well-defined groups of children are an important clinical or research outcome (Schendel et al. 2012).

One of the primary challenges of using ADI-R and ADOS scores to classify children with ASD is that results of the ADI-R and ADOS are not always consistent with each other and there is not a standard method to resolve discordance when one instrument suggests the presence of an ASD and the other instrument does not suggest presence of an ASD (de Bildt et al. 2004; Le Couter et al. 2008). There have been numerous proposals to resolve ADI-R and ADOS discordance or relax ADI-R criteria to include the broader spectrum of ASD in addition to the formerly defined Autistic Disorder (IMGSAC 2001; Risi et al. 2006; Sung et al. 2005). Some of these proposals overlap with one another (Risi et al. 2006; Sung et al. 2005) and define relaxed ADI-R criteria for ASD as meeting the cutoff score on the social deficits domain or communication deficits domain and being within two points of the cutoff score on the social or communication domain not met. These relaxed criteria have been shown to be both sensitive and efficient in detecting children with ASD in a variety of populations (de Bildt et al. 2013; Risi et al. 2006; Sung et al. 2005). However, these criteria have not been evaluated to determine whether they capture children with restricted interests and repetitive behaviors (RRB) noted on the ADOS or ADI-R, which is now a required diagnostic criteria in the Diagnostic and Statistical Manual of Mental Disorders [American Psychiatric Association (APA) 2013].

The Study to Explore Early Development (SEED) is a multi-site case–control study of children 30–68 months of age designed to explore the relationship between ASD phenotypes and various genetic and environmental risk factors (Schendel et al. 2012). A SEED workgroup consisting of ASD diagnostic experts convened to develop a final classification algorithm based on results of the ADI-R and ADOS and to address the complexities of classifying children with ASD based on ADI-R and ADOS scores. The objectives of this paper are to (1) describe the final classification algorithm based on results of the ADI-R and ADOS adopted in SEED, (2) examine the psychometric properties of different ASD classification methods, including the SEED method that incorporates rules for resolving ADI-R and ADOS discordance, and (3) determine whether RRB were noted for children defined as ASD in SEED who had instrument discordance resolved using ADI-R social and communication scores. We hypothesized that SEED ASD criteria would have a good balance of sensitivity and specificity using informed clinical judgment as the referent standard and classify more well-defined children than other ASD classification schemes.

A total of 3,769 index children were enrolled in SEED, 2,722 (72.2 %) completed a clinic visit, and 2,600 (68.9 %) completed enough of the clinic visit to be classified as ASD or non-ASD. The sample described in this paper is those 2,600 children who were classified as ASD or non-ASD. The SEED final classifications for these children were ASD (n = 707), DD (n = 995), and POP (n = 898). Children defined as DD had a variety of diagnosed conditions reported by parents on the caregiver interview: speech delay (59.3 %), developmental delay (36.2 %), non-specific developmental or learning problem (38.0 %), movement or coordination problem (14.6 %), sensory integration problem (10.5 %), hearing problem (9.0 %), behavior problem (8.5 %), birth defect (7.5 %), attention deficit hyperactivity disorder (7.4 %), Down syndrome (3.9 %), vision impairment (2.8 %), sleep problem (2.6 %), seizure disorder or epilepsy (2.9 %), cerebral palsy (2.4 %), self-injurious behavior (2.0 %), reading disorder (1.6 %), obsessive compulsive disorder (1.1 %), bipolar disorder (0.4 %), Fragile × syndrome (0.3 %), neurofibromatosis (0.3 %), reactive attachment disorder (0.2 %), Tourette syndrome (0.1 %), and tuberous sclerosis (0.1 %). These conditions were not mutually exclusive so more than one condition could be noted for each child.

The mean child age at time of clinic visit was 59.4 months (SD = 7.1 months). The racial distribution of the child sample was White (63.6 %), Black (15.9 %), multi-racial (11.7 %), Asian (4.1 %), not stated (4.3 %), and other (0.4 %). Hispanic ethnicity was reported in at least one parent in 15.2 % of the sample. Sixty-six percent of the sample was male; children defined as having ASD had a higher proportion of male children (82.0 % of 707) than children defined as DD (66.7 % of 995) or POP (53.1 % of 898), p < .001. Children defined as ASD had a higher proportion of ID as defined by the MSEL (62.4 % of 694) than children defined as DD (24.2 % of 988) or POP (2.8 % of 890), p < .001. The mean MSEL early learning composite for children in our sample were 66.9 points for children classified as ASD, 86.3 points for children classified as DD, and 102.3 points for children classified as POP; MSEL scores between 85 and 115 indicate average learning abilities.

Figure 1 shows final classification details for children defined as ASD, DD, and POP in SEED. There were 1,064 children who received a comprehensive evaluation for ASD and 707 (66.4 %) of these children were defined as ASD. Of the 707 children defined as ASD, 600 (84.9 %) had a previous ASD diagnosis, 97 (13.7 %) had a previous DD but not ASD diagnosis, and 10 (1.4 %) were identified from state vital records. Only 372 children were reported to have a known previous ASD diagnosis at the time of recruitment; the other 228 children with a previous ASD diagnosis had the diagnosis reported by a parent on the invitation call or during the caregiver interview.

There were 305 children defined as DD who received a comprehensive evaluation for ASD but did not meet SEED ASD criteria: 122 (40.0 %) of these children had a previous ASD diagnosis with ASD characteristics observed on the ADOS (n = 38), reported on the ADI-R (n = 26), or neither observed on the ADOS or reported on the ADI-R (n = 58) and 183 (60.0 %) of these children had a previous DD but not ASD diagnosis with ASD characteristics observed on the ADOS (n = 44), reported on the ADI-R (n = 19), or neither observed on the ADOS or reported on the ADI-R (n = 120). There were 52 children defined as POP who received a comprehensive evaluation for ASD but did not meet SEED ASD criteria: 10 had ASD characteristics observed on the ADOS, 3 had ASD characteristics reported on the ADI-R, and 39 did not have ASD characteristics observed on the ADOS or reported on the ADI-R.

A total of 1,017 children who completed a comprehensive evaluation for ASD also had a study clinician complete the OARS-adapted to gauge his or her clinical judgment of whether the child met criteria for ASD. Psychometric properties of the ADOS alone, ADI-R alone, concordant ADI-R and ADOS and SEED ASD criteria were conducted on the sample of children who completed a comprehensive evaluation for ASD, had a study clinician complete the OARS-adapted, and had a mental age of at least 24 months (n = 922). For these children, the ADOS alone yielded the highest sensitivity (.92) but the lowest specificity (.61) compared to other classification schemes (Table 2). The sensitivity and specificity of other classification schemes were relatively comparable, with SEED ASD criteria showing higher sensitivity than the ADI-R alone or concordant positive ADOS and ADI-R results (0.86, 0.77, and 0.75, respectively) and concordant positive ADOS and ADI-R results showing higher specificity than ADI-R alone or SEED ASD criteria (0.82, 0.73, and 0.74, respectively; Table 2). Concordant positive ADOS and ADI-R missed 62 more children defined as ASD by the clinician than SEED ASD criteria. Likewise, SEED ASD criteria classified 27 more children defined as non-ASD by the clinician than concordant positive ADOS and ADI-R.

Figure 2 displays how many of the 1,017 children who completed a comprehensive evaluation for ASD and had clinical judgment noted on the study form had concordant positive ADI-R and ADOS results, concordant negative ADI-R and ADOS results, and discordant ADI-R and ADOS results. Figure 2 also demonstrates the SEED criteria used to resolve ADI-R and ADOS discordance and the mean ADI-R and ADOS RRB scores for children who met each of the discordance criteria. It is important to note that only one of the 96 children who met SEED ADI-R and ADOS discordance criteria did not have any RRB noted on the ADI-R or ADOS. The endorsement rate (i.e., a score of other than 0 or none reported) of RRB noted on the ADI-R for children who had instrument discordance resolved was 58.9 % for repetitive use of objects, 57.9 % for sensitivity to noise, 49.4 % for unusual sensory interests, 40.0 % for circumscribed interests, 32.6 % resistance to changes in own routine or environment, 32.6 % for complex mannerisms, 30.9 % for hand and finger mannerisms, 22.1 % for unusual reactions to specific sensory stimuli, 21.1 % for unusual preoccupations, 20.9 % for unusual attachment to objects, 17.9 % for compulsions or rituals, 5.3 % for midline hand movements, and 3.2 % for resistance to changes in general environment. Children who had an ADI-R and ADOS discordance resolved were more likely to be defined as having ASD by the study clinician (71.9 %) than children who did not have an ADI-R and ADOS discordance resolved (43.4 %), p < .001.

Diagnostic instruments alone cannot replace informed clinical judgment when diagnosing children with ASD. However, use of diagnostic instruments to classify children with ASD may be advantageous when well-defined groups of children are an important clinical or research outcome. We found that SEED ASD classification based on results of both the ADI-R and ADOS had a good balance of sensitivity and specificity, which supports the utility of these standardized instruments to classify well-defined groups of children in clinical practice and research studies. Use of the SEED-specific ASD classification algorithm offers two major advantages over use of other instrument-based ASD classification schemes because it provides: (1) a method for resolving discordance between the ADI-R and ADOS that classifies more children defined as ASD by an experienced and reliable clinician than concordant positive ADOS and ADI-R results and (2) detailed classifications and sub-classifications that can be replicated in other studies. The SEED final classification algorithm may therefore help guide other clinical and research decisions when ASD risk is indicated on the ADOS but not the ADI-R or detailed sub-classifications are needed to explore ASD behavioral presentations and/or risks.

Our results support previous claims that the ADI-R and ADOS should be used together rather than alone since they offer additive contributions to ASD classification (Risi et al. 2006). For instance, we found high sensitivity but low specificity of the ADOS; sensitivity decreased but specificity improved when the ADOS and ADI-R were used together (concordant positive ADOS and ADI-R). Moreover, sensitivity continued to improve when SEED ASD criteria were applied. Specifically, SEED ASD criteria classified 62 more children with a mental age of at least 24 months than concordant positive ADOS and ADI-R, although there was a slight compromise in specificity (there was no compromise in PPV and an increase in NPV). SEED criteria also classified 87 children with ASD also classified as ASD by the study clinician but had a mental age less than 24 months (data on these children were not presented because clinical judgment was used to classify these children and also used as the referent standard for psychometric analyses). These findings are significant for shaping the future of population-based studies that seek an adequate balance of sensitivity and specificity and rely on sample size to yield the statistical power needed to conduct etiologic and other analyses.

As mentioned previously, the ADOS alone yielded the lowest specificity and highest false positive rate compared to any other classification scheme, which could be an artifact of the young mean age and cognitive abilities of our sample. Previous studies have found the probability of ADOS agreement with diagnostic criteria is inversely related to level of functioning for the child (de Bildt et al. 2004) and the mean cognitive standard score for children classified as ASD in SEED was 66.9 points. Yet the mean cognitive standard score for children in our sample is similar to other studies with similar samples of young children with ASD (Risi et al. 2006). Specificity of the ADOS and ADI-R improved in our sample when the instruments were used in conjunction rather than when they were used alone. Thus, our findings continue to support use of both the ADI-R and ADOS for evaluation of current developmental status and improvement of ASD classification.

Another advantage of the SEED final classification algorithm is the introduction of detailed classifications and sub-classifications that allow exploration of different pathways of development and the range of behavioral phenotypes. These classifications and sub-classifications are not available when children with ASD are classified using clinician judgment, diagnostic criteria, or one diagnostic instrument alone. We found that 16.9 % of children with a previous ASD diagnosis did not meet SEED criteria for an ASD, which could suggest improvement in symptoms or challenges with ASD assessment and diagnosis at young ages (Lord et al. 2006; Kleinman et al. 2008; Sutera et al. 2007; Turner et al. 2006; Wiggins et al. 2012). Conversely, we found many children met SEED criteria for an ASD or had ASD symptoms noted on the SCQ, ADOS, and/or ADI-R despite the fact they had not been recognized as having an ASD in community settings. Specifically, of the children defined as ASD in SEED, 84.9 % were identified with a previous ASD diagnosis (52.6 % were identified with a previous ASD diagnosis at the time of recruitment), 13.7 % were identified from community providers as a child with DD (but not ASD) and 1.4 % were identified from state vital records. Moreover, many children defined as DD and POP had ASD symptoms noted on the SCQ, ADOS, and/or ADI-R. These results highlight the heterogeneous nature of ASD and could indicate problems with differential diagnosis, continuous distribution of ASD symptoms in child samples, or selection bias among families who agreed to participate in SEED. Further research is needed to describe these different pathways of development, and the detailed nature of the SEED final classification algorithm allows further exploration of characteristics that distinguish children in one sub-classification from children in other sub-classifications. These findings also indicate caution is needed when classifying children for research based solely on existing diagnoses at the time of recruitment or lack of diagnoses at these young ages.

One challenge to using ADI-R and ADOS results to classify children with ASD is that results of these instruments do not always agree with one another. We found discordant ADI-R and ADOS results in 17.6 % of children administered a comprehensive developmental evaluation in SEED, which supports previous findings (de Bildt te al. 2004; Le Couter et al. 2008). Discordant ADI-R and ADOS results thus present a challenge to both clinicians and researchers who rely on these instruments for ASD classification. The discordance criteria adopted in SEED required the child to meet either the cutoff score in the social deficits domain or communication domain but not the RRB domain, even though RRB are now a necessary component of the ASD diagnosis (APA 2013). However, our analyses found that children who had ADI-R and ADOS discordance resolved had some RRB noted on both instruments; only one child who had ADI-R and ADOS discordance resolved did not have any RRB noted on either instrument. More than half of these children had repetitive use of objects and sensitivity to noise noted on the ADI-R. Additionally, these children were more likely to be defined as ASD by a study clinician than children who did not have an ADI-R and ADOS discordance resolved. These results could indicate that young children who meet ADOS ASD criteria and relaxed ADI-R criteria show social-communication and behavioral deficits reflective of ASD, even if clinically significant RRB have not fully manifested in the child or are not recognized and reported by a parent. These children should therefore still be considered for an ASD classification given the importance of early recognition of and intervention for all children with ASD.

A few limitations to our analyses warrant mention. First, clinical judgment determined by the OARS-adapted and results of the ADI-R and ADOS were not independent of one another, although clinical judgment took into account all information available to the clinician which is comparable to best estimate clinical diagnosis. The clinician who completed the OARS-adapted was often, but not always, the clinician who administered the ADOS and/or had access to ADI-R results. This clinician also had access to SCQ screen results, medical and psychiatric histories, and results of self-administered questionnaires that assessed the behavioral and social functioning of the child. Thus, clinical judgment was based on all available information instead of one or a few diagnostic instruments. Second, clinical judgment was used to define ASD in children with a mental age less than 24 months since low mental age deems the ADI-R invalid. Thus, the psychometric properties of SEED ASD criteria were not presented for these children. Future research is needed to examine the utility of ASD classification schemes for children with very low cognitive scores since ASD and ID often co-occur and may represent an important phenotype to guide etiologic analyses.

Despite these limitations, our findings support the use of ADI-R and ADOS scores in classifying children with ASD in clinical practice and research studies when creating well-defined groups of children is important to the diagnostic or treatment process or research design. SEED ASD criteria may detect more well-defined groups of children than other ASD classification schemes. Future research should explore whether SEED classifications yield children with different behavioral, developmental, and medical profiles so they can be used as important outcomes to explore the relationship between ASD phenotypes and etiologies.