The identified prevalence of autism spectrum disorders (ASD) has increased significantly over the last decade according to the Centers for Disease Control and Prevention’s Autism and Developmental Disabilities Monitoring (ADDM) Network (Centers for Disease Control and Prevention 2007a, b, 2009, 2012, 2014). The ADDM Network’s most recent estimate identified 1.47 % of 8-yearold children (one in 68) with an ASD in 2010 (Centers for Disease Control and Prevention 2014). A strength of the ADDM Network surveillance approach is that prevalence estimates are derived from a common methodology which is applied across study years by participating sites (Van Naarden Braun et al. 2007), thereby minimizing comparison errors based on variations in methodology. The approach uses a multisource, retrospective records review of existing health records (e.g. diagnostic and developmental assessments from psychologists, child psychiatrists, neurologists, developmental pediatricians, physical therapists, occupational therapists, speech/language pathologists) and, when available, education records. The Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, text revision (DSM-IV-TR) (American Psychiatric Association 2000) is used to define ASD case status. Children with ASD are identified according to DSM-IV-TR criteria through an independent clinician review of the developmental information in the records. An evaluation of the approach suggests that it is a cost-effective alternative to conducting a complete population screening and diagnostic clinical assessment of ASD (Van Naarden Braun et al. 2007). In a validation study, Avchen et al. (2011) reported fairly high specificity [0.96 (95 % CI 0.94–0.99)], positive predictive value [0.79 (95 % CI 0.66–0.93)], and negative predictive value [0.91 (95 % CI 0.87–0.96)] of the ADDM surveillance methodology compared to direct clinical assessment. The low sensitivity measured in the study [0.60 (95 % CI 0.45–0.75)] suggests that prevalence estimates derived from the ADDM-based ASD surveillance approach are conservative.

Despite the effort to maintain consistent methodology over time within and across ADDM sites, significant variation has been measured in ASD prevalence estimates between sites. In 2008, the Utah ADDM site (UT-ADDM) reported the highest estimated ASD prevalence (21.2 per 1,000 or 1 in 47) among all 14 participating sites (Centers for Disease Control and Prevention 2012), significantly exceeding prevalence estimates at all but two other sites (Arizona and New Jersey).

The complex nature of ASD, the absence of a biological marker, changes in diagnostic practices over time, and variation in the availability of diagnostic services complicates the pursuit of finding accurate ASD prevalence rates (Liptak et al. 2008). However, it is paramount that community diagnosticians, stakeholders and the general public have trust in the accuracy of the prevalence estimated by public health entities. In light of these challenges and UT-ADDM’s comparatively-high estimated ASD prevalence, a small validation study was conducted to compare ASD final case status as determined by UT-ADDM with ASD case status as determined by community provider, independent external expert reviewers (EERs). This study’s objectives were: (1) to measure agreement between UT-ADDM and EERs on final case status, ASD subtype classification (autism vs. ASD-NOS), degree of case certainty, and degree of case impairment; and (2) to examine characteristics of the records on which there was disagreement in ASD final case status between UT-ADDM and EERs.

Table 1 reports the inter-rater reliability calculations between the external expert reviewers and UT-ADDM. Inter-rater reliability was high between UT-ADDM and EER for two-level final case status (case vs. non-case): F (89, 270) = 21.6, ICC = 0.84 (95 % CI 0.79–0.88), p < 0.001. Whereas, poor agreement was measured for three-level final case status (Confirmed case vs. Suspected case vs. DNQ): F (89, 270) = 6.19, ICC = 0.57 (95 % CI 0.47–0.66), p < 0.001. Poor agreement was measured for ASD subtype classification (Autistic Disorder vs. ASDNOS): F (20, 63) = 2.64, ICC = 0.29 (95 % CI 0.08–0.55), p = 0.002, and agreement was null for noncase classification (Suspected Case vs. DNQ) between UT-ADDM and EER: F (57, 174) = 0.89, ICC = −0.03 (95 % CI −0.11–0.09), p = 0.69. Overall, measures of clinical sensitivity [0.70 (95 % CI 0.51–0.85)], specificity [0.97 (95 % CI 0.88–1.00)], positive predicted value [0.91 (95 % CI 0.72–0.99)], and negative predicted value [0.87 (95 % CI 0.76–94)] were fairly high.

At least one of the EERs disagreed with UT-ADDM on ASD final case status in nine out of 30 records that UT-ADDM classified as a Confirmed case. This represents a fair bit of disagreement across individual EERs. However, for only one Confirmed case did all three of the EERs disagree with UT-ADDM. For this child, all EERs classified the child’s status as “Suspected ASD” with multiple other disorders (i.e., intellectual disability, mood disorder, language delays) creating a complicated diagnostic picture. Table 2 presents composite record-level characteristics of the 21 records with a UT-ADDM final case status of Confirmed case for which there was perfect agreement between UT-ADDM and all three EERs, and the nine records with a UT-ADDM final case status of Confirmed case for which there was disagreement between UT-ADDM and at least one EER. Twenty-five out of the 30 children that were Confirmed ADDM ASD cases were male. Nineteen out of 21 (90 %) of the Confirmed cases for which there was agreement between UT-ADDM were male and six out of nine (67 %) of the Confirmed cases for which there was disagreement were male. Of the 21 Confirmed cases for which there was agreement between UT-ADDM and EERs, 62 % had a previous diagnosis of ASD on record and 43 % had an autism special education eligibility on record. Only 22 % of the nine Confirmed cases for which there was disagreement between UT-ADDM and EER had a previous ASD diagnosis on record and only 11 % had an autism special education eligibility. However, no statistically significant differences in characteristics were found between Confirmed cases for which there was agreement versus disagreement between EER and UT-ADDM (Table 2). In addition, there were no differences in mean IQ score between the Confirmed cases on which there was UT-ADDM and EER agreement (mean IQ = 75.4, SD = 17.4) versus the Confirmed cases on which there was UT-ADDM and EER disagreement (mean IQ = 64.5, SD = 22.14; p = 0.23). “Other disorder(s)” was indicated by at least one of the EERs on eight out of nine discordant records as a reason for not confirming the child as an ASD case. Alternative health conditions suggested by the EERs to explain the child’s observed behavior included intellectual/cognitive disability (5/9), anxiety (3/9), or language impairment (2/9). Insufficient information was also indicated as a reason for not confirming the child as an ASD case on eight out of nine discordant records. The average record quality was 4.16 for the 21 UT-ADDM Confirmed cases for which there was agreement between UT-ADDM and EER while the average record quality was 2.89 for the nine UT-ADDM Confirmed cases on which UT-ADDM and EER disagreed. Finally, EERs and UT-ADDM agreed on final case status for 58/60 non-cases (includes Suspected and DNQ). The two children confirmed as ASD cases by EER but not UT-ADDM were both classified as an ASD case by only one out of three EERs.

Two-level final case status (case vs. non-case) inter-rater agreement and corresponding clinical specificity, sensitivity, positive predictive value, and negative predictive value were fairly high between UT-ADDM and the three EERs providing convergent validation of the ADDM method for identifying children with ASD. In our study, among UT-ADDM Confirmed cases, at least one EER agreed with UT-ADDM clinician review on ASD final case status in 29 out of 30 records. Disagreement among EERs on final case status determination in nine out of 30 ADDM identified children with ASD indicates a somewhat high level of variation across individual EERs. Reasons cited by the EERs for not confirming the nine discordant cases as ASD included insufficient information and the presence of other disorders. Discordant records were also of low average quality compared to the 21 Confirmed cases on which there was agreement between UT-ADDM and the three EERs.

While two-level final case status inter-rater reliability between UT-ADDM and EER was excellent, characteristics of Utah’s ASD cases could make it difficult for EERs to confirm ASD case status through a retrospective record review process alone. Several key characteristics of UT-ADDM ASD cases differ from ASD cases identified at other SY2008 ADDM sites (Centers for Disease Control and Prevention 2012) and from cases described in previously published ASD studies (e.g. Fombonne 2009). Utah, along with the New Jersey ADDM site, identified a higher proportion of children with intellectual quotients (IQ) above 70 compared with any of the other sites participating in the ADDM network in SY2008. Eighty-seven percent of the UT-ADDM cases had an IQ >70 compared with 62 % of cases across all ADDM sites combined. Although ASD prevalence estimates were significantly higher for boys than girls among all ADDM sites combined (p < 0.01), the male-to-female prevalence ratio was lower in Utah (2.7) compared with all ADDM sites combined (4.6). Despite some differences in the characteristics of Utah’s ASD cases compared with other ADDM sites, no statistically significant differences in sex and average IQ were detected between Confirmed cases for which there was agreement and Confirmed cases for which there was disagreement between UT-ADDM and EER. This, as well as the high two-level final case status agreement measured between UT-ADDM and EER, suggests that the ADDM method is robust to variation in underlying case characteristics for ascertaining ASD.

We were not surprised to measure poor inter-rater reliability for ASD subtype (Autistic Disorder vs. ASD-NOS) between UT-ADDM and EER. The ADDM record review methodology is not intended to classify clinically-meaningful ASD subtypes, although the diagnostic subtypes indicated by community providers on comprehensive evaluations were reported in publications of prevalence findings (Centers for Disease Control and Prevention 2009, 2012, 2014). Even among research and clinical practices involving highly trained ASD diagnosticians distinguishing among ASD subtypes (Autistic Disorder, PDD-NOS, and Asperger’s Disorder) is known to be inconsistent (Lord et al. 2012; Risi et al. 2006). Changes to ASD classification in the newest iteration of the American Psychiatric Association’s ASD diagnostic criteria reflect this finding (American Psychiatric Association 2013).

An evaluation of differences in characteristics between records from UT-ADDM Confirmed cases for which there was agreement versus disagreement between UT-ADDM and EER found that a larger proportion of records on which there was agreement had previous indications of an ASD diagnosis and/or an autism special education classification, although these differences did not reach statistical significant. Discordant records were of lower average quality than records on which there was agreement. In addition, EERs cited insufficient information and the presence of another disorder as reasons for not classifying a child as a case. Overall, each individual EER identified fewer ASD cases than UT-ADDM. The reasons, cited above, that EERs provided for not confirming a child as an ASD case suggest that expert clinician review of retrospective records was conservative relative to UT-ADDM.

The measured prevalence of ASD varied widely across ADDM sites in 2008 (Centers for Disease Control and Prevention 2012). Although this study focuses on a single site that participated in the ADDM network in 2008, our findings concerning disagreement between EER and ADDM clinician reviewers may shed light on potential reasons for this phenomenon. Our study found that insufficient information and record quality influenced the EER’s determination of final case status. The amount and quality of the information contained in a child’s composite record is known to vary across ADDM sites (Van Naarden Braun et al. 2007) and has been suggested as an important factor driving variation and change in the measured prevalence of ASD in the ADDM network (Rice et al. 2010). In addition, disagreement in final case status determination among individual EERs and between EER and UT-ADDM may reflect the variability that exists, even among clinical experts practicing within the same community, in ASD diagnostic threshold based on a records review process. As such, this study’s findings highlight the importance of having a clear and consistent records review process shared across ADDM sites, the fidelity of which is maintained by regular reliability exercises.

A number of researchers have suggested that increases in ASD prevalence estimates are a result, in part, of diagnostic substitution and accretion (i.e., increasingly diagnosing autism in addition to other conditions previously diagnosed) (Bishop et al. 2008; King and Bearman 2009; Shattuck 2006a, b). Although not the focus of this study, several of the discordant cases involved children for which co-morbid intellectual disability, language impairment, sensory and/or psychiatric disorders were noted by the EERs as alternative health conditions. This indicates that clinical complexity may have impacted the EER’s case determination.

This study has several strengths including the evaluation of data collected in Utah as part of a population-based assessment of ASD prevalence. The strong agreement measured between UT-ADDM and EER on ASD final case status provides validation of the ADDM method to identify ASD cases which serve as the basis for estimating ASD prevalence in Utah. As ongoing clinician review quality control procedures are conducted among ADDM sites to enable comparisons of final case status across the ADDM Network, our Utah validation findings could have network-wide implications toward providing confidence in clinician review determination of ASD final case status at other ADDM sites. At the same time, some weaknesses are noted, including this study’s relatively small sample size. Also, validation was conducted using retrospective review of pre-existing records instead of via an independent clinical exam as was conducted in a previous ADDM Network validation study (Avchen et al. 2011).

Obtaining accurate estimates of ASD prevalence is imperative for determining the overall public health impact of ASD and planning for future diagnostic and treatment services to assist individuals with this increasingly identified condition. The approach used in this study was a means to validate UT-ADDM ASD final case status as identified using ADDM methodology. Overall, our findings suggest that children classified as ASD cases in the ADDM Network are also likely to be classified as ASD cases by independent experts, and provides support for UT-ADDM ASD prevalence estimates derived using the ADDM methodology.