In this issue of Fertility and Sterility, Stern et al. (1) assessed the validity of the birth outcome data reported to the Society for Assisted Reproductive Technology Clinic Outcome Reporting System (SART CORS) compared with the outcome data from vital records and the Massachusetts Birth Defects Monitoring Program (BDMP), the statewide population-based active birth defects surveillance system for a cohort of 9,092 assisted reproductive technology (ART) deliveries from 2004 to 2008 in Massachusetts. Compared with vital records, the SART CORS does a great job of reporting for some outcomes, with >99% concordance on delivery outcome (live birth/fetal death) and plurality. More than 90% of birth dates match, and most singleton birth weights (87%) are different by <50 g. Maternal race/ethnicity is missing two-thirds of the time in SART CORS, but when not missing, it is reasonably accurate, with 95% concordance. So far, so good. When we look at birth defects, it is entirely another matter.

The SART CORS reported 135 birth defects among 132 infants compared with 184 birth defects among 132 infants in the BDMP data. Although it seems at first glance that we might still be on fairly solid ground with the same number of infants identified; the problem was that only 51 infants were identified with birth defects in both datasets. Overall, SART CORS is missing 81 infants (61%) that have birth defects confirmed by BDMP, and a corresponding 81 cases in the SART CORS are unconfirmed by BDMP. Compared with the active surveillance of the BDMP as the criterion standard, ART clinics appear to be missing the mark. The problem starts with the categories of birth defects reported in SART CORS; rather than capturing diagnostic codes or detailed information, the system allows for the following predefined categories, with no instruction provided as to which specific diagnoses are to be considered as relevant to a given category: none, unknown, cleft palate, genetic defect, limb defect, cardiac defect, and other. Given these limited and undefined categories, and that the SART CORS data come from a variety of reporting sources, including medical records, provider reports and parent self-reports, it is not surprising that there are inaccuracies. We were a bit surprised to see that the errors went in both directions, with similar numbers of false positives and false negatives.

Major birth defects are common in aggregate, generally affecting ~3% of births (2), but specific birth defects are relatively rare. The most common defects are congenital heart defects (CHDs), with an overall prevalence of nearly 1% (3, 4), or approximately one-third of all babies with birth defects. But not all CHDs are diagnosed during the birth hospitalization. We might expect the SART CORS to underreport CHDs, and they do, with 11 cases (28%) compared with 40 in the BDMP. SART CORS also reports 14 unconfirmed CHDs, perhaps due to “rule out” conditions that were reported to the clinics but ultimately determined to be noncases. The Stern et al. report shows that reporting is not better for other defects. Sensitivity for specific defect groups ranges from 18% to 50%, and attempts to try and reconcile misclassified cases by means of searching vital records did not help resolve discrepancies.

Whether ART increases birth defect risk is an important question. In population research, two things drive the ability to see a significant effect: sample size and effect size. In this case, if the effects were large, they should be apparent by now, given the increased use of ART around the globe. So, let us suppose that the effects are small (if present at all) or effects are there for only a subgroup of patients. Then these classification errors are likely to lead to null results in a scientific investigation.

It seems to be time to question the utility of having ART clinics report birth defects. However well intentioned, the collection of these data without consistent methods, with limited specificity, and based on a variety of reporting sources is not working. It may not be worth the effort to collect and report poor-quality data, as opposed to investing in linkages with high-quality birth defects surveillance data. An old professor of one of us (P.M.) used to say that “some data beats no data” in public health and prevention, but when the data are of poor quality, it is time to consider moving to better systems.

The Centers for Disease Control and Prevention have supported the States Monitoring Assisted Reproductive Technology (SMART) Collaborative, which currently includes Massachusetts, Florida, Connecticut, and Michigan. This project supports linkage of ART data with other data systems, including vital records and birth defects registries (5). In addition to including other infant outcomes, this model is a stronger approach for surveillance, certainly for rare outcomes, such as birth defects, but also for other infant outcomes that are not captured in the current SART system.