In the United States (US), the National Center for Health Statistics (NCHS) calculates the maternal mortality ratio (MMR) using a specific set of International Classification of Disease (ICD) codes.^1,2 These codes (A34, O00-O95, O98-O99) are based on the World Health Organization definition of the death of a woman while pregnant or within 42 days of pregnancy that cannot be attributed to incidental or accidental causes.^3,4 All death certificates indicating pregnancy at time of death, pregnancy within 42 days of death, or obstetric-related cause of death are ascribed one of these codes, except for deaths due to external causes.⁵ Historically, changes in maternal death identification have led to increased success in capturing these deaths. Implementation of the 10th revision of the ICD accounted for a 13% increase in the number of maternal deaths counted from 1998 to 1999.⁵

Before 2003, 16 states included questions on their death certificates ascertaining whether a woman had been pregnant at the time of her death or within the preceding year.^1,2 These “pregnancy checkbox questions” were developed to improve identification of maternal deaths. A study among states using pregnancy checkboxes on their death certificates in 1991 and 1992 highlights the benefits of the checkbox in capturing maternal deaths.² Investigators found an additional 5%-40% of maternal deaths were identified by checkboxes, compared with only using pregnancy-related terms documented on death certificates.² Pregnancy checkboxes were added to the 2003 revision of the standard United States death certificate (Figure 1), which all states adopted in some form between 2003 and 2017.^6,7 The pregnancy checkboxes were expected to improve identification of maternal deaths and accuracy of the MMR.²

Although the addition of the pregnancy checkbox to the standard death certificate resulted in increased identification of maternal deaths, many researchers have raised questions about its accuracy.^6,8-10 The staggered adoption of the revised death certificate by states resulted in a lack of nationally comparable data, and NCHS has not released an official United States MMR since 2007.⁷ Instead, calculations of the MMR by researchers and others are based on the publicly available National Vital Statistics Systems data.¹¹

A 2017 report from four state Maternal Mortality Review Committees found that pregnancy checkboxes identified an additional 50% of deaths during pregnancy, 11% of deaths within 42 days of delivery, and 8% of deaths from 43 to 365 days postdelivery, compared with only using pregnancy-related terms found on the death certificate.¹² However, approximately 15% of checkbox-identified deaths had no verifiable evidence of pregnancy.¹² A Texas study found an 87% increase in maternal deaths comparing 2006-2010 to 2011-2015 data.¹³ A subsequent Texas study found that 50% of obstetric-coded deaths in 2012 showed no evidence of pregnancy within 42 days of death after data-matching—suggesting that the reported increase in maternal deaths was inflated by pregnancy checkbox errors.¹⁴ These studies question the accuracy of United States maternal death data, highlighting the necessity of data quality improvement measures.^{6,8-10,13,15,16,18}

Given concerns over the quality of the data collected by the pregnancy checkbox, four state health departments (Georgia, Louisiana, Michigan, and Ohio) implemented a quality assurance pilot in collaboration with Vital Records offices, Maternal and Child Health (MCH) programs, and technical assistance from staff at the Centers for Disease Control and Prevention (CDC). From January 2016 to March 2017, states reviewed all 2016 death certificates with a pregnancy checkbox marked—with the goal to amend vital statistics files sent to NCHS. States conducted monthly to bimonthly searches for birth or fetal death reports that matched death certificates to women of reproductive age, within a year of death. If no match was found, but a pregnancy checkbox was marked, an attempt was made to confirm the pregnancy by contacting the certifier or reviewing the decedent’s medical records. Participants from each state developed their own specific quality assurance processes based on existing practices, policies, and structures. CDC provided technical assistance at the states’ requests, including organizing quarterly calls to share updates and best practices, assisting in designing a template to track death certificates, and communicating findings to other stakeholders.

Throughout the quality assurance pilot, participants evaluated a total of 467 death certificates with a pregnancy checkbox marked. Of these deaths, 55% (n = 255) linked to a birth or fetal death certificate. After following up with the death certifiers of the remaining 212 records, 38% (n = 80) were confirmed to be pregnant, 17% (n = 35) were unable to be confirmed, and 46% (n = 97) were confirmed to not be pregnant (manuscript under review).

The purpose of this study is to describe opportunities and challenges to pilot implementation, project sustainability, and lessons learned—as identified by stakeholders who participated in the pilot.

From December 2017 to January 2018, the CDC project led identified key stakeholders to participate in an evaluation of the quality assurance pilot through focus groups. These group discussions provided a robust opportunity for analyzing the experiences, perspectives, and learnings of pilot participants. Participants had already established rapport with one another through the pilot’s quarterly calls, which facilitated rich and in-depth focus group discussions.

The researcher, who was not involved in the pilot, conducted three focus groups by means of teleconference. Each focus group was stratified by pilot role (Active Follow-up, MCH program, or Vital Records leadership) with typically one representative per state. Ten staff members across four states participated. Active Follow-up staff (n = 4) in the Vital Records Department performed linkages of vital records and case follow-up to verify pregnancy status, MCH program staff (n = 3) facilitated communication between offices, and Vital Records leadership staff (n = 3) designed and adapted the pilot process for their offices. These group compositions supported comparing experiences across shared roles and obtaining detailed information related to each unique state context. Discussion guides were developed with input from CDC staff to facilitate dialogue with respect to the pilot implementation, outcomes, and future quality assurance work. Institutional Review Board approval was obtained from Emory University, and all participants consented to focus group participation.

All focus group discussions were transcribed verbatim and analyzed using MAXQDA software (VERBI Software, Berlin, Germany). A codebook was developed with both deductive codes of expected themes and inductive codes derived from the data. Key themes were identified and compared across focus groups. All coded data were assessed using thematic analysis to discern consistent and unique themes across responses. Exemplar quotes were chosen to demonstrate source data.

Themes were identified across focus groups (Table 1) and explained in-depth with exemplar quotes for the following topics: pilot process, pilot outcomes, and future quality assurance.

The evaluation identified many successful outcomes of the quality assurance pilot, namely improved pregnancy checkbox data quality, improved timeliness for Maternal Mortality Review Committees, and increased prioritization of the quality assurance within state departments. Key challenges of pilot implementation included the high number of certifiers in each state, delay in certifier response time, staff turnover, competing priorities for both vital records and MCH departments, and lack of funding. States identified that several strategies assisted in overcoming these barriers: CDC engagement helped prioritize commitment to the pilot among state leadership, the multi-state effort assisted pilot development by sharing lessons learned and strategies, and communication with certifiers improved relationships and reduced certifier response time.

Participants provided salient recommendations to other states interested in initiating a pregnancy checkbox quality assurance process, including defining workflows, collaborating across departments, and committing time to contact certifiers for pregnancy confirmation. Uncertainty surrounding how NCHS codes the pregnancy checkbox suggests opportunities to increase knowledge of NCHS’s maternal death algorithm. In addition, states emphasized the importance of collecting quality data before sending to Maternal Mortality Review Committees for review and NCHS for coding. Lessons learned from this pilot provide guidance to increase collaboration between stakeholders in each state, prevent and/or correct certifier mistakes, improve timeliness of Maternal Mortality Review Committee review, and improve data quality processes at both the state and national level. A primary limitation of this evaluation is the use of focus groups instead of individual interviews, which may have affected how forthcoming participants chose to be in their responses. With 10 participants across the three focus groups, this evaluation may not be generalizable beyond the state agencies that participated.

The results of this pilot suggest that states may be able to achieve improved maternal death data by verifying a decedent’s pregnancy status through birth or fetal death linkage and certifier confirmation. The United Kingdom, described by experts as having the “gold standard for maternal health data,” implements a similar national vital records process wherein health care practitioners review medical records and additional reports to verify maternal deaths.⁶ Pilot participants and previous studies both suggest that additional certifier training, though challenging to deliver to the high number of state certifiers, could enhance efficacy of key pregnancy-related death indicators.^7,20,21 An ideal maternal mortality review process includes usage of the pregnancy checkbox and linkage to birth or fetal death records to capture true maternal deaths.^8,15,21 Vital statistics’ investment in providing quality pregnancy-related death data is essential to maternal death prevention efforts, as an accurate list of deaths enables the review committees to spend their limited time and resources on detailed investigations instead of case ascertainment.^14,22

Ultimately, quality vital records are a critical data source for public health purposes, including state perinatal quality improvement efforts such as perinatal quality collaboratives, fetal and infant mortality reviews, and Maternal Mortality Review Committees.²³ Participants in the pilot stated that conducting a quality assurance process for the pregnancy checkbox improves reliable state data, which influences state and national maternal mortality rates. As previous studies and pilot participants indicate, MMR data require especially rigorous quality validation because this key indicator of maternal health is calculated based on a small number of death occurrences.²⁰ Quality maternal health surveillance can improve national maternal health through accurate identification of maternal deaths, thorough maternal death case review, and timely dissemination of findings.¹