QRIS indicators are often publicly available and can be identified and accessed through the Quality Compendium’s Catalogue of QRIS Initiatives, a comprehensive database with information on all QRIS currently in place and with links to QRIS indicator documents or websites (Quality Compendium, 2020). We used the Quality Compendium’s database Catalogue of QRIS Initiatives to identify QRIS in place at any point during March 24^th through April 13^th of 2020 (Quality Compendium, ). During this time period, one author reviewed all state profiles on the Quality Compendium to 1) identify states with a statewide QRIS currently in place, and 2) systematically collect and compile documents or webpages outlining specific QRIS indicators for each state using the links provided in the Quality Compendium. A second author then reviewed QRIS webpages for each state QRIS to verify that all relevant and available QRIS indicator documents had been identified and downloaded. In one case, the research team contacted a state QRIS director directly by email to obtain QRIS indicator documents because they were not available or easy to locate through the Quality Compendium or on public websites.

Inclusion of state QRIS documents in the analysis was determined based on the following criteria: 1) initiative was listed in the Quality Compendium’s Catalogue of QRIS Initiatives; 2) initiative included statewide or universal regional QRIS indicators (for QRIS that operate at the regional/county level but with select indicators applied universally); 3) initiative was in place at any point within the defined period of data collection and was not in a pilot or developmental phase. We identified a total of 41 states operating QRIS for inclusion in the analysis, including D.C., and California’s universal indicators applied to regional QRIS implementation.

We conducted a directed content analysis of QRIS indicators for the 41 state initiatives identified. Content analysis is an approach to analyzing text data which can incorporate elements of qualitative and quantitative analysis. Directed content analysis is guided by a predefined theory or evidence from prior research, as opposed to conventional content analysis, which allows categorization and theory to emerge from the data (Hsieh & Shannon, 2005). Directed content analysis was selected as the methodological approach for this study, given our interest in identifying QRIS indicators consistent with pre-defined quality standards based on prior research in the fields of early childhood development (American Academy of Pediatrics et al., 2019). In order to identify, sort, and quantify QRIS indicators relevant to promotion of early childhood development and mental health, we employed directed content analysis by developing a codebook based on evidence-informed quality standards from Caring for Our Children (American Academy of Pediatrics et al., 2019) before engaging in the process of independent coding of state QRIS indicators, discussion and resolution of coding discrepancies, and codebook refinement by a team of three authors. Our iterative coding process was adapted from the template analysis approach, a method of coding qualitative data often applied to thematic analysis (Brooks et al., 2015).

The codebook was drafted and organized prior to coding the QRIS indicators, with revisions and refinements made throughout the iterative coding process. Deductive codes (developed a-priori) were derived from Caring for Our Children quality standards (American Academy of Pediatrics et al., 2019) relevant to promotion of early childhood development and mental health and were organized by category. To identify Caring for Our Children quality standards from which to derive codes for the analysis, members of the research team with expertise in early childhood development read through all standards twice and generated a list of those recognized as relevant to promotion of early childhood development and mental health in ECE. The team then reviewed and refined the list of selected standards before reformatting each standard description as a code definition and assigning each a brief code label. Two subject-matter experts external to the research team were then invited to review these preliminary codes and code definitions to ensure they were inclusive of ECE quality standards most salient to promotion of early childhood development and mental health, and to identify any critical gaps.

A total of 26 deductive codes were defined based on Caring for Our Children standards with subject-matter expert input. An additional 6 codes were developed inductively, (during the coding process) based on specific staff trainings, professional certificates, and classroom rating or assessment systems described within QRIS indicators. These trainings, certificates, or assessments were recognized by researchers as very likely to encompass one or more Caring for Our Children quality standards represented by the 26 deductive codes and were thus deemed relevant to include in data collection. The codebook was organized by sorting all inductive and deductive codes into five categories or coding themes: (1) developmental monitoring, screening, and referral (8 codes), (2) classroom procedures and environment (12 codes), (3) staff/caregiver training and support (6 codes), (4) health advocates and consultants (2 codes), and (5) parent/guardian engagement (4 codes). Table 1 lists all 32 primary codes (including 26 deductive and 6 inductive codes) and their definitions by theme, as well as examples of corresponding coded QRIS indicators; examples presented were selected as exemplary representations of the Caring for Our Children -derived codes. Functional subcodes were also generated to identify whether each coded indicator (or an equivalent indicator also meeting the code definition) applied to center-based ECE, family child care, or both settings.

Two researchers independently coded all indicators in the compiled state QRIS documents using Microsoft Excel. Researchers only coded indicators which were clearly identified in the documents as part of the state QRIS rating criteria. Indicators were coded regardless of whether conditions outlined in the codebook definition and the corresponding Caring for Our Children standard were fully or partially met. For example, the code opportunities for parents/ guardians to observe staff facilitating child development was derived from a Caring for Our Children quality standard which stated, “Parents/guardians should be given opportunities to observe staff members modeling healthy and safe behavior and facilitating child development,” whereas the code was applied to any QRIS indicator suggesting opportunities for parent participation in or observation of classroom or program activities.

The iterative, cyclical coding process consisted of the following steps: 1) independent coding of 2-3 states’ QRIS indicators by two authors; 2) discussion of coding and resolution of discrepancies by authors; and 3) changes to the codebook such as revision of codebook definitions or addition of inductive codes, as needed. Changes to code definitions were made throughout the coding process to improve coding efficiency and inter-coder agreement. All codebook revisions were approved by the entire research team. Once QRIS indicators for all states were independently coded and reviewed, the final codebook was then re-applied to all states’ QRIS indicators (Brooks et al., 2015). Inter-rater reliability was calculated following completion of the final round of coding based on independent re-coding of data from 20% of states by two researchers by dividing the total number of codes assigned to each states’ indicators by the number of codes in agreement with coding by a second researcher (McHugh, 2012). Inter-rater reliability calculated as percent coding agreement among researchers was 81.5%, pointing to reliability of results. Microsoft Excel Pivot Tables were used to generate counts of statewide QRIS with coded indicators by individual codes and by overarching category of promotion, stratified by program type (center-based, family child care, or unspecified). Researchers identified individual codes and categories of promotion most and least often employed by QRIS indicators. Patterns in the content and quantity of coded indicators were explored and described. This study was exempt from review by an Institutional Review Board due to use of publicly available data (online QRIS documents).

Many QRIS included indicators for developmental monitoring, screening, and referral, however, there was variability in the level of detail included in indicators relevant to this theme and many QRIS did not include indicators related to communication or collaboration with parents/guardians or other providers on developmental monitoring or screening. Some of the most frequently applied codes within this theme included caregivers/teachers monitor children’s development (n=31 or 91%), program utilizes validated developmental screening tool (n=23 or 68%), and if screening results in concern, child referred to their primary care provider (medical home) or to an appropriate specialist or clinic for further evaluation (n=21 or 62%). Fewer states included QRIS indicators coded as caregivers/teachers share results of developmental screening or monitoring with parents/guardians (n=17 or 50%), caregivers/teachers provide child development resource information to parents as needed (n=14 or 41%), caregivers/teachers gather input from parents/guardians to monitor children’s development (n=8 or 24%), and developmental screening includes parent/guardian consent or participation (n=3 or 9%). No states included indicators coded as caregivers/teachers gather input on development from healthcare providers, consultants, and/or other staff who know the child.

The level of detail offered by QRIS indicators coded as caregivers/teachers monitor children’s development differed by state. For example, California’s indicator describes the process, setting, and content of developmental assessments; “Assessment of children’s growth and development is an ongoing process and is conducted during children’s daily activities and routines to assess progress in the 4 domain areas of social, emotional, cognitive and physical development” Meanwhile, several other state standards coded as caregivers/teachers monitor children’s development offered less detail about the processes, but more about the intent, for example, “Developmental monitoring tools are used to provide early detection of health-related issues and developmental delays to support early intervention” (Georgia).

A large proportion of initiatives included indicators relevant to staff/caregiver training and support, however, many indicators did not specify topics or content of staff trainings, and fewer indicators were identified related to supporting staff health and wellbeing. The majority of initiatives (n=37 or 95%) were coded pre-service and/or continuing education training required but defined elsewhere and were not described in the QRIS indicator language, such as in the following example from Rhode Island’s indicators: “Professional Development Plan: All teachers/the educator have a written individual professional development plan aligned with RI’s Workforce Knowledge and Competencies.” Just over half of state systems (n=22 or 56%) included a QRIS indicator for caregivers/teachers complete continuing education training and/or orientation on child development, and 18 of those indicators clearly applied to both centers and family child care. For example, one of New Mexico’s QRIS professional development indicators for both centers and family child care stated, “At least one educator per classroom (preferably the lead educator) must have successfully completed: Child Growth, Development, & Learning.” QRIS indicators for twenty-one state systems included a requirement for caregivers/teachers have Child Development Associate credential (CDA). Twelve states mentioned implementation of and/or training related to Strengthening Families: A Protective Factors Framework, a framework which includes content related to building trusting, supportive relationships with parents/guardians (Harper Browne, 2016). Only nine state systems included at least one indicator for continuing education training/orientation covers relationships with families, including the following example from Illinois state indicators for family child care: “FCC [family child care] Provider has completed ExceleRate-approved training on family engagement and communication strategies.”

Seventeen state systems (n = 35%) included the QRIS indicator for specific policies to promote health of staff, such as family leave, sick time, and health insurance. However, only nine of those indicators clearly applied to family child care. Our codebook initially included a separate code for activities to promote health of staff, based on the Caring for Our Children standard “Health Promotion for the Staff,” which describes “activities such as health assessments, health education, help in accessing immunizations, health-related fitness activities, and time for staff to be outdoors” (American Academy of Pediatrics et al., 2019). However, our coding did not identify any QRIS indicators related to specific activities implemented by the program related to promotion of staff wellbeing and health. Though none were formally coded, we noted some examples of QRIS indicators describing less defined or structured opportunities for staff to engage in wellness activities such as the following examples from Oregon’s and Alaska’s standards, respectively:

As these examples did not describe specific activities, space, or time for staff health education, services, or relaxation distinct from time and space for planning and work-related activities, they were not formally coded as activities to promote health/wellbeing of staff.

The majority of state QRIS based indicators related to classroom procedures and environment, at least in part, on external rating or assessment systems, while many of the specific Caring for Our Children standards in this theme were rarely identified in indicators. Many QRIS included indicators on Environment Rating Scales (n=33 or 80%), and most of those applied to both centers and family child care (n=31 out of 33 or 94%). Environment Rating Scales are designed to assess quality of interactions between children and their classroom environment (Teachers College Press, 2022). Another external rating system, Classroom Assessment Scoring System^® (CLASS) (Teachstone, 2022), was referenced in indicators for 18 state QRIS (44%) for assessment of classroom interactions and relationships, though only 10 of those indicators clearly applied to family child care. The majority of state initiatives that assessed classroom relationships and interactions using CLASS, also used Environment Rating Scales to assess the environment (n=16 out of 18 or 89%).

Many state QRIS included indicators describing features of the planned, daily activities or curriculum, including that they foster developmental progress across domains (n=32 or 78%), are flexible to capture the interests and individual abilities of children (n=24 or 59%), and reflect the languages and cultures and languages of families served by the program (n=19 or 46%). The majority of state QRIS included staff to child ratios in indicators (n=23 or 56%), though only 14 of those applied to both family child care and centers.

Few state QRIS included specific indicators highlighting other aspects of the classroom environment and interactions with caregivers which promote early development and mental health, such as adults engage in frequent verbal exchanges with all children (n=6 or 15%), primary caregiving (n=6 or 15%), having developmentally appropriate discipline policies (n=6 or 15%), and continuity of care policy/procedure (n=5 or 12%). The following example from Montana’s rating system applied to both family child care and centers and was coded for primary caregiving and continuity of care (in addition to staff to child ratios): “A written staffing plan is in place assuring continuity of care (including a plan for substitute staff situations), appropriate adult to child ratios, appropriate group size, and that children are benefitting from having primary caregivers.” Also, few state initiatives included indicators coded as play is the foundation of the curriculum or activities (n=4 or 10%). In this example from Pennsylvania’s indicators, both learning through play and child development are addressed; “Lesson plans reflect a balance of activities that support developmentally appropriate learning through play.” Finally, Positive Behavioral Interventions and Support, often referred to as the Pyramid Model in early childhood settings, (Hemmeter et al., 2016) focuses on developmentally appropriate classroom management and behavioral supports and was included as a staff/caregiver training and support indicator for two state QRIS.

Caring for Our Children standards in this theme were least often represented in state QRIS indicators. Ten state QRIS (24%) included an indicator on engaging or partnering with a child care health consultant, with minimal differences by centers or family child care standards (Fig. 1). Caring for Our Children defines a child care health consultant as “a licensed health professional with education and experience in child and community health and child care and preferably specialized training in child care health consultation… [who] is not acting as a primary care provider at the facility but offers critical services to the program and families by sharing health and developmental expertise, assessments of child, staff, and family health needs and community resources (American Academy of Pediatrics et al., 2019).” Across state QRIS, descriptions of the role of child care health consultants differed, from providing little or no detail to describing engagement with child care health consultants as serving a specific purpose, such as establishing and maintaining health policies, completing injury prevention checklists or safety assessments, and assisting providers in developing strategies to support children’s health, development, and behavior. This range of child care health consultant role descriptions is illustrated by the following examples:

No states included QRIS indicators coded as program has designated health advocate. Caring for Our Children defines a health advocate as an “administrator or staff person…responsible for policies and day-to-day issues related to health, development, and safety of individual children, children as a group, staff, and parents/guardians. The primary contact for parents/guardians when they have health concerns, including health-related parent/guardian/staff observations, health-related information, and the provision of resources (American Academy of Pediatrics et al., 2019).”

The majority of QRIS included indicators about regular, planned communication with parents/guardians (n=32 or 91%) and program offers parent/guardian education about parenting or child development (n=28 or 80%). Fewer states specified regular, planned communication with parents/guardians about child’s development (n=12 or 34%) and that there should be opportunities for parents/guardians to observe staff facilitating child development (n=9 or 26%). There were minimal differences between centers and family child care indicators in this theme (Fig 1). For two states, the code opportunities for parents/guardians to observe staff facilitating child development, was coded for centers, but not for family child care. Researchers noted that some indicators coded for program offers parent/guardian education about parenting or child development suggested offering parents/guardians information about accessing services or resources to help ensure that children’s needs are met, such as this example from Alaska’s QRIS: “Program provides access to supports and community resources to assist families in: meeting their child’s needs; increasing their knowledge of child development; making social connections; and helping successfully transition their child to school.”

Findings of this study provide state and local leaders with information about how states have integrated evidence-informed standards for promotion of child development and mental health in QRIS indicators and highlight opportunities for further integration and research. These in-depth findings are a helpful companion to the snapshot of indicators available in the Quality Compendium. Findings also provide information on indicators for both center-based ECE and family child care (BUILD Initiative & QRIS National Learning Network, 2019 ; Meek, 2020; NASEM, 2019b). This study included all statewide QRIS indicators in place at the time the document searches were conducted (March and April of 2020), including California’s universal regional indicators. Our qualitative content methodology featured independent coding of the QRIS indicators and identification and resolution of coding differences by multiple independent researchers in order to enhance dependability and reduce subjective bias (Hsieh & Shannon, 2005; Ulin et al., 2005). Furthermore, calculations of inter-rater reliability indicated over 80% coding agreement among researchers, suggesting that the results are reliable and accurate (McHugh, 2012).

There are several limitations to note about this analysis. First, the codebook developed for this study may not be inclusive of all qualities of ECE environments and practices promotive of early childhood development and mental health. Given resource and feasibility limitations as well as the intended scope of this project, the research team had to be judicious while selecting relevant Caring for Our Children standards for inclusion in the codebook. Care was taken to identify the most salient Caring for Our Children standards for inclusion, and external subject-matter experts were consulted and invited to review those selected, in part, to address any potential bias on behalf of the research team during code development. Furthermore, Caring for Our Children standards are based, in part, on a limited body of peer reviewed scientific research in the ECE field demonstrating significant associations between ECE quality standards and desirable child outcomes (American Academy of Pediatrics et al., 2019; Soliday Hong et al., 2019). Therefore, there may be aspects of quality ECE that have not yet been identified for inclusion in Caring for Our Children standards, and additional research may be needed to support and enhance implementation of existing standards (Soliday Hong et al., 2019).

Second, the team made efforts to thoroughly search state QRIS websites and directly contacted programs when rating criteria were not available through the Quality Compendium or on public websites. Still, it is possible that additional information on rating criteria could have been updated elsewhere prior to the period of data collection and/or could have appeared in appendices or documents that were not included with the documents accessible to the study team. It is also important to note that coding and analysis was restricted to the indicators that were used to provide a score to determine QRIS ratings as this was an exploration of how specific child development and mental health standards are included in quality indicators. Caring for Our Children standards identified less frequently in QRIS indicators, such as health advocates and specific guidance for developmental monitoring and screening practices, may instead be supported in some states through QRIS guidance outside of the quality indicators assessed for this study. Content relevant to the standards coded for this project could have been present in introductory statements, descriptions of coaching available to participants or other initiatives described in the QRIS documentation. For example, given that health consultation services were described in some states’ QRIS documentation external to quality indicators, other relevant Caring for Our Children standards may also be represented elsewhere in QRIS guidance. Further, inclusion of Caring for Our Children standards in QRIS guidance (i.e., quality indicators and other documentation) does not necessarily translate to consistent or appropriate implementation by participating ECE programs in practice (Tout, 2017).

Finally, this analysis is not meant to be interpreted as a reflection of other strategies outside of QRIS that might be in place to address Caring for Our Children standards. There are many strategies outside of QRIS including licensing, technical assistance programs, and other initiatives a state may employ to address child development and mental health standards (BUILD Initative, 2020). It may be the case that state systems rely on technical assistance programs or other supports to ensure these components are in place. In 2021, nearly half of states reporting information to the QRIS Quality Compendium included licensing as the first step or lowest rating level in their QRIS (Quality Compendium, 2022a). Since we did not review licensing information, we cannot ascertain whether there are child development and mental health related standards in the first rating levels for those states. The purpose of licensing is to ensure basic protections for children in care, though many states use Caring for Our Children as a resource for developing rules and regulations for licensing (Quality Compendium, 2020). We also did not code and analyze state professional development system criteria that was not available in QRIS indicators. Similarly, we did not code and analyze the criteria in Environment Rating Scales, CLASS, or other classroom environmental assessments that many QRIS were using as the basis for ratings. Therefore, we cannot conclude from this analysis how state systems that relied on these assessments have addressed child development and mental health promotion standards in their QRIS rating criteria.

Additionally, our analysis identified 41 states with active QRIS as of March 24^th through April 13^th, 2020, based on the Quality Compendium, leaving 10 additional states that we did not investigate for QRIS implementation. As of December 2022, QRIS implementation varied across these states with two having QRIS in place (Alabama and Colorado), three working to develop QRIS (Kansas, West Virginia, and Wyoming), one using CLASS observations to evaluate classrooms (Florida), and three in development of quality improvement systems (Hawaii, Missouri, and South Dakota) (Quality Compendium, 2022b). Meanwhile, one state included in our analysis no longer has an active QRIS in place (Mississippi) (Quality Compendium, 2022b). Ongoing efforts to monitor and assess QRIS quality indicators and implementation structures may identify additional opportunities to integrate evidence-informed child development and mental health promotion standards.

Early childhood system and program leaders can use these findings to inform future QRIS and quality improvement work in their states or localities. Child development and mental health standards reflect essential components of quality ECE (American Academy of Pediatrics et al., 2019) and quality ECE is an important health equity strategy for reducing disparities (NASEM, 2019a). Information on how states have developed language for rating criteria on these standards can be helpful for updates or revisions to QRIS indicators or other quality initiatives. Furthermore, this work may inform efforts to assess or promote integration of evidence-informed standards in ECE quality initiatives beyond the U.S. context. For the child development and mental health related standards that fewer states included in their ratings, such as health consultants or advocates, leaders could consider whether to include these standards in future iterations of QRIS or whether other state initiatives, licensing, or other efforts adequately address these Caring for Our Children standard elements. Furthermore, future studies could examine the inclusion of these child development and mental health standards in licensing regulations, to understand the extent to which these standards may be a part of the foundational or first rating level of some QRIS and to examine potential variability in state approaches to these standards through licensing. Similar review of quality indicators within other common ECE quality assessments and initiatives, such as CLASS and Environment Rating Scales, could enhance understanding of their role in supporting early childhood development and mental health in ECE systems. Future research could also explore how different state ECE professional development systems address these child development and mental health standards to provide state leaders with more information to assist with planning and early childhood system development.

Additionally, researchers noted that some indicators coded for program offers parent/guardian education about parenting or child development included guidance for ECE programs related to offering parents/guardians information about accessing services or resources to help meet children’s basic needs, such as through social services, primary healthcare, and/or parent/guardian social support (see Table 1 examples). The guidance provided by these QRIS indicators is in alignment with a recent call to action from the Center for the Developing Child at Harvard University to expand the focus of the early childhood field toward addressing inequities which place disproportionate burdens on families with young children (Shonkoff, 2022). Such guidance also supports a preventive, integrated model of care by addressing contextual and multi-generational challenges which can impact early childhood mental health (Buka et al., 2022). As it was not systematically assessed or documented as part of our study, future inquiry may more directly assess the extent to which QRIS supports this recommendation to address structural, social, and family-level determinants of child mental health and development and underlying inequities. Future research could also examine the extent to which child development and mental health are promoted equitably through QRIS and to identify opportunities to ensure that QRIS guidance and resources are reaching ECE programs and communities with the greatest need (Jenkins et al., 2021; Meek, 2020). While expanding access to quality ECE is widely considered a promising strategy for promoting children’s mental and physical health equity on a population level (OSG, 2021) and QRIS has been suggested as a possible lever for doing so (NASEM, 2019b), potential barriers to equity in QRIS have been suggested (Jenkins et al., 2021; NASEM, 2019b). A recent study employed a nationally-representative sample of child care centers to identify characteristics of those engaged in QRIS and found that only 1/3 of child care centers participated in QRIS and that lower QRIS engagement was more likely among child care centers located in communities with larger proportions of Black residents, compared to those located in communities with low to moderate proportions of Black residents (Jenkins et al., 2021). Without equitably engaging ECE programs across diverse communities, QRIS may contribute to widening racial/ethnic disparities in access to high-quality ECE and associated positive health and developmental outcomes which extend beyond early childhood (BUILD Initiative, 2020; BUILD Initiative & QRIS National Learning Network, 2019; Jenkins et al., 2021; Jiashan & Natzke, 2021; Meek, 2020; NASEM, 2019b).

While our findings suggest that the majority of QRIS indicators relevant to child development and mental health were employed by QRIS as rating criteria for both FCCs and centers, an important factor for achieving equitable reach of the guidance (BUILD Initiative & QRIS National Learning Network, 2019 ; Jiashan & Natzke, 2021; NASEM, 2019b), future research may investigate and describe FCC participation in QRIS, as has recently been done for center-based programs (Jenkins et al., 2021). Furthermore, future research may elucidate opportunities for QRIS to enhance equitable access to quality ECE through other avenues, such as inclusion of ECE equity indicators, assessment of cultural relevance of indicators used to define ECE quality, restructuring of quality ECE systems away from rating-based systems, and directing funding resources toward supporting and training a diverse and culturally responsive early childhood workforce (Curenton et al., 2020; Jenkins et al., 2021; Meek, 2020; NASEM, 2019b).

Finally, states may consider equity implications for comprehensively embedding CFOC child development and mental health promotion standards in QRIS indicators and other measures of quality for ECE systems. For example, highly prescriptive definitions of quality across diverse communities may generate disproportionate barriers for some providers, which could lead to inequitable distribution of incentives, recognition, and support (Meek, 2020; Yazejian & Iruka, 2015). While lower quality ECE programs may be more likely to be located in communities with higher concentrated disadvantage, resource-related barriers to obtaining high quality ratings, such as lack of local referral networks to address indicated developmental delays, or limited funds for staff professional development, may exacerbate disparities (Hatfield et al., 2015). Barriers to achieving high quality ratings may also emerge if quality guidance is inflexible to accommodate cultural norms or practices (Meek, 2020). Meaningfully integrating equity considerations in all efforts to define and guide quality at the state level may help ensure that such efforts support, not hinder, availability of high quality ECE, particularly in communities that need it the most.