Public health researchers and practitioners have made the goals of increasing physical activity and improving healthy eating among youth a major national priority, recently through Michelle Obama’s Let’s Move Campaign [1]. These objectives are important given the rapid increase in childhood obesity in the United States, especially among Black and Latino children and those from economically disadvantaged backgrounds [2]. While current guidelines recommend children be physically active for at least 60 minutes each day, less than half of American children meet these standards [3-5]. Additionally, children seldom meet national dietary guidelines [6] and consumption of foods and drinks such as unhealthy snacks and sugary drinks has increased in recent years [7]. High obesity rates, low physical activity levels, and poor nutrition in childhood are all risk factors for chronic disease and higher obesity rates in adulthood [1].

Some of the most impactful strategies for improving youth activity and diet aim to create changes in school and afterschool environments. Interventions targeting these environments have been effective at increasing children’s physical activity and fitness [8-12], improving foods and beverages served and consumed by children [13-15], and decreasing obesity prevalence [13]. However, measuring physical activity and dietary intake with “gold standard” measures such as weighed plate waste and accelerometry requires training, time, and a substantial cost burden [16,17]. By contrast, lower cost surveillance measures for assessing trends in children’s physical activity and diet have typically relied on child and parent reports, which are subject to recall bias and have limited evidence for criterion validity [18-20]. While observational protocols have been developed for researchers to assess nutrition and physical activity among children in educational settings with less time and cost burden than “gold standard” measures [21,22], there has been limited focus on developing valid measures that practitioners can use. There is a need for tools teachers or afterschool staff can administer to assess program performance and evaluate progress as interventions are implemented. Measures that do not rely on trained research staff are also valuable for ongoing surveillance to simply and accurately track trends in program practices and child outcomes over time. In this paper, we aim to determine if a simple, low-cost, program-level, observational measure completed by afterschool program staff can validly assess afterschool physical activity and nutrition practices and child behaviors.

During phase 1, directors from 18 programs completed the practice assessment on five days each. One program completed OSNAP-OPAT on four days and another on only three days. In phase 2, most programs ran Monday-Thursday; thus, practice assessment and criterion data were usually collected on four days. Table 2 shows characteristics of the afterschool programs and the children who attended the programs. The typical program served about 50 children each day. Programs in this sample served more children of color (78%) than white children (22%) and 80% students at the school where the programs were based qualified for free/reduced price lunch.Table 2

Characteristics of afterschool programs and children where the Out-of-School Nutrition and Physical Activity Observational Practice Assessment Tool (OSNAP-OPAT) was completed

Directors who completed OSNAP-OPAT in phase 1 had an average of 3.5 (SD = 4.3) years in their role and an average age of 36 (SD = 10.6). Thirteen of 20 directors were women and 16 had a college degree. On average, the directors worked at the program 33 hours per week (SD = 9.4); some worked as full-time benefited employees, while other worked part-time.

This study establishes the criterion validity of a brief, low-cost, program-level observational measure of nutrition, physical activity, and screen time in afterschool programs that can be implemented by practitioners. When completed by afterschool directors, OSNAP-OPAT validly assessed serving water, fruits and vegetables, grains, and juice at snack as well as the amount of sugary drinks and snacks brought in from outside the program. Directors also validly assessed children’s water consumption. Reported physical activity participation was correlated with the proportion of children observed participating in MVPA and the amount of physical activity time reported was correlated with MVPA and VPA accelerometer estimates. The instrument validly assessed whether computer and television or movie time was offered during the program. Using OSNAP-OPAT, program directors were able to accurately assess program and participant practices related to obesity risk factors—healthy beverages, screen time, physical activity, and fruits and beverages—which have become the core components of recommended evidenced-based community interventions and programs [1,3].

Findings from this multi-phase study highlight important measurement considerations. In phase 1, dichotomous questions on the amount of time physical activity offered were not correlated with accelerometer data since most programs offered the targeted 30 minutes of daily physical activity and little variation existed in responses. When we changed items to a scale with five time intervals, physical activity items were significantly correlated with corresponding accelerometer estimates. Afterschool directors’ reports of the least amount of physical activity offered to any group of children were more highly correlated with children’s objectively measured physical activity levels than their reports of the physical activity of a typical child at the program. In phase 2, splitting a single sugary beverage question into three distinct items on juice, large juice (>4oz), and beverages with added sugar yielded valid assessments.

To date, limited research has sought to develop valid nutrition and physical activity measures that are easy for practitioners to use in free-living settings like afterschool programs with limited training and at low cost [21]. However, researchers have begun creating measures designed for early childcare staff [32,33]. The current study builds on this evidence base by using strong criterion measures over the course of one week to obtain a more precise estimate of nutrition and physical activity practices and behaviors. It also uses repeated assessment data to determine how one day of data performs in comparison to weekly criterion measures. While some items were significantly correlated with the criterion with one day of data, as expected the ranges of these estimates were wide. Whenever possible, researchers and program staff should use weekly averages, rather than one-day estimates. OSNAP-OPAT is low cost and requires minimal staff time to complete. Plus, the instructions for utilizing the measure are built into the tool, rather than requiring the extra expense and time of training. Its user-friendly format can help afterschool staff take action to improve their current practices. The measure and companion tools for identifying areas for improvement and developing action plans are available for free at www.osnap.org.

This study has several limitations. OSNAP-OPAT items completed by directors could not accurately assess the serving of snacks with trans fats nor children’s consumption of foods and beverages other than water. These items have been removed from the final measure. We encourage future research on the best way to capture accurate data on these items from practitioners in the field. The generalizability of our findings is limited due to the small sample of programs all located in Massachusetts who agreed to participate in a nutrition and physical activity intervention. However, the instrument was tested in programs sponsored by a range of organizations and in low income, racially diverse settings. While we were able to establish criterion validity—the extent to which OSNAP-OPAT performs similarly to “gold standard” measures of trained direct observation and accelerometry, assessing the measure’s content and construct validity were outside the scope of this study. Finally, because we initialized accelerometers using one-minute epochs according to the guidelines from the National Cancer Institute to be consistent with methods used in national surveillance, we were unable to analyze shorter physical activity intervals that may have captured intermittent physical activity. Recent research indicates that the Actigraph accelerometers and one-minute intervals capture children’s energy expenditure well compared to room calorimeters and doubly labeled water methods [25].

OSNAP-OPAT is a measure that can be used to validly assess program performance as interventions are implemented and evaluated in real world afterschool settings. This brief assessment tool will help researchers and practitioners gain an accurate assessment of afterschool program practices and child behaviors and target specific areas for improvement as they initiate and evaluate obesity prevention initiatives.