There has been a recent effort to differentiate between two types of translational research: Type I translational research focuses on discovery through clinical trials, whereas Type II examines the process by which efficacious practices, interventions, or treatments become implemented effectively in real-world settings.⁵ The current paper focuses on Type II translational research, which “is aimed at enhancing the adoption, implementation, and sustainability of evidence-based or scientifically validated interventions” ^{4(p 2)} and focuses on achieving broad, population-level effects. While there is an established literature supporting the efficacy (Type I) and effectiveness (one element of Type II research) of a variety of intervention approaches or programs, there has been less empirical work specifically on the process of translating efficacious practices into real-world settings or on how to support implementation and the scaling-up processes.⁶ When it comes to scale-up efforts in schools, there is limited empirical research on the extent to which prevention programs are adequately implemented and the association between implementation quality and student outcomes. This illustrates a clear need for additional research on the process of dissemination or planned diffusion ¹⁶ of evidence-based programs and whether the effects seen in randomized trials are replicated when brought to scale.¹⁷

The current paper focuses on the scale-up of SW-PBIS,¹⁴ which is a non-curricular, school-based prevention approach which aims to promote changes in staff behavior in order to positively impact student outcomes such as student discipline, behavior, and academic outcomes. SW-PBIS ¹⁴ is based on behavioral, social learning, and organizational behavioral principles. The model is implemented in all school contexts (classroom and non-classroom) with the aim of improving a school’s systems and procedures to prevent disruptive behavior and enhance the school’s organizational climate. The model follows the three-tiered prevention framework, where a universal system of support is integrated with selective and indicated preventive interventions for students displaying a higher level of need.¹⁸ Two recent randomized controlled effectiveness trials provide evidence of positive outcomes of the universal elements of the SW-PBIS model. Specifically, SW-PBIS has been shown to be effective at reducing student office discipline referrals and suspensions, and improving school climate.^19–22 Teachers in SW-PBIS schools also rate their students as needing fewer specialized support services, and as having fewer behavioral problems (e.g., aggressive behavior, concentration problems, bullying, rejection).^23,24 In addition, there are some favorable results from state-wide evaluations of SW-PBIS.^25,26 Taken together, these studies provide evidence that states can implement SW-PBIS on a large scale and that schools adopting SW-PBIS experience positive effects. Given the wide dissemination of SW-PBIS and previous research documenting its effectiveness, it is a particularly good candidate for Type II translational research focused on implementation quality in scale-up efforts.

Maryland has developed a coordinated system for implementation of SW-PBIS. Over the past 12 years, a collaboration between the Maryland State Department of Education, Sheppard Pratt Health System, and Johns Hopkins University^25–27 has trained a total of 877 schools (e.g., elementary, middle, high, alternative, special) in SW-PBIS, of which 740 (84 % of trained schools) are actively implementing and participating in the state initiative. This is made possible through the state-wide infrastructure, which includes a variety of core elements for dissemination,^7,28–30 including a consortium of stakeholders (e.g., educators, researchers, policymakers) who jointly coordinate, train, and support schools in the implementation of SW-PBIS. There are multiple levels of coordination (for details, see Barrett et al.²⁵) to promote high quality implementation. Similar systems of support have been utilized in other translational efforts to disseminate programs and achieve high fidelity (for examples, see Bloomquist et al.,³¹ Fixsen et al.,³² Spoth and Greenberg³³). The Maryland Initiative also maximizes the dissemination of SW-PBIS through the promotion of exchange between school practitioners, who may be more effective in shaping their colleagues’ opinions about SW-PBIS than the consortium,³⁴ and by utilizing coaches and district leaders.^16,34 Finally, there is ongoing data collection, evaluation, and technical assistance provided by the partners regarding implementation and outcomes.^29,30–33 The data from the current study come from the state’s evaluation efforts.

While several efficacy studies of behavioral or mental health prevention programs have documented an association between implementation and outcomes,^35–39 there have been relatively few studies which have examined the link between implementation and outcomes within the context of state-wide scale-up efforts. For example, research on the Triple P-Positive Parenting Program, which targets changes in child behavior through training parents to alter the home environment,⁴⁰ has found that the intensity of the program as well as the format (e.g., self-directed versus group format) was significantly associated with parent and child outcomes.⁴¹ Similarly, an evaluation of the Promoting Alternative Thinking Strategies (PATHS) social–emotional curriculum reported a significant interaction between implementation and contextual factors, like administrator support, on student behavioral and emotional outcomes.⁴² Taken together, the available research suggests a need for more empirical research on the association between implementation quality and outcomes when interventions are brought to scale.⁴³

When examining the association between implementation and outcomes, it is important to adjust for contextual factors, which may influence implementation, as well as the outcomes. For example, there is literature suggesting that a high rate of disorder or disorganization can impede successful implementation of programs and can negatively impact program outcomes, ^15,20,44 whereas a climate which encourages the adherence to implementation (or fidelity) may improve implementation. ^16,45 A previous study of SW-PBIS in elementary schools found that implementation fidelity was associated with school-level factors, such as the percent of certified teachers in the school;²⁷ however, the available district-level predictors were not associated with implementation. Also relevant was the number of years since training, such that schools that implemented the model longer achieved higher levels of fidelity²⁷ (see Rohrbach et al.³⁰ and Rogers³⁴). Together, these findings suggest that it is important to account for school-level contextual factors when examining the association between implementation and outcomes in scale-up efforts.

The current paper examined how the level of implementation of SW-PBIS related to student outcomes, while adjusting for school-level contextual factors, which are associated with both implementation quality and student outcomes. The data come from the state-wide evaluation of SW-PBIS, which is led by the PBIS Maryland Consortium. A variety of data elements are collected by the PBIS Maryland Consortium, including the implementation quality of SW-PBIS. The data reported in this paper focus on program implementation in the spring of 2009 (i.e., from the 2008 to 2009 year) and student outcomes in spring 2010 (i.e., from the 2009 to 2010 year), while controlling for predictor variables, which preceded each individual school’s year of training. Data from elementary and middle schools were examined, including traditional K-5 or K-6 elementary schools, K-8 schools, and middle schools with grades 5 or 6 to 8. High schools implementing SW-PBIS were excluded because the assessment of student outcomes varied substantially for this school level (i.e., different standardized testing approach).

The outcomes of interest were student achievement on the Maryland School Assessment (MSA) for math and reading, truancy rates (i.e., percent of students absent greater than 20 days in the school year), and suspensions (i.e., the total number of suspension events divided by the total number of students times 100). Baseline data for each outcome (i.e., achievement, truancy, and suspensions in the year prior to the school’s training in SW-PBIS) were controlled for. The level of implementation of SW-PBIS was assessed by three measures: the Implementation Phases Inventory (IPI),⁴⁶ the School-wide Evaluation Tool (SET),⁴⁷ and the Benchmarks of Quality (BoQ),^48,49

It was hypothesized that higher levels of implementation would be associated with higher levels of achievement and lower rates of truancy and suspensions. Based on the literature identifying potentially important contextual factors on implementation quality and student outcomes,¹⁵ a set of school-level variables (i.e., student enrollment; students per teacher; rates of mobility and teacher certification; and years since training) was controlled for. It was hypothesized that large school size and student to teacher ratio, lower rates of teacher certification, and high student mobility would be associated with poorer SW-PBIS implementation and outcomes, based on the use of these variables as proxies for disorder.¹⁵ On the other hand, we hypothesized that the longer the schools implemented SW-PBIS, the higher their implementation quality would be.⁵⁰

As a requirement of the PBIS Maryland Initiative, the IPI is completed bi-annually (fall and spring) by a district-appointed technical assistance provider (i.e., a SW-PBIS Coach, which in Maryland is often a school psychologist or counselor) and submitted electronically to the PBIS Maryland Consortium through the www.PBISMaryland.org web site. As noted above, the SET is completed by an external district assessor, and the BoQ is completed by the school’s SW-PBIS team; these data elements are completed annually in the spring and are also submitted electronically through the Consortium’s website. The non-identifiable school-level data have been approved for analysis in this study by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board.

The Mplus 6.1 statistical software ⁵⁴ was used to fit a structural equation model (SEM) ⁵⁵ in order to test the hypothesized associations between fidelity and student outcomes, while adjusting for covariates. Specifically, an SEM using maximum likelihood robust (MLR) estimation was fit. A confirmatory factor analysis (CFA) was conducted on the measurement model of implementation (i.e., the four IPI scales and the seven SET scales). A latent variable SEM approach was taken to allow for a reduction in the fidelity data dimensionality; to allow for a parsimonious, more interpretable model; and to eliminate concerns of multicollinearity due to the high correlations among the subscales within each measure (i.e., the IPI and SET).⁵⁶ The BoQ was modeled as a third manifest (i.e., observed) indicator of implementation fidelity, as only the overall score was available to the researchers. The two implementation factors and the observed BoQ scores were then used to predict student outcomes (i.e., math and reading academic performance, truancy, and suspensions), while adjusting for the school-level covariates (i.e., years since training, school enrollment, the student/teacher ratio, the percent of certified teachers, and student mobility) and the baseline outcome measures.

As schools were nested within districts, the clustering of schools within districts was accounted for using the Huber–White corrections to adjust the standard errors;⁵⁴ however, district-level covariates were not modeled due to the relatively small number of districts (i.e., 24), and because prior research using this dataset suggested that district covariates generally were not significantly associated with implementation.²⁷ Given that all of the schools in the study were from a single state, no state variables could be modeled. Model fit was determined through inspection of the Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and the Standardized Root Mean Residual (SRMR).⁵⁷ A value between .05 and .08 on the RMSEA is considered acceptable fit; a CFI and TLI of greater than .90 is considered acceptable; and the SRMR should be less than or equal to .08.⁵⁸

As described above, an SEM was fit to test the primary hypotheses regarding the association between the different sources of implementation fidelity data and student outcomes, while adjusting for school-level demographic factors and the baseline student outcomes. First, the measurement model was fit to verify the factor structure of the two latent implementation variables (i.e., the IPI and SET). The CFA indicated that these latent variables had adequate fit with an RMSEA=.045, CFI=.962, TLI=.951, and SRMR=.116 (see the Measurement Model section in Table 3 for factor loadings).

There are several limitations to consider when interpreting these findings. Type II translational research is often characterized as “messy,” ⁵ as it is difficult to implement carefully controlled designs when examining the real-world process of program implementation. In addition, the measures of contextual factors used were school-level proxies for disorder,⁶¹ rather than specific survey measures of students or staff members. On the other hand, multiple ratings of the implementation quality of SW-PBIS were used, which is a unique and strong aspect of this study.

The outcome and implementation data come from one time point (i.e., the Spring of 2010 and the 2008 to 2009 school year, respectively) which is 11 years after the state initiative began. This highlights two other common obstacles in translational research, including the extended amount of time it may take to disseminate an intervention or approach ^30,34 and the difficulty of assessment of a “moving target”.^{5(p 212)} By focusing on two school years (baseline and outcome), the data analysis is simplified and made more interpretable; however, future analyses should take into consideration patterns of implementation over time, beginning with the first year of implementation. The fact that different numbers of schools joined the initiative at different points across the 12-year effort makes such analyses complicated, as the data would need to be aligned by implementation year, rather than calendar year. Despite this, all school demographics modeled were from the year preceding the training year and the number of years since training was accounted for. It is possible that the associations between implementation fidelity and outcomes would vary at different points in the scaling-up process (e.g., if measured earlier in the statewide scale-up or in the future). This is an area for further research.

Given that this study occurred in one state, it is unknown whether the findings would generalize to other states where there may be different levels of support provided to schools implementing SW-PBIS and the data collected regarding implementation, the school context (e.g., requirements for teacher certification, varying school sizes or levels of mobility), and the tests of achievement outcomes would be different. Finally, non-implementing schools were not examined in this study, as implementation data were not collected from these schools. Therefore, it is unknown whether schools not trained in SW-PBIS implement similar strategies to trained schools or whether the use of SW-PBIS is superior to non-use within the context of a state-wide scale-up (i.e., randomized controlled trials have established its effectiveness on a smaller scale). Similarly, we were not able to track the implementation of other programs in combination with SW-PBIS (e.g., bullying prevention); this is, however, an important area for further research, as previous studies suggest that schools are likely implementing multiple prevention programs simultaneously.¹⁰

As noted above, the findings for the SET were less informative than the other two measures of implementation. This was a bit surprising given that the SET is the most established and most widely used measure of SW-PBIS fidelity, whereas the IPI and BoQ are newer measures. On the other hand, these two latter measures were developed in part to address some of the concerns regarding the SET related to a potential ceiling effects ⁵³ and the burden of administration by an outside assessor. The current findings suggest that the IPI has the best predictive validity of the three measures examined. This finding also highlights a practical barrier in conducting scale-up efforts and evaluating their effectiveness; what is practical (e.g., only collecting data using one implementation measure) and desired may not result in a comprehensive understanding of the outcomes. On the other hand, collecting multiple measures of implementation from different sources is often seen as burdensome and redundant by schools but may be important.