A total of nineteen programs were funded to implement the GOALS Approach as part of their HIV prevention contracts. Programs are housed at a mix of clinical sites (e.g., academic medical centers, community hospitals, community health centers; n = 9 programs) and non-clinical sites (e.g., community-based organizations; n = 10), spread across all five boroughs of New York City (2 in the Bronx, 4 in Brooklyn, 10 in Manhattan, 1 in Queens, and 2 on Staten Island). Programs vary by size, with 42% (n = 8) serving fewer than 1,000 clients per year and 32% (n = 6) serving over 5,000 clients per year. Funded programs focus on populations considered high priority by the NYC HD for access to HIV prevention services, including: Black, Indigenous, Hispanic/Latino, Asian, and other people of color (n = 18); lesbian, gay, bisexual, transgender, queer, and other sexual/gender minority (LGBTQ+) individuals (n = 8); youth and young adults ages 13-29 years (n = 9); Black and/or Latina women (n = 9); people who use substances and/or have substance use disorder (n = 5); people experiencing homelessness or housing instability and (n = 3); and people born outside the United States (n = 5). Each program designated 1-2 project leads who were responsible for program-level implementation and participated in cross-program learning collaboratives.

Data were collected on outcomes associated with each of the implementation strategies that comprised the NYC HD’s multi-level approach. Table 1 summarizes the data sources for each strategy, the implementation outcomes each strategy was designed to effect, and the metrics used to operationalize these outcomes for analysis.

We began by extracting data from NYC HD implementation check-ins to document enactment of systems-level activities necessary to support the multi-level implementation strategy. We present summary data on barriers and facilitators to systems-level activities, as well as actions undertaken by the NYC HD to overcome barriers they identified.

Next, using the metrics outlined in Table 1, we examined the impact of each implementation strategy on implementation outcomes. Analyses focused on adoption and penetration were conducted by examining longitudinal changes in metrics by quarter, focusing on the median and interquartile range (IQR) across programs. Perceived acceptability, appropriateness, and feasibility were evaluated by descriptive analyses of quantitative evaluation data, as well as thematic analysis (Terry et al., 2017) of qualitative themes extracted from learning collaborative discussion notes and slides. We used repeated measures correlation (rmcorr; Bakdash & Marusich, 2017) to examine both the association between training adoption and GOALS implementation and the association between GOALS implementation and HIV prevention intervention adoption within programs over time. Finally, we conducted stratified analyses comparing intervention adoption rates among programs with different GOALS implementation patterns (e.g., “high” versus “low” adopters).

Data on programs’ adoption of GOALS training are presented in Figure 2. The number of funded staff in each program ranged from seven to fourteen (median = 11, IQR 10–12), and the number of unfunded staff deemed “appropriate” for GOALS training ranged from 19 to 362 (median = 48, IQR 34-71). Across all programs, the percent of funded staff who completed GOALS training increased from a median of 67% in September 2022 (range 0-89%; IQR 25%-3%) to a median of 80% in October 2023 (range 67%-100%; IQR 75%-89 %). It is important to note that while the median increased by only 13 percentage points, the range and IQR narrowed substantially, reflecting an overall increase in training of funded staff across programs. We examined penetration by assessing program reports of training rates for all “appropriate” staff as of September 2023. Programs reported a median training rate for all staff of 76% (range 8%-100%, IQR 25% to 96%). One of the persistent challenges in maintaining high staff training percentages training was staff turnover; fluctuations in percentages reflect programs’ continual hiring of new staff that then needed to integrate the GOALS Approach training into their onboarding.

All programs demonstrated high-levels of learning collaborative adoption; 100% of programs prepared slides for both of their assigned sessions, and 95% of programs (18/19) attended both of their assigned sessions. One program missed one of its sessions, but submitted slides.

We examined staff evaluations of the GOALS Approach training to assess its impact on perceptions of acceptability, appropriateness, and feasibility of the GOALS Approach. As depicted in Table 2, staff had very positive reactions to the training objectives, presentation, and lead trainer. Over two-thirds of staff (77%; n = 230) reported that their knowledge or skills had increased after the training, and 42% (n = 126) reported that their knowledge or skills had increased by two or more points. Staff also indicated that they were likely to use the knowledge and skills from the training in their practice. In the qualitative comment field, staff emphasized the impact of the training on their language (“It will help me interact with [clients]in a non- stigmatizing but empowering way”; “I will engage with clients in the language that best allows for them to feel safe, open and valued”), their degree of comfort engaging in sexual history conversations (‘I will feel more comfortable having a conversation regarding sexual health”; “I am better prepared to ask sexual health questions”) and their work as a whole (“I now have more knowledge to offer within my position”; “The knowledge will improve my case management abilities”).

During the learning collaboratives, several key themes emerged in programs’ descriptions of barriers and facilitators to implementation of the GOALS Approach. Among programs who reported barriers, one of the most consistent themes was resistance to the training mandate. Programs who struggled with staff training reported that the time required was a “heavy lift” because it took time away from direct client care. In response to this barrier, the NYC HD provided a series of one-hour, in-person GOALS Approach training sessions to specific agencies in Q4 (June-August 2023). Programs also questioned the rationale behind GOALS Approach training for all funded staff, which applied even to those who did not provide HIV prevention interventions to clients (e.g., front-desk staff, data manager). And larger programs reported that low levels of buy-in from upper management (outside the funded program team) negatively impacted their ability to enforce GOALS Approach training requirements.

With regards to facilitators, programs who reported greater ease in enacting GOALS Approach with their staff reported strong buy-in from leadership as a critical facilitator and described the GOALS Approach as “in alignment” with their program mission. Other successful programs reported utilizing a “multi-pronged approach,” in which program leads began by securing buy-in and facilitating training among certain groups of staff (e.g., HIV testing staff, peer navigators) and then continuing to others, focusing on sharing “the why before the how” when they presented the requirement. Another facilitator was an increasing recognition among staff of the ways in which the GOALS Approach complemented or facilitated other components of care provided at the agency; buy-in for training was increased when staff began to see its ability to “increase collaboration between multiple disciplines.” Finally, many programs reported being able to use the “great feedback” about the GOALS Approach from clients to motivate their staff and increase training rates.

Data on program-level adoption of GOALS Approach during client visits by implementation quarter are presented in Figure 3. Across all programs, GOALS Approach implementation increased from a median of 68% of visits (range: 0-100%; IQR: 42%-84%) in September 2022 to a median of 89% of visits (range: 33%-100%; IQR: 75%-100%) in October 2023. We conducted repeated measures correlation (rmcorr) to examine the common within-program association between the percent of funded staff who received GOALS Training and the percent of visits at which the GOALS Approach was utilized by quarter. There was a significant repeated measures correlation between GOALS training and GOALS utilization, r_rm(71) = 0.34, 95% CI [0.11 – 0.53], p < .01.

Figure 4 presents median rates of GOALS utilization and offers of HIV testing, STI testing, and PrEP by quarter. Across all programs, the percent of visits during which HIV testing was offered increased from a median of 75% of visits (range: 0%-100%; IQR: 45%-99%) in September 2022 to a median of 91% of visits (range: 33%-100%; IQR: 78%-100%) in October 2023. The percent of visits during which STI testing was offered increased from a median of 72% of visits (range: 0%-100%; IQR: 31%-94%) in September 2022 to a median of 90% of visits (range: 33%-100%; IQR: 77%-100%) in October 2023. And the percent of visits during which PrEP was offered increased from a median of 63% of visits (range: 0%-100%; IQR: 32%-78%) in September 2023 to a median of 79% of visits (range: 0%-100%; IQR: 64%-91%) in October 2023. There was a significant repeated measures correlation between GOALS utilization and offers of HIV testing, r_rm(71) = 0.49, 95% CI [0.29 – 0.64], p < .001, between GOALS utilization and offers of STI testing r_rm(71) = 0.49, 95% CI [0.28 – 0.64], p < .001, and between GOALS utilization and offers of PrEP, r_rm(71) = 0.45, 95% CI [0.25 – 0.62], p < .001.

We then focused on quarters three through five, i.e., six months after the GOALS Approach training had become available, and categorized programs by their GOALS Approach implementation patterns. Programs were classified as “high” GOALS Approach implementers (n = 5) if their rate of the GOALS Approach implementation was above the median for all three quarters; programs were classified as “low” GOALS Approach implementers (n = 5) if their rate of GOALS Approach implementation was at or below the median for all three quarters. Programs were classified as “upward” GOALS Approach implementers (n = 4) if their rate of GOALS implementation started at or below the median in Q3 but increased to above the median by Q5. And programs were classified as “variable” GOALS Approach implementers (n = 4) if their rate of GOALS Approach implementation was above the median in some quarters and at or below the median in others.

Figure 5 presents rates of intervention adoption (provider offers of HIV testing, STI testing, and PrEP to clients) by GOALS Approach utilization patterns. In terms of HIV testing, high GOALS Approach implementers had the highest rates of intervention utilization, and low GOALS Approach implementers had the lowest rates. Programs with an upward trend of GOALS Approach implementation demonstrated an increase in HIV testing over time, as did programs with variable GOALS Approach implementation. A similar pattern was found for STI testing, although rates of intervention utilization among high GOALS Approach implementers remained much higher than the other three groups at each time point. In terms of PrEP, rates of intervention utilization increased sharply for the upward GOALS Approach implementer group, and decreased for the high GOALS Approach implementer group.

Finally, we reviewed data collected during the learning collaborative sessions to better understand the different implementation barriers and facilitators that distinguished among these four groups. High GOALS Approach implementer programs tended to have experience with the GOALS Approach prior to receiving NYC HD funding and had extremely high buy-in for the GOALS Approach among program leadership. In contrast, low GOALS Approach implementer programs expressed the greatest resistance to GOALS Approach training during the learning collaboratives, either because of concerns about time taken away from clinical care or lack of support from upper leadership. All five programs described competing requirements from other funding sources that made integration of the GOALS Approach into care more challenging. All four of the upward implementers reported initial challenges in the early learning collaboratives, but devised strategies to overcome them, primarily by increasing buy-in among both staff and leadership, and by using a more phased approach. All four of the variable GOALS Approach implementers were smaller agencies; three had not been previously funded by NYC HD for HIV prevention services and had the least experience in the delivery of sexual health care. And two of the variable GOALS Approach implementers provide services in multiple languages and needed to translate some of the GOALS Approach language recommendations for their clients.

It is important to note several unique aspects of Project PACE that facilitated the data collection and analysis presented in this paper. First, the implementation science research was developed at the same time as, and in concert with, the operationalization of the NYC HD strategies. This real-time collaboration allowed for the development of research methods that maximized the data available for analysis while prioritizing data collection strategies that were feasible and useful for implementation. For example, the question prompts given to programs during learning collaborative sessions were designed both to help the NYC HD support program implementation, and also to answer specific implementation science questions about barriers and facilitators. Second, PACE builds upon a decade-long partnership between NYC HD and HART, characterized by trust in the academic partners’ commitment not only to implementation research, but to implementation science, defined as the “application and integration of research evidence into practice and policy”(Glasgow et al., 2013). Project PACE is funded to provide regular data and feedback to the NYC HD, and NYC HD staff are actively involved in dissemination of research findings. As a result, PACE researchers were given access to all aspects of implementation data almost in real time, and were able to triangulate across data sources in this paper to better understand implementation and outcomes within and across systems-, program-, and provider-levels. These preliminary findings demonstrate the importance of robust and enduring partnerships between academic researchers and service providers for success of health system innovations and improvement of patient care and outcomes in HIV prevention.

There are several limitations to our analysis inherent in our implementation science study design. This preliminary analysis suggests that the implementation strategies impacted implementation outcomes, but precludes our ability to make a definitive causal claim. Our measurement of GOALS adoption relied on provider and program self-report, we did not include direct observation of providers to ensure fidelity to the approach. As noted above, providers were asked to indicate each client’s core sexual health concern (as a proxy for GOALS approach enactment), but it is possible that providers asked this question without using client-centered and affirming language. Future Project PACE analyses will focus on client-level survey data and provider interviews to better assess GOALS Approach fidelity.

It is important to note that this implementation strategy was being enacted in New York City, among agencies with high rates of HIV prevention intervention adoption at baseline. In addition, many programs were already using the GOALS Approach or had been exposed to it prior to the start of these funding contracts, because of its dissemination by the NYC HD and the New York State AIDS Institute, beginning in 2020. It is possible that the successes demonstrated by this multi-level implementation strategy would not translate to a different context. However, there was significant variability among programs in their levels of intervention adoption and implementation strategy utilization at baseline, and the most significant gains in intervention adoption were demonstrated among agencies with upward trajectories of GOALS Approach utilization.

Our analyses are based on data reported by agencies to the NYC HD and may be subject to error or bias. Programs are motivated to report successful implementation at both program and provider-levels, because of the systems-level mandates imposed by the NYC HD. To complement these data reported by programs, the NYC HD and Project PACE are also collecting client-level survey data, in which clients report on their perception of program staff’s comfort taking a sexual history, whether or not program staff used positive, non-stigmatizing language when discussing sexual health, and whether or not staff offered them specific HIV prevention interventions. These data were not yet available for the present paper, but future analyses conducted by Project PACE will incorporate client-level reports to examine the impact of the implementation strategy on the client experience and to validate program-level reports.

This analysis is limited in scope, presenting descriptive data from the first 18 months of project implementation. This initial analysis focused on the offer of HIV testing, STI testing, and PrEP as implementation outcomes, signifying greater provider adoption of evidence-based interventions in their practice. Future research must examine the impact of provider behavior on client-level outcomes, i.e., client-level uptake and sustainment of HIV prevention interventions as well larger-scale impacts on HIV incidence. While outside the purview of this pilot analysis, the five-year Project PACE study is funded to conduct interrupted time-series analyses to investigate these impacts. Project PACE is also conducting in-depth qualitative interviews with providers, program-level staff, and NYC HD staff about the implementation process, and these data will inform future mixed-methods analyses that can shed greater light on implementation barriers and facilitators, and their relationship to long-term outcomes.

Despite these limitations, our findings highlight the potential of a multi-level implementation strategy to have significant, positive impacts on adoption of HIV prevention interventions across a diverse set of programs in a jurisdiction with high HIV incidence. In addition, this analysis underscores the potential for community-academic partnerships to produce comprehensive, rigorous, and complex analyses of implementation strategies in real-world practice settings, in real time, highlighting the essential role of scientifically informed strategies in addressing public health challenges. We are hopeful that Project PACE will inspire development of similar partnerships to accelerate innovation and increase utilization of effective interventions that can have a sustained and positive impact on ending the HIV epidemic.