The human immunodeficiency virus (HIV) continues to affect African American/Black (hereafter referred to as Black) and Hispanic/Latino (hereafter referred to as Hispanic) populations in the United States at disproportionate rates in comparison to their White counterparts, particularly among sexual minority populations—gay, bisexual men, and other men who have sex with men (MSM) and transgender populations (CDC, 2019a,b,c, 2020a,b; Clark, Babu, Wiewel, Opoku, & Crepaz, 2017). Both Black and Hispanic MSM and transgender populations experience disparities in HIV-related outcomes along the HIV care continuum (HCC) (i.e., HIV diagnosis, linkage to care, retention in care, and viral suppression) (Beer, Oster, Mattson, & Skarbinski, 2014; Kalichman, Hernandez, Finneran, Price, & Driver, 2017; Levison, Levinson, & Alegría, 2018; McCree et al., 2019). Data also indicate that PrEP uptake among Black and Hispanic MSM and transgender persons is low relative to their White counterparts (Cahill et al., 2017; Eaton et al., 2017; Elopre, Kudroff, Westfall, Overton, & Mugavero, 2017; Kamitani et al., 2018; Poteat et al., 2019). Increasing PrEP uptake among those disproportionately affected by HIV could reduce disparities in HIV infection. Additionally, addressing social determinants of health (SDH) that can impede uptake of PrEP and other prevention and care services should be considered and evaluated to determine their effect on disparate HIV-related outcomes.

Social determinants of health result from the complex integration of overlapping social and economic systems and structures that are influenced by the distribution of resources, money, and policies (CDC, 2010). In the context of HIV, there are an array of SDH that adversely affect HIV-related outcomes especially for Black and Hispanic MSM and transgender populations. In addition to experiencing SDH such as unstable housing or lack of/insufficient health insurance, racial and ethnic minority MSM and transgender persons also deal with stigma and discrimination based on racial/ethnic and minority sexual and gender identities (Arnold, Rebchook, & Kegeles, 2014; Bowleg, 2012; Choi, Paul, Ayala, Boylan, & Gregorich, 2013; Frye et al., 2015; Quinn & Dickson-Gomez, 2016; Safer et al., 2016). Evidence suggests that social and structural determinants sustain and exacerbate disparities in HIV, preventing achievement in health equity for delivery of HIV prevention and care services (Millett, Flores, Peterson, & Bakeman, 2007, 2012; Yehia et al., 2015). Adopting health equity frameworks in our programmatic and evaluative activities may facilitate reduction of disparities for racial, ethnic, and sexual minorities (Braveman, 2003).

Health equity occurs when everyone has a fair opportunity to achieve their full health potential and includes the avoidance of unfair differences in health outcomes and access to healthcare among groups of people (World Health Organization, 2008). Given the pronounced disparities in HIV among Black and Hispanic MSM and transgender populations, we are far from attaining the goal of health equity in HIV. In the United States, it is well-documented that Black and Hispanic MSM and transgender persons encounter racial, sexual, and gender identity discrimination; HIV-related stigma; negative implicit biases of HCWs; as well as differential treatment due to structural barriers within civic institutions with longstanding histories of unfair, unjust, and unequal treatments (Aidala et al., 2016; Arnold et al., 2014; Blair et al., 2013; Eaton et al., 2015; Geter, Herron, & Sutton, 2018; Wohl et al., 2013). Some of these structural barriers include, but are not limited to, institutional racism, HIV criminalization laws, being uninsured or underinsured, and medical distrust (Bogart et al., 2016; Cahill et al., 2017; Feagin & Bennefield, 2014; Lehman et al., 2014; Patel et al., 2017; Quinn et al., 2018; Yehia et al., 2014). These structural level barriers perpetuate the disparities in HIV, and, as result, become challenges health departments must overcome to satisfactorily achieve desired HIV outcomes for these minority communities.

From 2015–2019, the Centers for Disease Control and Prevention (CDC) funded the PrEP Implementation, Data to Care, and Evaluation (PrIDE) demonstration project to support health departments with implementing PrEP and data-to-care (D2C) strategies prioritizing service provision for Black and Hispanic MSM and transgender persons. All participating health departments (hereinafter called funded recipients) incorporated health equity activities into their strategies for increasing PrEP knowledge, awareness, and uptake among the project’s priority populations. As a major component of the project, local site-specific evaluations (LE) allowed the funded recipients to conduct rigorous evaluations on various project strategies. These evaluations adhered to the steps and standards of the CDC Framework for Program Evaluation in Public Health (Milstein & Wetterhall, 1999). Of the 12 funded recipients, three recipients evaluated SDH-related strategies to continue making progress towards achieving health equity for the priority populations. This paper serves as an introductory paper that highlights overarching cluster evaluation findings across the three funded recipients that evaluated health equity activities in Project PrIDE. The objectives of this paper are to (a) describe the cluster evaluation approach used in Project PrIDE; (b) highlight overarching findings across three funded recipients that evaluated health equity strategies, lessons learned to inform the integration of health equity strategies in the provision of PrEP services and uptake; and (d) introduce individual PrIDE local evaluations on health equity strategies that will appear in this special issue.

In addition to the LE conducted by funded recipients, Project PrIDE used a cluster evaluation approach (Millet, 1995; Sanders, 1997). Cluster evaluation is an approach that looks across a group of programs/projects that share similarities and identifies common high-level themes that if confirmed across can have great significance (Bitar, Hbeichi, Al-Zou’bi, & Russon, 2015). This approach maximized opportunities for cross-validation among LE. Conducting the cluster evaluation was a collaborative process between funded recipients and CDC that occurred between 2017 and 2019. As part of this process, funded recipients identified questions to be addressed in evaluating their PrIDE strategies/activities and independently developed data collection instruments. Of these questions, across all the local evaluations, CDC identified those that were similar across project strategies. That is, all similar questions revolved around one of five project strategies (i.e., community engagement, provider capacity building, health equity, navigation, and media). Other questions instrumental to understanding the implementation of strategies and the evaluation process (e.g., What were lessons learned?) were also identified by both funded recipients and CDC.

CDC shared these grouped questions with the funded recipients, and in turn, the recipients reviewed and provided feedback regarding the cluster evaluation process (including evaluation questions, data collection and reporting, and a data use plan). Specifically, over a seven-month time period, CDC and funded recipients participated in monthly conference calls on these thematic clusters to share progress and findings.

In cluster evaluation, primary data collection methods such as interviews with evaluation staff and documentation of site visits tend to be used as data sources for analysis (Barley & Jenness, 1993; Bitar, Hbeichi, Al-Zou’bi, & Russon, 2014; Sanders, 2013); however, to prevent data burden for funded recipients, project documents became the main data source as secondary data can be used to answer evaluation questions (i. e., LE protocol, a preliminary LE report, final LE report, 2018 and 2019 annual progress reports, notes from site visits, monthly cluster calls, and annual project meeting presentations).

Document review analysis, a method used in evaluation, depends on the quality of the documents and how the documents assist with answering evaluation questions (Bowen Glenn, 2009; Caulley, 1983; Gross, 2018). As such, each LE protocol, LE preliminary report and final reports were assessed at CDC by a multidisciplinary review team against all quality evaluation standards (i.e., utility, feasibility, propriety, and accuracy) (Milstein & Wetterhall, 1999; Yarbrough, Shulha, Hopson, & Caruthers, 2011), feedback was provided, and these documents were not approved until all standards were met. In addition, the preliminary LE report was submitted to CDC to ensure that local evaluations were answering the evaluation questions, measuring progress, and pointing out needed revisions (e.g., gather more data, conduct a different analysis). This process was applied to the final LE report, which captured contextual factors that affected the project and LE (e.g., state and local policies affecting racial/ethnic, sexual and gender minorities, PrEP implementation maturity), facilitators and challenges of each evaluation stage (i.e., protocol development, implementation, analysis, utilization and dissemination) and findings per question, a separate section included the cluster evaluation questions, and a utilization plan. Additionally, specific questions regarding LE and cluster questions were shared in preparation for annual site visits and were discussed accordingly. Presentations at monthly cluster calls and annual project meetings also aided in this process.

Two CDC reviewers abstracted information from the cluster evaluation questions (see Table 1) and used a framework method with a simplified qualitative content analysis for identifying themes (Gale, Heath, Cameron, Rashid, & Redwood, 2013). This was an interactive process that also involved discussions to reach consensus about themes and conclusions and conferring with funded recipients as needed.

Of the 12 PrIDE funded recipients, the cluster approach identified three funded recipients that evaluated health equity-related strategies (i. e., Baltimore City Health Department, Louisiana Department of Health, and New York City Department of Health and Mental Hygiene; hereinafter referred to by the state or city they represent). These three Project PrIDE funded recipients addressed common overlapping barriers (e.g., medical distrust, discrimination due to race, gender, or sexuality), but the context in which these barriers affected HIV outcomes for the priority populations varied across the three. For instance, Baltimore City addressed historical and current medical distrust of local medical and public health establishments that exists among members of priority populations and negative implicit biases toward priority populations by some HCWs that affect the care of these patients. Louisiana and New York City addressed barriers with emphasis on minority racial, sexual, or gender identities. Louisiana recognized the importance of addressing institutional racism, homophobia, and transphobia within their local jurisdiction at the organizational level; whereas, New York City addressed structural barriers their transgender and gender non-conforming (TGNC) clients experience when accessing sexual health services.

Recipients conducted different outcome evaluations in the health equity cluster (e.g., utilizing pre-post methodology to evaluate change). Baltimore’s mixed-methods outcome evaluation focused on the use of a storytelling approach, Baltimore in Conversation, to reduce medical distrust and increase empathy among HCWs toward priority populations, and to improve HCWs’ understanding, beliefs, and practices related to racial/ethnic, sexual, and gender minority patients. Baltimore conducted five storytelling nights where HCWs (n = 55) participated in the storytelling sessions to enhance their understanding of the barriers clients encounter in seeking HIV prevention and care services. Louisiana conducted pre- and post-outcome evaluations of at least 10 trainings that addressed institutional racism, homophobia, and transphobia among its health department staff and collaborating partners (n = 242). Lastly, New York City conducted an outcome evaluation of a training for sexual health clinic (formerly known as Sexually Transmitted Diseases Clinics) staff to determine self-perceptions in providing culturally affirming care. Culturally affirming care occurs when clients receive appropriate services within their cultural context by having health care providers that are aware of and familiar with clients’ social context and culture, including sexual, gender, and racial identities (Furness et al., 2020; Pitts & Greene, 2020; Schilder et al., 2001).

Since Project PrIDE was funded by a programmatic, rather than a research award, the project did not utilize experimental or quasi-experimental designs. Table 1 summarizes the health equity strategies evaluated and audiences, and the cluster evaluation questions funded recipients addressed collectively. Additional information regarding the specific methodology, outcome measures used, and evaluation findings are detailed elsewhere (This Issue).

All the funded recipients targeted the same audience (i.e., providers) with their efforts. Two of three (Louisiana and New York City) used training as their strategy. These similarities allowed for a cluster evaluation and provided an opportunity to learn about groundbreaking strategies such as storytelling in Baltimore, a training policy in Louisiana’s health department, and a culturally affirming care training in New York City. Details on the health equity local evaluations are provided after this article in this special issue.

The health equity cluster provided an opportunity to use findings based on cross-cutting evaluation questions to assess and improve implemented strategies. Overall, the findings from the health equity cluster indicate that all recipients demonstrated progress towards their respective health-equity related outcomes (see Fields and Robinson in this special issue). For instance, post-event surveys of health care workers (HCWs) who participated in the storytelling events in Baltimore indicated an improvement in their empathy and understanding of their local priority populations’ experiences of stigma due to racial, gender, and sexual identities. In addition, qualitative analysis of HCWs written feedback/notes used while individuals shared their stories indicated HCWs learned new approaches for working with sexual and gender minority patients through the shared stories. This strategy for sharing experiential realities may even help to decrease medical distrust experienced by sexual and gender minority patients by improving patient-HCW relationship dynamics, resulting in, fewer barriers to HIV prevention and treatment.

The provision of staff trainings was successful at improving culturally affirming services and care in Louisiana and New York City. After undergoing the Undoing Racism^® workshop and trainings that address institutional racism and other discriminatory practices such as homophobia and transphobia (People’s Institute for Survival and Beyond, 2018), Louisiana saw significant increases in staff and community partners’ knowledge, motivation, and perceived behavioral skills needed to address these discriminatory practices in their jurisdiction. New York City observed positive perspectives among clients of sexual health clinics regarding receipt of culturally appropriate care. Additionally, New York City observed an increase in utilization of their sexual health clinics by both Black and Hispanic MSM and transgender clients following changes implemented to improve the cultural appropriateness of services provided. Some enhancements to improve culture appropriateness of services included role-based, skills-building, cultural sensitivity trainings for clinic staff, creating a welcoming environment, development of and regularly convening a transgender community advisory board, and implementing policies and procedures that better capture pertinent health data for these clients.

The findings from the three funded recipients demonstrate the value of multiple approaches to changing beliefs and perspectives of HCWs. These examples highlight both storytelling and direct HCW training that address social and structural barriers as promising strategies that can assist with reducing HIV-related disparities for priority populations.

The health equity cluster provided valuable lessons. One of the major lessons learned was that each funded recipient demonstrated the feasibility of evaluating innovative health equity strategies and organizational-level policies within the scope and timeframe of project PrIDE with technical assistance and support from CDC. Funded recipients routinely collect performance monitoring data; however, the LEs included in the health equity cluster showed that more rigorous evaluations can be conducted. Moreover, when these strategies respond to local needs, it enhances the utility of findings. Although funded recipients noted challenges clients encountered due to social and structural barriers (e.g., medical distrust, institutional racism), these evaluations emphasized the importance of serving clients holistically, rather than only focusing, for example, on PrEP uptake specifically. While project PrIDE was specific to HIV prevention and care activities (e. g., PrEP), the interventions evaluated illuminated the need to shift the paradigm from activities that are heavily focused on individuals to those that incorporate and evaluate the impact of organizations, systems, or policies on HIV-related outcomes; findings consistent with recent literature on addressing disparities among priority populations (Carter & Flores, 2019).

By conducting the cluster evaluation approach, we noted unexpected findings from LE. For instance, Baltimore City identified mental health services as an additional need for its priority populations. Although this is not a novel finding for the field, the prioritization of this local public health area of need might not have been discovered if not for the evaluation work performed. Identification of additional needs can sometimes be enhanced by combining intersectionality and health equity frameworks. Intersectionality frameworks—frameworks that assist with understanding how the combined effect of possessing multiple socially constructed identities (e.g., race, class, gender, etc.) shape an individual’s experience through the lens of privilege and oppression—can be combined with health equity approaches to better identify barriers and inform holistic application of strategies so health departments can better serve priority populations (Bowleg, 2012; Crenshaw, 1989; Weber, 2010). For instance, the priority populations for project PrIDE possess multiple minority identities (i.e., race, gender, and sexuality) and intersectionality frameworks can assist health department staff better identify and address complex barriers their clients may experience due to possessing more than one minority identity. Instead of focusing on one identity that presents barriers (e.g., race), HDs can integrate intersectionality and health equity frameworks to not only identify barriers to HIV prevention and care services, but also adequately address these barriers in the context of how their clients experience them due to the intersection of their racial, sexual, and gender identities (Corus & Saatcioglu, 2015).

Lastly, another major lesson from the three jurisdictions that constituted the health equity cluster was the value of conducting participatory evaluations (Garaway, 1995), which engages key stakeholders throughout the evaluation process (Chouinard, 2013). For instance, priority population members were critical to identifying areas of focus for the health equity strategies evaluated and assisting with the development of LE protocols, data collection tools, and data use plans particularly among priority population members. Furthermore, priority populations were engaged as member of consumer advisory boards that provided input and feedback throughout the evaluation process. For instance, priority population members participated in the storytelling events in Baltimore, where they shared their experiences in seeking HIV prevention and care services and the barriers encountered. Having these partners engaged in ongoing discussions about and data from the program evaluation activities may have been particularly relevant in developing programs conducted in this health equity cluster, yielding more useful results for all stakeholders involved.

Given the local success of the evaluation findings, the funded recipients provided insight on good practices to employ when conducting health equity-focused evaluations. Health departments and other public health entities can incorporate these practices when conducting evaluations that seek to improve health equity-related outcomes. First, engagement of the priority population(s) is paramount as these are critical stakeholders. The health equity cluster recipients found that collaborative and participatory approaches with priority populations not only strengthened the trust among priority population groups and HCWs, but in some cases helped to improve patient-HCW dynamics. Another good practice identified was to integrate programmatic and evaluation staff in the conduct of evaluation activities. Funded recipients found that ongoing collaboration with program and evaluation staff enhanced team cohesiveness, trust, and understanding of the evaluative tasks. Although this good practice is often common when conducting evaluations, the experience of the project funded recipients indicated that this practice is necessary, particularly when evaluating health-equity-specific strategies that impact priority populations.

Funded recipients also identified leadership buy-in in their health departments as an important component to help facilitate the evaluation objectives and the assurance that dedicated staff and resources will be available to conduct more rigorous evaluation activities. One example of leadership buy-in was the establishment of agency-wide health equity policies in Louisiana that shifted how health department staff and their community partners engaged in and provided services to priority populations. Another example of leadership buy-in was demonstrated in the sexual health clinics in NYC where the clinic leadership initiated processes to evaluate their standard of care to their sexual minority clients resulting in conducting a health-equity focused evaluation to improve upon that standard of care.

In addition to leadership buy-in, the two funded recipients that evaluated health equity relevant trainings in this cluster (i.e., New York City, Louisiana) identified on-going staff training via technical assistance and staff capacity building as important components to successfully improve equity in services and improve evaluation outcomes. The premise was that one training was not enough to provide culturally appropriate care and services. For example, funded recipients provided the initial trainings and offered booster or refresher trainings to address areas that training participants did not understand or had challenges implementing in the field. Ongoing trainings that address health equity are critical to achieve lasting favorable outcomes.

Lastly, there are opportunities (or local evaluations were opportunities) to use evaluation findings to highlight inequities for certain populations and incorporate health equity activities into all health department HIV programmatic efforts. Given that SDH are pervasive and contribute to unfavorable HIV outcomes for the priority populations, collection and use of SDH data are good practices to generate evidence regarding barriers, gaps, and areas of opportunity and improvement in the provision of HIV prevention and care services.

The following manuscripts represent two local health equity-related evaluations conducted in Project PrIDE by funded recipients in Baltimore City, where storytelling was implemented, and Louisiana where trainings about undoing racism, homophobia and transphobia took place among health department staff. Each manuscript provides detailed background information about HIV among racial/ethnic minorities and among MSM and transgender populations in their jurisdictions, the evaluation methodology utilized, findings, conclusions and lessons learned.