Among the 33 studies we examined, 11 (33%) were conducted before ART or prior to 1996 while 22 studies (67%) were conducted after ART. The majority (82%) of pre-ART studies were conducted in small group settings while the remaining studies were implemented as community-level interventions (18%). No individual-level interventions for MSM were conducted in this time period. In the post-ART period, half of the interventions were conducted as individual-level interventions while the other half were delivered in small groups. No post-ART studies were conducted as community-level interventions.

The majority of studies (n=26, 79%) were single-site studies, while 4 (12%) were multi-site studies. The 3 remaining studies were conducted nationwide using the internet [39, 40] and telephone [41]. The most commonly reported intervention sites were urban settings where higher proportions of MSM congregate: New York City, San Francisco, Los Angeles, and Seattle. Regionally, most interventions were conducted in the West, followed by the Northeast. The fewest interventions were conducted in the South.

In terms of MSM subpopulations, 6 studies (18%) exclusively focused on MSM of color and 7 studies (21%) intervened with substance-using MSM. Nine (27%) studies concentrated on HIV-positive MSM and 6 (18%) targeted HIV-negative MSM; the majority of studies (n=18, 55%) did not include HIV status as an eligibility criterion. A higher percent of MSM studies conducted post-ART focused on high-risk MSM when compared to the pre-ART period: 36% vs. 9% focused on HIV-positive MSM and 27% vs. 9% focused on substance-using MSM respectively.

Approximately 42% of the studies used wait-list groups as comparisons (n=14) while 11 studies (33%) used HIV-related comparison groups and eight (24%) used non-HIV related comparison groups. The majority of interventions were conducted in small groups (n=20, 61%). Eleven (33%) were individual- level interventions while two (6%) were community-level interventions. No interventions focused on couples. Nine (27%) studies reported power analysis results [42–50] (not shown in a table).

Seven studies that met all PRS efficacy criteria with the exception of finding a significant positive effect were classified as rigorous non-EBIs. All rigorous non-EBIs were conducted post-ART. When compared to EBIs, rigorous non-EBIs were significantly more likely to include substance use as an eligibility criterion and report study participants with a high school education or less. Also, rigorous non-EBIs were significantly less likely to be conducted in MSM settings and include sexual communication as part of the intervention in contrast to EBIs (Table 2).

Although these 11 studies found a significant positive effect, they did not meet other efficacy criteria. Having only an immediate follow-up assessment (n=4, 36%) [41, 55, 56, 62] was the most common reason for not meeting efficacy criteria. Other common reasons included having analytic sample sizes less than 40 per arm (n=2, 18%) [48, 55] and having less than 60% retention in at least 1 study arm (n=2, 18%) [39, 57]. The remaining interventions had other study limitations such as reported contradictory findings, substantial missing data, significant negative findings or re-assigned participants from the intervention arm to the control arm or excluded participants who missed interventions sessions from the analyses (n=4, 36%) [40, 61, 67, 69]. As shown in Table 1, the internet- and telephone-based interventions were classified as positive non-EBIs. Although not statistically significant, positive non-EBIs reported the lowest median number of intervention sessions, lowest median percent of blacks, and highest median percent of whites in their samples compared to EBIs, rigorous, and other non EBIs. They also reported the lowest median percent of study participants with a high school education or less. When positive non-EBIs were compared to EBIs, no statistically significant differences in study characteristics, intervention content, or participant characteristics emerged.

Besides lacking a significant positive effect for a relevant behavioral or biological outcome, these 6 studies also did not meet at least one other efficacy criterion. For other non-EBIs, reasons for not meeting PRS criteria were having analytic sample sizes less than 40 per arm (n=3, 50%) [42, 64, 65], having follow-up assessments less than 1 month (n=2, 33%) [47, 66], having significant negative findings (n=2, 33%) [47, 64] and biased allocation of participants to intervention and comparison arms [59]. Other non-EBIs reported the highest percent of interventions conducted in groups (n=5, 83%) compared to EBIs, rigorous non-EBIs, and positive non-EBIs. Other non-EBIs were also significantly less likely to pilot test interventions compared to EBIs (Table 2).

When we combined rigorous and other non-EBIs (n=13) or studies that did not find a significant positive intervention effect and compared this group with EBIs (not shown), we found similar results from previous comparisons. For example, rigorous and other non-EBIs combined were significantly more likely to include substance use (33% vs. 0, p = 0.05) and exclusively focus on HIV negative men (44% vs. 0, p=0.02) as eligibility criteria compared to EBIs. In addition, the combined group of rigorous and other non-EBIs was significantly less likely to be pilot tested (8 vs. 67%, p =0.01) and include sexual communication as part of the intervention (31 vs. 89%, p = 0.01) in contrast to EBIs. Finally, rigorous and other non-EBIs combined were more likely to use a non-HIV related comparison group in contrast to EBIs (46% vs. 0, p = 0.05).

Findings from this study suggest that it may be difficult to demonstrate significant positive changes in sexual behavior with MSM who experience substance use issues. MSM with substance use issues often experience other psychosocial health problems like depression and partner violence [70]. From a “syndemics” perspective, it may be difficult for MSM who experience multiple interconnected issues such as substance use, depression, a history of sexual child abuse or partner violence to change sexual risk behaviors [46, 70, 71]. Efficacious interventions for substance-using MSM who experience multiple vulnerabilities may require a comprehensive approach that addresses substance use specifically (e.g., methadone treatment, individual therapy, contingency management) [72], accounts for various MSM populations (e.g., MSM who use injection drugs, MSM who use recreational drugs on the weekends), and targets venues combining sexual activity and substance use [71]. Addressing interconnected syndemic factors (e.g., treating depression, counseling for childhood sexual abuse or partner violence) while using biomedical strategies (e.g., PrEP and/or PEP) and structural interventions (e.g., decreasing stigma and discrimination related to being gay) should also be a part of this comprehensive approach[72, 73].

For substance-using and other high-risk MSM, it may be more efficacious to use individual-level interventions rather than group-level interventions. In our review, a larger proportion of non-EBIs (71%) used groups to deliver the intervention compared to EBIs (33%). Delivering interventions to high-risk MSM in a group setting may have some unintended outcomes. Some evidence indicates that small group formats may actually reinforce and sustain risk behaviors for high-risk MSM through meeting similarly or more-risky men [14]. An advantage of one-on-one interventions such as EXPLORE is that they can be tailored to meet a high-risk individual’s specific prevention needs [74]. However, to have maximum impact, it may be necessary to use a combination approach that includes other kinds of strategies such as biomedical or structural interventions to reinforce engaging in safer behaviors for high-risk MSM.

Another factor that may contribute to interventions not finding effects is the type of comparison group. In our review, interventions that did not find significant positive effects were more likely to use a non-HIV comparison group (i.e., served as attention control) while EBIs were more likely to use wait-list or HIV-related comparison groups. In addition, the number of MSM studies that used wait-list controls decreased over time while more recent studies were more likely to use HIV-related comparison groups as demand controls. These findings suggest that using non-HIV specific comparison groups that focus on general health promotion or stress reduction may encourage comparison participants to think about and possibly make changes in sexual health behaviors [39, 68]. However, using wait-list and HIV-related comparison groups also have their disadvantages. Some researchers argue for using control groups with minimal or no treatment when conducting rigorous intervention evaluations [58], but the problem with this approach is that it does not control for attention or demand effects. Also, temporarily withholding interventions from individuals and communities severely impacted by HIV may be perceived as unethical. A potential disadvantage of using HIV-related comparison groups is the use of enhanced or exceptional HIV prevention programs or diluted versions of the new intervention as comparison groups that may greatly reduce the ability to detect effects [16]. Researchers should consider the advantages and disadvantages of using each type of comparison group and address issues related to controlling for attention and demand effects. Using a standardized comparison arm that the HIV prevention field could agree upon as a prevention standard for MSM can facilitate comparing intervention effects across studies.

One of the major reasons why interventions in our review did not meet efficacy criteria was not having at least 40 study participants per arm at follow-up assessment. Some of the interventions were conducted as small scale pilot studies to test feasibility and thus started with small sample sizes at baseline [42, 64]. Other interventions suffered from attrition problems [48, 65]. A prior power analysis can help to determine the adequate sample size for detecting an intervention effect but only 28% of the studies in our review reported power analysis results. More transparent reporting of power analyses would facilitate evaluation of evidence. Finally, recruiting and retaining MSM in HIV behavioral prevention studies, especially studies with a substance use focus, may be more challenging than in the past [75]. The HIV prevention field would greatly benefit from comparative research and a compendium of “best practices” specifically focusing on recruiting and retaining MSM in intervention studies [75]. Newer recruitment methods such as respondent-driven, venue-day-time, and internet sampling show promise [76, 77] but more research is needed to evaluate their strengths and limitations.

Having a follow-up assessment at least 1 month post-intervention was another common criterion that non-EBIs failed to meet in our review. These interventions only reported intervention effects immediately after the interventions were completed and thus the sustainability of the intervention effects is unclear. Given the emphasis on sustainable interventions in NHAS, HIV prevention researchers should consider longer follow-up assessments extending past immediate post- intervention to provide stronger evidence for sustainable changes in risk-reduction behavior.

A few intervention characteristics of EBIs that distinguished these interventions from those not meeting efficacy criteria might be helpful to consider when designing future behavioral interventions for MSM. For example, involving MSM in the design and implementation of interventions may be important. In our review, pilot testing and delivering the intervention through peers were more often characteristic of EBIs than non-EBIs. Pilot testing the intervention in the MSM community can provide information to improve the intervention. Using peers as intervention deliverers may help create an intervention environment that is safe and socially comfortable for MSM.

Researchers should also consider including intervention content on sexual communication, as it was associated with efficacious interventions. Other studies have also found sexual communication as an important component in behavioral interventions for MSM [11, 13]. Sexual negotiation and other communication skills such as HIV disclosure are critical if MSM rely on serostatus-based prevention strategies such as serosorting and strategic positioning to reduce their HIV risk. Innovative interventions are particularly important to help men with sexual negotiation and HIV disclosure skills in public and commercial sex environments where silence may be the norm [78]. As technological sites such as websites and phone apps become more popular to meet sexual partners among MSM, appropriate messages and strategies to enhance sexual communication will need to be developed with these new environments in mind.

There was an increase in the number of interventions conducted with high-risk MSM post-ART compared to pre-ART, but compared to the number of interventions PRS has evaluated, the number of interventions specifically developed for MSM of color is relatively few. Developing efficacious behavioral interventions for these highly impacted MSM subgroups, especially African American and Latino MSM is urgently needed. EBIs and non-EBIs reported similar median percentages for including black, Latino, and white MSM in their samples, but only 2 EBIs in our review exclusively focused on MSM of color [51, 53]. One was developed for Asian and Pacific Islander MSM and was conducted in the pre-ART era, while the other EBI is a group intervention designed for HIV-negative African Americans. Currently, there are no efficacious behavioral interventions that have been specifically developed for Latino MSM or those that exclusively focus on HIV-positive MSM of color. Researchers should consider addressing the cultural and contextual factors that have been associated with risk behavior with MSM of color [79, 80]. Acknowledging and addressing the health disparities as well as the economic and social inequities confronting MSM of color are also needed as a part of a comprehensive approach for reducing HIV in these groups.

In this review, the percentage of interventions that exclusively focused on HIV-positive MSM were not significantly different between EBIs and non-EBIs, but EBIs were significantly more likely than non-EBIs to exclusively focus on HIV-negative MSM. This finding suggests that it may be challenging to demonstrate significant positive effects with HIV-positive MSM compared to HIV-negative MSM. Several efficacious behavioral interventions have been developed for HIV-positive persons in general (http://www.cdc.gov/hiv/topics/research/prs/subset-best-evidence-interventions.htm#link2.6) and a large percentage of study participants in these studies were MSM. In our review, however, only 2 [50, 52] out of 9 interventions for HIV-positive MSM were efficacious. A factor that may contribute to non-efficacious behavioral interventions for HIV-positive MSM is the level of intervention. In this review, the majority of non-EBIs for HIV-positive MSM were group-level interventions. Intervening at the individual level reduces sexual risk for HIV-positive persons in general [81] and may be particularly important for HIV-positive MSM. As mentioned earlier, group-level interventions may inadvertently reinforce risky behavior for high-risk MSM and have fewer opportunities for specific tailoring that may be important for behavior change. Another factor that may explain why we found relatively few efficacious behavioral interventions for HIV-positive MSM is the intervention setting or where the intervention is delivered. Previous research shows behavioral interventions conducted in clinical settings where HIV-positive persons regularly receive medical or social services may be more effective in reducing sexual risk than non-clinical settings [81]. Future behavioral interventions to reduce sexual risk for HIV-positive MSM should consider individual-level interventions and delivering these interventions in settings where other services are also available.

HIV is greatly impacting MSM in the South [82], but we found the fewest interventions conducted there in our review. Because many MSM interventions have been developed and conducted in the West and Northeast, researchers should consider the cultural differences in the South (e.g., being located in the Bible belt) that can make HIV prevention more challenging for MSM and other vulnerable populations.

Our review also found that efficacious technology-based behavioral interventions have yet to be developed for MSM. Retention in these interventions may be challenging [39, 83] but behavioral interventions for MSM using digital media via the Internet, mobile phones, and computers have demonstrated the potential for increasing HIV testing [84], increasing HIV disclosure [85], and reaching high-risk MSM [86] and rural MSM [87].

Another gap in the MSM intervention literature is the lack of interventions for couples [88]. Some evidence indicates that most HIV transmissions occur within primary partner relationships [89] and, thus, developing couple-based behavioral interventions for MSM should be a priority. A recent pre-posttest design study that evaluated a couple-based intervention for methamphetamine-using black MSM found encouraging results for reducing sexual risk and illicit drug use [90].

Finally, early interventions that were evaluated as efficacious pre-ART should be replicated to determine if they are still efficacious post-ART [91]. It is important to recognize that interventions that worked 10 years ago may not be comparably effective today and that they should be continually assessed for effectiveness as they are implemented in practice [92].