Programs obtained site-specific workflow documentation for screening, brief intervention and referral procedures (e.g., when each service is conducted and by whom) using observation data and meeting notes to record SBI implementation processes; training attendance records to quantify number and roles of individuals trained; and de-identified electronic health record (EHR) data to measure documentation of SBI service delivery. Descriptive quantitative analyses were conducted using administrative and EHR data, provider surveys, and administrative reports to describe the population and outcomes.

Programs also provided supplemental data including: secondary data to contextualize findings across sites (n=2); environmental scans to describe SBI protocols prior to implementation (n=2); key informant interviews and focus groups of leadership and staff to contextualize data, describe SBI workflows and processes, and identify supports to promote efforts (n=3); site visits to document program fidelity (n=1); training observations to track process measures, provide quality improvement recommendations, and measure program fidelity (n=1); and, podcast metrics and Continuing Medical Education (CME) data to measure knowledge after listening to a project-developed podcast.

Provider-level data were collected via anonymous online staff surveys before and after SBI training (PPGNHAIK, PPSNE, BMC). Questions included skills, attitudes, and self-efficacy about SBI activities; training satisfaction; readiness to change practice behavior; facilitators and barriers to integration of SBI protocols and workflows; and FASD knowledge. Each site utilized questions tailored from validated SBIRT-focused surveys and/or developed custom items to correspond to the site activities. For example, PPSNE used questions derived from the Cutting Back SBI Program.¹⁷ A sample of questions for each site and topic area (including statistical findings) can be found in Online Supplemental Material 1. Unmatched survey data reflect participants who completed the pre- and post-training surveys. The Mann-Whitney U test (unmatched ordinal data) or the Wilcoxon Rank Sum test (matched data) were used with alpha set at 0.05.

Surveys for PPSNE and BMC were distributed via Qualtrics or paper surveys (pre-pandemic); PPGNHAIK’s surveys were distributed via the health center’s online learning management system. Pre- and post-training surveys were administered; unmatched samples measured changes in these factors and reported practice behaviors in participating providers. At BMC, 262 pre-training surveys were completed representing all 12 participating sites and 49 post-training surveys were completed 3–6 months post-training, representing five participating sites. At PPSNE, 140 pre-training surveys were completed representing all 15 health centers and 56 post-training surveys representing 12 centers were completed approximately 6 months after training. Data from PPGNHAIK comprised 198 pre-training surveys and 139 post-training surveys.

Qualitative data analysis allowed for triangulation of information from stakeholders through focus groups, key informant interviews, document review, and observations. Qualitative data was collected and analyzed by trained evaluators through a variety of strategies across sites including semi-structured interviews and focus groups. Site-specific Institutional Review Board approvals guided data collection with most interviews audio recorded and transcribed; in the absence of recording, detailed notes were taken. Semi-structured interview protocols were guided by implementation science and evaluation frameworks (e.g., the Consolidated Framework for Implementation Research).¹¹ The interview guides included questions related to protocols and workflows, internal and external contextual factors, patient interactions and provider self-efficacy, and barriers and facilitators to practice change. Convenience sampling was used to identify stakeholders who had relevant perspectives. Interview data were analyzed using thematic analysis.¹⁸ Content was coded by topic area and analyzed in Excel or NVIVO. Common themes that emerged across all programs, and site-specific themes uncovered during meetings and discussion, were included in the cross-site evaluation.

Although variation across programs in their implementation of alcohol SBI services existed, similarities emerged in targeting risk factors (alcohol and other substances) and services performed. All programs implemented evidence-based alcohol and drug SBI protocols, including prescreens (USAUDIT-C,¹⁹ Single Item Screening Questions for Drug²⁰ and Alcohol²¹ use), full screens (USAUDIT,¹⁹ WHO AUDIT,²² DAST-10,²³ ASSIST-FC²⁴), brief interventions (motivational interviewing style, brief negotiated interview,²⁵ FRAMES²⁶) and EHR-integrated referral procedures to local substance use disorder treatment services.

All programs provided preventive, primary, reproductive and gender-affirming care in diverse, underserved communities but with varying workflows and types of providers involved in care. At three of the four programs, clinicians conducted brief intervention following patient self-administered screening and electronic scoring or scoring by medical assistants. In one program (HFH), an interactive computer-based SBI program was used, supported by EHR best practice alerts for clinicians interacting with patients with moderate- or high-risk alcohol consumption.

Health systems with a centralized management structure (PPGNHAIK, PPSNE, HFH) were associated with a higher level of receptivity to SBI implementation and greater initial leadership involvement. Health centers independently managed within the system (BMC) experienced varying levels of receptivity in community health centers, necessitating further engagement strategies for buy-in.

Training format and content was similar across programs, before and during the pandemic. Pre-pandemic, most programs included in-person presentation-style education incorporating skills-building, practice, role-play, and video demonstrations, with a focus on preventing AEPs and FASDs. During the pandemic, all programs transitioned to live video synchronous training with three programs shifting to web-based asynchronous training. The fourth program provided support and continued technical assistance via virtual consultation. In three programs, training content was formalized and incorporated into health systems’ learning management system for sustainability and delivered using technology-based engagement strategies with some variation in systemization of adaptations due to health center disruptions. For example, when a state of emergency was declared, in-person experiences were limited to emergency- or essential-workers only, clinics temporarily shut down or closed, and shifts in protocols and workflows followed. Other disruptions included clinics adapting to technology-based care using telehealth, staff shortages, staff reallocation, and deployment of trained staff to other functions.

Across all programs, 1,259 individuals were trained in alcohol SBI service delivery representing: clinicians (physicians, physician assistants, nurse practitioners, certified nurse-midwives, and behavioral health clinicians; n=340); licensed and registered nurses and medical assistants (n=403); administrative and front desk staff (n=191); non-clinical leadership, including health system and/or project staff trainers (n=84); and other types of providers (n=5) (Table 3). These numbers represent unique individuals trained even though a majority attended multiple trainings (e.g., tailored follow-up training). Technical assistance was provided to health centers and providers based on identified need.

Two programs incorporated screening assessments in the EHR when electronic check-in was performed. EHRs were modified to accommodate screening scores, facilitate documentation of brief intervention, and provide clinical decision support, including facilitating referrals and billing processes. One program incorporated automated patient education materials. Technology was used to support SBI services, including integration of training videos and materials in two programs’ learning management system and, in one program, the development and dissemination of a podcast to educate providers on FASDs. One program delivered SBI services through the Computerized Intervention Authoring System program, fully integrated with the EHR to capture SBI delivery.

Program evaluation and dissemination efforts included utilizing de-identified EHR data to monitor program performance and provide regular feedback to stakeholders, relying on already established electronic systems, pre-existing collaborations and partnerships, project-specific data dashboards, and routine opportunities for meetings and feedback mechanisms.

Intermediate provider-specific practice changes, serving as proxies for longer-term change,²⁷ were evaluated in three of the four programs (PPGNHAIK, PPSNE, BMC) including: 1) self-reported assessment of skills in providing SBI services; 2) attitudes about the importance of conducting SBI services; 3) self-efficacy/confidence in conducting SBI activities; 4) readiness to change practice behavior; 5) training satisfaction; and, 6) knowledge about alcohol use limits and effects on pregnancy. Results below summarize a larger sample of questions; more details are available in Online Supplemental Material 1.

SBIRT service data were obtained from EHR records and used to measure program performance with 48 health centers reporting data across the four programs (Table 4). Three programs (PPGNHAIK, PPSNE, BMC) provided data for 36 months (July 1, 2019 through June 30, 2022); HFH began 18 months later on January 1, 2021. The target population were individuals aged 18 and older, presenting for an annual, preventive, or reproductive health care visit (308,938 individuals, ranging from 3,079 to 268,651). Populations served and screening information was tracked across years for comparison within each program, not across programs. The proportion of the target population screened for alcohol use ranged from 9.0% (HFH) to 89.5% (PPGNHAIK) across programs with 106,826 patients screened. The proportion of patients who screened positive for excessive alcohol use (n=10,087) ranged from 7.8% (BMC) to 26.7% (HFH). Programs had varying proportions of patients scoring in the high-risk range for alcohol use. All programs tracked BIs that were conducted for either alcohol or another substance. Proportions of those who screened positive for any substance who received a brief intervention ranged from 13.7% (BMC) to 61.5% (PPGNHAIK). Where possible, programs reported on screening, alcohol-only brief intervention, and referral rates for substances other than alcohol (Table 4).

Patient characteristic categories for race and ethnicity are reported in the manner in which they were collected from their collaborating health centers/sites. Three programs served predominantly female patients (97%−100%), while one (BMC) served more male patients (52.5%) (Table 5) with mean ages between 29 and 34 years old. HFH served a majority Black/African American (72.0%) population while PPGNHAIK served a majority White (68.6%) population. PPSNE’s patient population was primarily White (46.1%) but with the largest Hispanic population (24.7%) across programs, while BMC’s patient profile was 37.1% Black/African American and 24.6% White. Within the two Planned Parenthood affiliates, approximately 42% of patients received public assistance.