During in-depth interviews, we asked staff to describe any barriers or facilitators that they encountered in the preparation for and implementation of RCE. We also had staff describe how implementation strategies were used to address barriers to implementation. Finally, we asked staff to describe any systems changes that were necessary to prepare for or implement RCE and any adaptations they made to RCE and why. We assessed these variables at four timepoints: once during pre-implementation and in months 2, 4, and 6 of implementation.

We assessed acceptability, appropriateness, and feasibility of RCE by modifying Weiner and colleagues’ 4-item Acceptability of Intervention Measure (e.g., “RCE meets my approval.”), 4-item Intervention Appropriateness Measure (e.g., “RCE seems fitting.”), and 4-item Feasibility of Intervention Measure (e.g., “RCE seems implementable.”) [27]. We assessed implementation climate using six items adapted from Jacobs and colleagues’ measure of individuals’ perceptions of whether implementation of an innovation is rewarded, supported, and expected within their organization (e.g., “During the past month RCE project staff implementing RCE were expected to enroll a certain number of patients in RCE.”) [28]. We assessed organizational readiness using 12 items adapted from Shea and colleagues’ Organizational Readiness for Implementing Change measure (e.g., “During the past month RCE project staff implementing RCE are committed to implementing this intervention”) [29]. Each item was rated on a 5-point scale ranging from completely disagree (1) to completely agree (5). We also explored implementation context and outcomes during in-depth interviews with implementation staff. We assessed these variables at all seven timepoints: once during pre-implementation and monthly during the first six months of implementation.

For each consenting client, we calculated a fidelity score (0 to 7) that indicated the number of RCE visit components received. We operationalized fidelity as the percentage of clients who received all seven RCE visit components.

For each clinic, we calculated the proportion of clients linked to care within 3 days of referral to the RCE program during the implementation period. We operationalized referral date as the date clients self-referred into the RCE program or were connected to the RCE Concierge through a referral partner. We also provided each clinic with an Excel spreadsheet that had detailed instructions for gathering and inputting baseline linkage-to-care data. The spreadsheet contained formulas that calculated an aggregate linkage-to-care rate based on these inputs.

We had implementation staff rate each toolkit component’s usefulness in preparing for and/or implementing RCE on a 4-point scale ranging from not at all useful (1) to extremely useful (4). For each toolkit component, we also asked staff to assess how well it met each of its objectives on a scale of not at all (1) to very well (10). For select toolkit components, we asked implementation staff additional questions including whether they liked the component, the material in the component was easy to understand, and/or an appropriate amount of information was included in the component. Staff indicated their agreement with these items on a scale of strongly disagree (1) to strongly agree (4). Finally, for each toolkit component, we asked implementation staff about their perceived self-efficacy for performing key RCE behaviors covered in the component and their perceived likelihood of performing the key behaviors covered in the component on a 4-point scale ranging from strongly agree (1) to strongly disagree (4). Quantitative ratings of toolkit components informed the questions asked during follow-up, in-depth interviews. For instance, implementation staff were asked what they found useful or not useful, what could be improved, and any additional information that should be included in that component. We assessed these variables at four timepoints: once during pre-implementation and in months 1, 3, and 5 of implementation.

By evaluating the RCE implementation process, we were able to identify how the toolkit could be improved to mitigate implementation barriers and enhance facilitators and how program adaptations and systems changes supported RCE implementation (see Table 4).

Across the two clinics, most implementation staff perceived RCE to be acceptable (85%−100% agreeing or completely agreeing with each acceptability statement), appropriate (62%−100% agreeing or completely agreeing with each appropriateness statement), and feasible (86%−100% agreeing or completely agreeing with each feasibility statement). These positive perceptions of RCE were reiterated in the interviews, especially later in the implementation period, with staff noting how RCE provides clients with a welcoming introduction to HIV care. By contrast, early in the implementation period, some staff reported concerns about the time and effort that implementing RCE would require.

The majority of RCE implementation staff also agreed or strongly agreed with each statement for the implementation context domains of organizational readiness for implementing change (60%−100%) and implementation climate (66%−100%). Staff expanded on the implementation context in interviews, where they shared how the Clinic Champions played key roles ensuring staff buy-in and how open communication with clinic leadership was important for troubleshooting challenges. Staff from both clinics also noted that they had a method to reward or praise staff for their good work.

The first site (Clinic A) achieved 85% intervention fidelity—out of a total of 47 clients linked to care during implementation, four clients received 6 of 7 intended RCE services and three clients received 5 of 7 intended RCE services. Services not received during the RCE visit included laboratory tests to assess medical status (n = 3), a brief introduction to an HIV Specialist (n = 2), a discussion about insurance and financial assistance programs (n = 3), and referrals to wraparound support services (n = 2). The second site (Clinic B) achieved 100% fidelity, meaning all RCE clients (n = 60) received all intended services at their RCE visit.

To better support future implementing organizations to deliver RCE with fidelity, we tailored roleplay scenarios in our training modules and implementation manual to reflect real implementation challenges. For instance, our training module on delivering the RCE visit includes a roleplay scenario in which a client shares that they do not have insurance and another scenario in which an HIV Specialist is not available to perform a full medical visit.

To help alleviate concerns about the effort required to implement RCE, we offered tips throughout the training modules and implementation manual on how staff can integrate RCE without disrupting their overall workflow. We also modified questions in the readiness worksheet to help implementing organizations identify channels for communicating with clinic leadership and for rewarding staff for their efforts implementing an innovation.

At the first site (Clinic A), linkage to care (i.e., the proportion of clients whose RCE visit was within 3 days of referral) increased from 68% at baseline to 77% during the implementation period. However, this increase was not statistically significant (see Table 5). At this site, two new staff members joined the team approximately halfway through the implementation period, and their role involved intake of RCE referrals and scheduling of RCE visits. Thus, we also assessed change in linkage to care from baseline to the first versus second half of RCE implementation. Clinic A achieved 53% linkage to care during the first half of implementation and 90% linkage to care during the second half of implementation. There was a statistically significant increase in linkage to care from baseline to the second half of implementation (X² = 4.06, p < 0.05; see Table 5).

The second site (Clinic B) integrated RCE with an existing rapid ART program and had a very high linkage-to-care rate at baseline. We found that linkage to care decreased from 98% at baseline to 90% during the implementation period. This decrease was not statistically significant (see Table 5). As part of their adoption of RCE, this site partnered with a health center in the area to reach new clientele and increase referrals into the RCE program. All clients not linked to care within 3 days (6 of 60 total RCE clients) were referred by this partnering health center. In interviews, implementation staff noted unanticipated barriers to seeing clients from this health center within 3 days of referral—namely, issues with transportation (a 45- to 60-minute trip via public transportation) and finding a time when the RCE client could meet with their provider.

By testing each component of the toolkit with key implementation staff—via surveys and follow-up, in-depth interviews—we were able to identify critical refinements to make the toolkit more helpful and easier to use and to ensure that the toolkit met its key objectives of supporting adoption and effective implementation of RCE (see Table 6). Key findings included shortening the orientation video and creating a client-facing video to promote RCE; making the readiness worksheet more reflective of clinic challenges around staffing, workflow, and providing supportive services; streamlining trainings and adding realistic roleplay scenarios; making language overall more accessible; and shortening materials for maximum utility.

Our implementation evaluation in two distinct healthcare settings helped us identify critical refinements to the implementation toolkit to support uptake and successful implementation of the RCE program. A key finding was the need to make toolkit components more succinct and user-friendly for busy implementation staff. This finding is supported in the literature by a recent qualitative study of clinic staff’s perspectives of intervention toolkits, in which staff shared their desire for toolkits to contain brief resources in multiple formats with high functionality [20]. To address this need, we made toolkit components shorter and/or easier to navigate, without removing any information required for successful implementation of the RCE program. We streamlined content by writing more concisely, leveraging bulleted lists, removing redundancies, and depicting information visually wherever possible. In line with research findings that 5- to 20-minute videos work best for maximizing adult learners’ attention [31], we made our training content more engaging and digestible by turning three time-intensive, recorded training videos (1 to 2 h each) into a set of six brief training modules (15 to 20 min each). We focused each module on a discrete part of the RCE program (e.g., patient-centered care strategies, referral process, RCE visit, outreach process, monitoring). Finally, we turned our comprehensive implementation manual into an interactive PDF with navigation features to help implementation staff easily find the specific information they are looking for.

Another key refinement to the toolkit was providing more context on how implementation staff could adapt RCE for their specific setting. This was addressed by acknowledging that the referral process may need to be slightly different depending on the source of referral (e.g., having a different process for someone who tests positive for HIV in an inpatient setting within the healthcare system than for someone who tests positive at an affiliated mobile testing unit) and that implementation roles are flexible (e.g., a single staff may fulfill multiple roles and/or multiple staff may serve in the same role). Importantly, the ability to adapt an innovation like RCE for local use may increase its acceptability among implementation staff [22].

We found that at both clinic sites, staff reported readiness to implement RCE and perceived that implementation of RCE was rewarded, supported, and expected. Staff across both clinics also reported high acceptability (i.e., the extent to which RCE is perceived to be agreeable, palatable, or satisfactory), appropriateness (i.e., the perceived relevance or compatibility of RCE within a setting and/or the perceived fit of RCE to address a particular problem), and feasibility (i.e., the perceived ability of RCE to be successfully implemented within a setting) of RCE. In interviews, staff underscored the importance of open communication with leadership and having a system in place to praise or reward staff. Thus, we added questions to the readiness worksheet to help future implementing organizations identify channels for communicating with clinic leadership and for staff acknowledgements and rewards, in preparation for RCE.

We identified just a few challenges with program fidelity at one clinic (Site A). Of the 47 clients linked to care during implementation at this site, seven did not receive one or more of the intended RCE services. Services not received included laboratory tests to assess medical status, a brief introduction to an HIV Specialist, a discussion about insurance and financial assistance programs, and referrals to wraparound support services. This may have been due, at least in part, to staffing shortages that this clinic reported as a barrier to implementation. Across nearly all segments of the healthcare workforce, staffing shortages and employment turnover has not yet fully recovered from the COVID-19 pandemic [32, 33]. To help future implementing organizations prepare for turnover among implementation staff, we expanded the readiness worksheet to include strategies to ensure staff coverage for RCE (e.g., building redundancy into roles). Moreover, our efforts to streamline and shorten the toolkit components will support onboarding of new staff in preparation for and throughout program implementation.

To further support organizations with effective implementation of RCE, our toolkit follows two best practices for achieving intervention fidelity: standardizing training and monitoring intervention delivery [34]. To standardize training, our toolkit includes a comprehensive implementation manual, companion quick guide, and series of training modules that offer (1) step-by-step guidance on how to implement the RCE program and (2) roleplay scenarios that support practical application of this guidance. Our toolkit also includes a report card to help implementing organizations monitor progress and identify and correct any issues with fidelity. Moreover, as a result of our implementation evaluation findings, we added “real-world” tips and/or questions across toolkit components to prompt consideration of how to incorporate delivery of key RCE services into existing workflows. We also incorporated strategies that implementation staff in this pilot study identified as key facilitators to RCE implementation (e.g., having one or more backup RCE Concierges to ensure adequate coverage and support for RCE clients).

Our study offers preliminary evidence that the RCE implementation toolkit can support effective implementation of RCE and bolsters evidence regarding the impact of RCE on linkage to HIV care. Overall, both clinics implemented the RCE program with a high degree of fidelity—Site A achieved 85% fidelity and Site B achieved 100% fidelity. This suggests that the toolkit was effective in helping clinics prepare for and implement the Red Carpet Entry program. As a result of adopting and implementing the RCE program, Site A saw an improvement in linkage to care from 68% at baseline to 77% during the implementation period. Furthermore, Site A was able to achieve 90% linkage to care during the second half of the implementation period, after addressing staffing shortages. While Site B did not see an improvement in linkage to care as a result of RCE, they integrated RCE into an existing rapid ART program and had a very high baseline linkage-to-care rate (98%). These findings highlight the potential impact of the RCE program in healthcare settings in which linkage to care is a concern, specifically settings in which there is no existing rapid or same-day ART program.

There are several limitations in this study. We conducted this implementation evaluation study at just two clinic sites with a small number of implementation staff. While these sites represented distinct settings—an infectious disease practice within a university health system and a local health department—the toolkit refinements we made as a result of the implementation evaluation may not capture the implementation strategies, adaptations, and systems changes needed to make RCE work in other healthcare settings. While we collected data from all key implementation staff and clinic leaders at the participating sites, we did not include implementation-adjacent staff, such as front-desk staff. Our small sample of participants further limits the generalizability of the toolkit refinements made. Further, we did not assess whether the refinements we made to the toolkit following our implementation evaluation increased perceived effectiveness of the toolkit among implementation staff and/or improvement in implementation and health service-related outcomes. Finally, data collection methods relied on self-report and findings subject to recall and social desirability bias. The field of dissemination and implementation would benefit from future research on whether approaches such as ours yield improvements in toolkit development and effectiveness, and subsequent improvements in implementation effectiveness.