HIV treatment programs in sub-Saharan Africa currently deliver antiretroviral treatment (ART) and pre-ART care to millions of patients (1). However, the long-term success of HIV treatment programs depends on identifying models of treatment delivery that provide quality care at a low cost. For many of the most important aspects of HIV treatment delivery, the smallest unit which allows variation in delivery approaches is the treatment facility. For instance, health systems integration, health worker team composition, and location of treatment are inherent characteristics of facilities that cannot be varied at the level of individual patients. Many other factors, such monitoring and screening approaches, which theoretically could be varied at the level of the patient, will be impractical or impossible to vary within a facility for managerial or political reasons. Because treatment facility is often the smallest unit that allows for randomization, cluster randomized controlled trials (cRCTs) have become a methodological mainstay for HIV treatment implementation research, as evidenced by their use in trials investigating testing and counseling (2), treatment as prevention (3, 4), provision of care by peer health workers (5, 6) and tuberculosis prevention (7).

Unlike traditional randomized trials, cRCTs randomize entire pre-existing groups, such as health centers or neighborhoods, to different study arms. cRCTs are advantageous when interventions are naturally implemented at the group level, when denying some group members access to the intervention would be challenging or unacceptable, or to minimize contamination of study arms (8–10). However, combining group-level randomization with individual-level analysis creates challenges for study design. Because subjects from the same cluster often share characteristics such as beliefs, behaviors, or environments, their outcomes are often correlated. Consequently, each subject in a cRCT provides less independent information than a subject in a comparable individually randomized trial. In the analysis phase, ignoring the correlation between cluster members will increase the type-I error rate. However, if the correlation between cluster members is accounted for in the analysis phase but ignored during the design phase, sample size calculations will underestimate the true number of subjects required to adequately power the cRCT, increasing the type-II error rate. (11–13).

Researchers commonly adjust for this correlation by multiplying the sample size required for a comparable individually randomized trial by the “Design Effect,” also known as the “Variance Inflation Factor.” The Design Effect is a function of the number of individuals per cluster, m, and the intra-cluster correlation coefficient (ICC), a parameter that describes the proportion of the total variation in outcomes that is due to variation between clusters (9). DesignEffect=1+(m−1)∗ICC

Typically, researchers designing cRCTs rely on previously published ICCs for their sample size calculations. However, because ICCs depend on a study’s outcome, design, analytic plan, and population, researchers often struggle to find published ICCs that apply to their study (14). Despite calls for increased publications of ICCs for a range of outcomes, (15) fewer than 20% of both health-related and HIV-related cRCTs report their ICCs (16, 17).

Only three previous studies have attempted to fill this gap by reporting ICCs for a range of HIV-related outcomes. Of these studies, two focused on sexual attitudes and behaviors among adolescents and young adults in the United States (18, 19). One study calculated ICCs for outcomes among people living with HIV, but included only a limited number of clinical outcomes and calculated ICCs using baseline data alone, which is applicable to future cross-sectional studies but not necessarily prospective studies (20). This paper aims to provide estimates of the ICC for HIV-related clinical outcomes calculated both cross-sectionally and longitudinally among clinics supporting patients enrolled in a large urban HIV care and treatment program in Dar es Salaam, Tanzania.

Of the 109,943 patients enrolled in MDH since 2004, 109,320 had information on their site of enrollment and 73,862 of the 74,067 patients who were initiated on ART over follow-up had information on their site of ART initiation. Table 1 presents the distribution of patients and patient visits by site of enrollment and ART initiation. A detailed description of the baseline sociodemographic and clinical characteristics of the study population has been discussed elsewhere (26, 27). Over the first 2 years of follow up, only 4.6% of total patient-visits occurred at a site that was different from the patients’ original site of enrollment. Among those patients who were initiated on ART, only 8.4% of patient-visits occurring between the date of ART initiation and the date two years after the patient first became eligible for immunologic failure or second line-drugs took place at a site that was different from the patients’ original site of ART initiation.

Table 2 presents descriptive statistics and ICCs for general health outcomes among all patient. The ICCs in Table 2 ranged from 0.0050 (95% CI: 0.0022, 0.0110) for receiving tuberculosis treatment and 0.0544 (0.0343, 0.0850). Table 3 presents descriptive statistics and ICCs for nutritional outcomes among all non-pregnant patients. The ICCs for nutritional outcomes ranged more widely than for the general health outcomes spanning from 0.0641, (95% CI: 0.0366, 0.1100) for overweight at baseline to 0.0009, (95% CI: 0.0003, 0.0030) for the 6-month cumulative incidence for obesity. For both general health outcomes and nutritional outcomes, ICCs for a given outcome tended to be higher at enrollment than for subsequent visits. However, among subsequent visits, the ICCs tended to increase with time since enrollment.

Table 4 presents descriptive statistics and ICCs for plasma lipid level outcomes among patents. Because fewer individuals provided data on plasma lipid level measurements, fewer sites met the criteria for inclusion in the estimation of the ICCs, and the confidence intervals for lipid level ICCs tended to be relatively large. In some cases, the confidence intervals for the ICCs could not be estimated; however, the point estimates of these ICCs were investigated and judged to be reasonable given site-specific data. The ICCs for lipid level outcomes tended to be smaller than the ICCs reported in Table 2, ranging from 0.0207 (95% CI: 0.0067, 0.0622) for prevalence of low HDL at enrollment to 0 for the prevalence of high total cholesterol at 24 months, cumulative incidence of high LDL levels at 12 and 24 months, and change in ALT from baseline to 12 months. As in Table 2, the point estimate for the ICC of an outcome at enrollment tended to be higher than the ICC for that outcome over follow-up.

Table 5 presents descriptive statistics and ICCs for different definitions of immunologic failure among ART initiated patients. Cumulative incidences for immunologic failure were calculated starting from the first date of eligibility for immunologic failure, which occurs 168 days after ART initiation. Despite having similar clinical significance, the five definitions of immunological failure had relatively variable point estimates for the ICC, although their confidence intervals often overlapped substantially. The ICCs for the cumulative incidence of immunologic failure by two of the three single-item definitions, CD4+ cell count <100 and 50% drop in CD4+ count from peak value, decreased over time. In contrast, the ICCs for the cumulative incidence of immunologic failure defined as a return to pre-ART baseline CD4+ count or lower increased over time. Immunologic failure by the Tanzanian criteria and by any criteria, both of which are composite outcomes including at least one definition of immunologic failure for which ICCs increased over time as well as the definition of immunologic failure for which ICCs decreased over time, did not show a monotonic trend.

Table 6 presents descriptive statistics and ICCs for additional clinical outcomes among ART-initiated patients. ICCs ranged from 0.0707 (95% CI: 0.0448, 0.1099) for the 24-month cumulative incidence of non-adherence to 0 for the change in ALT IU/L 12 months from ALT initiation. As seen previously, ICCs for a given outcome tended to increase with time of ART initiation. However, ALT>40 IU/L and ALT>120 IU/L, the only outcomes in Table 6 for which prevalence at ART initiation was available, had lower ICCs for prevalence at ART initiation than for cumulative incidences over follow-up.

This paper reports ICCs for key clinical outcomes among a large cohort of HIV-positive adults. This cohort allowed for the calculation of ICCs for many novel outcomes. Furthermore, while previous papers reporting ICCs for HIV-related outcomes relied exclusively on data available at baseline (18–20), this paper estimated ICCs using both cross-sectional and longitudinal data. Because most cRCTs focus on longitudinal outcomes, these ICCs may better reflect future researchers’ chosen study designs than ICCs calculated from baseline data alone.

For many binary outcomes, the ICC for prevalence at baseline was larger than the ICCs calculated using longitudinal data. There are three explanations for this pattern. First, the ICC for a binary outcome is dependent on the probability of that outcome (35). All else being equal, we would expect the ICCs for prevalence at baseline to be greater than the ICCs for cumulative incidences whenever baseline prevalence was greater than later cumulative incidences, as was the case for many outcomes in our study. Second, when calculating the ICCs for incident outcomes, we excluded prevalent cases. Afterwards, the remaining population of at-risk patients at each clinic became more similar to each other than the initial populations at each clinic had been, reducing both between-clinic variation and the ICC. Third, it has been noted that modeling time when calculating the ICC tends to meaningfully reduce ICC estimates (14). While time was not explicitly in our calculations, several of our outcomes, such as the cumulative incidences and changes from baseline values, investigated changes in patients’ status from baseline. By looking at changes in patients’ status over time, we were able to incorporate time into the definition of our outcome. This implicit modeling of time may also help explain why the ICCs for prevalence tended to be higher than ICCs for cumulative incidence and why the ICCs for baseline values tended to be higher than the ICCs for changes from baseline over time. Regardless of the precise mechanisms, the differences between ICCs calculated using baseline data and ICCs calculated using longitudinal data were often large enough to have substantial implications for sample size calculations, highlighting the importance of publishing ICCs for outcomes that are relevant for longitudinal study designs rather than ICCs calculated using baseline data alone.

The ICCs for the different definitions of cumulative incidence of immunologic failure did not demonstrate a consistent trend over time. While ICCs for the cumulative incidence of immunologic failure by the definitions of CD4+ cell count <100 and 50% drop in CD4+ count from peak value increased over time, the ICCs for the cumulative incidence of immunologic failure defined as a return to pre-ART baseline CD4+ count or lower decreased over time. While it is unclear why the trend for ICCs over time should be different for immunologic failure defined as a return to pre-ART baseline CD4+ count or lower than for most other outcomes included in this paper, the high 6-month cumulative incidence of return to pre-ART baseline CD4+ count or lower relative to the other single-item definitions of immunologic failure may play a role. For the immunologic failure defined by either Tanzanian criteria or by any criteria, the lack of a monotonic increasing trend in ICCs over time may reflect the fact that both outcomes include a return to pre-ART baseline CD4+ count or lower in their composite definition of immunologic failure.

The largest ICC observed in our study was for the 24-month cumulative incidence of non-adherence. This finding is consistent with previous observations that behavioral outcomes tend to have higher ICCs than physiologic outcomes. Our results are also similar to previously published estimates from Zhang et al. (20), who estimated ICCs from HIV clinics in Kenya, Namibia, and Tanzania. Their unadjusted estimates for ICCs for any missed medication does in the past 30 days as reported verbally (ICC=0.029, 95% CI: 0.014, 0.069) and using a visual analog scale, (ICC=0.041, 95% CI: 0.021, 0.095) overlap with our ICCs for ART non-adherence. Their confidence intervals for the ICC for CD4+ count <200 (ICC=0.019, 95% CI: 0.009, 0.048) also overlapped with our confidence intervals of the ICC for CD4+ cell count <100. While their unadjusted ICC for CD4+ count (ICC=0.017, 95% CI: 0.007, 0.043) was somewhat lower than our ICC for CD4+ count at baseline, the confidence intervals overlapped with our ICCs for change in CD4+ count 6, 12, and 24 months from enrollment.

Over the time periods included in our analysis, only 4.6% of patient-visits occurred at sites other than the site of enrollment among the general population of MDH adult patients, and only 8.4% of patient-visits occurred at sites other than the site of ART initiation among patients who initiated ART. Because this variation in visit site could influence the ICCs, we removed visits that occurred at sites other than the site of enrollment or ART initiation and reran some of our analyses. After deleting those visits that took place other than at the site of enrollment or ART initiation, ICCs mostly decreased, usually by a little, but occasionally by a lot. This is somewhat surprising since mixing of patients between facilities should tend to dampen between-facilities variation in outcome rates, making them more alike to one another.

Although many ICCs reported in this paper appear small, sample size calculations, especially for binary outcomes, are sensitive to small changes in the ICC. Consider a hypothetical 2-armed cRCT conducted in Dar es Salaam designed to reduce the 24-month cumulative incidence of immunologic failure from 46% to 36%. Assuming equal sized intervention and control arms, a naïve sample size calculation assuming an ICC of zero would estimate that a study with 758 patients would achieve 80% power (36). However, if the true ICC for the study population were 0.0102, which reflects our estimate of the ICC of the 24-month cumulative incidence of immunologic failure by any definition and also approximates the mean of the ICCs reported in this paper, a study with 50 patients per cluster would require 1200 patients to achieve 80% power, over 1.5 times as many as would have been required in an individually randomized setting. A study with 100 individuals per cluster would require 1600 patients to achieve 80% power, which is over twice as many patients as were required in calculations that ignored the ICC. These scenarios demonstrate how failing to account for seemingly small ICCs can result in severely underpowered studies.

Because sample size calculations are sensitive to small changes in the ICC, we urge researchers to be cautious when selecting published ICCs to use in their sample size calculations. As demonstrated by the variation in point estimates of the ICCs for different definitions of immunologic failure, small differences in the definition of the outcome can meaningfully change the point estimates of ICCs. Therefore, we join other authors in recommending that researchers designing RCTs conduct sensitivity analyses to evaluate their anticipated power under the full range of ICCs suggested by the confidence intervals (9, 37).

This paper calculates the confidence intervals for ICCs using a formula that is based on a normality assumption. Because binary outcomes cannot satisfy this normality assumption, the confidence intervals reported in this paper are not strictly valid for binary outcomes. This normality assumption also may not be satisfied for all continuous outcomes. Because the calculations of ICC intervals can be very sensitive to violations of the assumptions of normality, these confidence intervals should be considered an approximation of the range of values that could be encountered in practice (38).

Donner and Klar (9) have previously advised against overestimating the stability of ICCs calculated from fewer than 40 clusters. Although 47 sites are included in this study, most of our ICCs were calculated using fewer than 40 sites due to the exclusion of sites with fewer than 5 events for binary outcomes, 10 measurements for continuous outcomes, or which were extreme outliers. The MDH clinics are part of a complex health system, and some sites serve few patients while others face special circumstances causing them to act as outliers. Failing to use exclusion criteria sometimes resulted in unstable or implausible estimates for some ICCs. Despite this limitation, the MDH cohort is one of the largest cohorts of HIV-positive people, and few other cohorts are better suited to estimating these ICCs. For example, the median number of sites used in the calculation of our ICCs is greater than the total number sites available to Zhang et al. (20).

Our ICCs were calculated among adults attending CTCs in in Dar es Salaam, Tanzania. These findings will be most useful for researchers randomizing at the clinic level in urban sub-Saharan African settings, but may also be useful in other resource-limited settings. Researchers can also use our estimates to calculate coefficients of variation for binary outcomes, which have been used in alternative formulas of estimating sample sizes for cRCTs (39). For binary outcomes, the relationship between the coeffient of variation and the ICC is described using the following equation: ICC=k2(π1−π)where k is the coefficient of variation and π is the probability of the outcome (40). However, these ICCs cannot be generalized to all contexts. In particular, ICCs for binary outcomes can only be generalized to contexts with similar prevalence or cumulative incidences (35).

cRCTs provide unique opportunities to evaluate HIV-related interventions, especially for HIV treatment implementation research. However, researchers can effectively leverage this study design only if they have access to applicable ICCs needed to accurately predict the sample size needed to design well-powered, efficient studies. There is a critical need to provide future researchers with ICCs for a diverse range of HIV-related clinical outcomes. Despite some limitations, these estimates provide valuable information, especially given the current scarcity of ICCs for HIV-related outcomes. Future researchers conducting cRCTs should consider publishing ICCs for not only their primary outcomes but also for secondary outcomes to address this persistent gap in the literature.