Key populations, such as female sex workers (FSW) and men who have sex with men (MSM), are difficult to survey because of the lack of a clearly defined sampling frame. In many cases, surveying these populations is desirable so that program planners, or health service providers can provide targeted services, assess the effectiveness of their programs, and determine the prevalence of diseases (Lansky et al., 2007; UNAIDS and Organization, 2010; Johnston et al., 2013). In 2014, the Joint United Programme on HIV/AIDS (UNAIDS) set three targets, called the 90-90-90 targets, to achieve by 2020: that 90% of all people living with HIV will know their status, 90% of all people with diagnosed HIV infection will receive sustained antiretroviral therapy, and 90% of all people receiving antiretroviral therapy will have viral suppression. Surveying key populations is essential to properly characterizing the progress toward reaching the 90-90-90 targets (UNAIDS, 2014; Hladik et al., 2017).

A link-tracing network sampling method called respondent-driven sampling (RDS) has been widely used to study hard-to-reach populations due to the hidden nature of these populations (Heckathorn, 1997; Johnston et al., 2006, 2015; Malekinejad et al., 2008). RDS starts with a small sample (usually a convenience sample) of initial respondents selected by investigators called seeds. These seeds are given a small number of coupons (typically 2–5) which they can give to other members of the population. Those who receive the coupons may present themselves at the survey office to be added to the study. After participating in the survey, individuals are given coupons to recruit others into the study. This process continues, growing the recruitment chain over successive waves (each step in the recruitment chain after the seed). Participants receive incentives both for participating in the study and for recruiting others into the study (Heckathorn, 1997).

RDS has become more widely adopted because it allows researchers to survey hard-to-reach populations without a sampling frame (Malekinejad et al., 2008). Unlike another link-tracing sampling method called snowball sampling in which respondents are asked to nominate possible recruits to the researcher, in RDS, respondents themselves are responsible for recruiting others into the study by handing out the coupons that they receive. This means that respondents are not required to divulge any information about others in the population, lessening confidentiality concerns, though more of the burden of recruitment falls on the respondent (Heckathorn, 1997).

RDS is not without its limitations. Because the respondents themselves are essentially in charge of the sampling, we must make many assumptions not only about the sampling process, but also about the network structure of the overall population (Heckathorn, 1997; Gile and Handcock, 2010; Gile et al., 2015). For example, respondents may choose to only give coupons to people they are closest to, which could oversample from certain subgroups due to homophily effects Paquette et al. (2011). Similarly, respondents may only know others who are similar to themselves, leaving few options in who they can recruit. Respondents may also recruit as is convenient, possibly resulting in samples that do not cover the study area. Since estimation methods using RDS make many sampling and network assumptions about the under-lying RDS surveys, violations of these assumptions might call into question the validity of estimates based on RDS data (Wang et al., 2005; Handcock and Gile, 2011; Salganik et al., 2011; White et al., 2012; Salganik, 2012; Gile et al., 2015). Gile et al. (2015) note that traditional statistical diagnostic tests are not appropriate because of the unknown dependence between recruiter and recruit. Therefore, we must develop new diagnostic measures that are both informative about the RDS process as well as easy to use during the actual data collection to help researchers as they run their studies. While previous work on RDS diagnostics has been done by Gile et al. (2015), Lansky et al. (2012), Liu et al. (2012), there are still gaps in the diagnostics, particularly in exploring the assumption that all units within the population are connected. To date, there have been no RDS diagnostics developed that use respondent latitude and longitude information.

The aim of this paper is to build on existing diagnostics of RDS performance by showing how diagnostics with geographical data can be used to provide more insight into the RDS process than information collected through typical survey questions. We start by taking some of the diagnostics first developed by Gile et al. (2015) — convergence plots which show the change in a weighted statistic as additional respondents are added, as well as bottleneck plots which show the same, except split up by the recruitment trees started by each seed – and expanding on them by looking at rolling values of statistics in addition to the cumulative values. Then, we explore the use of these diagnostics using geographic data, using the idea of a convex hull to track how much of the city an RDS survey traverses using both the cumulative and rolling average versions of the convergence and bottleneck plot. To demonstrate the utility of these diagnostic plots, we apply these to FSW and MSM in Kampala, Uganda, two survey populations for which geographic data were collected.

In Section 2, we introduce the data that we will be looking at. In Section 3, we discuss the assumptions in RDS. In Section 4, we develop the diagnostics and apply them to the sampled FSW and MSM populations. In Section 5, we include a discussion of some limitations of these diagnostics and in Section 6, we make concluding remarks.

To develop the RDS diagnostics in this paper, we used data collected in the 2012 Crane Survey, a key population focused survey in Kampala, Uganda run in collaboration between the Makerere University School of Public Health, the Ministry of Health, and the Centers for Disease Control and Prevention and funded by the US Government President’s Emergency Plan for AIDS Relief (Hladik et al., 2017; Doshi et al., 2018). The Crane Survey used RDS to survey key populations at high risk for HIV, including female sex workers (FSW), men who have sex with men (MSM), drug users (DUS), and other populations. For our applications, we focused on FSW and MSM since they had the largest sample sizes, and we were able to create more substantial examples using them. In this paper, we focus on two groups from the Crane Survey: FSW and MSM. Table 1 shows the characteristics of the RDS run as part of the Crane survey for both FSW and MSM. The eligibility criteria for FSW was women 15 years or older¹ residing in greater Kampala who had sold sex to men in the past 6 months, and the eligibility criteria for MSM were biological males 18 years or older residing in greater Kampala who had had anal sex with another man within the last six months. Exclusion criteria included being unable to understand the language (Luganda or English), coupon receipt from a stranger (rather than someone known to them), or being from outside the study area of greater Kampala.

For the majority of participants, the data were collected using audio computer assisted self-interview (ACASI) in Luganda or English. ACASI is an survey data collection system in which respondents listen to questions through headphones and mark answers on a computer. This allowed respondents to answer questions in a secure, private setting compared to face-to-face interviews, allowing them to answer sensitive questions with more candor. Data were collected on demographics, HIV-related behavior, STD symptoms, as well as various HIV-related knowledge, attitudes and practices, and mental health questions. The key innovation that provides much of the focus of the paper was the geographical data collected in these surveys. In a separate face-to-face interview, respondents were also asked to place a pin on an offline Google Map of Kampala marking their primary location of occupation (FSW) or socialization (MSM). Staff ensured geographic orientation by pointing out the survey office location as well as well-known Kampala landmarks. The maps that the respondents were shown were clean and did not contain any pins placed by previous respondents. The latitude and longitude values of the pin locations were then recorded. We note that though the latitude and longitude values calculated were quite exact, the pin locations themselves might not be, since respondents were simply asked to place the pin by sight. We did not require a greater level of precision than this, because in our applications, we focused on how the recruitment throughout the city as a whole. If needed, staff assisted respondents who had trouble finding “their” locations on the map.

For our applications, respondents who had pin locations outside of the city boundaries of Kampala, Uganda were also omitted, even though members who resided in greater Kampala were included in the overall survey. Respondents were only asked to record publicly accessible locations such as commercial venues or open air street locations. No private residential information was recorded. Collected geographic data were stored on password restricted computers accessible to survey staff and investigators on a need-to-know basis. Further discussion of how researchers may choose to approach collecting geographic data in general is provided in Section 4.5.

Even though RDS is useful for surveying hidden populations, there are many assumptions that must be made to use RDS data for prevalence and size estimates. In this paper, we will be focusing on two key types of assumptions: seed dependence and population structure.

One of the main ideas behind employing RDS was that having long chains of recruitment reduced the dependence of the samples on the initial seeds (Heckathorn, 1997). This seed dependence can be affected by the existence of recruitment homophily (where members of the social network tend to form ties with other members similar in characteristics to themselves) and of bottlenecks (where two parts of a social network are separated by just a few nodes). Though the samples might be influenced by the seeds to begin with, estimators typically assume that long chains remove seed dependence. The samples can then be considered to have been sampled independent of the initial seed, and assume the bias that comes with the initial convenience sample has been eliminated. That is, even though the seeds are selected from a convenience sample, the final sample is expected to be representative of the full population (Volz and Heckathorn, 2008; Gile and Handcock, 2010; Heckathorn, 2011; Gile et al., 2015).

Separate, but related, is the issue of having a clearly defined population that is being sampled. Notably, when using RDS, the geographical boundaries of the population can become uncertain. Since the respondents are put in charge of finding new candidate participants, they may recruit members from outside the study area, or only members within a certain subset of the study area (e.g. a particular community or neighborhood). While it is relatively easy to exclude those who report being outside of the geographical area of interest (e.g. a city or metropolitan area), finding out whether the sampling is able to reach everyone within the city is harder.

This can be thought of as a sort of selection error—the error that results as respondents self-select into the study. Even though RDS is designed specifically around self-selection as candidate participants choose to go to the study site and be included, we still have to be wary of a combination of coverage error and non-response error affecting our estimates. While we do not have a sampling frame, we do have a target population, and we are assuming that RDS could potentially reach everyone in this target population, and that we are able to exclude people who are not in this population.

In this way, we have some assumptions about the overall population structure that, in most applications, are simply assumed to be true (Gile et al., 2015). First, we assume that there is a connected network of individuals in the population. Sometimes, this is close to trivial—in the case of men who have sex with men (MSM), for example, we are only interested in the population as it relates to the heightened risk for HIV. Therefore, missing out on a pair of exclusive partners who do not have any contact with others and do not have HIV might be acceptable. However, it is possible that, due to some large social or geographical barriers, there are several large groups of people within the same city who do not have any connection to each other. In this case, an RDS chain that starts with a seed in one part of the city would (theoretically) not be able to traverse to everyone in the population, and the real population it would be drawing from would be the subset that was connected.

We note that this last point is important not only as it relates to estimating population parameters such as HIV and other STI prevalence measures, but also as it relates to population size estimation using RDS data (see Handcock et al., 2015; Crawford et al., 2018 for population size estimation methods using RDS data). While it is possible that, if the disjoint components in the population network have similar population characteristics such as HIV prevalence, the estimates could quite possibly be unaffected, population size estimation is definitely affected by the disjoint nature of the population, and identifying whether the RDS is capable of reaching all members of the population is crucially important in determining what exactly the size estimate represents. It is also quite unlikely that disjoint populations would also exhibit the same exact prevalence characteristics in reality.

Gile et al. (2015) recommend using convergence plots and bottleneck plots to determine whether the RDS recruitment chains reach convergence (i.e. they reach a point at which there is little to no seed dependence) and whether the seeds converge to the same value (i.e. the bottleneck effect is weak enough that different seeds are not sampling from effectively different sub-populations). Convergence plots show the sample size on the horizontal axis and (typically) a weighted statistic on the vertical axis. In this way, the convergence plot shows the value of the sample statistic as each subsequent respondent is added to the study, and as such, the order in which the sample statistics are shown is the chronological order. Because of this, we might have a case in which respondents in different waves (that is, number of steps removed from the seed) enter the study at a similar point in time. Bottleneck plots are similar to convergence plots, but plotted by seed so that differences across the various chains can be observed.

These are two of the plots recommended by Gile et al. (2015) and we refer to them as the cumulative convergence and bottleneck plots. In addition to these, we will look at sample statistics with a rolling window (as opposed to finding the cumulative statistic), helping us see whether most of the recruitment of people with certain characteristics is happening earlier or later in the sample. This achieves a similar goal as the convergence plots, but allows for more clarity in changes. The added information is particularly apparent when using the convex hull introduced in Section 4.4. In all cases, for the rolling average, we used a window of size 25. That is, each value of the average is calculated using the most recent 25 respondents at that point in time. We note that while this number was chosen arbitrarily, one could very easily use different values for the size of the rolling window, which would, in essence, change the smoothness of the rolling average plots. In our diagnostic plots, we did not observe any difference besides smoothness when varying the size of the rolling window. In applications with other RDS data, we suggest trying different values to find one that shows trends most clearly.

One limitation of the convex hull diagnostic plots is that they can be hard to interpret as the study is ongoing. Researchers may consider including questions in their surveys asking about how many people the respondents know in different neighborhoods to supplement these diagnostics and be more proactive in determining whether a dip in rolling convex hull area is cause for concern or not. For example, if researchers see such a dip and notice that the recent respondents generally said they do not know very many people outside of their own neighborhood, researchers might consider adding seeds from different geographical areas.

Investigators should balance knowledge of the geographical area of interest with the information provided by the diagnostics in this paper. Notably absent are the respondent pin locations displayed on a map. Due to privacy concerns, diagnostics with actual pin locations placed on a map were not deemed suitable to be shared with a larger audience. However, researchers during the RDS collection may use the actual locations placed on a map, and looking at these locations, along with the convex hull, on a map over time could provide additional insights into the recruitment process. A survey item asking about neighborhood of residence could achieve similar goals as the geographic data, but it would not be as fine-grained and could fail to detect cases in which disjoint networks exist in the same neighborhood.

In addition, for the diagnostics we have presented in this paper, we chose to omit those with pin locations outside of the city limits of Kampala, Uganda. However, one might be interested in exactly these cases, to identify when recruitment jumps outside of some geographical boundary.

The use of convex hulls is not perfect. Venues for socialization and work are not placed uniformly within a city, and the structure of streets and walkways can greatly affect the true distance between two points. In other words, there is no easy way to determine what a target convex hull size should be. Instead, we suggest looking at trends to determine whether there is reason to be concerned about recruitment getting stuck in certain areas, or whether the seed dependence is very high.

Furthermore, the usage of these convex hulls does not guarantee that researchers will be able to detect whether the population social network is connected. Disjoint networks might occupy the same geographic space, but be separated by social class or other demographic characteristics, such as education level. However, since part of how the population is defined is according to the geographic location (e.g. all FSW older than 15 years of age who had sold sex to a male within the last six months within the greater Kampala area), we believe examining the convex hull diagnostic plots is highly useful for evaluating the “selection error” that might occur with RDS.

For future extensions to these diagnostics, one area of interest might be in including study site. We did not consider study site in our diagnostics, but the location of whether respondents were required to go to take the survey could play a huge role in the recruitment. This is particularly pertinent in very large cities, where multiple study sites might be necessary, and researchers may be concerned about the level of geographic coverage of the RDS study.

Based on these results, we recommend using geographical data, when available, as another diagnostic measure for RDS, as they can help determine the effective reach of these surveys. We suggest using the area of the convex hull with convergence and bottleneck plots, as well as both cumulative and rolling versions of each of these plots, to better determine the range of the RDS and whether the seed dependence is overcome as the RDS progresses. In addition, we recommend using rolling plots alongside cumulative plots in general to more easily identify trends as they occur.