Eligibility criteria were defined using the PICOS (Population, Interventions, Comparisons, Outcomes, Study Design) format. Studies evaluating the accuracy and/or precision of any CD4 technology commercially available at the time of the review were considered eligible for inclusion. Currently, no “gold” standard technology or internationally recognised reference preparation exists for CD4 enumeration, and a wide range of flow cytometric technologies have been used as comparators [6].

For the purposes of this review, we included studies that used as reference technologies any flow cytometric method considered to be acceptable by the WHO HIV diagnostics working group named in the review protocol (S1 Methods).

Studies were identified by searching two electronic databases—MEDLINE and EMBASE, scanning the reference lists of the Nature supplement Evaluating diagnostics: the CD4 guide, and by inviting the WHO working group, whose members are authors of the Nature supplement to identify relevant studies for the review [6,8].

We used the following search terms to search the electronic databases: “CD4”, “technolog*”, “methodolog*”, “techn*”, “method*”, “test”, “evaluation”, “validation”, “accuracy”, “comparison”, “efficacy”, “performance”, “reproducibility”, “precision”, “flow AND cytometry”. Subject headings included: “CD4 antigen”, “antigens, CD4”, “CD4 lymphocyte count”, “CD4-positive T-lymphocytes”, “technology”, “methodology”, “technique”, “evaluation”, “clinical evaluation”, “economic evaluation”, “evaluation research”, “evaluation and follow up”, “instrument validation”, “validation process”, “validation study”, “evaluation studies as topic”, “validation studies”, and “flow cytometry”. Full electronic search strategies and review protocols are detailed in S1 Methods.

Articles were exported from the search database to EndNote and screened for relevance (Fig. 1). Data were extracted by two independent reviewers (SG and KS) and disagreements resolved through consensus.

The following data were extracted: study location, index test, reference test, and population (HIV positive or HIV positive and negative). Data on accuracy and precision included bias or mean difference and limits of agreement, misclassification probabilities (when sensitivity or specificity values were given, misclassification probabilities were calculated), and coefficient of variation. Where possible, HIV positive data alone were extracted. Where this was not possible, combined HIV positive and negative data were extracted. Only data within the clinically relevant range (thresholds of 200, 350, 500 cells/μl) were extracted as per the inclusion criteria.

Studies should report not only percent misclassification around clinically important CD4 cell thresholds (e.g., 200, 350 or 500 cells/μl), but should also report the magnitude of these misclassifications.

The secondary outcome measure addressed precision or reproducibility. Precision is particularly important when following a patient’s serial measurements using the same technology. Precision can be measured within-laboratory or between-laboratories and is expressed as percent coefficient of variation (%CV).

Studies meeting inclusion criteria were also assessed for bias and quality on ten points drawn from the STARD guidelines (Fig. 2) [9]. This review has been reported following the PRISMA statement guidance for reporting of systematic reviews [10,11].

This systematic review was first performed in July 2009. Of the 433 studies in the search, 345 were excluded as they were not performance evaluations. After further triage, 20 studies that measured bias, misclassification and/or %CV were accepted for inclusion in this review (PRISMA flow diagram, Fig. 1). A second search was conducted in April 2013 using the same search strategy and review protocol with the goal of capturing more POC CD4 enumeration technologies in the review. An additional 12 studies were included. A summary of data extracted from all eligible studies with data on bias, misclassifications and/or %CV is shown in S1 Dataset.

A summary of study characteristics is shown in Table 6.

At least one published performance evaluation study was found for each of the following technologies: BC Cytospheres [12–18], Dynal Dynabeads [14,18–22], Guava PCA [23–29], Partec CyFlow instruments [16,26,30–36], BD FACSCount [14,37–42]; BD Trucount tubes [43–46], BC Flow-Count fluorospheres [47], Cytognos Perfect-Count microspheres [48], SP Flow Cytometry using flow rate calibration [49–52]; BC PLG FlowCARE CD4 [53], Sysmex PocH-100i with Dynal Dynabeads [54], i+MED CD4 Select [55], Alere Pima Analyzer [56–58,61], PointCare NOW [60], Apogee Auto40 [59,63,64] and MBio Snap Count (MBio Diagnostics, Inc., Boulder, CO, USA [MBio]) [62].

No published, peer-reviewed performance evaluations of the commercially available technology Partec miniPOC were found by our literature search, and the MBio assay is not yet commercially available.

The findings of the quality assessment of included studies are summarised in Fig. 2. Most studies reported the index test (test under evaluation) and the reference standard in sufficient detail to be reproduced, but few studies reported whether staff at the evaluation sites were proficient at performing the reference standard and/or sufficiently trained on performing the index test. Few studies reported internal quality controls being performed during the evaluation period. Without these quality measures, it would be difficult to differentiate whether the bias or misclassification between the index and reference tests was due to differences in inherent test characteristics or to operator error.

Manufacturer involvement was evident in a number of studies. Seven studies declared one or more authors to be affiliated with the manufacturer of the index test [12,17,39,42,46,54]. Four studies were partially sponsored by the manufacturer [25,26,45,53]. One study stated that the manufacturer’s site was one of the study sites [42], and four studies declared donation of reagents or equipment by the manufacturer [15,29,38,40]. A further four studies could be considered to be calibration or test developers’ papers [30,49,51,55]. In the absence of definitions for a sponsored study versus an independent evaluation, it is not clear to what extent the inclusion of manufacturers as co-authors of papers influenced the study results.

As there is no international standard for CD4 enumeration, a variety of reference standard technologies were used for evaluating the performance of new CD4 technologies, making it difficult to pool data on bias and misclassification across all studies.

Bias (mean difference) data were collated and represented graphically but only from studies that compared the index tests mean difference to the same reference technology (Table 6 and Fig. 3a, b). FACSCount and FACSCalibur were the most common reference technologies.

Bias was evaluated at CD4 thresholds of 200 cells/μl and 350 cells/μl using the BD FACSCount as a reference method. For the Pima Analyser, at CD4 counts at both <350 cells/μl and >350 cells/μl, the studies found that the Pima Analyser consistently underestimated CD4 counts by-16.6 cells/μl (limits of agreement-88.4, +55.3 cells/μl at <350 cells/μl) and at-70.7 cells/μl (limits of agreement-216.5, +75 cells/μl) [58]. However, when the Guava Easy CD4 was compared to the FACSCount, the bias for CD4 counts <350 cells/μl was +13 cells/μl (limits of agreement-27, +53) and for cell counts >350 cells/μl, bias was-45.3 cells/μl [23,27].

The FACSCalibur, in comparison to the FACSCount, overestimated CD4 counts by +13.1 cells/μl at values of <350 cells/μl [41]. However, bias estimation of manual bead based methods showed underestimation of CD4 counts at <350 cells/μl of-35.2 cells/μl (limits of agreement-164.9,+ 94.6) and -0.4 cells/μl (limits of agreement-126, +125.2) for BC Cytospheres and Dynal Dynabeads respectively [18].

When the overall bias for all CD4 technologies was calculated, this ranged from-70.7 to +47 cells/μl for CD4 counts >350 cells/ μl and -35.2 to +13.1 cells/μl for CD4 counts <350 cells/μl when compared to the FASCount as a reference method.

We then progressed to study bias data using a threshold of 200 cells/μl compared to FACSCount as a reference method. A total of six publications were identified covering the following technologies, the Apogee Auto 40 [59], Guava Easy CD4 [23,25–27], and Partec CyFlow Counter [26,52].

The four studies that had reported data at <200 cells/μl and compared the Guava Easy CD4 to the FACSCount, all had a positive bias that ranged from +10 to +45.5 cells/μl [23,25–27]. One study had data for CD4 counts >200 cells/μl and had a bias of +44.9 cells/μl (limits of agreement-112.6 to + 212.3) [25]. It is interesting to note that all studies reporting the performance of the Guava Easy CD4 showed that this assay overestimated CD4 counts compared to the FACSCount [23,25–27].

Two studies compared the Partec CyFlow Counter to the FACSCount. One showed an over-estimation by +0.8 cells/μl (limits of agreement-21.7 to +23.2) while the other showed an underestimation by-5.8 cells/μl (limits of agreement-37.6 to +25.9) [26,52].

Overall when comparing the technologies to the FACSCount as a reference method, bias ranged from +44.9 to -12.1 for CD4 counts >200 cells/ μl and +45.5 to -5.8 for CD4 counts <200 cells/ μl. Bias using FACSCalibur as a reference method showed similar results to that using FACSCount (Fig. 3a and b).

Fig. 4 showed the range of misclassifications at thresholds of 200 and 350 cells/μl of new CD4 assays compared to different reference standards.

Nine studies provided data on misclassification probabilities using a cut-off of 350 cells/μl [16,18,27,56,60–64]. Data were available on the following assays: BC Cytospheres [16,18], Dynal Dynabeads [18], Guava Easy CD4 [27], Partec CyFlow Counter [16], Pima Analyzer [56,61], PointCare NOW [60], MBio Snap Count [62], and Auto40 Flow Cytometer [63,64].

Two studies [63,64] evaluated the Auto40 Flow Cytometer compared to the FACSCalibur in Cameroon. One study [63] reported upward and downward misclassification probabilities of 8% and 2%, respectively, while another [64] found the likelihood of under-treatment (upward misclassification) to be 3% and the probability of over-treatment (downward misclassification) to be 2%.

Karcher et al. conducted a large trial in field conditions in Uganda comparing BC Cytospheres and the Partec CyFlow Counter to DP flow cytometry [16]. HIV positive patients were recruited, the majority of whom had CD4 counts within a range from 0 to1200 (median 332 cells/μl). Of samples with CD4 counts <350 cells/μl measured by DP flow cytometry, BC Cytospheres misclassified 16% of the patients as having CD4 counts of >350 cells/μl, thereby denying them of eligibility for treatment. Similarly for those with CD4 counts > 350 cells/μl measured by DP flow cytometry, BC Cytospheres misclassified 20%, indicating that 20% of patients not qualifying for treatment using the reference test would have done so if BC Cytospheres were used. Karcher et al. also compared the Partec CyFlow to the DP flow cytometer and found that of samples with counts <350 cells/μl, 29% were misclassified as having >350 cells/μl by the Partec CyFlow Counter, and 7% of samples with CD4 counts >350 cells/μl were misclassified downward as being <350 cells/μl [16].

Lutwama et al. conducted a large study of manual technologies in Uganda; they recruited only HIV positive patients, the majority of whom had CD4 counts within the clinically important range (range in study: 0–900 cells/μl) using the reference standard technology [18]. Of samples with counts of <350 cells/μl using the reference standard technology, BC Cytospheres misclassified only 1% as >350 cells/μl. However, of those with counts >350 cells/μl, 68% were misclassified as <350 cells/μl (indicating that 68% of patients not qualifying for treatment using the reference test would have done so if BC Cytospheres were used). Dynal Dynabeads had upward and downward misclassification probabilities of 6% and 30% respectively.

Renault et al. (2010) conducted a comparison study between the Guava Easy CD4 and FACSCount. Across a range of CD4 (0–1100 cells/μl), the upward and downward misclassification was calculated and found to be 6.1% and 9.4%, respectively [27].

One study reported on evaluations of the PointCare NOW assay (this instrument has since been re-marketed/rebranded as HumaCount CD4now (Human Diagnsotics Worldwide mbH, Weisbaden, Germany) in five countries (Mozambique, Belgium, Canada, USA and South Africa) [60]. Mozambique, Belgium, Canada and USA used the FACSCalibur as a reference standard while South Africa compared the PointCare NOW to the Epics XL. Upward and downward misclassification were reported by country: Mozambique, +51%, -20%; Belgium, +62%, -4%; Canada, +50%, -0%; USA, +0%, -3%; and for South Africa, +64%, -6%. Overall misclassification was also calculated, and it was found that testing with PointCare NOW would have led to 47% of patients with CD4 counts less than 350 cells/μl not eligible to receive treatment (upward misclassification) and 6% of patients with CD4 counts greater than 350 cells/μl eligible to receive treatment (downward misclassification).

Of the two studies evaluating the Pima Analyzer, Herbert et al. (2013) used the BC Cytomics FC 500 as a reference standard while Jani et al. compared the Pima Analyzer to the BD FACSCalibur [56,61]. Across the clinically relevant range (60–1200 cells/μl), Herbert et al. found the upward and downward misclassification to be 6.1% and 9.4% respectively. MBio Snap Count was evaluated by Logan et al. (2013) and compared to the FACSCalibur [62]. Of the 94 samples, 2.1% were misclassified upward and 3.2% were misclassified downward at a threshold of 350 cells/μl.

Thirteen studies presented misclassification data using a CD4 cut-off of 200 cells/μl; these are presented in Fig. 4b [12,13,15,17,18,20,23,28,36,56,60,61,63,64].

Even through misclassification probabilities can be influenced by the number of patients with CD4 counts close to the threshold in each study, Pointcare NOW showed an overall tendancy towards upward misclassification at both thresholds of 200 and 350 cells/μl. Most other technologies showed misclassification probabilities of <10%.

Forty-four percent of studies reported within-laboratory precision of absolute CD4 count measurement, using replicates of fresh whole blood^. [15,17,19,21,24,29,32–34,38–42,45–47,50,54,56,58] Fig. 5 shows the inter- and intra assay variations expressed as % CV for the Apogee Auto assay and the intra-assay variation for the mBio SnapCount [62,63].

Five studies reported between-laboratory precision using whole blood. Two were studies evaluating BD FACSCount, [39,42] one evaluated Panleucogating, [50] and two were studies evaluating bead-based SP technology (BD Trucount tubes and BC Flow-Count fluorospheres) [46,47]. Gernow et al. studied the reproducibility of BC Cytospheres and found poor precision, with a coefficient of variation of 58% [15]. The study by Landay et al., however, found precision levels more in keeping with the other technologies [17].

Overall, in studies addressing between-laboratory precision, SP flow cytometry using BD Trucount tubes or BC Flow-Count fluorospheres showed less inter-laboratory variability than the DP comparators. [46,47] In addition, Denny et al. demonstrated improved inter-laboratory precision using DP Panleucogating compared with technologies which included DP or SP conventional flow cytometry [50]. External quality assurance data showed that the BD FACSCount, which is most commonly used a reference standard for CD4 assay evaluations, has within-laboratory and between-laboratory precision of 15% or less [38,39,42,46].

Limitations include the fact that we only included papers published in the English language, and we may have overlooked data because of this limitation. Limiting the search to the peer-reviewed literature may have overlooked robust evaluations conducted by national reference facilities or similar institutions.

It is important to consider that the reference standard technologies themselves are not perfect. Misclassification assumes that the reference result is accurate, i.e., the closest approximation to the truth. Thus, a result considered as a misclassification may in fact be correct. The reference technology if performed once may give a result of 340cells/ul, but if performed in duplicate using the same specimen may give results of 340 and 360. It may be important for the reference technology to be performed in duplicate and only when a concordant result is obtained around a threshold of 350 can it be used as the reference standard for the new test.

What constitutes an “acceptable” margin of error and misclassification probabilities around a threshold remain undefined and may vary among sites, depending on local factors such as the distribution of CD4 counts among asymptomatic patients, how often patients undergo repeat CD4 testing, and the implications of potential overtreatment (e.g., cost, long term risk of drug toxicity). However, given the move towards earlier treatment and the use of better-tolerated, less toxic drugs, misclassification that results in overtreatment may be more acceptable than would have previously been the case. It is relatively straightforward for national programmes to decide which technology best fits their needs based purely on cost and operating characteristics; it may be harder to decide what performance characteristics are acceptable, and harder still to obtain data on test performance to inform choice.

Given the potential for testing error, laboratory participation in EQA programmes and access to quality control (QC) reagents are essential. EQA information is not mentioned in the publications included in the review. Without this information, the proficiency of the laboratory staff performing the testing may have contributed to the errors and variation in addition to the assays themselves.