Self-reported adherence, based on detailed questionnaires, is considered the most feasible method for assessing adherence in resource-poor settings [1, 2]. Self-reported data are relatively low cost to collect and do not involve complicated field logistics or invasive procedures such as blood sampling. However, despite its many advantages, self-reported adherence is prone to several sources of bias [2–4]. Data may be subject to recall bias if patients do not accurately recall their treatment history, including the number of pills taken, day and time of each dose, and when the full course was completed. Social desirability bias may also occur if patients provide perceived expected responses in order to avoid being seen as negligent.

Several other methods have been used to complement or even replace patient recall. Many studies verify self-reported adherence by counting the number of pills remaining in packaging, although this is not always accurate as patients may remove pills without taking them, or packaging may not be available for inspection [5, 6]. Electronic methods of assessing adherence have been used extensively in chronic diseases, mostly in the form of pill containers with caps that record the day and time the container is opened, such as Medication Events Monitoring Systems (MEMS) [4, 7]. While not feasible for routine clinical practice in many parts of the world, MEMS have been used to validate other adherence measures in studies of adherence to antimalarial drugs [8, 9], tuberculosis treatment [10], and antiretroviral therapy [11].

Artemisinin-based combination therapies (ACTs) are first line treatment for malaria in most endemic countries and are increasingly obtained by patients seeking treatment in both public and private health sectors. Good patient adherence is required to maximise their clinical impact and minimise the rate of development of drug-resistance [12–14]. A number of studies assessing adherence to ACTs have been conducted in recent years, with results showing that anywhere between 39% and 100% of patients can be considered adherent [5, 6], reflecting both genuine differences in adherence as well as variation in study design and measurement methods. The majority of these studies relied on self-report with or without pill count. One study in Malawi used MEMS containers and found 100% adherence to artemether-lumefantrine (AL) by self-report and lower adherence (92%) by MEMS [8]. However, ACTs are now typically dispensed in blister packs designed to improve adherence [15] and look considerably different than MEMS containers. This may result in over-estimated adherence, as patients using MEMS are likely to be aware that their adherence is being monitored and their experience is no longer comparable to that of patients receiving unit doses in their customary packaging.

In this study, set in southern Tanzania, the validity of self-reported patient adherence to AL was assessed using a novel customised electronic monitoring device—termed smart blister packs—that looked identical to regular AL packs, but contained a device that registered the date and time each pill was removed from the pack. This is the first study to our knowledge to employ this technology under routine conditions to investigate adherence to antimalarial treatment.

Interviews were conducted with 1,204 patients from 117 outlets (five ADDOs were closed or refused to participate). Blister packs were not available for collection from 256 patients (21%). Smart blister packs that had been damaged or from which data were not extractable were collected from 252 patients (21%). In total, data were extracted from the blister packs of 696 patients (58%) (Fig 2). Due to infeasible dosing patterns or errors in smart blister pack technology, 55 patients were excluded from the analysis of timely completion, but not the analysis of completed treatment, since pills remained in their blister packs. For self-report, this applied to 18 patients who reported taking a subsequent dose before an earlier dose. For smart blister pack data, this applied to 10 patients with timestamps recorded before the treatment was dispensed or after the pack was collected, and 27 patients with a different number of pills observed by the study team at scanning than the number read by the software.

Table 1 shows characteristics of patients with both smart blister pack data and self-reported data available and patients with only self-reported data available (results comparing patient characteristics and adherence to AL across outlet types are presented elsewhere [21]). Based on the recommended age groups for AL blister packs in Tanzania, 27% of the 696 patients with both smart blister pack and self-reported data available were in the youngest age group (under three years), 24% were three years to under eight years, 6% were eight years to under 12 years, and 43% were 12 years and above. Fewer patients with only self-reported data available were under three years (21%) and more were 12 years and older (50%). Taking the first dose at the outlet was reported by 23% of patients with both smart blister pack and self-reported data available and 17% with only self-reported data available (p = 0.0281), but all other characteristics were similar for patients with and without both types of data. Given that some differences were observed, comparisons between methods were restricted to patients with both self-report and smart blister pack data available.

This paper examines the validity of self-reported adherence in comparison with smart blister packs that recorded the day and time pills were removed from packaging. Timely completion (37% by self-report and 24% by smart blister packs) was much lower than completed treatment (64% by self-report and 67% by smart blister packs). No difference was observed between self-report and smart blister pack data for the percentage of patients completing treatment, but timely completion was lower when assessed using smart blister pack data (OR = 0.36, 95% CI: 0.29, 0.42, p<0.0001). Smart blister pack data showed that, even among patients who completed treatment, 33% did not take the correct number of pills for all doses, and 60% did not take each dose at the correct time interval.

This study has several limitations. First, recovery of blister packs might have been higher had dispensers and patients been more aware of the aims of the study. However, to avoid artificially increasing adherence, we intentionally did not tell dispensers or patients that the objective of the study was to assess adherence. Secondly, dispensers did not always accurately record the time that drugs were dispensed, despite daily visits by the study team and attempts to clarify times that were unclear. We therefore concluded that the data were not sufficiently robust to assess the timeliness of the first dose.

For both self-report and smart blister pack data, we allowed a margin of error around the correct time intervals when assessing timely completion. Clocks are not commonly used in rural Mtwara, and Swahili times of day which each cover several hours were therefore used for self-reported timely completion. Recorded timestamps plus or minus four hours were used to define acceptable limits for smart blister pack data. Among patients with timely completion by both methods, Swahili times of day corresponded well with timestamps, so it is unlikely that the definitions of timely completion affected the difference between methods. However, more stringent definitions for both methods might have resulted in lower levels of timely completion. Variability in adherence definitions has been frequently noted as a challenge for comparing adherence results across studies [5, 6, 25].

Self-report and smart blister pack data have various advantages and disadvantages for assessing patient adherence to ACTs. Self-reported adherence is collected through interviews, which are relatively inexpensive and are a widely accepted method of collecting data in many populations, including in southern Tanzania. Nonetheless, collecting accurate self-reported data can be challenging. Patients may have provided expected answers on the number of pills taken for each dose, or not remembered, and interviewers reported that patients often seemed confused about when and how each dose was taken. Some patients reporting taking more pills in total than the number in the blister pack, and some reported taking a subsequent dose before a previous dose. In this study, specific questions were asked about each of the six intended doses. However, patients may have taken pills grouped in a way that differed from the intended doses in the blister packs, and may therefore have taken more than six actual doses, as smart blister pack data indicated was the case. If any actual doses beyond the six intended are not captured by self-report interviews, this could underestimate the percentage completing treatment. In order to avoid some of these challenges, data collection tools need to be improved and evaluated, for example, including both open and closed-ended questions to identify actual doses taken [6].

As part of both the completed treatment and timely completion definitions, self-report was verified by counting pills remaining in packaging for the 83% of patients that presented blister packs. Very few patients (9/696) reported completing treatment but had pills remaining, similar to the high concordance between self-report alone and pill count described in a review of adherence to medication across diseases [3]. While this suggests that definitions incorporating pill counts may not differ from self-report alone, the patients who did not present blister packs reported higher completed treatment than those who did present blister packs (85% vs. 68%). This suggests that requesting blister packs may reduce over-reporting of adherence, unless patients for whom blister packs were not available were more likely to have finished packs and disposed of them. Blister packs were requested at the end of the questionnaire, which means patients could have removed pills from the pack after completing the interview but prior to presenting the pack.

Compared to smart blister packs, self-report generally over-reported timely completion, consistent with most other studies of electronic monitoring of adherence across diseases [4]. However, 27% (41/155) of patients with timely completion by smart blister pack data did not report timely completion (S2 Table). This is likely due to patient confusion when reporting dose history, which was noted frequently by study staff, or possibly removing pills for later consumption. Studies of adherence to tuberculosis treatment and antiretroviral therapy have also documented that some patients non-adherent by self-report were adherent by electronic pill containers [10, 11]. Our smart blister pack data revealed how some patients took pills in more than six actual doses, or removed a large number of pills at once, including removal of all pills remaining in the packaging on the date of the interview—possibly at the time of the interview.

The smart blister packs were easy to assemble, the software and portable scanner were straightforward to use, and the packs were designed to look identical to regular AL packs commonly available in Tanzania. However, they had a slight bulge at the top of the package for a small chip that stored data. Some patients noticed this and opened up the packaging to investigate, damaging the capacity to record and extract timestamps. Several patients were alarmed when the chip was discovered, requiring the study team to provide full explanations to participants and village leaders. Other blister packs were destroyed when children were allowed to play with them after completion of treatment, or when dropped in water or fires before, during, or after taking pills.

Although smart blister packs accurately recorded when pills were removed from the packaging, it was also possible for a timestamp to be recorded for a pill if pressure had been applied and a pill was partially removed, even if the seal was not obviously broken. While it was evident when timestamps were recorded before the treatment was dispensed to the patient or after the pack was collected, this could also have occurred in the middle of treatment, which could have altered timely completion estimates, though this is not thought to have occurred frequently. In addition, for some patients, the number of pills observed by the study team at scanning did not correspond with the number read by the software. Future smart blister pack designs should be improved to reduce the chances that timestamps would be recorded when pills weren’t completely removed and to reduce the bulge in the packaging so that patients are less likely to notice and tamper with the packaging.

Another approach to getting insights into adherence is through the measurement of drug levels in patients’ blood during follow-up. In the case of ACTs, this has been approached by measuring the concentration of the non-artemisinin partner drug since the artemisinin component is absorbed and eliminated rapidly [26]. We collected blood spots on filter paper for the assessment of lumefantrine concentrations. However, analysis 19–24 months after collection found lumefantrine concentrations below the lower limit of quantification, reflecting the need for filter papers to be stored at appropriate temperatures and ideally analysed within 4–6 weeks post-collection [27].

Moreover, studies of adherence to AL have not found significant differences in blood lumefantrine concentrations between patients considered adherent and non-adherent by self-report [8, 28–30]. Some studies have reported significant differences in lumefantrine concentrations (usually measured 7 days after initiation of treatment) between supervised versus non-supervised patients [31–33]. These and other studies have not reported differences in treatment failure based on cut-offs of 280 ng / ml [29, 31–33] or 175 ng / ml [8], which have been previously found to predict recrudescence [12, 26]. In addition, lumefantrine is known to have high inter-individual metabolic variation, with factors such as weight, age, pregnancy, and fat intake affecting absorption [34–36]. As a result, while biological measures of lumefantrine concentration might provide some additional objective information about adherence, they can be difficult to obtain, process and interpret in follow-up studies.

Finally, methods for measuring adherence should be based on levels and components of adherence that are important for effectiveness. For example, it might be expected that completing treatment is more important than exact timing of doses. The recommended regimen for AL was developed in early trials showing that a six-dose regimen of AL taken twice per day, with the second dose taken after eight hours, resulted in higher cure rates than a four-dose regimen [34, 37, 38]. However, it is unclear how strictly dose intervals must be adhered to in order for treatment to be effective. Patients who obtain ACTs in the evening may be less likely to adhere to correct dose timing than patients who obtain ACTs in the morning, and it is unclear to what extent this matters. As adherence needs to be defined and measured depending on what is required for drug effectiveness, the importance of exact dose timing must be clarified.

Accurate measurements of patient adherence are important for developing strategies to assure the effectiveness of ACTs. While self-reported data along with examination of packaging might be sufficient to assess completion of treatment, patient reports of timely completion appear to be affected by both recall and/or social desirability bias. Smart blister packs provided slightly lower, and potentially more accurate, estimates of the number of pills taken for each dose and the time intervals between doses. In settings where data on dose timing are considered important for clinical outcomes, smart blister packs may be a useful tool, though innovations to make them more robust and discreet would be useful. Improved methods for collecting self-reported data are also likely to enhance the accuracy of measured patient adherence to treatment. Finally, a clearer rationale for what is considered adequate adherence is required.