IntroductionScientific articles and popular press publications have revealed broad public skepticism about scientific findings (1). This attitude persists despite efforts to make research more accessible and immediate. The quality of behavioural research specifically, including its reproducibility (2), has also been called into question.
The efficacy of behavioural weight loss interventions in particular has been questioned. Benchmark studies such as the Diabetes Prevention Program (3), the Finnish Diabetes Prevention Study (4), the Da Qing Study (5) and the Look Ahead study (6) have found lifestyle interventions to be efficacious and associated with numerous long-term benefits, including sustained weight loss, improvements in metabolic measures such as blood pressure, fitness, glycemic control and cholesterol levels (6), and decreased incidence of type 2 diabetes. A large number of efficacy and effectiveness studies have attempted to replicate and/or adapt the interventions used in these studies across diverse settings and populations and using a range of different intervention modalities (e.g. individual counselling, group counselling, web-based, multi-method approaches), number of visits and treatment durations (7–10).
Although not all studies have observed statistically significant or clinically meaningful weight loss, a systematic review and meta-analysis systematic review of 80 studies with a minimum 1-year follow-up found that behavioural weight loss interventions aimed at reducing caloric intake and increasing caloric expenditure produce modest weight loss (7). The US Preventive Services Task Force recommends that clinicians screen all adults for obesity and offer or refer patients with body mass indexes (BMI) of 30 or higher to intensive behavioural treatment (Grade B recommendation), a recommendation that has been adopted by the American Academy of Family Physicians (11). Despite this evidence, behavioural weight loss interventions have been referred to as ineffective because of the high prevalence of weight regain and lacking methodological rigour in trials (12–15). In depth examination of the methodological quality of trials in this area might increase the veracity of findings within science and in public opinion.
Given the importance of quality in informing practice, policies and public perceptions, evaluation of the body of literature testing behavioural weight loss interventions from a quality perspective is critical to advance the field. The goal of this systematic review was to assess the methodological quality, of published studies testing the effectiveness of behavioural interventions for weight loss among adults and to examine whether quality was associated with report of a statistically significant weight loss outcome, strength of intervention effectiveness and sample size.
MethodsSearch strategyLiterature searches for behavioural weight loss trials published between January 1, 2009 and December 31, 2014 were conducted using PUBMED, MEDLINE and PSYCINFO search engines. Search terms included: ‘weight’, ‘loss’, ‘overweight’, ‘obese’, ‘intervention’ and ‘trial’. The reference lists of studies that met inclusion criteria (described below) were searched to identify additional articles.
Inclusion and exclusion criteriaStudy inclusion criteria included: (i) weight or BMI as a primary or secondary outcome; (ii) intervention that tested a behavioural approach to dietary and/or physical activity change for the purpose of weight loss; (iii) inclusion of an adult (age 18+), non-hospitalized or institutionalized sample, (iv) a minimum sample size of 100 total participants at baseline; (v) a minimum of 6 month follow-up period and (vi) publication in English in a peer-reviewed journal. Exclusion criteria included: (i) study aim was to evaluate efficacy of a pharmacological or surgical intervention; (ii) basic science study; (iii) policy and/or environmental intervention only; (iv) evaluation of an already implemented program; (v) survey development study; (vi) weight maintenance or weight gain prevention was the outcome and (vii) secondary/mediation analysis of a trial. For interventions that had multiple publications meeting inclusion criteria, only the article with the final data assessment point was reviewed.
Initial article review processTo determine whether articles met the study inclusion and exclusion criteria, an initial title review of all articles generated from the search was performed by one author. Two authors (SL and CH) then independently reviewed the abstracts of articles that passed initial review. Inter-rater reliability was calculated (93.5%). The authors met to discuss articles with discrepant codings to achieve consensus. Full text articles were pulled for those deemed to meet inclusion criteria based upon abstract review.
CodingThree authors (SL, CH, MW) developed the initial coding criteria and coding form. This form was pilot tested on five articles by four authors (SL, CH, MW, DE). These rating forms were reviewed by the study team item by item. Items that yielded any discrepancies were discussed to clarify meaning and wording. This process was repeated on a new set of five articles until 100% agreement was achieved on all data elements. A protocol document that described coding rules for each item was then created for use by coders. Each article had two reviewers. The reviewers met to compare ratings, code inter-rater agreement (IRA) for each item and discuss inconsistencies to come to consensus.
Coding criteriaStudy elements abstracted included intervention characteristics, study design characteristics and methodological quality indicators.
Intervention characteristicsassessed included the intervention delivery setting (health care, community organization, worksite, church, college, neighbourhood, academic/lab, combination) (IRA = 86.0%), behavioural target(s) of the intervention (diet, exercise or both) (IRA = 96.5%), intervention modality (group or individual; in-person, phone, web, other technology) (IRA = 100%) and number of intervention sessions offered/dose (IRA = 100%).
Study design characteristicsFactors related to study methodology included number of study conditions (IRA = 100%), comparison condition (IRA = 98.8%), level of randomization (IRA = 100%), sample size at baseline (IRA = 100%) and final follow-up assessment time period (IRA = 100%), and final follow-up assessment time period in months (IRA = 98.8%). Weight-related outcomes used in the study included change in weight (in both lbs. and kgs.), change in BMI, percent of weight change and achieving a specific weight loss percentage (typically 5% of baseline weight) (IRA = 86.2%).
Study effectivenesswas assessed in two ways. First, we abstracted whether or not a statistically significant result (p < .05) was reported for any weight-related outcome at the final assessment point (IRA = 98.8%). Second, within each intervention condition, baseline values and results at the final assessment point were abstracted for each weight-related outcome that was reported (IRA = 100%). Types of outcomes included were change in weight (in either pounds or kilogrammes), change in BMI, change in percentage weight loss and the percentage of the sample that achieved a pre-defined weight loss threshold (e.g. 5% of baseline weight). For intervention conditions that included a given outcome, we calculated the median and interquartile range across those results. Then, within each intervention condition that included change in weight or BMI, a Cohen’s D statistic was computed in order to standardize estimates across the multiple ways in which weight-related outcomes were assessed. Cohen’s D was calculated as the standardized mean change from baseline in weight or BMI. Mean change in weight or BMI at the last follow-up visit was divided by the standard deviation of the baseline measure. The effect size was very similar for studies reporting both change in weight and change in BMI. Because of this, when both were reported, we used effect size for change in weight in our analyses as this was the most commonly reported outcome measure.
Methodological qualityWe included ten indicators reflecting key components of methodological and/or reporting quality (16). Quality indicators were an amalgam of known standards for rating intervention studies that were adapted by the study team to be specific for behavioural weight loss interventions (17–19). Individual indicators, rather than a summary score, were used so that specific elements could be assessed (17,18). The quality indicators selected therefore covered these two areas and were as follows:
Study design was assessed (randomized controlled trial (RCT), non-randomized design with control group, single arm design). RCTs were considered the gold standard. Level of randomization (i.e. individual participant, provider, site) was also assessed (IRA = 100%).
We assessed the measurement approach used to assess weight and/or height data. Options included: measured by research staff, obtained from medical or other records, self-report or a combination of these (IRA = 100%).
Whether or not power/sample size calculations were reported was assessed (yes/no) (IRA = 95.3%).
We defined ‘intent-to-treat’ (ITT) analysis as whether or not participants’ data were included in the intervention condition to which they were randomized in the analysis, regardless of whether or not they participated in/attended that intervention/program/ treatment. ITT was considered independently of how missing data at follow-up was handled. Because of discrepancies in how investigators define ITT (20–23), text was reviewed for a clear definition of ITT consistent with the one used in this study and CONSORT-type diagrams and denominators in tables were reviewed to establish whether the ITT criteria was met. ITT was classified as yes or no (IRA = 95.3%).
Loss to follow-up rate was assessed by determining the percentage of individuals at baseline who completed assessments at the final follow-up point. We used the most conservative approach, not taking into consideration specific study designed exclusion criteria (e.g. some studies excluded participants who became pregnant or began taking weight gain medications), we used this approach to apply a consistent definition across studies (IRA = 93.5%). While there is no universal consensus on what loss follow-up rate results in biassed study results, we considered less than 25% to indicate high quality (24), as suggested by the Methodological Quality Rating Scale.
Missing data strategy refers to the approach(es) used to handle missing data at follow-up assessments. Classifications were based on standards described in 2010 by the National Research Council (25) and included: no or minimal (>95% of the baseline sample included in the final follow-up time point) missing data (26), multiple imputation, generalized estimating equations or other non-parametric model-based approaches, inverse probability weighting, maximum likelihood/parametric models, imputed weight gain carried forward; single imputation (i.e. last or baseline carried forward) and complete case analysis with and without comparisons to baseline sample). We additionally assessed whether sensitivity analyses were performed using more than one method (IRA = 96.5%).
The sampling strategy used to enrol participants was classified as being from a known population (i.e. having a denominator) or not (IRA = 91.9%). Among those studies with a known population, we assessed whether comparisons were made between the included sample and the known population (IRA = 100%).
We assessed whether data were reported in participant treatment receipt or adherence was reported (IRA = 97.7%).
Last, we assessed whether attempts to ensure intervention fidelity were described or reported (IRA = 95.3%). We additionally assessed the reporting/methods used for fidelity which was categorized as none stated, followed a standardized protocol, or quality review by study staff.
AnalysisThe distribution of intervention characteristics, study design characteristics and each of the nine methodological quality indicators were computed among the included studies. To determine whether or not methodological quality was associated with intervention effectiveness, bivariate comparisons were made to assess the associations of the sum of quality indicators with whether or not the study reported a statistically significant outcome at the final assessment point (t-test) and the strength of association/Cohen’s D statistic (using Pearson’s correlation coefficient). The latter was computed only among those studies that included change in weight (lbs. or kgs.) or BMI as an outcome. The bivariate association of the study sample size and the sum of number of quality indicators was also assessed (one-way ANOVA).
ResultsStudy selectionA total of 5,499 non-duplicate articles were retrieved from the initial search strategy (Fig. 1). Of these, 4,765 were excluded based on title review; 734 abstracts were reviewed. Of these, 543 were excluded and 191 articles were selected for full review. An additional six articles were identified from reference review for full review, resulting in 197 articles for full review. Of these, 107 were deemed ineligible. The most common reasons for ineligibility included: follow-up period of less than 6 months (n = 30), results presented in another publication (n = 17) and methods paper (n = 9). A total of 90 articles were included in this review; 156 unique interventions were tested in these 90 studies. A list of these articles is included in the online Appendix.
Study characteristicsTable 1 describes the intervention characteristics of the included studies. Most targeted both diet and physical activity (93.3%). The most common primary intervention delivery settings were health care (38.9%) and academic research labs (36.7%). A range of intervention delivery modalities were used, with many studies reporting multiple modalities within a single intervention condition and/or across intervention conditions. More than half of the interventions (51.3%) offered more than 20 sessions.
Table 2 presents a summary of the design characteristics and outcomes of the included studies. The majority included two treatment conditions (66.7%). A wide variety of comparison conditions were used, with comparative effectiveness (i.e. comparing one intervention approach to other) (48.9%) (either with or without a no intervention control group), usual care (22.2%) and materials only (20.0%) being the most common. Most studies included more than one outcome measure. The majority of the 90 studies included change in weight and/or BMI as outcomes (87.7%). Among those that did not include such a measure, outcomes included proportion of participants who reached a certain threshold of weight loss (typically 5%) (1.1%); percentage of weight loss achieved (8.9%); or both a threshold of weight loss and weight loss percentage (2.2%).
One hundred forty of the 156 unique interventions included in the 90 studies presented change in weight as an outcome (in pounds or kilogrammes). Converting each of these to pounds, median weight loss in the 140 interventions was −5.13 (interquartile range (IQR) = −10.14 to −2.55). The median change in BMI, among the 86 interventions that assessed it, was −.80 (IQR = −1.8 to −.30). With respect to change in percentage weight loss (included in 46 interventions), the median was −4.7% (IQR = −1.68% to −6.4%). Among studies that included proportion achieving a specified weight loss threshold, a median of 36.0% of participants did so (IQR = 21% – 49%). Cohen’s D could be calculated for 76 of the 79 studies that assessed change in weight or BMI; these included 133 unique weight loss interventions. Mean Cohen’s D statistic was −.26 (sd = .24; median = −.21). Almost half (47.4%) demonstrated no effect; 36.1% a small effect; 12.8% a medium effect and 3.8% a large effect.
Methodological qualityA summary of the methodological quality indicators is presented in Table 3. Quality indicators that were most commonly utilized included a randomized controlled trial design (100% (individual level randomization 95.6%, site level randomization 4.4%), objectively measured weight/height (96.7%) and description of an ITT analytic approach (86.7%). Most studies did not describe fidelity assessments (65.6%), with 15.5% reporting following a standardized protocol and 18.9% reporting quality review by the study team. Sixty-two percent (62.2%) of studies had a follow-up retention rate of more than 25%. The majority of studies reported using a strategy to address loss to follow-up (either primary or sensitivity analysis) or having minimal missing follow-up data (69.9%). A variety of approaches to loss to follow-up were used; the most common were last/baseline value carried forward (35.6%) and multiple imputation (24.5%), either alone or as a sensitivity analysis to a primary analysis that used a complete case approach. Overall, the mean number of quality indicators achieved was 6.3, with a median of 6 (range 2–9).
No differences in methodological quality were observed between studies that reported a statistically significant reduction in a primary outcome at final assessment point (mean number of quality indicators = 6.3 (SD = 1.5)) and those that did not (mean = 6.0 (SD = 1.4); p = .31). We also found no association between methodological quality and effect size (Pearson’s correlation coefficient = 0.07) among those studies for which a Cohen’s D statistic could be calculated. No differences in quality indicators were observed by study sample size (mean number of quality indicators for sample size 0–199 = 6.1 (SD = 1.4); 200–399 = 6.3 (SD = 1.5); 400 or greater = 6.2 (SD = 1.4); p = 0.88).
DiscussionIn this study, the methodological quality of contemporary trials testing the efficacy and effectiveness of behavioural weight loss interventions among adults was examined, with an emphasis on key indicators of internal and external validity. A total of 90 articles using a wide range of intervention modalities, comparison conditions and conducted in a variety of settings were included. Overall, the studies were of moderate to high quality (median of six out of nine quality indicators), with considerable variation across indicators.
Internal validity indicators that were assessed included key features that have potential to bias the results of weight loss trials. Most studies used a protocol that required weight and height to be objectively measured, a positive finding in light of the recognized bias in the use of self-report or other methods in attaining these measures (27,28). Most studies additionally clearly stated that ITT analysis was completed, that is they were included in the analysis in the intervention group to which they were assigned regardless of their participation in or adherence to the intervention. Studies that did not use an ITT approach typically excluded participants who did not meet a specified intervention participation benchmark, which has potential to overestimate intervention effects.
While the majority of studies used a strategy to address loss to follow-up, more than one-fifth did not, instead using a complete case analysis approach. This can lead to reduced power and selection bias, particularly if loss to follow-up is differential across conditions or related to weight loss. Traditional clinical trial methodology additionally call for the most conservative approach to addressing loss to follow-up be used in order to decrease the probability of committing a type 1 or 2 error. In this review, we found that single imputation methods, such as last observation carried forward and baseline observation carried forward were most commonly used, which has been true in other areas. Such approaches, however, are typically not recommended as a primary approach for handling missing data because they can introduce bias (25). This may be particularly true with weight-related outcomes. On average, American adults are estimated to gain 1 to 2 pounds per year (29). Thus, baseline and last value carried forward approaches, the mostly commonly used in the studies included in this review and the easiest to implement statistically, may not be the most conservative approach. Another potential loss to follow up strategy is to impute an average weight gain to coincide with estimated population weight gain. However, only one study included in this review used this approach. More advanced statistical approaches such as multiple imputation, model-based approaches, while more computationally intensive, are increasingly considered a gold standard. In a meta analysis of pharmaceutical randomized controlled trials published between 2000 and 2006 that assessed weight loss or weight gain, Elobeid and colleagues compared various loss to follow-up analysis approaches on trial results. The authors concluded that multiple imputation and model based approaches are the most appropriate for weight loss trials (30). Less than 30% of studies included in this review used such a strategy.
There have been calls within the field of behavioural medicine and related disciplines to increase reporting of indicators that relate to project implementation and external validity in an effort to better promote the translation of research into practice and enhance the public health impact of research-developed interventions (31–33). In this review, only 32.7% of studies reported on intervention fidelity, despite these calls to action and the availability of frameworks and standards for fidelity reporting (34–36). Only 17.3% including objective reviews such as reviewing sessions in person or via audiotape for protocol adherence; other methods reported included describing training of study staff, use of standardized protocols, selecting intervention staff with specialized degrees and supervision of interventionists. It is possible that quality control and fidelity checks were in place, but not reported in many studies, particular given journal restrictions. Intervention adherence rates were reported by 76.7% of studies. We found that only 41.1% of studies samples from a known population, with the majority enrolling convenience samples. Of these, only one study made comparisons between the known population and the included sample. Thus, most studies were not designed to assess reach, and among those that were, only one attempted to describe the success in achieving a representative population. The results of these three indictors (fidelity, participation and reach/representativeness) are similar to those reported in a systematic review of childhood obesity prevention interventions published from 1980 to 2008 (37).
We found that methodological quality was not associated with whether or not a statistically significant weight loss result was reported, effect size, or sample size. This indicates that differences in methodological quality are not driving inconsistencies in translation of efficacy findings to effectiveness studies. Instead, such differences are likely a function of intervention design, intensity and/or delivery and the associated challenges of achieving behaviour change across diverse populations and settings. Mean weight loss in the studies included was 6.79 pounds (sd = 6.70) relative to 14.33 (sd = 10.36) in the Diabetes Prevention Program and 9.23 (sd = 11.24) in the Finnish Diabetes Prevention Program, both of which found significant associated reductions in diabetes risk of 58% (38). While weight losses in effectiveness studies were 47–74% of those of these larger efficacy trials and the degree to which this magnitude of weight loss affects diabetes risk is unknown, it would seem premature to dismiss these interventions as ineffective. Two main criticisms have been launched against behavioural weight loss interventions. The first is that weight regain is extremely common and is an indicator of treatment failure (13,14). Although weight regain is common over the long-term, the Diabetes Prevention Program found that 10 years after the lifestyle intervention, in spite of average weight regain of 71%, reduction in incidence of diabetes was 34% relative to control (39). The Da Qing Diabetes Study found a reduction in diabetes incidence of 43% relative to control at 20 years although did not report weight regain rates (40). Weight loss may not be the best measure of impact of lifestyle interventions, although it is the most feasible measure of impact in shorter term effectiveness studies, which poses a challenge to this field. It should also be noted that termination of treatment for many conditions results in a return of symptoms. Lifestyle interventions may not be a permanent cure for obesity, but this is an extraordinarily high bar for any preventive intervention (41). Chronic care models are certainly needed. The second criticism of weight loss interventions is that they lack methodological rigour (12). Our findings show that recent trials are of moderate to high quality.
This study must be viewed in the context of its limitations. Only studies published in English were included. Publication bias is a well-known limiting factor in synthesizing research evidence that could impact the conclusions of this study (42). Studies with positive or significant results are more likely to be published in the scientific literature, and selected outcomes with null findings are more likely to be intentionally excluded from published reports. We additionally restricted articles to those with at least 100 people and six months or more follow-up time. The former was done to eliminate pilot type studies which typically are not designed with maximal methodological rigour and are not powered to find statistically significant results. The later restriction was improved because we were interested in studies designed to assess weight loss beyond the short-term impact of the intervention. We examined whether studies were statistically significant, rather than using an indicator of whether or not clinically meaningful weight loss was achieved. This was done because vast heterogeneity in how results were reported made it impossible to operationalize clinically meaningful results. Only a select number of quality indicators were included, indicators were limited to those deemed by the study team to be most likely to impact study bias. Lastly, we were only able to assess what was reported in the included studies. It is possible that a given study could have included/addressed a given quality indicator and the investigators chose not to report it for reasons such as journal word count restrictions, response to journal editors and/or reviewers and the investigators’ simply not perceiving it to be important.
In summary, this systematic review of behavioural trials testing weight loss interventions found this body of literature to be of high quality for specific quality indicators, including study design and measurement. However, specific areas in need of improvement were identified. This review identified a need for utilization of more methodologically rigorous statistical approaches to address loss to follow-up in such trials, more representative sampling approaches and better reporting of fidelity, in particular. These are potential areas of improvement for this field of research that are relevant to investigators, journal editors and reviewers.