IntroductionBackgroundWhen the possible causes of a patient’s condition vary geographically, knowledge about local scale disease incidence could help steer clinicians towards the most likely diagnosis. Children with peripheral facial palsy pose a diagnostic challenge, because optimal management at the point of care requires correctly identifying the etiology for the palsy. Rapid point-of-care testing for Lyme disease is not available, so diagnostic test results, if ordered, often are not known for several days, leaving clinicians to choose a treatment strategy without confirmatory serology. Over-diagnosis of Lyme is associated with excessive antibiotic use, and under-diagnosis with progression to more complications. At one time otitis media accounted for most identifiable cases of facial palsy in children.1 Infections with Borrelia burgdorferi have increased over the past several decades, so Lyme disease, the most frequently reported vector-borne disease in the United States, now accounts for a substantial proportion of cases in endemic areas.2–4 Antivirals and corticosteroids may be helpful in adults with facial palsy.5 For children with Lyme facial palsy, early initiation of appropriate antibiotics is the optimal strategy, and according to the American Academy of Pediatrics Red Book Committee on Infectious Diseases, steroids should not be given.6
ImportancePrediction rules traditionally factor in historical elements, physical exam findings and sometimes seasonality to identify the correct cause of the facial palsy, but to date, none of the rules have incorporated residential location as a predictor.7 Epidemiological context– the recent regional incidence of a disease, often calculable from clinical or public health datasets – may be an important predictor, in the absence of timely diagnostic data.8, 9 Even within endemic areas, Lyme disease incidence varies by location and season, in part because the irregular local and regional distribution of ticks depends on landscape ecology, and micro- and macrometerologic conditions.10, 11 While prior studies have considered ecologic and entomologic risk to generate community-level Lyme prevention recommendations and vaccination strategies,12–15 this analytic approach has not yet extended to manipulating formal management algorithms for symptomatic patients by combining geographic risk with clinical features.
Goals of this InvestigationTo optimize management of peripheral facial palsy in children, clinical decision models would incorporate local epidemiological risk to differentiate Lyme disease from other etiologies. Taking a novel approach, integrating epidemiological information about location and season with traditional clinical variables, we sought to create a model to improve diagnostic accuracy and management of children with peripheral facial nerve palsy. We hypothesize that quantitative use of the patient’s geographic risk of Lyme disease would improve the accuracy of diagnosis.
MethodsDesign, setting and subjectsOur sample was a retrospective cohort of children under 20 years old presenting to the emergency department (ED) of Children’s Hospital Boston, a large, urban tertiary care hospital, from 1995–2007. The study site ED volume exceeds 50,000 patients annually. We only included children residing in Massachusetts.
Selection of ParticipantsED visits of patients with peripheral facial nerve palsy were identified by a computer-assisted key word screening tool and regular expression matching from all ED visits at the study site during the study period.16 We included only those children with facial palsy who were evaluated for Lyme disease (either by obtaining Lyme serology or by the presence of erythema migrans rash). Patients were excluded from the analysis if they had any of the following characteristics causing facial palsy: congenital facial palsy, known central nervous system malignancy, known history of herpes simplex virus, surgery near the facial nerve within one week of presentation, or Todd’s or more generalized paralysis, including hemiparesis.
Case definitionA child was defined to have Lyme disease according to the CDC definition: presence of erythema migrans lesion or serologic evidence of infection with Borrelia burgdorferi via the two-tiered testing strategy.17 Children were only classified positive in our study if the Western blot was positive using the laboratory reference standards. Offsite commercial laboratory personnel (ARUP – Salt Lake City; Immugen – Norwood, MA) performed serologic testing for B burgdorferi. Patients with positive enzyme-linked immunosorbent assay and negative Western blot or no Western blot performed were not considered positive for Lyme disease.18
Predictor variables and data collectionDemographics, onset and duration of symptoms, clinical features, laboratory data and treatment data were collected for each patient via comprehensive chart review by two investigators specializing in pediatric emergency medicine (LEN, ADT). Signs and symptoms included headache, fever, muscle aches, joint pains, rash and potential exposures such as tick bites. Laboratory data were reviewed for Lyme test results. Treatment data included type and duration of treatment with antibiotics or steroids. To assess inter-rater reliability, an independent abstractor specializing in pediatric emergency medicine (AMF) reviewed eight percent of charts chosen at random.19, 20 Candidate predictors with kappa statistics with a lower limit of 95% confidence interval of > 0.4 were considered for the multivariate analyses.21 Visit date and county of residence for each patient were obtained from the chart review. County-level annual Lyme incidence was calculated from available public health surveillance data from the Massachusetts Department of Public Health Office of Integrated Surveillance Informatics Services.2, 22–24 These data were used to calculate the average Lyme incidence over the prior three years in the home county for each patient. For example, for a patient presenting from Essex County in 2004, the incidence in that county was averaged from 2001–2003.
Building the decision modelsThree decision models were built with clinical and epidemiological variables: 1) Clinical model – candidate predictors included traditional elements – data on demographics, history and physical exam; 2) Epidemiologic model – candidate predictors included the timing of presentation (month or season) and the incidence variables associated with the county of residence; and 3) Contextualized model – variables not included in the prior two models still qualified for inclusion into this model, which combined clinical and epidemiological predictors.
Univariate and multivariate analysisUnivariate and multivariate analytic techniques were used to identify predictors of Lyme disease among patients with peripheral facial palsy. Significance of association of categorical variables with Lyme disease was tested by Chi square. Continuous variables (i.e. average county incidence of Lyme disease in prior 3 years) were dichotomized at categorical cutoffs (e.g. average incidence > 20 cases/100,000 people). Recursive partitioning was used to identify thresholds for testing univariate and multivariate associations.
In the multivariate analyses, candidate variables were entered into a backward stepwise logistic regression to identify independent predictors of patients with Lyme disease. P value cutoffs for entry and departure for the multivariate regression models were 0.25 and 0.10, respectively. The final models contained variables where p<0.05.
Several seasonal variables were considered independently for entry into the models. A range of cutoffs was considered to define patients who presented in “Lyme season,” (June–October, May–December, June–November), because “Lyme season” varies by geography, climate, suitability for tick populations and annual trends.25 For the spatial variables, the annual county Lyme disease incidence in each of the prior three years and the overall three year average incidence were considered as independent predictors. Recursive partitioning was used to identify a cutoff for the three year average Lyme incidence in the county of residence, and this cutoff was considered as an independent candidate predictor. Final models underwent bootstrap validation. Predictors selected in over 50% of 1000 bootstrap analyses were retained in the final models.26–28
Methods of measurement of model performanceSensitivity, specificity, positive and negative predictive values, and area under the ROC curve, were used to compare performances of the models. Actual management by pediatric emergency medicine experts was compared to management guided by the decision models. Correct management of Lyme facial palsy was defined as use of a correct antibiotic for a correct duration and omission of corticosteroids and antivirals, as defined by the expert panel in the American Academy of Pediatrics Red Book Committee on Infectious Diseases.6
The Committee on Clinical Investigation of Children’s Hospital Boston approved the study.
ResultsFrom 1995–2007, there were 609,671 visits to this emergency department for patients under age 20 years. Table 1 displays the characteristics of the 264 patients (0.04% of all ED visits) who presented with peripheral facial palsy, were evaluated for Lyme disease and met study criteria. Patients evaluated for Lyme disease (n=264) were similar to those not evaluated for Lyme disease (n=156) with respect to age, gender and presence of fever and were more likely to have headache (28% vs 12 %, p=0.001) and present during Lyme season (49% vs 31%, p=0.001). The patients came from the nine Eastern-most of the 14 counties in Massachusetts. Figure 1 shows county-level average incidence of Lyme disease for Massachusetts over one three year period of the study.
Development of clinical decision modelUnivariate analysisPatients with Lyme were more likely to be male, have a history of fever, headache, systemic symptoms like myalgias and arthritis and no history of trauma to the face or head (Table 1). There were no significant differences between those with and without Lyme for age, neck pain, or otitis media. Exposure to tick bite was not captured in the vast majority of charts, and so could not be considered for the analyses.
Multivariate analysisIn the clinical model, headache (OR 4.4, 95% CI 2.2–7.5) was the most significant predictor of Lyme facial palsy, followed by fever (3.3, 1.6–7.1) (Table 2). Presence of either of these two predictors identified children with Lyme with 60% sensitivity, 79% specificity, and area under the ROC curve (AUC) of 0.71. The PPV was 61% and the NPV was 79%.
Development of models incorporating epidemiological context: Selection of seasonal variableUnivariate analyses were conducted using a range of cutoffs to define Lyme season. Recursive partitioning identified candidate cutoffs for Lyme season. Patients with Lyme disease were more likely to present during any of the defined Lyme seasons. The Lyme season defined as “June–November” showed a stronger association for Lyme than “June–October” or “May–December” so for further analyses, June–November was used as Lyme season.
Selection of spatial variableUnivariate analysis was used to examine associations between Lyme disease and Lyme incidence rates in the patient’s home county. Recursive partitioning identified cutoffs to classify 3 year county average incidences as high or low risk. The low risk cutoff occurred when the average three year Lyme incidence for a county was less than four cases/100,000 people. Annual incidence and three year average incidence were associated with Lyme disease, but the cutoff incidence of >4 cases/100,000 people was the strongest spatial predictor, and was retained as the spatial predictor for the rest of the analyses.
The best epidemiological model contained two variables—Lyme season (June–November) and high-risk home location (three year average county-specific Lyme incidence > 4 cases/100,000 people). Lyme season (OR 25, 95% CI 8.6–107) and high-risk home location (20, 7.4–68) were both very strong predictors with odds ratios above 20 (Table 2). The AUC for this model was 0.84.
The contextualized model considering all clinical and epidemiologic variables regardless of whether they entered into the previous models contained four variables: fever (OR 3.9, 95% CI 1.5–11), headache (OR 2.7, 1.3–5.8), Lyme season (OR 25, 8.3–113) and high-risk home location (OR 18, 6.5–69) (Table 2). The AUC for this model was 0.89. This model was 100% sensitive and 24% specific with a PPV of 42% and NPV of 100%.
ValidationAll predictors from the multivariate analyses were validated by the bootstrap method and retained in the final models. High-risk location was selected in over 99%, Lyme season in over 97%, fever in over 81% and headache in over 77% of 1000 bootstrap analyses.
Measurement of model performanceAdding epidemiologic factors (seasonal and spatial variables) to the clinical model improved the AUC from 0.71 to 0.89, whereas adding clinical factors to the epidemiological model improved the AUC more modestly, from 0.84 to 0.89. Figure 2 illustrates the risk of Lyme facial palsy based on the presence of high-risk predictors. Of the 264 patients in the study, 134 presented from high-risk locations during Lyme season, and 87 (65%) had Lyme. In contrast, 7/130 (5%) patients who presented without both high risk location and season were positive for Lyme. A total of 69 patients presented from high-risk locations during Lyme season without either clinical predictor (fever or headache), and 35 (51%) of these patients had Lyme. Of the 65 patients who presented with fever, headache or both, in conjunction with high-risk location and Lyme season, 52 (80%) had Lyme. The combinations that included both season and location identified 87/94 (93%) of Lyme cases. Finally, none of the 42 patients without any of the four identified risk factors had Lyme disease.
Comparing clinician performance with decision modelsWe compared the proportion of children with facial palsy empirically treated with the appropriate medications by attending physicians in the pediatric emergency department with hypothetical outcomes generated by the three models. These physicians treated 68/94 (72%) Lyme disease patients with the correct type of antibiotics and without steroids or antivirals. The epidemiologic and contextualized models did not miss any cases of Lyme disease.
DiscussionTo date, clinical decision rules have relied on clinical factors and to a much lesser extent, seasonality. In the case of Lyme disease, clinicians may informally consider exposure and location when determining the cause of facial palsy, but there are currently no mechanisms that formally facilitate integration of this important contextual information. To the extent that clinicians use contextual epidemiological information to help guide decision making, they tend to use it informally and to rely on personal or pooled collective experiences to reason about diagnosis, testing and treatment.29–31 While most clinicians certainly often intuitively integrate geographic information into their diagnostic workup for Lyme disease, we show that a quantitative, formal integration of geographically based incidence improves diagnosis and treatment.
Within endemic regions of the United States, selected states have higher Lyme rates, and within those states, there is significant variation by county. Our findings support a general approach of estimating clinical risk of disease at the point of care, accounting for recent spatial incidence. This approach emphasizes applying epidemiologic context to the clinical decision making process rather than relying solely on history, physical exam, heuristics and preliminary diagnostic test results.9, 32, 33 Improved collaboration between public health departments and clinicians, the maturation of electronic health records, and advances in disease surveillance and automated reporting now increase the feasibility of delivering readily available and easily computed relevant public health information to clinicians at the point of care.34–36
Previously, we showed that epidemiological information about meningitis from a single hospital provides valuable epidemiological context and enhances a decision model for distinguishing aseptic from bacterial meningitis.8 We have also illustrated how an external public health surveillance source improves a clinical decision model, by incorporating state-wide “epidemiological context.”37 Now, for the first time, we show how spatial incidence data improves the ability of a model to identify cases of an infectious disease. In our analyses, epidemiological context variables like season and home location were stronger than any clinical predictor in identifying patients with Lyme facial palsy, building upon a previous clinical model that did not consider home location.7 Epidemiological context was especially powerful when combined with clinical factors. These findings stress the importance of “situational awareness” in clinical settings. Understanding the epidemiological context in which a patient presents may provide vital information about the etiology of the patient’s problem, but currently, valuable spatiotemporal data are not formally processed, considered, utilized or integrated into the clinical decision-making process.
Clinical and public health datasets offer synergistic information that can be leveraged to generate and refine clinical decision algorithms. Public health data have not typically contributed information to generate decision models because while they contain records about those with confirmed disease, they provide little if any information about those without the disease of interest. This creates unique challenges to the integration of public health data into decision models, which rely on rich information about patients both with and without the disease.38, 39 To capitalize on the use of public health data, we relied on incidence rates to develop and refine a decision model for Lyme disease, a condition with significant morbidity and of increasing public health importance.
LimitationsExternal validation should be considered prior to integration into a clinical setting, as the performance of predictive indices may deteriorate in subsequent validation studies.40 Our study was confined to Massachusetts, a state endemic for Lyme disease. Specific definitions of thresholds may vary when more geographically diverse data are considered. Second, this study occurred at a single ED. However, this site provides care for 75% of the children who live in and around this large metropolitan area. Third, residential county was taken from ED registration data, which may not accurately reflect home location for patients with multiple home addresses. This residential location also does not reflect exposures during travel, but does represent the best available exposure data. More accurate information about patient home location would probably strengthen the accuracy of a model incorporating residential location. Fourth, due to the retrospective nature of the study, we only were able to include patients in whom a diagnosis of Lyme disease was considered, and not patients with subtle presentations where the clinician did not consider Lyme as an etiology for facial palsy. Lyme incidence data are county-level, and do not account for variation within county, so future investigations using larger data sets, might provide adequate power to obtain finer spatial resolution. For example, availability of zip code level incidence data might provide more refined risk stratification. Future studies could also incorporate surrogate markers for local disease incidence, such as vector surveys and canine serosurveys. Last, compliance with mandatory reporting requirements by laboratories and clinicians is highly variable41, so underreporting of Lyme disease is a limitation.