Multiple searches were conducted between September 2013 and August 2014 using disparate keywords to identify all possible English-language PHI and GHI peer-reviewed literature indexed in PubMed or MEDLINE that was published between January 1, 2012 and September 30, 2014. Supplemental articles were gathered by reviewing the bibliographies of selected articles and by soliciting suggestions from PHI and GHI working group members using AMIA listservs.

For PHI articles, we used the following MEDLINE Subject Headings (MeSH) keywords in various combinations: public health informatics, public health, informatics, and information system. Use of the MeSH headings was preferred, although articles were also identified using the keywords to look within article titles and abstracts. For GHI articles, we used the following keywords: informatics, telemedicine, information system, developing countries, global, national health programs, low resource, and resource-limited. Given the recent emergence of GHI in the literature, MeSH headings for the field are limited; only some of these are official MeSH headings. Furthermore, the searches in 2014 expanded the keyword list to include the name of every LMIC as defined by the World Bank as well as other terms that better reflected GHI activities happening internationally, such as mobile health (mHealth), electronic health records and electronic medical records. This significantly increased the list of candidate articles, but it was determined by CDC health librarians to be the most sensitive method for identifying informatics articles published in LMIC nations. A complete listing of the dates and queries used to identify articles is available in Appendix A.

We used a variety of inclusion and exclusion criteria to narrow the lists of candidate articles. To be included and classified as a PHI article: (a) informatics, information science, or computing had to be the primary subject of the study or a main component of the study methodology; and (b) the article needed to focus on a topic related to public health practice or research. We used a broad lens to assess each article’s relevance to public health practice and research, including: 1) activities conducted by, with, or involving a local, state or federal health agency; 2) assessment and monitoring of disease and health outcomes; 3) primary and secondary prevention of disease; 4) social determinants of health as well as health disparities; and 5) development of the public health workforce including PHI education and competencies. Articles were excluded from the PHI group if the research occurred principally within a LMIC nation. We further excluded articles that did not constitute original research such as letters, editorials, perspectives, opinions, whitepapers, comments, and study protocols.

For a publication to be included and classified as a GHI article, informatics, information science, or information systems had to be the primary subject of the study or a main component of the study methodology. Additionally, the primary research must have focused on activities taking place within an LMIC, and also have a focus on a global health topic. Global health topics follow the framework of Koplan et al. [4] and include: 1) health issues that transcend national boundaries; 2) development and implementation of solutions that often require global cooperation; 3) embraces both prevention of populations and clinical care of individuals; and 4) promote health equity among nations and for all people. Articles were excluded from the GHI group if they did not constitute original research, such as letters, editorials, perspectives, opinions, whitepapers, comments, and study protocols.

Multiple volunteers from the PHI and GHI working groups assisted with review of candidate articles. Two authors (BED and JP) divided the candidate articles into relatively equal numbers and assigned them to reviewers. Reviewers examined titles, abstracts, and other article metadata, recommending inclusion or exclusion based on the defined criteria. Each article was reviewed by at least two individuals, and disagreements were resolved after further review by the team leads (BED and JP).

Selected articles were abstracted by multiple individuals using a set of criteria established by the PHI and GHI teams. For PHI, trainees in PHI programs volunteered to read through the full text of each selected article and summarize key metadata. They examined the information system implemented or evaluated (e.g., syndromic surveillance system, immunization information system); method of data capture (e.g., electronic health record, mobile device); impact of the technology on practice or research; barriers to adoption; size of the cohort or population; study locations; and jurisdiction of practice (e.g., local health department, ministry of health). Summarized information was captured in a spreadsheet and reviewed by the PHI team leads (BED and HK).

For GHI, volunteers had graduate degrees in public health, informatics or informatics-related field and experience in applying ICT in global settings. They reviewed the full text of each selected article published before 2014 (N=83). Given the expanded size of the corpus in 2014 (N=199), volunteers were asked to examine the abstract and full text if possible. Reviewers categorized and summarized the articles using the following metadata: article type (e.g., review, research, methods, models); main keywords; main objective; principal findings; possible impact on the practice of GHI; and other notes. Like PHI, these observations from the reviewers were captured in a spreadsheet and reviewed by the GHI team lead (JP) to identify themes and trends.

PHI articles focus on a wide range of information systems (Figure 3), including: measuring population health using electronic health data (Pop Health / Quality Indicators; N = 20), electronic laboratory reporting (ELR) along with communicable disease case reporting (N=13); syndromic surveillance (N=11), and immunization information system (IIS; N=6). Examining trends across the three years reveals that articles shifted slightly away from a focus on quality indicators towards ELR and case reporting. Yet all four systems remained in the ‘Top 4’ each year. These are all common information systems or uses of electronic data found in public health practice. They are also specific functions called out in the U.S. Centers for Medicaid and Medicare Services’ (CMS) meaningful use program, which incentivizes the adoption and use of electronic health record systems in health care delivery systems and organizations [5,6]. Other common information systems used by governmental public health agencies but not appearing frequently in the selected articles includes chronic disease registries (e.g., state cancer registry), vital information systems (e.g., death registry), and water quality monitoring systems (e.g., toxicology information system).

The year 2012 contained a significant number of ‘other’ articles which is indicative of the broader search strategy employed at the beginning of our project that became more focused over time. Articles in the other category included topics such as the application of geographic information systems to public health practice [7]; PHI education, training and workforce development [8-10]; online information resources [11,12]; and social networking [13,14]. While relevant to the discipline of PHI, we did not classify them as focused on an information system used in routine public health practice.

In Figure-4, we summarize the methods used in the PHI articles. While 43 articles used a quantitative methodology, just nine articles employed a controlled experimental or comparative design such as the examination of a new influenza-like-illness classifier [15] or changes in syndromic surveillance system use following a change [16]. Twenty articles present initial findings from a pilot study that measured a system’s usability [12] or use [17,18] among early adopters. Twelve articles were even more exploratory in nature, outlining simply the design process for a new system [19-21] or presenting the characterization of a new data source [22,23]. Twenty-two articles were systematic reviews, including a review of syndromic surveillance classifiers [24], the use of IIS for research [25], the use of social networking sites in public health [13], and information needs of public health practitioners [26]. Case studies were also present in 16 articles, summarizing the design or implementation of a PHI system within a single health department or group of organizations [27,28]. Ten studies were surveys, which employ quantitative methods to analyze respondents’ answers to questions about PHI training programs [9]; the role of governance in guiding adoption and use of PHI systems [29]; public health engagement in health information exchange [30]; or characterizing the quality of data in an electronic information system [31,32].

In Figure-5 we summarize the jurisdictions involved in the implementation or use of the information system described in the selected articles. There is a noticeable trend in the data away from articles that describe systems designed, implemented, or used at the federal level towards systems used at the local or state level. The more recent state and local health department articles tended to focus on information systems implemented in compliance with the CMS meaningful use program: ELR and case reporting [33,34]; syndromic surveillance [16], and quality reporting [21]. More recent articles also synthesize lessons and implementation strategies at the local level for ELR, IIS, and other meaningful use related information systems [35,36].

Fourteen articles describe information systems deployed or used by hospitals or health systems. For example, three studies examined information systems in U.S. hospitals used by infection preventionists to monitor and control outbreaks within a hospital or health system [30,37,38]. Other studies examined the use of oncology information systems, in combination with as well as independent from cancer registries, to create databases for studying cancer epidemiology and outcomes [39,40]. While clinical organizations’ roles in the design, maintenance or use were emphasized in these articles, the studies either used public health data resources like cancer registries, or they otherwise involved governmental public health agencies in the design or implementation of the system. Other articles employed surveys or interviews that included governmental public health agencies in combination with hospitals or health systems to examine larger topics within the field of biomedical informatics including but not limited to comparative effectiveness research [41,42].

The GHI literature spans a broad range of technologies, health service delivery areas, and foci. Reviewers identified several high-level and recurring themes in the GHI literature through full article review (2013) and abstract review (2014). Major themes that spanned both years of the review included: telemedicine, mHealth, and eHealth. Emerging themes, especially in the 2014 review, included: surveillance, decision support, and geographical information systems (GIS). These themes are summarized in Figure-6. Furthermore, reviewers noted that the overall volume of articles increased in the 2013-2014 review. Although the overall number of articles increased, the relative proportion of each major theme remained similar.

Over the two years of review, the theme of telemedicine remains the largest portion of GHI-related literature. Telemedicine applications bridge healthcare delivery needs when providers and specialists are not physically present in a specific region. Telemedicine has also been a source for access to specialists in fields, such as dermatology and pediatric cardiology, when a country has limited number of specialists available for consultation. Another application of telemedicine technology is to provide continuous medical education to healthcare providers in remote regions. The telemedicine literature in a global context has focused on the development and evaluation of telemedicine programs in various clinical contexts, generally describing implementations of telemedicine and the impact of these tools [43-60]. The literature suggests that when telemedicine solutions are implemented with adequate infrastructure, financial and local clinical support they are found to be effective in meeting healthcare delivery goals and impacting healthcare providers’ ability to learn of new medical practices, although garnering this support is challenging in LMICs [52,53,56,58].

As mobile technologies have proliferated throughout LMICs, their use in a variety of health-related activities has been studied [61-72]. As rapidly as mobile technology use has grown, the mHealth literature also has rapidly increased. Much of the mHealth literature explored the effectiveness of mHealth interventions, with emphasis on using mobile technologies to deliver health-promoting messages, to improve clinic visit attendance for culturally sensitive health issues such as breast cancer, and to increase patient retention in care, treatment, and prevention programs. The mHealth literature focused on scaling up and evaluating mHealth interventions increased in 2014, suggesting growth in maturity for this area of GHI research. mHealth interventions increasingly span numerous areas of clinical and public health focus, including psychiatric disorders, total health/wellness, surveillance of rabies, infant feeding, drug adherence, and measuring the impact of perinatal interventions. Some studies evaluated mobile survey platforms, generally acknowledging the value of this technology for survey distribution [73-88]. The literature also described mobile applications used to improve specific clinical activities (e.g. waiting time, order entry) [74,83,89-93].

The final major theme, eHealth, focuses on the development, use and impact of electronic health record (EHR) systems on LMICs. Some studies in this category aimed to demonstrate the impact of eHealth applications on patient and population outcomes within various LMIC contexts [47,89,94-97]. Other studies explored more foundational informatics aspects, such as data quality in eHealth applications, factors that lead to adoption of EHR systems, use of standardized minimum data sets to assist with the electronic exchange of clinical data, and development and use of open source, standardized EHRs . Determining the extent of adoption and use of EHRs in LMICs and understanding the level of computer literacy as a barrier to adoption continued in the literature over the two years [98-116]. Policy research related to e-health was not highly present in the literature; we identified just one systematic review of policy issues [98].

We further examined the methodology used in each selected article, summarized in Figure-7. Our methods for conducting the review varied from 2013 to 2014, yet in each year we sought to distinguish evaluation articles from reviews and methods papers. Review articles, both robust systematic reviews and weaker review articles spanned several topics, including the impact of mHealth interventions on healthcare quality, a general review of mHealth and its potential, the use of open-source EHR systems, health information systems in Sub-Saharan Africa countries, mobile phone interventions for consumer health, and overcoming shortages of human resources in LMICs [74,81,84,89,99,100]. We further identified a group papers that focused on new methods of informatics research. Finally we identified model papers, which emphasized the development of new models of evaluation or health information technology (HIT) development.

The largest category of articles included a mix of various primary studies. In addition to studies that assessed the impact of ICT on patient and population outcomes, themes that arose during our discussion of the articles were assessments of infrastructure needed for system implementation and readiness of an environment, such as a clinical facility or a group of system users, to adopt the system. Because access to electricity, telecommunications, hardware and other aspects of infrastructure are sometimes constrained in LMICs, we noted another theme aimed to measure the use and viability of ICT. Specifically in 2014 we observed this theme of enumerating infrastructure challenges and proposing solutions to them; topics included measuring system and power outage, security and privacy enhancements, cloud-based migrations and implementations, measuring a hospital’s ability to outsource ICT effectively, and data transformations or mapping to multiple standards [101-107]. Emphasis was often placed on open-source applications that can be distributed across multiple communities and nations [52,108-113]. Discussions examined different aspects of infrastructure as either facilitator or barriers to adoption [94,114-116].

We further identified a set of articles (N=7) we refer to as “readiness studies,” or studies that assess training; availability of health care workers, health delivery facilities, or ICT; phone ownership; and data access in LMICs [102,114-119]. These studies specifically identify the extent to which an environment within an LMIC is prepared to adopt some form of HIT.

Finally, as we opened our search in 2014 to include all countries designated as LMICs according to the World Bank, we observed new topics surface in the literature. Specifically, we observed the presence of imaging informatics publications released in 2014 (N=4). These publications described the improvement of imaging services by integrating systems, reducing cost, using mobile devices, applying new imaging techniques in LMICs, and exploring new methods to identify and compare data in images [120-124]. Reviewers also identified an additional theme related to emergency services supported by information systems in LMICs [125-127].

At the turn of the twenty-first century, PHI efforts around the world were characterized by a focus on the core public health function of monitoring populations: early detection of bioterrorism [128], such as the Anthrax attacks in the U.S [129] and the Tokyo subway attacks [130], as well as global health threats such as SARS [131] and the H1N1 pandemic [132]. While the threat of a large-scale epidemic has not diminished in recent years, as evidenced in 2014 by MERS [133,134] and Ebola [135], the scope of PHI has broadened to support the full range of functions performed by governmental public health authorities. As evidenced in this review, PHI today contributes not only to disease detection [15,136] but also to the delivery of public health care services [137-139], the measurement of population health indicators within and across jurisdictions [21,140], investigating environmental hazards [141], and prevention of disease [142-144].

PHI research to a large extent, just like in many other areas of biomedical research, follows policy and funding patterns within a nation or region. The last few years have witnessed a global recession with shrinking public health budgets and a sharp decline in funds allocated to public health preparedness. Funding patterns explain, in part, what appears to be a more balanced portfolio of PHI activities with a strong but smaller emphasis on surveillance. In developed nations with a strong integrated public health system such as Canada [145,146], Australia [147], Taiwan [27] and several European nations [148-150], PHI activities include an emphasis on integrating data and supporting bi-directional communication between clinical and public health services. Whereas in the US, PHI efforts have focused largely on EHR incentive policies that seek to improve how public health agencies gather the data they need to monitor the health of populations [29,33,151]. Thus to stimulate new advancements in PHI, especially in the U.S., policymakers must encourage the development or broader implementation of information systems that span the clinical and public health continuum to support the full range of public health functions. Support for this expanded view of PHI can be found in both the Council of State and Territorial Epidemiologists’ ‘Blueprint version 2.0’ statement on the future of surveillance [152], as well as the workshop report from the Robert Wood Johnson Foundation on the future of PHI [153].

The focus of GHI research intersects at the crossroads of clinical and public health informatics in the context of LMICs. Given a global emphasis on the development and implementation of national eHealth strategies, recent GHI articles reflect an increase in the development, implementation and use of telemedicine, mHealth, EHR, and other forms of healthcare ICT in LMICs on all continents. Furthermore, the limited studies assessing outcomes or impact of these systems on patient and population health suggests that adoption of EHR and other ICT systems in LMICs may be accelerating at a pace that exceeds the rate at which the academic community can evaluate them.

Despite a change in our search methodology, we observed similar patterns across all publication years. First, there is significant growth in the publication of GHI articles. While this may be partly due to our change in search strategy to include specific LMIC country names, we believe there is true growth in the development, implementation and use of ICT in LMICs across the globe. External forces, including the World Health Organization (WHO) eHealth Technical Advisory Group (TAG) formed in 2013, are encouraging LMICs to adopt and implement national eHealth strategies that will strengthen health systems through the use of ICT including telemedicine and EHR systems [154,155]. Second, despite growth in articles many studies continue to focus on implementation rather than outcomes. We observed numerous, although sometimes subtle, descriptions of implementation challenges. For example, reviewers noted that organizational culture can create barriers, which impact the adoption and implementation of informational tools, was often mentioned in the literature, although this subject is not treated methodologically as the focus of many publications [81,117]. Finally, GHI research is intrinsically linked to public health practice in LMICs, creating the link between the use of ICT and use of clinical and other data for public health activities such as surveillance, health monitoring and public health program evaluation [156-160].

The literature suggests that the use of ICT will continue to expand in LMICs. This will likely generate new studies on their impact on patient and population health. As these studies are planned and executed, our review suggests the following. First, we note a general dearth of rigorous study designs. Better designed studies that follow the Good Evaluation Practices in Health Informatics (GEP-HI) [161] and are reported using the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) guidelines [162,163] will support future efforts that can more rigorously systematically review and synthesize outcomes. More rigorous review and synthesis will provide stronger evidence that many policymakers seek as they contemplate policies and funding beyond initial investments in eHealth systems [164]. In addition, given a rich dialogue in the Discussion sections of many articles on the barriers and facilitators of adoption, we suggest that future research more rigorously examine the implementation of ICT in LMICs. There exists a growing body of implementation science literature in the developed world [165,166], which would be strengthened by contributions from LMICs if implementation were more rigorously studied in parallel with outcomes. Furthermore, we observed that many studies involve partners from multiple universities, research institutes, and countries with diverse funding streams. Finding harmony across fragmented stakeholders appears to be a goal of the applied work taking place in these LMICs, yet this is not emphasized as a measurable component of research in the current body of literature. Future studies should consider studying this dimension of GHI implementation as a facilitator or barrier.

Our review of the PHI and GHI literature contains several limitations. First, our review of the literature was limited to articles published in English, which limited our ability to identify and read relevant articles published in other languages (the Iceberg effect). Second, our review was limited to primarily peer-reviewed articles indexed in MEDLINE, which limited our ability to identify relevant articles published in newer journals or journals which do not meet the scientific scope and quality metrics established by the MEDLINE Literature Selection Technical Review Committee. GHI article review for 2014 was limited to abstract review of selected articles.

Although our purpose differs from that of the International Medical Informatics Association’s IMIA Yearbook of Medical Informatics, many of our methods are quite comparable [167]. And like IMIA, as we continue to evolve the Year in Review session for AMIA’s annual symposium, we will seek to refine our methods. We will consider additional sources beyond MEDLINE to identify more articles from related disciplines which are not indexed in PubMed such as the computer, information, and decision sciences. We will further explore tools such as BibReview to enhance our ability to identify, select and summarize recent publications in the disciplines of PHI and GHI.