Clinical data sources are composed of patient-level data that are generated from interaction with and assessment of patients and are typically abstracted from institutional electronic health records (EHRs). They can be collated at the institution-level by health systems, by government agencies (e.g., Surveillance, Epidemiology, and End Results [SEER] by the National Cancer Institute [NCI]), by professional societies through participation that may or may not be tied to accreditation or data access (e.g., the American College of Surgeons [ACS] and the Society of Thoracic Surgeons [STS]), and by vendors that source data from multiple institutions sharing a common EHR platform (e.g., Epic Cosmos).¹ In addition, EHR-based networks such as All of Us by the National Institute of Health (NIH) and PCORNet by the Patient-Center Outcomes Research Institute (PCORI) work across different EHR platforms using a Common Data Model for reporting.² All of these sources are distinct from claims-based data sets, which are generated from billing and insurance-related transactions rather than from interactions with people, and are addressed in a different guide in this series. Depending on the variables available, clinical datasets can be used to conduct a wide variety of health services research (HSR) including observational cohort studies, comparative effectiveness research (CER), and economic evaluations. In the case of clinical data sources for which prospective data collection can be centrally coordinated, they can also be used to conduct pragmatic clinical trials (e.g., ACS National Surgical Quality Improvement Program [NSQIP],³ Abdominal Core Health Quality Collaborative [ACHQC]).⁴ In this manuscript, we describe the strengths and weaknesses of clinical sources of “big data.” Specifically, we focus on datasets not previously included in the 2018 JAMA Surgery Practical Guide to Surgical Data Sets Series.³ A more expansive list is available on the website of the Surgical Outcomes Club.

Clinical datasets vary significantly based on included populations, how and from where content is collected, and data elements they contain. These factors may also limit generalizability and contribute to selection bias and should be considered carefully when deciding which dataset to use for a given research question. For example, data in the STS General Thoracic Surgery Database is provided by only 40–50% of eligible centers, thus clinical outcomes may be less generalizable than those conducted through the STS Congenital Heart Database, which is sourced from over 95% of the centers performing these operations.⁵ The Department of Veterans Affairs (VA) Corporate Data Warehouse (CDW) and the Million Veterans Program (MVP), although robust, may have limited generalizability given the preponderance of older men among the US veteran population.⁶ If datasets are procedure-based (e.g., Carotid Endarterectomy Registry from the Society of Vascular Surgery [SVS] Vascular Quality Initiative [VQI]) rather than disease-based, CER comparing surgical and non-surgical management is not possible. Finally, many datasets lack patient-reported outcomes (PROs) and long-term data on survival or recurrence, all of which should be considered alongside short-term outcomes (e.g., 30-day readmission, perioperative complications) when assessing a surgical procedure.

The quality of data collection and collation varies across datasets. Data in the VA Surgical Quality Improvement Program (VASQIP) and NSQIP (modeled after VASQIP) are collected and iteratively updated by trained clinical reviewers.² The National Cancer Database (NCDB) is sourced from tumor registries of institutions that are accredited by the Commission on Cancer (CoC) and must submit data to maintain CoC accreditation and enable NCDB access for investigators.³ Although tumor registrars are expected to reference the Facility Oncology Registry Data Standards (FORDS) manual while coding and entering registry data, no central review of NCDB data is performed, so coding errors occurring at the institutional level may be perpetuated in the NCDB.

Most of the data in clinical sources is primary, i.e., comes from patients themselves via self-report (e.g., race, gender identity, PROs, survey responses), objective measurements (e.g., labs, radiology exams), and/or direct observation by clinicians (e.g., palpable lymphadenopathy). However, a small proportion of data in clinical sources may be secondary, i.e., collected by a third party, and sometimes linked to primary data (e.g., administrative Medicare claims data [secondary] linked to SEER data [primary], vital status information from the National Death Index and State Vital Statistic Registries [secondary] linked to data in the VA CDW [primary]), with varying levels of transparency as to what is primary and what is secondary data. Because all research using clinical datasets involves information collected from human participants, investigators should always seek approval from their local institutional review board. If only deidentifiable data is used, exempt status is typically granted. Some clinical datasets, including ones that include deidentified but potentially identifiable data, require data use agreements (DUAs) either with participating institutions and/or the organization overseeing the dataset or with individual investigators. Precluding the publication of table cells below a certain number of participants (e.g., 10 for ACHQC and NCDB) is another measure intended to limit the likelihood of participant re-identification.

Some clinical data sources require that all analyses be conducted by credentialed analysts and biostatisticians within the administering organization or may require that all analyses be done within a specified trusted research environment. Others, such as SEER, require specific software to view the data. As data in the VA system is spread across over 60 databases within the CDW, researchers typically need to work with VA-affiliated analysts with programing expertise to cross-match the various databases at the level of patients and timeframes.⁶ Finally, large sample sizes may potentially lead to overestimation of statistical significance and data precision even when there is little or no clinically meaningful significance.

Care must be paid to both the population in a clinical dataset as well as the origin of collected data to anticipate and avoid selection bias. Regression and other techniques can address many but not all sources of confounding. Careful dataset selection based on the question and population of interest is the best way to minimize selection bias and to ensure the validity, generalizability, and real-world applicability of the research findings. For example, hospital-based databases such as the NCDB are not appropriate for describing population-wide, epidemiological trends. In addition for some research questions, particularly for rare conditions or in marginalized groups, mixed-methods analyses incorporating qualitative approaches may be more appropriate than running regression analyses on large but non-representative samples. With greater focus on redressing disparities and promoting equity in surgical care, clinical data are increasingly used to identify ways in which traditionally marginalized groups experience disparities in disease prevalence, access to diagnosis/treatment, and outcomes. It is important, however, that investigations regarding disparities be pursued with intentionality and not as an analytical afterthought. For such work, as for all high-quality HSR, a statistical analysis plan should be drafted prior to study initiation that makes clear the hypotheses to be tested, the variables to be included, and the a priori subset and interaction analyses that are planned to avoid “fishing expeditions.” Furthermore, investigators should be aware of how nonrandom missingness may disproportionately affect minoritized groups, potentially amplifying bias and distorting results.