The first case study, 1999–2018 NHANES data linked to HUD data through 2019, is an example of data blended through record linkage that is used primarily for research purposes. NHANES is a series of nationally representative cross-sectional surveys of the U.S. civilian, noninstitutionalized population that is selected using a complex, multistage probability design [2, 10] and consisting of about 5,000 persons from 15 different counties each year and released in two-year increments. The survey includes a standardized health examination, laboratory tests, and questionnaires that cover various health-related topics.

HUD is the primary federal agency responsible for overseeing domestic housing programs and policies and is responsible for administrating various housing and community development programs [11]. HUD administrative data systems contain housing, income, and program participation data for recipients of Housing Choice Vouchers (HCV), Public Housing (PH), and Multifamily (MF) programs.

Only NHANES participants who provided consent for person level data linkage as well as the necessary personally identifiable information (PII), such as name and date of birth, were eligible for linkage. NHANES participants under 18 years of age at the time of the survey were considered linkage eligible if they met linkage eligibility criteria and consent was provided by their parent or guardian [12].

The quality of linked data is a combination of the quality of each source, the quality of the blending method, and the quality of the resulting data for the intended purpose. For this case study, the authors focused on the quality of the blending method and the resulting linked data.

Table 2 provides information about the linked NHANES-HUD data and summarizes its identified threats to quality for the dimensions of the FCSM Framework. The quality of these data in many dimensions was considered high for research purposes, including for relevance, accuracy, scientific integrity, computer and physical security and confidentiality. Linked NHANES-HUD data have supported numerous studies ranging from blood lead levels among children receiving federal housing assistance to the association of cigarette smoking and asthma among people receiving housing assistance [13,14] (relevance). Benchmarking measurements are in line with estimates from the full HUD population and linkage errors are low [12] (accuracy). As a result, the scientific integrity of the linked NHANES-HUD data is high.

Data are available through the NCHS RDC, a secure computing environment, to protect confidentiality of respondents. However, due to the processing time to release survey data, the time to coordinate data sharing agreements and complete the linkage process, the data were considered low on timeliness. For example, as of the writing of this paper the linked NHANES-HUD data only included survey data through 2018 and administrative data through 2019, and therefore, could not be used for COVID-19 related research (see NCHS Data Linkage table [Reviewed 2022 December 28; cited 2023 May 23], Available from https://www.cdc.gov/nchs/data/datalinkage/LinkageTable_1.pdf.)

The second case study considered was on the use of outdoor, or ambient, air quality data that could be blended with health data, including health survey data and administrative records, for conducting research to improve understanding of the association between air pollution and health. Studies have generally demonstrated a negative association of poor air quality on health and on health disparities, though specific air quality measures, health outcomes, and study designs differ among studies [15,16].

Air quality data are collected, processed, and analyzed for many reasons. For instance, air quality measurements from monitors are routinely collected by the U.S. Environmental Protection Agency for compliance purposes and can have limitations when used for tracking individual- or community-level exposures at continuous spatiotemporal scales. Modeled predictions can augment gaps in monitoring data and can estimate exposures over an area, but prediction accuracy can vary based on several factors, including assumptions centered around meteorology and emissions. Other approaches to estimating air pollution exposures include the use of satellite-based remote sensing and low-cost sensors. Remote-sensing data, which are available in a timely manner, can be used to create measures of air quality and to develop emission inventories that support modeled estimates. Low-cost air sensors, including ‘purple’ air monitors, that are part of citizen science networks can help characterize local air quality levels with reasonable accuracy.[17] Furthermore, air pollution data collected during and after wildfires, dust storms, volcanic eruptions and other air quality events or natural disasters also provide information on air quality.[18] In addition, any or all of these individual air quality data sources can be combined with or without other environmental parameters (e.g., temperature, land use) for generating ‘fused’ or ‘blended’ air pollution data for use in public health research and surveillance.[19] Specifically, newer approaches to combining various air quality data sources, either in a statistical framework or using an artificial intelligence (AI) based framework, have gained wide-spread acceptance in the environmental health community.[20]

Air quality data can be blended with health and other data by merging aforementioned air quality data sources by geography or locational attributes (e.g., municipal indicators, such as county, grids, latitude and longitude), by time (e.g., annual, monthly, daily, hourly), and by other indicators, such as whether an air pollution metric exceeds a certain threshold.

As with all blended data, the quality of air pollution data blended with health data are a combination of the quality of each source, the quality of the blending method, and the quality of the resulting data for the intended purpose. For this case study, the authors focused on the quality of the air pollution data and the derived estimates of air quality, and the geography and temporal resolutions needed for blending.

Table 2 summarizes the air pollution data as considered for this case study and their quality for the dimensions of the FCSM Framework. The quality of these data in many dimensions is high, including relevance, accuracy, and scientific integrity. Blended air quality and health data have been used in numerous studies that have informed decisions ranging from determining air quality standards to guidance for high-risk groups. As a result, the relevance of air quality data for blending was considered high. Measurements and model-based estimates are considered accurate and are typically based on current technologies for collection and processing (high scientific integrity). However, different modeling assumptions and analytic decisions may increase the threat to coherence.

The third case study evaluated 2013–2014 PAM data collected in NHANES (Table 2) [9]. The personal monitoring devices used in NHANES to collect the data were worn on the wrist for one week and measured body movements at the wrist, including those like the swinging of the arm during activities such as walking or jogging, intensity of movement over time, and the amount of time spent doing different levels of physical or sedentary activity.

The use of PAM data is complicated by the high dimensionality of the data, requiring very large data files that include detailed information on several dimensions of activity and movement that, as of this writing, require extensive computing resources and expertise to develop of metrics for analysis. Blending these large, high-dimensional data adds additional complexities arising from the blending methods and the planned analyses.

While the data source for this case study was the NHANES PAM data, some of the assessments provided here could apply to other sources of person-level data obtained through wearable, personal monitoring devices, not necessarily collected through NHANES. PAM data are collected for other reasons, including research studies [e.g., 21, 22] and patient health monitoring.

Table 2 summarizes PAM data as considered for this case study and its quality for the dimensions of the FCSM Framework. The quality of these data in many dimensions is high, including relevance, accuracy, scientific integrity, computer and physical security, and confidentiality. NHANES PAM data were collected and processed using scientifically validated protocols and IT procedures that increase their accuracy and scientific integrity and protect confidentiality and data security [9]. Creating summary statistics or other dimension reduction approaches using scientifically sound methods, including ones that rely on machine learning, for pre-processing the PAM data can improve their accessibility. Blended PAM and mortality data have been used in studies of mortality and cognitive function [21,22]. As a result, the relevance of PAM data for blending is high. However, timeliness may be low in the data quality assessment because of the periodic collection of some NHANES components as well as the time it takes to process and prepare the data for release. As of the writing of this manuscript the most recent collection and release of PAM data are from the NHANES 2013–2014 cycle. Measurements and model-based estimates are considered accurate and are typically based on current technologies for collection and processing.

Data from each case study were considered relevant for answering health-related research questions, and that the relevance of input sources increased when blended. Published studies directly or indirectly are one way to evaluate the data’s relevance for health research and were identified for each case study. Threats to relevance occur when data do not align with the most pertinent research questions; for example, if person-level air quality data are needed but only area-level are available and therefore are used as a proxy for person-level data. Threats to relevance may also increase as threats to other dimensions increase, including threats to timeliness, accuracy, and scientific integrity.

There were two types of accessibility issues considered for these case studies. The first was the complexity of the input or blended data. The more complex the input data and resulting blended data, the more difficult they are to successfully use. The complexity of linked, blended, and large monitoring data reduces their accessibility for users who may not be familiar with methods or tools for analyzing big data. This threat is reduced through interdisciplinary research teams, by thorough documentation (including but not limited to analytic guidelines and web tutorials), and through increased and varied research uses, where published examples can illustrate best-practices for analysis.

The second type of accessibility considered was how the data can be obtained and used. The data may have access restrictions due to disclosure risks and, as a result, be more difficult to obtain for analysis (e.g., analyses must be conducted in a secure RDC). Advances in synthetic blended data or the availability of public use ‘feasibility files’ (files that provide a limited set of variables that can be used to determine the maximum available sample size for each linked file) may reduce this threat for some uses by providing users with limited public use data that can be used for developing analytic plans and testing code.

Timeliness is the length of time between the event or phenomenon the data describe and their availability. Preparing data sharing agreements, standardizing the final adjudicated data, processing complex data, such as modeled or very large files, and performing record linkages or implementing other blending methods all add to the time it takes to release a file. This complexity in data production can cause threats to data quality for research that depends on timeliness. Innovations in data science and statistics may reduce these threats by facilitating greater automation and efficiency of these data production processes, as can advances in data sharing and governance that are being implemented throughout the federal government which may facilitate more timely and efficient data sharing [23,24].

With blended data, threats to punctuality are related to delays in any of the components of blended data, from data sharing agreements and input data processing to data blending and dissemination. The multiple steps needed to produce blended data increase the risks of missing deadlines. Punctuality differs from timeliness by its focus on expected timelines for the data’s availability, rather than the correspondence between the data’s reference time and current time. Data users expect delivery of data at a given time and may plan their research accordingly. If data producers fail to deliver the data as planned, this becomes a quality issue which can negatively affect research. Given the tremendous size of the data file (over 1 terabyte compressed), for example, the PAM data were significantly delayed in their release. Mitigations include efficiencies in processing that are realized through technology, scientific development, and subject matter experience.

In general, threats to granularity for all data include reduced sample size and lack of detailed information available that permits analysis of small population subgroups and small-scale geographic locations. When blending data, threats to granularity from reduced sample size can result from linking or blending data, as the size of the blended data for analysis is a function of the size of the smaller data input excluding records that cannot be blended. For example, NHANES data prior to linkage have limited granularity due to sample size and degrees of freedom (only about 5,000 participants a year from 15 PSUs), compounded when samples are limited to individuals who are linkage-eligible. With only about 10 million people in the U.S. receive federal rental assistance [12], the linked NHANES-HUD data subsequently have very little granularity as the number of linked respondents can be small; see the NCHS match rate table for the NHANES-HUD linked data here: https://www.cdc.gov/nchs/data/datalinkage/NCHS-HUD-Match-Rate-Tables-final.pdf, accessed 5/3/2023.

However, blending air quality data with survey data is not dependent on linkage-eligibility requirements that are needed for person-level linkage and does not necessarily decrease granularity relative to the input sources since air quality data provide information at various geographic levels, particularly when model-based methods are employed and the data are available for all or nearly all respondents.

Threats to accuracy and reliability are numerous. New tools are being developed to automate and standardize many measures of accuracy to enable regular assessments. Key threats to accuracy for the case studies include missing item or record level data; potential bias due to linkage (e.g., differential linkage eligibility or quality); insufficient metadata, lack of complete addresses for geocoding; mis-aligned geographies, including those based on administrative (e.g., U.S. Federal Information Processing Standards (FIPS)-based) identifiers that change; modeling error, and monitor disruptions (air quality data); and possible device malfunction (PAM). Threats to accuracy can also be reduced by blending, where information may be more accurate in one source compared to the other and can be used to improve the accuracy of a given item. For example, race/ethnicity data obtained by self-report in a survey may be more accurate than the race/ethnicity data collected on death certificates when mortality data are linked [25] whereas Medicaid or Social Security Administration program participation may be more accurately determined through linkage rather than survey self-report.[26, 27]

Within the domain of Objectivity, accuracy and coherence are related and threats and mitigations to accuracy identified above can also relate to coherence. For these case studies, coherence was evaluated in different ways for data alignment to related sources of information. Threats to coherence for linked data may increase with the potential of bias from linked data when not all records are eligible for linkage so the population used for estimation with the linked data may not be the same as that in the administrative data. This threat means that estimates from the linked data may not align with estimates from the administrative data. This threat for linked data could be mitigated by adjustments to sample weights or other analytic approaches but the success of the mitigation would depend on the extent of the bias and available information for adjustment. Threats to the coherence of blended air quality data arise from the use of different modeling approaches, different data processing decisions, monitor-specific data, and different blending decisions which could lead to some variation in inference among related studies. For both the linked NCHS-HUD data and the air quality data use-cases mitigations of these threats include the use of common definitions, scientifically valid statistical methods, standard approaches for modeling and other analytic processes, and the provision of detailed documentation so users can understand and interpret differences among sources.

Threats to scientific integrity can increase with outside interference and the use of outdated or insufficiently rigorous scientific methods. No threats to scientific integrity were identified for these case studies.

Threats to credibility arise from threats to the producers’ reputations. No threats to credibility were identified for these case studies.

No threats to computer and physical security were identified for these case studies. However, evaluation of these case studies relied on the practices and standards of each data provider for assurances on threats to computer and physical security.

In general, threats to confidentiality increase when using blended data by increasing the amount of information available about a survey participant, program recipient, or geographic location. Blending air quality data to person-level data, for example, increases threats of person-level disclosure even when actual locations are removed from the data file as people in areas with extreme values of air quality could be identified. Confidentiality threats are increased by information available on the internet, including historical information and information that may be collected in the future, that can be combined with person-level information. Threats are mitigated through restricted access and may be mitigated through innovations in synthetic data.