Cognitive Interviewing is a research method used to 1) identify problems participants may experience with survey questions, 2) study survey item construct validity, and 3) examine differences in thought processes in response to survey items across different demographic, linguistic, or cultural groups. ³⁰ Cognitive interviews facilitate understanding regarding the performance of the drafted survey questions; specifically, if the respondents understand the questions according to their intended design, and if accurate answers are given based on that intent. Utilizing this method with a diverse group of participants yields deep, contextual insight into how respondents interpret questions, consider relevant aspects of their lives and formulate responses based on those considerations. ³¹The technique can be used through a descriptive process for nascent survey items, or using a reparative approach to revise established items. Here, we utilized a descriptive process with an expert panel of 18 transdisciplinary participants, and a reparative approach with a final 2 additional participants.

Purposeful and snowball recruitment techniques were employed to assemble an expert panel of 20 members. We intended to sample a group of people who had a high likelihood of experiencing the disasters or hazards in the survey items we were testing. Thus, rather than future survey respondents, we sampled disaster experts at this stage due to the total number of disaster/hazard terms and the geographic variability in frequency of impact. Our strategy intentionally over-sampled (up to 50%) those with Black and/or African American racial, biracial or multiracial identities. See Appendix A for more information on our rationale for including 20 expert panelists. Inclusion criteria were a nationally or internationally recognized expert in their disaster-related discipline as evidenced by publications, awards, and/or fellowships and professional work experience as a first responder or disaster responder, public health, home health, emergency nursing, disaster nursing, and health care management expertise, and reside in the U.S. Our professional network enabled access to national experts through which we recruited. We recruited the first 18 participants for the full iteration of procedures, and a final 2 panel participants for a reparative approach after final item revision.

Using the Department of Health and Human Services regulations found at 45 CFR 46.104(d)(2), a commercial institutional review board (IRB) reviewed this protocol and study materials and determined that this research was exempt from IRB oversight.

An item pool was created from existing organizational hazard vulnerability assessment’s (Kaiser Permanente, Risk Identification and Site Criticality Toolkit and Centre for Research on the Epidemiology of Disasters listings) and a literature review for use at the household level. ^32–34 A semi-structured cognitive interview guide was designed by a member of the research team (JC) with extensive experience in emergency nursing and instrument development and pilot tested with a consulting member of the expert panel.

Interviews were conducted by the same member of the research team (TA) from December 2021 through May 2022. The purpose of the interviews was to ascertain perceptions of standard hazard vulnerability assessment domain items and responses when applied to the household level. Participants provided informed written and verbal informed consent in advance of the interview. Refer to Appendix A for additional detail regarding the interview structure.

Demographic information collected included sex at birth, current gender, age, racial and ethnic identities, language spoken in the home, veteran status, household member veteran status, and highest completed level of education. All demographic information was self-reported by participants.

The videorecording of the interview was utilized as the raw data for analysis. No transcription was used in order to fully incorporate non-verbal information in the analytic process. ³⁵ Interviewer notes augmented the videorecording data. Theme and pattern coding were used for analyses (Figure 1). Refer to Appendix A for additional detail regarding data analyses.

All participants confirmed they were located within the U.S. or U.S. territories at the time of the interview. The interviews were conducted over the web and recorded (audio and video, as available). We allotted 90-minutes for each meeting with expert panelists. Here, we report the results of our thematic and pattern coding analysis. ³⁶

This transdisciplinary panel represented a variety of occupations, including epidemiology, chemistry, fire service, first responder, nursing, academic professor, consulting and roles of disaster planning and response throughout all phases of the disaster management cycle. Panelists’ areas of professional expertise reflected disaster-related leadership, expertise and service that spanned all levels of government (local/state/national/international) and included (but were not limited to) sectors of public health, emergency preparedness, management and response, emergency medical services, global health security, non-profit engagement and health care leadership.

Table 1 lists the demographic characteristics of the expert panel. Of the initial 18 interviews completed with a descriptive approach, seven (38.8%) expert panelists identified as cisgender men. Ten (55.5%) expert panelists identified as cisgender women, and one respondent chose not to report their sex or gender. Eight (44.4%) expert panelists were between the ages of 30–49, and five (27.7%) were between the ages of 50–64, and another five (27.7%) were 65 and older. Eight expert panelists (44.4%) endorsed a Black/African American racial identity. Of these, 2 participants indicated additional racial identities (Biracial, White/Caucasian, Native American, or a combination) to Black/African American. Eight expert panelists (44.4%) endorsed a Caucasian racial identity. Of these, 1 participant indicated an additional racial identity of Native American in addition to White/Caucasian. One expert panelist endorsed a Native Hawaiian/Other Pacific Islander racial identity, and the final expert panelist interviewed with a descriptive approach vocalized the desire for an “Other” or blank/fill-in category in response to this question. All participants spoke English, with two participants also speaking other languages in their home (Arabic and French). Collectively, panelists mapped their answers pertaining to personal or professional experience of the 60 hazards/disasters to every region of the Mainland U.S., Alaska, Hawai’i, Puerto Rico and the Virgin Islands. International geographies mentioned, as participants cognitively mapped their experience with each hazard or disaster term, included Japan, Afghanistan, Honduras, Haiti, West Africa and China. The demographics of the two expert panelists interviewed with a reparative approach are included in Table 1 as well.

The results of the pattern coding we utilized are depicted on Table 2. Here, we used pattern coding to assess for the potential for racial data bias and ascertain group differences between those with Black/African American racial identities and those who did not report any Black/African American racial identity.

Overall, across participants and items, four overarching themes emerged in the general conceptualizations examined in response to the items worded as, “In your lifetime, have you ever been directly impacted by [disaster/hazard/event term]?” These themes were the disaster characteristics, resources, individual life facet, and felt effect. Figure 2 depicts a flowchart of these themes with subthemes. Participants consistently mapped their stories and responses along this overarching flowchart as part of both their comprehension of the question and in judging or justifying their retrieved information as warranting the response as either yes or no. For example, one participant endorsed a secondary, emotional impact to the active shooter disaster/hazard. Although they were not physically present for the actual event, they felt emotionally impacted, although “not directly,” while supporting family members (including minor children) who were processing and psychologically recovering following this event. In addition, participant’s cognitive walk-through of individual life facets was also used as an aid in the retrieval of relevant information from memory.

Of the 60 disaster terms, the cognitive interviewing and analysis process revealed potential problems with several of the disaster, hazard, and response terms. These terms require elimination, replacement, or further revision in our validation process and survey instrument development. The findings of the three terms we identified as most problematic or with the most disparate interpretations by participants are summarized in the following paragraphs. These terms were trauma, radiation exposure, and criminal acts of intent. The findings specific to these terms are summarized below, integrating a synthesis of the empirical data from direct interview quotes, field notes and theme coding.

As identified in our pattern coding results (Table 2), we noted several areas that flag the need for further investigation into possible racial disaster disparities. We found greater than 30% difference, with a greater proportion of participants with all other racial identities reporting experiences with chemical exposure outside structures, other utility failure, and weapons, compared to participants with Black/African American racial identities. These items were considered for elimination and/or combination with other terms for an instrument relevant to those with Black/African American racial identities. We found a difference greater than or equal to 25% for water dam failure, evacuation, external flood, suspicious package/substance, and transportation failure with a greater proportion of those with Black/African American racial identities reporting an experience with these events, compared to others. In contrast, a greater proportion of those with other racial identities reported experiences with explosion, hazardous material incident, very important person situation, or other hazard than participants with Black/African American racial identities. Our findings align with previously published literature on the impact of historical structural racism for communities of color increasing the risks of negative health impacts related to flooding, transportation, and evacuation.^14–17 Segregated neighborhoods and social networks generate racial disparities on a macro level ²⁰ and perpetuate increased disaster-related racial disparities. Residents in low-income racially segregated communities, disproportionately overrepresented by Black residents, experience higher incidents of disaster toxic exposure, ⁴² and worsened housing and resource recovery after disaster.⁴³ Some people with Black/African American racial identities may both reside in high-risk disaster areas and have the least amount of resources to protect themselves and their families against or recover from climate change related hazards or disasters.⁴⁴ Given the longstanding history of structural racism and unequal race-related wealth distribution in the U.S., multi-level conceptualizations and research designs are required to better understand disaster-related racial disparity, especially when collectively investigating climate change, health equity and household-level disaster preparedness. ^20–23

In addition to no household-level hazard vulnerability assessment, our team found no cognitive interview reports online through QBank in the last 5 years that investigate data or racial bias for individuals or communities with Black/African American racial identities. ³⁷ In the context of health disparity research with substance abuse, Burlew et al ⁴³ notes the importance of adequate measurement and cultural equivalence/appropriateness when working to eliminate health disparities, beyond a person’s primary language. Investigating the relevance of different constructs for specific groups, and how they are understood and interpreted, is vital to ensure adequate measures are being used in disaster research with marginalized populations.^30,45 There is an obvious need for instrument development and assessment to eliminate data gaps and data collection most relevant to communities of color as those most at risk to climate change impacts in the U.S. Valid instruments are needed to assess the disproportionate impacts of climate change on communities of color and inform climate equity interventions.

When considering transferability, the findings reported here present a cognitive map and schema from an expert perspective relevant to all-hazards experiences, as well as those specific to disaster sub-types. These themes may be considered as an initial theory of a cognitive process map when considering individual and household vulnerability. We also identified several problematic hazard, disaster, and event terms that might not be interpreted by survey-takers as intended. Other measurements utilizing terms such as trauma, radiation exposure, or criminal acts of intent may require further validity testing and refinement in research and practice. We further identified several hazard, disaster, or event terms that require clarification, elimination, replacement, or further revision in our validation process and survey instrument development. These are common terms, often used in organizational hazard vulnerability assessments, that may have relevance to other disaster and climate change research teams seeking to focus their work on individual or household units of analysis.