Explaining the Unseen: Multimodal Vision-Language Reasoning for Situational Awareness in Underground Mining Disasters

Details

• Personal Author:
  Jewel, Mizanur R. ; Elmahallawy, Mohamed ; Madria, Sanjay ; Frimpong, Samuel
• Description:
  Underground mining disasters produce pervasive darkness, dust, and collapses that obscure vision and make situational awareness difficult for humans and conventional systems. To address this, we propose MDSE, Multimodal Disaster Situation Explainer, a novel vision-language framework that automatically generates detailed textual explanations of post-disaster underground scenes. MDSE has three-fold innovations: (i) Context-Aware Cross-Attention for robust alignment of visual and textual features even under severe degradation; (ii) Segmentation-aware dual pathway visual encoding that fuses global and region-specific embeddings; and (iii) Resource-Efficient Transformer-Based Language Model for expressive caption generation with minimal compute cost. To support this task, we present the Underground Mine Disaster (UMD) dataset-the first image-caption corpus of real underground disaster scenes-enabling rigorous training and evaluation. Extensive experiments on UMD and related benchmarks show that MDSE substantially outperforms state-of-the-art captioning models, producing more accurate and contextually relevant descriptions that capture crucial details in obscured environments, improving situational awareness for underground emergency response. The code is at Github. Description provided by NIOSH
• Subjects:
  • Mining
  • Occupational Health
  • Safety
• Keywords:
  • Mining industry
  • Automation
  • Situational awareness
  • Mine disasters
  • Underground mines
  • Imaging techniques
  • Emergency response
• ISBN:
  9798331555115
• ISSN:
  2642-9381
• Document Type:
  Text
• Funding:
  Cooperative Agreement
• Genre:
  Proceedings
• Place as Subject:
  Kansas City Region OSHA ; Missouri ; San Francisco Region OSHA ; Washington
• CIO:
  National Institute for Occupational Safety and Health (NIOSH)
• Topic:
  Workplace Safety & Health
• Location:
  North America
• Pages in Document:
  13 pdf pages
• NIOSHTIC Number:
  nn:20071823
• Citation:
  2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 6-10, 2026, Tucson, Arizona. Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE), 2026 Mar; :1324-1333
• Email:
  frimpong@mst.edu
• Federal Fiscal Year:
  2026
• NORA Priority Area:
  Mining
• Performing Organization:
  Missouri University of Science and Technology
• Peer Reviewed:
  True
• Start Date:
  2023/09/01
• End Date:
  2027/08/31
• Download URL:
  https://stacks.cdc.gov/view/cdc/260046/cdc_260046_DS1.pdf
• File Type:
  [PDF - 36.73 MB]
• Collection(s):
  National Institute for Occupational Safety and Health
• Main Document Checksum:
  urn:sha-512:606219558cdad253cd197b2d5d43ffb99b8db03d99dff513131ea74e9ee7b19a5e6c7bbf8070d95c848d8ceced09dfbcc00f96cca70375792e28f1b0e51e710b