Enhancing Fire and Smoke Segmentation with Cross-Attention Side-Adapter Network

Fire and smoke segmentation is crucial for disaster management and emergency response. Existing fire and smoke segmentation methods predominantly rely on conventional deep learning models such as U-Net. However, a key challenge in fire and smoke segmentation is the inherent uncertainty in target scale—segmentation targets can range from large-scale fire or smoke regions to minute initial flames or smoke particles. This multi-scale characteristic poses additional challenges, as traditional models often struggle to balance global feature extraction for large targets with fine-grained feature representation for small targets, leading to missed or inaccurate detections. Furthermore, existing fire segmentation datasets exhibit limited diversity, resulting in models trained with conventional methods that lack generalization ability in cross-domain applications. To overcome these limitations and enhance model performance, this study proposes an optimized Side Adapter Network SAN that integrates cross-attention mechanisms and a CrossViT architecture to improve feature extraction across different target scales. Specifically, the proposed approach employs cross-attention mechanisms to enhance information exchange between CLIP and the side network, while CrossViT effectively strengthens the side network’s capability in capturing fine-grained image details. Experimental results demonstrate that, compared to traditional CNN and Transformer-based models, the optimized SAN achieves significant improvements in accuracy for fire and smoke detection and segmentation tasks. Moreover, due to its strong open-vocabulary semantic segmentation capability, the model exhibits robust generalization in cross-domain applications, enabling it to effectively handle complex environments and diverse fire scenarios.