CLIP Guided UNet for Smoke Segmentation

Smoke segmentation aims to
classify pixels with the smoke label, which is a downstream task of semantic segmentation. Convolutional neural networks (CNNs) have made great progresses in semantic segmentation. UNet is a widely used CNN-based semantic segmentation model. However, the conventional CNNs are inherently limited to local receptive fields that only provide short-range contextual information. Pretrained Vision-Language Models (VLMs) such as CLIP have learned rich semantics from web-scale image-text pairs. Inspired by this, we propose a novel CLIP guided UNet framework (CGUnet) for smoke segmentation, which merits the global and rich context of CLIP and the precise localization of UNet. Specifically, we design a CLIP guided cross attention (CGCA) module, in which the CLIP feature is used as the query, and the visual features of UNet as the Key and the Value. We conduct experiments on two public smoke segmentation datasets. Our method achieves SOTA results on both datasets, outperforming other methods.