GFSM-DETR: A Gated Frequency-Spatial Modulation Detector for Real-Time Small Face Detection

Real-time and accurate dense small-scale face detection constitutes a critical technology in computer vision. However, in complex environments, this task presents significant challenges, including high target density, minute scales, frequent mutual occlusions, and background noise interference. These factors often cause models to lose critical fine-grained features, leading to severe missed detections. To address these challenges, this paper proposes a high-accuracy real-time detector (GFSM-DETR) specifically optimized for dense small-scale face detection. First, the GFSM-Fusion module is designed to utilize synergistic representation learning across the spatial and frequency domains, effectively extracting and amplifying high-frequency features of small targets while suppressing background noise. Furthermore, the ECA-CATM module is constructed to facilitate noise-resistant global context modeling through multi-stage information interactions within the spatial and channel domains combined with an additive attention mechanism. In addition, the proposed Focal-PIoUv2 loss function utilizes a dual-constraint mechanism to effectively filter gradient interference generated by low-quality predicted boxes. On the SCUT-Head and Brainwash datasets, the proposed model significantly outperforms the baseline RT-DETR and a series of mainstream detectors, achieving mAP50 scores of 95.2% and 95.7%, respectively. Compared to leading mainstream detectors including YOLOv11m and YOLO26m, our model realizes mAP50 gains of 2.5% and 2.0%, along with Recall improvements of 3.9% and 2.0% across the two datasets. These results indicate that the proposed model provides a high-accuracy, low-cost solution for real-time deployment on edge and mobile devices.