Roberta-MHARC: Enhanced Telecom Fraud Detection with Multi-head Attention and Residual Connection

Telecom fraud has become one of the hardest-hit areas in the criminal field. With the development of artificial intelligence technology, telecom fraud texts have become highly concealed and deceptive. Existing prevention methods, such as mobile phone number tracking, detection, and traditional machine learning model text recognition, lack real-time performance in identifying telecom fraud. Furthermore, due to the scarcity of Chinese telecom fraud text data, the accuracy of recognizing Chinese telecom fraud text is not high. In this paper, we design a telecom fraud text decision-making model Roberta-MHARC based on Roberta combined with multi-head attention mechanism and residual connection. First, the model selects some categories of data in the CCL2023 telecom network fraud data set as basic samples, and combines it with the collected telecom fraud text data to form a five-category covering impersonating customer service, impersonating leadership acquaintances, loans, public security fraud, and normal text data set. Secondly, during the training process, the model adds a multi-head attention mechanism and improves the training speed through residual connections. Finally, the model improves its accuracy on multi-classification tasks by introducing an inconsistency loss function alongside the cross-entropy loss. Experimental results show that our model achieves good results on multiple benchmark datasets.