Enhancing Vulnerability-Fixing Commit Classification: The Synergy of User-Guided and LLM

With the increasing complexity of software development environments, identifying and fixing vulnerabilities has become a key aspect of software maintenance. One way to improve the efficiency and effectiveness of vulnerability-fixing is to classify vulnerability-fixing commits. However, the existing vulnerability-fixing classification methods are limited to code language, code length, commit dataset, ambiguous and domain specialized commits, which leads to low precision. In this paper, we propose a user-guided classification method for vulnerability-fixing commits. For ambiguous and domain specialized commits, we incorporate human involvement and timely intervention in the process of fine-tuning the BERT model. Furthermore, a large language model LLM is employed to address the challenges posed by the variant code language and length. Experiment results show that our approach significantly improves the performance of commit classification. The user-guided BERT message classifier accuracy increases by 2～5 compared with baseline methods after 10 iterations of human participation. Based on the TensorFlow dataset, the patch classifier using LLM outperforms HERMES by 11.6 in terms of F1-score. In summary, our overall classification which combined the results of message classifier and patch classifier outperforms the HERMES by 14.6 and VulCurator by 5.6 .