ProtoHGC: Heterogeneous Graph Contrastive Learning with Prototype-Regularized Classification for Transaction Fraud Detection

Transaction fraud detection on digital payment platforms poses three inter-twined challenges: sparse supervision, severe class imbalance, and complex relational dependencies among accounts and transaction events. This paper studies fraud detection on a heterogeneous transaction graph constructed from the public PaySim simulator. To this end, we propose ProtoHGC, a framework that integrates heterogeneous graph contrastive pre-training with prototype-regularized classification. ProtoHGC employs a multi-scale en-coder that aggregates first-order neighborhood features and second-order meta-path-based context, coupled with node-level and subgraph-level con-trastive objectives to enhance representation quality under limited supervi-sion. Building on the learned embeddings, class-specific prototypes for normal and fraudulent behaviors enforce intra-class compactness and inter-class separation. Experiments on PaySim demonstrate that ProtoHGC con-sistently outperforms GCN, GAT, GraphSAGE, and MLP baselines across all tested hidden dimensions in terms of AUC-ROC, PR-AUC, accuracy, re-call, and F1-score. We explicitly note that the current evaluation is limited to a single synthetic dataset; few-shot adaptation, cross-domain generaliza-tion, and multimedia-specific fraud scenarios remain unvalidated.