Transformer-Based Anomaly Detection in Deep Reinforcement Learning

Despite promising potential of Deep Reinforcement Learning DRL to adapt to various tasks, it remains highly vulnerable to adversarial attacks or anomalous observation signals. Existing research on DRL robustness does not fully safeguard against all types of adversarial perturbations and disturbances, which hampers its use in critical real-world systems and applications, such as smart grids, traffic control, and autonomous vehicles. In these contexts, anomalous states can cause decision-making errors that may be exacerbated in subsequent actions, resulting in irreparable damage. To promptly detect anomalous states and prevent greater losses, we propose the Transformer-Based Anomaly Detection T-BAD framework. Utilizing the transformer's ability to handle sequential data robustly, T-BAD enables real-time detection of anomalous states and actions. Specifically, first, our approach involves collecting trajectory data from the DRL algorithm used by the agent and training a transformer module to accurately model the agent's actions. For anomaly detection, we input k consecutive states and k-1 consecutive actions excluding the current action into the transformer, and then compare its output with the agent's action. The difference between them indicates whether DRL model is influenced by interference or adversarial attacks. Extensive experiments across multiple scenarios in Atari and Mujoco demonstrate that our T-BAD outperforms existing baselines in anomaly detection while also possessing some capacity for anomaly correction.