SpikeRWKV:Energy-efficient Large Language Model with Spiking Neural Network

Spiking Neural Networks SNNs , as the third generation of neural networks, hold great promise for enhancing the energy efficiency of large language models LLMs due to their event-driven computation. However, their naive application in large-scale models typically depends on binary spike simulations over long time steps, making it challenging to balance performance and energy consumption. To address this issue, we propose a Multi-head Spike Encoding scheme with three advantages. First, it enables parallel spike processing to accelerate computation Second, it supports precise representation of positive and negative spikes Third, it mitigates energy surges caused by high-frequency spikes through hierarchical spike decomposition. To demonstrate the effectiveness of our encoding scheme, we introduce SpikeRWKV, an SNN-based adaptation of the RWKV language model. Experimental results demonstrate that SpikeRWKV significantly enhances performance on natural language understanding NLU tasks, achieving a $3.15 times$ reduction in energy consumption compared to the baseline, along with an 8.3 lower perplexity and 5.7 improvement in bits-per-character BPC . Furthermore, SpikeRWKV is $3.88 times$ more energy-efficient than its non-spiking counterpart.