PolyRec : Polynomial Attention for Enhanced Sequential Recommendation

The self-attention mechanism has been widely adopted in sequential recommendation due to its powerful capability in modeling long-range dependencies. However, as the number of attention layers increases, user embedding vectors tend to collapse into a low-dimensional subspace. This collapse of embedding space leads to overly concentrated user distributions, resulting in the dimensional collapse phenomenon. The increased similarity among user embedding representations makes it challenging for the model to distinguish between different users, ultimately causing recommendation results to become homogenized. To mitigate this issue, we propose a novel sequential recommendation model named Polynomial Attention for enhanced Sequential Recommendation PolyRec , which alleviates spatial collapse and improves the distribution of user representations. Firstly, the model can better capture high-order structural information through the incorporation of high-order polynomial terms. Simultaneously, leveraging the orthogonality and optimal approximation properties of Chebyshev coefficients stabilizes the parameter training process and enhances the representation capability of the attention mechanism. Furthermore, we conduct a theoretical analysis to demonstrate that during neural networks' aggregation of target information, feature representations are prone to being squeezed by noise and redundant information, thereby exacerbating dimensional collapse. Therefore, by introducing Fourier transforms, we truncate the traditional residual connections in the frequency domain. This approach effectively retains more important information, thereby alleviating the over-squashing phenomena. Experimental evaluations on four benchmark datasets demonstrate that our model outperforms other baseline methods in recommendation accuracy.