RanpCode: Rank-based Pruning after Complete CP Decomposition for Model Compression

Traditional convolutional neural networks CNNs architecture has limitations such as over-parameterization and a large computational demand. One effective approach to address these issues is to replace convolutional kernels with their low-rank tensor approximations. Among the various methods available, Canonical Polyadic CP tensor decomposition stands out as particularly promising. However, employing CP decomposition for CNNs compression presents two major challenges. First, numerical optimization algorithms used to fit convolutional tensors can cause rank-one tensors to cancel each other out, leading to instability and complicating the fine-tuning of the resulting model. Second, determining the appropriate rank for CP decomposition is inherently complex. To overcome these challenges, we propose RanpCode, a novel compression method based on CP decomposition. This method incorporates specially designed numerical fitting techniques to ensure complete CP decomposition and address instability issues. Furthermore, it employs a rank pruning scheme to automatically determine the optimal rank for each layer, with the rank globally optimized and adjusted according to the desired compression rate. Our evaluations on popular CNNs architectures for image classification demonstrate that RanpCode achieves higher compression rates while maintaining superior accuracy.