SOT-BFP: Structure-aware Orthogonal Transformation for LLM Block Floating-Point Quantization

Block floating-point (BFP) quantization is an attractive solution for low-precision inference in large language models (LLMs) due to its wide dynamic range. However, its accuracy is fundamentally constrained by the shared exponent within each block: large within-block magnitude variation causes severe mantissa truncation for small values, leading to noticeable performance degradation. We propose SOT-BFP, a simple yet effective framework that mitigates this exponent-dominated truncation via structure-aware orthogonal transformation. Specifically, SOT-BFP first applies channel permutation to reorganize block composition, and then performs a block-wise orthogonal Hadamard transform to smooth within-block magnitude variation. This design constructs more favorable block structures for BFP quantization, while remaining exactly invertible in full precision and requiring no change to the BFP format or training pipeline. Experiments on widely used LLMs under post-training quantization show that SOT-BFP consistently improves downstream accuracy and reduces perplexity over standard BFP with negligible runtime overhead.