Shadow Model Craft: An Efficient Framework for Privacy-Preserving Inference

Privacy-preserving inference PPI has become a critical requirement in Machine Learning as a Service MLaaS , where both user inputs and model parameters are sensitive assets. Existing cryptographic-based approaches, such as those relying on homomorphic encryption or secure multi-party computation MPC , often suffer from substantial computational and communication overhead, making them impractical for large-scale deep learning models. In this paper, we propose a novel and efficient framework that protects model and input privacy while significantly improving inference efficiency. The core of our approach is textit{Shadow Model Craft}, a structural model decomposition strategy inspired by secret sharing. Instead of encrypting model parameters, we distill the original model into multiple lightweight shadow models with disjoint functionality and distribute them across non-colluding servers. Each server performs inference over secret-shared inputs using plaintext model fragments, thus eliminating the need for encrypted model parameters. Our design allows local execution of linear operations, further reducing inference latency. Experiments on CIFAR-10 and ImageNet demonstrate that our framework achieves strong privacy guarantees with up to 90 model compression and over remarkable speedup compared to other sota works, all while maintaining competitive inference accuracy. This work offers a practical and scalable solution for secure deep learning inference in real-world deployments.