QM-ToT: A Medical Tree of Thoughts Reasoning Framework for Quantized Model

Large language models (LLMs) have achieved substantial progress in biomedical question answering. However, real-world medical applications often operate under resource-constrained settings, where model quantization is a practical necessity for local and privacy-preserving deployment. In such settings, the inherent complexity of clinical reasoning further amplifies the performance degradation of quantized LLMs. To address these issues, we propose Quantized Medical Tree of Thought (QM-ToT), an agentic reasoning framework that enables autonomous, feedback-driven clinical reasoning. QM-ToT operates as a reasoning agent that decomposes complex medical problems into hierarchical plans and autonomously navigates the solution space under dual-evaluation feedback. This framework facilitates substantial performance improvements in INT4-quantized models on the challenging MedQA-USMLE dataset. Specifically, we demonstrate a remarkable accuracy increase from 34\% to 50\% for the LLaMA2-70b model and from 58.77\% to 69.49\% for LLaMA-3.1-8b. Besides, we also proposed an effect data distillation method based on QM-ToT. Compared to the traditional distillation method, we achieved an improvement of 135.7\% while using only 9.8\% of the data.
This work, for the first time, showcases the potential of ToT to
significantly enhance performance on complex biomedical tasks, establishing a crucial foundation for future advances in deploying high-performing quantized LLM in resource-limited medical settings.