Dynamic Low-Rank Routing for Efficient Multi-Task Instruction Fine-Tuning of Large Language Models

Abstract

This paper proposes a multi-task instruction fine-tuning method that integrates low-rank adaptation with dynamic routing to address the challenges of insufficient adaptation efficiency and severe task conflicts in large-scale language models. The method first applies low-rank decomposition to restrict parameter updates to a low-dimensional subspace, which reduces redundant computation and storage costs while preserving semantic representation capacity. On this basis, a dynamic routing mechanism is introduced to adaptively select optimal paths and task-specific modules according to input instruction features, enabling differentiated modeling for different tasks within a shared representation space. This mechanism effectively alleviates performance interference and gradient conflicts commonly encountered in multi-task training, ensuring stable coordination across tasks. To achieve unified optimization, a joint objective function is constructed by combining task loss, low-rank update regularization, and routing distribution constraints, thereby improving stability and controllability during training while maintaining performance. The experimental design covers multi-dimensional analyses of hyperparameter sensitivity, environmental sensitivity, and data sensitivity, verifying the superiority of the proposed method in key metrics such as instruction accuracy, average multi-task performance, parameter efficiency, and task conflict. The results demonstrate that the method achieves efficient fine-tuning and robust performance in complex task environments, providing a solution that balances efficiency and stability for multi-task instruction modeling.