Adaptive Scheduling for Multi-Model Collaborative Distributed Inference under Resource Heterogeneity and Dynamic Workloads

Abstract

This paper addresses the scheduling complexity of distributed inference systems in multi-model collaborative scenarios under resource heterogeneity, model diversity, and dynamic request conditions. An adaptive scheduling method for distributed inference systems with multi-model collaboration is proposed. From a system-level perspective, the method integrates model selection and resource allocation into a unified scheduling decision process. Coordinated management of multi-model inference workflows is achieved through comprehensive characterization of system states. The proposed scheduling mechanism dynamically adjusts its decisions in response to changes in the operating environment and maintains a reasonable execution order under shared resource constraints, thereby improving overall system performance. Comparative experiments conducted in multi-model collaborative inference settings demonstrate that the proposed method achieves clear advantages in resource utilization efficiency, load distribution rationality, system stability, and scheduling responsiveness. The method effectively mitigates performance degradation caused by resource contention and model heterogeneity during parallel multi-model execution. The results indicate that a unified adaptive scheduling design plays an important role in supporting complex intelligent inference services and provides valuable guidance for the engineering implementation and optimization of distributed inference systems.