Learning-Driven Collaborative Scheduling for Multi-Tenant Distributed Inference Services

Abstract

To address the scheduling complexity of multi-tenant distributed inference services under shared resources, model heterogeneity, and dynamic request conditions, this paper proposes a learning driven collaborative scheduling mechanism that models inference scheduling from a holistic system perspective. The approach abstracts system states and formulates scheduling decisions as a learnable mapping process. This enables the scheduling policy to dynamically adapt to workload variations and resource conditions during continuous operation. To capture intrinsic relationships in multi-tenant concurrent scenarios, collaborative utility modeling is incorporated into the scheduling objective. This guides scheduling behavior toward coordinated constraints among resource utilization efficiency, service stability, and tenant fairness. The proposed framework avoids strong dependence on specific execution details and provides a unified system-level treatment of request assignment and resource scheduling. It mitigates performance degradation caused by resource contention and workload fluctuation in multi-tenant parallel inference. System-level evaluation in multi-tenant distributed inference scenarios demonstrates that the method achieves strong overall performance in service response, system throughput, operational stability, and tenant fairness. These results confirm the effectiveness and practicality of learning driven collaborative scheduling in complex inference service environments.