Automatic Orchestration of Enterprise ETL Processes via Graph Neural Network and Large Language Model Collaborative Reasoning

Abstract

As data source types, field semantics, quality rules, and task dependencies in enterprise data platforms become increasingly complex, traditional ETL process orchestration methods relying on manual configuration and static rules struggle to meet the demands for automation, consistency, and reliability in large-scale enterprise data processing. Addressing the limitations of traditional ETL process orchestration methods, which suffer from insufficient cross-source relationship identification, inadequate business semantic understanding, and the ever-increasing complexity of contract types, field semantics, quality rules, and task dependencies, this paper proposes an automatic orchestration method based on collaborative reasoning using graph neural networks and a large language model. This method first models data sources, fields, rules, and tasks as a unified ETL state graph and uses relation-aware graph representation learning to capture high-order relationships between field mappings, task dependencies, and rule constraints. Subsequently, it constructs a structural semantic joint context by combining business requirement text, data pattern information, and rule descriptions, enabling the large language model to complete task decomposition, process reasoning, and candidate ETL process generation under graph-structured evidence constraints. Furthermore, the method introduces constraint consistency verification and a multi-criteria scoring mechanism to screen candidate processes based on dependency rationality, rule coverage, and execution feasibility, thereby improving the stability and reliability of the automatic orchestration results. Comparative experiments based on publicly available enterprise data integration scenarios show that the proposed method has good overall performance in terms of process generation correctness, dependency consistency, rule coverage, and execution success rate, and can provide effective support for enterprise data warehouse construction, automated data governance, and intelligent data engineering.