The role of AI in modern data engineering: automating ETL and beyond

Kasireddy, Janardhan Reddy (2025) The role of AI in modern data engineering: automating ETL and beyond. World Journal of Advanced Engineering Technology and Sciences, 15 (1). pp. 1206-1221. ISSN 2582-8266

Artificial intelligence is transforming data engineering by enhancing traditional Extract, Transform, Load (ETL) processes with adaptive, self-optimizing systems. As organizations confront growing data volumes and complexity, AI offers solutions that extend beyond conventional approaches, introducing capabilities for automated schema detection, intelligent data quality management, performance optimization, and natural language interfaces. These advancements enable dynamic adaptation to changing data structures, sophisticated anomaly detection, resource allocation optimization, and more intuitive human-system interactions. Across financial services, manufacturing, and healthcare sectors, AI-driven data pipelines demonstrate substantial improvements in fraud detection, IoT data processing, and patient data harmonization. While challenges persist in explainability, training data requirements, governance, and skill transitions, the future points toward augmentation rather than replacement—creating synergistic partnerships between human expertise and machine intelligence that combine strategic thinking with pattern recognition at scale.