AI-Driven Strategic Modernization of Legacy ETL Workflows on Serverless Cloud Platforms

Abstract

Legacy ETL systems create considerable operational overhead with manual capacity management, rigid scheduling models, and inadequate cloud integration support. The proposed framework in this article presents a systematic solution for migrating legacy ETL processes to serverless cloud environments, overcoming essential shortfalls through AI-aided workflow classification, natural language processing-assisted code translation, and event-oriented orchestration. Unlike previous lift-and-shift or containerized models with operational inflexibilities, this framework embraces fully managed serverless services integrated with machine learning functions. The result is considerable cost savings, performance enhancements via distributed processing engines, and improved data freshness. The framework involves automated inventory documentation via metadata-driven models, AI-aided complexity stratification across heterogeneous data, Apache Spark-based code refactoring using Business Process Model and Notation templates, and reliable, fault-tolerant, trigger-activated orchestrations. The model ensures improved reliability, scalability, and standards compliance while eliminating infrastructure-related operational issues. Challenges in metadata completeness gaps, translation processes with semantic drift, artificial intelligence-related bias, and operational constraints are addressed comprehensively through complete provenance tracking and parallel validations. The proposed framework outperforms current serverless cloud-migration solutions through intelligent automation and pattern-driven optimization. In an anonymized enterprise case study, the framework reduced end-to-end batch processing time and infrastructure cost while improving workflow success rates and migration throughput compared to the legacy environment. A formal evaluation section details workflow classification quality, translation accuracy, and operational improvements observed during the migration program.