Intelligent fault detection in snowflake-based big data pipelines using federated machine learning

Goli, Harsha Vardhan Reddy (2025) Intelligent fault detection in snowflake-based big data pipelines using federated machine learning. Global Journal of Engineering and Technology Advances, 23 (2). pp. 215-221. ISSN 2582-5003

This article introduces a federated machine learning (FML) framework for detecting faults and anomalies in Snowflake-powered Big Data pipelines. Traditional fault detection systems typically rely on centralized log ingestion, which raises concerns about privacy and latency. In contrast, the proposed FML-based approach enables individual data nodes to train local models on telemetry and workload metadata, such as query failures, slowdowns, and unexpected I/O patterns. These local models then collaborate in a privacy-preserving manner to create a robust global anomaly detection system. Using synthetic workloads designed to simulate financial and healthcare data lakes, this study demonstrates that the FML approach improves fault detection precision by 22% compared to conventional centralized monitoring solutions. The system integrates seamlessly with Snowflake’s metadata and query profiling layers, using external functions and Snowpipe for real-time data ingestion. Additionally, the researchers developed a Snowflake-native dashboard that visualizes detected anomalies and recommends mitigation strategies. The paper concludes with a discussion on the broader impact of secure, distributed AI systems in enterprise data management, illustrating how combining Snowflake’s cloud scalability with federated learning can enhance fault detection, reduce downtime, and pave the way for autonomous data operations in modern data ecosystems.