Fault-Tolerant Data Engineering for Real-Time ETL in Multi-Source Enterprise Systems

Abstract

Real-time ETL is essential for enterprise systems that continuously integrate data from applications, databases, APIs, message queues, cloud services, and event streams. However, multi-source ETL pipelines are highly exposed to source outages, network delays, transformation errors, node failures, duplicate records, and partial destination commits. This article presents a fault-tolerant data engineering architecture for real-time ETL in multi-source enterprise systems. The proposed architecture combines source adapters, stream normalization, layered fault detection, adaptive checkpointing, transaction-aware replay, failover routing, consistency validation, and operational monitoring. The simulated results show that ETL processing latency decreased from 185 ms to 92 ms, fault recovery time decreased from 46 s to 8 s, and data loss rate decreased from 3.8% to 0.2% under the full fault-tolerant architecture. Throughput stability also improved from 72.4% to 96.1%, while checkpoint recovery success increased from 68.5% to 98.4% and consistency preservation improved from 79.2% to 99.1%. These findings indicate that fault tolerance should be embedded across ingestion, transformation, recovery, routing, and loading stages rather than added only as a post-failure recovery function. The proposed architecture provides a practical foundation for reliable, scalable, and continuously available real-time ETL in enterprise data environments.