Event Driven System Design with High Availability - Nikhil Sagar Miriyala (International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2024)
Modern event-driven architectures rely on frameworks like Apache Kafka to enable real-time asynchronous processing. However, ensuring high availability across distributed systems, particularly in a multi-region environment, poses challenges such as avoiding duplication of data and achieving seamless failover with minimal manual intervention. This paper proposes a robust solution that leverages AWS Managed Streaming for Apache Kafka (MSK) to design an active-active highly available system. By implementing a smart consumer-based idempotent mechanism with a globally shared database and message-level UUIDs, the system ensures zero message duplication while maintaining fault tolerance. This design effectively minimizes latency, supports system resilience, and addresses the challenges of distributed event streaming systems. Practical considerations, including scalability, failure recovery, and latency optimization, are also discussed to provide a comprehensive guide to the deployment of resilient event-driven architectures in cloud-based infrastructures.