SlideShare una empresa de Scribd logo
1 de 30
Descargar para leer sin conexión
Fan-out, fan-in & the multiplexer
Replication recipes for global platform distribution
Introduction
Alex Hilton
Engineering Manager
Foreign Exchange Team
Passions
- ! Family
- 🍹 Mixology
- 🎲 Online Catan
- 🚀 Building the worlds next gen financial infrastructure
+ +
Global Platform Distribution Challenge
How best to centrally manage events generated in remote clusters?
HQ
Distribution across remote clusters yields great portability
Global Platform Distribution Challenge
How best to centrally manage events generated in remote clusters?
• Why does this challenge exist?
• Kafka distribution best practices & mistakes
• Mirror Maker replication recipes
• Topic Multiplexing
• Separate global distribution concerns from application domain
• Improve observability & monitoring
1
2
3
Information Distribution Challenge
Foreign Exchange
Platform
FX Partners
(Centralized)
Globally distributed APIs:
• Payment
• Transfer
• Card
• Wallet
Information Distribution Challenge
Foreign Exchange
Platform
Desired Experience:
🌏 Local
🚀 Fast
⭐ High availability
Our Constraints:
🎯 Central connections
Aggregation opportunities
Key Learning – Kafka promotes decomposition
Isolation benefits
• Single responsibility
• Single writer
From monolith… …to pluggable microservice pipeline
Why is Kafka a good fit? Pluggability
Kafka is a universal application dependency
Workflows modelled as chain of microservices
Pluggability benefits
- Unburden critical path
- Support innovation
- Composable capability
- Portability
API on Kafka Pattern
Price distribution
API on Kafka Pattern
• Push state & all decision making to very edge of system
• Fast API response time
🚀 Fast (API latency)
• Read API driven from in-memory state
• Write API publish instructions as events
⭐ High availability
• Convenient horizonal scalability
• Reduce dependencies on critical path
🌏 Local
• How to solve…? Instruction execution
Learnings – Simplify the critical path for API responses
• Burden Client with Kafka RPC
• Exactly once semantics
Do Don't
• Embrace eventual consistency
• Enforce idempotency
Global Distribution Pattern 🌏
Global Distribution Pattern
• Remote Kafka clusters power global API distribution
• Mirror maker (v2) used to synchronize clusters
• Fan out vs Fan in
🚀 Fast (API latency)
• Internalize the cost of WAN distribution
⭐ High availability
• Opportunities for DR / cross region fallback (cross cluster)
🌏 Local
• We can deploy APIs near customers
• Kafka yields application portability & composability
M
irror m
aker
M
irror m
aker
Multi-cluster Challenges - Fan out
Fan out replication Pattern
• Mirror maker pull model
• Avoid monolithic replication processor
• MM instance deployed to each remote region
• Topic data is cloned (but not offsets).
• Topic name remains the same
• Offsets vary
Mirror Maker Best Practice - Isolation
MM Isolation Best Practice
• Separate MM instances for each region
• Isolate meta data per source-destination combination
• Version meta data to support case reset / remodel
⭐ Benefits
• Limit blast radius of cluster stability issues
• Flexibility to shutdown / maintain regions
• Migration pathway to support MM upgrades
MM Kafka cluster naming convention:
version.source.dest.[dest,src]
v5 . HQ . US . dest
Sample Mirror Maker Metadata
Fan in… and the multiplexer
Multi-cluster Challenges - Fan in
Fan in challenge
Compromised isolation on merged topic
• Harder to reconcile / monitor
• Violate single writer principal
Multi-cluster Challenges - Fan in
Fan in – better isolation
Isolation ✅
• Region, topic, mirror maker
Portability ❌
• Application & env burden
• API & topic region
• Processor consumption
Scalability 😐
• Multiple workflows
Monitoring 😐
• How to monitor replication during off peak periods?
Fan in – the naïve way
Multiplexing to the rescue!
a method by which multiple analog or digital signals are combined into one signal over a shared medium.
The aim is to share a scarce resource.
Multiplexing to the rescue!
Could multiplexing techniques help us to abstract distribution / replication concerns away from
application / business domain?
Apply multiplex concept to fan-in challenge
We can solve isolation, distribution, portability, replication monitoring
concerns without burdening our application
• Multiplex service can interlace all source topics into a single channel
• Simpler replication, easier to isolate, easier to monitor
• Demux service can deinterlace multiplexed byte stream
Multiplexing unburdens application domain from distribution concern
Isolation ✅
Portability ✅
Scalability ✅
Monitoring ❓
Generating Mux Stream & Partitioning
Multiplexing is not difficult – entire mux service is 300 lines of Kotlin
Respect source partition sequence.
PartitionKey = srcTopic + srcPartition
Can use ByteArray serdes to copy
record key & value to mux topic.
No need to use original schema
Kafka headers provide convenient way
to carry original context to demux
(the source partition & topic name)
Kafka Header Metadata
Improving observability & monitoring
Throughput and latency metrics
• Kafka mux headers carry source region to help improve event observability
• Demultiplexer exposes latency / throughput metrics to power rich dashboard
Improving observability & monitoring
Improving observability & monitoring
Dead replication alerts
• Synthetic ‘keepalive’ events generated to keep replication & monitor warm during off-peak time
• Synthetic events internalized to multiplexer / demultiplexer & do not affect business workflow
Mux Dashboard
Kafka Distribution Patterns – Fan in Fan out & the Multiplexer
✅ Isolation
✅ Portability
✅ Scalability
✅ Monitoring
🌏 Local
🚀 Fast
⭐ High availability
Fan In + Multiplexer
Fan Out
Conclusions
- Install Kafka cluster in each customer region
- Keep API processing simple
- Strive for strong isolation between regions
- Separate application & distribution concerns
What Next…?
Support Multiple HQ Regions
Share The Multiplexer Code
Read more on our Blog!
medium.com/airwallex-engineering
Feedback / questions: alex.hilton@airwallex.com
airwallex.com/careers

Más contenido relacionado

La actualidad más candente

Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...
Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...
Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...HostedbyConfluent
 
Event Sourcing & CQRS, Kafka, Rabbit MQ
Event Sourcing & CQRS, Kafka, Rabbit MQEvent Sourcing & CQRS, Kafka, Rabbit MQ
Event Sourcing & CQRS, Kafka, Rabbit MQAraf Karsh Hamid
 
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...HostedbyConfluent
 
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyEvent Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyBob Rhubart
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...confluent
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streamsconfluent
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent
 
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...Chris Richardson
 
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and DevelopersApache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developersconfluent
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETconfluent
 
Building a Streaming Microservice Architecture: with Apache Spark Structured ...
Building a Streaming Microservice Architecture: with Apache Spark Structured ...Building a Streaming Microservice Architecture: with Apache Spark Structured ...
Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks
 
Kafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presentedKafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presentedSumant Tambe
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 
How Orange Financial combat financial frauds over 50M transactions a day usin...
How Orange Financial combat financial frauds over 50M transactions a day usin...How Orange Financial combat financial frauds over 50M transactions a day usin...
How Orange Financial combat financial frauds over 50M transactions a day usin...StreamNative
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystemconfluent
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connectconfluent
 
[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자Oracle Korea
 

La actualidad más candente (20)

Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...
Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...
Evergreen: Building Airbnb’s Merge Queue With Kafka Streams with Janusz Kudel...
 
Event Sourcing & CQRS, Kafka, Rabbit MQ
Event Sourcing & CQRS, Kafka, Rabbit MQEvent Sourcing & CQRS, Kafka, Rabbit MQ
Event Sourcing & CQRS, Kafka, Rabbit MQ
 
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...
3 Kafka patterns to deliver Streaming Machine Learning models with Andrea Spi...
 
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu KrishnaswamyEvent Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
Event Driven Architecture (EDA) Reference Architecture | Anbu Krishnaswamy
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streams
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
 
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...
Oracle CodeOne 2019: Descending the Testing Pyramid: Effective Testing Strate...
 
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and DevelopersApache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developers
 
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NETStream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
 
Building a Streaming Microservice Architecture: with Apache Spark Structured ...
Building a Streaming Microservice Architecture: with Apache Spark Structured ...Building a Streaming Microservice Architecture: with Apache Spark Structured ...
Building a Streaming Microservice Architecture: with Apache Spark Structured ...
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafka
 
Kafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presentedKafka tiered-storage-meetup-2022-final-presented
Kafka tiered-storage-meetup-2022-final-presented
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 
How Orange Financial combat financial frauds over 50M transactions a day usin...
How Orange Financial combat financial frauds over 50M transactions a day usin...How Orange Financial combat financial frauds over 50M transactions a day usin...
How Orange Financial combat financial frauds over 50M transactions a day usin...
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
 
[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자[Main Session] 카프카, 데이터 플랫폼의 최강자
[Main Session] 카프카, 데이터 플랫폼의 최강자
 

Similar a Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
 
The evolution of data center network fabrics
The evolution of data center network fabricsThe evolution of data center network fabrics
The evolution of data center network fabricsCisco Canada
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stackNitin Mehta
 
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBMAvailability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBMHostedbyConfluent
 
Taming the cost of your first cloud - CCCEU 2014
Taming the cost of your first cloud - CCCEU 2014Taming the cost of your first cloud - CCCEU 2014
Taming the cost of your first cloud - CCCEU 2014Tim Mackey
 
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018Amazon Web Services Korea
 
Istio as an enabler for migrating to microservices (edition 2022)
Istio as an enabler for migrating to microservices (edition 2022)Istio as an enabler for migrating to microservices (edition 2022)
Istio as an enabler for migrating to microservices (edition 2022)Ahmed Misbah
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesQAware GmbH
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesJosef Adersberger
 
Caching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session ICaching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session IVMware Tanzu
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithMarkus Eisele
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.Taras Matyashovsky
 
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...Peter Broadhurst
 
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaEvent Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaDataWorks Summit
 
Legion - AI Runtime Platform
Legion -  AI Runtime PlatformLegion -  AI Runtime Platform
Legion - AI Runtime PlatformAlexey Kharlamov
 
DockerCon EU 2015: Using Docker and SDN for telco-grade applications
DockerCon EU 2015: Using Docker and SDN for telco-grade applicationsDockerCon EU 2015: Using Docker and SDN for telco-grade applications
DockerCon EU 2015: Using Docker and SDN for telco-grade applicationsDocker, Inc.
 

Similar a Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex (20)

Microservices.pdf
Microservices.pdfMicroservices.pdf
Microservices.pdf
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
 
The evolution of data center network fabrics
The evolution of data center network fabricsThe evolution of data center network fabrics
The evolution of data center network fabrics
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
 
Hacking apache cloud stack
Hacking apache cloud stackHacking apache cloud stack
Hacking apache cloud stack
 
Designing microservices
Designing microservicesDesigning microservices
Designing microservices
 
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBMAvailability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM
Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM
 
Taming the cost of your first cloud - CCCEU 2014
Taming the cost of your first cloud - CCCEU 2014Taming the cost of your first cloud - CCCEU 2014
Taming the cost of your first cloud - CCCEU 2014
 
Newt global meetup microservices
Newt global meetup microservicesNewt global meetup microservices
Newt global meetup microservices
 
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018
Amazon EKS 그리고 Service Mesh (김세호 솔루션즈 아키텍트, AWS) :: Gaming on AWS 2018
 
Istio as an enabler for migrating to microservices (edition 2022)
Istio as an enabler for migrating to microservices (edition 2022)Istio as an enabler for migrating to microservices (edition 2022)
Istio as an enabler for migrating to microservices (edition 2022)
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Caching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session ICaching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session I
 
Stay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolithStay productive_while_slicing_up_the_monolith
Stay productive_while_slicing_up_the_monolith
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
 
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...
IBM IMPACT 2014 - AMC-1882 Building a Scalable & Continuously Available IBM M...
 
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaEvent Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache Kafka
 
Legion - AI Runtime Platform
Legion -  AI Runtime PlatformLegion -  AI Runtime Platform
Legion - AI Runtime Platform
 
DockerCon EU 2015: Using Docker and SDN for telco-grade applications
DockerCon EU 2015: Using Docker and SDN for telco-grade applicationsDockerCon EU 2015: Using Docker and SDN for telco-grade applications
DockerCon EU 2015: Using Docker and SDN for telco-grade applications
 

Más de HostedbyConfluent

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsHostedbyConfluent
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...HostedbyConfluent
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...HostedbyConfluent
 
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...HostedbyConfluent
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixHostedbyConfluent
 
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouseHostedbyConfluent
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...HostedbyConfluent
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformHostedbyConfluent
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningHostedbyConfluent
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...HostedbyConfluent
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...HostedbyConfluent
 
Configuring Kafka Connect To Be Successful At Scale
Configuring Kafka Connect To Be Successful At ScaleConfiguring Kafka Connect To Be Successful At Scale
Configuring Kafka Connect To Be Successful At ScaleHostedbyConfluent
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsHostedbyConfluent
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...HostedbyConfluent
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...HostedbyConfluent
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleHostedbyConfluent
 
Dynamic Change Data Capture with Flink CDC and Consistent Hashing
Dynamic Change Data Capture with Flink CDC and Consistent HashingDynamic Change Data Capture with Flink CDC and Consistent Hashing
Dynamic Change Data Capture with Flink CDC and Consistent HashingHostedbyConfluent
 
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...HostedbyConfluent
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2HostedbyConfluent
 

Más de HostedbyConfluent (20)

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
 
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at Netflix
 
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse
(real-time)²: Real-time data for real-time analytics with Kafka and ClickHouse
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment Platform
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine Learning
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
 
Configuring Kafka Connect To Be Successful At Scale
Configuring Kafka Connect To Be Successful At ScaleConfiguring Kafka Connect To Be Successful At Scale
Configuring Kafka Connect To Be Successful At Scale
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered Applications
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
 
Streaming is a Detail
Streaming is a DetailStreaming is a Detail
Streaming is a Detail
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at Scale
 
Dynamic Change Data Capture with Flink CDC and Consistent Hashing
Dynamic Change Data Capture with Flink CDC and Consistent HashingDynamic Change Data Capture with Flink CDC and Consistent Hashing
Dynamic Change Data Capture with Flink CDC and Consistent Hashing
 
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...
A Glide, Skip or a Jump: Efficiently Stream Data into Your Medallion Architec...
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
 

Último

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 

Último (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 

Fan-out, fan-in & the multiplexer: Replication recipes for global platform distribution | Alex Hilton, Airwallex

  • 1. Fan-out, fan-in & the multiplexer Replication recipes for global platform distribution
  • 2. Introduction Alex Hilton Engineering Manager Foreign Exchange Team Passions - ! Family - 🍹 Mixology - 🎲 Online Catan - 🚀 Building the worlds next gen financial infrastructure + +
  • 3. Global Platform Distribution Challenge How best to centrally manage events generated in remote clusters? HQ Distribution across remote clusters yields great portability
  • 4. Global Platform Distribution Challenge How best to centrally manage events generated in remote clusters? • Why does this challenge exist? • Kafka distribution best practices & mistakes • Mirror Maker replication recipes • Topic Multiplexing • Separate global distribution concerns from application domain • Improve observability & monitoring 1 2 3
  • 5. Information Distribution Challenge Foreign Exchange Platform FX Partners (Centralized) Globally distributed APIs: • Payment • Transfer • Card • Wallet
  • 6. Information Distribution Challenge Foreign Exchange Platform Desired Experience: 🌏 Local 🚀 Fast ⭐ High availability Our Constraints: 🎯 Central connections Aggregation opportunities
  • 7. Key Learning – Kafka promotes decomposition Isolation benefits • Single responsibility • Single writer From monolith… …to pluggable microservice pipeline
  • 8. Why is Kafka a good fit? Pluggability Kafka is a universal application dependency Workflows modelled as chain of microservices Pluggability benefits - Unburden critical path - Support innovation - Composable capability - Portability
  • 9. API on Kafka Pattern Price distribution API on Kafka Pattern • Push state & all decision making to very edge of system • Fast API response time 🚀 Fast (API latency) • Read API driven from in-memory state • Write API publish instructions as events ⭐ High availability • Convenient horizonal scalability • Reduce dependencies on critical path 🌏 Local • How to solve…? Instruction execution
  • 10. Learnings – Simplify the critical path for API responses • Burden Client with Kafka RPC • Exactly once semantics Do Don't • Embrace eventual consistency • Enforce idempotency
  • 11. Global Distribution Pattern 🌏 Global Distribution Pattern • Remote Kafka clusters power global API distribution • Mirror maker (v2) used to synchronize clusters • Fan out vs Fan in 🚀 Fast (API latency) • Internalize the cost of WAN distribution ⭐ High availability • Opportunities for DR / cross region fallback (cross cluster) 🌏 Local • We can deploy APIs near customers • Kafka yields application portability & composability M irror m aker M irror m aker
  • 12. Multi-cluster Challenges - Fan out Fan out replication Pattern • Mirror maker pull model • Avoid monolithic replication processor • MM instance deployed to each remote region • Topic data is cloned (but not offsets). • Topic name remains the same • Offsets vary
  • 13. Mirror Maker Best Practice - Isolation MM Isolation Best Practice • Separate MM instances for each region • Isolate meta data per source-destination combination • Version meta data to support case reset / remodel ⭐ Benefits • Limit blast radius of cluster stability issues • Flexibility to shutdown / maintain regions • Migration pathway to support MM upgrades MM Kafka cluster naming convention: version.source.dest.[dest,src] v5 . HQ . US . dest
  • 15. Fan in… and the multiplexer
  • 16. Multi-cluster Challenges - Fan in Fan in challenge Compromised isolation on merged topic • Harder to reconcile / monitor • Violate single writer principal
  • 17. Multi-cluster Challenges - Fan in Fan in – better isolation Isolation ✅ • Region, topic, mirror maker Portability ❌ • Application & env burden • API & topic region • Processor consumption Scalability 😐 • Multiple workflows Monitoring 😐 • How to monitor replication during off peak periods?
  • 18. Fan in – the naïve way
  • 19. Multiplexing to the rescue! a method by which multiple analog or digital signals are combined into one signal over a shared medium. The aim is to share a scarce resource.
  • 20. Multiplexing to the rescue! Could multiplexing techniques help us to abstract distribution / replication concerns away from application / business domain?
  • 21. Apply multiplex concept to fan-in challenge We can solve isolation, distribution, portability, replication monitoring concerns without burdening our application • Multiplex service can interlace all source topics into a single channel • Simpler replication, easier to isolate, easier to monitor • Demux service can deinterlace multiplexed byte stream
  • 22. Multiplexing unburdens application domain from distribution concern Isolation ✅ Portability ✅ Scalability ✅ Monitoring ❓
  • 23. Generating Mux Stream & Partitioning Multiplexing is not difficult – entire mux service is 300 lines of Kotlin Respect source partition sequence. PartitionKey = srcTopic + srcPartition Can use ByteArray serdes to copy record key & value to mux topic. No need to use original schema Kafka headers provide convenient way to carry original context to demux (the source partition & topic name)
  • 25. Improving observability & monitoring Throughput and latency metrics • Kafka mux headers carry source region to help improve event observability • Demultiplexer exposes latency / throughput metrics to power rich dashboard
  • 27. Improving observability & monitoring Dead replication alerts • Synthetic ‘keepalive’ events generated to keep replication & monitor warm during off-peak time • Synthetic events internalized to multiplexer / demultiplexer & do not affect business workflow
  • 29. Kafka Distribution Patterns – Fan in Fan out & the Multiplexer ✅ Isolation ✅ Portability ✅ Scalability ✅ Monitoring 🌏 Local 🚀 Fast ⭐ High availability Fan In + Multiplexer Fan Out Conclusions - Install Kafka cluster in each customer region - Keep API processing simple - Strive for strong isolation between regions - Separate application & distribution concerns
  • 30. What Next…? Support Multiple HQ Regions Share The Multiplexer Code Read more on our Blog! medium.com/airwallex-engineering Feedback / questions: alex.hilton@airwallex.com airwallex.com/careers