Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Connect @
Twitter Scale
Ryanne Dolan @dolanRyanne
Jordan Bull @jordangbull
2
Project: CloudShuttle
Kafka
BigQuery
DC 1
Cache
MySQL
DC 2
Manhattan
Cloud Region 1
PubSub
BigQuery
Cloud Region 2
PubSu...
3
Connectors
PubSubSource PubSubSink
ManhattanSink
NighthawkSink
TwemcacheSink
KinesisSource
KinesisSink
MySQLSource
Mirro...
Text Boxes
4
A Brief History of
Connect @Twitter
5
First pass: one Connect
cluster per Kafka cluster
Kafka Connect GCP PubSub
6
Enter the PubSub
Source Connector
Kafka Connect GCP PubSub
7
Challenges with the
Topology
● Operational Burden
● Capacity Modeling
● Large Blast Radius
8
Our Solution
Kafka Connect GCP PubSub
PubSub Source
PubSub Sink
Text Boxes
9
Rejectected
Alternatives
10
Megacluster
Main Kafka
11
Miniclusters
Text Boxes
12
Operating
Connect as
a Service
Code on background
13
Configuration
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.strea...
14
Monitoring SLOs
15
SLO: Latency
Kafka Connect Destination
T1 T2
16
SLO: Delivery
Kafka Connect Destination
%
17
Additional Alarms
● Failed Tasks
● Config Load Errors
18
KIPs!
● IdentityReplicationPolicy (382) ✅
● Record Rate Limiting (731)
● Delete Committed Records (762)
● Latency Metri...
Thank You
19
Ryanne Dolan @dolanRyanne
Jordan Bull @jordangbull
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

Download to read offline

Twitter has one of the largest Kafka fleets in the world, handling hundreds of millions of events per second. In order to operate Kafka Connect at this scale, we've had to get creative. In this talk we'll present some of the problems we've run into with Kafka Connect, and how we've engineered around them.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

  1. 1. Connect @ Twitter Scale Ryanne Dolan @dolanRyanne Jordan Bull @jordangbull
  2. 2. 2 Project: CloudShuttle Kafka BigQuery DC 1 Cache MySQL DC 2 Manhattan Cloud Region 1 PubSub BigQuery Cloud Region 2 PubSub ?
  3. 3. 3 Connectors PubSubSource PubSubSink ManhattanSink NighthawkSink TwemcacheSink KinesisSource KinesisSink MySQLSource MirrorSink Kafka MirrorSource
  4. 4. Text Boxes 4 A Brief History of Connect @Twitter
  5. 5. 5 First pass: one Connect cluster per Kafka cluster Kafka Connect GCP PubSub
  6. 6. 6 Enter the PubSub Source Connector Kafka Connect GCP PubSub
  7. 7. 7 Challenges with the Topology ● Operational Burden ● Capacity Modeling ● Large Blast Radius
  8. 8. 8 Our Solution Kafka Connect GCP PubSub PubSub Source PubSub Sink
  9. 9. Text Boxes 9 Rejectected Alternatives
  10. 10. 10 Megacluster Main Kafka
  11. 11. 11 Miniclusters
  12. 12. Text Boxes 12 Operating Connect as a Service
  13. 13. Code on background 13 Configuration cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000 cps.streamingPull.parallelStreams: 4 cps.streamingPull.maxAckExtensionMs: 10000 cps.streamingPull.maxMsPerAckExtension: 10000 kafka.record.headers: true kafka.partition.scheme: kafka_partitioner header.converter: org.apache.kafka.connect.storage.StringConverter key.converter: org.apache.kafka.connect.storage.StringConverter value.converter: org.apache.kafka.connect.converters.ByteArrayConverter cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000 cps.streamingPull.parallelStreams: 4 cps.streamingPull.maxAckExtensionMs: 10000 cps.streamingPull.maxMsPerAckExtension: 10000 kafka.record.headers: true kafka.partition.scheme: kafka_partitioner header.converter: org.apache.kafka.connect.storage.StringConverter key.converter: org.apache.kafka.connect.storage.StringConverter value.converter: org.apache.kafka.connect.converters.ByteArrayConverter cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000 cps.streamingPull.parallelStreams: 4 cps.streamingPull.maxAckExtensionMs: 10000 cps.streamingPull.maxMsPerAckExtension: 10000 kafka.record.headers: true kafka.partition.scheme: kafka_partitioner header.converter: org.apache.kafka.connect.storage.StringConverter key.converter: org.apache.kafka.connect.storage.StringConverter value.converter: org.apache.kafka.connect.converters.ByteArrayConverter cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000 cps.streamingPull.parallelStreams: 4 cps.streamingPull.maxAckExtensionMs: 10000 cps.streamingPull.maxMsPerAckExtension: 10000 kafka.record.headers: true kafka.partition.scheme: kafka_partitioner header.converter: org.apache.kafka.connect.storage.StringConverter key.converter: org.apache.kafka.connect.storage.StringConverter value.converter: org.apache.kafka.connect.converters.ByteArrayConverter cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000 cps.streamingPull.parallelStreams: 4 cps.streamingPull.maxAckExtensionMs: 10000 cps.streamingPull.maxMsPerAckExtension: 10000 kafka.record.headers: true kafka.partition.scheme: kafka_partitioner header.converter: cps.streamingPull.enabled: true cps.streamingPull.flowControlMessages: 10000
  14. 14. 14 Monitoring SLOs
  15. 15. 15 SLO: Latency Kafka Connect Destination T1 T2
  16. 16. 16 SLO: Delivery Kafka Connect Destination %
  17. 17. 17 Additional Alarms ● Failed Tasks ● Config Load Errors
  18. 18. 18 KIPs! ● IdentityReplicationPolicy (382) ✅ ● Record Rate Limiting (731) ● Delete Committed Records (762) ● Latency Metrics (767)
  19. 19. Thank You 19 Ryanne Dolan @dolanRyanne Jordan Bull @jordangbull

Twitter has one of the largest Kafka fleets in the world, handling hundreds of millions of events per second. In order to operate Kafka Connect at this scale, we've had to get creative. In this talk we'll present some of the problems we've run into with Kafka Connect, and how we've engineered around them.

Views

Total views

48

On Slideshare

0

From embeds

0

Number of embeds

4

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×