Far from a controversial choice, Kafka is now a technology developers and architects are adopting with enthusiasm. And it’s often not just a good choice, but a technology enabling meaningful improvements in complex, evolvable systems that need to respond to the world in real time. But surely it's possible to do wrong! In this talk, we'll look at common mistakes in event-driven systems built on top of Kafka: - Deploying Kafka when an event-driven architecture is not the best choice. - Ignoring schema management. Events are the APIs of event-driven systems! - Writing bespoke consumers when stream processing is a better fit. - Using stream processing when you really need a database. - Trivializing the task of elastic scaling in all parts of the system. It's highly likely for medium- and large-scale systems that an event-first perspective is the most helpful one to take, but it's early days, and it's still possible to get this wrong. Come to this talk for a survey of mistakes not to make.

1. Event-Driven Architecture Done Right
And several ways to do it wrong
Tim Berglund
Director of Developer Advocacy, Con
f
luent
@tlberglund

2. What is Event-Driven Architecture?

3. An Event-Driven Reference System
Order
Web UI
Orders
Service
Users
Web UI
Users
Service
Payment
Gateway
Payments
Service
Warehouse
Control
Shipping
Service

4. How can this go wrong?

5. How can this go wrong?
1. You just shouldn’t
2. You should have used more databases
3. You think schema doesn’t matter
4. You write bespoke consumers
5. You trivialize scaling

6. 1When you shouldn’t

7. When are events a bad idea?
• Small scale
• High uncertainty

10. 2
When you should have used more
databases

11. Legacy is life
• Old paradigms are supplemented, not supplanted
• Change data capture lets data-in-motion and data-at-rest coexist
• It’s okay if you have databases doing important work
Application

12. Single column text slide
Subhead Mark Pro Bold 32 pt
Body copy would go here.
This would just be to show how longer copy would look. Mark Pro Narrow Light 24 pt.
• Bullet Level 1
• Bullet Level 2

13. Con
f
luent Developer
developer.con
f
luent.io

14. Users
Web UI
Users
Service
ment
way
ments
vice
Warehouse
Control
Shipping
Service
Don’t overuse KTables
• Lookup by ID
• Lookup by username
• Lookup by email

15. 3
When you think schema doesn’t
matter

16. Schema Matters
Order
Web UI
Orders
Service
Users
Web UI
Users
Service
Payment
Gateway
Payments
Service
Warehouse
Control
Shipping
Service

17. Order
Web UI
Orders
Service
Users
Web UI
Users
Service
Payment
Gateway
Payments
Service
Warehouse
Control
Shipping
Service
Fraud
Detection Customer
Loyalty

20. • Data ownership by domain
• Data as Product
• Data available everywhere
• Federated data governance
Data Mesh Principles
https://bit.ly/2UBChMU

21. Schema Matters
Order
Web UI
Orders
Service
Users
Web UI
Users
Service
Payment
Gateway
Payments
Service
Warehouse
Control
Shipping
Service

22. 4
When you go to the tailor for your
consumers

23. Consumers - Scaling Out

24. Consumers - Scaling Out

25. try (final KafkaConsumer<String, Payment> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Collections.singletonList(TOPIC));
while (true) {
ConsumerRecords<String, Payment> records = consumer.poll(100);
for (ConsumerRecord<String, Payment> record : records) {
String key = record.key();
Payment value = record.value();
System.out.printf("key = %s, value = %s%n", key, value);
}
}
}

27. Consider the problems to solve here…
Order
Web UI
Orders
Service
Users
Web UI
Users
Service
Payment
Gateway
Payments
Service
Warehouse
Control
Shipping
Service

28. State management

29. Ka
f
ka Connect
Ka
f
ka Streams
ksqlDB

30. 5
When you think scaling will be
trivial

31. Scale is dif
f
icult
1. How much scaling will you really do?
2. How much elasticity will you need?
3. Will you scale storage?
4. Will you scale compute?

32. Scaling Compute

33. CREATE TABLE movie_ratings AS
SELECT title,
SUM(rating)/COUNT(rating) AS avg_rating,
COUNT(rating) AS num_ratings
FROM ratings
LEFT OUTER JOIN movies
ON ratings.movie_id = movies.movie_id
GROUP BY title;

35. • Make a bigger cluster
• Add storage tiers
Scaling storage

36. • Bigger cluster
• Tiered storage
Scaling Storage

37. SOURCE: https://www.righto.com/2014/10/how-z80s-registers-are-implemented-down.html

40. How can this go right?
1. Use events when you should
2. Embrace databases appropriately
3. Manage your schemas
4. Use appropriate compute frameworks
5. Managed services whenever you can

41. Thank you!
@tlberglund
Con
f
luent Developer
developer.con
f
luent.io