Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

1
Building Reactive Distributed Systems
For Streaming Big Data, Analytics & ML
Helena Edelson @helenaedelson Reactive Summit 2016

@helenaedelson #reactivesummit
Distributed Systems & Big Data Platform
Engineer, Event-Driven systems, Streaming
Analytics, Machine Learning, Scala
Committer: FiloDB, Spark Cassandra Connector,
Kafka Connect Cassandra
Contributor: Akka (Akka Cluster), prev: Spring
Integration
Speaker: Reactive Summit, Kafka Summit, Spark
Summit, Strata, QCon, Scala Days, Philly ETE
2
I play with big data
twitter.com/helenaedelson
slideshare.net/helenaedelson
linkedin.com/in/helenaedelson
github.com/helena
and think about FT

If you’re here you’ve taken the red pill

It’s Not Easy Being…
Globally Distributed
Eventually Consistent
Highly Available
While handling TBs of data
Self Healing
Fast
4

Massive event spikes & bursty traffic
Fast producers / slow consumers
Network partitioning & out of sync systems
DC down
Failure before commit
DDOS'ing yourself - no backpressure
Lack of graceful lifecycle handling, i.e. data loss
when auto-scaling down
5

Beware the C word
Consistency?
6
Convergence?

@helenaedelson #reactivesummit 7
Complexity, or…

http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/42
The Real Topic
^Other

“Everything fails, all the time”
Start with this premise
Build failure, chaos routing
and intelligence into your
platform infrastructure
Be fault proactive vs just
reactive
9
- Werner Vogels

The matrix is everywhere…
It is a world where humans
vs machines need to be
directly involved in health of
systems, respond to alerts…

Log analysis tools like
Splunk perpetuate this
illusion…

Self Healing, Intelligent Platforms

Self Healing, Intelligent Platforms
Imagine a world
where engineers designed software
while the machines did triage, failure analysis and
took action to correct failures
and learned from it…

Alerting and Failure Response
The New Boilerplate

Building Systems That Learn
You have systems receiving, processing & collecting data. They can provide insight into
trends, patterns, and when they start failing, your monitoring triggers alerts to humans.
Alternative
1. Automatically route data only to nodes that can process it.
2. Automatically route failure contexts to event streams with ML algorithms to learn:
What is normal?
What is an anomaly?
When these anomalies occur, what are they indicative of?
3. Automate proactive actions when those anomalies bubble up to a fault-aware layer.
15

Deﬁne Failures vs Error Fault
Tolerance Pathways & Routing
Failures - more network related, connectivity, …
Errors - application errors, user input from REST
requests, config load-time errors…
Escape Loops and Feedback system
Built-in failure detection

I want to keep all my apps
healthy, handle errors with a
similar strategies, and failures with
a similar strategy…
…and share that data across
datacenters with my multi-dc
aware kernel to build smarter,
automated systems.

In A Nutshell
Event
Stream
Petabytes Multi-DC
Aware Mgmt
Kernel
Clustered
Apps
Clustered
Apps
Clustered
AppsClustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps

In A Nutshell
Multi-DC
Aware Mgmt
Kernel
Clustered
Apps
Clustered
Apps
Clustered
AppsClustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps
Danger
Sauce
Event
Stream
Petabytes

Multi-DC
Aware Mgmt
Kernel
20
X-DC-Aware Kernel
Clustered
Apps
Clustered
Apps
Clustered
AppsClustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps
Clustered
Apps
Event
Stream
Petabytes
Danger
Sauce

I want to
route failure context
(when a node is alive
enough to talk)
to my.failures.fu Kafka topic for
my X-DC kernel to process through
my ML streams
to learn about usage,
traffic, memory patterns and
anomalies, to anticipate failures,
intercept and correct
Without Human Intervention

Akka Cluster it. The cluster knows
the node is down. Many options/
mechanisms to send/publish failure
metadata to learning streams.
If it’s a network failure and my node can’t
report to the kernel?
Bring it

I think I’ll start with a
base cluster-aware Akka Extension for
my platform.
Any non-JVM apps can share data
with these via Kafka, ZeroMQ,
Cassandra…
Sew easy.
Might sneak it in without a
PR.

Then I can bake my fault tolerance
strategy & route awareness in every
app & node.
And make it know when to use
Kafka and when to use Akka for
communication.

And all logic and flows for graceful handling of
node lifecycle are baked in to the framework too.
Dude, that’s tight.
Graceful Lifecycle

Akka Cluster
Node-aware cluster membership
Gossip protocol
Status of all member nodes in the ring
Tunable, automatic failure detection
No single point of failure or bottleneck
A level of composability above Actors & Actor
Hierarchy
26

Akka Cluster has tunable Gossip.I
can partition segments of my node ring
by role, and route data by role or
broadcast.
Tunable Gossip

Akka Cluster has
AdaptiveLoadBalancing logic.
I can route events away from nodes in trouble and
directed to healthier nodes. Many options, highly
configurable, good granularity.
AdaptiveLoadBalancing*
Router
Cluster Metrics API

Dynamically Route by Node Health
Cluster Metrics API & AdaptiveLBR
Leader
Let’s pummel my cluster
with critical data !

Oh look,
slow consumers or something in GC
thrash…
Leader
Uses EWMA algo to weight decay of health data

Leader
The
Cluster leader ‘knew’ the red
node was nearing capacity & routed my
traffic to healthy nodes.

Reactive - Proactive
32

Proactive - Parallelized
Akka Cluster on nodes
Cluster LB routers
Node router Actors
Kafka replication & routing
across datacenters
33

I drank the cool aid,
put on the costume,
now what…

Embrace Uncertainty

So many opportunities for
failure in this statement
Eventually Consistent Across DCs
37

there is no now…
Embrace Asynchrony & Location Transparency
…or where

There Is No Now
39
US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Akka
micro
services
Akka
micro
services
Akka
micro
services
FiloDB Cassandra
Akka
micro
services
Akka
micro
services
Akka
micro
services
Spark
Compute Clusters
Analytics/Timeseries/ML Storage Storage Clusters
S3/
Cold
Raw Event Stream
Event Stream
DC-1
DC-2
Akka Cluster-ed
apps, partitioned
by Role=service Raw Event Replay
Flink
Backpressure

US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Akka
micro
services
Akka
micro
services
Akka
micro
services
FiloDB Cassandra
Akka
micro
services
Akka
micro
services
Akka
micro
services
Spark
Compute Clusters
Analytics/Timeseries/ML Storage Storage Clusters
S3/
Cold
Raw Event Stream
Raw Event Replay
Event Stream
But I Can Replay Future[Then]
DC-1
DC-2
Akka Cluster-ed
apps, partitioned
by Role=service
Backpressure

Replaying Data from Time (T)
General time - We updated some compute
algorithms which run on raw and aggregated
data. I need to replay some datasets against the
updated algos
Specific time - A node crashed or there was a
network partitioning event and we lost data
from T1 to T2. I need to replay from T1 to T2
without having to de-dupe data
41

Cassandra Bakes Clustered
Ordering Into Data Model
CREATE TABLE timeseries_t (
id text,
year int,
month int,
day int,
hour int,
PRIMARY KEY ((wsid), year, month, day, hour)
) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC,
hour DESC);
42
Now the read path queries automatically return sorted by most
recent. No sorts to do or ordering in your code.
Can also easily bucket in cold storage for faster reads.

US-East-1
MirrorMaker
EU-west-1
ZK
ZK
Compute Clusters
Event Stream
Build Apps Not Clusters?
App clusters
vs
Infra clusters
Actors &
Kafka Streams
Actors &
Kafka Streams
Actors &
Kafka Streams
Event Stream
Event Stream

Kafka Streams Key Features
Secure stream processing using Kafka security
Elastic and highly scalable
Fault-tolerant
Stateful and stateless computations
44

Kafka Streams Key Features
Interactive queries
Time Model & Windowing
Supports late-arriving and out-of-order data
Millisecond processing latency
At-least-once processing guarantees
exactly-once is in progress :)
45

val builder = new KStreamBuilder() 
// Consume from topic x in the stream, from any or one DC
val kstream: KStream[K,V] = builder.stream(des, des, “fu.input.topic")
// Do some analytics computations
// Publish to subscribers in all DCs in the stream
kstream.to(“serviceA.questionB.aggregC.topic”, …)
// Start the stream
val streams = new KafkaStreams(builder, props)
streams.start()
// Do some more work then close the stream
streams.close()
Kafka Streams: KStream
46

Kafka Streams: KTable
val locations: KTable[UserId, Location] = builder.table(“user-locations-topic”)
val userPrefs: KTable[UserId, Prefs] = builder.table(“user-preferences-topic”)
// Join detailed info from 2 streams as events stream in
val userProfiles: KTable[UserId, UserProfile] = 
locations.join(userPrefs, (loc, prefs) -> new UserProfile(loc, prefs))
// Compute statistics
val usersPerRegion: KTable[UserId, Long] = userProfiles
.filter((userId, profile) -> profile.age < 30)
.groupBy((userId, profile) -> profile.location)
.count()

val streams = new KafkaStreams(builder, props)
streams.start()
// Called when a stream thread abruptly terminates
// due to an uncaught exception.
streams.setUncaughtExceptionHandler(myErrorHandler)
Kafka Streams DSL Basics
48

Integrating Intelligence
For Fast Data and Feedback Loops

Translation Layer
50
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Compute Clusters
Event Stream
DC-1
Raw Data
Cassandra
Spark SQL
Spark MLLib
Spark
Streaming
Kafka

Translation Layer
51
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Compute Clusters
Event Stream
DC-1
Raw Data
Cassandra
To Columnar Format
Reads chunks,
translates to spark
rows
Spark MLLib
Spark SQL
Spark
Streaming
Kafka
FiloDB

Translation Layer
52
US-East-1
MirrorMaker
EU-west-1
PubSub
Apps
PubSub
Apps
PubSub
Apps
Compute Clusters
Event Stream
DC-1
Raw Data
Cassandra
To Columnar Format
Reads chunks,
translates to spark
rows
Spark MLLib
Spark SQL
Spark
Streaming
Kafka
FiloDB
Faster
Ad Hoc
Querying
Faster
ML
Feedback
Instead of slowing things
down, it makes it faster.

Your Akka Cluster got in my
Reactive Distributed Analytics
Database!
Walking with a jar of PB as one does…

FiloDB: Reactive
Scala & SBT
Spark (also written in scala)
Akka actors & Akka Cluster for Coordinators
Futures for IO
Typesafe Config
54

import filodb.spark._
KafkaUtils.createDirectStream[..](..)
.map(_._2)
.map(FuEvent(_))
.foreachRDD { rdd =>
sql.insertIntoFilo(rdd.toDF, “fu_events”, database = Some(“org1Keyspace”))}
55
FiloDB API:
Append only, to existing dataset
Write: Kafka, Spark Streaming,
Spark SQL & FiloDB

Read: FiloDB & Spark SQL
56
val df1 = sqlContext.read.format(“filodb.spark”)
.option(“dataset”, “aggrTable”)
.load()
val df2 = sqlContext.filoDataset(“aggrTable”, database = Some(“ks2”))
More typesafe than the
dataframe read (df1)

Determine
Train the
model
Train on historic data
Score
Based on Objectives
Proactive
Responses
React in real time
Machine Learning

What do I want to predict & what can I learn from the data?
Which attributes in datasets are valuable predictors?
Which algorithms will best uncover valuable patterns?
58
Optimization &
Predictive Models

I need fast ML feedback loops,
what tools should I pick?
Depends…

Read from stored training data
into MLLib
60
import org.apache.spark.mllib.stat.Statistics
val df = sqlContext.filoDataset(“table”, database = Some(“ks”))
val seriesX = df.select(“NumX”).map(_.getInt(0).toDouble)
val seriesY = df.select(“NumY”).map(_.getInt(0).toDouble)
val correlation: Double = Statistics.corr(seriesX, seriesY, “pearson”)
Calculate correlation between multiple series of data

Read from the stream to MLLib
& store for later work in FiloDB
61
val stream = KafkaUtils.createDirectStream[..](..)
.map(transformFunc)
.map(LabeledPoint.parse)
val trainingData = sqlContext.filoDataset(“trainings_fu")
val model = new StreamingLinearRegressionWithSGD()
.setInitialWeights(Vectors.dense(weights))
.setStepSize(0.2).setNumIterations(25)
.trainOn(trainingData)
model
.predictOnValues(stream.map(lp => (lp.label, lp.features)))
.insertIntoFilo(“predictions_fu")

Parting Thoughts

Don’t take when someone says X
doesn’t work at face value.
It may not have been the right choice
for their use case
They may not have configured/
deployed/wrote code for it properly for
their load
All technologies are optimized for
a set of things vs every thing
Invest in R&D
63
Cultivate Skepticism

Modular
does one thing
Decoupled
knows one thing
Reusable & Extendable
when not final or
private for a reason
64
Keep Everything Separate

Did you just duplicate codez?

Versus duplicating, with disparate
strategies, all over your teams & orgs
Build A Reactive / Proactive Failure & Chaos
Intelligence System for platform infrastructure

Byzantine Fault Tolerance?
67
Looks
like I'll miss
standup

twitter.com/helenaedelson
slideshare.net/helenaedelson
github.com/helena
Thanks!

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

Similar to Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning (20)

Recently uploaded

Recently uploaded (20)

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning