Maybe you’ve heard of Mesos—that thing that you can run Hadoop on. I think it powers Twitter? Isn’t it an Apache project, or something? In this talk, we’ll learn all about Mesos—what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.

1. Mesos:
The
Datacenter
Opera1ng
System
David
Greenberg
Two
Sigma

2. Who
am
I?
• Architected
project
to
build
a
massive
Mesos
cluster
• Building
custom
framework
and
leveraging
open
source

3. The
Plan
What
is
Mesos?
How
can
I
use
Mesos?
How
can
I
build
on
Mesos?

4. What
is
Mesos?

5. A
long
1me
ago…
Are
you
done
with
the
machine?
I
need
to
load
my
cards.
Lol
no;
maybe
tomorrow.

6. 1957
Oh
man!
Let’s
all
share
the
computer,
AT
THE
SAME
TIME!
John
McCarthy
Popularized
Timesharing

7. A
long
1me
ago…
Are
you
done
with
the
Hadoop
cluster?
I
need
to
run
my
analy1cs
job.
Lol
no;
maybe
tomorrow.

8. 2010
Oh
man!
Let’s
all
share
the
cluster,
AT
THE
SAME
TIME!
Ben
Hindman
Popularized
Mesos

9. Good
ideas
today
mirror
good
ideas
of
yesteryear

10. Mesos:
an
Opera1ng
System

11. Isola1on

12. Resource
Sharing

13. Common
Infrastructure
• read(),
write(),
open()
• bind(),
connect()
• apt-­‐get,
yum
• launchTask(),
killTask(),
statusUpdate()
• Docker

14. Distributed
System*
Anatomy
Workers
Coordinator
*
Excluding
peer-­‐to-­‐peer
systems

15. Sta1c
Par11oning
Coordinator
(Hadoop)
Coordinator
(Storm)

16. Mesos:
a
Level
of
Indirec1on
Mesos
(slaves)
Coordinator
Mesos
(master)
Coordinator

17. Mesos:
a
Level
of
Indirec1on
Mesos
(slaves)
Coordinator
Mesos
(master)
Coordinator

18. Mesos:
a
Level
of
Indirec1on
Mesos
(slaves)
Coordinator
Mesos
(master)
Coordinator

19. Mesos:
a
Level
of
Indirec1on
Mesos
(slaves)
Coordinator
Mesos
(master)
Coordinator

20. Mesos:
a
Level
of
Indirec1on
Mesos
(slaves)
Coordinator
Mesos
(master)
Coordinator

21. Coordina1ng
Execu1on
≈
Scheduling

22. s/Coordinator/Scheduler/
Mesos
(slaves)
Coordinator
Mesos
(master)

23. s/Coordinator/Scheduler/
Mesos
(slaves)
Scheduler
Mesos
(master)

24. Apache
Hadoop
Mesos
(slaves)
JobTracker
(Scheduler)
Mesos
(master)

25. Distributed
System
≈
(Mesos)
framework

26. a
Mesos
framework
is
a
distributed
system
that
has
a
coordinator

27. a
Mesos
framework
is
a
distributed
system
that
has
a
coordinator

28. a
Mesos
framework
is
a
distributed
system
that
has
a
scheduler
a

29. a
Mesos
framework
is
an
app
for
your
cluster

30. How
can
I
use
Mesos?

31. Tons
of
Flexibility!

32. Jenkins
• Con1nuous
build
server
• Just
install
a
plugin!

33. Hadoop
• Mul1-­‐cluster
isola1on
• Fast
startup
• Just
run
the
repacked
Cloudera
CDH
4.2.1
MR1
distribu1on
for
Mesos

34. Marathon
• PaaS
on
Mesos
• init.d
for
the
cluster
• Docker
support
• Scales
at
the
click
of
a
budon
• Manages
edge
routers
-­‐
HAProxy

35. Chronos
• Distributed
cron
• Supports
job
dependencies
• REST
API

36. Aurora
• Advanced
PaaS
on
Mesos
• Powers
Twider
• Supports
phased
rollouts
• Supports
complex
deployments

37. Spark
• In
memory
Map
Reduce,
built
for
“Medium
Data”
• Supports
SQL
as
well
as
Java,
Python,
and
Scala
• Designed
for
interac1ve
analysis
via
REPL

38. How
do
I
use
these?
• Free
online
interac1ve
tutorials!
– hdp://mesosphere.io/learn
• Covers
all
of
the
previously
men1oned
and
many
more

39. How
can
I
build
on
Mesos?

40. Cluster
Manager
Status
Quo
Applica?on/Human
Specifica1on
Cluster
Manager
The
specifica1on
includes
as
much
informa1on
as
possible
to
assist
the
cluster
manager
in
scheduling
and
execu1on

41. Cluster
Manager
Status
Quo
Applica?on/Human
Cluster
Manager
Wait
for
task
to
be
executed

42. Cluster
Manager
Status
Quo
Applica?on/Human
Result
Cluster
Manager

43. Problems
with
Specifica1ons
① Hard
to
specify
certain
desires
or
constraints
② Hard
to
update
specifica1ons
dynamically
as
tasks
execute
and
finish/fail

44. An
Alterna1ve
Model
Scheduler
Mesos
request
3
CPUs
2
GB
RAM
• A
request
is
purposely
simplified
subset
of
a
specifica1on
• It
is
just
the
required
resources
at
that
point
in
)me

45. What
should
you
do
if
you
can’t
sa1sfy
a
request?

46. What
should
you
do
if
you
can’t
sa1sfy
a
request?
①
Wait
un?l
you
can
…

47. What
should
you
do
if
you
can’t
sa1sfy
a
request?
①
Wait
un?l
you
can
…
②
Offer
best
you
can
immediately

48. What
should
you
do
if
you
can’t
sa1sfy
a
request?
①
Wait
un?l
you
can
…
②
Offer
best
you
can
immediately

49. Mesos
Model
Scheduler
Mesos
offer
hostname
4
CPUs
4
GB
RAM
• Resources
are
allocated
via
resource
offers
• A
resource
offer
represents
a
snapshot
of
available
resources
that
a
scheduler
can
use
to
run
tasks

50. An
Analogue:
non-­‐blocking
sockets
Applica?on
Kernel
write(s, buffer, size);!

51. An
Analogue:
non-­‐blocking
sockets
Applica?on
Kernel
42 of 100 bytes written!!

52. Mesos
Model
offer
hostname
4
CPUs
4
GB
RAM
offer
hostname
4
CPUs
4
GB
RAM
offer
hostname
4
CPUs
4
GB
RAM
Scheduler
Mesos
offer
hostname
4
CPUs
4
GB
RAM
Scheduler
uses
the
offers
to
decide
what
tasks
to
run

53. Mesos
Model
Scheduler
Mesos
Scheduler
uses
the
offers
to
decide
what
tasks
to
run
“Two-­‐level
scheduling”
task
3
CPUs
2
GB
RAM

54. Two-­‐level
Scheduling
• Mesos:
controls
resource
alloca+ons
to
schedulers
• Schedulers:
make
decisions
about
what
tasks
to
run
given
allocated
resources

55. Two-­‐level
Scheduling
Elsewhere
• Mesos
influenced
by
opera1ng
system
supported
user-­‐space
scheduling
– E.g.
green
threads,
gorou1nes
• Mesos
is
designed
less
like
a
“cluster
manager”
and
more
like
an
opera1ng
system
(or
kernel)

56. Language
Bindings

57. Should
I
build
it
on
Mesos?
• Theme
of
MesosCon:
it’s
easy
to
build
frameworks
• Open
source
and
proprietary
frameworks
are
being
created
all
the
1me
– Two
Sigma
– Neplix
– Twider
– Hubspot

58. But
should
I
really
build
it
on
Mesos?
• Most
users
just
use
Marathon,
Hadoop,
Spark,
and
Chronos
• Why
did
we
build
our
own?
– Exo1c
workload

59. The
Plan,
redux
What
is
Mesos?
How
can
I
use
Mesos?
How
can
I
build
on
Mesos?

60. Ques1ons?
Thank
you