A presentation from #CCCEU13. It is based on Forthscale`s method of running scalable infrastructure with the help of monitoring systems. Now updated as for OpsTalk TLV

1. Managing scalable infrastructure
with monitoring

2. What is ForthScale?
Scalable
infrastructure experts
company
kick ass team of
skilled ninjas
●
●
●
●
●
Consulting
Design
Integration
Implementation
Management

3. Groundlaying
●
●
●
●
●
Semantics (intent or a desired state)
Dynamics (actual behavior)
Dev think mostly about semantics
Ops think mostly about dynamics
DevOps = complete picture
● Thank you Mark Burgess

4. Groundlaying
● Complexity is THE source of randomness.
● Complexity is the birthplace of chaos
● We all know the KISS principle but fail it.
The maintenance theorem says: you can't
really control anything over time. Best you can
do is to keep it roughly in balance.

5. Groundlaying
"Equilibrium" replaces determinism as the most
important idea in science. It is the definition of
dynamical stability.

6. What is scaling
Scaling is switching multiple Equilibrium points
in accordance with current resources demand.

7. Groundlaying
● People tend to use modern IaaS same way
they used to operate hardware systems.
● If you do not take true advantage of IaaS
why pay it`s premia?
● If you are paying for it, maybe it could be a
good idea to actually use it?

8. Basics - Operator goals
Use IaaS to improve infrastructure lifecycle
management.
●
●
●
●
Provide availability
Keep user experience
Manage costs
Account for usage

9. Cases
Mainly running an operation with fluctuating
resource consumption.
● Web app (externally or internally hosted)
● Mobile app (externally hosted)
● Intranet app (internally hosted)
● Data analysis (externally or internally
hosted)

10. Basics - a perfectly good stack
●
●
●
●
Clear policy on resource consumption
Stack parts horizontal scalability
Non sticky nodes (for easy replacements)
Simply orchestrated

11. The Forthscale way
What is the best way to allocate
resources?
Monitoring stack and learning what
resources are needed and when

12. Controller building blocks
● CloudStack allows to manage allocation of
computation resources (IaaS).
● Munin monitoring collects analytical data
about infrastructure performance.
● Ansible for infrastructure orchestration.

13. CloudStack
●
●
●
●
Is a cloud orchestration platform
Manages computing resources allocation
Has an extensive API
Yes we can use other platforms

14. Munin
Munin is an open source client /server
monitoring application that presents output in
graphs through a web interface
yes, any monitoring will do.

15. Munin
●
●
●
●
●
●
●
highly configurable
stores to rrd
makes graphs
shows trends
can alert
very very very simple to deploy
text based configuration

16. Cloudstack + Munin = 日本のスタイル
●
●
●
●
●
Set of plugins done with CloudMonkey
Can store all metrics provided by api
Correlate IaaS, OS and AppStack issues
Can use Nagios or similar for alerting
But why just alert then you can handle?

17. Handling
●
●
●
●
●
Alert issue
Re-allocate resources (increase / decrease)
Switch resources between components
Replace components / nodes
Report issue

18. Handling - Orchestration
Issues can arise on any level
●
●
●
●
IaaS resources
OS
Middleware
Continuous Integration
Better use orchestration tool that supports any
dynamic, any part of the stack.

19. Ansible
Why ? “why not just use Ansible instead?”
case.
● Simple
● Agentless
● Powerfull
● Python

20. CloudStack+Ansible=No limits
Generate playbooks to:
●
●
●
●
Add or remove nodes
Change nodes sizes
Upgrade components or application
Add / remove nodes to / from arrays

21. CloudStack+Munin+Ansible
Scalable infrastructure what:
● Operates on true metrix.
● Efficient and accountable.
● Available, Scalable and Redundant.

22. Use scenarios - Vertica
Vertica is a SQL compliant grid database.
1. Adding or removing nodes based on amount
of resources needed.
2. Changing nodes sizes based on amount of
resources needed.
3. Proactive policy for resource allocation.

23. Use scenarios - version release
New version can also lead to changes in
computing power requests.
1.
2.
3.
4.
Building a stage environment.
Executing a load test.
Comparing monitoring metrics.
Adjusting resources together with
deployment

24. Forthscale contacts
Naor Weissmann
http://www.linkedin.com/in/naorweissmann
IL: +972 (0)505 980005
US: +1 303 5867379
UK: +44 (0)203 1292157
Email: naorw@forthscale.com
Web site: http://www.forthscale.com

25. Feel free to contact me any time
Thank you
for your time.
I hope that at least I’ve
raised some questions.