As infrastructure scales, simple tasks become increasingly difficult. For large infrastructures to be manageable, we use automation. But automation, like any power tool, comes with its own set of risks and challenges. Automation should be handled like production code, and great care should be exercised with power tools. This talk will cover how SaltStack is used at LinkedIn and offer tips and tricks for automating management with SaltStack at massive scale including a look at LinkedIn-inspired Salt features such as blacklist and prereq states. It will also cover Salt master and minion instrumentation and a compilation of how not to use Salt.
SaltConf keynote review - Thomas Jackson, LinkedInSafety with Power toolsAs infrastructure scales, simple tasks become increasingly difficult. For large infrastructures to be manageable, we use automation. But automation, like any power tool, comes with its own set of risks and challenges. Automation should be handled like production code, and great care should be exercised with power tools. This talk will cover how SaltStack is used at LinkedIn and offer tips and tricks for automating management with SaltStack at massive scale including a look at LinkedIn-inspired Salt features such as blacklist and pre-req states. It will also cover Salt master and minion instrumentation and a compilation of how not to use Salt.
Professional, fewer cats
How long we’ve been using itTom is embarrased that we started so early 0.8.9runners just addedOutputters just addedCross calling salt modules using __salt__0.9.9Highstate test=TrueExternal pillarmInion swarm
We all manage some service, so lets talk about it
To get some context, I’m going to talk a little about the main service I support
When I started ATS was new– so we had a lot of manual things ;)First question is going to be…
Consistency problems (missing a file, or a package)Missing a log entry
Really are useful and AMAZINGLY simple
Taken fromwikipedia, TODO: get a better one?
Remote- faster than old one 30m
14m
Tools to use all the toolsOne such tool is Salt, as with any other tools there are some things to keep in mind while using
Similar to power tools
Other automaton(Don’t step on toes)Clear lines of ownership
JMX example
Nice doc string!Whatup with decorator?
startJVM WAT?Our feature: limit memory consumption on module load in *nix (modules_max_memory)
Consume event bus to get information on jobs running
Well, not necessarily..
Lots of features (and more coming I’m sure)Lets take some time to talk about 2
Migrating because we had re-implemented states before it existed, had wrappers to do OOR/IR operationsAnother example of where something you think you need, others want too!
No only new features, but we find + fix bugs too!Find the root cause, usually simpler than you’d think