1. A ot e
buM
• L u S s mA m is a r
in x yt
e d in t t
ro
H s a da dF te o 2K s
ub n n a r f id
h
• D v p , Po u titH cs n
eO s rd c y ak a d
iv
T o , T e igL b wk
o l h B e o si
s
2. G o in S l k
rw g p nu
Tl R t h a - G r innen tn l
y r us m n am I ra a
e c t io
3. O : ( uin a o t e
H d rg n ua ) g
I o ’t a tole a ol
d n wn t iv in wr d
wh u S l k
ito t p n.
u
I o ’t a tole a ol
d n wn t iv in wr d
wh u S l k
ito t p n.
u
4. B cs r
akt y
o
• Fe int c intl in 0 9
re s n e s ld 2 0
a ae
S g I t c o C nrlo sr r
in l n a e n e t L g ev
e sn a e
• U ga e oE t pis
p rd t ne re
r
5. L vl
ee2
• S lS l k nod d a dint c
p p n o t e ict s n e
it u e a
L e s oewe e b G r in o n c
icne vr h l d y am C n et
m
• L it v ibita due
im e is il n s
d y
6. S p r o l kI t c r
u e C oS i n r t
su o
IF Y HAVE MORE INPUTS THAN LICENSE
OU
YOU’RE GONNA HAVE A BAD TIME
7. P n o E p nio
l f xas n
a r
• D c e t m k a p a n oe o ut
eid d o ae p l t m r rb s
ic io
R a te o u e t io
ed h D c m na n
t
• .c n 2 1
of01
8. E t pis Ac it tr
ne re rhe ue
r c
O te
ul
in
O te
ul
in
• P p e D p y et
upt el mn
o
Irs u tr L yu
n at c e ao t
f r u
• Gta
oc s
h
F tr P n
uue l s
a
9. Ppe
upt
• S ac , I ee a dF r ad r r “tr- e”
erh n xr n owre ae unky
d
e: in l e p n::in ee ...d n
x c d s l k d xr o e
u u
• R ay wsm f F r ad r
el A eo e o owres
l r
W y o ue p n D p y e t a a e?
h n t s S l k e l m n M ng r
u o
11. H w e s Sl k
o W Ue p n
u
W bA cs L g
e ces o s
• I en l p la n u it
n raA p t A d s
t ic io
• W d w S cr Eet
in o s euit vns
y
12. Wy Le p n
h Iik S l k
u
M ks sr H p y
ae U es a p
R aT e a
elim D t
a
N A en te
o lra s
t iv
13. Gta
oc s
h
D n I e ao o d t oe N S
o ’t d x l f a vr F
n t a
S ae K o l g B n lT e yc
h rd n w d e u d im S n
e e
T ga dS ac p r isio s
a n erh em s n
14. F tr P n
uue l s
a
S a C nrl yt L g in
cl e t S s m o g g
e a e
M r S l krma sr ee p r O
oe p n f
u o U e/ vl e P V
D o
A d io aI us
d itn l p t
n
Ta in
rin g
15. T s n Ave
ip a d dic
• W I vn Ft f W d w E e t -
M E e t ilro in o s vns
e r
ht:/.c /ernc
t / og x r
p t F
• S l k ae nwr
p nb s A s es
u
I started as an IT intern my mentor had a free copy of version 2.x running on the log server. I was tasked with finding a solution for SOX & PCI requirements. (Which was mind expanding for an intern, to say the least) Worked with purchasing to get a small license for the enterprise features. My project ended up piping Splunk output into a python program that no one but I understood that printed out a text report that (I felt at least) was superior to the one in place at the time. (Big surprise, didn’t end up using it).
When I came back there was some cursory interest in the app, but no major users and no project champion. Welcome, back Tyler... Splunk Expert (by Default). I was also attached to Garmin Connect, which is our awesome fitness tracking site, after getting more comfortable in my settings, I began to integrate the site logs into Splunk
Obvious, but this was my experience during the first dedicated instance. We had a small license and it was all being used by Garmin Connect. It really wasn’t taking hold like I knew it could.
After I became more comfortable in my position, I felt compelled to make the application more robust and widespread. I went to .conf last year, attended some training sessions and read up on the Administration documentation.
Overview of the Current Architecture Elements, will then go in depth a bit more on each subject.
Puppet makes deployment simple. Servers are built with one include statement. Forwarders are split up based on role and inputs. Customize the inputs a bit if necessary and include the splunk forwarder class in the puppet node definition.
Describe layers and functions. Search is load balanced. Search, Index and Forwarders are horizontally scalable. Network/Taiwan instances aren’t pictured but are separate dedicated instances. Will move the network index into the main infrastructure real soon now.
Feature Tracking Incident Management We don’t have a wide variety if inputs into Splunk at the moment. We currently use it on all of the major IT web applications to obtain service metrics, track new features and diagnosing issues in Production. The developers are also starting to cater their applications to output Splunk friendly logs Windows security events are queried via WMI ad filtered to specific IDs, this helps keep the volume down while delivering value for the Windows guys.
Ease of configuration, having the one stop shop for user-land configs. LDAP integration is super simple. Able to generate detailed reports and drill into the data on the fly is a killer feature and something that you simply won't find with any other application. User community and Documentation. There are no real alternatives to Splunk. Some tools touch on some of the features gained with the app, but there is no offering that matches what splunk can give you. I’ve tried SEC, logwatch, Logstash, and Spiceworks. None were as user friendly and robust as Splunk.
Keeping up with the demand. From a license and user request perspective, I limited amount of time to handle the requests at hand. Familiar position for me at least, but a good problem to have. Mounted Bundles must have the same time across the board. Watch your permissions on saved searches and tags. They are usually private when I share them with another user and they cannot access.
Currently only one centralized syslog server, want to scale it out and put a farm of syslog servers behind a load balancer. Splunk will be the defininitive timeline for syslog events. Read about Deployment Server but passed on it at the time. Would like to pick it back up and see how it could be beneficial. Add additional inputs to the application I’ve been tasked with training my coworkers on how to use the application. Once they pick it up and figure it out, they can do awesome things.