Slides from my talk "Application Logging With The ELK Stack"

1. Application Logging
With The ELK Stack
@bwaine - #DPC15
Monday, 29 June 15

2. 2
Ben Andersen-Waine
Software Engineer
Contractor
Deployed ELK To Prod
Numerous Times
Monday, 29 June 15

3. Logging?
Monday, 29 June 15

4. System Logs
Monday, 29 June 15

5. 5
Monday, 29 June 15

6. Application Log
Monday, 29 June 15

7. Debug Information - Errors (connections,
uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls,
Event Triggers
Business Events - Purchases, Logins,
Registrations, Unsubscribes
7
Application Log
Monday, 29 June 15

8. ssh webserver@mydomain.net
tail -f /var/log/nginx/my-site.access.log
tail -f /var/log/my.application.log
ssh data@mydomain.net
tail -f /var/log/mysql/mysql.log
ssh q@mydomain.net
tail -f /var/log/rabbitmq/nodename.log
8
Keeping Track Of All This....
Monday, 29 June 15

9. 9
The Elk Stack
Monday, 29 June 15

10. Monday, 29 June 15

11. 1) Monolog
2) Everything else....
11
PHP Logging Tools
Monday, 29 June 15

12. 1) Monolog: Loggers And Handlers
2) Monolog:Tags & Formatters
3) Logging business events
12
Basic Logging Examples
Monday, 29 June 15

13. use MonologLogger;
use MonologHandlerFingersCrossedHandler;
use MonologHandlerStreamHandler;
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);
$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
13
Monday, 29 June 15

14. // Set A Log Level
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger
$appLog = new Logger('AppLog');
14
Monday, 29 June 15

15. $strHandler
= new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler
= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');
$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
15
Monday, 29 June 15

16. EG 2:Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
16
Monday, 29 June 15

17. // Create A Logger
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
17
Monday, 29 June 15

18. $id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
18
Monday, 29 June 15

19. 2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424
19
Log Levels
Monday, 29 June 15

20. 2013 - PSR03 - PHP Logging Interface Standard
http://www.php-fig.org/psr/psr-3/
20
PSR3
Monday, 29 June 15

21. EG 3: Event Logging
use MonologLogger;
use SymfonyComponentEventDispatcherEventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener(
"business.registration.post",
function () use ($busLog) {
$busLog−>info("Customer registered");
}
);
$dispatcher−>dispatch("business.registration.post");
Monday, 29 June 15

22. Logstash Architecture
1. Logstash Shipper ships logs to
logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic
Search
4. Kibana exposes a web interface
to Elastic Search data
Monday, 29 June 15

23. Logstash Architecture
Monday, 29 June 15

24. Why not rate the talk now BEFORE
the demo?
24
https://joind.in/talk/view/14235
Monday, 29 June 15

25. ELK Demo
25
1) Discover Data (search / diagnose)
2)Visualize Data
3) Produce A Dashboard
4) Demonstrate ‘the new hotness’ of Kibana 4
Monday, 29 June 15

26. 26
https://github.com/LoveSoftware/
getting-started-with-the-elk-stack
Monday, 29 June 15

27. Monday, 29 June 15

28. Monday, 29 June 15

29. Monday, 29 June 15

30. Monday, 29 June 15

31. Logstash Config
31
Monday, 29 June 15

32. Logstash Collecting
{
"network": {
"servers": [ "logs.logstashdemo.com:5000" ],
"timeout": 15,
"ssl ca":
"/etc/pki/tls/certs/logstash−forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/nginx/helloapp.access.log"
],
"fields": { "type": "nginx−access" }
}
]
}
32
Monday, 29 June 15

33. Logstash Processing
input {
lumberjack {
port => 5000
ssl_certificate =>
"/etc/pki/tls/certs/logstash−forwarder.crt"
ssl_key =>
"/etc/pki/tls/private/logstash−forwarder.key"
}
}
Input
33
Monday, 29 June 15

34. Logstash Processing
Filtering
filter {
if [type] == "nginx−access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
34
Monday, 29 June 15

35. Logstash Processing
Output
output {
elasticsearch { host => localhost }
}
35
Monday, 29 June 15

36. Groking
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}
%{WORD:method}
%{URIPATHPARAM:request}
%{NUMBER:bytes}
%{NUMBER:duration}
Monday, 29 June 15

37. 37
Hey Ben....
Have you got time for that
gratuitously flashy geo data demo?
Monday, 29 June 15

38. Monday, 29 June 15

39. Logging Ideas
Release Marker
Error rates of various applications over time
Latency in various percentiles of each application tier
HTTP Responses: 400 series responses
HTTP Responses: 500 series responses
Auto git blame production errors
Auth and Syslogs
39
Monday, 29 June 15

40. Go Forth And Log....
BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS
40
Monday, 29 June 15

41. Questions?
41
Monday, 29 June 15

42. https://joind.in/talk/view/14235
42
Monday, 29 June 15