20180510 Kubernetes 開源容器 技術論壇 2018 任博煜(工研院)

1. Kubernetes monitoring by Prometheus and Grafana
Kevin Jen
Div X/ICL
2018/04/27

2. Copyright 2016 ITRI 工業技術研究院
個人簡介
• 任博煜 Kevin Jen
• 工研院資通所
• 資料中心與雲端應用軟體組
• 協助整合以及維運由Kubernetes
搭建的 ITRI DNN Farm

3. Copyright 2016 ITRI 工業技術研究院
Tools
Prometheus
This is the central piece, it contains the time series database and the logic of scraping stats from
exporters (see below) as well as alerts.
Node-exporter
A Node Exporter to export system metrics in a Prometheus-compatible format.
There are many other types of expoter ,like container expoter or snmp expoter.
Nvidia-smi-exporter
nvidia-smi metrics exporter for Prometheus
Grafana
A web-based graphical dashboard builder that supports Prometheus

4. Copyright 2016 ITRI 工業技術研究院
What is Prometheus
• Prometheus is an open-source systems monitoring and alerting toolkit originally built
at SoundCloud.
• Since its inception in 2012, many companies and organizations have adopted Prometheus,
and the project has a very active developer and user community. It is now a standalone open
source project and maintained independently of any company.
• To emphasize this and clarify the project's governance structure, Prometheus joined
the Cloud Native Computing Foundation in 2016 as the second hosted project
after Kubernetes.

5. Copyright 2016 ITRI 工業技術研究院
Features
• A multi-dimensional data model (time series identified by metric
name and key/value pairs)
• A flexible query language to leverage this dimensionality
• Targets are discovered via service discovery or static configuration

6. Copyright 2016 ITRI 工業技術研究院
Grafana
• Grafana is an open source metric analytics & visualization suite.
• Prometheus delegates the heavy lifting of proper graphical displays and dashboards to
Grafana.
• Grafana features pluggable panels and data sources allowing easy extensibility and a
variety of panels, including fully featured graph panels with rich visualization options.
• Easily create alert rules from within the UI and have them be continually evaluated by the
Grafana backend.

7. Copyright 2016 ITRI 工業技術研究院
部署
kubectl create -f node-exporter.yaml
kubectl create -f rbac-setup.yaml.yaml
kubectl create -f prometheus-config-map.yaml
kubectl create -f prometheus-deploy.yaml
kubectl create -f grafana-deploy.yaml
git clone https://github.com/kevin7674/prometheus.git
cd Prometheus
下載
部署
檢查
kubectl get pod

8. Copyright 2016 ITRI 工業技術研究院
使用RBAC創建Cluster Role並設置訪問權限
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: ["", "extensions", "apps"]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
- deployments
- services
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: default

9. Copyright 2016 ITRI 工業技術研究院
node-exporter.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: node-exporter
spec:
template:
metadata:
labels:
app: node-exporter
name: node-exporter
spec:
containers:
- image: prom/node-exporter:latest
name: node-exporter
ports:
- containerPort: 9100
hostPort: 9100
name: scrape
hostNetwork: true
hostPID: true
restartPolicy: Always
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
app: node-exporter
name: node-exporter
name: node-exporter
spec:
clusterIP: None
ports:
- name: scrape
port: 9100
protocol: TCP
selector:
app: node-exporter
type: ClusterIP

10. Copyright 2016 ITRI 工業技術研究院
prometheus-ConfigMap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: default
data:
prometheus.yml: |-
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https

11. Copyright 2016 ITRI 工業技術研究院
prometheus-ConfigMap.yaml
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics

12. Copyright 2016 ITRI 工業技術研究院
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
prometheus-ConfigMap.yaml

13. Copyright 2016 ITRI 工業技術研究院
prometheus-ConfigMap.yaml
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::d+)?;(d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name

14. Copyright 2016 ITRI 工業技術研究院
prometheus-deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: prometheus-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus:v1.8.2
args:
- "-config.file=/etc/prometheus/prometheus.yml"
- "-storage.local.path=/prometheus/"
- "-storage.local.retention=240h"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
nfs:
server: 10.241.1.11
path: "/nfs/prometheus“
nodeSelector:
release: production
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
name: prometheus
name: prometheus
spec:
selector:
app: prometheus-server
type: NodePort
ports:
- name: prometheus
protocol: TCP
port: 9090
nodePort: 30008

15. Copyright 2016 ITRI 工業技術研究院
nvidia_smi_exporter
到每台GPU-node做以下步驟
1.下載並執行nvidia_smi_exporter
git clone https://github.com/kevin7674/nvidia_smi_exporter.git
cd nvidia_smi_exporter
./nvidia_smi_exporter 9101 &
2.修改 vi /etc/rc.local
./nvidia_smi_exporter 9101 &
修改Prometheus-configmap.yaml
- job_name: "nvidia_smi_exporter"
static_configs:
- targets: [‘Node_IP:9101']
- targets: [‘Node_IP:9101']

16. Copyright 2016 ITRI 工業技術研究院
grafana-deploy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: grafana
#namespace: prometheus
labels:
app: grafana
spec:
replicas: 1
template:
metadata:
labels:
app: grafana
spec:
containers:
- image: grafana/grafana:3.1.0
name: grafana
ports:
- containerPort: 3000
hostPort: 3000
env:
- name: GF_AUTH_BASIC_ENABLED
value: "false"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: Admin
- name: GF_SERVER_ROOT_URL
value: /api/v1/proxy/namespaces/prometheus/services/grafana/
volumeMounts:
- name: grafana-persistent-storage
mountPath: /var/lib/grafana
volumes:
- name: grafana-persistent-storage
nfs:
server: 10.241.1.11
path: "/nfs/grafana“
nodeSelector:
release: production
apiVersion: v1
kind: Service
metadata:
name: grafana
#namespace: prometheus
labels:
app: grafana
spec:
ports:
- port: 80
targetPort: 3000
nodePort: 30000
selector:
app: grafana
type: LoadBalancer

17. Copyright 2016 ITRI 工業技術研究院

18. Copyright 2016 ITRI 工業技術研究院
Query

19. Copyright 2016 ITRI 工業技術研究院
Excute graph
Refer to the official website
https://prometheus.io/docs/querying/basics/

20. Copyright 2016 ITRI 工業技術研究院
Slice & Dice
container_memory_usage_bytes{name="consul-server"}

21. Copyright 2016 ITRI 工業技術研究院
Aggregation sum(container_memory_usage_bytes{name=~"^consul"})
avg(container_memory_usage_bytes{name=~"^consul"}) by (name)

22. Copyright 2016 ITRI 工業技術研究院
sum by (pod_name)( rate(container_cpu_usage_seconds_total{image!="", pod_name!=""}[1m] ) )

23. Copyright 2016 ITRI 工業技術研究院

24. Copyright 2016 ITRI 工業技術研究院
Data source

25. Copyright 2016 ITRI 工業技術研究院
Dashboard
Creating a new dashboard and add a graph panel.

26. Copyright 2016 ITRI 工業技術研究院
Edit

27. Copyright 2016 ITRI 工業技術研究院
Creating a Prometheus graph by querying metrics.

28. Copyright 2016 ITRI 工業技術研究院
平台基礎資源監控
監控項目包含:
• Prometheus 透過node-exporter做為agent，收集node上的資源使用狀況:
(服務節點數量、CPU使用率、Memory使用量、Filesystem Space、Host
Temperature、Network I/O)
• Kubernetes和基礎服務是否存活: (kubelet、Registry、API Server、NFS
Server)

29. Copyright 2016 ITRI 工業技術研究院

30. Copyright 2016 ITRI 工業技術研究院
Container監控
• Prometheus透過kubernetes API得到container資源使用狀態。
• Container監控項目 : (Pod名稱、數量、使否存活、CPU使用率、
Memory使用量、Filesystem、I/O)

31. Copyright 2016 ITRI 工業技術研究院

32. Copyright 2016 ITRI 工業技術研究院
GPU監控
• 每一台GPU Node利用nvidia-smi exporter做為agent回報資訊，
再由prometheu統一收集、達到監控。
• GPU監控項目: (GPU使用率、GPU溫度)

33. Copyright 2016 ITRI 工業技術研究院

34. Copyright 2016 ITRI 工業技術研究院

35. Copyright 2016 ITRI 工業技術研究院

36. Copyright 2016 ITRI 工業技術研究院
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: alertmanager-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: alertmanager
image: quay.io/prometheus/alertmanager:v0.10.0
args:
- "-config.file=/etc/prometheus/config.yml"
- "-storage.path=/alertmanager"
ports:
- containerPort: 9093
protocol: TCP
name: http
volumeMounts:
- name: alertmanager-config-volume
mountPath: /etc/prometheus
volumes:
- name: alertmanager-config-volume
configMap:
name: alertmanager
nodeSelector:
release: production
Alertmanager-deploy.yaml

37. Copyright 2016 ITRI 工業技術研究院
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: default
data:
prometheus.yml: |-
global:
scrape_interval: 15s
rule_files:
- '/etc/prometheus-rules/alert.rules'
alerting:
alertmanagers:
- static_configs:
- targets: ["10.241.1.11:30009"]
修改prometheus-ConfigMap.yaml

38. Copyright 2016 ITRI 工業技術研究院
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-alert-rules
data:
alert.rules: |-
ALERT GPUHighTemperature
IF nvidia_temperature_gpu{instance="10.241.1.22:9101",gpu="Tesla V100-SXM2-16GB[0]"} > 25
FOR 1m
ANNOTATIONS {
summary = "{{$labels.instance}}: GPU Temperature High",
description = "{{$labels.instance}}: This GPU Temperature has really high for over 1m",
}
Alert-rules-configmap.yaml

39. Copyright 2016 ITRI 工業技術研究院

40. Copyright 2016 ITRI 工業技術研究院
Grafana Alert

41. Copyright 2016 ITRI 工業技術研究院
Grafana Alert

42. Copyright 2016 ITRI 工業技術研究院

43. Copyright 2016 ITRI 工業技術研究院
Alert Mail設置
修改 grafana image 設定發信功能
docker run -d -p 3000:3000 monitoringartist/grafana-xxl
docker ps
docker exec -ti <container_ID> sh
apt-get update
apt-get install vim
vim /usr/share/grafana/conf/defaults.ini
SMTP / Emailing [smtp]
enabled = true
host = smtp.gmail.com:587
user = itriccmabin@gmail.com
password =“XXXX"
cert_file =
key_file =
skip_verify = false
from_address = itriccmabin@gmail.com
from_name = DNN_Monitor
[emails] welcome_email_on_sign_up = false templates_pattern = emails/*.html

44. Copyright 2016 ITRI 工業技術研究院

45. Copyright 2016 ITRI 工業技術研究院