Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
The Snake and the Butler
Next
Download to read offline and view in fullscreen.

Share

Kubernetes Colorado - Kubernetes metrics deep dive 10/25/2017

Download to read offline

A deep dive of the metrics exposed in Kubernetes.

Related Books

Free with a 30 day trial from Scribd

See all

Kubernetes Colorado - Kubernetes metrics deep dive 10/25/2017

  1. 1. Reveal Your Deepest Kubernetes Secrets Metrics Kubernetes Colorado - 10/25/2017 Bob Cotton - FreshTracks.io
  2. 2. About Me ● Co-Founder - FreshTracks.io - A CA Accelerator Incubation ● bob@freshtracks.io ● @bob_cotton
  3. 3. Agenda ● Short overview of Prometheus ● Sources of metrics ○ Node ○ kubelet and containers ○ Kubernetes API ○ etcd ○ Derived metrics (kube-state-metrics) ● The new K8s metrics server ● Horizontal pod auto-scaler ● K8s cluster hierarchies and metrics aggregation
  4. 4. Prometheus
  5. 5. Prometheus - A Short Overview ● Based on monitoring patterns at Google ● Open sourced by SoundCloud in early 2015 ● Second project admitted to the Cloud Native Computing Foundation after Kubernetes ● Adoption surge is tracking Kubernetes ○ 63% of teams using Kubernetes use Prometheus ● Metric discovery and collection ● High-performance timeseries database ● Alert definition and generation
  6. 6. Prometheus - Unique Features ● Supports “Metrics 2.0” - i.e. metrics contain any number of label/values ● “Pull based” metrics collection ● Service discovery mechanism ● Rich set of “exporters” ● Extremely high-performance TSDB ● PromQL ● Alert Manager ● Easily installable from Helm ● Grafana typically used for visualization
  7. 7. Labeled Series - aka Metrics 2.0 APIServiceRegistrationController_adds{ instance="172.20.107.203:443", job="kubernetes-apiservers"} 1 apiserver_request_count{ client="kubectl/v1.7.5 (darwin/amd64)kubernetes/17d7182", code="200", contentType="application/json", instance="172.20.78.248:443", job="kubernetes-apiservers", resource="pods", verb="GET"} 20 apiserver_request_count{ client="kubectl/v1.7.5 (darwin/amd64) kubernetes/17d7182", code="200", contentType="application/json", instance="172.20.78.248:443", job="kubernetes-apiservers", resource="pods", verb="LIST"} 30 172_20_78_248.apiserver.pods.get.request_count = 20
  8. 8. Prometheus K8s API Server TSDB Kublet (cAdvisor) node-exporter kube_state_metrics App containers other exporters node_exporter App containers Kublet (cAdvisor) Service Discovery
  9. 9. Prometheus Exposition Format and Exporters ● The Prometheus exposition format - Text over http ○ Efforts to make it a standard ● Supported by Sysdig and the TICK collector ● Close to 100 exporters for various technologies ● The jmx_exporter can cover any Java/JMX application ● https://prometheus.io/docs/instrumenting/exporters/ Demo ● “Official Exporters” ○ node_exporter ○ jmx_exporter ○ snmp_exporter ○ haproxy_exporter ○ cloudwatch_exporter ○ collectd_exporter ○ mysql_exporter ○ memcached_exporter
  10. 10. Sources of Metrics in Kubernetes
  11. 11. Host Metrics from the node_exporter ● Standard Host Metrics ○ Load Average ○ CPU ○ Memory ○ Disk ○ Network ○ Many others ● ~1000 Unique series in a typical node Demo Node node_exporter /metrics
  12. 12. Container Metrics from cAdvisor ● cAdvisor is embedded into the kubelet, so we scrape the kubelet to get container metrics ● These are the so-called “core” metrics ● For each container on the node: ○ CPU Usage (user and system) and time throttled ○ Filesystem read/writes/limits ○ Memory usage and limits ○ Network transmit/receive/dropped Demo Node node_exporter /metrics kubelet cAdvisor
  13. 13. Kubernetes Metrics from the K8s API Server ● Metrics about the performance of the K8s API Server ○ Performance of controller work queues ○ Request Rates and Latencies ○ Etcd helper cache work queues and cache performance ○ General process status (File Descriptors/Memory/CPU Seconds) ○ Golang status (GC/Memory/Threads) Node node_exporter kubelet cAdvisor Any other Pod /metrics API Server
  14. 14. Etcd Metrics from etcd ● Etcd is “master of all truth” within a K8s cluster ○ Leader existence and leader change rate ○ Proposals committed/applied/pending/failed ○ Disk write performance ○ Network and gRPC counters
  15. 15. K8s Derived Metrics from kube-state-metrics ● Counts and meta-data about many K8s types ○ Counts of many “nouns” ○ Resource Limits ○ Container states ■ ready/restarts/running/terminated/waiting ○ _labels series just carries labels from Pods ● cronjob ● daemonset ● deployment ● horizontalpodautoscaler ● job ● limitrange ● namespace ● node ● persistentvolumeclaim ● pod ● replicaset ● replicationcontroller ● resourcequota ● service ● statefulset
  16. 16. Sources of Metrics in Kubernetes ● Node via the node_exporter ● Container metrics via the kubelet and cAdvisor ● Kubernetes API server ● etcd ● Derived metrics via kube-state-metrics
  17. 17. Scheduling and Autoscaling i.e. The Metrics Pipeline
  18. 18. The New “Metrics Server” ● Replaces Heapster ● Standard (versioned and auth) API aggregated into the K8s API Server ● In “beta” in K8s 1.8 ● Used by the scheduler and (eventually) the Horizontal Pod Autoscaler ● A stripped-down version of Heapster ● Reports on “core” metrics (CPU/Memory/Network) gathered from cAdvisor ● For internal to K8s use only. ● Pluggable for custom metrics
  19. 19. Feeding the Horizontal Pod Autoscaler ● Before the metrics server the HPA utilized Heapster for it’s Core metrics ○ This will be the metrics-server going forward ● API Adapter will bridge to third party monitoring system ○ e.g. Prometheus
  20. 20. Kubernetes Hierarchy and Aggregation
  21. 21. Core Metrics Aggregation ● K8s clusters form a hierarchy ● We can aggregate the “core” metrics to any level ● This allows for some interesting monitoring opportunities ○ Using P8s “recording rules” aggregate the core metrics at every level ○ Insights into all levels of your Kubernetes cluster ● This also applies to any custom application metric Demo Namespace Deployment Pod Container
  22. 22. FreshTracks.io is Hiring! https://searchjobs.ca.com/go/Accelerator/3279000/
  23. 23. Questions?
  24. 24. Resources ● Prometheus.io ● Core Metrics in Kubelet ● Kubernetes monitoring architecture ● What is the new metrics-server?
  • return001

    Aug. 22, 2020

A deep dive of the metrics exposed in Kubernetes.

Views

Total views

1,644

On Slideshare

0

From embeds

0

Number of embeds

7

Actions

Downloads

19

Shares

0

Comments

0

Likes

1

×