In part 1 of this series, I discussed the rise of microservice architecture and the reliance on Kubernetes and Docker for container orchestration and management. I also shared some of the challenges these new technologies present and what sources of data we need in order to effectively monitor our Kubernetes environments. In my concluding comments, I mentioned that Prometheus provides an excellent starting point for collecting and analyzing this data. In this post, I’ll give a brief overview of Prometheus, illustrate how to collect Kubernetes data with Prometheus, and discuss some of the pros and cons of the Prometheus approach.
In the next installment of this series, I’ll address some of those cons, illustrating how Sensu and Prometheus are complementary.
Prometheus is an open source monitoring tool originally built at SoundCloud. Now a standalone open source project, Prometheus followed in Kubernetes’ footsteps to join the Cloud Native Computing Foundation (CNCF) in 2016.
One of the most unique attributes about Prometheus is that it uses a pull-based architecture — as opposed to pushing metrics to the monitoring tool, it pulls metrics from services. Prometheus also has a pushgateway, meaning it does support “push” for certain metrics for when the pull model doesn’t work.
One of the most interesting things about the Prometheus pull-based model is the dependency on “metric exporters,” which are used to expose metrics for services and systems. Prometheus exporters can be standalone applications (akin to single-purpose monitoring agents), or they can be built into your applications and services via language-specific client libraries (like a modern APM solution).
Prometheus components include:
Prometheus architecture and ecosystem components. Source.
There’s a Kubernetes operator that’s been created for Prometheus, which automates a majority of the deployment process. If you’re going to use Prometheus for monitoring Kubernetes, use the operator. A related project called kube-prometheus offers a packaged solution for deploying and managing Prometheus on Kubernetes, combines the Prometheus operator with a collection of manifests that includes pre-configured Grafana dashboards and a few Prometheus exporters. By itself, the operator doesn’t really deploy a complete monitoring solution; kube-prometheus takes it one step further by being prescriptive in what you monitor and how. And — in addition to pre-configuring alertmanager — it automatically deploys kube-state-metrics, which is pretty rad. Effectively, you can run this one sweet command to get started with Prometheus on Kubernetes.
Here are the steps we recommend for collecting Kubernetes data with Prometheus:
Note: this is by no means a full Prometheus installation guide (Prometheus has its own). kube-prometheus is a packaged solution for monitoring Kubernetes and your labeled applications but it’s just a starting point.
Now that I’ve given you a sense of how to monitor Kubernetes with Prometheus, I’ll lay out the benefits and shortcomings of Prometheus for monitoring.
The benefits of Prometheus include:
The fact that Prometheus’ data collection model defaults to time series presents a double-edged sword: on the one hand, you can easily standardize collecting data in a particular way, which helps simplify things; on the other, you’ve now limited yourself to a constrained data model that might be missing some context.
This leads me to the cons of Prometheus:
In my next post, I’ll share best practices for monitoring your applications running on Kubernetes (including the daemonset and sidecar pattern), show how to use Sensu to scrape Prometheus metrics, and address the cons outlined above — illustrating how Sensu and Prometheus are complementary, forming a comprehensive solution for monitoring Kubernetes and the applications running on it.