How to Set Up Kubernetes Monitoring

Where should Kubernetes monitoring begin?

I have seen plenty of Kubernetes clusters where every pod was reported as Running while customers were already receiving errors. The scheduler was happy. The application was not. CPU throttling, memory pressure, rising HTTP errors, and a nearly full filesystem can all hide behind a green-looking pod list.

If you discover those symptoms only after a user opens a ticket, you are investigating an incident rather than monitoring the service. My usual starting point is the Prometheus, Grafana, and Alertmanager stack, with kube-state-metrics for Kubernetes object state.

Let me put it this way: monitoring is not a contest to produce the largest number of graphs. I want to know which signals deserve to wake me at 3 a.m., which ones need a daytime review, and which ones are only useful during troubleshooting. A pod restart matters. The restart count, its owning Deployment, and the effect on users matter more.

What each component actually does

These components are related, but they do different jobs. Keeping that distinction clear makes troubleshooting much less confusing.

Metrics Server and Prometheus are not interchangeable. Metrics Server provides a current resource-use view through the Kubernetes API. Prometheus stores historical time series, collects application metrics, and evaluates detailed alert rules.

A working kubectl top nodes command is useful. It is not proof that monitoring is complete.

Check the cluster before installing anything

I test monitoring changes on the Proxmox cluster in my home lab before touching production. My second-hand HP EliteDesk has taken plenty of failed chart upgrades so customer workloads did not have to. That is a much cheaper place to learn what a bad values file does.

First confirm that the API server, nodes, and Helm client are usable:

kubectl cluster-info
kubectl get nodes -o wide
kubectl version
helm version

The first command checks API connectivity. The node listing shows scheduling capacity and node conditions, while the version commands tell you what clients and servers you are working with. Kubernetes client output varies by release, so I do not rely on one exact format.

Create a dedicated namespace for the stack:

kubectl create namespace monitoring
kubectl get namespace monitoring

If the namespace already exists, AlreadyExists is harmless. In a production workflow, I normally manage this through a manifest or GitOps repository instead of treating imperative commands as the source of truth.

Installing kube-prometheus-stack with Helm

The kube-prometheus-stack chart from the Prometheus Community packages Prometheus Operator, Prometheus, Grafana, Alertmanager, Node Exporter, and kube-state-metrics together. The exact resources and chart defaults change over time, so I pin the chart version rather than allowing an unexpected update to alter CRDs during maintenance.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm search repo prometheus-community/kube-prometheus-stack

Record the chart version shown by the search command. Then inspect its values before installing:

helm show values prometheus-community/kube-prometheus-stack > kube-prometheus-stack.values.example.yaml

For a disposable test cluster, the defaults are enough to get started:

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

This creates a Helm release called monitoring. I would not use those defaults for a production cluster. Storage, resource requests, retention, access, and alert routing should be deliberate.

A starting values file

The following configuration is only a starting point. Replace standard with a StorageClass that exists in your cluster, as shown by kubectl get storageclass. On some managed clusters, the default StorageClass has a different name; on a bare-metal cluster, there may be none at all.

grafana:
  admin:
    existingSecret: grafana-admin
  persistence:
    enabled: true
    storageClassName: standard
    size: 10Gi
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 40GB
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        cpu: 2
        memory: 4Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

This keeps Grafana settings, Prometheus data, and Alertmanager state on persistent volumes. It does not make the data a backup. I still back up the underlying storage or export the configuration separately.

Disk sizing depends on scrape interval, retention, metric cardinality, and the number of targets, not just pod count. I learned that after treating a small cluster as if every metric had the same storage cost.

Install or update the release with the values file:

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values values.yaml \
  --wait \
  --timeout 10m

The --wait flag waits for resources to become ready, but it cannot repair a missing StorageClass or an unschedulable pod. If Helm times out, inspect pods, events, and PVCs before running the command again.

Verify the installation instead of trusting Helm

Start by watching the monitoring namespace:

kubectl get pods -n monitoring -w

Most components should reach Running. A short-lived setup job may show Completed. A pod stuck in Pending usually points to insufficient resources, an unsuitable node, or a PVC that cannot bind.

Check storage, Services, and recent events:

kubectl get pvc -n monitoring
kubectl get svc -n monitoring
kubectl get events -n monitoring --sort-by=.lastTimestamp

Messages such as FailedScheduling, FailedMount, and image-pull errors usually narrow the problem quickly. They have saved me from repeatedly reinstalling a chart when the real issue was a missing disk provisioner.

For an initial test, use port forwarding. Run each command in a separate terminal:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://127.0.0.1:9090 and http://127.0.0.1:3000 locally. Port forwarding is suitable for a temporary check, not for publishing Grafana to the internet. Use a VPN, bastion host, or authenticated reverse proxy instead. The same principle applies to the panel access described in Secure Remote Access to Hosting Panels with VPN and Bastion Hosts.

Run the up query in Prometheus. A value of 1 means that Prometheus reached a target; 0 means the scrape failed. Change the time range and confirm that samples are arriving. A dashboard that loads is not evidence that its queries are returning current data.

Expose application metrics, not only cluster metrics

Node and Kubernetes object metrics tell me what the platform is doing. They do not tell me whether an order endpoint is slow or whether checkout is returning errors. The application needs to expose request count, latency, error rate, and other service-specific measurements, commonly through a /metrics endpoint.

With Prometheus Operator, a ServiceMonitor describes how that endpoint should be scraped. The Service must have a matching label and a named metrics port:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders-api
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: orders-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

The release: monitoring label must match the selector used by the Prometheus resource created by the Helm release. If you changed that selector in values.yaml, use the matching value here. The Service in the production namespace also needs a port named metrics; a ServiceMonitor can look perfectly valid and still produce no target without it.

Check the Prometheus Status and Targets pages. If the target is not UP, check the Service selector, DNS name, port, path, and NetworkPolicy. The endpoint does not need to be exposed outside the cluster. Prometheus only needs network access to it.

Alert rules should lead to an action

Writing Prometheus rules without configuring Alertmanager leaves the job half finished. Prometheus evaluates the expression. Alertmanager groups, silences, and routes the resulting notifications.

A for period prevents a brief fluctuation from waking someone unnecessarily. This example reports a sustained gap between available and desired Deployment replicas:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: application-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: application.rules
      rules:
        - alert: DeploymentReplicasMismatch
          expr: kube_deployment_status_replicas_available{namespace="production"} < kube_deployment_spec_replicas{namespace="production"}
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Deployment has fewer pods than expected"
            description: "The number of available pods for {{ $labels.deployment }} has been low for 10 minutes."

        - alert: InstanceDown
          expr: up{job=~"critical-.*"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Critical monitoring target is unreachable"
            description: "The target {{ $labels.instance }} cannot be scraped."

The first rule measures reduced Deployment capacity. The second is deliberately scoped to jobs named critical-*; applying up == 0 to every target can create a noisy alert storm during planned maintenance.

Store SMTP credentials and webhook tokens in a Kubernetes Secret or a dedicated secret-management system. Do not place them directly in a values file committed to Git. After configuring a receiver, trigger a controlled test alert and verify the message arrives.

A silent Alertmanager is a fire alarm with no battery.

Prometheus has a cardinality problem to manage

Prometheus needs monitoring too. When its memory use keeps climbing, I first inspect retention, scrape volume, and label cardinality. A unique URL, customer ID, or order number used as a label can create a new time series for every request.

These two labels have very different consequences:

http_requests_total{path="/orders/847291"}
http_requests_total{route="/orders/:id"}

The first can produce one series per order. The second normalizes the route and is much easier to control. Metric names and labels deserve review during application development; removing a high-cardinality metric after Prometheus is already under pressure is unpleasant.

If the Prometheus pod restarts, inspect the pod and its previous container logs:

kubectl describe pod -n monitoring -l app.kubernetes.io/name=prometheus
kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus --previous
kubectl top pod -n monitoring

The --previous flag shows logs from the container instance that exited. If the pod was OOMKilled, increasing the memory limit may only postpone the same failure. Check retention, scrape interval, and labels first.

What a DDoS night taught me about useful signals

During my first major DDoS incident, node dashboards looked reasonably normal while application errors climbed. At first glance, the traffic increase could have been genuine demand. When we compared request sources and response codes, a large portion of the traffic clearly did not resemble normal visitor behavior.

We enabled rate limits, geographic filtering, and connection limits at the ingress layer. I also had to explain one point to the customer: not every HTTP request represents a real visitor.

Since that night, I have kept ingress 5xx rate, request volume, response time, and external availability separate from node-health alerts. Prometheus target failures, Grafana data-source failures, and Alertmanager delivery failures deserve their own checks.

The dashboards were not the problem. We were watching the wrong layer.

Security, access, and retention

Grafana dashboards can expose namespace names, node names, service names, and metric labels that should not be public. I disable anonymous access and avoid exposing Grafana directly to the internet.

A ServiceAccount with broad Kubernetes API permissions creates another unnecessary risk. Review the RBAC objects installed by the chart and grant only the access the components need.

NetworkPolicy introduces a practical trap. If you enable restrictive policies, confirm that Prometheus can reach kube-state-metrics, Node Exporter, and application metric Services. I usually begin with the required paths documented and tested, then tighten policies gradually. Blocking everything at once makes the failure harder to locate.

Retention has a storage cost. Seven to fifteen days may be a reasonable starting range for a small cluster, but the correct value depends on your incident history and disk capacity. Set both retention and retentionSize; do not let the Prometheus volume grow without a boundary.

Operating the stack after installation

A dashboard is not a maintenance plan. Every week, review alert history, silences, disk usage, and the alerts that resulted in no action. If the same rule wakes someone for a condition nobody can act on, change the query or remove the rule.

When you create a maintenance silence, give it an expiry and a reason. A silence without an end time is just a polished way of disabling an alarm.

After chart or application changes, I check:

• Prometheus target status
• Grafana data-source access
• Test notifications through Alertmanager receivers
• PVC utilization
• Pod restart counts and OOMKilled events
• Node disk, CPU, and memory pressure
• Ingress HTTP 5xx rate and response time

Keep an external synthetic check as well. Every pod can be healthy while the payment endpoint is broken. An outside HTTP check combined with internal metrics gives you two different views of the same service.

The separation between a warning and an affected user journey matters outside Kubernetes too. The method described in WordPress Site Health: How to Read and Fix the Report applies here: identify the component, then identify the user flow it affects.

In my lab, I install chart updates into a separate namespace, fire a test alert, and compare dashboard queries before touching the live release. I once spent hours chasing an apparently empty graph before noticing that the new Deployment no longer matched the query’s label selector. The graph was honest. My assumption was not.

Frequently asked questions

What is the difference between Prometheus and Metrics Server?

Metrics Server exposes current CPU and memory usage through the Kubernetes API and supports commands such as kubectl top and features such as the Horizontal Pod Autoscaler. Prometheus stores time-series data, collects application metrics, and evaluates alert rules.

Is Grafana required for Kubernetes monitoring?

No. Prometheus queries and Alertmanager can work without Grafana. Grafana is useful because it puts related metrics into dashboards that are faster to read during an investigation.

How long should Prometheus data be retained?

That depends on cluster size, disk capacity, scrape volume, and the historical questions you need to answer. Starting with seven to fifteen days and watching disk usage is safer than allowing unlimited retention.

How can I reduce the number of alerts?

Check whether several rules report the same event. Add a for period for short-lived conditions, separate critical alerts from informational ones, and remove alerts that never require an action. The best alert is not the loudest one; it is the one that tells me what needs doing next.