Prometheus + Grafana Setup on Kubernetes

Step-by-step guide to set up Kubernetes monitoring with Prometheus and Grafana on Docker Desktop's built-in Kubernetes cluster, including Helm installation, configuration, and dashboard setup.

Prerequisites

Docker Desktop with Kubernetes enabled
kubectl configured and working
Helm 3 installed
At least 4-6 GB RAM allocated to Docker Desktop (Settings → Resources)

Verify your cluster

bash

1kubectl cluster-info
2kubectl get nodes
3helm version

Step 1: Add Helm Repository

bash

1helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2helm repo update

Step 2: Create Values File

Save this as monitoring-values.yaml:

yaml

1# ============================================
2# kube-prometheus-stack — Full Monitoring Setup
3# Docker Desktop Local Development
4# ============================================
5
6# --- Prometheus ---
7prometheus:
8  prometheusSpec:
9    retention: 72h
10    resources:
11      requests:
12        memory: 512Mi
13        cpu: 250m
14      limits:
15        memory: 2Gi
16    # Scrape all ServiceMonitors across all namespaces
17    serviceMonitorSelectorNilUsesHelmValues: false
18    podMonitorSelectorNilUsesHelmValues: false
19    ruleSelectorNilUsesHelmValues: false
20  service:
21    type: NodePort
22    nodePort: 30090
23
24# --- Grafana ---
25grafana:
26  adminPassword: "admin"   # Change this!
27  service:
28    type: NodePort
29    nodePort: 30300
30  # Enable persistence so dashboards survive restarts
31  persistence:
32    enabled: true
33    size: 1Gi
34  # Additional dashboard providers
35  dashboardProviders:
36    dashboardproviders.yaml:
37      apiVersion: 1
38      providers:
39        - name: 'grafana-dashboards-kubernetes'
40          orgId: 1
41          folder: 'Kubernetes Views'
42          type: file
43          disableDeletion: true
44          editable: true
45          options:
46            path: /var/lib/grafana/dashboards/grafana-dashboards-kubernetes
47  # Community dashboards (dotdc set — best for K8s)
48  dashboards:
49    grafana-dashboards-kubernetes:
50      k8s-views-global:
51        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json
52        datasource: Prometheus
53      k8s-views-namespaces:
54        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json
55        datasource: Prometheus
56      k8s-views-nodes:
57        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json
58        datasource: Prometheus
59      k8s-views-pods:
60        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json
61        datasource: Prometheus
62      k8s-system-api-server:
63        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json
64        datasource: Prometheus
65      k8s-system-coredns:
66        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json
67        datasource: Prometheus
68
69# --- Alertmanager ---
70alertmanager:
71  service:
72    type: NodePort
73    nodePort: 30093
74
75# --- Node Exporter (host-level metrics) ---
76# FIX for Docker Desktop: disable hostRootFsMount to avoid
77# "path / is mounted on / but it is not a shared or slave mount" error
78prometheus-node-exporter:
79  hostRootFsMount:
80    enabled: false
81nodeExporter:
82  enabled: true
83
84# --- kube-state-metrics (K8s object metrics) ---
85kubeStateMetrics:
86  enabled: true
87
88# --- Scrape targets ---
89kubelet:
90  enabled: true
91kubeApiServer:
92  enabled: true
93kubeControllerManager:
94  enabled: true
95kubeScheduler:
96  enabled: true
97kubeProxy:
98  enabled: true
99kubeEtcd:
100  enabled: true
101coreDns:
102  enabled: true
103prometheus-node-exporter:
104  hostRootFsMount:
105    enabled: false

Step 3: Install the Stack

bash

1helm install monitoring prometheus-community/kube-prometheus-stack \
2  --namespace monitoring \
3  --create-namespace \
4  -f monitoring-values.yaml

Wait for all pods to be ready:

bash

1kubectl -n monitoring get pods -w

Expected pods (all should be Running):

monitoring-grafana-*
monitoring-kube-prometheus-operator-*
monitoring-kube-prometheus-prometheus-*
monitoring-kube-prometheus-alertmanager-*
monitoring-kube-state-metrics-*
monitoring-prometheus-node-exporter-*

Step 4: Access the UIs

Service	URL	Credentials
Grafana	http://localhost:30300	admin / admin
Prometheus	http://localhost:30090	(no auth)
Alertmanager	http://localhost:30093	(no auth)

If NodePort doesn't work, use port-forward:

bash

1kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring
2kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring
3kubectl port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 -n monitoring

Step 5: Verify Prometheus Targets

Open Prometheus: http://localhost:30090
Go to Status → Targets
Verify these targets are UP:
- kubernetes-apiservers
- kubernetes-nodes
- kubernetes-nodes-cadvisor
- kubernetes-service-endpoints
- kube-state-metrics
- node-exporter
- prometheus
- alertmanager
- grafana

Step 6: Explore Grafana Dashboards

Built-in Dashboards (auto-installed)

Navigate to Grafana → Dashboards. You'll find these folders:

Default folder — kubernetes-mixin dashboards:

Kubernetes / Compute Resources / Cluster
Kubernetes / Compute Resources / Namespace (Pods)
Kubernetes / Compute Resources / Namespace (Workloads)
Kubernetes / Compute Resources / Node (Pods)
Kubernetes / Compute Resources / Pod
Kubernetes / Compute Resources / Workload
Kubernetes / Networking / Cluster
Kubernetes / Networking / Namespace (Pods)
Kubernetes / Networking / Namespace (Workload)
Kubernetes / Networking / Pod
Kubernetes / API Server
CoreDNS
Node Exporter / Nodes
Node Exporter / USE Method / Cluster
Node Exporter / USE Method / Node
Alertmanager Overview
Prometheus Overview

Kubernetes Views folder — dotdc community dashboards:

k8s-views-global (cluster overview)
k8s-views-namespaces
k8s-views-nodes
k8s-views-pods
k8s-system-api-server
k8s-system-coredns

Optional: Import More Dashboards Manually

In Grafana → Dashboards → Import → enter dashboard ID:

ID	Name	Focus
315	Kubernetes Cluster Monitoring	Classic cluster view
6336	Kubernetes Pods	Pod-level detail
11802	K8s Cluster with SLO views	SLO / error budgets
17375	K8s Resource Monitoring	Requests vs limits

What Metrics You Get

From node-exporter (host-level):

CPU usage per core
Memory usage / available / cached
Disk I/O (reads, writes, latency)
Network traffic (bytes in/out per interface)
Filesystem usage
System load averages

From kube-state-metrics (K8s objects):

Pod status (running, pending, failed, restarts)
Deployment replicas (desired vs available)
Container resource requests and limits
Job/CronJob status and duration
PVC status and capacity
Node conditions and allocatable resources
Namespace resource quotas

From kubelet/cAdvisor:

Container CPU usage (actual vs requested)
Container memory usage (RSS, cache, working set)
Container network I/O
Container filesystem reads/writes
Pod CPU/memory throttling

From API server:

Request rate and latency by verb/resource
Request error rate
Etcd request duration
Workqueue depth and processing time

From CoreDNS:

DNS query rate by type
DNS response codes
DNS latency histogram

Useful PromQL Queries

Try these in Prometheus (http://localhost:30090/graph):

promql

1# CPU usage by pod
2sum(rate(container_cpu_usage_seconds_total{namespace!=""}[5m])) by (namespace, pod)
3
4# Memory usage by pod
5sum(container_memory_working_set_bytes{namespace!=""}) by (namespace, pod)
6
7# Pod restart count
8kube_pod_container_status_restarts_total
9
10# Node CPU usage %
11100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
12
13# Node memory usage %
14(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
15
16# Network receive rate by pod
17sum(rate(container_network_receive_bytes_total[5m])) by (namespace, pod)
18
19# Disk I/O
20rate(node_disk_read_bytes_total[5m])
21rate(node_disk_written_bytes_total[5m])
22
23# API server request rate
24sum(rate(apiserver_request_total[5m])) by (verb, resource)
25
26# Top 10 pods by CPU
27topk(10, sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace, pod))

Troubleshooting

Pods stuck in Pending

bash

1kubectl -n monitoring describe pod <pod-name>
2# Usually a resource issue — increase Docker Desktop RAM

Targets showing as DOWN in Prometheus

bash

1# Check if kube-proxy metrics are exposed
2# On Docker Desktop, some control plane targets may be unreachable
3# This is normal — kube-scheduler, kube-controller-manager, and etcd
4# may not expose metrics by default on Docker Desktop

Grafana shows "No Data"

Check that Prometheus datasource is configured (Grafana → Connections → Data Sources)
The datasource URL should be: http://monitoring-kube-prometheus-prometheus.monitoring:9090
Verify the dashboard variable selectors (namespace, pod) have values

Reset everything

bash

1helm uninstall monitoring -n monitoring
2kubectl delete namespace monitoring
3# Then reinstall from Step 3

Upgrade the Stack

bash

1helm repo update
2helm upgrade monitoring prometheus-community/kube-prometheus-stack \
3  --namespace monitoring \
4  -f monitoring-values.yaml

Summary

After completing this guide you have:

Prometheus scraping all cluster metrics
Grafana with 20+ pre-built dashboards
Alertmanager for alert routing
node-exporter for host metrics
kube-state-metrics for Kubernetes object metrics
Full coverage: CPU, memory, disk, network, pods, nodes, API server, DNS

All accessible locally: