The standard kube-prometheus install is a good starting point, but on a bare-metal cluster it usually stops one step short of being conveniently usable.

That missing step is what this note was about: getting Prometheus and Grafana onto stable external IPs through MetalLB, and fixing the RBAC when the default permissions were not enough.

1. Install kube-prometheus

The base install was:

1
2
3
4
5
6

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus

kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/

That gets the stack into the cluster, but it does not guarantee that the services are externally reachable the way you want.

2. Deal with the Network Policies

The source note explicitly removed the monitoring namespace network policies because they blocked the external access pattern that was needed:

1
2
3

kubectl get networkpolicies.networking.k8s.io -n monitoring \
  | awk '{print $1}' \
  | xargs kubectl delete networkpolicies.networking.k8s.io -n monitoring

I would treat that as a temporary or environment-specific choice, not a universal best practice. Sometimes the right move is to modify those policies instead of deleting them.

3. Add MetalLB

The MetalLB install flow was:

1
2
3

helm repo add metallb https://metallb.github.io/metallb
helm repo update
helm install --wait metallb metallb/metallb --namespace loadbalancer --create-namespace

Then define an address pool and an advertisement:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: monitoring-pool
  namespace: loadbalancer
spec:
  addresses:
    - 192.0.2.50-192.0.2.90
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: monitoring-pool-adv
  namespace: loadbalancer
spec:
  ipAddressPools:
    - monitoring-pool

The real addresses from the note are replaced here with documentation-safe examples.

4. Convert the Monitoring Services to LoadBalancer

Once MetalLB is ready, exposing the services is simple:

1
2

kubectl patch svc -n monitoring prometheus-k8s -p '{"spec":{"type":"LoadBalancer"}}'
kubectl patch svc -n monitoring grafana -p '{"spec":{"type":"LoadBalancer"}}'

That is the point where the monitoring stack stops feeling theoretical and starts feeling usable.

5. Patch the RBAC if the Default Install Is Not Enough

The stock prometheus-k8s role in the note needed extra resource permissions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

rules:
  - apiGroups: [""]
    resources:
      - nodes/metrics
      - pods
      - endpoints
      - services
    verbs:
      - get
      - list
  - nonResourceURLs:
      - /metrics
    verbs:
      - get

This is exactly the kind of thing polished install guides tend to skip. In real clusters, the “works on paper” RBAC is sometimes not enough for the observability you actually want.

Closing Thought

The install itself is not the interesting part here. The interesting part is what made the stack practical:

  • the monitoring namespace was reachable
  • MetalLB supplied stable service IPs
  • Prometheus had the permissions it needed

That is the difference between “deployed” and “actually usable.”