Centralized Logging & Metrics for Multi‑Cluster Kubernetes with Loki, Promtail, and Prometheus

Loki as a single source of truth for all logging needs across various systems

Logs and metrics are the two unshakeable pillars of observability. Logs give you timestamped facts; metrics reveal long‑term behaviour through time‑series aggregation. In a single‑cluster Kubernetes setup, coupling Promtail → Loki for logs and Prometheus → Grafana for metrics is almost a copy‑paste exercise.

But what happens when you run five EKS clusters (prod, staging, dev, QA, sandbox) or split clusters by business unit? Duplicating every Loki, Prometheus, and Grafana stack is expensive and painful to maintain. The smarter pattern is to centralise all telemetry and visualise everything from one Grafana instance.

This guide provides a step-by-step explanation of how to do that.

What is Loki?

Loki is Grafana’s horizontally scalable log datastore. Unlike Elasticsearch‑style systems, Loki indexes only metadata (labels) and stores the raw log lines cheaply in object storage (S3, MinIO, GCS). That design means:

• Lower costs – perfect for African startups watching every naira.

• Prometheus‑like labeling – log queries feel familiar if you already know PromQL.

• Multi‑tenant by design – critical for centralising logs from multiple clusters without blending data.

Loki Deployment for Log Storage

A single, highly available Loki cluster will live in a central AWS EKS or EC2 environment.

# Add Helm repo
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Values highlights (loki-values.yaml) # storageConfig.bucketNames.chunks: loki-chunks # storageConfig.bucketNames.ruler: loki-ruler # storageConfig.tsdb_shipper.active_index_directory: /data/tsdb # s3.region: us-east-1 # s3.endpoint: s3.amazonaws.com

helm install loki grafana/loki-stack \ --namespace observability \ --values loki-values.yaml

• Binary vs microservices mode: Use single‑binary for dev/test; microservices for production to scale reads/writes independently.

• Object storage: Point Loki at S3, MinIO, or Backblaze B2 to decouple storage from compute and cut cost

Promtail Deployment for Log Collection

Promtail runs as a DaemonSet in each Kubernetes cluster, tailing /var/log/containers/*.log and shipping to central Loki.

# In every cluster
helm install promtail grafana/promtail \
  --namespace observability \
  --set "loki.serviceName=loki-gateway.loki.svc.cluster.local" \
  --set "extraScrapeConfigs[0].job_name=kubernetes-pods" \
  --set "extraScrapeConfigs[0].pipeline_stages[0].docker=true" \
  --set "extraLabels.cluster=$(CLUSTER_NAME)"

Heads‑up: Promtail is on the deprecation path. You can swap in Grafana Alloy (successor to the OpenTelemetry Collector) with identical label‑enrichment shown below.

---

Configuring Promtail (or Alloy) for Multi‑Cluster

Add a cluster label so queries can filter per cluster:

# promtail extraScrapeConfigs
- job_name: kubernetes-pods
  pipeline_stages:
    - docker: {}
    - labels:
        cluster: ${CLUSTER_NAME}   # e.g., prod-us-east-1, staging-eu-west-1

If you need stricter tenancy separation, enable Loki multi‑tenant mode and give each cluster a unique tenant ID + API token.

Security Tips

• Use mTLS between Promtail/Alloy and Loki Gateway.

• Scope API tokens per cluster to write‑only permissions.

---

Metrics with Prometheus & Remote Write

For metrics, keep a lightweight Prometheus in each cluster to scrape exporters locally. Then remote‑write to a long‑term store such as Grafana Mimir or Cortex (both speak Prometheus Remote Write).

remote_write:
  - url: https://mimir-gateway.observability.svc.cluster.local/api/v1/push
    write_relabel_configs:
      - action: add
        replacement: ${CLUSTER_NAME}
        target_label: cluster

This pattern avoids cross‑cluster scraping latency and lets each Prometheus act as a high‑resolution cache.

---

Querying Logs & Metrics in Grafana

1. Add Loki & Prometheus/Mimir data sources in Grafana.

2. Create a dashboard‑level variable called cluster populated by label values (label_values(cluster)).

3. Sample LogQL query:

   {app="payments", cluster="$cluster"} |= "ERROR"
   

4. Sample PromQL query:

   sum by(pod) (rate(http_requests_total{cluster="$cluster"}[5m]))
   

5. Build multi‑cluster heatmaps and alert rules that fire if any prod cluster error‑rate exceeds a threshold

---

Conclusion

Setting up observability for multiple Kubernetes clusters doesn’t mean copy‑pasting an entire monitoring stack everywhere. By centralising Loki for logs and using Prometheus → Mimir remote‑write for metrics, African startups—and teams worldwide—can:

• Save money on duplicate infrastructure.

• Simplify operations with one Grafana to rule them all.

• Gain holistic insights across environments and business units.

Ready to ship? Grab the Helm values from the GitHub repo linked below and start logging smarter today.

© 2025 ewere.tech – Feel free to share, but please credit the source.