When Prometheus is used for monitoring and observability, the major focus is on server metrics. Prometheus goes beyond that. It is a time series database engine, so any type of data that is a plot between metrics and time is where Prometheus shines. It collects data per time and can give aggregated metrics or point-in-time metrics for the data that has been collected. In this article I shall be explaining a fast and easy way to deploy the Synthetics Monitoring solution for Prometheus called the blackbox-exporter. But before I go further, I need to explain what Synthetic monitoring is
What is Synthetic Monitoring
Synthetic monitoring is a proactive technique that simulates user interactions, API requests, and network behavior to continuously test applications’ and services’ availability, performance, and functionality. Running scripted tests from various locations and environments helps detect downtime, latency issues, and failures before they impact real users.
This approach ensures a smooth user experience, improves incident response times, and aids in optimizing system reliability.
The basic operation of synthetic monitoring is to continuously probe a particular URL with the expectation of getting an HTTP 200 response, which is an acknowledgment that the service is running as it should. Any other response received when a service probed automatically means that the service is not responsive and indicates there might be a problem with the services.
Different services can be used to achieve this result which are not within the Prometheus eco system. Some of proprietary, and free. The following is a list of synthetic monitoring services, or in a simpler form, Uptime Monitoring Services
- Uptime Robot (free for the first 50 monitors)
- SatusCake (free for the first 3 monitors)
- Uptime Kuma (open-source)
- Pingdom (30-day free trial)
- Betterstack (free for the first 10 monitors)
- Amazon CloudWatch Synthetics (no free monitors)
A monitor is a configuration entry for performing an uptime or health check on a URL. A monitor is made up of different parameters such as the URL to configure, port (if needed), the check interval (how frequently should the URL be checked if it is up or down), notification channel (where to send a notification when the URL is down); slack, email, Teams etc, region (this means where the URL will be pinged from).
The Uptime checking tools are great, but they pose a new tool in the observability stack that needs proper management. If I were to choose from any of these, I would go for Statuscake and Betterstack. The reason I would choose both of them is the Terraform support they both have. Statuscake has a well-structured and working Terraform provider, which can help manage the life cycle of monitors in Statuscake. Though the last update for the provider was made a year ago, it is still valid and works. It can be integrated as part of a deployment workflow to create monitoring when deploying a service, and easy to destroy when the service is no longer useful.
Just like Statuscake, Betterstack has a functioning Terraform provider, which is from the official owners of the product. At the time of this writing, the last update on the provider was made about 7 days ago, which means regular updates are made on the provider. This ensures the stability of the provider and the resources within the provided, making Betterstack a better option for implementing Infrastructure as Code with Synthetic monitoring.
Now that we understand Synthetic monitoring and other existing tools that can be used for it. Let us see how this same setup can be configured in Prometheus without the need for third-party tools. I think it makes sense to do Synthetic monitoring in Prometheus if you already have it running, especially if you are using kube-prometheus-stack. With kube-prometheus-stack, It is easier, faster, and cheaper to run and maintain. If you have never deployed kube-prometheus-stack. The next step will explain it; if not, you can skip forward to the Blackbox Exporter step.
Deploy kube-prometheus-stack
kube-prometheus-stack is an open-source configuration where Prometheus and AlertManager configurations have been pre-defined for Kubernetes metrics collection and alerting. For metrics collection, kube-prometheus-stack does not require any further configuration for it to collect the necessary metrics from the important components of a Kubernetes cluster. Alertmanager in the kube-prometheus-stack has many possibilities of alerts pre-configured, from KubeDeploymentReplicasMismatch, to KubeContainerWaiting, KubeDeploymentRolloutStuck, KubePodNotReady, KubePodCrashLooping, NodeSystemSaturation, and dozens more. These can be helpful to know the behaviour of the components running within the Kubernetes cluster.
One more amazing feature of kube-prometheus-stack is that it installs a bunch of CRDs and a Kubernetes Operator for managing AlertManager and Prometheus configuration. With these CRDs, there is no need to configure AlertManager or Prometheus via the alertmanager.yaml or prometheus.yaml file anymore. AlertRules and Prometheus ScrapeConfigs can be done via the CRDs as YAML manifests. The following is an example of a CRD that creates a new PrometheusRule without any change to the Alertmanager configuration file.
Sample PrometheusRule Config via CRD
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: example-alert-rule
namespace: monitoring
spec:
groups:
- name: example.rules
rules:
- alert: HighCPUUsage
expr: avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8
for: 2m
labels:
severity: critical
annotations:
summary: "High CPU usage detected"
description: "CPU usage on instance {{ $labels.instance }} has been above 80% for more than 2 minutes."
Installing kube-prometheus-stack
Pre-requisties
- A Kubernetes cluster v1.29+ preferably
- Helm installed via helm.sh
Run the following command to install kube-prometheus stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 70.3.0
The commands above will install Prometheus, AlertManager, and Grafana into the Kubernetes cluster. One interesting mention is that the Grafana installed has a pre-configured dashboard for cluster metrics, node metrics, pod metrics, and more. The data on the dashboard are from the metrics collected from the Prometheus that has been installed.
Now that we have kube-prometheus stack running, let us install and configure the blackbox exporter for synthetics using the Probe CRD that has been installed.
Deploy Blackbox Exporter
Before deploying the blackbox exporter, let us get a description of what it is. Prometheus Blackbox Exporter is a specialized exporter designed to probe and monitor network endpoints using protocols like HTTP, HTTPS, TCP, ICMP (ping), and DNS. Unlike standard exporters that collect internal system metrics, Blackbox Exporter actively checks whether a service is reachable, responding correctly, and meeting performance expectations. It works by receiving probe requests from Prometheus, executing tests based on predefined modules, and exposing metrics such as response time, status codes, SSL certificate validity, and connectivity success or failure. Common use cases include website uptime monitoring, API health checks, SSL expiration tracking, and network latency measurement. By integrating Blackbox Exporter with Prometheus, teams can set up alerts to detect service outages and network issues in real time, ensuring system reliability and availability.
Installing Blackbox Exporter
- A Kubernetes cluster v1.29+ preferably
- Helm installed via helm.sh
- kube-prometheus-stack installed and running in the cluster
Run the following command to install kube-prometheus stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install my-prometheus-blackbox-exporter prometheus-community/prometheus-blackbox-exporter --version 9.4.0
Configure Probe with URLs
The following is a simple configuration to monitor www.ewere.tech and www.medium.com/mycloudseries. The probe can be configured to monitor multiple URLs at the same time.
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: blackbox-probe
labels:
release: prometheus-stack
spec:
jobName: k8s-blackbox-scraper
interval: 60s
module: http_2xx
prober:
url: prometheus-blackbox-exporter.default.svc:9115
targets:
staticConfig:
static:
- https://www.ewere.tech
- https://www.medium.com/mycloudseries
This configuration defines a Probe resource in the observability namespace using the kube-prometheus-stack CRD. It sets up a Prometheus job named k8s-blackbox-scraper, which runs every 60 seconds to monitor the availability of two websites: https://www.ewere.tech and https://www.medium.com/mycloudseries. The probe uses the Blackbox Exporter service at prometheus-blackbox-exporter.default.svc:9115 to check if these URLs return a 2xx HTTP status code (indicating success). The http_2xx module ensures that the target websites are reachable and responding correctly. If any of the specified endpoints fail to return a successful response, Prometheus can log the issue and trigger alerts based on additional alerting rules. This setup is useful for external uptime monitoring, allowing teams to detect and respond to potential website outages.
NB: It is important to take note of the label.release value. This value must match the label.release value of the kube-prometheus-stack that was deployed earlier. If the values do not match, Prometheus will not be able to scrape the metrics exposed by the blackbox exporter. Also, the prober.url values must be the service URL of blackbox exporter that has been deployed previously.
What is an Uptime setup without a dashboard to see when the service is up or down and to see other relevant metrics like SSL expiry date, service response time, and probe duration.
Configure Grafana Dashboard
Grafana Library has various dashboards for the blackbox exporter. Any of the existing options can be used and imported into your existing Grafana dashboard to view the metrics exposed by the blackbox exporter and scraped by Prometheus. This link contains a list of dashboards that can be configured. The screenshot below is a sample of a blackbox exporter dashboard showing some relevant metrics of a service.
Conclusion
If you already run Prometheus in your environment, this is a cheaper alternative to StatusCake, UptimeRobot, and even Uptime Kuma. It keeps everything about your observability in a single environment and makes troubleshooting easier and more efficient. There is no need to jump around multiple tools to understand the behaviour of your application; with the Grafana dashboard, you can have a holistic view of everything at a glance.
