In a previous article, I explained three ways to store Prometheus metrics for longer than the default 15-day retention period. This article will continue on that premise and explain more about the different deployment methods for Thanos and the ones that should be used. But before we go into details, a little recap of what Thanos is and its advantages would be helpful to set the context.
The Thanos service logo has the same color as the Thanos villain character from the Marvel comics.
So what is Thanos?
According to the home page of the Thanos website: thanos.io. Thanos is an open-source, highly available Prometheus setup with long-term storage capabilities. Thanos can do this because it can write the data into an object storage (S3-compatible storage) engine that can keep the data for a longer period. This data is queryable anytime it is needed. While Thanos can write to a long-term storage system, it can also read real-time data from Prometheus as the data is being scraped by Prometheus. It has the remote_read and remote_write mechanisms for accessing the data from Prometheus.
Thanos is best deployed in a microservice approach because it is made up of different components that perform specific tasks. This means each component can be scaled independently of the other. These are the major components that make up a production-grade Thanos setup:
- Thanos Sidecar: a pod that runs with Prometheus to pull data from Prometheus as it is being scraped
- Thanos Compactor: compresses data that has not been accessed for some time in the object storage service to save storage space.
- Thanos Ruler: alert rules manager for Thanos, similar to AlertManager in Prometheus
- Thanos Query: the service that processes PromQL queries sent via the browser
- Thanos Gateway: the proxy service for sending requests to the query engine
- Thanos Store: the gateway that communicates with the object storage service.
- Thanos Bucket Web: is used to inspect bucket blocks in the form of an interactive web UI
Now that we know the different components that make up a Thanos setup, let us talk about the various methods of collecting metrics from Prometheus and querying that data in Thanos:
What are the Deployment options?
The three ways to collect metrics from Prometheus to Thanos. These methods are in two major categories: remote_read and remote_write. They are as follows:
- Sidecar Only
- Sidecar and Query Frontend
- Remote Write
Sidecar Only
This is the most popular method of deploying Thanos. In this method, every other component of Thanos is deployed separately using the official Thanos Helm chart. During the deployment of Prometheus, we will need to deploy Thanos as a sidecar to run with Prometheus. Thanos will also be configured to write data to the object storage as it receives it. The querier will then be configured to connect with the Thanos sidecar to query more recent data while data is still written to the object storage service by the sidecar. Older metrics can be queried via the querier, but through the Storage gate to the object storage engine.
The following diagram depicts the system just described:
Press enter or click to view the image in full size
The diagram above shows how data flows from the Thanos sidecar to the S3 bucket, and the Querier connecting to the Thanos sidecar for real-time data. It also depicts that the compactor connects to the object storage to compact old data for improved efficiency.
This deployment can be done using the Kube-Prom Stack Helm Chart. Two configurations will be enabled: the Thanos sidecar container and the object storage credentials for the sidecar
In the values file, add these modifications to enable Thanos’s sidecar
thanos:
objectStorageConfig:
key: thanos-storage-config.yaml
name: thanos-storage-config
The configuration above will ensure the Thanos sidecar sends data to the object storage configured within it. We shall be using S3 in this example, but this configuration supports the following object storage services:
- S3 (or S3-compatible like Linode Object Store, MiniO)
- GCS
- Azure
- Openstack Swift
- Tencent OSS
- AliYun OSS
- Baidu BOS
- Filesystem
- Oracle Cloud Infrastructure Object Storage
Thanos will need permission to write data to the object storage. We shall be using Amazon S3 as our example:
type: S3
config:
bucket:
endpoint: s3.us-west-2.amazonaws.com
access_key: ABCDE
secret_key: 12$5%
Encode the contents of the file above using the following base command cat s3-config.yaml | base64 . Copy the contents of the output and replace them with <base64-encoded-string-s3-config> In the following file below:
apiVersion: v1
kind: Secret
metadata:
name: thanos-storage-config
data:
thanos-storage-config.yaml: <base64-encoded-string-s3-config>
Apply the file to your Kubernetes cluster to create the secret. This will enable Thanos to write data to the object storage service. This data can be retrieved via the Thanos Storegateway and queries via the Thanos Querier. But for real-time data, the Thanos querier will be configured to read data directly from the sidecar, so there can be a one-source-truth for real-time data and data stored in object storage. The single source of truth for querying data is the Thanos querier in the end. The following configuration needs to be added to the Thanos Querier so that it can read real-time data.
--store=thanos-store.monitoring.svc.cluster.local:10901
--store=prometheus-operated.monitoring.svc.cluster.local:10901
The configuration above enables Thanos to read real-time data from the Prometheus Kubernetes service that has been deployed along with the Kube-Prometheus helm chart. These changes will automatically be added when the Thanos sidecar is enabled; if not, you will have to edit it in the, thanos-query-deployment in the section spec.spec.containers args query.
Cons
- Complex setup with multiple Thanos sidecars
Pros
- Multiple points for writing to the object storage share the load
- Multiple access points for real-time data
The next method of getting data out of Prometheus for Thanos to query is the sidecar with a query frontend.
Sidecar with Queryfrontend
This setup technique is very similar to the previous one just deployed, where Thanos sidecar is enabled on Prometheus instances. The sidecar will write metrics collected from Prometheus to the S3 bucket.
What is different in this setup is that, apart from the Thanos sidecar that is deployed, there will be multiple query frontends deployed for each Prometheus instance. For each instance of the Thanos sidecar, there will be a Thanos Querier deployed.
The following architectural diagram explains the flow of data between the different components
Press enter or click to view the image in full size
From the diagram, all the Queriers connect to a single Querier where all the data is aggregated and can be queried from and not the individual Queriers.
Cons
- Complex Implementation
- More overhead
Pros
- Provides high availability for sidecar and queriers.
The last option is the most straightforward and likely the easiest to set up out of the two already mentioned deployment methods. Let’s get to it and see.
Remote Write
This uses the remote_write feature, which exists in Prometheus. Other methods of deployment listed above do not have anything to do with the Prometheus instance, but the remote write does. It requires changes to the Prometheus configuration. A URL where Prometheus will send metric data needs to be configured in Prometheus.
On the Thanos side, one component of Thanos will need to be configured/enabled for metrics to be collected. This component is called the Thanos Receiver. The URL from the Thanos receiver is the URL that will be configured on the Prometheus remote_write URL configuration. This means that real-time data will be sent from Prometheus to the Thanos receiver and then queried via the Thanos Querier.
Thanos Storegateway will be configured with the object storage service because the receiver will store non-real-time data to the object storage via the storegateway and retrieve from the object storage service via the storegateway when the data is needed.
The following diagram describes the remote_write architecture
Press enter or click to view the image in full size
From the diagram above, we seem to have introduced a single point of failure on the Thanos receiver, which is receiving data from multiple Prometheus instances. But the receiver has a feature called the Hashring configuration. This feature, when enabled, will allow Thanos receiver to load balance write requests and ensure high availability of the receiver, thereby mitigating the single point failure problem.
Cons
- Possibility of a single point of failure if not well-configured
Pros
- Much easier to set up compared to the previous mentions
Conclusion
All the deployment methods have their pros and cons. You can choose whichever option to go with, depending on your environment and resources. For a small to medium system that might not have a lot of resources (Human and Infrastructure), the remote write might be the go-to. On the other hand, if there are surplus resources, then the Thanos sidecar approach won’t be bad, because the remote write creates a single-point-of-failure scenario, although the Thanos receiver can be scaled independently to also handle that with the Hashring configuration.
In the end, the choice of deployment to be used here will be based on preference, although I would recommend the Remote Write approach.
Further Reading / References:
https://zapier.com/blog/five-recommendations-when-running-thanos-and-prometheus/
