In a previous article, we talked about the importance of being able to manage an application and have visibility into the behavior of the application. In that post, the focus was on the metrics that can be collected from applications and the environment they run. In this post, we shall be talking about another type of telemetry or signal that needs to be collected to understand the behavior of applications. This type of signal is called a log.

What is a Log?

It is a record of events, actions, or messages generated by software or hardware for monitoring and debugging purposes. Logs help you track all the output of operations of your code and associate timestamps to each operation which makes it easy to know when an event happened. Logging is managed by the developer during the process of building an application. The better the logging approach used on the application the better that can be collected by log collection tools.

Now that we understand what a log is. How does Loki come into the mix?

When logs are created they need to be processed (collected, stored, and visualized) in a central location for analytics and debugging. Grafana Loki can be used to process logs produced by applications.

What is Grafana Loki

Grafana Loki is a log aggregation system designed to collect, store, and query logs. It is optimized for efficiency and scalability, focusing on indexing metadata rather than full log content, which makes it cost-effective compared to traditional logging solutions. It is horizontally scalable (uses a microservice approach for its

What is Grafana Loki Used For?

Loki is primarily used for log monitoring, allowing users to search and analyze logs in combination with metrics and traces. It integrates seamlessly with Grafana, enabling a unified dashboard for visualizing logs alongside metrics, making troubleshooting and monitoring efficient.

Key Components of Grafana Loki

Loki Server: The core component that collects and stores log data. It ingests log streams and indexes metadata for fast searching.

Promtail: An agent that runs on client machines, is responsible for discovering log files, reading them, attaching labels, and sending them to Loki.

LogCLI: A command-line interface tool used for querying logs stored in Loki.

Grafana: The visualization layer where logs can be viewed, searched, and correlated with metrics. Grafana provides the user interface for querying Loki.

How to Deploy Grafana Loki

Loki can be deployed in two major forms:

Single Binary
Distributed Setup

Single Binary refers to deploying Loki using a single executable file on a virtual machine. This setup is ideal for small-scale applications where the daily log volume is relatively low. If the total log data does not exceed 5GB per day, this deployment option is suitable for Loki.

The Distributed setup is for a more scalable approach. When dealing with Terabytes of logs this is the best approach to use. This setup is designed to ensure the horizontal scalability and high availability of Loki deploying Loki as multiple microservices to handle higher loads.

This model is mostly used when deploying Loki in a Kubernetes cluster. It is the most efficient way to deploy Loki to be horizontally scalable and highly available. It deploys Loki into different unique components with each component handling a specific part of the logging process. From the collection process to the storage process, the retrieval process, and compression operation, they are all handled by different components of the Loki setup which de-couples the operations and ensures they all run independently, can be scaled independently, and work in parallel without one operation interfering with another operation.

Key microservices include:

Distributor: Receives and validates incoming logs from sources like Promtail, Fluentd, OTel, Logstash, Grafana Alloy, and Fluentbit.

Ingester: Stores log data temporarily and eventually flushes it to long-term storage.

Querier: Handles user queries by fetching and filtering log data.

Query Frontend: Sits in front of the Querier, managing and optimizing large-scale queries.

Storage/Storegateway: Use scalable storage backends like S3, GCS, or Cassandra for log data.

Compactor: Compresses logs stored in the object storage periodically to reduce its size and improve retrieval efficiency

Load Balancing: Use a load balancer (like NGINX or HAProxy) in front of Distributors for traffic distribution.

High Availability: Deploy Loki components in clusters with redundancy to avoid single points of failure.

Scaling Horizontally: Add more instances of the core services (Distributor, Ingester, and Querier) as needed. This can be managed via Kubernetes for dynamic scaling.

Storage Backend: Choose scalable storage like Amazon S3, Google Cloud Storage, or Azure Blob Storage for storing log data, allowing efficient scaling.

General Use Cases for Grafana Loki

Centralized Logging: Aggregate logs from various sources like servers, containers, and cloud environments for centralized monitoring.

Incident Response: Investigate incidents by correlating logs with metrics and traces, quickly identifying the root cause.

DevOps & SRE Monitoring: Integrate logs with metrics to get a holistic view of infrastructure and application health, aiding in proactive monitoring.

Debugging and Troubleshooting: Use Loki to search for errors, warnings, and specific events in logs, simplifying the debugging process.

Security Monitoring: Track access logs and other security-related logs for threat detection and compliance purposes.

Conclusion

Loki can be applied in various use cases and has a very small memory footprint, due to the indexing technique used in its architecture. Loki is very flexible and has a vast range of applications for both small, medium, and large-scale organizations. It is quite cost-effective and easy to manage over a long period. Low barrier of entry and continuous maintenance is quite low.

Observability 101: What is Grafana Loki

Prometheus 3.0 Released: A Landmark in Cloud-Native Monitoring

Observability 101: What is Prometheus

Synthetics Monitoring in Prometheus using blackbox-exporter in Kubernetes

Subscribe to our Newsletter