One of the issues auditors find disturbing with logs is leaking sensitive information such as API keys and secrets. Secrets can come from numerous sources, and it can be difficult to anticipate which applications or services might accidentally reveal sensitive data. Ensuring a business complies with regulatory policies requires that sensitive credentials are not visible in the logs of application. The easiest way this is done is by the developer redacting it via code or truncating the information from being visible or totally removing it from the application logs via what is written to stdout. If the logging system for an application is not standardized and centralized, it means that every application code will need this implementation to ensure all logs stored and viewed meet the compliance requirements which can be daunting and difficult to implement and track over time.
Alloy to the Rescue
Redacting secrets during the ingestion and processing of logs presents a practical, scalable, and proactive approach to minimizing the risk of sensitive data exposure. By intercepting and sanitizing logs as they are received—before they are stored, forwarded, or displayed—organizations can significantly reduce the chances of secrets being accidentally leaked through logs, whether it’s API keys, passwords, tokens, or other sensitive credentials. Recognizing the importance of this safeguard, the Security Operations team at Grafana Labs is developing an experimental component called loki.secretfilter as part of Grafana Alloy.
The loki.secretfilter aims to provide a flexible and powerful way to detect and redact secrets in real-time, regardless of the source of the logs. This allows teams to implement consistent security practices without needing to deeply customize every application’s logging behavior. By embedding this functionality directly into the observability pipeline, it acts as an additional line of defense—helping organizations maintain compliance, avoid data breaches, and reduce the operational burden of secret management in logs.
How it works
As an Alloy component, loki.secretfilter offers flexibility in integration within your infrastructure. Alloy collects logs from various sources, processes them, and forwards them to your log management system. When used with Loki, loki.secretfilter can be seamlessly incorporated into this workflow to filter log lines, regardless of their origin or content.
To detect sensitive information, loki.secretfilter utilizes the gitleaks.toml configuration file from the open-source project Gitleaks. This file contains a comprehensive set of predefined patterns for identifying secrets across various formats and frameworks. Users can also provide a custom gitleaks.toml file to tailor the detection patterns to their specific needs. Each log line is analyzed individually, using the regular expressions defined in the configuration to identify potential secrets. When a secret is found—and unless it appears on an allowlist—it is replaced with a redacted string. This redaction is customizable and may include information like the type of secret (as determined by the matching regex) and, optionally, a hash of the original secret.
It’s important to note that loki.secretfilter is currently designated as experimental. Experimental components are subject to frequent changes and may be removed without a direct replacement. To use this component, the stability.level flag must be set to experimental. While loki.secretfilter enhances log security by redacting sensitive information, it has certain limitations. Personally Identifiable Information (PII) isn’t currently within its scope, and some secrets might remain undetected. Additionally, the component may generate false positives or redact more information than necessary. Therefore, it shouldn’t be solely relied upon for redacting sensitive information. Furthermore, loki.secretfilter operates on log lines and doesn’t scan labels or other metadata.
Once redacted, the logs can be ingested by Loki, where users can query for the redaction string. This allows teams to identify secrets that might have been exposed before the component was enabled, and take corrective actions—such as turning off verbose or debug logging in misbehaving tools.
Including a hashed version of the secret in the redaction string adds an extra layer of traceability. This enables engineers with access to the secrets management system to match the hash in the logs with the actual stored secrets, helping pinpoint which specific secret was leaked—without ever exposing the sensitive value itself.
Sample Loki Alloy Configuration
This configuration outlines a log processing pipeline in Grafana Alloy designed to collect local log files, redact sensitive information, and forward the sanitized logs to a Loki instance for storage and analysis.
local.file_match "local_logs" {
path_targets = "<PATH_TARGETS>"
}
loki.source.file "local_logs" {
targets = local.file_match.local_logs.targets
forward_to = [loki.secretfilter.secret_filter.receiver]
}
loki.secretfilter "secret_filter" {
forward_to = [loki.write.local_loki.receiver]
redact_with = "<ALLOY-REDACTED-SECRET:$SECRET_NAME:$SECRET_HASH>"
}
loki.write "local_loki" {
endpoint {
url = "<LOKI_ENDPOINT>"
}
}
Here’s a breakdown of each component and its role in the redaction process:
local.file_match “local_logs”:
- Purpose: Identifies and targets specific log files on the local system for collection.
Configuration:
- path_targets: Specifies the file paths to monitor for log entries. Replace “<PATH_TARGETS>” with the actual paths of the log files you intend to collect.
loki.source.file “local_logs”:
- Purpose: Reads the log files identified by the local.file_match component.
Configuration:
- targets: References the targets defined in local.file_match.local_logs.targets, indicating which log files to read.
- forward_to: Directs the collected log entries to the next component in the pipeline, in this case, the loki.secretfilter.
loki.secretfilter “secret_filter”:
- Purpose: Scans incoming log entries for sensitive information, such as API keys, passwords, or tokens, and redacts them before they are forwarded.
Configuration:
- forward_to: Specifies the next destination for the processed logs, here being the loki.write component.
- redact_with: Defines the string that will replace any detected sensitive information. The placeholder “<ALLOY-REDACTED-SECRET:$SECRET_NAME:$SECRET_HASH>” can be customized to include:
- $SECRET_NAME: The type or name of the detected secret.
- $SECRET_HASH: A hash of the secret’s value, aiding in identifying the specific secret without exposing its actual content.
loki.write “local_loki”:
- Purpose: Sends the sanitized log entries to a Loki instance for storage and subsequent analysis.
Configuration:
- endpoint: Contains the URL of the Loki server where the logs will be sent. Replace “<LOKI_ENDPOINT>” with the actual endpoint of your Loki instance.
Conclusion
This feature of Grafana Alloy will improve business compliance to regulatory requirements. It is quite easy to implement as it does not require so much configurations or changes to both Alloy and Loki only to Alloy. This will further increase the security of logs viewed by developers in an organization to avoid engineers from viewing sensitive credentials on the logs of an application.
