How to Include Raw Log Content in Alert Annotations When Configuring Alerts with Loki

When configuring alerts with Loki, the current best practice for including raw log content in alert annotations is to avoid embedding complete log lines directly in alert notifications. This is because Loki alerting operates on time-series data derived from logs, not the raw log lines themselves. Including highly dynamic log content in alert labels or annotations can lead to high-cardinality issues, resulting in a large number of unique alerts and potential performance problems.

Instead, the recommended approaches are:

• Use annotations such as summary or description to clearly explain the alert, and include a Grafana Explore link pre-filled with a LogQL query when needed. This allows responders to quickly access the relevant log lines without embedding them directly in the alert message.
• If you must include log content, you can use Promtail pipelines to extract specific fields from logs as labels, but you should be very careful to avoid high-cardinality data.

Here is a summary of this guidance:

│ … Create your alert, write a good summary, write a good description, and then link to a LogQL query in Grafana Explore where you can view the logs. If you really, really need to, you can try extracting parts of the log line as labels, but you have to be very careful with highly dynamic data because you’ll end up with hundreds or thousands of alerts. … My recommendation today is to try to avoid putting log lines into alerts, and that’s exactly why.
│
│ Grafana Loki: Best Practices for Recording and Alerting Rules (Loki Community Call February 2025)

Currently, there is no built-in, safe way to automatically include complete raw log content in Loki alert annotations. The recommended workflow is to provide context and link to the relevant logs rather than embedding the logs themselves. This approach is both scalable and user-friendly.