Grafana Series - Unified Dashboard - 11 - Seamless Navigation from Logs to Traces

Series Articles

• Grafana Series Articles

Overview

As described in the previous article Grafana Series - Unified Dashboard -1- Introduction, understanding all related data—and the relationships between them—is critical for quickly resolving incidents and identifying the true root cause of unexpected system behavior. Grafana enables teams to seamlessly visualize and navigate across all their data in one place.

The most typical example is Grafana Labs’ LGTM stack, which includes:

• Loki (Logging)
• Grafana (Visualization)
• Tempo (Tracing)
• Mimir (Metrics)

Through the following technical details, seamless visualization and navigation across Logging, Tracing, and Metrics can be achieved:

• Metrics -> Logs: Based on service discovery and unified labels
• Logs -> Metrics: Based on extracting metrics via LogQL
• Logs -> Traces: Based on derived fields or automated logging
• Traces -> Logs: Based on labels
• Traces -> Metrics: Based on metrics generated from spans
• Metrics -> Traces: Based on Prometheus Exemplars

The diagram below illustrates this:

Even without adopting Grafana Labs’ solution, a certain degree of seamless navigation can still be achieved.

For example:

• Logging with EFK
• Tracing with Jaeger

If the logs also contain trace_id, you can at least achieve seamless navigation from Logs to Traces via trace_id.

│ 🐾Notes:
│
│ The prerequisite is that the logs contain a trace_id related field.
│ Of course, having trace_id in the logs means the developers have planned the log format well, which greatly simplifies the configuration of navigation logic on the operations side.
│ In most cases, you need to combine fields/labels such as cluster, namespace, app, pod, etc. for querying (rather than locating directly via trace_id).

Hands-On: Logs (ElasticSearch) to Traces (Jaeger)

│ 📝Notes:
│
│ Hands-on scenario: Locate the corresponding Trace for an error log entry.

First, update the ES data source configuration to add Data links related settings:

• Field: trace_id (configure based on your actual setup; it could also be traceId, etc.)
• Internal link: ✔️, and select a Trace (Jaeger) data source
• Query: ${__value.raw}

After that, you can use the ES Log Quick Search Dashboard to filter ERROR logs. Click to expand the details, and you’ll find that trace_id now has a navigation link, as shown below:

Clicking it will navigate directly to the Explore -> Jaeger page with the error trace_id, displaying both the service topology and the trace waterfall view, as shown below:

🎉🎉🎉

Summary

At this point, we have completed a very simple one-way navigation from Logs (ElasticSearch) to Traces (Jaeger). Even this alone can greatly facilitate troubleshooting and problem analysis.

To achieve bidirectional navigation across Metrics, Logs, and Traces, specific solutions are required for all three, along with more fine-grained configuration across data sources, Dashboards, and Queries. This includes but is not limited to:

• The cross-linking configuration between the three as mentioned above
• Configuring Data links, Derived fields, Exemplars, Trace to logs, Trace to Metrics, etc. on each data source
• Adding Text panels or similar to Dashboards, combined with Variables for more user-friendly navigation
• Configuring panel links for more user-friendly navigation
• …

This article serves as a starting point—hopefully it inspires you to build more polished and practical unified monitoring dashboards with seamless navigation.