Grafana Series - Unified Display - 3 - Prometheus Dashboards

Series Articles

• Grafana Series Articles

Background Knowledge

Prometheus Template Variables

You can use variables instead of hard-coded details such as server, app, and pod_name in metric queries. Grafana lists these variables in dropdown select boxes at the top of the dashboard, allowing you to change the data displayed in the dashboard. Grafana refers to these as template variables.

Query Variables

Use Query type variables to query Prometheus for a list of metrics, labels, or label values.

Select a Prometheus data source query type and enter the required input:

Variable Syntax

The Prometheus data source supports two variable syntaxes in the Query field:

• $, e.g.: rate(http_requests_total{job=~“$job”}[$_rate_interval]) — this is easier to read and write, but does not allow you to use a variable in the middle of a word. In most cases, this syntax is sufficient.
• [[varname]], e.g.: rate(http_requests_total{job=~“[[job]]”}[$_rate_interval])

If you enable the Multi-value or Include all option, Grafana converts labels from plain text to regex-compatible strings, which requires you to use =~ instead of =. Therefore, when using PromQL with Grafana variables, it is recommended to use =~.

Annotations

Annotations overlay rich event information on top of graphs. You can add annotation queries in the Annotations view of the Dashboard menu.

Prometheus supports two ways to query annotations:

• A regular metric query
• Prometheus pending and firing alerts (see Inspecting Alerts During Runtime), for example: ALERTS{alertname=“”, alertstate=“”, }

The Step option is useful for limiting the number of events returned from your query.

Ad Hoc Filters Variable

Prometheus supports a special ad hoc filters variable type that you can use to temporarily specify any number of label/value filters. These filters are automatically applied to all your Prometheus queries.

Hands-On

Let’s pick a dashboard directly from Grafana Dashboards — JVM(Micrometer) (ID: 4701) — for demonstration purposes.

│ 📝Notes:
│
│ This is one of the huge benefits of using Grafana — its rich ecosystem.
│ We generally don’t need to build dashboards from scratch; we can stand on the shoulders of giants and make partial adjustments.

The final result looks like this:

Detail Optimizations

Here are some small dashboard optimization tips to share:

• For variables, be cautious when enabling the Multi-value or Include all options to avoid querying excessive amounts of data.
• For a single dashboard with a large number of panels, you can add Rows based on panel type and collapse some Rows. This enables lazy loading, reduces query volume, and improves user experience.
• For the time range selector in the upper right corner, keep it as small as reasonably possible. If the query volume is particularly large, consider disabling auto refresh.

Configuring JVM Restart Annotation

In this dashboard, you can configure an Annotation for JVM Restart. Here is a configuration example:

• Name: Restart Detection
• Data source: Prometheus
• Enabled: ✔️
• Color: Red
• Query
  • Expr: resets(process_uptime_seconds{app=“$app”, pod_name=“$pod_name”}[1m]) > 0 — this is based on process_uptime_seconds.
  • Step: 1m
• Field formats — the title and text fields can use variables, e.g.:
  • Title: Restart
  • Tags: restart-tag
  • Text: uptime reset

The final result looks like this (JVM Restart has not been reproduced here), demonstrated using Grafana Play’s demo:

Variables

Label Value

The specific configuration is as follows. For example, to select based on the value of app, configure it like this:

Name: app
Type: Query
Query: label_values(app)

If you want to add another variable pod_name that filters based on the app result, configure it as follows:

Name: pod_name
Query: label_values(jvm_memory_used_bytes{app=“$app”}, pod_name)

If you need to adjust variables according to your actual situation — for example, changing the pod_name variable to an instance variable — the instance variable would be:

Name: instance
Query: label_values(jvm_memory_used_bytes{app=“$app”}, instance)

You can then go to the dashboard’s Settings -> JSON Model and batch replace pod_name and pod_name="$pod_name" with instance and instance="$instance".

Interval

You can also set a time interval (Interval) as a variable. The configuration is as follows:

• Name: interval
• Type: Interval
• Values: different time intervals separated by , e.g.: 5s,10s,30s,1m,10m,30m,1h,6h,12h,1d,7d,14d,30d

Queries within the dashboard can then use $__interval or $__interval_ms.