Grafana Series - Unified Display - 8 - ElasticSearch Log Quick Search Dashboard

Series Articles

• Grafana Series Articles

Overview

We created a similar log quick search dashboard based on ElasticSearch, inspired by this article: Grafana Series (Part 12): How to Create a Grafana Dashboard for Searching Logs Using Loki.

The final result looks like this:

│ 📝Notes:
│
│ I actually created 2 dashboards based on ElasticSearch:
│
│ * One for searching Applog
│ * One for searching accesslog
│
│ The explanation below covers both dashboards.

This time we won’t go into every detail — only selected key points will be covered.

Prerequisites

Creating Queries

Use custom JSON strings to write queries where the field is mapped as a keyword in the Elasticsearch index mapping.

If the query involves a multi-field with both text and keyword types, use “field”: “fieldname.keyword” (sometimes fieldname.raw) to specify the keyword field in your query.

Query

Query	Description
{“find”: “fields”, “type”: “keyword”}	Returns a list of field names with index type keyword.
{“find”: “terms”, “field”: “hostname.keyword”, “size”: 1000}	Returns a list of keyword values using terms aggregation. The query uses the current dashboard time range as the time range query.
{“find”: “terms”, “field”: “hostname”, “query”: ‘’}	Returns a list of keyword field values using terms aggregation with the specified Lucene query filter. The query uses the current dashboard time range as the query time range.

The terms query has a default limit of 500 results. To set a custom limit, set the size property in your query.

Variable Syntax

Panel titles and metric queries can reference variables using several different syntaxes:

• $varname — This syntax is easy to read, but it doesn’t allow you to use a variable in the middle of a word. Example: apps.frontend.$server.requests.count
• ${var_name} — Use this syntax when you want to interpolate a variable in the middle of an expression.
• ${var_name:} — This format gives you more control over how Grafana interpolates values.
• [[varname]] — Not recommended. Deprecated old syntax that will be removed in a future release.

Advanced Variable Format Options

The format for variable interpolation depends on the data source, but in some cases you may want to change the default format.

For example, the default format for the MySQL data source joins multiple values in a comma-separated, quoted manner, like: ‘server01’, ‘server02’. In some cases, you may want an unquoted comma-separated string, like: server01,server02. You can achieve this with the advanced variable formatting options listed below.

General Syntax

Syntax: ${var_name:option}

You can test formatting options on the Grafana Play site.

If any invalid formatting option is specified, glob is the default/fallback option.

CSV

Formats a multi-value variable into a comma-separated string.

servers = ['test1', 'test2']
String to interpolate: '${servers:csv}'
Interpolation result: 'test1,test2'

Distributed - OpenTSDB

Formats a multi-value variable in OpenTSDB’s custom format.

servers = ['test1', 'test2']
String to interpolate: '${servers:distributed}'
Interpolation result: 'test1,servers=test2'

Double Quote

Formats single-value and multi-value variables into a comma-separated string, escaping " with " within individual values, and wrapping each value in “” quotes.

servers = ['test1', 'test2']
String to interpolate: '${servers:doublequote}'
Interpolation result: '"test1","test2"'

Glob - Graphite

Formats a multi-value variable as a glob (for Graphite queries).

servers = ['test1', 'test2']
String to interpolate: '${servers:glob}'
Interpolation result: '{test1,test2}'

JSON

Formats a multi-value variable into a comma-separated string.

servers = ['test1', 'test2']
String to interpolate: '${servers:json}'
Interpolation result: '["test1", "test2"]'

Lucene - Elasticsearch

Formats a multi-value variable in Lucene format for Elasticsearch.

servers = ['test1', 'test2']
String to interpolate: '${servers:lucene}'
Interpolation result: '("test1" OR "test2")'

URL Encode (Percentencode)

Formats single-value and multi-value variables for use in URL parameters.

servers = ['foo()bar BAZ', 'test2']
String to interpolate: '${servers:percentencode}'
Interpolation result: 'foo%28%29bar%20BAZ%2Ctest2'

Pipe

Formats a multi-value variable into a pipe-separated string.

servers = ['test1.', 'test2']
String to interpolate: '${servers:pipe}'
Interpolation result: 'test1.|test2'

Raw

Turns off data source-specific formatting, such as single quoting in SQL queries.

servers = ['test.1', 'test2']
String to interpolate: '${var_name:raw}'
Interpolation result: 'test.1,test2'

Regex

Formats a multi-value variable into a regex string.

servers = ['test1.', 'test2']
String to interpolate: '${servers:regex}'
Interpolation result: '(test1\.|test2)'

Single Quote

Formats single-value and multi-value variables into a comma-separated string, escaping ’ with ' within individual values, and wrapping each value in ’ quotes.

servers = ['test1', 'test2']
String to interpolate: '${servers:singlequote}'
Interpolation result: "'test1','test2'"

Sqlstring

Formats single-value and multi-value variables into a comma-separated string, escaping ’ with ‘’ within each value, and wrapping each value in ’ quotes.

servers = ["test'1", "test2"]
String to interpolate: '${servers:sqlstring}'
Interpolation result: "'test''1','test2'"

Text

Converts single-value and multi-value variables to their text representation. For a single variable, it returns just the text representation. For multi-value variables, it returns the text representations joined with +.

servers = ["test1", "test2"]
String to interpolate: '${servers:text}'
Interpolation result: "test1 + test2"

Query Parameters

Formats single-value and multi-value variables into their query parameter representation. Example: var-foo=value1&var-foo=value2

servers = ["test1", "test2"]
String to interpolate: '${servers:queryparam}'
Interpolation result: "servers=test1&servers=test2"

Configuring Variable Selection Options

Selection Options is a feature you can use to manage variable option selections. All selection options are optional and are turned off by default.

Multi-value Variables

Interpolating a variable with multiple selected values is tricky because it’s not straightforward how to format multiple values into a string that is valid in the given context where the variable is used. Grafana tries to solve this by allowing each data source plugin to inform the templating interpolation engine what format to use for multiple values.

│ 📝Notes:
│
│ The Custom all value option on the variable must be empty for Grafana to format all values into a single string. If left empty, Grafana concatenates (joins together) all values in the query, similar to value1,value2,value3. If a custom all value is used, the value will be something like * or all.

Multi-value Variables with Prometheus or InfluxDB Data Sources

InfluxDB and Prometheus use regex expressions, so the variables host1, host2, host3 would be interpolated as {host1,host2,host3}. Each value is regex-escaped.

Multi-value Variables with Elastic Data Source

Elasticsearch uses Lucene query syntax, so the same variable would be formatted as (“host1” OR “host2” OR “host3”). In this case, each value must be escaped so that it only contains Lucene control words and quotes.

Include All Option

Grafana adds an All option to the variable dropdown. If the user selects this option, all variable options are selected.

Custom All Value

This option is only visible when Include All option is selected.

You can enter regex, globs, or Lucene syntax in the Custom all value field to define the value of the All option.

By default, the All value includes all options in a combined expression. This can become very long and cause performance issues. Sometimes it may be better to specify a custom all value, such as a wildcard.

To have custom regex, globs, or Lucene syntax in the Custom all value option, it is never escaped, so you will need to consider what constitutes a valid value for your data source.

ElasticSearch Template Variables

Choosing a Variable Syntax

As mentioned above, the Elasticsearch data source supports multiple variable syntaxes in the query field.

When Multi-value or Include all value options are enabled, Grafana converts the tags from plain text to Lucene-compatible conditions. That is, it implicitly converts $varname to ${varname:lucene}.

Hands-on

1. Identify the Index Fields

First and foremost, you need to identify which fields the index has and which keywords are available, then select some fields and keywords as variables. You can query and experiment directly through the Kibana interface.

For this example, the selected fields are:

• app_name
• level
• request_path (🐾 Through repeated use in Kibana, I found that request_path.keyword should be used instead of request_path when querying)
• request_method
• status_code

2. Create Variables

app_name

Configuration:

• Name: app_name
• Type: Query
• Data source: ES
• Query: {“find”: “terms”, “field”: “current_app_name”}. Additionally, for nested usage, you can use something like {“find”: “terms”, “field”: “pod_name”, “query”: “app_name:$app_name”}

request_path

Configuration:

• Name: request_path
• Type: Query
• Data source: ES
• Query: {“find”: “terms”, “field”: “request_path.keyword”, “query”: “app_name:$app_name”}
• Multi-value: ✔️
• Include All option: ✔️
• Custom all value: *

🐾 Note that Custom all value is used here, so the final Query All expression becomes: request_path.keyword:* instead of request_path.keyword:( OR …)

request_method

request_method has only a few common values:

• GET
• POST
• DELETE
• HEAD
• PUT
• PATCH
• OPTIONS

So it can be set as a Custom variable with the following configuration:

• Name: request_method
• Type: Custom
• Values separated by comma: GET,POST,DELETE,HEAD, PUT,PATCH,OPTIONS
• Multi-value: ✔️
• Include All option: ✔️
• Custom all value: *

level

Log levels can use the Custom variable type directly:

• Name: level
• Type: Custom
• Values separated by comma: INFO, WARN, ERROR,FATAL
• Multi-value: ✔️
• Include All option: ✔️

If you only care about error logs, the default value for the level variable can be set to have both ERROR and FATAL selected.

status_code

Here, the status_code variable will be used with Lucene’s range syntax [] (inclusive of both start and end numbers), so Custom all value and Variable syntax configuration are involved.

• Name: status_code
• Type: Custom
• Values separated by comma: 200 TO 299, 300 TO 399, 400 TO 499, 500 TO 599
• Include All option: ✔️
• Custom all value: 200 TO 599 (📝Note: This covers all HTTP status codes from 200 to 599)

To use it in a Query later, the usage is:

status_code:[${status_code:raw}]

Using ${status_code:raw} directly, the interpolation becomes:

status_code:[200 TO 299]
status_code:[200 TO 599]

This achieves the expected ES query behavior.

filter

Finally, an Ad hoc filters variable is added to allow users to perform more custom filtering.

• Name: filter
• Type: Ad hoc filters
• Data source: ${datasource}

This will be automatically applied to all queries in the dashboard. A typical use case:

For request_path, you may need to filter out monitoring/health check requests (containing keywords like info, health, metric, etc.), and you can save this filter as the default variable value.

3. Panels

The dashboard consists of only 2 panels:

• Top: Time series — displays a log bar chart with color coding: INFO logs in green, WARN logs in yellow, ERROR and FATAL logs in red.
• Bottom: Logs

Time Series Panel

As shown below:

This can be achieved with the following queries:

app_name:$app_name AND level:($level AND INFO)
app_name:$app_name AND level:($level AND ERROR or FATAL)

The $level AND INFO syntax is a workaround to make the Time series panel update when the level variable changes.

Another important point is that the Metric should be set to Count (log entry count) rather than Logs (actual log content).

Additionally, you need to configure Override -> Color as follows:

Finally, if the bars are too dense, you can adjust the Interval as shown in the 3 Colors Time series panel image to change the time interval. In this example, it’s set to 1m.

Logs Panel

In the Logs panel, you can also make various adjustments based on your actual needs.

As shown below, you can adjust the log display settings:

• Time: Whether to show timestamps
• Unique labels: Whether to add labels to each log entry
• Common labels: Whether to add common labels in the top-left corner of the logs panel for all logs
• Wrap lines
• Prettify JSON: JSON formatting
• Enable log details: Enable viewing log details
• Deduplication: Log deduplication, with the following methods:
  • None: No deduplication
  • Exact: Exact deduplication
  • Numbers: Deduplication that treats different numbers as the same category
  • Signature: Deduplication based on computed signatures
• Order: Sorting order.

Additionally, since ES log details may contain many fields we don’t care about, such as _source, _id, etc., you can use Transform to adjust the output. See the following image:

Summary

This article is one of the key entries in this series. It covers a wealth of practical details, including:

• ES Query
• Variable syntax
  • Variable raw syntax
  • Lucene - Elasticsearch syntax
  • …
• Multi-value Variables
• Include All option
• Custom all value
• Ad hoc filters Variable
• ES Metric Type
  • Count
  • Logs
  • …
• Adjusting query time intervals
• Logs panel settings
• Panel Transform

Hope this helps.