Grafana Series - Unified Display - 2 - Prometheus Data Source

Series Articles

• Grafana Series Articles

Grafana Prometheus Data Source

Grafana ships with built-in support for Prometheus. This article covers some of the options, variables, queries, and other features specific to the Grafana Prometheus data source (including Prometheus-compatible implementations such as Thanos, Mimir, etc.).

Configuring the Prometheus Data Source

Here are a few important settings worth highlighting:

• URL: The URL of the Prometheus Server, e.g. http://prometheus.example.org:9090 (no path or other information needed)
• Manage alerts via Alerting UI: When enabled, this integrates with AlertManager, allowing Grafana to display and manage AlertManager. Once enabled, you can select the corresponding Prometheus AlertManager data source.
  • Alertmanager data source: A dropdown to select the corresponding Prometheus AlertManager data source.
• Type: The type of Prometheus Server; options include Prometheus, Cortex, Thanos, and Mimir. When selected, the version field will attempt to auto-populate using the Prometheus buildinfo API. Some Prometheus types, such as Cortex, do not support this API and must be filled in manually.
• Disable metrics lookup: Checking this option disables the metric selector and metric/label support in the query field autocomplete. This can help if you are experiencing performance issues with a larger Prometheus instance.
• Custom query parameters: Add custom parameters to the Prometheus query URL. For example, timeout, partial_response, dedup, or max_source_resolution. Multiple parameters should be concatenated with &.
  • e.g. max_source_resolution=5m&timeout=10

Configuring Exemplars

│ 📝Note:
│
│ Available in Prometheus v2.26+ and Grafana v7.4+.
│ As described in the previous article, Exemplars are used for seamless navigation from Metrics to Traces.

Grafana 7.4 and later can display Exemplars data alongside metrics in both “Explore” and “Dashboards”. Exemplars data links higher-cardinality metadata from specific events to traditional time series data.

• Internal link: Enable this option if you have an internal link. When enabled, a data source selector will appear. Select the backend tracing data store for your exemplar data, such as the Tempo or Jaeger data source within Grafana.
• Data source: (Only visible when “Internal link” is enabled) Select the backend tracing data store for your exemplar data.
• URL: (Only visible when “Internal link” is disabled) Define the full URL for the external link. You can interpolate from this field using ${__value.raw}.
• Label name: Add a name for the traceID attribute of the exemplars.

Prometheus YAML Configuration Example

A typical example is as follows (based on Grafana 9):

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # Access mode - proxy (server in the UI) or direct (browser in the UI).
    access: proxy
    url: http://localhost:9090
    jsonData:
      httpMethod: POST
      manageAlerts: true
      prometheusType: Prometheus
      prometheusVersion: 2.37.0
      cacheLevel: 'High'
      exemplarTraceIdDestinations:
        # Field with internal link pointing to data source in Grafana.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          name: traceID

        # Field with external link.
        - name: traceID
          url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D'

EOF