How to Monitor OpenSearch in K8s

Overview

OpenSearch is seeing increasing adoption, but its ecosystem is not yet fully mature.

For the following scenario:

• Monitoring containerized OpenSearch or OpenSearch running in K8s

I checked and found that the official project does not yet provide a comprehensive solution.

This article explains how to monitor OpenSearch in K8s, covering the full pipeline from installing the exporter plugin to metrics collection and visualization.

Introduction to OpenSearch

• OpenSearch is an open-source distributed search engine (forked from a specific version of Elasticsearch) that supports use cases such as fast, scalable full-text search, application and infrastructure monitoring, security and event information management, and operational health tracking.
• OpenSearch offers a variety of features and plugins to help you index, secure, monitor, and analyze your data.
• OpenSearch includes a demo configuration so you can get up and running quickly, but before using OpenSearch in production, you must manually configure the security plugin with your own certificates, authentication methods, users, and passwords.
• OpenSearch is backed by AWS, and all components are available on GitHub under the Apache License Version 2.0.

Introduction to Prometheus Exporter Plugin for OpenSearch

• The Prometheus Exporter plugin exposes OpenSearch metrics in Prometheus format.
• The plugin version must exactly match the OpenSearch version, so you need to keep the prometheus-exporter-plugin-for-opensearch version in sync with the OpenSearch version.
• The plugin can be installed on each OpenSearch node that you want Prometheus to scrape.
• The plugin can be configured via static and dynamic settings in config/opensearch.yml.
• Metrics are available directly at http(s)://:9200/_prometheus/metrics.

📚️ References

This article uses 2 resources:

• OpenSearch
• Prometheus Exporter Plugin for OpenSearch

Implementation

Two approaches:

1. Build a custom image that includes the prometheus-exporter plugin
2. Install the prometheus-exporter plugin via the OpenSearch Helm Chart

(Approach 1) Build and Use a Custom Image with the prometheus-exporter Plugin

│ 📝Notes:
│
│ This example uses opensearch:2.12 as the version

Dockerfile contents:

FROM opensearchproject/opensearch:2.12.0
LABEL maintainer="cuikaidong@foxmail.com"
ARG EXPORTER_PLUGIN_URL="https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch/releases/download/2.12.0.0/prometheus-exporter-2.12.0.0.zip"
RUN opensearch-plugin install -b ${EXPORTER_PLUGIN_URL}

│ 📝Notes
│
│ If the download times out during docker build, you can replace the EXPORTER_PLUGIN_URL line with a proxy URL (not detailed here).
│ Alternatively, download the file first, then COPY it into the image and run:
│ opensearch-plugin install -b file:///path/to/prometheus-exporter-2.12.0.0.zip

Build and push the image:

docker build -t xxxxx/opensearch:2.12.0-prometheus-exporter -f ./Dockerfile
docker push xxxx/opensearch:2.12.0-prometheus-exporter

│ 📝Notes
│
│ You can use a CI/CD pipeline to automatically build new images as OpenSearch and prometheus-exporter-plugin-for-opensearch are updated.
│ I believe that as the OpenSearch ecosystem matures, there will be OpenSearch images that already include the exporter.

For containerized or K8s-based OpenSearch, simply replace the image with the custom-built one that includes the prometheus-exporter.

For example:

Originally:

image: opensearchproject/opensearch:2.12.0

Change to:

image: xxxx/opensearch:2.12.0-prometheus-exporter

(Approach 2) Use the OpenSearch Helm Chart

If you are running OpenSearch in K8s, you can also use the OpenSearch Helm Chart, which includes the ability to install third-party plugins. The relevant values.yaml is as follows:

## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
plugins:
  enabled: true
  installList:
    - https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch/releases/download/2.12.0.0/prometheus-exporter-2.12.0.0.zip

│ 📚️Reference:
│
│ OpenSearch Helm Chart

Modify the prometheus-exporter Configuration

Additionally, you can modify the prometheus-exporter configuration as needed. For detailed configuration instructions, see:

• prometheus-exporter-plugin-for-opensearch config

Example configuration:

Append the following to config/opensearch.yml:

plugins.security.disabled: true
prometheus.indices_filter.selected_indices: "log-*,*log,*log*,log*-test"
prometheus.indices_filter.selected_option: "STRICT_EXPAND_OPEN_FORBID_CLOSED"

│ 📝Disclaimer
│
│ plugins.security.disabled: true is optional and allows accessing the plugin URL over HTTP. Not recommended for production. Use only for quick verification.
│
│ prometheus.indices_filter.selected_indices is provided as a reference only. Adjust as needed.
│
│ prometheus.indices_filter.selected_option uses the default configuration. Please read the details and adjust as needed.

After modifying the configuration, restart the container for the changes to take effect.

Verify the Plugin Is Enabled

Metrics are available directly at:

http(s)://opensearch-host:9200/_prometheus/metrics

As an example, you will get output like the following:

# HELP opensearch_process_mem_total_virtual_bytes Memory used by ES process
# TYPE opensearch_process_mem_total_virtual_bytes gauge
opensearch_process_mem_total_virtual_bytes{cluster="develop",node="develop01",} 3.626733568E9
# HELP opensearch_indices_indexing_is_throttled_bool Is indexing throttling ?
# TYPE opensearch_indices_indexing_is_throttled_bool gauge
opensearch_indices_indexing_is_throttled_bool{cluster="develop",node="develop01",} 0.0
# HELP opensearch_jvm_gc_collection_time_seconds Time spent for GC collections
# TYPE opensearch_jvm_gc_collection_time_seconds counter
opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="old",} 0.0
opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="young",} 0.0
# HELP opensearch_indices_requestcache_memory_size_bytes Memory used for request cache
# TYPE opensearch_indices_requestcache_memory_size_bytes gauge
opensearch_indices_requestcache_memory_size_bytes{cluster="develop",node="develop01",} 0.0
# HELP opensearch_indices_search_open_contexts_number Number of search open contexts
# TYPE opensearch_indices_search_open_contexts_number gauge
opensearch_indices_search_open_contexts_number{cluster="develop",node="develop01",} 0.0
# HELP opensearch_jvm_mem_nonheap_used_bytes Memory used apart from heap
# TYPE opensearch_jvm_mem_nonheap_used_bytes gauge
opensearch_jvm_mem_nonheap_used_bytes{cluster="develop",node="develop01",} 5.5302736E7
...

Collect Metrics with Prometheus

(This is a reference example only; adjust as needed.) Append the following under the scrape section in your Prometheus configuration:

- job_name: opensearch
  metrics_path: /_prometheus/metrics
  relabel_configs:
    - replacement: '<your-instance-name>'
      target_label: node
  static_configs:
    - targets: ['<your-host-name>:9200']

Configure Prometheus Rules and Alerts

Here is a simple example. If you are already using OpenSearch, you likely have existing ES-related rules and alerts that can be adapted with minor modifications.

alert: OpenSearchYellowCluster
for: 5m
annotations:
  summary: At least one of the clusters is reporting a yellow status.
  description: '{{$labels.cluster}} health status is yellow over the last 5 minutes'
  runbook_url: ''
labels:
  severity: warning
  '': ''
expr: |
  opensearch_cluster_status == 1

View in Grafana

You can use the following Grafana Dashboard:

• https://grafana.com/grafana/dashboards/20827-opensearch/

The result looks like this:

More OpenSearch dashboards can be found by searching for the keyword “OpenSearch” at https://grafana.com/grafana/dashboards/.

Summary

How to monitor containerized or K8s-based OpenSearch?

1. First, install the OpenSearch Prometheus Exporter plugin using one of 2 methods:
   1. Build a custom image that includes the OpenSearch Prometheus Exporter plugin
   2. Install via the OpenSearch Helm Chart
2. Configure Prometheus scrape config
3. Configure Prometheus Rules and Alerts
4. View in Grafana

That’s it.