Standardizing Global Large-Scale Hybrid Cloud Kubernetes Prometheus Monitoring with GitOps Automation

Background

Current State

1. Company overview:
   1. A PaaS/SaaS company with global operations spanning Southeast Asia, South Asia, Middle East, Europe, Africa, Americas, East Asia, etc.
   2. Dozens of production Kubernetes clusters, over 100 clusters total including non-production (multiple cluster types across various public clouds, dedicated clouds, private clouds, data centers, etc.)
   3. Continuous cost optimization efforts since the pandemic.
2. Monitoring overview — due to historical reasons and cost considerations:
   1. Built on deeply customized native Prometheus with some self-developed exporters/service discovery; not using kube-prometheus-stack (incompatible, would increase costs)
   2. Monitoring coverage: Kubernetes/pods/various middleware/microservices/URLs, etc.
   3. One Prometheus monitoring stack per cluster
   4. Compute and storage resources allocated to monitoring are constrained
   5. Deployment method: Ansible for initial monitoring component installation, followed by Jenkins DevOps CI/CD for automated releases

In summary, the monitoring system can be characterized as:

1. Global
2. Large-scale
3. Hybrid cloud
4. Kubernetes-native
5. Low-cost

Problem

A recent incident where insufficient monitoring coverage (specifically, a cluster missing URL monitoring configuration) led to missed alerts prompted a thorough post-mortem. The core issues can be summarized as:

1. Lack of a single source of truth — monitoring configurations are scattered across clusters, leading to version inconsistencies and rule omissions;
2. Configuration drift from manual operations — unable to synchronize global cluster state in real time, limiting fault detection capabilities.

To prevent recurrence, the following improvement plan was developed:

Adopt a standardized monitoring architecture centered on GitOps (Git as the single source of truth) + Prometheus Operator, detailed as follows:

1. Root Causes and Improvement Direction

1. Current Challenges

   • Fragmented management: Prometheus monitoring configurations across hundreds of global clusters still partially rely on manual maintenance, prone to rule omissions and inconsistent thresholds.
   • Manual management risks: Manual management of monitoring components, configurations, and thresholds carries risks of stale or misconfigured settings (as seen in the recent incident).
   • Monitoring data noise: Inconsistent configurations lead to frequent false positives/missed alerts, impacting incident response efficiency.

2. Target Architecture

   • Single Source of Truth: Manage all monitoring configurations (Prometheus rules, ServiceMonitors, AlertManager, etc.) through a Git repository, eliminating manual intervention.
   • GitOps automated reconciliation and self-healing: Leverage ArgoCD and related GitOps tooling for real-time configuration synchronization, ensuring cluster state matches Git declarations.
   • Centralized observability: Standardize deployments via Prometheus Operator; if needed, consider integrating Thanos/Cortex/Mimir for cross-cluster monitoring data aggregation in the future.

2. Technical Implementation Path

1. GitOps (Git as the Single Source of Truth) Standardized Workflow
   • GitOps: Store all monitoring resources (Prometheus CRDs, Grafana dashboards) in a Git repository with version control and code review mechanisms ensuring change traceability.
   • Automated reconciliation: Use ArgoCD and related GitOps tooling to watch for Git repository changes and automatically push them to clusters, preventing manual errors (referencing Red Hat OpenShift GitOps best practices).
   • Emergency fix process: All production changes must go through Git commits; only the Git repository serves as the modification entry point, eliminating “temporary patches.”
2. Prometheus Operator Enhanced Capabilities
   • Unified deployment templates: Use Helm Charts to package the Prometheus Stack (AlertManager, BlackBox Exporter, etc.), ensuring consistent versions and configurations across clusters.
   • Dynamic service discovery: Automatically discover microservice endpoints via ServiceMonitor, avoiding omissions caused by manually adding Exporters.

3. Expected Benefits

1. Reduced operational risk: Configuration drift reduced by over 90%, with fully automated management of monitoring components, thresholds, and configurations.
2. Improved incident response: Centralized alert views and standardized rules reduce MTTD (Mean Time to Detect) by 50%.
3. (TBD) Cost optimization: Avoid redundant monitoring component development, improve resource utilization by 30% (through Prometheus federation cluster optimization for data storage, e.g., Thanos/Cortex/Mimir).

4. Next Steps

1. Pilot phase: Plan to set up a temporary environment for a period of PoC validation, producing standardized templates and automation pipelines.
2. Global rollout:
   1. Build a dedicated monitoring management cluster.
   2. Phased migration to the GitOps (Git as the single source of truth) + Prometheus Operator architecture; given the scale, this is expected to require sustained investment.
3. Training and collaboration: Organize internal team knowledge-sharing sessions to align on GitOps (Git as the single source of truth) + Prometheus Operator collaboration standards (branching strategies, project structure strategies, review processes, etc.).

📚️ References

• OpenShift GitOps Recommended Practices | Red Hat Developer
• Lightning Talk: Best Practices on Organizing GitOps Repositories - Konstantinos Kapelonis, Codefresh