Connecting Tailscale to Prometheus Monitoring

This article was last updated on: November 9, 2026 pm

Background

The National Day holiday is finally here! Staying home tinkering with my HomeLab, K8s, and NAS server cluster feels great! I really got a lot done this holiday — readers who are interested can look forward to a flood of upcoming posts; if you can’t wait, you can go straight to my repo east4ming/homelab2 and browse the PRs and commits.

The homelab2 environment isn’t large: 4 nodes (3 masters + 1 worker), Ubuntu 26.04.1, K3s v1.36.5, and everything that can run is running on it. It’s these 4 machines that I wrapped in a layer of tailscale-operator (plus the native Linux tailscale), using ProxyGroup to funnel both Ingress and egress traffic into the tailnet.

And the problems followed: I had no idea what was happening in that Tailscale layer.

  • Is the tailscaled on the nodes reporting health messages?
  • Are any packets being dropped, and what’s the reason?
  • The proxy Pods are up, but are they actually connected to the control plane?
  • Has DERP secretly switched me to a distant region?

Without metrics, all of this could only be checked by eyeballing tailscale status manually, then SSHing in to investigate once something broke. That obviously wasn’t acceptable. So I took advantage of the free time during the National Day holiday to get Tailscale metrics monitoring implemented.

This post focuses mainly on the monitoring implementation and the trade-offs involved, and lays out the whole pipeline along the way.

📝Disclaimer:

All conclusions in this article are based on hands-on testing in this specific homelab2 environment. Tailscale iterates quickly, so please defer to the source code of the version you’re running.

Final Result

Tailscale Dashboard 1

Tailscale Dashboard 2

Tailscale Prometheus Alertrule

Tailscale Alert 1

Tailscale Alert 2

Overall Design: Two Pipelines, Each Doing Its Job

Let’s start with the conclusion: this solution isn’t “one mechanism to rule them all” — it is explicitly split into two pipelines.

Pipeline Scrape Target Mechanism Target Count
Node side tailscaled on the 4 nodes Ansible-deployed systemd service + Prometheus static scrape 4
Cluster side Proxy Pods managed by ProxyGroup ProxyClass (metrics + ServiceMonitor) + operator auto-discovery 8

The node side doesn’t use ServiceMonitor because tailscaled runs on the host machine, outside K8s; the Prometheus Operator simply can’t see it, so it has to go through static additionalScrapeConfigs (though it could also be based on Prometheus node scrape). The cluster side is the opposite: the proxy Pods are created and destroyed dynamically by the operator, so hardcoding IPs is just digging a pit for yourself — instead, ServiceMonitor must be used so the operator declares them itself.

The boundary between the two pipelines is clear, and that’s a point I want to emphasize.

Node Side: Read-Only Endpoint + Static Scraping

The node-side implementation is very short — essentially a single command wrapped in a systemd service:

1
2
3
4
5
6
7
8
9
10
11
# /etc/systemd/system/tailscale-metrics.service
[Unit]
Description=Tailscale read-only metrics endpoint
After=tailscaled.service

[Service]
ExecStart=/usr/bin/tailscale web --readonly --listen <node-lan-ip>:5252
Restart=always

[Install]
WantedBy=multi-user.target

Two details:

  1. --readonly: without this flag, tailscale web exposes a fully writable control interface. Putting that on a monitoring port is no different from hanging the admin console on the public internet. Read-only mode exposes only metrics and status, giving a clear security boundary.
  2. Bind <node-lan-ip> rather than 0.0.0.0: this keeps the exposure model consistent with the existing node-exporter — reachable only on the internal network, with a dedicated UFW allow rule. I also verified along the way that 100.100.100.100:5252 (Tailscale’s own local API) returns 000, showing that this web endpoint is independent of the tailscaled local API and won’t be hit by accident.

In practice this endpoint returns 12 metric families / 27 series; the ones I actually use are these:

  • tailscaled_advertised_routes / tailscaled_approved_routes — whether routes have been approved, at a glance
  • tailscaled_health_messages{type=...} — health issues reported by tailscaled itself
  • tailscaled_home_derp_region_id — which region the home DERP is in
  • tailscaled_inbound/outbound_bytes_total, ..._packets_total — traffic
  • tailscaled_inbound/outbound_dropped_packets_total{reason=...} — dropped packets, with a reason label; this one is my favorite
  • tailscaled_serve_inbound/outbound_bytes_total — traffic from tailscale serve

On the Prometheus side, you just add a static job in kube-prometheus-stack’s additionalScrapeConfigs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
- job_name: tailscale-nodes
  static_configs:
    - targets: ["192.168.3.226:5252"]
      labels:
        node: n100-jumper-0  # label each node individually for easy identification
    - targets: ["192.168.3.174:5252"]
      labels:
        node: n100-jumper-1
    - targets: ["192.168.3.158:5252"]
      labels:
        node: n100-jumper-2
    - targets: ["192.168.3.154:5252"]
      labels:
        node: n100-cheshi-0
    - targets: ["192.168.3.246:5252"]
      labels:
        node: casey-desktop

📝Notes:

This doesn’t go through node-exporter’s textfile collector. Two reasons: first, I didn’t want to touch node-exporter’s startup parameters and hostPath mounts (changing one thing drags in a whole batch); second, textfile has a staleness problem with file writes — if tailscaled dies but the file is still there, the metrics “lie”. A separate job either scrapes or it doesn’t, and the semantics are clean.

Cluster Side: ProxyClass and the Rejected “Default” Approach

On the cluster side, the idea is to add a dedicated ProxyClass:

1
2
3
4
5
6
7
8
9
apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: tailscale-metrics
spec:
  metrics:
    enable: true
  serviceMonitor:
    enable: true

Then have the two ProxyGroups ingress-proxies (4 replicas, carrying 22 Ingresses) and egress-proxies (4 replicas) explicitly reference it via spec.proxyClass: tailscale-metrics. The operator then creates a <sts>-metrics Service (port 9002) and a same-named ServiceMonitor for each proxy, and ultimately all 8 targets are up=1.

So far so good. The real pitfall is whether to set it as the default ProxyClass.

There’s a proxyConfig.defaultProxyClass in the Helm values, which translates into the operator’s PROXY_DEFAULT_CLASS environment variable. Intuitively this is the least-effort approach: set the default once, and all proxies automatically carry metrics, with no need to modify each ProxyGroup individually.

I almost did exactly that. But after digging through the source code, I found something wrong.

This logic in cmd/k8s-operator/sts.go:1045: for Ingress proxies annotated with tailscale.com/experimental-forward-cluster-traffic-via-ingress, the operator explicitly logs “metrics not supported” and skips them. But reconcileMetricsResources() still dutifully creates a metrics Service and ServiceMonitor for them.

The result: the 8 standalone Ingress proxies with that annotation produce 8 permanently up=0 targets.

And those 8 up=0 targets directly trigger the TargetDown alert that ships with kube-prometheus-stack. I confirmed by testing in the test environment that the alert indeed goes into pending.

🤔 A “more elegant” default configuration in exchange for 8 targets that will never be healthy and a constant stream of false-positive noise. That math doesn’t work out no matter how you do it.

So the final approach is: don’t set a default ProxyClass; instead, have the two ProxyGroups reference it explicitly.

Explicitly Excluded Scope: No Scraping, and No Pretending to Scrape

Continuing through the source code, there are two more upstream limitations that must be explicitly excluded:

  1. Standalone Ingress proxies with the experimental-forward-cluster-traffic-via-ingress annotation (8 of them) — as described above, unsupported.
  2. Standalone egress proxies (10 of them) — in cmd/k8s-operator/metrics_resources.go, egress proxies just return nil. Note that this doesn’t raise an error; it simply has no metrics, a case of “silently no metrics”.

Point 2 is different in nature from point 1: point 1 produces bad up=0 targets, while point 2 simply has no target. So from the “don’t manufacture false positives” standpoint, the egress proxy side is safe.

I also excluded the operator Deployment’s own metrics — tailscaled_* is what we want; the operator’s own Go runtime metrics aren’t very valuable and are pure noise.

Alerts: 3 New + Reusing Existing Ones

I added 3 new alert rules:

  • TailscaleNodeHealthMessages — node-side tailscaled_health_messages is non-zero
  • TailscaleNodeDroppedPackets — abnormal node-side packet drop rate, with a reason dimension
  • TailscaleProxyHealthMessages — health messages from cluster-side proxies

For scrape interruptions, I didn’t define duplicate alerts. kube-prometheus-stack already has TargetDown / KubePodNotReady / KubeStatefulSetReplicasMismatch, which cover it well. Writing another set of exporter-down alerts would just make the same thing fire twice, and that would drive me crazy.

📝Notes:

I’ve always followed this principle: if existing alert semantics can be reused, never define duplicates. More alerts isn’t better — more accurate alerts is better.

Grafana Dashboard: Why I Maintain It Myself

Here’s the research conclusion up front, so you don’t have to dig through it again:

  • There is no dashboard on grafana.com based on the client-side tailscaled_* metrics (api/dashboards?name=tailscale returns no exact matches)
  • 24177 / 24178 look related but depend on the API-based grafana/tailscale-exporter (which requires a Tailscale API Key); the metric names don’t match, so they’re unusable
  • Zydepoint/Tailscale-dashboard does use tailscaled_*, but only from the node perspective, and it isn’t on grafana.com

So the dashboard is maintained in the repo: system/monitoring-system/files/dashboards/tailscale.json, 14 panels, with a sidecar ConfigMap and the uid set to tailscale. Both the node perspective and the proxy perspective have data.

It’s not that I’m in love with writing JSON — it’s purely that there was nothing ready-made available, and it’s a single sentence to the AI.

Deployment Paths and That Credential Pitfall

This setup spans three deployment paths, and the boundaries need to be clear:

  1. system/monitoring-system → via ArgoCD
  2. CRs in the tailscale namespace (ProxyClass, ProxyGroup) → via metal/roles/tailscale (the operator itself is managed by Ansible, because Tailscale is installed before ArgoCD.)
  3. Node-side systemd service → reuse the single node-level convergence role prerequisites (the UFW exception lives there too)

Verification

1
2
3
4
5
# end-to-end
curl -s http://<node-ip>:5252/metrics | head   # all 4 return 200
# Prometheus API
# up{job="tailscale-nodes"}  → all 1
# up{job=~"ts_.*"}           → all 1

The success criteria are a few hard indicators: all 4 node targets + 8 proxy targets are UP, with no Tailscale target at up=0; the 3 alert rules are loaded with no persistent false positives; both the Grafana node and proxy panels have data; and port 5252 is read-only and not listening on 0.0.0.0.

There’s one more I care about a lot: IaC + GitOps — that is, re-syncing ArgoCD + re-running Ansible reproduces the entire configuration, with no manual kubectl apply outside of Git. If this one isn’t met, everything above is pointless no matter how nicely it’s done.

Summary

Back to the title. Why did we abandon the default ProxyClass? Because its “elegance” is fake:

  • 👍 Set the default once; all proxies automatically get metrics; minimal configuration
  • 👎 Creates permanently up=0 targets for 8 Ingress proxies with a specific annotation
  • 👎 Directly triggers the existing TargetDown alert, continuously false-positive, polluting the alert channel

Switching to explicit references from the two ProxyGroups costs two extra lines of YAML and buys zero false positives and zero bad targets. I don’t think this trade-off leaves any room for hesitation.

A few principles worth taking away:

  1. The first principle of adding monitoring is “don’t create noise” — a permanently failing target is worse than no monitoring at all, because it trains the on-call person to ignore alerts.
  2. When upstream behavior is in doubt, read the source code; don’t guess. That line in sts.go:1045 is more reliable than any documentation.
  3. Design in layers, each doing its job: things on the host go through static scraping, things in K8s go through ServiceMonitor; don’t force them into one mechanism.
  4. Reuse alerts whenever possible; duplicate definitions only make the same failure fire twice.
  5. Keep credential boundaries engraved in your mind — a command that “looks harmless” can wipe out production credentials.

Knowledge gained from paper is ultimately shallow; only by practicing it yourself can you truly understand it. The docs say “metrics are supported”, but hands-on testing tells you “only some proxies are supported”; that gap in between can only be closed by verifying line by line yourself.

That’s all.

📚️References