Node IPv6 Prefix Drift: The Silent Cross-Node Pod Network Failure Behind a "Just an Alert"
This article was last updated on: November 9, 2026 pm
Introduction
During the previous National Day holiday I upgraded the homelab to IPv4+IPv6 dual stack, and recently I updated and restarted the 4 nodes again. The result: a kubelet alert, repeating every 10 seconds.
1 | |
The first reaction is usually that the configuration is out of sync — just restart k3s and be done with it.
What the evidence actually showed was far more serious: cross-node IPv6 pod communication was already broken; the business side simply hadn’t reported it.
Between the alert and the real blast radius, there is an entire layer of difference.
Symptoms
Time: 2026-10-29.
During the same round of restarts, the ISP re-delegated the IPv6 prefix:
- Old:
240e:xxxxxxxx:ae43::/64 - New:
240e:xxxxxxxx:6440::/64
The cluster is a 4-node k3s dual stack, Ubuntu 26.04, kernel 7.0.0-42.
The node-ip in /etc/rancher/k3s/config.yaml contained the old GUA as a literal.
After the new prefix was delegated, that address no longer existed on any interface.
kubelet validates secondaryNodeIP every 10 seconds; on failure, the related fields are simply not written to Node status.
The Node object’s IPv6 InternalIP stayed on the old address.
Root Cause
The node’s IPv6 identity was bound to the GUA literal handed out by the ISP.
The GUA comes from the carrier, and when the prefix rotates is entirely up to the ISP.
On home broadband, prefix rotation is the norm, not an anomaly.
Hard-coding node identity to an address the carrier can change at any time is a flawed design in itself.
Evidence: The Real Impact Beyond the Alert
The alert is only the entry point; the thing that actually matters is Cilium on the network side.
The Cilium agent reads node addresses at startup and writes them into the CiliumNode object.
Once the old address was written in, auto-direct-node-routes could not generate cross-node IPv6 direct routes.
Before the fix, ip -6 route contained only the local node’s pod /64 — not a single cross-node route.
Cross-node IPv6 pod communication was effectively broken.
The IPv6 EndpointSlices for pods on the cluster (semaphore, mysql) pointed to an address that no longer existed.
📝Notes:
Failures like this don’t trigger business alerts. The pod itself is Running; what’s broken is the cross-node path.
Design Trade-offs
Four directions laid out for comparison:
| Option | Approach | Cost |
|---|---|---|
| A | Switch node IPv6 to a site ULA; keep the GUA on the interface only for egress / NAT | One-time change, no longer drifts with the ISP |
| B | Recompute the GUA on every start via systemd ExecStartPre |
Restart on every drift; Node / Endpoint all change |
| C | Request a static prefix from the ISP | Cleanest, but usually unobtainable on home broadband |
| D | Fall back to IPv4 single stack | Conflicts with the existing dual-stack spec; pods must be recreated |
Option A was chosen.
My reasoning: node identity only needs to be stable and reachable within the site; it does not need to be tied to the carrier’s prefix.
The GUA stays on the interface, egress and NAT work as usual.
The pod /56 and Service /112 subnets are unchanged, so the business side notices nothing.
The chosen ULA is fdb5:xxxxxxxx:1::/64, with host bits ::11 through ::14:
::11→ n100-jumper-0::12→ n100-jumper-1::13→ n100-jumper-2::14→ n100-cheshi-0
Implementation: A Few Changes
- Edit
/etc/netplan/60-homelab-ula.yamlto write the fixed, pre-planned ULA IPv6 address, then runnetplan apply
The YAML content:
1 | |
The IPv4 address and default route are both preserved; there is no need to rewrite the cloud-init-generated 50-cloud-init.yaml. An example of that file’s YAML content:
1 | |
📝Notes:
netplan merges multiple files sorted by filename;
60-homelab-ula.yamlsorts after50-cloud-init.yaml.
Making It Take Effect: Restarting k3s Is Not Enough
Restarting k3s without restarting the cilium agent: this is the easiest step to miss in the whole remediation.
After editing config.yaml and restarting k3s, the Node object’s InternalIP updates. But cilium-dbg node list still shows the old address. The cross-node routes are still missing. The Cilium agent only reads Node addresses at startup and writes them into CiliumNode; when the Node updates, it does not re-read them on its own. Each cilium agent must be restarted individually.
After the restart, the IPv6 column in cilium-dbg node list shows each node’s own ULA, and ip -6 route contains direct routes of the form via <peer ULA>.
📝Notes:
Restart one node at a time; don’t stop them all at once. While an agent is stopped, that node’s pod network is briefly interrupted.
Verification
Production acceptance covered 6 dimensions:
- The IPv6 InternalIP of all 4 Nodes is the planned ULA, status Ready
- The
secondaryNodeIPcount is 0 - The IPv6 column in
cilium-dbg node listshows each node’s own ULA - 3 cross-node direct routes appear
- Cross-node pod IPv6 connectivity works; the three business namespaces are Running and HTTP is available
TestNodeAddressingPASS
A read-only check was also performed on the router side: the LAN is a br-lan bridge with no ULA configuration and no route or neighbor entry for that /64. Inter-node IPv6 uses Layer 2 forwarding, so the router needs no changes.
Summary
Lessons learned from this incident:
- Node IPv6 identity should not be bound to the ISP-delegated GUA; reachability within the site is enough
- An alert’s severity and its actual blast radius do not necessarily match — gather the evidence yourself
- The Cilium agent reads Node addresses only at startup; Node updates do not propagate to it automatically
- netplan’s
addressesacross multiple files is append semantics, so a standalone file can carry just the ULA
Beaten and battered a thousand times, still it stands firm; let the winds blow from east, west, south, or north.
IPv6 GUA prefixes will drift; the architecture should not drift with them.
That’s all.