k3s + Cilium + Tailscale Dual-Stack Network Overhaul: Evaluation, Experimentation, and Production Rollout

Introduction

The National Day long holiday has finally arrived! Staying home tinkering with my HomeLab, K8s, and NAS server cluster feels great! I really messed around with a lot of things this holiday; interested readers can look forward to a flood of follow-up posts, and if you can’t wait, you can also go straight to the PRs and commits in my repo - east4ming/homelab2.

This should be the last post in the 2026 National Day series.

The motivation was simple: that homelab k3s cluster at home had been running for over two years, always IPv4-only. The ISP gave a /56 IPv6 prefix, static routes were configured on the router, but the cluster just wouldn’t budge. Every time I tried to reach it from outside with curl -6, I had to put a NAT layer in front, and the more I looked at it, the more awkward it felt.

IPv6 is the inevitable trend, so I decided to convert the entire cluster to IPv4/IPv6 dual-stack.

Then I discovered this was far messier than I imagined. The official k3s documentation basically doesn’t endorse “enabling dual-stack in place on an existing IPv4-only cluster.” Note: it’s not “unsupported,” it’s “we don’t guarantee it” — and the difference between those two is huge.

So this article isn’t a success-story tutorial, but a record of a process “from uncertainty to certainty”: how to turn unknown risks into quantifiable, deterministic behavior using local same-version experiments, how to step into pitfalls in production and climb back out, and finally those pitfalls that have nothing to do with the network protocol itself.

📝Disclaimer: This article is based on my own homelab production cluster (4 nodes), not a toy environment. But a homelab is still a homelab, so please don’t copy this directly to your company’s core production cluster; proceed at your own risk.

⚠️Risk Warning

k3s: Does not support enabling dual-stack in place on an existing cluster

https://docs.k3s.io/networking/basic-network-options (Dual-stack section) original text:

Dual-stack networking must be configured when the cluster is first created.
It cannot be enabled on an existing cluster once it has been started as IPv4-only.

Cilium 1.20 likewise has no “single-stack → dual-stack” migration documentation

Searching the Cilium docs:
only 4 files mention dual-stack (BGP×2, LB IPAM, ENI), and there is no dual-stack guide or migration page at all.
The only online IPAM migration is cluster-pool → multi-pool, and it explicitly states “In case of failure, no rollback is available.”
→ Combined with k3s’s prohibition,

Conclusion: In-place dual-stack has no vendor documentation backing it; it can only be treated as “prove it yourself.”

Final Result

Background: What the Cluster Looks Like

Let me set the stage first, otherwise the pitfalls that follow won’t make sense:

Component	Version/Configuration
k3s	v1.36.5+k3s1, 4 nodes (3 server + 1 agent mixed)
CNI	Cilium 1.20.2, originally ipam.mode=kubernetes
Service network	10.43.0.0/16
Pod network	10.42.0.0/16
Other	Tailscale Operator v1.102.4, Rook-Ceph, ArgoCD
Control plane	disable-cloud-controller: true

Before the overhaul I evaluated three routes:

• Route A: Full dual-stack for control plane + Pod + Service (not endorsed by the vendor)
• Route B: Dual-stack only at the node and tailnet level
• Route C: Node + tailnet dual-stack only, keeping the cluster interior IPv4

At first I actually chose C, because the phrase “not endorsed by the vendor” weighed too heavily. But then I thought: without Pod/Service dual-stack, half the effort is wasted — containers are still IPv4-only, so what’s the point of IPv6?

So I went back to Route A, but split it into A′: P1 makes the control plane dual-stack first, P2 lets Cilium allocate IPv6 PodCIDRs. Two steps, each independently rollback-able.

Step 1: The Vendor Says No, So I Experiment Myself

There was only one big unknown risk: if the apiserver’s --service-cluster-ip-range is changed from IPv4-only to dual-stack, will it refuse to start? Or will it keep running with the old Services?

Nobody gives an answer, because nobody is willing to try it on a production cluster. So locally in Docker, I spun up a k3s of exactly the same version as production, first built the IPv4-only cluster, then changed the flags and restarted, as a controlled experiment. E1 through E14, fourteen rounds in total.

The conclusion was better than I expected:

1. Three flags must be changed together: cluster-cidr, service-cidr, node-ip. IPv4 comes first, and there must be no space after the comma. Miss one, or get the order wrong, and the apiserver fails fast.
2. The failure mode is deterministic. It doesn’t corrupt etcd, it isn’t half-dead; it errors out and exits at startup. This property matters a lot, because it means failure is recoverable.
3. On success, startup completes in about 15 seconds.
4. The measured blast radius for existing Services is exactly one object: kube-dns. Because k3s automatically derives a second cluster-dns and sets it to RequireDualStack.

Points 3 and 4 are what gave me the confidence to act. If the blast radius had been “rebuild all Services,” I wouldn’t have done this at all.

📝Notes: There’s a counterintuitive point here — many people assume that changing ServiceCIDR will reassign all ClusterIPs. Testing shows it doesn’t. The ipFamilyPolicy of existing Services is set back to SingleStack, and the ClusterIP stays unchanged. Dual-stack is opt-in, not a global switch.

Step 2: Cilium IPAM Is the Real Deep Pit

Control-plane dual-stack is just the appetizer; Pod dual-stack is the main course.

My initial naive plan was: annotate the Node with network.cilium.io/ipv6-pod-cidr and let Cilium add an IPv6 PodCIDR — done, right?

Testing disproved it. Under Cilium 1.20.2 this annotation simply doesn’t take effect.

Then can I directly modify Node.spec.podCIDRs? No, this field is immutable. Then delete and recreate the control-plane Node to let k3s reassign? Also no, blocked by k3s’s finalizer.

So the conclusion is firm: under ipam.mode=kubernetes, Pod dual-stack is a dead end. Because Cilium requires every enabled address family to have a PodCIDR; if one is missing, the agent panics + CrashLoopBackOff, and Pod networking across the entire cluster goes down. This is no joke.

The only path is switching to Cilium’s own IPAM:

# values.yaml excerpt
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: ["10.42.0.0/16"]
    clusterPoolIPv6PodCIDRList: ["fd00:42::/48"]
ipv6:
  enabled: true

Key point: switching ipam.mode and setting ipv6.enabled=true must be done in the same helm upgrade. Doing it in two steps leaves an intermediate state where all agents are down.

Silent Blocking by Stale CiliumNode

After the switch, a new problem appeared: the DS stalled at 2/4, and newly started agents reported required IPv6 PodCIDR not available.

The cause is that the leftover CiliumNode.spec.ipam.podCIDRs on the node contains only IPv4; the operator sees it — “this node has already been allocated” — and silently skips it, so the IPv6 PodCIDR can never be added.

The fix is to patch manually:

kubectl patch ciliumnode <node> --type=merge -p '{
  "spec":{"ipam":{"podCIDRs":["10.42.x.0/24","fd00:42::x:0/64"]}}
}'

Be careful to preserve the original IPv4 mapping, otherwise all the static routes on the router have to be changed and the workloads rebuilt. This pitfall deserves special mention: the operator’s “already allocated” judgment is silent — no error, no alert, it just does nothing.

Step 3: Pitfalls Unrelated to the Network Protocol

At this point the protocol-level issues were basically resolved. But what actually blocked me during the production rollout were the following.

Pitfall 1: Node Has No IPv6 InternalIP

Because the cluster has disable-cloud-controller: true, when node-ip is dual-stack, k3s deliberately does not pass it to kubelet. As a result, the Node’s InternalIP is still IPv4-only.

It must be added explicitly:

kubelet-arg:
  - node-ip=192.168.x.x,fd00::x

There’s another verification pitfall: kubectl get nodes -o wide only renders one InternalIP, making it look like it didn’t take effect.
Example result:

kubectl get nodes -o wide
NAME            STATUS   ROLES                       AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION             CONTAINER-RUNTIME
n100-cheshi-0   Ready    <none>                      685d   v1.36.5+k3s1   192.168.3.154   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.3.4-k3s1.36
n100-jumper-0   Ready    control-plane,etcd,master   685d   v1.36.5+k3s1   192.168.3.226   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.3.4-k3s1.36
n100-jumper-1   Ready    control-plane,etcd,master   685d   v1.36.5+k3s1   192.168.3.174   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.3.4-k3s1.36
n100-jumper-2   Ready    control-plane,etcd,master   685d   v1.36.5+k3s1   192.168.3.158   <none>        Ubuntu 26.04.1 LTS   7.0.0-38-generic (amd64)   containerd://2.3.4-k3s1.36

You need:

kubectl get nodes -o custom-columns='NAME:.metadata.name,IPs:.status.addresses[*].address'
NAME            IPs
n100-cheshi-0   192.168.3.154,240e:3a3:...d,n100-cheshi-0
n100-jumper-0   192.168.3.226,240e:3a3:...f,n100-jumper-0
n100-jumper-1   192.168.3.174,240e:3a3:..6b,n100-jumper-1
n100-jumper-2   192.168.3.158,240e:3a3:..5b,n100-jumper-2

Pitfall 2: The Operator Erases the restartedAt Annotation

Existing Pods won’t be rewritten to dual-stack by Cilium; they must be recreated. The usual approach is kubectl rollout restart. But for workloads managed by an operator (KubeVirt, Tailscale ProxyGroup, and the like), the operator erases the restartedAt annotation, so the restart does nothing.

Instead, delete the Pods directly and let the controller recreate them according to the current template:

kubectl delete pod -n <ns> --all

Pitfall 3: The GitOps Client Must Be Dual-Stacked First

This is a delivery-chain coupling issue, and it’s very subtle. ArgoCD’s repoURL points to gitea’s ClusterIP. If you change the gitea-http Service to PreferDualStack first, while ArgoCD’s Pods are still IPv4-only, DNS returns both A and AAAA, the client resolves AAAA first, and then hits ENETUNREACH — git clone fails immediately.

The order can’t be reversed: dual-stack the client Pods first, then change the corresponding Service.

📝Notes: Service dual-stack is opt-in. After enabling dual-stack, newly created Services are still SingleStack by default; an IPv6 ClusterIP only appears on Services that explicitly declare PreferDualStack or RequireDualStack. Conversion can be done with kubectl patch; the original IPv4 ClusterIP is fully preserved, no rebuild needed.

Final State and Rollback Plan

Actual data after the overhaul:

Metric	Result
Pod dual-stack	168/168 (no IPv4-only Pods in the cluster)
Services with IPv6 ClusterIP	29 (28 bulk-converted + 1 earlier pilot)
Node InternalIP	All 4 are IPv4 + IPv6
Cilium DS	4/4
Ceph	HEALTH_OK (22.20k objects / 81 GiB / 81 pgs active+clean / 4 OSD up/in)
PVC	50/50 Bound
ArgoCD	36/36 Synced

This change was high-risk, so full preparation was mandatory:

• Backups! Backups! And more backups!
• A foolproof rollback plan

Unfinished Items

To be honest, a few things weren’t done cleanly:

• 7 Services can’t be declared in the Git repository and can only be remedied with idempotent patches
• metrics-server didn’t retroactively get an IPv6 ClusterIP
• Some ipFamilyPolicy values were written back to the repository rather than being repository-driven, creating a configuration drift risk

None of these affect current operation, but they’re technical debt to keep in mind.

Summary

Looking back, the most valuable part of this overhaul wasn’t “how to configure dual-stack,” but compressing the vendor-unendorsed gray area into deterministic behavior through same-version controlled experiments. The official docs say “not guaranteed”; the experiments say “it fails fast, doesn’t damage etcd, and the blast radius is 1 object” — the information content of those two statements differs enormously. The former keeps you from acting; the latter tells you how to act and how to retreat.

As for the non-network pitfalls (operators erasing annotations, GitOps chain coupling), they all point to the same thing: the success or failure of a large-scale network overhaul often lies not in the network itself, but in your fine-grained grasp of control-plane startup order, controller behavior, and delivery-chain coupling.

The wheels of history roll forward. IPv6, this thing, comes slowly, but it can’t be stopped — so embrace it proactively.

That’s all.

📚️References

• k3s Official Documentation - Dual-Stack Networking
• Cilium Official Documentation - IPAM Modes
• Cilium Official Documentation - IPv6 and Dual-Stack
• Tailscale Kubernetes Operator