Cilium Series Part 7: Switching Cilium's NodePort Implementation from SNAT to DSR

Series Articles

• Cilium Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium can already effectively improve network performance. However, by toggling different Cilium modes and enabling various features, you can further enhance Cilium’s network performance. Tuning options include but are not limited to:

• Enable Native Routing
• Fully replace KubeProxy
• Switch IP Address Masquerading to eBPF-based mode
• Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Disable Hubble (not recommended — observability is more important than a marginal performance gain)
• Set MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)
• (Optional for advanced users) Adjust eBPF Map Size
• Linux Kernel optimization and upgrade
  • CONFIG_PREEMPT_NONE=y
• Other:
  • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
  • Set CPU to performance mode
  • Stop irqbalance and pin NIC interrupts to specific CPUs

Where network/NIC/OS conditions permit, we enable as many of these tuning options as possible. Related optimizations will be covered in subsequent articles. Stay tuned.

Today we will switch Cilium’s NodePort implementation from SNAT to DSR to improve network performance.

Test Environment

• Cilium 1.13.4
• K3s v1.26.6+k3s1
• OS
  • 3 Ubuntu 23.04 VMs, Kernel 6.2, x86

Direct Server Return (DSR)

By default, Cilium’s eBPF NodePort implementation operates in SNAT mode. This means that when external traffic arrives at a node, if the node determines that the backend for a load balancer, NodePort, or service with an external IP resides on a remote node, it redirects the request to the remote backend on its behalf by performing SNAT. This requires no additional MTU changes. The trade-off is that replies from the backend need to take an extra hop back to the original node, where reverse SNAT translation is performed before the packet is returned directly to the external client.

As shown below, Cilium’s eBPF NodePort implementation running in SNAT mode:

$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose
...
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                eth0 192.168.2.3 (Direct Routing)
  Mode:                   SNAT

In SNAT mode, inbound traffic when the NodePort backend pod is on a different node:

Outbound traffic:

This can be changed to dsr via the loadBalancer.mode Helm option, so that Cilium’s eBPF NodePort implementation runs in DSR mode. In this mode, the backend replies directly to the external client without the extra hop — that is, the backend replies using the service IP/port as the source. DSR currently requires Cilium to be deployed in native routing mode, meaning it does not work with any tunnel mode.

DSR mode traffic flow:

Another advantage of DSR mode is that it preserves the client’s source IP, allowing policy matching on the backend node. This is not possible in SNAT mode. Since a given backend may serve multiple services, the backend needs to know which service IP/port to use in its reply.

Note that because Cilium-specific IP options may be dropped by the underlying network fabric, DSR mode may not work in certain public cloud provider environments. If the backend resides on a node far from the node handling a given NodePort request, and you experience service connectivity issues, first check whether the NodePort request actually reaches the node containing the backend. If not, switching back to the default SNAT mode is recommended as a workaround.

Additionally, in certain public cloud provider environments that enforce source/destination IP address checks (such as AWS), these checks must be disabled for DSR mode to work.

Steps to Enable DSR

In a kube-proxy-free environment with DSR mode only, the Helm configuration example is as follows:

helm upgrade cilium cilium/cilium --version 1.13.4 \
    --namespace kube-system \
    --reuse-values \
    --set loadBalancer.mode=dsr

│ 🐾Warning
│
│ Prerequisites:
│
│ 1. Native Routing is enabled
│ 2. Cilium has fully replaced KubeProxy

Verification

$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose|grep DSR
  Mode:                   DSR

Performance Improvement

For performance improvements, refer to the official benchmark:

👍️👍️👍️

Summary

In this article, we switched Cilium’s NodePort implementation from SNAT mode to DSR mode. Compared to SNAT mode, DSR has clear advantages when the NodePort backend pod is on a different node:

• At least one fewer network hop
• Client source IP is preserved

However, pay attention to the prerequisites when enabling DSR in public cloud environments.

At this point, the following performance tuning items have been completed:

• ✔️ Enable Native Routing
• ✔️ Fully replace KubeProxy
• ✔️ Switch IP Address Masquerading to eBPF-based mode
• ✔️ Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Set MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)

📚️References

• DSR Mode - Kubernetes Without kube-proxy — Cilium 1.13.4 documentation
• Cilium 1.10: WireGuard, BGP Support, Egress IP Gateway, New Cilium CLI, XDP Load Balancer, Alibaba Cloud Integration and more