Cilium Series Part 9: Switching Host Routing to BPF-Based Mode

Series Articles

• Cilium Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium already effectively improves network performance. However, by toggling different Cilium modes and enabling additional features, you can further enhance Cilium’s network performance. Tuning options include but are not limited to:

• Enable Native Routing
• Fully replace KubeProxy
• Switch IP Address Masquerading to eBPF-based mode
• Run Kubernetes NodePort in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Disable Hubble (not recommended — observability is more important than a marginal performance gain)
• Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)
• (Optional for advanced users) Adjust eBPF Map Size
• Linux Kernel optimization and upgrade
  • CONFIG_PREEMPT_NONE=y
• Other:
  • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
  • Set CPU to performance mode
  • Stop irqbalance and pin NIC interrupts to specific CPUs

When network/NIC/OS conditions permit, we enable as many of these tuning options as possible. Related optimizations will be covered in subsequent articles. Stay tuned.

Today we’ll tune Cilium by enabling Host Routing to completely bypass iptables and the upper host stack, achieving faster network namespace switching than regular veth device operations.

Test Environment

• Cilium 1.13.4
• K3s v1.26.6+k3s1
• OS
  • 3 Ubuntu 23.04 VMs, Kernel 6.2, x86
  • 3 Debian 10 development boards, Kernel 4.19, arm64

eBPF Host-Routing

Even though Cilium uses eBPF for network routing, by default network packets still traverse some parts of the node’s regular network stack. This means all packets still pass through all iptables hooks, which add significant overhead. For exact data from the test environment, refer to TCP Throughput (TCP_STREAM) and compare the results for “Cilium” and “Cilium (Legacy Host Routing)”.

Details are as follows:

Single-Stream:

Multi-Stream:

eBPF-based host routing was introduced in Cilium 1.9 to completely bypass iptables and the upper host stack, achieving faster network namespace switching than regular veth device operations. If the kernel supports this option, it is automatically enabled. To verify whether your installation is using eBPF host routing, run cilium status in any Cilium pod and look for the line reporting the “Host Routing” status, which should show “BPF”.

Below is the performance improvement after eBPF-based host routing was introduced in Cilium 1.9:

As the initial results show, when using Cilium 1.9 (and later) eBPF extensions on a v5.10 kernel, single-stream TCP throughput for Pod-to-remote-Pod sessions under direct routing doubles compared to having both directions handled by host stack forwarding. Similarly, TCP request/response transaction performance for Pods in the test improved by nearly 3x when bypassing the host stack.

Requirements

• Kernel >= 5.10
• Direct-routing configuration or tunneling
• eBPF-based kube-proxy replacement
• eBPF-based masquerading

Implementation

As mentioned above, “if the kernel supports this option, it is automatically enabled”.

Let’s check the situation for Kernel >= 5.10:

Kernel >= 5.10

$ kubectl -n kube-system exec ds/cilium -- cilium status |grep "Host Routing"
Host Routing:            BPF

As shown above, in the Kernel >= 5.10 environment — “3 Ubuntu 23.04 VMs, Kernel 6.2, x86” — BPF-based Host Routing is already automatically enabled.

│ 📝Notes
│
│ According to the previous article - Bypass iptables Connection Tracking: when eBPF Host Routing is unavailable, network packets still traverse the regular network stack in the host namespace, and iptables adds significant cost.
│ Therefore, in the “3 Ubuntu 23.04 VMs, Kernel 6.2, x86” environment, there is actually no need to configure “Bypass iptables Connection Tracking”.

Kernel < 5.10

$ kubectl -n kube-system exec ds/cilium -- cilium status |grep "Host Routing"
Host Routing:            Legacy

As shown above, in the Kernel < 5.10 environment — “3 Debian 10 development boards, Kernel 4.19, arm64” — the Host Routing feature is in Legacy mode.

│ 📝Notes
│
│ According to the previous article - Bypass iptables Connection Tracking: when eBPF Host Routing is unavailable, network packets still traverse the regular network stack in the host namespace, and iptables adds significant cost. This traversal cost can be minimized by disabling connection tracking requirements for all Pod traffic, thereby bypassing the iptables connection tracker.
│ Therefore, in the “3 Debian 10 development boards, Kernel 4.19, arm64” environment, it is necessary to configure “Bypass iptables Connection Tracking”.

Summary

This article covers tuning Cilium by enabling Host Routing to completely bypass iptables and the upper host stack, achieving faster network namespace switching than regular veth device operations.

The prerequisite is Kernel >= 5.10. (In environments where Host Routing cannot be enabled, you can configure “Bypass iptables Connection Tracking” to improve performance.)

At this point, the following performance tuning items have been completed:

• ✔️ Enable Native Routing
• ✔️ Fully replace KubeProxy
• ✔️ Switch IP Address Masquerading to eBPF-based mode
• ✔️ Run Kubernetes NodePort in DSR (Direct Server Return) mode
• ✔️ Bypass iptables Connection Tracking
• ✔️ Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)

📚️References

• Host-Routing - Tuning Guide — Cilium 1.13.4 documentation
• CNI Performance Benchmark — Cilium 1.13.4 documentation
• Cilium 1.9: Maglev, Deny Policies, VM Support, OpenShift, Hubble mTLS, Bandwidth Manager, eBPF Node-Local Redirect, Datapath Optimizations, and more