Cilium Series Part 10 - Enabling IPv6 BIG TCP and Jumbo Frames

Series Articles

• Cilium Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium can already effectively improve network performance. However, by toggling different Cilium modes and enabling additional features, you can further enhance Cilium’s network performance. Tuning options include but are not limited to:

• Enable Native Routing
• Fully replace KubeProxy
• Switch IP Address Masquerading to eBPF-based mode
• Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Disable Hubble (not recommended — observability is more important than a marginal performance gain)
• Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)
• (Optional for advanced users) Adjust eBPF Map Size
• Linux Kernel optimization and upgrade
  • CONFIG_PREEMPT_NONE=y
• Other:
  • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
  • Set CPU to performance mode
  • Stop irqbalance and pin NIC interrupts to specific CPUs

When network/NIC/OS conditions are met, we enable as many of these tuning options as possible. Related optimizations will be covered one by one in subsequent articles. Stay tuned.

Today we tune Cilium by enabling IPv6 BIG TCP, which allows the network stack to prepare larger GSO (transmit) and GRO (receive) packets, reducing the number of stack traversals to improve performance and latency.

Test Environment

• Cilium 1.13.4
• K3s v1.26.6+k3s1
• OS
  • 3 Ubuntu 23.04 VMs, Kernel 6.2, x86

IPv6 BIG TCP

IPv6 BIG TCP allows the network stack to prepare larger GSO (transmit) and GRO (receive) packets, reducing the number of stack traversals to improve performance and latency. It reduces CPU load and helps achieve higher speeds (i.e., 100 Gbit/s and above). To allow these packets to pass through the stack, BIG TCP adds a temporary Hop-By-Hop header after the IPv6 header and strips it before transmitting the packet over the wire. BIG TCP can operate in a dual-stack setup where IPv4 packets use the old limit (64k) and IPv6 packets use the new, larger limit (192k). Note that Cilium assumes the default kernel values of 64k for GSO and GRO and only adjusts them when necessary — that is, if BIG TCP is enabled and the current GSO/GRO maximum is less than 192k, Cilium will attempt to increase these values; if BIG TCP is disabled and the current maximum is greater than 64k, Cilium will attempt to decrease these values. BIG TCP does not require changes to the network interface MTU.

Requirements

• ✔️ Kernel >= 5.19
• ✔️ eBPF Host Routing
• ✔️ eBPF-based kube-proxy replacement
• ✔️ eBPF-based masquerading
• ✔️ Tunnel and encryption disabled
• ❌ Supported NICs: mlx4, mlx5

Since I don’t have mlx4 or mlx5 NICs available, I was unable to perform hands-on testing this time.

To enable IPv6 BIG TCP:

helm install cilium cilium/cilium --version 1.13.4 \
  --namespace kube-system \
  --set tunnel=disabled \
  --set bpf.masquerade=true \
  --set ipv6.enabled=true \
  --set enableIPv6Masquerade=false \
  --set enableIPv6BIGTCP=true \
  --set kubeProxyReplacement=strict

Note that after toggling the IPv6 BIG TCP option, Kubernetes Pods must be restarted for the changes to take effect.

To verify that your installation is running with IPv6 BIG TCP, run cilium status in any Cilium pod and look for the line reporting “IPv6 BIG TCP” status — it should show “enabled”.

Changing MTU to Jumbo Frames

While we’re at it, let’s also mention changing the MTU to jumbo frames.

The Maximum Transmission Unit (MTU) can have a significant impact on configured network throughput. Cilium will automatically detect the MTU of the underlying network devices. Therefore, if the system is configured to use jumbo frames, Cilium will automatically use jumbo frames as well.

To benefit from this, make sure your system is configured to use jumbo frames (if your network allows it).

Since my network devices cannot be configured for jumbo frames, I was also unable to perform hands-on testing this time.

Summary

This article covers tuning Cilium by enabling IPv6 BIG TCP, which allows the network stack to prepare larger GSO (transmit) and GRO (receive) packets, reducing the number of stack traversals to improve performance and latency.

However, the prerequisites are Kernel >= 5.19 and specific NIC support, so hands-on verification was not possible this time.

Another tuning option is changing the MTU to jumbo frames to improve network throughput, but this requires the network to support it.

At this point, the performance tuning items that have been verified in practice are:

• ✔️ Enable Native Routing
• ✔️ Fully replace KubeProxy
• ✔️ Switch IP Address Masquerading to eBPF-based mode
• ✔️ Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• ✔️ Bypass iptables Connection Tracking
• ✔️ Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• ❌ Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19, supported NICs: mlx4, mlx5)
  • Unable to verify due to lack of supported NICs
• ❌ Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)

📚️ References

• IPv6 BIG TCP - Tuning Guide — Cilium 1.13.4 documentation