Cilium Series Part 6: Switching IP Masquerading from IPtables to eBPF

Series Articles

• Cilium Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium can already effectively improve network performance. However, by switching Cilium modes and enabling additional features, you can further enhance Cilium’s network performance. Tuning options include but are not limited to:

• Enable Native Routing
• Fully replace KubeProxy
• Switch IP Masquerading to eBPF-based mode
• Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Disable Hubble (not recommended — observability is more important than a marginal performance gain)
• Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)
• (Optional for advanced users) Adjust eBPF Map Size
• Linux Kernel optimization and upgrade
  • CONFIG_PREEMPT_NONE=y
• Other:
  • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
  • Set CPU to performance mode
  • Stop irqbalance and pin NIC interrupts to specific CPUs

When network/NIC/OS conditions permit, we enable as many of these tuning options as possible. Related optimizations will be covered in subsequent articles. Stay tuned.

Today we will enable Cilium eBPF IP masquerading mode to improve network efficiency.

Test Environment

• Cilium 1.13.4
• K3s v1.26.6+k3s1
• OS
  • 3x Ubuntu 23.04 VMs, Kernel 6.2, x86

IP Masquerading

The IPv4 addresses used by Pods are typically allocated from RFC1918 private address blocks and are therefore not publicly routable. Cilium automatically masquerades the source IP address of all traffic leaving the cluster to the node’s IPv4 address, since the node’s IP is already routable on the network. As shown below:

For IPv6 addresses, masquerading is only performed when using the iptables implementation.

To disable this option:

• For IPv4 traffic leaving the host, use the option enable-ipv4-masquerade: false.
• For IPv6 traffic, use the option enable-ipv6-masquerade: false.

Configuration

Setting the Routable CIDR

The default behavior excludes any destination within the local node IP allocation CIDR range. If Pod IPs are routable across a broader network, you can specify that network using the option: ipv4-native-routing-cidr: 10.0.0.0/8 (or ipv6-native-routing-cidr: fd00::/100 for IPv6 addresses). In this case, all destinations within that CIDR range will not be masqueraded.

Setting the Masquerade Interface

See the section below on configuring the masquerade interface.

eBPF-Based IP Masquerading Mode

If not explicitly specified, the default is IPTables-based IP masquerading mode. This is the traditional implementation and works on all kernel versions. The default configuration can be checked as follows:

$ kubectl -n kube-system exec ds/cilium -- cilium status | grep Masquerading
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]

The eBPF-based implementation is the most efficient implementation. It requires Linux kernel 4.19 and can be enabled via the bpf.masquerade=true helm option.

The current implementation depends on the BPF NodePort feature. This dependency will be removed in the future (GitHub issue 13732).

The specific command is:

helm upgrade cilium cilium/cilium \
   --namespace kube-system \
   --reuse-values \
   --set bpf.masquerade=true

Masquerading can only be performed on NIC devices running the eBPF masquerade program. This means that if the egress NIC device runs the program, packets sent from Pods to external addresses will be masqueraded (to the IPv4 address of the egress NIC device). If not specified, the program will automatically attach to the NIC devices selected by the BPF NodePort device detection mechanism. To change this manually, use the devices helm option. Use cilium status to determine which NIC devices the program is running on:

$ kubectl -n kube-system exec ds/cilium -- cilium status | grep Masquerading
Masquerading:            BPF   [eth0]   10.0.0.0/22 [IPv4: Enabled, IPv6: Disabled]

As shown in the output above, IPv6: Disabled — this is because eBPF-based masquerading does not currently support IPv6 traffic.

From the output above, the program is running on the eth0 NIC device.

eBPF-based masquerading can masquerade packets for the following IPv4 L4 protocols:

• TCP
• UDP
• ICMP (Echo request and Echo reply only)

By default, all packets sent from Pods to IP addresses outside the ipv4-native-routing-cidr range are masqueraded, except for packets destined for other cluster nodes. The excluded CIDR is shown in the cilium status output above (10.0.0.0/22).

For more fine-grained control, Cilium implements ip-masq-agent in eBPF, which can be enabled via the ipMasqAgent.enabled=true helm option.

The eBPF-based ip-masq-agent supports setting nonMasqueradeCIDRs and masqLinkLocal options in the configuration file. Packets sent from Pods to destinations belonging to any CIDR in nonMasqueradeCIDRs will not be masqueraded. If the configuration file is empty, the agent provides the following non-masquerade CIDRs:

• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16
• 100.64.0.0/10
• 192.0.0.0/24
• 192.0.2.0/24
• 192.88.99.0/24
• 198.18.0.0/15
• 198.51.100.0/24
• 203.0.113.0/24
• 240.0.0.0/4

│ 📝Note
│
│ The configuration using ip-masq-agent will not be demonstrated here. Interested readers can try it on their own.

Additionally, if masqLinkLocal is not set or is set to false, 169.254.0.0/16 is appended to the non-masquerade CIDR list.

The agent uses Fsnotify to track configuration file updates.

The following example shows how to configure the agent via a ConfigMap and verify it:

apiVersion: v1
  kind: ConfigMap
  metadata:
    name: ip-masq-agent
  data:
    config: |
      nonMasqueradeCIDRs:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16
      masqLinkLocal: true

$ kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.13.4/examples/kubernetes-ip-masq-agent/rfc1918.yaml

$ # Wait ~60s until the ConfigMap is propagated into the configuration file

$ kubectl -n kube-system exec ds/cilium -- cilium bpf ipmasq list
IP PREFIX/ADDRESS
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

Alternatively, when installing Cilium via Helm, you can configure the above ip-masq-agent with --set ipMasqAgent.config.nonMasqueradeCIDRs=‘{10.0.0.0/8,172.16.0.0/12,192.168.0.0/16}’ and --set ipMasqAgent.config.masqLinkLocal=false.

Summary

In this article, we switched IP masquerading from IPTables to eBPF-based mode. Compared to the IPTables mode, the eBPF-based implementation is the most efficient implementation.

At this point, the following performance tuning has been completed:

• ✔️ Enable Native Routing
• ✔️ Fully replace KubeProxy
• ✔️ Switch IP Masquerading to eBPF-based mode
• Run Kubernetes NodePort implementation in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Change MTU to jumbo frames (requires network conditions to allow it)
• Enable Bandwidth Manager (requires Kernel >= 5.1)
• Enable BBR congestion control for Pods (requires Kernel >= 5.18)
• Enable XDP acceleration (requires native XDP driver support)

📚️References

• Masquerading — Cilium 1.13.4 documentation