Cilium Series Part 11: Enabling Bandwidth Manager

Series Articles

• Cilium Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium already effectively improves network performance. However, by toggling different Cilium modes and enabling additional features, you can further enhance Cilium’s network performance. Tuning options include but are not limited to:

• Enabling Native Routing
• Fully replacing KubeProxy
• Switching IP Address Masquerading to eBPF-based mode
• Running Kubernetes NodePort in DSR (Direct Server Return) mode
• Bypass iptables Connection Tracking
• Switching Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• Enabling IPv6 BIG TCP (requires Linux Kernel >= 5.19)
• Disabling Hubble (not recommended — observability is more important than a marginal performance gain)
• Changing MTU to jumbo frames (requires network conditions to allow it)
• Enabling Bandwidth Manager (requires Kernel >= 5.1)
• Enabling BBR congestion control for Pods (requires Kernel >= 5.18)
• Enabling XDP acceleration (requires native XDP driver support)
• (Optional for advanced users) Adjusting eBPF Map Size
• Linux Kernel optimization and upgrade
  • CONFIG_PREEMPT_NONE=y
• Others:
  • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
  • Setting CPU to performance mode
  • Stopping irqbalance and pinning NIC interrupts to specific CPUs

When network/NIC/OS conditions permit, we enable as many of these tuning options as possible. Related optimizations will be covered in subsequent articles. Stay tuned.

Today we’ll tune Cilium by enabling Bandwidth Manager to manage network traffic more efficiently, improving overall application latency and throughput.

Test Environment

• Cilium 1.13.4
• K3s v1.26.6+k3s1
• OS
  • 3 Ubuntu 23.04 VMs, Kernel 6.2, x86

Bandwidth Manager

Cilium’s Bandwidth Manager is responsible for managing network traffic more efficiently, with the goal of improving overall application latency and throughput.

In addition to natively supporting Kubernetes Pod bandwidth annotations, the Bandwidth Manager (first introduced in Cilium 1.9) also sets up Fair Queue (FQ) queueing disciplines on all externally-facing network devices to support TCP stack pacing (e.g., EDT/BBR), and configures optimal server-level sysctl settings for the network stack.

The Bandwidth Manager’s functionality focuses on two aspects: from the upper-layer protocol perspective and from the queueing discipline perspective.

When the Bandwidth Manager is enabled, it switches the TCP congestion control algorithm to BBR by default, achieving higher bandwidth and lower latency, especially for internet-facing traffic. It configures the kernel network stack with more server-oriented sysctl settings that have been proven beneficial in production environments. It also reconfigures the traffic control queueing discipline (Qdisc) layer to use multi-queue Qdiscs and Fair Queue (FQ) on all externally-facing network devices used by Cilium. After switching to Fair Queue, the Bandwidth Manager also implements support for Earliest Departure Time (EDT) rate limiting with the help of eBPF, and now natively supports the kubernetes.io/egress-bandwidth Pod annotation.

This also eliminates the need for the bandwidth CNI plugin chain, as that plugin uses TBF (Token Bucket Filter), which has scalability limitations. With the EDT-based model, global locking at the Qdisc layer can be avoided, especially with multi-queue NICs. Cilium’s eBPF datapath classifies network traffic into per-node aggregates, then enforces user-defined kubernetes.io/egress-bandwidth rates by setting earliest departure timestamps on egress network packets shortly before handing them to the FQ leaf Qdiscs. The Qdiscs maintain per-flow state and schedule packet departure times based on their timestamps, ensuring packets are not sent before the time specified by the timestamp. Thanks to eBPF’s flexibility, classification of Pod aggregates works not only with direct routing but also with tunneling or when using L7 proxies.

Compared to using HTB (Hierarchical Token Bucket) for rate limiting, evaluations of application latency with eBPF and FQ show significant reductions in CPU utilization while improving transmission latency. When eBPF and FQ are used together, the 95th percentile latency is reduced by approximately 20x, and the 99th percentile latency is reduced by approximately 10x.

Requirements

• Kernel >= 5.1
• Direct-routing configuration or tunneling
• eBPF-based kube-proxy replacement

Hands-on: Enabling Bandwidth Manager

To enable the Bandwidth Manager:

helm upgrade cilium cilium/cilium --version 1.13.4 \
  --namespace kube-system \
  --reuse-values \
  --set bandwidthManager.enabled=true 

Verification

Status Verification

To verify that your installation is running with the Bandwidth Manager, run cilium status in any Cilium pod and look for the line reporting “BandwidthManager” status, which should show “EDT with BPF”.

Specifically:

$ kubectl -n kube-system exec ds/cilium -- cilium status | grep BandwidthManager
BandwidthManager:           EDT with BPF [CUBIC] [eth0]

Bandwidth Manager Functional Verification

Below is an example of deploying application Pods with egress bandwidth limited to 50 Mbit/s using the kubernetes.io/egress-bandwidth annotation:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
   name: iperf3
   labels:
      app: iperf3
spec:
   selector:
      matchLabels:
        app: iperf3
   template:
      metadata:
         labels:
            app: iperf3
         annotations:
            kubernetes.io/egress-bandwidth: '50M'
      spec:
         containers:
         -  name: iperf3
            image: clearlinux/iperf:3
            command: ['/bin/sh', '-c', 'sleep 1d']
            ports:
            - containerPort: 5201
EOF

The result is as follows:

$ k3s kubectl get pod -o wide -l app=iperf3
NAME           READY   STATUS    RESTARTS   AGE    IP           NODE          NOMINATED NODE   READINESS GATES
iperf3-g54rg   1/1     Running   0          2m9s   10.0.0.127   cilium-62-1   <none>           <none>
iperf3-4fwnf   1/1     Running   0          97s    10.0.1.101   cilium-62-2   <none>           <none>
iperf3-688m4   1/1     Running   0          65s    10.0.2.247   cilium-62-3   <none>           <none>

Select one pod as the server (the one on the cilium-62-2 node) and another as the client (the one on the cilium-62-3 node).

The command to run on the server (iperf3-4fwnf):

kubectl exec -it iperf3-4fwnf -- iperf3 -s -f M

The command to run on the client (iperf3-688m4):

$ kubectl exec -it iperf3-688m4 -- iperf3 -c 10.0.1.101 -f m
Connecting to host 10.0.1.101, port 5201
[  5] local 10.0.2.247 port 33300 connected to 10.0.1.101 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.51 MBytes  63.0 Mbits/sec    0    385 KBytes
[  5]   1.00-2.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   2.00-3.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   3.00-4.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   4.00-5.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   5.00-6.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   6.00-7.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   7.00-8.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   8.00-9.00   sec  5.65 MBytes  47.4 Mbits/sec    0    385 KBytes
[  5]   9.00-10.00  sec  6.46 MBytes  54.2 Mbits/sec    0    385 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  59.2 MBytes  49.7 Mbits/sec    0             sender
[  5]   0.00-10.02  sec  57.0 MBytes  47.8 Mbits/sec                  receiver

iperf Done.

The measured bandwidth is 49.7 Mbits/sec (sender) and 47.8 Mbits/sec (receiver), confirming that the 50 Mb/s bandwidth limit is effectively enforced. 👍️👍️👍️

Summary

This article continues tuning Cilium by enabling the Bandwidth Manager to manage network traffic more efficiently, improving overall application latency and throughput. We also verified the bandwidth limiting functionality through hands-on testing.

At this point, the following performance tuning items have been validated:

• ✔️ Enabling Native Routing
• ✔️ Fully replacing KubeProxy
• ✔️ Switching IP Address Masquerading to eBPF-based mode
• ✔️ Running Kubernetes NodePort in DSR (Direct Server Return) mode
• ✔️ Bypass iptables Connection Tracking
• ✔️ Switching Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
• ❌ Enabling IPv6 BIG TCP (requires Linux Kernel >= 5.19, supported NICs: mlx4, mlx5)
  • Unable to verify due to lack of supported NICs
• ❌ Changing MTU to jumbo frames (requires network conditions to allow it)
• ✔️ Enabling Bandwidth Manager (requires Kernel >= 5.1)
• Enabling BBR congestion control for Pods (requires Kernel >= 5.18)
• Enabling XDP acceleration (requires native XDP driver support)

📚️ References

• Bandwidth Manager - Tuning Guide — Cilium 1.13.4 documentation
• Bandwidth Manager - Cilium 1.9: Maglev, Deny Policies, VM Support, OpenShift, Hubble mTLS, Bandwidth Manager, eBPF Node-Local Redirect, Datapath Optimizations, and more