Cilium Series Part 5: Replacing KubeProxy with Cilium

This article was last updated on: May 17, 2026 am

Series Articles

Introduction

Switching the Kubernetes CNI from other components to Cilium already provides a noticeable boost in network performance. However, by toggling different Cilium modes and enabling additional features, you can further improve Cilium’s network performance. Tuning options include but are not limited to:

  • Enable Native Routing
  • Fully replace KubeProxy
  • Switch IP Address Masquerading to eBPF-based mode
  • Run Kubernetes NodePort in DSR (Direct Server Return) mode
  • Bypass iptables Connection Tracking
  • Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
  • Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
  • Disable Hubble (not recommended — observability is more important than a marginal performance gain)
  • Change MTU to jumbo frames (requires network conditions to allow it)
  • Enable Bandwidth Manager (requires Kernel >= 5.1)
  • Enable BBR congestion control for Pods (requires Kernel >= 5.18)
  • Enable XDP Acceleration (requires native XDP driver support)
  • (Optional for advanced users) Adjust eBPF Map Size
  • Linux Kernel optimization and upgrade
    • CONFIG_PREEMPT_NONE=y
  • Other:
    • tuned network-* profiles, e.g.: tuned-adm profile network-latency or network-throughput
    • Set CPU to performance mode
    • Stop irqbalance and pin NIC interrupts to specific CPUs

When network/NIC/OS conditions permit, we enable as many of these tuning options as possible. Related optimizations will be covered one by one in subsequent articles. Stay tuned.

In the previous article, we enabled Cilium Native Routing, which resulted in a significant improvement in network throughput.

Today, we will use Cilium to fully replace KubeProxy, creating a Kubernetes cluster without KubeProxy, thereby drastically reducing iptables rule chains (and netfilter) for comprehensive network performance improvements.

Test Environment

  • Cilium 1.13.4
  • K3s v1.26.6+k3s1
  • OS
    • 3 Ubuntu 23.04 VMs, Kernel 6.2, x86

Background

In Kubernetes clusters, Kube Proxy makes heavy use of iptables. At larger cluster scales, thousands or even tens of thousands of iptables rules can severely degrade Kubernetes network performance, leading to slow network request responses.

An example of a large number of iptables rule chains:

So Many Iptables Rules

What Kube Proxy Does

Kube Proxy is responsible for traffic routing in the following areas:

  1. ClusterIP: In-cluster access via ClusterIP
  2. NodePort: Access from inside and outside the cluster via NodePort
  3. ExternalIP: External access via external IPs
  4. LoadBalancer: External access via LoadBalancer

Cilium fully implements all of these features with significant performance improvements. The official Cilium benchmark results are as follows:

NodePort Latency Performance

Performance is even better with DSR enabled:

NodePort Latency Performance with DSR

Implementation Steps

Let’s proceed with the replacement. Cilium’s eBPF kube-proxy replacement works in both direct routing and tunnel modes.

Reinstall K3s

1
2
# Server Node
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --flannel-backend=none --disable-network-policy --disable=servicelb --prefer-bundled-bin --disable-kube-proxy' INSTALL_K3S_VERSION=v1.26.6+k3s1 sh -

Notes:

  • –disable=servicelb K3s servicelb is not a standard Kubernetes component. To reduce interference, we remove it first.
  • –disable-kube-proxy Disables Kube Proxy

Reinstall Cilium

Depending on the situation, you may need to uninstall Cilium first:

1
helm uninstall cilium -n kube-system

Reinstall with the kubeProxyReplacement parameter:

1
2
3
4
5
6
7
8
9
10
11
helm install cilium cilium/cilium --version 1.13.4 \
   --namespace kube-system \
   --set operator.replicas=1 \
   --set k8sServiceHost=192.168.2.43 \
   --set k8sServicePort=6443 \
   --set hubble.relay.enabled=true \
   --set hubble.ui.enabled=true \
   --set tunnel=disabled \
   --set autoDirectNodeRoutes=true \
   --set ipv4NativeRoutingCIDR=10.0.0.0/22 \
   --set kubeProxyReplacement=strict

Notes:

  • kubeProxyReplacement=strict Sets kube-proxy replacement to strict mode. By default, Helm sets kubeProxyReplacement=disabled, which only enables in-cluster load balancing for ClusterIP services.

Basic Verification

Verify after installation:

1
2
$ kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
KubeProxyReplacement:    Strict   [eth0 192.168.2.3 (Direct Routing)]

Use --verbose to view full details:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose
...
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                eth0 192.168.2.3 (Direct Routing)
  Mode:                   SNAT
  Backend Selection:      Random
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Disabled
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767)
  - LoadBalancer:   Enabled
  - externalIPs:    Enabled
  - HostPort:       Enabled

Hands-on Verification

Next, we can create an Nginx deployment, then create a new NodePort service and verify that Cilium has correctly installed the service.

Create the Nginx Deployment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
EOF

Next, create a NodePort service for these two instances:

1
2
$ kubectl expose deployment my-nginx --type=NodePort --port=80
service/my-nginx exposed

Check the NodePort service port and other details:

1
2
3
$ kubectl get svc my-nginx
NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
my-nginx   NodePort   10.43.204.231   <none>        80:32727/TCP   96s

Using the cilium service list command, we can verify that Cilium’s eBPF kube-proxy replacement has created the new NodePort service. In this example, a service with port 32727 was created (on NIC device eth0):

1
2
3
4
5
6
7
$ kubectl -n kube-system exec ds/cilium -- cilium service list
ID   Frontend             Service Type   Backend
...
32   192.168.2.3:32727    NodePort       1 => 10.0.0.70:80 (active)
                                         2 => 10.0.2.96:80 (active)
33   0.0.0.0:32727        NodePort       1 => 10.0.0.70:0 (active)
                                         2 => 10.0.2.96:80 (active)

Additionally, we can use iptables in the host namespace to verify that there are no iptables rules for this service:

1
2
3
casey@cilium-62-1:~$ sudo iptables-save | grep KUBE-SVC
[sudo] password for casey:
casey@cilium-62-1:~$

The empty result above confirms that there are no KUBE-SVC related iptables rules.

We can test using curl against NodePort, ClusterIP, PodIP, etc.:

1
2
3
4
5
6
7
8
9
10
11
node_port=$(kubectl get svc my-nginx -o=jsonpath='{@.spec.ports[0].nodePort}')
# localhost+NodePort
curl 127.0.0.1:$node_port
# eth0+NodePort
curl 192.168.2.3:$node_port
# ClusterIP
curl 10.43.204.231:80
# Local PodIP
curl 10.0.0.70:80
# PodIP on another Node
curl 10.0.2.96:80

│ 📝Note
│
│ The last two commands work because we previously enabled Native Routing.

All requests succeed:

1
2
3
4
5
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

🎉🎉🎉

Summary

Kube Proxy’s heavy use of iptables negatively impacts network performance in large-scale Kubernetes clusters. By fully replacing Kube Proxy with Cilium, you can significantly improve Kubernetes network performance for ClusterIP, NodePort, LoadBalancer, externalIPs, and more.

So far, the following performance tuning items have been completed:

  • ✔️ Enable Native Routing
  • ✔️ Fully replace KubeProxy
  • Switch IP Address Masquerading to eBPF-based mode
  • Run Kubernetes NodePort in DSR (Direct Server Return) mode
  • Bypass iptables Connection Tracking
  • Switch Host Routing to BPF-based mode (requires Linux Kernel >= 5.10)
  • Enable IPv6 BIG TCP (requires Linux Kernel >= 5.19)
  • Change MTU to jumbo frames (requires network conditions to allow it)
  • Enable Bandwidth Manager (requires Kernel >= 5.1)
  • Enable BBR congestion control for Pods (requires Kernel >= 5.18)
  • Enable XDP Acceleration (requires native XDP driver support)

📚️References