In this chapter, we will install Cilium directly into a Kubernetes cluster.
The components and versions used in our experiment are:
Cilium 1.13.4
K3s v1.26.6+k3s1
OS
Debian 10, Kernel 4.19.232, arm64
Ubuntu 23.04, Kernel 6.2, x86
│ 📝Notes:
│
│ As mentioned in the previous article, Cilium has strict requirements on the Linux Kernel version. Version 1.13.4 recommends Kernel ≥ 5.10 (using the latest LTS stable Kernel), with a minimum of Linux Kernel 4.19.57.
│ So we chose 2 operating systems — one that only meets the minimum Kernel requirement, and one with the newest possible Kernel — to see what differences there are in installation and functionality.
Cilium Installation Methods
Cilium supports 2 installation methods:
Cilium CLI
Helm chart
The CLI tool makes it easy to get started with Cilium, especially when you’re just beginning to learn. It uses the Kubernetes API directly to inspect the cluster corresponding to the existing kubectl context and selects appropriate installation options for the detected Kubernetes implementation.
The Helm Chart method is suitable for advanced installations and production environments that require fine-grained control over the Cilium installation. It requires you to manually select the optimal datapath and IPAM mode for your specific Kubernetes environment.
System Requirements
To install Cilium, the minimum system requirements are as follows:
Host must be AMD64 or AArch64 (i.e., arm64) architecture
For the first installation, we use the Cilium CLI method. The OS is: Debian 10, Kernel 4.19.232, arm64.
Installing K3s
We install the Kubernetes cluster via K3s. The specific command is as follows:
1 2
# Server Node curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC='--write-kubeconfig-mode=644 --flannel-backend=none --disable-network-policy --prefer-bundled-bin' INSTALL_K3S_VERSION=v1.26.6+k3s1 sh -
│ 📝Notes:
│
│ Several mainstream Linux distributions ship iptables versions that contain a bug causing duplicate rules to accumulate, which negatively impacts node performance and stability. For how to determine if you are affected, see issue #3117.
│ K3s includes a properly functioning iptables version (v1.8.8). You can have K3s use the bundled iptables version by starting K3s with the --prefer-bundled-bin option, or by uninstalling the iptables/nftables packages from the OS.
│ The --prefer-bundled-bin flag has been available since the 2022-12 releases (v1.26.0+k3s1, v1.25.5+k3s1, v1.24.9+k3s1, v1.23.15+k3s1).
Verification:
1 2 3 4
$ systemctl status k3s ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled) Active: active (running)
1 2 3
$ k3s kubectl get node NAME STATUS ROLES AGE VERSION linaro-alip NotReady control-plane,master 3d1h v1.26.6+k3s1
🐾 Note that since Flannel is not installed and Cilium has not been installed yet, the node status should be: NotReady.
Installing Cilium CLI
1 2 3 4 5 6 7
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
Verification:
1 2 3 4 5
$ cilium version cilium-cli: v0.15.2 compiled with go1.20.4 on linux/arm64 cilium image (default): v1.13.4 cilium image (stable): v1.13.4 cilium image (running): 1.13.4
Run the following command to verify that the cluster has proper network connectivity:
1 2 3 4 5 6 7 8
$ cilium connectivity test --request-timeout 30s --connect-timeout 10s ℹ️ Monitor aggregation detected, will skip some flow validation steps ✨ [k8s-cluster] Creating namespace for connectivity check... (...) --------------------------------------------------------------------------------------------------------------------- 📋 Test Report --------------------------------------------------------------------------------------------------------------------- ✅ 69/69 tests successful (0 warnings)
│ 🐾Warning:
│
│ When installing in China, due to network restrictions, some tests may fail (e.g., accessing 1.1.1.1:443). See the example below for details.
│ This is expected behavior.
│ The connectivity test requires at least two worker nodes to successfully deploy in the cluster. The connectivity test pods will not be scheduled on nodes running in the control plane role. If you haven’t configured two worker nodes for your cluster, the connectivity test command may stall while waiting for the test environment deployment to complete.
Example — actual test results when running in China:
$ kubectl -n kube-system exec ds/cilium -- cilium status Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init) KVStore: Ok Disabled Kubernetes: Ok 1.26 (v1.26.6+k3s1) [linux/arm64] Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"] KubeProxyReplacement: Disabled Host firewall: Disabled CNI Chaining: none CNI Config file: CNI configuration file management disabled Cilium: Ok 1.13.4 (v1.13.4-4061cdfc) NodeMonitor: Listening for events on 4 CPUs with 64x4096 of shared memory Cilium health daemon: Ok IPAM: IPv4: 9/254 allocated from 10.0.0.0/24, IPv6 BIG TCP: Disabled BandwidthManager: Disabled Host Routing: Legacy Masquerading: IPtables Controller Status: 48/48 healthy Proxy Status: No managed proxy redirect Global Identity Range: min 256, max 65535 Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 11.68 Metrics: Disabled Encryption: Disabled Cluster health: 1/1 reachable (2023-07-19T12:25:40Z)
A few things to note here:
datapath mode: tunnel: For compatibility reasons, Cilium defaults to the tunnel (VXLAN-based) datapath mode, which is an overlay network architecture.
KubeProxyReplacement: Disabled: Cilium has not fully replaced kube-proxy. We will cover how to achieve this replacement in a future article.
IPv6 BIG TCP: Disabled: This feature requires Linux Kernel >= 5.19, so it is disabled on Kernel 4.19.232.
BandwidthManager: Disabled: This feature requires Linux Kernel >= 5.1, so it is currently disabled.
Host Routing: Legacy: Legacy Host Routing still relies on iptables and has weaker performance; however, BPF-based host routing requires Linux Kernel >= 5.10.
Masquerading: IPtables: There are several IP masquerading methods: eBPF-based and iptables-based. The default is iptables-based, but eBPF-based is recommended.
Hubble Relay: disabled: Hubble is also disabled by default.
The most important characteristic of Cilium is its performance, so we will cover how to enable all performance-enhancing features one by one in subsequent articles.
Installing Cilium Hubble
1 2 3 4 5 6 7 8 9 10 11 12 13
$ cilium hubble enable --ui ✨ Patching ConfigMap cilium-config to enable Hubble... 🚀 Creating ConfigMap for Cilium version 1.13.4... ♻️ Restarted Cilium pods ⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)... 🚀 Creating Peer Service... ✨ Generating certificates... 🔑 Generating certificates for Relay... ✨ Deploying Relay... ✨ Deploying Hubble UI and Hubble UI Backend... ⌛ Waiting for Hubble to be installed... ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret ✅ Hubble was successfully enabled!
You should notice that the Cilium DaemonSet is running on all nodes in the cluster, while the cilium-operator Deployment is running on a single node.
Congratulations! 🎉🎉🎉 You have now installed Cilium to provide connectivity for your Kubernetes cluster.
Summary
In this article, we covered the quick installation process for Cilium.
To install Cilium, certain basic requirements must be met, among which Cilium has relatively strict requirements on the Linux Kernel version.
Using cilium install, we installed K3s v1.26.6+k3s1 and Cilium 1.13.4 on a Debian 10, Kernel 4.19.232, arm64 machine, enabled Hubble, and verified the setup. 🎉🎉🎉