Cilium Series Part 3: Core Components and Key Concepts of Cilium

Series Articles

• Cilium Series Articles

Introduction

Now that the installation is complete, let’s take a look at the components and key concepts of Cilium.

Cilium Components

As mentioned above, installing Cilium deploys several runtime components (some of which are optional). What does each one do?

Cilium Operator

The Cilium Operator can be thought of as the management plane or operational plane of Cilium. It is not in the critical path of any forwarding or network policy decisions. If the Operator is temporarily unavailable, the cluster can generally continue to operate.

Cilium Agent

The Cilium Agent runs as a DaemonSet, so there is a Cilium Agent pod running on every node in the Kubernetes cluster. The agent performs most of the Cilium-related work:

• Interacts with the Kubernetes API server to synchronize cluster state
• Interacts with the Linux kernel — loading eBPF programs and updating eBPF maps
• Communicates with the Cilium CNI plugin executable via a filesystem socket to receive notifications about newly scheduled workloads
• Creates DNS and Envoy Proxy servers on demand based on the required network policies
• Creates the Hubble gRPC service when Hubble is enabled

Cilium Client

Each pod in the Cilium Agent DaemonSet includes a Cilium Client executable that can be used to inspect the status of the Cilium Agent and eBPF map resources installed on that node. The Client communicates with the Cilium Agent’s REST API through the DaemonSet pod.

│ 🐾Warning:
│
│ This is not shown in the diagram above.
│ The Cilium Client is not the Cilium CLI.
│ It is different from the Cilium CLI tool executable installed on your workstation or bastion host. The Cilium Client executable is included in every Cilium Agent pod and can be used as a diagnostic tool when needed to help troubleshoot Cilium Agent issues. During normal operations, you will rarely interact with the Cilium Client, but in some labs, we will use it to inspect the internals of Cilium’s network state when working with certain Cilium features.

Cilium CNI Plugin

The Cilium Agent DaemonSet also installs the Cilium CNI plugin executable onto the Kubernetes host filesystem and reconfigures the node’s CNI to use it. The CNI plugin executable is separate from the Cilium Agent and is installed as part of the Agent daemon initialization (i.e., the install-cni-binaries init container). When needed, the Cilium CNI plugin communicates with the running Cilium Agent via a host filesystem socket.

Hubble Relay

When Hubble is enabled as part of a Cilium-managed cluster, the Cilium Agent running on each node will restart to enable the Hubble gRPC service, providing node-local observability (with some performance overhead, but it’s worth it). To achieve cluster-wide observability, a Hubble Relay Deployment along with two additional services — the Hubble Observer service and the Hubble Peer service — are added to the cluster.

The Hubble Relay Deployment provides cluster-wide observability by acting as an intermediary between the cluster-wide Hubble Observer service and the Hubble gRPC service provided by each Cilium Agent. The Hubble Peer service allows Hubble Relay to detect when new Hubble-enabled Cilium Agents come online in the cluster. As a user, you typically interact with the Hubble Observer service through the Hubble CLI tool or Hubble UI to gain insight into the cluster’s network traffic as observed by Hubble.

Cluster Mesh API Server

The Cluster Mesh API Server is an optional deployment that is only installed when the Cilium Cluster Mesh feature is enabled. Cilium Cluster Mesh allows multiple clusters to share Kubernetes services.

Cilium Cluster Mesh deploys an etcd key-value store in each cluster to hold information about Cilium identities. It also provides a proxy service for each etcd store. Cilium Agents running in any member of the same cluster mesh can use this service to read global information about Cilium identity state across the entire mesh. This enables the creation and access of global services that span the cluster mesh. Once the Cilium Cluster Mesh API Server is available, Cilium Agents running in any Kubernetes cluster that is a member of the Cluster Mesh can securely read information from each cluster’s etcd proxy, gaining awareness of the Cilium identity state across the entire mesh. This makes it possible to create global services that span the cluster mesh.

Key Concepts of Cilium

Cilium Endpoints

In simple terms, a Cilium Endpoint is a Pod along with its assigned IP address.
Think of it as roughly equivalent to a Kubernetes Endpoint, but with more information.

Example:

$ k3s kubectl get ciliumendpoints.cilium.io -A
NAMESPACE     NAME                                      ENDPOINT ID   IDENTITY ID   INGRESS ENFORCEMENT   EGRESS ENFORCEMENT   VISIBILITY POLICY   ENDPOINT STATE   IPV4         IPV6
default       my-nginx-754c4d44b4-949p2                 1238          30982         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.77
kube-system   local-path-provisioner-76d776f6f9-d75kl   2157          12431         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.133
default       my-nginx-754c4d44b4-zs6fl                 205           30982         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.125
kube-system   coredns-59b4f5bbd5-9q24h                  646           55256         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.82
kube-system   svclb-traefik-29b9c193-67lk8              2511          4457          <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.211
kube-system   traefik-57c84cf78d-wqq48                  1515          52140         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.218
kube-system   metrics-server-566c578db4-h59fh           4083          892           <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.158
kube-system   hubble-ui-68fb44f6f5-69ggm                2960          40657         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.6
kube-system   hubble-relay-5f68b89b76-pzkb5             483           37481         <status disabled>     <status disabled>    <status disabled>   ready            10.0.0.119

Here you can see that the two my-nginx pods have different endpoint IDs, but share the same identity: 30982.

Cilium Identity

A key concept that enables Cilium to work efficiently is the concept of Identity. All Cilium Endpoints have a label-based identity.

Cilium identities are determined by labels and are unique across the entire cluster. Endpoints are assigned an identity that matches their security-relevant labels, meaning all endpoints sharing the same set of security-relevant labels will share the same identity. The unique numeric identifier associated with each identity is used by eBPF programs for fast lookups in the network data path, which is also the foundation for Hubble’s Kubernetes-aware network observability.

When network packets enter or leave a node, Cilium’s eBPF programs map the source and destination IP addresses to their corresponding numeric identity identifiers, and then determine what data path actions to take based on policy configurations that reference these numeric identity identifiers. Each Cilium Agent is responsible for updating the identity-related eBPF maps with the numeric identifiers associated with endpoints running locally on the node, by observing updates to relevant Kubernetes resources.

Example:

$ k3s kubectl get ciliumidentities.cilium.io -A
NAME    NAMESPACE     AGE
55256   kube-system   2d23h
892     kube-system   2d23h
12431   kube-system   2d23h
4457    kube-system   2d23h
52140   kube-system   2d23h
30982   default       33h
40657   kube-system   37m
37481   kube-system   37m

Summary

This article followed up while the topic was fresh to introduce the core components and key concepts of Cilium, laying the groundwork for our deeper exploration of Cilium in upcoming articles. 💪