Several Approaches to Encrypting K8s Secrets

Introduction

You’ve probably heard this not-so-secret secret many times — Kubernetes Secrets are not encrypted! Secret values are stored in etcd as base64 encoded strings. This means anyone with access to your cluster can easily decode your sensitive data. Anyone? Yes, almost anyone can, especially when the cluster’s RBAC is not properly configured. Anyone can access the API or access etcd. It could also be someone authorized to create a Pod or Deployment in a Namespace, who then uses that permission to retrieve all Secrets in that Namespace. How do you ensure that Secrets and other sensitive information (such as tokens) on your cluster are not leaked? In this blog post, we’ll discuss several approaches to encrypting application Secrets when building, deploying, and running applications on K8s.

K8s Secrets

Applications running on a Kubernetes cluster can use Kubernetes Secrets, eliminating the need to store sensitive data such as tokens or passwords in the application code.

The typical workflow for Secrets within a default Kubernetes cluster is as follows:

1. Dev phase: Application developers using CI/CD treat git as the source of truth for managing configurations deployed to the cluster. Access control helps secure access to the repository, but this alone doesn’t always ensure that sensitive application information won’t be leaked.

2. Ops phase: The API server creates Kubernetes Secret resources on the cluster. You can read more about the Secret lifecycle here. Secrets stored in etcd can be consumed by application pods in one of three ways:

   1. As files in a volume mount for one or more containers.
   2. As container environment variables.
   3. Used by the Pod’s kubelet when pulling images.

In all three cases, the Secret values are decoded before use.

So now that we know how it works, why is base64 encoding alone not sufficient?

Why Base64 Encoding Is Not Encryption

Base64 encoding is a binary-to-text encoding scheme that represents 24-bit binary data as 6-bit base64 digits. It is used to transmit large amounts of data over the network, especially large files such as image files. Its primary function is to provide data integrity during transmission over the network. To be clear, encoding is not encryption.

Try this on any Linux terminal.

$ echo -n 'not encrypted' | base64
bm90IGVuY3J5cHRlZA==

$ echo -n 'bm90IGVuY3J5cHRlZA==' | base64 --decode
not encrypted

As shown above, whether during transmission of Secrets to the cluster or when used on the cluster, anyone with access to your system can easily decode your Secrets.

The Problem

As a DevSecOps administrator, you clearly face two challenges:

1. How to encrypt and manage sensitive data outside the cluster — that is, before it enters the cluster during the build and deploy phases?
2. How to protect sensitive data while applications are running inside the cluster?

Here are several approaches to encrypting K8s Secrets.

Encrypting Secrets Before Deploying to the Cluster

As a developer pushing code to a git repository (a.k.a. the “source of truth” for your application), you can encrypt the sensitive information used by your application before pushing it to the git repository. Below are two common methods for encrypting secrets before they are committed to a git repository and deployed to an OpenShift cluster:

Using Bitnami Sealed Secrets

Bitnami Sealed Secrets Overview

Bitnami Sealed Secrets: “Encrypted Secrets” for Kubernetes.

Typical use case:

The problem: “I can manage all my K8s configs in git, except Secrets.”
The solution: Encrypt your Secret into a SealedSecret, which is safe to store even in a public repository. A SealedSecret can only be decrypted by the controller running in the target cluster — no one else (not even the original author) can obtain the original Secret from a SealedSecret.

Bitnami Sealed Secrets Workflow

An example workflow using Bitnami Sealed Secrets:

1. The cluster administrator deploys the Sealed Secrets controller on the K8s cluster.
2. The developer installs the kubeseal CLI on their local machine.
3. The developer creates a Secret resource, which is then encrypted or sealed by the kubeseal CLI using a key fetched from the controller at runtime. For network-restricted environments, the public key can also be stored locally and used by kubeseal. Kubeseal will create a SealedSecret custom resource.
4. The developer pushes this CR to their git repository.
5. A CD tool such as ArgoCD can be used to deploy the CR on the cluster.
6. The controller detects the SealedSecret resource and decrypts it using the private key on the cluster.

Using KSOPS/Mozilla SOPS

KSOPS/Mozilla SOPS Overview

• Mozilla SOPS - sops is an editor of encrypted files that supports YAML, JSON, ENV, INI, and BINARY formats, and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
• KSOPS - A flexible Kustomize plugin for SOPS-encrypted resources.

KSOPS/Mozilla SOPS Workflow

If you use Argo CD to deploy applications in Kubernetes, you can use the Kustomize SOPS plugin, which decrypts resources encrypted with SOPS.

On the cluster, the administrator will:

1. Deploy ArgoCD.
2. Generate keys using age.
3. Create a secret storing the public and private keys in a specific (e.g., GitOps) Namespace.
4. Customize Argo CD to use the Kustomize SOPS plugin.
5. Push the public key to the Git repository.

The developer will:

1. Create a Secret on the local console.
2. Use the SOPS CLI to download the public key and encrypt the secret.
3. Generate a KSOPS YAML with the encrypted Secrets and push it to the Git repository.

ArgoCD decrypts the secret files using KSOPS before deploying the Secrets on the cluster.

Summary

Both methods above use asymmetric encryption to encrypt secrets. Both provide a way to decrypt sensitive data before it is deployed to the cluster as Secrets. Sealed Secrets integrates natively with Kubernetes. SOPS/KSOPS can work independently and does not require a controller on the cluster. Additionally, Sealed Secrets uses strong crypto such as AES-256-GCM, while SOPS uses gpg and age. SOPS provides integration with cloud provider KMS, whereas SealedSecrets does not yet, but plans to implement it in the future (see here). SOPS can encrypt not only Secret values but also YAML, JSON, env var, and binary values, making it suitable for encrypting Helm charts as well.

However, as you can see, once encrypted data enters the cluster, it is decrypted before use. So this essentially only solves part of the problem. Next, we need to look at how to protect this data within the cluster. Let’s explore the different options for encrypting data on the cluster.

Encrypting Secrets on the K8s Cluster

K8s etcd Encryption Options

By default, the K8s container platform does not encrypt etcd data. However, native K8s and some K8s distributions provide options to enable etcd-based encryption.

Here are some relevant reference documents:

1. Native K8s: Encrypting Secret Data at Rest | Kubernetes
2. OpenShift: Encrypting etcd data | Security and compliance | OpenShift Container Platform 4.13
3. K3s: Secrets Encryption | K3s

Readers can explore these for further details.

Using KMS Providers for Data Encryption

In addition to the etcd (at-rest) encryption options above, native K8s and some K8s distributions also provide (dynamic) data encryption using KMS providers.

Here are some relevant reference documents:

1. Native K8s: Using a KMS provider for data encryption | Kubernetes
2. GKE: Encrypting Secrets at the Application Layer | Google Kubernetes Engine (GKE) | Google Cloud
3. Amazon EKS: Enabling secret encryption on an existing cluster - Amazon EKS
4. Using Alibaba Cloud KMS for Secret Encryption at Rest (alibabacloud.com)

Public Cloud/Private Cloud/Data Center Disk Encryption Options

Node-level encryption using EBS in K8s across public cloud/private cloud/data centers can provide an additional layer of encryption. Here are some public cloud examples:

1. AWS: When hosting a K8s cluster on AWS, you can enable Amazon EBS encryption to provide encryption for EC2 instances. Amazon EBS encryption uses AWS KMS keys when creating encrypted volumes and snapshots. It uses AES-256-XTS for block cipher encryption. When you create an encrypted EBS volume and attach it to a supported instance type, the following types of data are encrypted:

   • Data at rest inside the encrypted volume
   • All data moving between the volume and the instance
   • All snapshots created from the encrypted volume
   • All volumes created from those snapshots

2. Azure: Provides encryption options for Azure Managed Disks connected to Azure Key Vault.

3. Google provides encryption options for Google Cloud Storage. Both use AES 256 keys by default, but customer-managed and customer-supplied keys can also be used, with KMS integration.

Secrets Using Third-Party Secrets Store Integration

A key reason for choosing a third-party Secrets store is to ensure that the lifecycle of Secrets is managed outside the cluster through a centralized Secrets storage solution. The authentication and authorization policies and procedures provided by these Secrets stores differ from those on the cluster and may be better suited for controlling application data access. Most of these solutions also provide envelope encryption and HSM support required by regulatory bodies. Popular solutions include HashiCorp Vault, CyberArk Conjur, AWS Secret Store, Azure Key Vault, Google Secret Manager, 1Password, and more.

Sidecar Solutions

Solutions like Vault can be used to inject specific Secrets into application pods. In this case, the sidecar/init container is responsible for authenticating with the Secret Provider, and the application can then use the returned Secrets as needed. The connection to the Provider is made over TLS to ensure the security of Secret retrieval. Vault provides additional security through response wrapping, which allows you to pass credentials without intermediaries being able to see them. Customers choosing these solutions can decide whether to store secrets on the cluster or off the cluster. Typically, if customers have been using Vault for their infrastructure and other application needs, they tend to integrate with these solutions for a seamless secret management experience on K8s.

Secrets Store CSI (SSCSI) Driver and Provider Solutions

The Secrets Store CSI Driver allows Secrets and other sensitive information to be mounted as volumes into application pods. The Secrets Store CSI Driver communicates with providers via gRPC to retrieve Secret content from external Secrets stores specified in SecretProviderClass custom resources. Once the volume is attached, the data is loaded into the container’s filesystem. Unlike the sidecar solutions described above that bring in Secret content from a specific provider, the SSCSI driver can be configured to retrieve Secret content from multiple different Secret Providers. For more information on how the driver and providers work, see here.

Customers who prefer not to store secrets in etcd as Kubernetes Secrets primarily choose SSCSI for the following reasons:

• They may have strict compliance requirements, making it necessary to store and manage secrets only in a central store rather than in the cluster.
• They may be deploying workloads in environments where the control plane is not managed by them, so they want full control over their workload secrets without trusting the platform administrator. For example, customers deploying workloads into tenants on a managed service provider cluster, or into a cloud platform where the control plane is not managed by them.

The SSCSI driver does not directly provide a way to protect non-volume-mounted secrets, such as those needed as environment variables or image pull secrets, or Secrets you might create directly on the cluster for managing Ingress certificates. However, you can use the sync secrets feature, which creates Kubernetes Secrets and then provides support for Secrets used as environment variables.

External Secrets Operator (ESO)

External Secrets Operator (ESO) is a user-friendly solution for syncing secrets from external secret management solutions into Kubernetes Secrets. ESO runs as a Deployment resource in the Kubernetes cluster, using CustomResourceDefinitions (CRDs) to configure access to Secret Providers via SecretStore resources and to manage Kubernetes Secret resources via ExternalSecret resources.

Customers choose ESO in the following scenarios:

• They need easy integration with the platform and developer-friendly usage.
• They have high trust in the cluster’s control plane — especially regarding how etcd encryption is configured or how RBAC is managed on the cluster.
• They have multi-cluster use cases for secret management and need cross-cluster secret integration.
• They need to manage platform Secrets for non-application purposes, such as secrets for Ingress, automation, and image pulling.
• They need to modify Secrets on the cluster and provide templates for specific applications.
• Last but most importantly, their use cases require Secrets on the cluster.

Integrating Applications with HSM (Hardware Security Module)

For the highest level of security, integrate applications or K8s with HSM (Hardware Security Module). Details omitted.

Conclusion

Today, we explored the various encryption options available for K8s and how each option protects sensitive data. You can make an informed choice based on your use case and actual situation.

Here are some personal recommendations from the author, for reference only:

• If you primarily use AWS, choose based on your security level: EBS encryption or KMS encryption.
• If you use K8s in a data center and have Secrets to manage outside the K8s cluster, HashiCorp Vault is recommended.
• If you use K8s in a data center and only need K8s Secrets encryption, etcd at-rest encryption and ESO are both viable options to consider.