Rook-Ceph v1.20.0 CSI ServiceAccount Naming Mismatch Bug and Fix

Preface

Do NOT upgrade the rook-ceph helm chart to v1.20.0

A hard lesson learned! It nearly took down my Homelab and caused data loss.

Here’s what happened:

1. Renovate detected a new rook-ceph helm chart version v1.20.0
2. CI helm diff passed without any anomalies
3. Git merged to main branch, ArgoCD upgraded to v1.20.0
4. After upgrade, discovered 2 ctrlplugin pods couldn’t start due to missing SAs
5. Searched GitHub issues for a solution, found this issue: v1.19.0 failed to delete volumeattachment due to secret name missmatch. The suggested fix was to rollback. (My mistake — in the heat of the moment I didn’t notice this issue was for a different version. Mine was v1.20.0.)
6. (My mistake) Git reverted and pushed, ArgoCD synced. Some deployments got permanently stuck.
7. (My mistake) Deleted the stuck deployments, but ArgoCD couldn’t recreate them after deletion.
8. (My mistake) All cluster storage went unhealthy. Git became unreachable too (it used rook-ceph storage).
9. (My mistake) Disabled ArgoCD auto-sync and related features.
10. Reinstalled v1.20.0 and applied a temporary workaround.

Background

After upgrading rook-ceph from v1.19.6 to v1.20.0, Ceph mon data was upgraded to Tentacle ondisk format (v20) and cannot be downgraded back to v19.x. Additionally, the v1.20.0 CSI operator has a ServiceAccount naming inconsistency bug.

Bug Symptoms

After upgrading to v1.20.0, the following Deployments and DaemonSets cannot create Pods:

Component	Error
rook-ceph.cephfs.csi.ceph.com-ctrlplugin	serviceaccount “cephfs-ctrlplugin-sa” not found
rook-ceph.rbd.csi.ceph.com-ctrlplugin	serviceaccount “rbd-ctrlplugin-sa” not found
rook-ceph.cephfs.csi.ceph.com-nodeplugin	nodes is forbidden: … cannot get resource “nodes”
rook-ceph.rbd.csi.ceph.com-nodeplugin	nodes is forbidden: … cannot get resource “nodes”

Root Cause

The CSI operator (ceph-csi-controller-manager) in Rook v1.20.0, when creating CSI Deployments/DaemonSets, references ServiceAccount names using the legacy naming style (without the ceph-csi- prefix):

• cephfs-ctrlplugin-sa
• rbd-ctrlplugin-sa
• cephfs-nodeplugin-sa
• rbd-nodeplugin-sa

However, the Helm chart creates ServiceAccounts using the new naming style (with the ceph-csi- prefix):

• ceph-csi-cephfs-ctrlplugin-sa
• ceph-csi-rbd-ctrlplugin-sa
• ceph-csi-cephfs-nodeplugin-sa
• ceph-csi-rbd-nodeplugin-sa

The name mismatch causes Pods to fail ServiceAccount validation.

RBAC Permission Gap Chain

The legacy-style SAs created by copying are missing three layers of RBAC permissions, each causing different issues:

Layer	Missing Binding	Affected Component	Symptom
1. ClusterRoleBinding (cluster resources)	ceph-csi-{type}-ctrlplugin-crb, ceph-csi-{type}-nodeplugin-crb	ctrlplugin, nodeplugin	Cannot list csinodes/volumeattachments/persistentvolumes
2. RoleBinding (Leader Election)	ceph-csi-leader-election-rolebinding	ctrlplugin	CSI attacher/resizer/snapshotter cannot acquire leader lease, volume attach/detach times out
3. ClusterRoleBinding (Node permissions)	ceph-csi-{type}-nodeplugin-crb, {type}-csi-nodeplugin	nodeplugin	Cannot get nodes information

Cluster Environment

• Rook version: v1.20.0
• Ceph version: v20.2.1 (Tentacle)
• CSI Operator: quay.io/cephcsi/ceph-csi-operator:v1.0.1
• Kubernetes: k3s
• Deployment method: Helm umbrella chart (rook-ceph + rook-ceph-cluster + snapshot-controller)

Workaround

Step 1: Create the Missing ServiceAccounts

Copy the existing new-style SAs to create the legacy-style SAs:

# Create ctrlplugin SAs
kubectl get sa -n rook-ceph ceph-csi-cephfs-ctrlplugin-sa -o yaml \
  | sed 's/name: ceph-csi-cephfs-ctrlplugin-sa/name: cephfs-ctrlplugin-sa/' \
  | kubectl apply -f -

kubectl get sa -n rook-ceph ceph-csi-rbd-ctrlplugin-sa -o yaml \
  | sed 's/name: ceph-csi-rbd-ctrlplugin-sa/name: rbd-ctrlplugin-sa/' \
  | kubectl apply -f -

# Create nodeplugin SAs
kubectl get sa -n rook-ceph ceph-csi-cephfs-nodeplugin-sa -o yaml \
  | sed 's/name: ceph-csi-cephfs-nodeplugin-sa/name: cephfs-nodeplugin-sa/' \
  | kubectl apply -f -

kubectl get sa -n rook-ceph ceph-csi-rbd-nodeplugin-sa -o yaml \
  | sed 's/name: ceph-csi-rbd-nodeplugin-sa/name: rbd-nodeplugin-sa/' \
  | kubectl apply -f -

Step 2: Bind RBAC Permissions (All Three Layers)