Rook-Ceph Data Repair in Practice

This article was last updated on: November 9, 2026 pm

Introduction

The National Day holiday is finally here! Nothing beats staying home and tinkering with my own HomeLab, K8s, and NAS server cluster. I really messed with a lot of things during this long holiday — readers who are interested can look forward to a flood of upcoming posts, and if you can’t wait, you can go straight to the PRs and Commits in my repo east4ming/homelab2.

Today’s post is the first one.

Here’s what happened: over the National Day holiday, I upgraded all the nodes in my HomeLab’s Rook-Ceph cluster from Ubuntu 24.04 to 26.04.1. One of them, n100-cheshi-0, ran into trouble during the upgrade, and no matter how many times I rebooted, it wouldn’t come back. In a moment of 🧠 heat and panic, I cut the power outright — GG 😭 — and it stayed down for hours. Fortunately, I got it fixed in the end. (That’s another article too — stay tuned.)

After the node came back online, I assumed Ceph would recover on its own, since the disks were all still there and no PGs were reporting errors. Then I ran ceph -s — well, well: only 3 of the 4 OSDs were online. OSD 1 was lying down in down/out, playing dead, and ceph osd df showed its SIZE as 0 B.

Honestly, at first my head was full of worst-case scenarios: “the disk died,” “BlueStore metadata is corrupted,” “I’ll have to rebuild the OSD.” It took me half the night to realize that the 4 TB of data on the disk was completely fine — it was the Rook operator that had decided it “didn’t recognize” it.

This article documents the full path from diagnosis to repair. The focus is on the diagnostic approach: using a three-way FSID comparison plus source-code verification to turn what looked like data corruption into an expired field in a Secret.

📝 Disclaimer:

  • This article is not an ad or a promotion — it’s purely a record of lessons learned the hard way.
  • The environment is a 4-node K3s + Rook-Ceph (4 OSD) homelab, not production. It’s only meant to give readers some ideas — don’t copy it blindly.

Background: Is autoout the Culprit?

Let’s first clarify why an OSD gets kicked out. Ceph has a parameter mon_osd_down_out_interval, defaulting to 600 seconds. If an OSD stays unreachable longer than that, the monitor marks it out and zeroes out its CRUSH weight — this is what’s called autoout.

The node was powered off for hours, so OSD 1 had long exceeded the timeout, and therefore:

  1. OSD 1 was autoout’ed and its CRUSH weight zeroed;
  2. Its data was migrated to the remaining 3 OSDs;
  3. The cluster returned to HEALTH_OK, with all 81 PGs active+clean.

Taken on its own, this is designed behavior, not a failure. The real problem came afterward: the node came back, but OSD 1 did not heal itself.

Criterion 1: A Missing Deployment ≠ a CrashLooping Pod

My first move was to look at the Pods:

1
kubectl -n rook-ceph get deploy | grep osd

The output contained only rook-ceph-osd-0/2/3; there was no rook-ceph-osd-1 Deployment at all.

That signal is very telling. Here’s how Rook goes about managing OSDs:

Symptom Meaning
Deployment exists, Pod is CrashLoop / Pending The operator accepted the disk; the problem is at runtime, in scheduling, or with the device
Deployment doesn’t exist at all The operator never adopted the disk during the preparation phase; the Pod was never created

In other words, OSD 1 was not a runtime failure — the operator deliberately skipped it. That distinction pulled me straight from “the disk is broken” to “why won’t the operator recognize it.”

The 0 B SIZE for OSD 1 in ceph osd df follows the same logic: the disk isn’t gone, it simply was never brought under management.

Criterion 2: The osd-prepare Log Is the Decisive Evidence

Rook’s OSD adoption flow is: the rook-ceph-osd-prepare job scans the disks first and decides whether they’re usable; only then does the operator create the corresponding Deployment. So the real answer lives in the prepare job’s logs.

1
kubectl -n rook-ceph logs job/rook-ceph-osd-prepare-n100-cheshi-0

The key lines (simplified):

1
2
3
4
5
6
osd_id: 1
type: bluestore
ceph_fsid: abb2c4e2-12a3-4b93-8d90-f2eaf2290901
...
skipping osd.1 ... belonging to a different ceph cluster
0 ceph-volume raw osd devices configured on this node

When I saw belonging to a different ceph cluster, I was a bit baffled: where would I have gotten a second Ceph cluster?

But the log also told me the metadata on the disk was valid and intact: osd_id 1, type bluestore, ceph_fsid abb2c4e2-…. The problem wasn’t the disk — it was that Rook believed the “current cluster’s FSID” didn’t match the FSID on the disk.

Three-Way FSID Comparison

Following that thread, I laid the FSIDs from three sources side by side:

Source Command / Location FSID
Ground truth of the running cluster ceph fsid abb2c4e2-12a3-4b93-8d90-f2eaf2290901
OSD 1 disk BlueStore metadata ceph_fsid in the prepare log abb2c4e2-… (consistent)
rook-ceph-mon Secret .data.fsid f568f7c3-5603-4108-99ff-3742cf008a83 (stale)

Put those three rows together and the answer emerges:

  • Cluster ground truth = disk metadata = abb2c4e2-… ✅
  • Only the fsid in the rook-ceph-mon Secret was the old f568f7c3-… ❌

The operator took the stale FSID from the Secret, compared it against the disk, naturally concluded “this isn’t my disk,” and skipped adoption. The disk, the data, the PGs — all fine.

Source-Level Verification: Why Doesn’t This Field Self-Correct?

Here I got a bit more careful: what if it was just a coincidence, and the field would drift again next time? So I dug into the Rook upstream source at pkg/operator/ceph/controller/cluster_info.go and searched for fsidSecretNameKey. There are only three occurrences:

  • L52: constant definition
  • L154: read (used when constructing ClusterInfo)
  • L423: write, and only when the Secret doesn’t exist and a new cluster is being created

In other words, no task ever “auto-corrects fsid when it detects a mismatch.” Once it’s written wrong, it stays wrong. This also explains why OSD 1 could never self-heal after the node recovered: this is not a failure that waiting can fix.

📝 Notes: This conclusion gave me confidence in the entire repair plan. Changing the value in the Secret is enough; the disk doesn’t need to be touched.

The Fix: patch, Not delete

The fix is a single command. The principle is to change only the fsid field, leaving ceph-secret, ceph-username, and mon-secret untouched.

1
2
3
NEW_FSID=$(ceph fsid)
kubectl -n rook-ceph patch secret rook-ceph-mon --type merge \
  -p "{\"data\":{\"fsid\":\"$(echo -n "$NEW_FSID" | base64 -w0)\"}}"

Then have the operator reload ClusterInfo:

1
kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator

🐾 Caution: Do not, for the sake of convenience, delete and recreate the rook-ceph-mon Secret. It carries a DisasterProtectionFinalizer, and deleting it would very likely be treated by the operator as bootstrapping a “new cluster” — that would be a real disaster.

Verification: From 0 devices to 1 devices

After restarting the operator, check the prepare log again:

1
1 ceph-volume raw osd devices configured on this node

Going from 0 to 1 is the signal of successful adoption. Next:

1
2
3
kubectl -n rook-ceph get deploy rook-ceph-osd-1   # 1/1 Ready
ceph osd tree                                      # osd.1 up/in
ceph osd df                                        # crush weight 0.86850

OSD 1 re-registered automatically and its CRUSH weight was restored, with no manual ceph osd in or ceph osd crush reweight at any point.

Side Effect: A Rolling Restart of All OSDs

When the operator pushed out the new OSD configuration, it triggered a rolling restart of all 4 of my OSDs. The cluster briefly entered HEALTH_WARN:

Metric Value
osds down 1
objects degraded 11648 / 44646 ≈ 26.090%
pgs degraded 23

The moment I saw 26% degraded, my heart skipped a beat 😅. But this is a normal transitional state: throughout, all PGs still met their replica requirements — no undersized or incomplete appeared. The cluster recovered on its own after about 1–2 minutes.

Closing the Loop: HEALTH_OK ≠ Data Intact

In a power-loss scenario, HEALTH_OK is the last thing you should trust. A healthy PG state only means “enough replicas” — not that “the data wasn’t corrupted.”

So I ran a deep-scrub across all 81 PGs:

1
2
ceph pg deep-scrub $(ceph pg ls | awk 'NR>1{print $1}')
ceph health detail

Result: all deep-scrub ok, 0 inconsistent objects, HEALTH_OK. Only then was the job truly done.

Preventive Measures

A few lessons from the trenches, for your reference:

  1. Before upgrading, run ceph osd set noout, and unset noout when done, to prevent autoout from triggering a full data migration;
  2. Prepare a UPS for node upgrades, and don’t force a shutdown on impulse — a few-hundred-dollar UPS is far cheaper than rebalancing several TB. This time, the power cut was the lesson;
  3. Back up the rook-ceph-mon Secret so you have a baseline to compare against when something goes wrong;
  4. Always deep-scrub after recovery — don’t just see HEALTH_OK and shut down your computer.

Summary

Looking back, the “devil” in this failure was hiding in a field no one had ever noticed. The disk was intact, the data was intact, the PGs were intact — what was broken was the stale FSID in the rook-ceph-mon Secret, which will never auto-correct. And the “read-only, never write” behavior in the Rook upstream source meant it could only be fixed by hand.

The criterion “a missing Deployment ≠ a CrashLooping Pod” saved me at least a night of thrashing around. The three-way FSID comparison plus source-level verification were the two key steps that turned “mysticism” into “certainty.”

Knowledge gained from books is ultimately shallow; only by practicing it yourself do you truly understand. OSD 1 was never sick at all — it had just been misidentified. Once that was clear, the fix was a single kubectl patch — concise and elegant.

That’s all.

📚️ References