Ubuntu 24.04 LTS In-Place Upgrade to 26.04 LTS: 4 N100 Nodes, Real-World Pitfalls, and a Must-Check Checklist

Introduction

The National Day holiday is finally here! There’s nothing better than staying home and tinkering with my HomeLab, K8s, and NAS server cluster! I really got a lot done this holiday—readers who are interested should look forward to the upcoming flood of posts; if you can’t wait, you can go straight to the PRs and commits in my repo east4ming/homelab2.

Those 4 N100 mini PCs in my Homelab run K3s + Rook Ceph and had been sitting on Ubuntu 24.04 LTS (Noble Numbat) all along. Once 26.04 LTS (Resolute Raccoon) came out, I was itching to upgrade—after all, 24.04’s support window is ticking away day by day, and sooner or later it has to be done.

📝Disclaimer:

• This article is not a “translation of the official upgrade guide”; it’s a retrospective after actually rolling through all 4 nodes
• The generic tutorials that say just run do-release-upgrade and then reboot don’t cut it on a cluster with Ceph

What really made me decide to write this article was the chain of failures after the upgrade that “looked unrelated but were actually one chain”: /etc/sysctl.conf deleted → inotify limits rolled back → fwupd failing every hour → KubeVirt virt-handler CrashLoopBackOff; plus a Ceph OSD stuck in down+out, whose root cause turned out to be a stale fsid in the mon Secret.

Why Did This Upgrade Take So Long?

According to Canonical’s official release cadence, the release dates for Ubuntu 26.04 LTS and its first point release, 26.04.1, are as follows—and the upgrade path for 24.04 LTS users was indeed delayed by more than a month.

📅 Release Dates for Each Version

• Ubuntu 26.04 LTS (Resolute Raccoon): officially released on April 23, 2026.
• Ubuntu 26.04.1 LTS: released on August 27, 2026. As the first point release, it rolls up the security updates and bug fixes since the initial April release.
• 24.04 LTS → 26.04.1 LTS upgrade path: officially enabled on September 29, 2026. It landed right on the National Day holiday—perfect!

⏳ Why the Upgrade Was Delayed

Although 26.04.1 was released at the end of August, Canonical did not immediately enable automatic upgrades for 24.04 LTS users. From August 27 to September 29, the upgrade path was delayed by about 33 days.

This delay is a standard strategy Canonical uses to ensure upgrade stability; the specific reasons are as follows:

1. Waiting for the first point release: Under Ubuntu’s Long Term Support (LTS) policy, the automatic upgrade path from one LTS release (e.g. 24.04) to the next (e.g. 26.04) is usually not enabled immediately when the initial version ships, but waits until the first point release (i.e. 26.04.1) is out. This gives Canonical months to fix serious issues found in the initial release and ensure the reliability of the cross-release upgrade.

2. Additional critical backports: This is the direct reason the upgrade path was not enabled on the day 26.04.1 was released. Canonical engineer Oliver Reiche explained on the mailing list that they needed to deploy some extra “backport” fixes before enabling automatic upgrades. These fixes mainly target rust-coreutils, the core system toolset rewritten in Rust, and address “regressions” in recent versions (i.e. previously working functionality that got broken). Stability of core system tools is critical, so Canonical chose to fix these potential risks first before pushing the upgrade to the wider 24.04 user base.

3. Phased rollout: Even after the upgrade path is officially enabled, Canonical rolls out upgrade notifications gradually over several days to avoid overloading servers and to let early adopters report issues.

Overall, this delay was technical and prioritized stability. For production environments, waiting for the officially recommended upgrade path is the safer choice, because standard security maintenance for Ubuntu 24.04 LTS continues until May 2029—there’s no need to rush the upgrade.

Upgrade Path: No Skipping Levels

Let’s get the path straight first—there’s no trickery here:

Current Version	Can You Upgrade Directly to 26.04 LTS?	Notes
24.04 LTS	✅ Yes	24.04 is the direct upgrade baseline for 26.04; one step does it
22.04 / 20.04	❌ No	You must first upgrade to 24.04 LTS, then to 26.04
25.10	✅ Yes	Non-LTS goes through the normal upgrade path

26.04 LTS is supported until 2031-04, meaning these 4 nodes won’t need another major OS version bump for the next 5 years—that matters a lot for a Homelab.

As for the upgrade path, do-release-upgrade only offers LTS → LTS upgrades when Prompt=lts. If it still tells you “no new release found” (for example, the first point release hasn’t entered the regular channel yet), then use:

sudo do-release-upgrade -d

📝Notes: -d uses the devel channel and will give you a release that hasn’t officially entered the regular upgrade channel yet. Fine for tinkering at home, but for production, wait for the point release.

Before the Upgrade: Protect What You Can

The core idea in one sentence: the nodes themselves can be lost, but etcd and cluster data cannot. All configuration lives in the Ansible + GitOps repo, so nodes can be reinstalled; but etcd snapshots and Ceph data have no second copy.

On the cluster side I did three things:

# 1. Take a manual etcd snapshot
sudo k3s etcd-snapshot save --name pre-26.04

# 2. Archive node status
kubectl get nodes -o wide > ~/pre-upgrade-nodes.txt

# 3. Archive Ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s > ~/pre-upgrade-ceph.txt

On the node side, first bring 24.04 up to date and reboot once, then start the cross-release upgrade. The order is a rolling upgrade, workers first and control-plane last, one machine at a time—after each node, wait for the node to be Ready and pods to stabilize before touching the next one.

General Upgrade Steps

# 1. Backup！Backup！Backup！
# 2. Update Ubuntu 24.04 LTS
# 2.1 apt
sudo apt update
sudo apt upgrade
# （optional）remove packages that are no longer needed
sudo apt autoremove --purge
# 2.2 snap
sudo snap refresh

sudo reboot

# 3. Check the Upgrade Prompt Setting 
sudo sed -i 's/Prompt=.*/Prompt=lts/' /etc/update-manager/release-upgrades

# 4. Upgrade to Ubuntu 26.04 LTS from 24.04 LTS
sudo do-release-upgrade

# Choose yes or no as prompted

# 5. Post-Upgrade
sudo snap refresh
sudo apt update
sudo apt upgrade
# optional
sudo apt autoremove --purge

# Finished🎉

⚠️ Don’t Rush to Run apt autoremove --purge

This one deserves its own section, because it contradicts what the generic tutorials advise.

On the nodes in this repo, the linux-firmware dependency chain has been deliberately modified—what’s installed is a hollow shell package, linux-firmware-minimal (that’s another story, stay tuned). This means that if you casually run apt autoremove --purge, there’s a good chance it takes the i915 / Wi-Fi firmware with it, and then you’re in for a reinstall or a story about plugging in an Ethernet cable.

If you really must clean up, first confirm the firmware package you want to keep is marked as manual:

apt-mark showmanual | grep linux-firmware

Post-Upgrade Must-Check Checklist

Below are the issues that actually occurred after the upgrade—not “ones you might run into”. I’ve ordered them by severity.

1️⃣ /etc/sysctl.conf Was Deleted Outright (Biggest Trap)

Ubuntu 26.04’s procps marks /etc/sysctl.conf as remove-on-upgrade. During the upgrade it is deleted outright, even if you modified it, even if you chose “keep the current version”.

The consequence is that any Ansible task that writes kernel parameters into this file silently stops working after the upgrade:

Parameter	Before Upgrade	After Upgrade (Reverted to Default)	Impact
fs.inotify.max_user_instances	8192	128	fwupd.service / fwupd-refresh.service fail every hour
net.ipv6.conf.all.forwarding	1	0	IPv6 forwarding stops working
net.ipv6.conf.all.accept_ra	2	1	RA acceptance behavior changes

The inotify item also triggered a chain reaction: on nodes where inotify instances were exhausted, KubeVirt’s virt-handler exited fatally during startup, showing as CrashLoopBackOff with Failed to create an inotify watcher in the logs.

The right approach: put the parameters in a dedicated file under /etc/sysctl.d/, for example 90-homelab-prerequisites.conf. After fixing it, delete the CrashLooping virt-handler Pod and it will come back up on its own.

📝Notes: Don’t try to “add it back” in sysctl.conf—it gets deleted every upgrade. /etc/sysctl.d/ is the right way.

2️⃣ Third-Party apt Sources Wiped to a Comment

do-release-upgrade can’t migrate third-party one-line .list files into deb822-format .sources. What it does instead: it empties the original file down to a single comment line and moves the original content to a same-named .disabled file.

That’s how the Tailscale source disappeared this time, and the upgrade tool conveniently flagged tailscale as an “obsolete package”.

When restoring it, note that the suite must be changed to the new codename resolute, and first confirm that upstream has actually published that suite—otherwise apt update will fail outright.

3️⃣ logrotate Fails Wholesale Due to Duplicate conffiles

After choosing “keep the current version”, conffiles removed by the release upgrade are left behind; two packages each keep a copy, their globs are identical, logrotate flags a duplicate, and then everything fails:

cloud-init-base:1 duplicate log entry for /var/log/cloud-init.log

This is Launchpad #2152107. The fix is simple: rename /etc/logrotate.d/cloud-init with a .disabled suffix (logrotate ignores that suffix).

4️⃣ netplan gateway4 Deprecation Warning

netplan.io 1.2 still accepts gateway4, but every netplan generate emits a warning:

gateway4 has been deprecated, use default routes instead

I’ve already changed the new install template to use routes:. But for nodes already running with working networking, don’t modify the cloud-init-generated netplan just to silence this warning—zero benefit, non-trivial risk.

5️⃣ Verifying Cluster Components One by One + One Ceph OSD down+out

On the cluster side I went through everything one by one: systemctl --failed expected to be empty, k3s status, all nodes Ready, Pods not in Running/Completed, ceph -s, KubeVirt virt-handler.

The result: osd.1 on n100-cheshi-0 was stuck in down+out. See the previous article for the specific fix.

How to Answer the Upgrade Prompts

Jotting down my choices this time, for reference:

1. Continue: y
2. Automatically restart services: Yes (choosing No means confirming each service individually, and it’s easy to miss one)
3. Locally modified configuration files: conservatively choose “keep the current version”, but you must diff each one
4. Remove obsolete packages: y

The upgrade logs for all 4 nodes are in /var/log/dist-upgrade/ (apt-term.log, main.log); when troubleshooting leftover issues, this is first-hand evidence.

IaC-Side Adjustments to Sync

See the code here: https://github.com/east4ming/homelab2/commit/e7044ced769a43cc2d28fa5bd7c98c15a729574d

Nodes are disposable only if the repo keeps up. What needed syncing this time:

• pxe_server/defaults/main.yml: tags / checksums for the ISO and netboot.xyz squash resources
• ubuntu.ipxe.j2: added a resolute menu entry and the :resolute_amd64 / :resolute_arm64 branches
• user-data.j2: libpcre3 / libpcre3-dev are removed in 26.04 (PCRE1 was dropped); the dnsutils transitional package is gone too, replaced by bind9-dnsutils—if you don’t change this, a fresh PXE install fails at the packages: stage; netplan gateway4 → routes
• prerequisites/tasks/main.yml: sysctl parameters moved to /etc/sysctl.d/
• scripts/firmware-slim and the 24.04 version descriptions in the docs

Conclusion

When it comes to cross-LTS upgrades, generic tutorials cover “how to get the system upgraded”, while the part that actually eats your time is “is the cluster still alive after the upgrade”. Among these checks, item 1 is the one I think most deserves a note of its own: a failure chain triggered by a silently deleted config file—the kind you’d never think of unless you read the logs.

📝Notes: The conclusion is actually quite plain—nodes can be lost, data cannot. So taking an etcd snapshot and archiving ceph -s before the upgrade are baseline actions, and checking items one by one after the upgrade is far more reliable than “it looks like there are no errors”.

The rear waves of the Yangtze push on the ones before, and each generation outdoes the last. System versions will keep changing; as long as you keep the evidence you should keep and put the parameters that belong in /etc/sysctl.d/ in the right place, that’s enough.

📚️References

• Official guide for upgrading from Ubuntu 24.04 LTS to 26.04 LTS
• Launchpad #2152107 - logrotate duplicate log entry for cloud-init.log
• netplan reference - gateway4 deprecation and default routes