No Application Could Reach the Database: An AppArmor Stacking Incident Misdiagnosed as a Network Failure

On 2026-10-29 18:08:56, four Ubuntu 26.04 nodes finished apt upgrade, which moved linux-image-generic from 7.0.0-38 to 7.0.0-42, and then rebooted them one by one.

After the reboot, Pods in three namespaces fell into CrashLoop one after another: rsshub (TTRss, my RSS reader; the database is PostgreSQL), lobe-chat (LobeHub, an AI application; the database is PostgreSQL), and semaphore (a third-party Ansible UI; the database is MySQL).

The errors reported by the applications were highly misleading: casdoor (used for authentication and login) in lobe-chat reported dial tcp 10.43.36.163:5432: connect: no route to host, while semaphore kept repeating Pinging database.

The first instinct was the network. In the end, that judgment turned out to be only half right. The real problem was that after the Ubuntu kernel upgrade, signal delivery inside containerd containers was intercepted by AppArmor stacking.

Symptoms: Several Clues That Contradict Each Other

1. The database container logs were almost empty, with no stack trace of a startup failure.
2. Databases without a readiness probe still showed Running 1/1.
3. semaphore and semaphore-mysql used hostNetwork, so their processes showed up directly in the host’s ps.

The third point is the easiest one to be misled by. Seeing the processes on the host makes it look like a host-level issue.

The real signal was that no route to host appeared on an in-cluster ClusterIP.

Ceph stayed HEALTH_OK throughout (storage was fine), and the Service’s EndpointSlices were normal too.

If the database is not Ready, the Service has no endpoints, so the applications failing to connect is merely a downstream symptom.

Layer One: 100% CPU in the Process Table

On the host, gosu mysql and gosu postgres each consumed about 100% CPU, yet neither mysqld nor postgres had started.

gosu is what the official images’ entrypoint uses to drop privileges. That it was stuck meant the container never exec’d the real database process. That also explains the empty database logs mentioned earlier.

Layer Two: perf and strace Point to Signals

The call stack captured by perf:

main.SetupUser
  -> syscall.Setgid
    -> syscall.AllThreadsSyscall
      -> runtime.osyield

strace analysis found nothing but endless sched_yield(). In other words, gosu was waiting for a result that would never come.

Go’s AllThreadsSyscall broadcasts SIGSETXID to every thread in the process, using that signal to make each thread complete setgid/setuid in sync. When signal delivery fails, the wait loop never exits.

At this point, the suspect shifted from the network to “whether signals can be delivered at all”.

Layer Three: bpftrace Catches EPERM

One kretprobe produced the following output:

bpftrace -e 'kretprobe:apparmor_task_kill { printf("%s %d\n", comm, retval); }'

gosu’s return value was -13, i.e. EPERM, and there were no signal_generate events at all.

Verifying directly inside the container: kill -TERM on a sibling process returned Permission denied, and the target process was still alive.

This proved that AppArmor was blocking signals in the middle.

Root Cause: Labels No Longer Match After Stacking

The task’s actual AppArmor label looked like this:

cri-containerd.apparmor.d//&unconfined

//& is the stacking notation, meaning an additional unconfined layer on top of the CRI profile.

containerd’s CRI profile contains rules such as signal ... peer=cri-containerd.apparmor.d, which rely on exact label matching to allow signal delivery inside the container. Once the label took the stacking form, it no longer matched.

With kernel 7.0.0-42 and the default sysctl parameter kernel.apparmor_restrict_unprivileged_unconfined=1, the way container profiles are transitioned was changed to stacking. The result: any signal delivery inside a container returns EPERM.

📝Notes: gosu was not the only victim. glibc’s multi-threaded setuid broadcast and postgres’s own backend signals all take the same path. This also explains why “database containers all failed to start” rather than an issue with one particular image.

Controlled Experiment: Ruling Out Storage, CNI, and Capabilities One by One

Same node, same mysql:8 image, changing only the container configuration:

Configuration	gosu behavior
privileged: true	Normal
appArmorProfile: Unconfined	Normal
capabilities.add: [ALL]	Normal
capabilities.add: [CHOWN]	Still spinning
kernel.apparmor_restrict_unprivileged_unconfined=0	Normal

The first three rows are workarounds used only for evidence gathering. The fourth row shows this has nothing to do with capabilities.

After the fix, both ClusterIP and PodIP were reachable.

The Fix: One sysctl Line, but It Must Be Persisted to a File

Temporary stopgap:

sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0

Added that line to the sysctl loop and persisted it in:

/etc/sysctl.d/90-homelab-prerequisites.conf

The 90- in the filename is deliberate. Ubuntu’s own /usr/lib/sysctl.d/10-apparmor.conf sorts earlier, so only a 90- file that sorts later can override it.

Why the Pods Had to Be Recreated

The AppArmor label is fixed at the moment the container is created; changing the sysctl does not make already-running containers heal themselves.

This time I recreated lobe-chat/{postgresql,casdoor,lobe}, semaphore/{semaphore-mysql,semaphore}, and rsshub/database-postgres; the remaining Pods were left to roll naturally.

Why the Other Options Were Rejected

1. Roll the kernel back to 7.0.0-38. The old kernel is still in /boot, and this preserves Ubuntu’s default hardening. The cost is rebooting four nodes in sequence, plus an apt pin to keep the upgrade from coming back.
2. Use Unconfined / privileged / capabilities: ALL per Pod. This only treats the symptom, and it strips away a layer of isolation.
3. Modify the containerd CRI profile to add signal rules for the stacking peer. The profile is loaded from containerd’s built-in copy, and k3s offers no override hook, so this path is a dead end.
4. Turn AppArmor off entirely. The blast radius is too large; rejected.

Verification Checklist

1. This sysctl is 0 on all four nodes.
2. No CrashLoop and no non-Running Pods in the three namespaces.
3. Two pg_isready checks and one mysqladmin ping pass.
4. lobe / casdoor / semaphore all return HTTP 200.
5. kill -TERM inside the container returns kill_rc=0, and the target process is TERMINATED.

Risks and Follow-ups

Disabling this restriction amounts to rolling back one of Ubuntu’s hardening measures. Containers in the cluster do not use unprivileged userns, so the exposure is limited and acceptable for a while. It should be re-evaluated once the kernel is fixed.

The stacking behavior in 7.0.0-42 is incompatible with the signal rules in containerd’s CRI profile. Both Ubuntu and containerd have received this feedback, and most likely containerd will fix it; going forward I will check the containerd and k3s changelogs every month. Related bug reports:

• apparmor: allow signals and ptrace for stacked profile peers | containerd
• Regression 7.0.0-42/-43: Go 1.24 programs livelock in setuid (gosu); official DB Docker images hang on start | Ubuntu

Summary

After the Ubuntu kernel upgrade, signal delivery inside containers was intercepted by AppArmor stacking: gosu spun, database logs were empty, and applications reported no route to host. This post documents the investigation from the process table and perf through strace to bpftrace, along with the fix — persisting the sysctl and recreating the Pods — and continues to track the upstream fix.

📚️References

• AppArmor Wiki
• containerd CRI configuration documentation
• sysctl.d man page
• On-site evidence gathered during this incident (perf / strace / bpftrace output, controlled experiment table)