K8s Best Practices Updated: Use Node Swap!

This article was last updated on: November 9, 2026 pm

Overview

K8s best practices have been updated:

Provided the following conditions are met:

  • Kubernetes 1.34 or later
  • The node’s system disk is local SSD or NVMe high-speed storage
  • cgroup v2

use node swap!

To enable it, the recommended kubelet config addition is:

1
2
3
failSwapOn: false
memorySwap:
  swapBehavior: LimitedSwap

👍Benefits

👍️Pros

  • Saves money: eligible workloads can swap out idle anonymous pages under memory pressure, so pod density goes up significantly;
  • Better stability: memory spikes now have a buffer, and being OOM-killed is no longer the only option;
  • Small change footprint: the host swap file, partitions, and /etc/fstab are untouched — the only change is a single kubelet config item;
  • Observability is already in place: use node.status.nodeInfo.swap.capacity and kubectl top nodes --show-swap to inspect swap.

📝Notes:

  • Only Burstable containers with request < limit benefit; Guaranteed workloads see no difference at all;
  • The scheduler is not swap-aware; capacity planning still follows the node’s physical memory, and swap cannot be counted toward it;
  • What gets swapped out is anonymous pages, which hit the disk as I/O — do not enable this if the system disk is not local SSD or NVMe high-speed storage.

The common community view is: treat swap as insurance against memory spikes, not as a replacement for RAM.

What follows is my hands-on summary from the Homelab:

Homelab Cluster Overview

Four N100 mini PCs, cgroup v2, and the host /swap.img has long been swaponed (if swap isn’t enabled, you’ll need to enable it first in practice). They run k3s v1.36.5+k3s1.

NodeSwap went GA in Kubernetes v1.34, so let’s enable it and see.

Swap Defaults

kubelet’s memorySwap.swapBehavior defaults to NoSwap.

Under that default, even with swap enabled on the host and even if you set failSwapOn: false, the container cgroup’s memory.swap.max is still 0.

📝Notes: failSwapOn: false controls “whether kubelet should refuse to start when the node has swap”; swapBehavior controls “whether containers can use swap”. Two switches, two different things.

On my Homelab cluster, the host-level swapon, partitions, and /etc/fstab are all already in place. All that’s missing is that one line in kubelet.

Three Things to Confirm Before You Start

  1. cgroup v2: stat -fc %T /sys/fs/cgroup returns cgroup2fs;
  2. Swap is enabled on the host: swapon --show produces output — on my Homelab cluster it’s /swap.img;
  3. The Kubernetes version corresponding to your k3s version already includes NodeSwap GA (>=1.34).

If the prerequisites aren’t met, changing the kubelet config is meaningless.

Which Pods Does LimitedSwap Actually Apply To

The upstream-recommended value is LimitedSwap. It is not an “open the floodgates” policy.

The applicable boundaries are defined upstream:

  1. It only takes effect for Burstable containers;
  2. and requires memory request != limit;
  3. Guaranteed and BestEffort are always NoSwap;
  4. Critical pods (IsCriticalPod) are also NoSwap, regardless of QoS.
Pod QoS request vs limit Swap allowed
Burstable request != limit ✓ LimitedSwap
Burstable request == limit ✗ NoSwap
Guaranteed request == limit ✗ NoSwap
BestEffort neither set ✗ NoSwap
Any IsCriticalPod ✗ NoSwap

The quota is computed with this formula:

1
containerSwapLimit = (containerMemoryRequest / nodeTotalMemory) × totalPodsSwapAvailable

The smaller the request, the less room there is to swap out. This is deliberate: swap exists only to absorb memory spikes and does not participate in capacity planning.

🐾 Note: The scheduler is not swap-aware. The memory it sees is still the node’s physical RAM. Don’t treat swap as “extra memory” to oversell, or the node will eventually wobble.

Implementation: Modifying the k3s kubelet config

On every start, k3s rewrites /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf.

To override the defaults, you have to drop a file that sorts after it into the same directory.

The filename must end in .conf, and files are loaded in lexicographic order. Here I use 90-swap.conf.

1
2
3
4
5
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
memorySwap:
  swapBehavior: LimitedSwap

Write failSwapOn: false explicitly; don’t rely on the default.

Do not overwrite or delete 00-k3s-defaults.conf; k3s manages that one itself.

Rollout

Restart kubelet — for k3s, that means restarting the k3s service.

Rollout order: workers first, then masters one at a time.

Between each node, confirm kubectl get nodes shows everything Ready and the etcd members are healthy before touching the next one. Restarting multiple masters at once briefly loses etcd quorum.

The whole process restarts no nodes and evicts no pods.

Verification Checklist

Writing the config isn’t the end of it; walk through four commands one by one.

  1. The config kubelet actually has in effect:
1
2
kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" \
  | jq '.kubeletconfig | {failSwapOn, memorySwap}'

Expect memorySwap.swapBehavior == "LimitedSwap".

Example:

1
2
3
4
5
6
7
❯ kubectl get --raw "/api/v1/nodes/n100-cheshi-0/proxy/configz" | jq '.kubeletconfig | {failSwapOn, memorySwap}'
{
  "failSwapOn": false,
  "memorySwap": {
    "swapBehavior": "LimitedSwap"
  }
}
  1. Node swap capacity is reported, not unknown:
1
2
3
4
5
❯ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.swap.capacity}{"\n"}{end}'
n100-cheshi-0	8589930496
n100-jumper-0	4294963200
n100-jumper-1	4294963200
n100-jumper-2	4294963200

All four nodes should have a value.

  1. Swap usage is visible:
1
2
3
4
5
6
❯ kubectl top nodes --show-swap
NAME            CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)
n100-cheshi-0   671m         16%      11088Mi         35%         0Mi           0%
n100-jumper-0   853m         21%      8065Mi          53%         0Mi           0%
n100-jumper-1   759m         18%      6317Mi          41%         0Mi           0%
n100-jumper-2   466m         11%      7270Mi          47%         0Mi           0%

Look at the last two columns — SWAP(bytes) SWAP(%).

  1. The most important one: get inside a container and check the cgroup:
1
kubectl exec <pod> -- cat /sys/fs/cgroup/memory.swap.max

Compare two pods:

  • A Burstable pod with request 64Mi / limit 256Mi has memory.swap.max > 0;
  • Pods with request == limit, and pods with neither request nor limit set, have memory.swap.max == 0.

If these two sets of numbers line up, the boundaries are working as expected. Example:

Burstable Pod:

1
2
cat /sys/fs/cgroup/memory.swap.max
36200448

Pod with neither request nor limit set:

1
2
cat /sys/fs/cgroup/memory.swap.max
0

I couldn’t find a pod with request == limit in my homelab cluster at the time, so I won’t show that here.

Summary

  1. NodeSwap went GA in v1.34, but the default swapBehavior is still NoSwap; setting only failSwapOn: false will not give containers any swap quota;
  2. Only one config item needs to change: memorySwap.swapBehavior: LimitedSwap;
  3. The effect is scoped to Burstable with request != limit, and the quota is allocated in proportion to the request’s share of node memory;
  4. k3s delivers it through the drop-in directory: .conf suffix plus a lexicographic position after 00-k3s-defaults.conf;

That’s all.

📚️References