How to Find Unused Resources in a K8s Cluster?

This article was last updated on: November 9, 2026 pm

Introduction

The National Day holiday is finally here! Staying home and tinkering with my HomeLab, K8s, and NAS server clusters feels great! I really got a lot done this holiday — readers who are interested should look forward to a flood of upcoming posts; if you can’t wait, you can go straight to the PRs and commits in my repo east4ming/homelab2.

My Virgo OCD and cleanliness compulsions kicked in, and I wanted to clean up the “unclaimed” resources in my Homelab cluster.

Honestly, once a cluster has been running for a while, everyone ends up in the same place: a pile of PVCs sitting there unused, a Service with no Pod behind it, ConfigMaps left over from the previous deployment, Deployments with 0 replicas whose objects still exist. None of this affects the business, but it’s unpleasant to look at, and it wastes storage and quota. Checking them one by one with kubectl get by hand? Across dozens of namespaces, that would take until next year.

So this practice came about: introduce yonahd/kor in a GitOps way, installing a “searchlight for unused resources” in the cluster. The process went smoothly overall, but I hit a very typical pitfall along the way: silent failure. No error, no alert — the dashboard was just empty. This article is mainly about that pitfall.

📝Disclaimer:

Everything in this article comes from hands-on testing in my own HomeLab; it is not an advertisement or a promotion. kor is an upstream open-source project, and this article merely records my experience using it.

Final Result

Kor dashboard 1

Kor dashboard 2

Kor dashboard 3

Kor Prometheus Alertrules

I haven’t received a single alert since configuring the alerting rules, so there are no related screenshots this time.

What kor Is, and How I Planned to Use It

kor is a Kubernetes unused-resource scanner that can detect orphaned Services, Deployments, ConfigMaps, Secrets, PVCs, HPAs, PDBs, and a whole roster of other resource types.

It has two modes:

Mode Form Output Enabled this time
prometheusExporter Long-running Deployment, exposes metrics Scraped by Prometheus ✅ Enabled
cronJob Scheduled Job Report output to logs / Slack ❌ Disabled

The reason CronJob mode was turned off is very practical: I don’t have Slack credentials here, so the report could only go into the Pod logs, which is no better than just looking at the metrics. So in the end I kept only the exporter mode.

The deployment form is a wrapper chart, placed in system/kor/:

1
2
3
4
5
6
7
8
# system/kor/Chart.yaml
apiVersion: v2
name: kor
version: 0.1.0
dependencies:
  - name: kor
    version: 0.2.16
    repository: https://yonahd.github.io/kor

Key trade-offs in the values:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
kor:
  cronJob:
    enabled: false
  prometheusExporter:
    enabled: true
    image:
      repository: yonahdissen/kor
      tag: v0.6.9
      pullPolicy: IfNotPresent   # replaces the upstream default of latest + Always
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 256Mi
  exporterInterval: 30m          # upstream default is 10m, relaxed to 30m

I changed the default configuration in two places:

  1. Pinned the image to v0.6.9 and changed imagePullPolicy to IfNotPresent. The upstream default is latest + Always, which in a Homelab means pulling the image on every restart, with the version liable to change at any moment. Pinning it plus IfNotPresent saves both effort and bandwidth.
  2. Relaxed exporterInterval from 10 minutes to 30 minutes. The reason is that each scan round performs a LIST across the entire cluster, which puts real pressure on the API Server. There’s no need to scan every 10 minutes in a Homelab; 30 minutes is plenty.

Measured steady-state overhead: about 10MiB of memory / 24m CPU, a 10.5MB image, and about 3 minutes for the first scan. This overhead is essentially negligible.

Integrating with Existing Prometheus and Grafana: Zero Changes Throughout

The nicest part this time: I didn’t touch any existing Prometheus / Grafana configuration.

Scraping relies on the ServiceMonitor bundled with the chart.

The dashboard lives in system/monitoring-system/ and is loaded by the existing Grafana sidecar.

The delivery path is clean too: no new ArgoCD Application manifests; the ApplicationSet’s git generator covers system/*, and the namespace is taken from the directory name, kor.

After going live, I confirmed:

  • up{job="kor-exporter"} = 1, with no Prometheus configuration changed
  • kubernetes_orphaned_resources produced 925 series after the first scan
  • exported_namespace had 41 distinct values

The Pitfall: Silent Failure Where namespace Is Always kor

Everything went smoothly up to that point; the pitfall was in adapting the dashboard.

kor’s metrics include the namespace where the resource lives. I dug through the kor source code, saw that the label name was namespace, and so very naturally wrote the dashboard query as:

1
sum by(namespace) (kubernetes_orphaned_resources)

Deployed, opened the dashboard.

Empty.

No error, no alert, the query returns legally — there’s just no data. My first reaction was that the dashboard JSON was wrong, but after checking for a long while I found no problem. Then I queried directly in Prometheus:

1
kubernetes_orphaned_resources

The results came back: every series did have a namespace label, but its value was always kor.

Another label, exported_namespace, had 41 distinct values — that was the real resource namespace.

The cause lies in prometheus-operator’s behavior: it attaches a set of target labels to every scraped series, including namespace=<the namespace where the exporter runs>, which is kor. The metric’s own label of the same name, namespace, conflicts with it and is therefore renamed to exported_namespace.

📝Notes: This is a general Prometheus rule: when a target label conflicts with a sample label, the sample label is prefixed with exported_. It’s not specific to kor; any exporter can run into it.

This is a classic silent failure: no error, no alert, perfectly valid query syntax — the result is just empty or always a single value. If you don’t go and check the actual value distribution of the live series, reading the source code and docs alone will never reveal it.

The fix is simple: change the query back to exported_namespace (or specify honorLabel):

1
sum by(exported_namespace) (kubernetes_orphaned_resources)

After the change I re-verified: sum by(exported_namespace) returned 42 groups — back to normal.

Safety Boundary: Provide Clues Only, Humans Decide on Cleanup

I think this section matters more than the technical details.

kor does have deletion capability, via the --delete flag. I didn’t configure it, nor enable it. The RBAC verbs are only:

1
verbs: ["get", "list", "watch"]

No delete, update, or patch. That is to say, even if kor wanted to delete something, it couldn’t.

There are two reasons:

  1. I position it as a “searchlight”, not a “janitor”. This feature’s job is to provide clues, not to take automated action.
  2. kor’s resource determination has known upstream false positives: Secrets / ConfigMaps referenced by CRDs or by dynamically loaded configuration are misjudged as “unused”. Deleting them automatically would be an incident.

So the cleanup process is: a human looks at the dashboard → a human judges → a human executes manually.

For the same reason, several other options were rejected:

  • CronJob mode: no Slack credentials, so a report that goes into the logs is as good as nonexistent.
  • kor’s automatic deletion: requires write permissions and adds false-positive risk; not worth it.
  • krew / local binary installation: outside GitOps, so configuration drift goes unmanaged.

kor Results

Unused resources found by Kor:

kind Count kind Count kind Count
ReplicaSet 486 Crd 127 Secret 90
ConfigMap 68 Job 40 Pvc 30
ClusterRole 26 VolumeAttachment 25 Service 11
ClusterRoleBinding 10 RoleBinding 7 ServiceAccount 4
StorageClass 1

The largest namespaces were rook-ceph (141), monitoring-system (62), kubevirt (43), rsshub (43), and tailscale (40).

Categorization and “Can It Be Deleted?”

The criteria come down to two easily confused dividing lines:

  1. Derived objects (those with ownerReferences) only look at the owner: RSs are created by Deployments, Jobs by CronJobs, PVCs by DataVolumes. Their labels are inherited from the parent object’s template — 258 of the 486 RSs carry argocd.argoproj.io/tracking-id, and judging by labels would misclassify them as “GitOps-managed” and make you afraid to clean them up.
  2. Only declarative objects are judged by GitOps markers: anything owned by ArgoCD/Helm will be recreated if deleted, which only creates drift.
1
2
3
Class A  Safe to batch delete      557   RS 486 / Job 32 / VolumeAttachment 25 / PVC 7 / Service 7
Class B  Needs manual confirmation  36   RBAC+CM 17 / VolSync cache 17 / prod-vm-datadisk / virt-exportproxy
Class C  Do not delete             332   Crd 127 / Secret 90 / GitOps-managed 85 / operator-owned 24 / DataVolume 5 / StorageClass 1

Safe to Delete (Class A)

Object Count Why it’s safe Real benefit
ReplicaSet (old revisions) 486 replicas=0 and all status counters are 0; deleting won’t recreate them, only loses rollout undo history Noise reduction only
Job (completed) 32 27 renovate history + kube-bench + 4 CronJob history Noise reduction only (logs kept for now)
VolumeAttachment 25 References PVs that no longer exist; CSI leftovers Zero — see below
Woodpecker PVC 7 Orphaned pipeline workspaces from 2026-05, no Pods, no owner Nominally 70 GiB, measured ~0.5 GiB
Woodpecker Service 7 Leftover headless Services from the same batch of pipelines Noise reduction only

The benefits must be stated honestly: etcd is only 71 MiB, and the 557 objects are mostly historical noise; rbd du measured the 7 WP PVCs at only about 0.5 GiB (one at 0 B, six at 80–84 MiB, RBD thin provisioning). Those 25 stale VAs bring no space savings whatsoever.

But I’m a Virgo with OCD — knowing something is useless and not cleaning it up makes me uncomfortable. 🤣

Needs Manual Confirmation (Class B)

I looked through it carefully and found 4 leftover loft-cluster-* ClusterRoles, plus loft-cluster-admin, loft-cluster-authenticated, vc-v-v-1-v-loft-default-v-v-v-1, and their 3 Bindings (that software was deleted long ago).
Deleted them — feels great~

Absolutely Must Not Delete (the three most critical)

  • kube-system/k3s-etcd-snapshot-s3-config — deleting it stops the cluster’s etcd backups dead; this is the last line of defense.
  • *-backup-repository / volsync-src-* — Restic repository passwords; once deleted, historical backups can never be decrypted again.
  • tailscale/<domain> TLS Secret — referenced by the Ingress’s spec.tls; deleting it breaks HTTPS on the spot.

None of the 127 CRDs get deleted: deleting a CRD cascades to its CRs, and they all belong to installed operators/charts, so they’d be recreated anyway. kubevirt-test/prod-vm-datadisk stays too — the datadisk of virtualmachine/prod-vm (Stopped) points to it.

How to Handle kor False Positives?

Two approaches:

  • Label the false-positive resources with kor/used=true
  • Add the --exclude-labels flag to the kor command: this time I specified tailscale.com/managed=true — more scalable than labeling them one by one

Adding kor Alerting Rules

Rule Expression What it covers
KorAuditNoData absent(kubernetes_orphaned_resources) for 1h Scraping hasn’t stopped but the metrics are all gone → scan stuck / RBAC broken (TargetDown won’t catch this)
KorUnusedResourcesGrowing 7-day growth > 150, for 6h Cleanup reminder
KorStaleVolumeAttachments count(...{kind="VolumeAttachment"}) > 10 for 12h CSI/node leftovers; the annotation asks to also check for orphaned Ceph images
KorUnmountedPvcSpike Unmounted PVCs (excluding VolSync cache) growing by > 10 in 7 days Volumes hold provisioning quota / workloads aren’t mounting their volumes

Deliberately no absolute-count alerts: hands-on testing shows this scale is affected by three things — the --exclude-labels configuration, cluster history (old RSs/completed Jobs only accumulate), and kor’s false positives. “There are hundreds or thousands of them” isn’t necessarily a fault; “it grew by 150 in a week” is.

Summary

The core takeaways from this practice:

  1. Use kor to scan Kubernetes for unused resources
  2. Configure Grafana dashboards to display them
  3. Manually judge and clean up the genuinely unused resources
  4. Filter out false-positive configurations
  5. And configure appropriate alerting rules based on the actual situation

In the end, it gave this Virgo (OCD + cleanliness freak) peace of mind! 😄

📚️ References