Moving Certificate Renewal from crontab to a Kubernetes CronJob

📝Notes: This article covers the migration design and verification strategy for moving from Linux crontab to a K8s CronJob. The issuance flow with acme.sh + DNSPod, and the push details for Tencent CDN and Qiniu, were already covered in another post, so here they are only mentioned as background and glossed over.

Background: renewal ran on a desktop machine

Continuing from last time, the old setup can be summed up in three lines:

• One Ubuntu desktop machine,
• One ~/.acme.sh,
• One crontab line.

It worked, but it had plenty of problems:

1. Single point of failure. If the machine went to sleep, had its disk replaced, or had its OS reinstalled, the renewal chain broke. And once the chain breaks, the certificate is headed for expiry.
2. Not observable. Whether crontab ran or failed could only be checked by digging through logs. No Job history, no failure counts, and no place to hook alerts into.
3. Configuration drift. There was no single source of truth linking that one crontab command to the actual script, so after one change you could never quite say what had been changed at the time.

Why a CronJob

The Kubernetes CronJob is the only scheduler in this environment that is both “declarative” and “keeps a history”.

Here is what you get:

• It is written into Git and goes through the GitOps review process, so configuration drift drops to zero.
• Job history is retained, so both successes and failures can be checked with kubectl get jobs, without logging into some machine to dig through logs.
• Retry counts, concurrency policy, and timeouts are all fields, so you do not have to write your own shell checks.
• Logs go to Loki, metrics go to Prometheus, and alerts go to Alertmanager — all the pipelines already exist.

CronJob structure

The skeleton looks roughly like this; adjust the fields to your actual environment.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cert-renew
  namespace: certs
spec:
  schedule: "17 3 * * *"
  timeZone: "Asia/Shanghai"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 3600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: cert-renew
          containers:
            - name: acme
              image: neilpang/acme.sh:latest
              command: ["/bin/sh", "-c"]
              args:
                - |
                  acme.sh --issue --dns dns_dp \
                    -d ewhisper.cn -d '*.ewhisper.cn'
              envFrom:
                - secretRef:
                    name: acme-dns-credential
              volumeMounts:
                - name: acme-home
                  mountPath: /acme.sh
          volumes:
            - name: acme-home
              persistentVolumeClaim:
                claimName: acme-home

The details are as follows:

• concurrencyPolicy: Forbid is mandatory. If two runs overlap, they write the DNS TXT records at the same time, overwrite each other, and issuance fails outright.

• startingDeadlineSeconds: 3600 provides a one-hour tolerance window. A brief cluster outage then does not cost you an entire run.

• The timeZone field requires K8s 1.27 or later (I just upgraded to 1.37 yesterday, the latest stable version of k3s). On older versions you have to fall back to the TZ environment variable inside the container.

• restartPolicy: Never together with backoffLimit: 2 stops after two failures, preventing endless retries from hammering the DNS API.

Secret management: only placeholders in the repository

You really, really need to be careful with secret management to avoid leaks:

• Only placeholders go in the repository.
• Secrets are managed separately through the External Secret Operator

📌Example: the point of placeholders is that the spec can be committed, CI can run, and PRs can be reviewed, while the actual Secret never enters Git history.

Secret mounting approach: the CronJob mounts the environment variables through a Secret, readOnly, never written to the image layer, and never appearing in the plaintext output of kubectl describe pod.

Go-live checklist

Before the first run, go through these one by one:

1. Confirm that what is mounted in the Pod is the real secret value, not the placeholder from the repository.
2. Give the ServiceAccount only the required verbs; being able to read Secrets is enough, do not grant *.
3. Confirm that concurrencyPolicy: Forbid is in effect.
4. Make sure startingDeadlineSeconds is not less than the duration of one run.
5. Trigger the first run manually; do not just sit and wait for the schedule.
6. Record the current certificate fingerprint before migrating, as a baseline for comparison.
7. Keep crontab for at least one full cycle before deleting it.
8. Wire Job failure alerts into Alertmanager.

The migration is finally complete. 🎉🎉🎉

Summary

For the migration from Linux crontab to a K8s CronJob, the YAML part can be written by following the official documentation. The hard part is the detailed considerations that come up in practice.

This article summarized my steps for migrating certificate renewal from Linux crontab to a Kubernetes CronJob: the design, secret management, and pre-launch checks. The benefits are:

• More reliable and stable
• Historical logs (Loki) and alerting (AlertManager)
• GitOps: git is the single source of truth.

The related code has been packaged as a Helm chart; anyone interested can take a look: https://github.com/east4ming/homelab2/tree/master/apps/acme-ewhisper

I hope it helps.

🎉🎉🎉

📚️References

• Kubernetes CronJob official documentation
• acme.sh project homepage