Kubernetes Disaster Recovery and etcd Backup

By Kokil Thapa | Last reviewed: September 2026

Kubernetes disaster recovery and etcd backup decide whether you rebuild a cluster in hours or lose weeks of work. Every Deployment, Secret, ConfigMap, and RBAC rule lives in etcd—the cluster's single source of truth. Application data sits elsewhere on PersistentVolumes. A solid etcd data store strategy covers both layers. This guide walks through snapshot commands, restore paths, Velero layering, and runbooks you can test before a real outage.

Why does etcd matter for Kubernetes disaster recovery and etcd backup?

etcd is a distributed key-value store. It holds every Kubernetes object the API server knows about. Lose etcd without a backup and you lose the cluster brain—even if worker nodes and disks still run.

That split confuses teams early. etcd stores desired state: which Pods should run, which Services exist, which Ingress rules apply. It does not store MariaDB rows, uploaded PDFs, or WooCommerce product images. Those live on PersistentVolumes or external databases.

On production systems I've maintained, the first DR question is always the same: what are we actually trying to recover? Control plane state, application data, or both? Answer that before you pick tools.

Managed Kubernetes—EKS, GKE, AKS—hides etcd from you. The cloud provider handles control-plane backups. Self-managed clusters on Ubuntu servers, bare metal, or tools like Kubespray put etcd backup squarely on your team. That is where Linux system administration skills meet cluster operations.

What etcd actually stores

etcd holds resource definitions under /registry prefixes. That includes Namespaces, Deployments, StatefulSets, Services, Ingresses, NetworkPolicies, and CustomResourceDefinitions. Secrets and TLS certificates live here too—encrypted at rest if you enabled encryption providers.

It does not hold container image layers. Those pull from a registry on restore. It does not hold block storage contents. A restored Pod spec pointing to a PVC still needs the volume data intact.

How do you back up etcd in a Kubernetes cluster?

For self-managed clusters, etcd snapshot backup is the canonical control-plane recovery method. The official Kubernetes documentation describes snapshot and restore procedures for stacked and external etcd topologies.

Run snapshots from a host that can reach etcd endpoints with valid TLS certificates. On stacked control-plane nodes, etcd often listens on 127.0.0.1:2379. External etcd clusters expose member IPs on port 2379.

Manual snapshot with etcdctl

Install etcdctl matching your etcd server version. Version mismatch between client and server causes subtle failures. Check the running version first:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status --write-out=table

Create the snapshot:

ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd/snapshot-$(date +%F-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Verify integrity immediately. A corrupt snapshot discovered during an outage is useless:

ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd/snapshot-2026-09-10-0200.db -w table

Automated etcd backup script

Schedule snapshots with cron or systemd timers on each control-plane node. Copy files off-node within minutes. Local-only backups die with the server.

#!/bin/bash
set -euo pipefail
BACKUP_DIR="/var/backups/etcd"
RETAIN_DAYS=14
STAMP=$(date +%F-%H%M)
SNAP="${BACKUP_DIR}/snapshot-${STAMP}.db"

ETCDCTL_API=3 etcdctl snapshot save "${SNAP}" \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL_API=3 etcdctl snapshot status "${SNAP}" -w table
rclone copy "${SNAP}" remote:k8s-etcd-backups/ --s3-no-check-bucket
find "${BACKUP_DIR}" -name 'snapshot-*.db' -mtime +${RETAIN_DAYS} -delete

The rclone step mirrors patterns from rsync versus rclone for server backups. Object storage with versioning adds another safety net. Budget roughly Rs 2,000–5,000/month (~USD 15–37) for modest snapshot retention on S3-compatible storage.

1. Confirm etcd TLS certificate paths on your control-plane nodes.
2. Install a matching etcdctl client binary.
3. Run a manual snapshot and verify with snapshot status.
4. Automate via cron with retention and off-site replication.
5. Alert if the backup job fails or the snapshot size drops sharply.

Pair this with broader server backup habits from Ubuntu server backup strategies and automated server backup setup. etcd snapshots are one file in a wider safety net.

What is the difference between etcd backup and Velero for Kubernetes disaster recovery?

Teams often ask whether Velero replaces etcd backup. It does not. The two tools solve different problems and work best together.

Velero excels at namespace-level recovery and cross-cluster migration. Read the dedicated Velero backup and restore guide for install and schedule details. etcd excels at full control-plane disaster recovery when the entire API layer is gone.

On a booking platform like Adventure Third Pole Trek, etcd holds Deployment specs and Ingress rules. Velero—or MySQL dumps—protects booking records and customer data. Skip either layer and your DR plan has a hole.

When to use which tool

• etcd only: Small self-managed cluster, fast full-cluster rebuild, no granular namespace needs.
• Velero only: Managed Kubernetes where etcd is hidden; accept you cannot rebuild the control plane yourself.
• Both: Self-managed production clusters running stateful apps—this is the correct default in 2026.

Volume snapshots add a third path for databases. See volume snapshots in Kubernetes and OpenEBS storage patterns for CSI snapshot workflows.

How do you restore a Kubernetes cluster from an etcd snapshot?

etcd restore is destructive on the target data directory. Treat it as a last-resort full rebuild—not a quick rollback. Read the official Kubernetes etcd backup and restore documentation before your first drill.

The etcd project also documents disaster recovery scenarios in its operating guide for recovery. Cross-check both sources. Kubernetes wraps etcd with specific path and flag requirements.

Restore procedure outline

1. Stop the API server and kubelet on all control-plane nodes.
2. Stop etcd on the node you are restoring to.
3. Clear or rename the existing etcd data directory.
4. Run etcdctl snapshot restore with correct --name, --initial-cluster, and --initial-advertise-peer-urls flags.
5. Restart etcd, then the API server, then kubelet.
6. Verify cluster health with kubectl get nodes and kubectl get pods -A.

ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd/snapshot-2026-09-10-0200.db \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380 \
  --data-dir=/var/lib/etcd-restored

Common mistakes kill restores silently. Wrong member name. Mismatched peer URLs. Restoring a snapshot from a three-node cluster onto a single-node test bed without adjusting flags. Stale certificates after IP changes.

I've seen teams spend hours on restore failures that a documented runbook would have prevented. Write every flag value down before the outage.

Post-restore checks

After etcd restore, confirm these items before declaring success:

• All control-plane nodes report Ready.
• kube-system Pods reach Running state.
• Critical application Deployments match expected replica counts.
• DNS and Ingress controllers respond correctly.
• Recent Secrets and ConfigMaps exist—compare resource counts against backup metadata.

If application Pods start but serve stale or empty data, your etcd restore worked but PV or database recovery did not. That is a data-layer problem, not a control-plane one. Consult PersistentVolume lifecycle docs and your database restore procedures.

What should a Kubernetes disaster recovery runbook include?

A runbook turns panic into steps. Store it outside the cluster—Git repo, wiki, or printed copy. If the cluster is down, you cannot read Confluence inside it.

Structure your runbook around recovery objectives. Define RPO (how much data you accept losing) and RTO (how fast you must be back). A law-firm portal tolerates four hours differently than a live payment gateway.

Runbook sections to document

• Cluster inventory: control-plane IPs, etcd topology, Kubernetes version, CNI plugin.
• Certificate paths: exact files for etcdctl TLS flags.
• Backup locations: local paths, S3 buckets, retention windows.
• Restore commands: copy-paste blocks with placeholders clearly marked.
• Application dependencies: external MySQL hosts, Redis endpoints, payment gateway callback URLs.
• Smoke tests: URLs and kubectl checks that prove the platform works.
• Escalation: who approves a full restore versus partial Velero recovery.

Align cluster DR with wider org strategy from cloud backup and disaster recovery planning and multi-cloud disaster recovery. Kubernetes is one tier in a stack that includes DNS, TLS certs, and registries.

Secure the runbook itself. It contains Secret references and infrastructure details. Restrict access with the same care as Kubernetes RBAC hardening policies. Validate runbook JSON snippets with a JSON formatter before storing them—typos in backup metadata files cause confusion during incidents.

GitOps workflows complicate recovery slightly. If Argo CD manages cluster state, understand whether etcd restore or Git is the source of truth after recovery. See Argo CD GitOps patterns for sync behaviour after a partial restore.

How often should you test Kubernetes disaster recovery?

Untested backups are wishful thinking. Schedule DR drills at least quarterly for production clusters. Monthly is better for revenue-critical platforms.

Each drill should have a scoped goal. Q1: restore etcd to an isolated VM and verify object counts. Q2: Velero namespace restore. Q3: full simulated control-plane loss. Q4: cross-region failover if multi-site.

Game-day checklist

1. Announce the drill window to stakeholders.
2. Spin up an isolated recovery environment—never test restore on live etcd first.
3. Restore the latest snapshot and measure elapsed time against RTO.
4. Compare resource counts: kubectl get all -A | wc -l versus pre-backup baseline.
5. Run application smoke tests against restored services.
6. Document gaps and update the runbook same day.

Track restore duration trends. Slow restores usually mean oversized etcd databases or missing indexes—not backup tool failure. Troubleshoot with Kubernetes troubleshooting techniques during drills, not during real outages.

For Laravel workloads on Kubernetes, confirm environment ConfigMaps and Secrets after restore. Application-level cache clears may be needed. The Kubernetes for Laravel guide covers common deployment patterns that break silently after state recovery.

Ongoing support and maintenance contracts should explicitly include DR drill scheduling. DR is not a one-time setup task. It is recurring operational work with real calendar slots.

Key Takeaways

• etcd backup captures cluster metadata only—PersistentVolumes and external databases need separate backup paths.
• Schedule automated etcdctl snapshot save jobs, verify each snapshot, and replicate files off-site within minutes.
• Combine etcd snapshots with Velero and volume snapshots for full Kubernetes disaster recovery coverage.
• Document restore commands, certificate paths, and member names in an offline runbook before you need them.
• Run quarterly DR drills on isolated infrastructure and measure restore time against your RTO target.
• Managed Kubernetes hides etcd—use Velero and cloud-native backup services instead of fighting provider boundaries.

People Also Ask

Can you restore etcd to a different Kubernetes version?

Restore etcd snapshots to the same minor Kubernetes version when possible. Skipping major versions during disaster recovery invites API compatibility issues. Upgrade after the cluster is healthy, not during the initial restore.

How long are etcd snapshots valid?

Snapshots do not expire technically, but older snapshots miss recent Deployments, Secrets, and CRD changes. Match retention to your RPO. Most production teams keep 14–30 days of hourly or six-hourly snapshots online.

Does etcd backup include PersistentVolume data?

No. etcd stores PVC objects—the claim definitions—not the bytes on disk. You need CSI volume snapshots, Velero with restic, or database-native dumps to recover actual file and row contents.

What happens if all three etcd members fail simultaneously?

Restore from the most recent verified snapshot onto a rebuilt member, then rejoin or rebuild additional members per your topology docs. Without any off-site snapshot, the cluster state is unrecoverable regardless of running worker nodes.

Build a recovery plan before you need one

Kubernetes disaster recovery and etcd backup are not optional extras for self-managed clusters. They are the difference between a bad afternoon and a lost platform. Start with automated etcd snapshots today. Layer Velero for namespace granularity. Write the runbook offline. Schedule your first drill this quarter.

Need help designing DR for a production cluster or migrating stateful apps safely? Contact us to discuss enterprise application deployment and recovery planning—or browse the blog for deeper Kubernetes operations guides.