Velero: Backup and Restore Kubernetes

By Kokil Thapa | Last reviewed: August 2026

Running production workloads on Kubernetes without a verified backup strategy is a liability waiting to materialize. While etcd snapshots protect cluster state, they do not capture application data, persistent volumes, or cross-cluster migration requirements. Implementing Velero: Backup and Restore Kubernetes provides the necessary abstraction layer to secure both cluster resources and persistent storage against accidental deletion, corruption, or regional outages. For teams managing critical infrastructure, understanding this tool is as fundamental as configuring reliable CI/CD pipelines or securing application deployments.

How does Velero: Backup and Restore Kubernetes actually work?

Understanding the internal mechanics of Velero prevents misconfiguration during high-stress recovery scenarios. Unlike simple database dumps or raw disk imaging, Velero operates as a control-plane orchestrator that coordinates multiple subsystems. When you initiate a backup, the Velero server pod queries the Kubernetes API server for resource definitions (Deployments, Services, ConfigMaps) and simultaneously instructs storage providers to snapshot attached volumes.

The process follows a strict sequence that ensures consistency. First, Velero discovers all resources matching your backup criteria (namespace labels, resource types, or selectors). Second, it serializes these resources into JSON/YAML tarballs and uploads them to your configured object storage bucket. Third, if persistent volumes are included, it triggers either a CSI VolumeSnapshot or a filesystem-level backup via the node-agent DaemonSet. Finally, it records the backup status and expiration metadata back into the cluster as Custom Resources.

A common mistake I see in production environments is assuming Velero backs up the actual etcd database. It does not. Velero reconstructs cluster state by re-applying Kubernetes manifests. This distinction matters because custom resources managed by operators outside the standard API may require explicit annotation or plugin support to be captured correctly. Always verify that your specific CRDs are included in backup logs before trusting the system for disaster recovery.

How do you install and configure Velero with S3-compatible storage?

Installation requires three components: the Velero CLI on your local machine, the Velero server deployment in the cluster, and credentials for your object storage provider. In 2026, Helm is the standard installation method, replacing the deprecated `velero install` CLI command for production setups. The current stable release is Velero v1.16.x, which requires Kubernetes 1.28+ and supports modern CSI snapshot APIs.

Preparing S3-compatible storage credentials

Whether you use AWS S3, MinIO, DigitalOcean Spaces, or a Nepal-based cloud provider's object store, Velero needs IAM-style credentials. Create a file named credentials-velero:

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

For non-AWS providers, you must also specify the region and S3 URL in your Helm values. Never commit these credentials to Git. Use sealed secrets, external-secrets operator, or inject them directly during the Helm install step.

Helm installation with production defaults

Create a values.yaml that enables node-agent for filesystem backups and configures your specific storage backend:

configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: my-k8s-backups
      config:
        region: us-east-1
        s3ForcePathStyle: true
        s3Url: https://nyc3.digitalocean.com
  volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: us-east-1

deployNodeAgent: true
nodeAgent:
  privileged: true
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 1Gi

credentials:
  useSecret: true
  secretName: cloud-credentials

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.11.0
    volumeMounts:
      - mountPath: /target
        name: plugins

Install using the official chart repository:

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
helm install velero vmware-tanzu/velero \
  --namespace velero \
  --create-namespace \
  --values values.yaml \
  --wait

After installation, verify connectivity immediately. Running velero backup-location get should show "Available" status within 60 seconds. If it shows "Unavailable", check network policies, credential formatting, and bucket permissions. I have debugged dozens of failed installations where the issue was simply a missing s3:ListBucket permission on the IAM policy. Velero requires read/write/list access to function.

What is the difference between CSI snapshots and filesystem backups?

Choosing the wrong volume backup method is the most frequent cause of slow restores and inflated storage costs. Velero supports two distinct approaches, each with different performance characteristics and infrastructure requirements. Understanding this trade-off is essential when architecting cloud hosting solutions for clients with varying budget and RPO constraints.

In practice, I recommend CSI snapshots as the default for any cluster running on major cloud providers (AWS EBS, GCP PD, Azure Disk) with a compatible CSI driver. The performance difference is dramatic: a 500GB PostgreSQL volume takes seconds to snapshot via CSI but hours via filesystem backup. However, filesystem backups remain indispensable for NFS volumes, hostPath mounts, or migration scenarios where source and destination clouds differ.

When using filesystem backups, ensure node-agent pods have sufficient CPU and memory limits. Under-resourced agents cause backup timeouts on large volumes. Set requests to at least 500m CPU and 512Mi RAM per node, scaling up based on concurrent backup load. Monitor node-agent logs for "context deadline exceeded" errors, which indicate resource starvation rather than network issues.

How do you schedule automated backups and verify integrity?

Manual backups provide false confidence. Production systems require automated schedules with verification loops. Velero uses Schedule resources to define cron-based backup policies, but scheduling alone is insufficient without integrity testing.

Creating tiered backup schedules

Implement a multi-tier strategy that balances RPO, storage costs, and recovery granularity:

# Hourly namespace-level backups for critical apps
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: production-hourly
  namespace: velero
spec:
  schedule: "0 * * * *"
  template:
    includedNamespaces:
      - production-*
    snapshotVolumes: true
    ttl: 24h0m0s
    storageLocation: default
    volumeSnapshotLocations:
      - default

# Daily full-cluster backup with 30-day retention
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: cluster-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    snapshotVolumes: true
    ttl: 720h0m0s
    excludeResources:
      - events
      - events.events.k8s.io

Exclude ephemeral resources like Events, ReplicaSets, and EndpointSlices unless you have specific forensic requirements. These resources regenerate automatically and bloat backup storage unnecessarily.

Automated restore verification

A backup you cannot restore is worthless. Implement weekly automated restore tests into a separate validation namespace or staging cluster. Use Velero's --namespace-mappings flag to avoid overwriting production:

velero restore create verify-$(date +%Y%m%d) \
  --from-backup production-hourly-20260822-020000 \
  --namespace-mappings production-app:verify-app \
  --wait

After restore completes, run health checks against the restored workloads. Verify pod readiness, database connectivity, and data integrity checksums. I integrate this verification into DevOps automation pipelines that alert on failure. Silent backup corruption is worse than no backups at all because it creates complacency.

How do you troubleshoot failed restores and partial backups?

Restore failures are inevitable in complex environments. Systematic debugging separates quick recoveries from extended outages. Velero exposes detailed logs and describe commands that pinpoint root causes when interpreted correctly.

1. Check restore status first: Run velero restore describe <name> to see warnings and errors summary. Partial failures show as warnings; complete failures show as errors.
2. Inspect pod logs: kubectl logs -n velero deploy/velero reveals API-level issues like RBAC denials or CRD conflicts. Node-agent logs (kubectl logs -n velero ds/node-agent) expose volume mount or snapshot problems.
3. Validate resource compatibility: Restores fail when target clusters lack required CRDs, operators, or storage classes. Pre-flight checks should verify these dependencies exist before initiating restore.
4. Review hook execution: If using pre/post-restore hooks for database quiescing, check hook container logs. Failed hooks often leave applications in inconsistent states even when the restore reports success.
5. Examine download URLs: For manifest-only issues, download the backup tarball via velero backup download and inspect contents locally. Corrupted uploads or incomplete transfers become visible here.

One recurring issue involves PersistentVolumeClaim binding during restore. If the target cluster uses different storage classes or availability zones, PVCs may remain pending indefinitely. Always map storage classes explicitly using --storage-class-mappings when restoring across environments. Another gotcha: restoring StatefulSets with ordered pod creation can timeout if headless services aren't created first. Velero handles this automatically in most cases, but custom operators sometimes interfere with resource ordering.

For partial backups where some volumes succeeded but others failed, treat the entire backup as compromised. Do not attempt selective volume restoration unless you fully understand application consistency boundaries. It is safer to restore from the last known-good complete backup and replay transaction logs than to risk data corruption from mismatched volume timestamps.

Implementing Reliable Velero: Backup and Restore Kubernetes Strategies

Effective Velero: Backup and Restore Kubernetes implementation requires treating backups as production infrastructure, not an afterthought. Start with CSI snapshots for performance-critical workloads, supplement with filesystem backups for portability, and automate verification weekly. Document your RTO/RPO targets explicitly and test against them quarterly under realistic conditions. Remember that backup reliability depends entirely on restore testing frequency. If you need assistance designing disaster recovery architectures or auditing existing Kubernetes backup configurations, reach out to discuss your infrastructure requirements.