August 21, 2026
11 min read
By Kokil Thapa | Last reviewed: September 2026
Every pod label, Secret, and node heartbeat lives in one place: the key value store Kubernetes uses as its brain. That store is etcd. Lose it, and you lose the cluster—not your container images, but every Deployment, Service, and RBAC rule that makes the cluster run. Application teams often ignore the control plane until API calls start timing out. Platform engineers cannot afford that blind spot. Whether you run self-hosted clusters or managed EKS, understanding etcd is what separates a quick restore from a multi-day rebuild. The same discipline applies when you ship scalable tech solutions for startups on top of Kubernetes.
Quick answer: Kubernetes cluster data is stored in etcd, a distributed key value store that uses the Raft consensus algorithm. Every API write is replicated to a quorum of members before commit, giving linearizable reads and atomic writes across the control plane.
Where is Kubernetes cluster data stored?
On exam questions and in production runbooks, the answer is the same: kubernetes cluster data is stored in etcd. Not on worker nodes. Not in your container registry. Not in Prometheus or your application database.
etcd holds the entire Kubernetes object model. That includes Pods, Deployments, Services, ConfigMaps, Secrets, Namespaces, RBAC policies, and Custom Resource Definitions. The kube-apiserver is the only component that talks to etcd directly. Controllers, schedulers, and kubelets read state through the API server, never from etcd itself.
Think of etcd as the authoritative ledger. Worker nodes are disposable. etcd is not. If you wipe a node and rejoin it, Kubernetes rebuilds workloads from etcd state. If you wipe etcd without a valid backup, the cluster is gone.
Managed services hide etcd from you. Amazon EKS, Google GKE, and Azure AKS run etcd on your behalf. Self-managed clusters—kubeadm, kubespray, bare-metal installs—put etcd on your shoulders. For a broader control-plane picture, see the guide on Kubernetes architecture and node roles.
What makes etcd the right key value store for Kubernetes?
etcd is not a general-purpose database. It is a strongly consistent, distributed key value store built for coordination and metadata. That design matches what Kubernetes needs: small objects, frequent watches, and strict ordering guarantees.
Each key-value pair in etcd maps to a Kubernetes resource path. Values are stored as JSON or protobuf, depending on the API version. Clients subscribe to key prefixes through watches. When a Pod status changes, etcd notifies watchers, and controllers react.
Compare etcd to alternatives you might know from web development:
| Store | Consistency | Best fit | Kubernetes use? |
|---|---|---|---|
| etcd | Strong (linearizable) | Cluster metadata, coordination | Yes — default datastore |
| Redis | Eventual (single primary) | Cache, sessions, pub/sub | No — wrong consistency model |
| Consul | Strong (Raft) | Service discovery, KV | Rare — not upstream default |
| MySQL / PostgreSQL | Transactional (ACID) | Application data, reports | No — too slow for watch-heavy API |
Kubernetes chose etcd because of Raft consensus, native watch support, and proven production history. The upstream project documents this in the official Kubernetes etcd administration guide. For deeper internals, the etcd project documentation covers API v3, leases, and transactions.
Do not treat etcd like Redis or MySQL. I've seen teams store large blobs in Custom Resources and wonder why the API server crawls. etcd holds metadata. Application data belongs in database-driven application layers or object storage.
How does the Kubernetes key value store maintain consistency?
etcd implements the Raft consensus protocol. Every write goes to a leader node first. The leader replicates the entry to followers. A quorum must acknowledge before the write commits. For a three-node cluster, two nodes must agree. For five nodes, three must agree.
This is strong consistency, not eventual consistency. After a successful write, every subsequent read returns that value or a newer one. Split-brain is avoided mathematically as long as fewer than half the members fail at once.
Write latency follows network round-trip time between members. On a multi-AZ cluster, one slow link can push commit latency from 5 ms to 80 ms. Raft waits for the slowest quorum member on every write. Adding nodes does not always speed things up. More members mean more coordination per commit.
If a follower falls far behind, the leader sends a snapshot instead of incremental log entries. That causes temporary I/O pressure on both sides. Monitor snapshot transfer counts as an early warning sign. The original Raft paper from Stanford remains the best reference for the algorithm mechanics: Raft consensus specification.
What are the correct etcd backup and restore procedures?
etcd data loss equals cluster loss. You cannot reconstruct cluster state from worker nodes alone. Volume snapshots are not enough. They capture filesystem blocks, not guaranteed Raft log consistency across members.
The only safe method is etcdctl snapshot save. It talks to etcd and produces a consistent point-in-time export of the keyspace.
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-*.db --write-out=table
0 */4 * * * root /usr/local/bin/etcd-backup.sh >> /var/log/etcd-backup.log 2>&1Restore is destructive. Stop the kube-apiserver first. Never restore into a running cluster. The sequence: stop API server, stop etcd, wipe data dirs, restore snapshot, restart etcd, restart API server. Test restores quarterly on a staging cluster. An untested backup is wishful thinking.
Velero backs up Kubernetes resources to object storage. It does not replace etcd snapshots for full disaster recovery. Use both: Velero for namespace-level recovery, etcd snapshots for control-plane rebuilds. See Velero backup and restore for Kubernetes and cloud disaster recovery planning for the full picture.
Wire backup scripts into your CI/CD pipeline automation so etcd snapshots run on the same schedule as application deploys.
How do you tune etcd performance for production workloads?
Most etcd problems are disk problems. etcd uses bbolt with frequent fsync calls. Slow disks kill write throughput before CPU or RAM become bottlenecks. NVMe SSDs are mandatory for production in 2026. Network-attached storage with high latency is a common failure mode on cloud VMs.
| Metric | Healthy | Warning | Critical |
|---|---|---|---|
| WAL fsync p99 | < 10 ms | 10–50 ms | > 100 ms |
| Backend commit p99 | < 25 ms | 25–100 ms | > 250 ms |
| Leader elections | Rare | Multiple per hour | Continuous flapping |
| DB size | < 4 GB | 4–8 GB | > 8 GB quota |
| Snapshot transfers | Near zero | Occasional | Sustained |
Compaction removes old revision history, but the database file keeps growing until you defragment. Schedule defrag during low-traffic windows. Defrag one member at a time. Never defrag all members simultaneously.
ETCDCTL_API=3 etcdctl defrag --endpoints=https://etcd-1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
ETCDCTL_API=3 etcdctl endpoint health --clusterWatch object sizes too. Kubernetes stores ConfigMaps and Secrets in etcd. A Secret approaching 1 MB causes read amplification across the API server. Keep large payloads in S3 or a database. Store only references in etcd. Review Secrets and ConfigMaps best practices before stuffing megabytes into the cluster store.
When should you scale etcd from three to five nodes?
A default three-node etcd cluster tolerates one failure. A five-node cluster tolerates two. That sounds better, but every write now needs three acknowledgments instead of two. Write latency rises with member count and network distance.
Scale to five nodes when:
- Your cluster spans three or more availability zones and must survive a zone loss
- Mission-critical workloads exceed your single-node recovery SLA
- The cluster exceeds roughly 5,000 nodes or sustains over 10,000 API requests per second
- Compliance requires N+2 redundancy on control-plane components
Do not scale to five nodes because bigger feels safer. On a legal-tech platform I worked on, three nodes handled sensitive document workflows fine. Peak API traffic stayed around 800 req/s. Five nodes would have added coordination overhead without real benefit. Measure first with etcdctl endpoint status --write-out=table.
For HA control-plane design beyond etcd sizing, read highly available Kubernetes control plane setup.
How do you monitor etcd health before outages happen?
Reactive alerts on etcd unavailability arrive too late. By then, recovery options shrink fast. Proactive monitoring tracks leading indicators: WAL fsync latency, backend commit duration, leader election frequency, and database size growth.
Prometheus with etcd metrics remains the standard stack in 2026. Alert on these thresholds:
- WAL fsync p99 above 50 ms for five minutes: Disk degradation. Check I/O contention and co-located workloads.
- Leader elections above two per hour: Network instability or CPU throttling. Inspect inter-node latency and cgroup limits.
- DB size above 7 GB: Schedule defrag within 24 hours. Default quota is 8 GB.
- Sustained snapshot transfers: A follower is lagging. Check bandwidth and disk read speed on the leader.
Align etcd runbooks with broader DevOps automation practices. Document exact etcdctl commands before a 3 AM page. When the API server is flapping, nobody wants to grep docs for certificate paths.
For the full metrics stack, see Prometheus and Grafana monitoring setup. Use the JSON formatter tool to inspect exported manifest payloads during incident triage.
When etcd symptoms surface as Pod failures, the guide on debugging CrashLoopBackOff in Kubernetes helps separate app issues from control-plane problems.
Key Takeaways
- Kubernetes cluster data is stored in etcd—the distributed key value store behind the API server, not on worker nodes.
- Back up with
etcdctl snapshot saveon a schedule; test restores on staging clusters at least quarterly. - Prioritize NVMe disk latency over CPU when tuning etcd; WAL fsync p99 above 100 ms is a critical risk.
- Keep ConfigMaps and Secrets small; etcd is for metadata, not bulk application storage.
- Scale to five etcd nodes only when fault-tolerance requirements justify higher write latency.
- Monitor WAL fsync, leader elections, and DB size as leading indicators before the API server fails.
People Also Ask
Kubernetes cluster data is stored in which of the following?
etcd. Every Kubernetes object—Pods, Deployments, Services, Secrets, ConfigMaps, nodes, and RBAC rules—persists in etcd. Worker nodes hold container runtime state only. The container registry holds images. Neither can rebuild cluster configuration after etcd loss.
Is etcd a SQL or NoSQL database?
Neither in the traditional sense. etcd is a distributed key value store with strong consistency guarantees via Raft. It supports range queries, watches, and transactions on keys. It is optimized for coordination metadata, not relational queries or analytics workloads.
Can you run Kubernetes without etcd?
Not in upstream Kubernetes. etcd is the only supported datastore for standard clusters. Some experimental projects explore alternative backends, but production Kubernetes—self-managed or managed—relies on etcd for all control-plane state.
Who manages etcd on managed Kubernetes services?
The cloud provider runs and patches etcd for you on EKS, GKE, and AKS. You still inherit etcd limits: object count, request rate, and Secret size caps. Misconfigured workloads can still overwhelm the API server even when you never touch etcd directly.
Build a Control Plane You Can Trust
The key value store Kubernetes depends on is not invisible infrastructure. It is the ledger that defines your entire cluster. Strong consistency buys reliability, but only if you respect disk requirements, backup discipline, and proactive monitoring. Master etcd fundamentals before chasing advanced features like learner nodes or cross-region replication.
If you need help designing production Kubernetes infrastructure—etcd backup validation, performance tuning, or HA control-plane architecture—Linux system administration and cluster operations cover the full stack. We have shipped booking platforms like Adventure Third Pole Trek on resilient infrastructure. For a broader DevOps path, see how to become a DevOps engineer in 2026 or CKA exam preparation.
Contact us to discuss your cluster architecture, or reach out directly about etcd operations and backup validation. Reliable control planes are engineered, not accidental.
Frequently Asked Questions
Etcd is a distributed, strongly consistent key-value store that serves as the single source of truth for all Kubernetes cluster state. It stores configuration data, node status, pod definitions, and secrets. Without a healthy etcd cluster, the Kubernetes API server cannot function, making it the most critical infrastructure component to protect and monitor in any production environment.
Etcd provides strong consistency via the Raft consensus protocol, which is essential for distributed system coordination. Unlike SQL databases optimized for complex queries and transactions, etcd excels at fast reads/writes of small key-value pairs and reliable leader election. Kubernetes requires this specific consistency model to prevent split-brain scenarios where multiple controllers attempt conflicting state changes simultaneously across nodes.
Always run an odd number, typically three or five nodes. This ensures the Raft consensus algorithm can tolerate failures while maintaining quorum. A three-node cluster survives one failure; five nodes survive two. Never run even numbers, as they provide no additional fault tolerance over the next lower odd number but increase latency and resource costs without improving availability guarantees.
Etcd is sensitive to disk I/O latency and requires dedicated SSDs, never HDDs. For clusters under 100 nodes, allocate 2-4 vCPUs, 8GB RAM, and NVMe storage with sub-10ms write latency. Larger clusters need proportional scaling. Avoid sharing disks with other workloads. In my experience managing infrastructure, insufficient disk performance causes more etcd instability than CPU or memory constraints in real deployments.
Use etcdctl snapshot save to create point-in-time backups, storing them encrypted off-cluster. Automate this via CronJob or systemd timer every few hours. Test restores quarterly in staging. During restoration, stop all API servers first, restore to a single node, then rejoin peers. Never skip backup verification. On legal-tech portals I maintain, untested backups caused extended outages during recovery attempts.
Technically possible but strongly discouraged for production. Running etcd as pods introduces circular dependencies: if the cluster fails, etcd fails, preventing recovery. External etcd clusters on dedicated VMs or bare metal provide isolation from Kubernetes failures. For Nepal-based clients with limited infrastructure budgets, I sometimes accept stacked etcd on control-plane nodes but always with dedicated disks and rigorous monitoring.
Track key metrics: WAL fsync duration (should stay under 10ms), backend commit latency, leader elections count, and db size. Use Prometheus with etcd-exporter and set alerts on WAL latency exceeding 50ms or repeated leader changes. Check disk space regularly as etcd compaction failures cause unbounded growth. In production systems I manage, proactive metric alerting prevents cascading failures far better than reactive troubleshooting after outages occur.
Common causes include slow disks, excessive key churn, large value sizes, and insufficient compaction. Frequent leader elections indicate network partitions or disk stalls. Keyspace bloat from uncollected revisions degrades read performance. Monitor compaction status and enable auto-compaction. On high-traffic eCommerce platforms, I have seen etcd slowdowns trace back to misconfigured retention policies creating massive historical revision accumulation that overwhelmed storage IOPS.
Enable TLS for all peer and client communication using certificates signed by a trusted CA. Encrypt sensitive values at rest using Kubernetes EncryptionConfiguration with AES-CBC or secretbox providers. Restrict etcd port access via firewall rules to only API servers. Rotate certificates before expiry. For legal-tech systems handling sensitive documents, I enforce mTLS between all components and audit etcd access logs for unauthorized connection attempts.
The cluster becomes read-only and eventually completely unavailable once quorum is permanently lost. API servers reject all write operations. Recovery requires manual intervention: either restore from backup or add new members to regain quorum. This is why odd-numbered clusters and geographic distribution matter. In disaster scenarios I have handled, quorum loss meant hours of downtime because backups were outdated or restoration procedures were undocumented.
Perform rolling upgrades one node at a time, always maintaining quorum. Upgrade minor versions sequentially, never skipping majors. Verify cluster health between each node upgrade using etcdctl endpoint health. Backup immediately before starting. Downgrade is unsupported, so test thoroughly in staging first. On shared infrastructure deployments, I schedule upgrades during low-traffic windows and keep rollback snapshots ready, though actual rollbacks remain risky and complex.
Stacked etcd runs on the same nodes as Kubernetes control-plane components, simplifying management but coupling failures. External etcd runs on separate infrastructure, providing fault isolation at higher operational cost. Stacked suits smaller clusters under 50 nodes. External is mandatory for large-scale or compliance-sensitive environments. For Nepal Gift Card platform, we chose stacked topology to reduce hosting costs while accepting tighter failure domains appropriate for that scale.
Leader storms indicate network instability, disk latency spikes, or clock skew. Check inter-node latency with ping and traceroute. Verify NTP synchronization across all nodes. Examine WAL fsync metrics for disk bottlenecks. Review system logs for OOM kills or kernel panics. Temporarily increase election timeout if network jitter is confirmed. In one production incident, mismatched MTU settings between cloud provider networks caused packet fragmentation triggering persistent election failures until corrected.
K3s uses SQLite for single-node simplicity. kine translates etcd API calls to PostgreSQL or MySQL, enabling familiar database backends. Apache Zookeeper predates etcd but lacks its Kubernetes-native integration. No alternative matches etcd's maturity for standard Kubernetes. For specialized Nepal-based deployments with existing PostgreSQL expertise, I have evaluated kine but ultimately recommended standard etcd due to better community support and proven reliability under diverse failure conditions.
Cloud-managed etcd costs USD 50-150 monthly (NPR 6,500-20,000) depending on size. Self-hosted on three t3.large EC2 instances with EBS gp3 volumes runs approximately USD 120 monthly (NPR 16,000). Factor in engineering time for maintenance. Budget projects in Nepal often use stacked topology on existing control planes to avoid dedicated infrastructure costs. Always calculate total ownership including monitoring, backup storage, and personnel expertise rather than just compute expenses.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
