August 21, 2026
10 min read
Table of Contents
By Kokil Thapa | Last reviewed: August 2026
A single point of failure in your cluster's management layer turns minor hardware issues into full outages. Building a highly available Kubernetes control plane eliminates this risk by distributing the API server, scheduler, controller manager, and etcd across multiple nodes with proper load balancing. While my daily work centers on Laravel and PHP infrastructure, the principles of redundancy and graceful degradation I apply to CI/CD pipeline automation translate directly to Kubernetes architecture. This guide covers the exact configuration patterns needed for production resilience in 2026.
Quick answer: A highly available Kubernetes control plane requires at least three control-plane nodes running stacked etcd, an external or internal load balancer distributing traffic to all API servers, and kubeadm configured with
--upload-certs for secure certificate distribution during node joins.How do you architect a highly available Kubernetes control plane?
The foundation of any resilient cluster is understanding what components actually require redundancy. The control plane consists of four critical services: kube-apiserver (the only entry point for all operations), etcd (the distributed state store), kube-scheduler (assigns pods to nodes), and kube-controller-manager (reconciles desired state). In a highly available Kubernetes control plane, each must survive individual node failures without interrupting cluster operations.
You have two deployment topologies to choose from. Stacked etcd runs the etcd member as a static pod on the same node as other control plane components — simpler to operate but couples compute and storage failure domains. External etcd separates the data store onto dedicated nodes, providing better isolation at the cost of six machines minimum. For most teams managing their own infrastructure, stacked etcd with three nodes provides sufficient resilience while keeping operational complexity manageable.
The load balancer is non-negotiable. Worker nodes, kubectl clients, and CI pipelines must reach the API server through a single stable endpoint. When one control plane node fails, the load balancer stops routing traffic to it within health-check intervals. HAProxy, Nginx, or cloud-provider LBs all work; the key is configuring TCP health checks against /livez on port 6443 rather than simple port connectivity.
How do you configure HAProxy for Kubernetes API server load balancing?
The load balancer sits between every client and your API servers. Misconfiguration here causes intermittent failures that are notoriously difficult to debug. On bare metal or VMs without cloud LB integration, HAProxy remains the standard choice for 2026 deployments due to its transparent TCP proxying and robust health checking.
Install and configure HAProxy 2.8+
<!-- /etc/haproxy/haproxy.cfg -->
frontend k8s-api
bind *:6443
mode tcp
option tcplog
default_backend k8s-api-backend
backend k8s-api-backend
mode tcp
option tcp-check
tcp-check connect port 6443
tcp-check send GET\ /livez\ HTTP/1.1\r\nHost:\ localhost\r\nConnection:\ close\r\n\r\n
tcp-check expect string ok
balance roundrobin
server cp1 10.0.1.10:6443 check inter 5s fall 3 rise 2
server cp2 10.0.1.11:6443 check inter 5s fall 3 rise 2
server cp3 10.0.1.12:6443 check inter 5s fall 3 rise 2The health check configuration matters more than the balancing algorithm. Using /livez instead of raw TCP connect ensures the API server process itself is responsive, not just that the port is open. The fall 3 rise 2 parameters prevent flapping: a server must fail three consecutive checks before removal and pass two before re-addition. Set inter 5s to detect failures within 15 seconds maximum.
Enable the stats interface for operational visibility during incidents:
listen stats
bind *:8404
mode http
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:secure-password-change-meOn Ubuntu 24.04, enable and start the service with systemctl enable --now haproxy. Verify backend health immediately after configuration changes using echo "show stat" | socat stdio /run/haproxy/admin.sock before directing production traffic.
How do you initialize a stacked etcd cluster with kubeadm?
kubeadm automates the complex certificate generation and etcd bootstrapping that previously required manual PKI management. The critical step many guides omit is creating a proper kubeadm-config.yaml before initialization — interactive flags become unmanageable for HA setups.
Create the kubeadm configuration
# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.31.0
controlPlaneEndpoint: "k8s-api.example.com:6443"
networking:
podSubnet: "10.244.0.0/16"
serviceSubnet: "10.96.0.0/12"
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.0.1.10
bindPort: 6443The controlPlaneEndpoint must resolve to your load balancer VIP or DNS name — never a specific node IP. This is what gets embedded in certificates and kubeconfig files. Initialize the first node with kubeadm init --config=kubeadm-config.yaml --upload-certs. The --upload-certs flag encrypts and uploads CA certificates to a temporary secret, allowing subsequent control plane nodes to join securely without manual certificate copying.
Join additional control plane nodes
The init output provides a join command with a certificate decryption key. Run this exact command on each additional control plane node:
kubeadm join k8s-api.example.com:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:xxxx... \
--control-plane --certificate-key xxxxx...Verify etcd cluster health after each join:
kubectl exec -n kube-system etcd-cp1 -- \
etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health --clusterAll three endpoints must report healthy before proceeding. If a join fails partway through, run kubeadm reset on the failed node before retrying — partial state causes cryptic TLS errors.
What are the common failure modes and recovery procedures?
Understanding how your highly available Kubernetes control plane fails is more important than knowing how it succeeds. In practice, I've seen three categories of issues repeatedly during production deployments and maintenance windows.
etcd quorum loss
With three nodes, losing one maintains quorum (2/3). Losing two destroys the cluster. Monitor etcd member count actively:
kubectl get pods -n kube-system -l component=etcd -o wide
# Alert if Ready count < 2If a node suffers permanent hardware failure, remove it from the etcd membership before rebuilding:
# Get member ID from healthy node
kubectl exec -n kube-system etcd-cp1 -- etcdctl member list
# Remove failed member
kubectl exec -n kube-system etcd-cp1 -- etcdctl member remove MEMBER_ID_HEX
# Then rebuild node and rejoin normallyNever restart a permanently failed etcd member without removing it first — stale members can corrupt cluster state when they attempt to rejoin with outdated WAL logs.
Certificate expiration
kubeadm-generated certificates expire after one year by default. Check expiry dates proactively:
kubeadm certs check-expirationRenew all certificates before expiry with kubeadm certs renew all, then restart static pods by moving manifests out of /etc/kubernetes/manifests/ and back in. Automate this check in monitoring — certificate expiry is the most common cause of sudden HA cluster outages in environments without active maintenance.
Load balancer misrouting
When HAProxy health checks pass but API requests fail intermittently, the issue is often asymmetric routing or firewall rules blocking return traffic. Test from each worker node individually:
curl -k https://k8s-api.example.com:6443/livez
# Must return "ok" from every nodeIf specific nodes fail while others succeed, check UFW/firewalld rules, SELinux/AppArmor profiles, and network interface binding. This diagnostic approach mirrors troubleshooting patterns I use when debugging Laravel API connectivity issues across distributed infrastructure.
| Failure Scenario | Detection Method | Recovery Action | Max Tolerable Duration |
|---|---|---|---|
| Single CP node down | HAProxy backend marked DOWN | Automatic failover; repair offline node | Indefinite (quorum maintained) |
| etcd leader election | API latency spike 2-10s | Automatic; no action required | <30 seconds |
| Certificate expired | kubectl returns x509 error | kubeadm certs renew + pod restart | Minutes (manual intervention) |
| Two CP nodes down | API server read-only or unavailable | Restore from etcd snapshot | Hours (data loss possible) |
| LB health check false positive | Intermittent 503 errors | Fix /livez endpoint or adjust check | Until corrected manually |
How do you monitor and maintain long-term control plane health?
Deploying HA is a one-time event; maintaining it requires ongoing discipline. The practices below come from operating production clusters where downtime directly impacts business revenue — similar to the reliability standards required for DevOps automation in Nepal's growing tech sector.
Essential monitoring targets
- etcd cluster size: Alert when ready members drop below 2. Use Prometheus query
count(etcd_server_has_leader{job="kube-etcd"} == 1) < 2. - API server request latency: P99 should stay under 1s for mutating operations. Sustained elevation indicates etcd disk I/O saturation or network issues.
- Certificate expiry: Alert at 30 days remaining. Automate with cert-manager or cron-based
kubeadm certs check-expirationparsing. - Load balancer backend health: Track HAProxy backend UP/DOWN transitions. Frequent flapping suggests network instability or resource exhaustion on control plane nodes.
- etcd disk WAL fsync duration: Values exceeding 10ms indicate slow storage. Control plane nodes need SSD/NVMe — spinning disks cause cascading timeouts.
Backup strategy
etcd snapshots are your disaster recovery lifeline. Schedule automated snapshots every 4-6 hours:
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Retain last 12 snapshots, delete older
find /backup -name "etcd-snapshot-*.db" -mtime +3 -deleteStore backups off-cluster. A storage failure that takes down etcd will also destroy co-located backups. Use S3, GCS, or a separate NFS mount. Test restoration quarterly — untested backups provide false confidence.
Upgrade procedure
Upgrade control plane nodes sequentially, never simultaneously. Drain workloads (even though control plane nodes shouldn't run user pods), cordon, upgrade kubeadm/kubelet/kubectl packages, run kubeadm upgrade apply on the first node then kubeadm upgrade node on subsequent nodes, uncordon, and verify etcd health before proceeding to the next. Budget 30-45 minutes per node for production clusters.
Conclusion
A highly available Kubernetes control plane demands deliberate architecture choices, disciplined configuration, and proactive maintenance. Three stacked etcd nodes behind a properly health-checked load balancer handles most production workloads reliably when you understand the failure modes and recovery procedures outlined above. Skip shortcuts on certificate management, backup testing, or monitoring — these are where HA clusters silently become single points of failure. If you're planning a production Kubernetes deployment or need help hardening an existing cluster's control plane, reach out to discuss your infrastructure requirements.
Frequently Asked Questions
A configuration with three or more etcd members and multiple API servers behind a load balancer, ensuring cluster management survives individual node failures without downtime.
Minimum three nodes to maintain etcd quorum during one failure; five nodes tolerate two simultaneous failures but increase operational cost and complexity significantly.
Etcd uses Raft consensus requiring (N/2)+1 votes; even numbers provide no extra fault tolerance over the next lower odd number while increasing split-brain risk.
In my experience managing infrastructure, stacked etcd works reliably for clusters under 50 nodes and simplifies operations. External etcd adds deployment complexity but isolates API server load from consensus traffic, making it necessary only when control plane resources face contention or you manage multiple clusters sharing etcd backing stores. For most Nepal-based deployments on limited budgets, stacked reduces both cost and failure domains.
Use kube-vip or keepalived for bare metal VIPs, or cloud provider LBs for managed environments. The load balancer must forward TCP 6443 to all healthy API servers and perform active health checks against /livez endpoints. Avoid DNS-only round-robin as clients cache stale IPs during failover. On Ubuntu servers I manage, kube-vip in ARP mode provides sub-second failover without external dependencies, costing zero additional NPR compared to managed alternatives.
Remaining nodes maintain etcd quorum and continue serving API requests through the load balancer. Automatic leader election occurs within seconds. Workloads remain unaffected as worker nodes communicate via the VIP. However, certificate renewal, scheduled jobs, and controller reconciliation pause briefly. Monitor etcd member health immediately; running degraded with two of three nodes means losing another causes total cluster failure. Always replace failed nodes before performing maintenance elsewhere.
Three minimal VMs (2 vCPU, 4GB RAM) cost roughly Rs 15,000–25,000/month (~USD 110–185) on typical cloud providers. Bare metal requires higher upfront investment but eliminates recurring compute costs. Budget an additional 20% for load balancer, storage, and monitoring overhead.
Yes, using kubeadm upgrade workflow. Add new control plane nodes sequentially with kubeadm join --control-plane, verify etcd membership and API server health after each addition, then update your load balancer backend pool. Never remove the original node until at least two new members are healthy and synced. Back up etcd snapshots before starting. This process takes 30–60 minutes per node depending on network and storage performance.
Disk latency exceeding 10ms causes leader elections and API timeouts. Network partitions between members break quorum. Clock skew over 500ms corrupts consensus. Insufficient disk space triggers compaction failures. In production systems I have debugged, slow NVMe degradation was the silent killer—monitoring showed healthy CPU/memory while etcd fsync durations climbed gradually. Always use dedicated SSDs, enable etcd metrics, and alert on WAL fsync p99 latency exceeding 5ms.
Use etcdctl snapshot save on a single member during low-traffic periods; never backup all members simultaneously. Store encrypted snapshots off-cluster. Restoration requires stopping all API servers, restoring to one member with etcdctl snapshot restore, then rejoining other members as fresh learners. Test restores quarterly in staging. On legal-tech portals handling sensitive documents, I automate daily snapshots to S3-compatible storage with versioning, ensuring recovery point objectives stay under one hour regardless of failure type.
Each API server needs certs signed by the same CA with SANs covering the VIP, individual node IPs, and localhost. Etcd members require separate peer and client certs. Kubelet and controller-manager need distinct identities. Certificate rotation must happen before expiry across all nodes simultaneously or rolling with load balancer draining. Use cert-manager or kubeadm's built-in rotation. Expired certs cause silent API failures that mimic network issues; always validate cert validity during troubleshooting.
Track etcd member count, leader changes, DB size, and WAL fsync latency via Prometheus. Alert on API server request duration p99 exceeding 1s, failed authentication attempts, and node NotReady status. Use Grafana dashboards specifically designed for HA control planes showing quorum status visually. Synthetic probes hitting the VIP endpoint catch load balancer misconfigurations that individual node metrics miss. Logging aggregation helps correlate events across nodes during incidents. Without proper observability, HA becomes false confidence rather than genuine resilience.
Only if downtime directly loses revenue or violates compliance. Single-node clusters with regular backups recover in minutes for non-critical workloads. HA adds significant operational burden: certificate management, upgrade coordination, troubleshooting complexity. For most Nepal SMB sites I build, managed Kubernetes or simpler orchestration suffices. Reserve HA for platforms where four-hour recovery windows are unacceptable, like payment processing or legal service portals with strict SLAs. Calculate actual business impact before investing in HA infrastructure.
Etcd peers need reliable low-latency connectivity on ports 2379 and 2380. API servers communicate on 6443. All nodes require stable DNS resolution and synchronized time via NTP. Cross-datacenter deployments introduce latency that breaks Raft timing assumptions; keep all control plane nodes within the same region or availability zone. Firewall rules must allow bidirectional traffic between members. Network jitter causes more HA failures than hardware faults in my experience; use dedicated VLANs or security groups isolating control plane traffic from application workloads.
Upgrade one node at a time: drain, cordon, upgrade kubeadm/kubelet/kubectl, uncordon, verify health, repeat. Etcd upgrades follow separate compatibility matrices. Never upgrade multiple nodes simultaneously. Pre-upgrade validation with kubeadm upgrade plan catches breaking changes. Maintain rollback capability by keeping previous binaries accessible. Schedule upgrades during maintenance windows despite HA claims; cascading failures during upgrades cause extended outages. Document exact versions tested together. On production clusters, I stage upgrades in identical test environments first, validating workload compatibility before touching live systems.
