kube-controller-manager and Control Loops

By Kokil Thapa | Last reviewed: September 2026

When a Deployment rolls out three replicas but only two Pods stay Running, something must notice the gap and act. That job belongs to kube-controller-manager and Control Loops — the control-plane component that watches the Kubernetes API and drives the cluster toward the state you declared. If you run Laravel on Kubernetes, operate a Linux-managed production stack, or debug why a Service never gets Endpoints, understanding these loops separates guesswork from engineering. This guide maps the reconcile pattern, lists the built-in controllers, and shows how to trace failures before they become outages.

What is kube-controller-manager and how do control loops work?

kube-controller-manager is a control-plane binary. It bundles dozens of small controllers into one process. Each controller implements the same pattern: watch, compare, act, repeat.

That pattern is a control loop. You declare desired state in YAML. The API server stores it in etcd. A controller reads that record, inspects what actually runs, and patches the API when they diverge. The loop never exits. It runs until someone deletes the object or shuts down the manager.

This design mirrors how you might keep a web app healthy outside Kubernetes. A cron job checks queue depth. A script restarts PHP-FPM when memory spikes. Kubernetes formalises that feedback cycle at cluster scale. The difference is uniform APIs, shared watch machinery, and idempotent handlers.

For broader context on where this component sits, read the Kubernetes architecture control-plane and nodes overview. The manager sits beside kube-apiserver and etcd. It does not schedule Pods directly — that is kube-scheduler — and it does not route packets — that is kube-proxy.

The four steps every control loop follows

Every built-in controller follows the same reconcile skeleton, even when the objects differ.

1. Watch — Subscribe to add/update/delete events for one resource type through a shared informer cache.
2. Enqueue — Push the object key (namespace/name) onto a rate-limited work queue when something changes.
3. Reconcile — Read desired spec and observed status, then compute the delta.
4. Act — Create, update, or delete dependent objects via the API until the delta is zero or an error blocks progress.

Errors do not crash the process. The item re-queues with exponential backoff. That resilience is why a transient etcd blip rarely requires a manual restart. Persistent errors surface as Events on the affected object and as log lines on the control-plane node.

How does kube-controller-manager reconcile desired vs actual state?

Reconciliation is the heart of kube-controller-manager and Control Loops. Desired state lives in spec fields. Actual state lives in status fields and in related objects the controller creates.

Consider a Deployment with replicas: 3. The deployment controller sees a ReplicaSet whose .status.readyReplicas is 2. It does not patch the Deployment spec. Instead it ensures the ReplicaSet spec asks for three Pod templates. The ReplicaSet controller then creates a third Pod. kubelet on a worker node starts the container. The kubelet posts Pod status back to the API. On the next reconcile pass, the deployment controller reads readyReplicas = 3 and stops.

Level-triggered logic drives this behaviour. Edge-triggered scripts fire once on change. Control loops re-evaluate on every sync, even if the triggering event was missed. That property matters after network partitions or apiserver restarts.

The kube-apiserver guide explains optimistic concurrency and resourceVersion. Controllers rely on both. A stale read triggers a retry rather than corrupt state.

Shared informers and why they matter

Running a separate watch stream per controller would crush apiserver. Instead, kube-controller-manager uses shared informers. One list-and-watch call per resource type populates a local cache. Every controller reads from that cache.

Benefits include lower apiserver load and faster reconcile reads. Trade-offs include slight staleness — usually milliseconds — and the need to understand resync intervals. A full resync re-queues every cached object periodically. That catches missed events and heals silent drift.

Idempotency and finalizers

Reconcile handlers must be idempotent. Calling the same logic twice should not create duplicate Pods or double-scale a ReplicaSet. Controllers achieve this with deterministic names, owner references, and generation-aware status updates.

Finalizers extend the loop. When you delete a Pod backed by a PersistentVolumeClaim, the PV protection controller may hold the object until storage is released. The object stays in a terminating phase until every finalizer clears. That is still reconciliation — desired state becomes "gone," but prerequisites must finish first.

Which controllers run inside kube-controller-manager?

The upstream binary ships many controllers. Cloud providers may run additional ones out-of-tree, but the core set below covers day-to-day cluster behaviour. Official reference: the kube-controller-manager command reference on kubernetes.io.

Some clusters split cloud-specific loops into cloud-controller-manager. On AWS or GCP that binary handles routes, load balancers, and node lifecycle hooks tied to the provider API. The split keeps vendor code out of the core binary while preserving the same watch-compare-act pattern.

Networking controllers interact closely with kube-proxy iptables vs IPVS modes. The Service controller publishes EndpointSlices. kube-proxy on each node programs rules from those slices. A break anywhere in that chain looks like "Service has no endpoints" in kubectl output.

Controllers you can disable selectively

Large platforms sometimes disable individual controllers with --controllers flags when an out-of-tree replacement exists. Disabling service or deployment on a live cluster is almost never correct. Disabling cloud-specific loops during migration is more common.

# Static pod manifest excerpt — controller-manager container args
- command:
  - kube-controller-manager
  - --cluster-cidr=10.244.0.0/16
  - --service-cluster-ip-range=10.96.0.0/12
  - --leader-elect=true
  - --controllers=*,bootstrapsigner,tokencleaner
  - --bind-address=127.0.0.1
  - --secure-port=10257

The --leader-elect=true flag is critical for HA setups. Only one replica holds the lease at a time. Standby instances idle until the leader stops renewing. Details appear in the highly available Kubernetes control plane article.

How do you configure and debug kube-controller-manager in production?

On kubeadm clusters the manager usually runs as a static Pod under /etc/kubernetes/manifests/kube-controller-manager.yaml. Managed services hide the manifest but expose the same flags through their control-plane settings.

Health checks and metrics

The secure port defaults to 10257 on modern releases. Liveness and readiness probes hit /healthz. Prometheus scrapes /metrics when you enable RBAC for the scraping ServiceAccount.

# Confirm the leader pod and recent logs (control-plane node)
kubectl -n kube-system get pods -l component=kube-controller-manager
kubectl -n kube-system logs kube-controller-manager-control-plane1 --tail=100

# Hit health endpoint from the node (replace with your node IP)
curl -k https://127.0.0.1:10257/healthz

Watch for repeated reconcile errors on one controller name. A flood of FailedCreate Events on a ReplicaSet often traces back to admission webhook rejection — covered in the admission controllers and webhooks guide.

Common failure modes

• Leader election loss — Clock skew or etcd latency prevents lease renewal. Standby takes over after --leader-elect-lease-duration expires.
• RBAC denial — Custom roles that trim control-plane permissions break controllers silently until logs show Forbidden errors.
• Quota exhaustion — ReplicaSet controller loops forever if Namespace resource quotas block Pod creation.
• Orphaned objects — Removing owner references leaves garbage the collector never deletes.
• Version skew — Controller-manager more than one minor version behind apiserver triggers unsupported field errors.

When you operate mixed stacks — Laravel APIs on Kubernetes plus bare-metal cron — treat controller logs like application logs. Ship them to your central store. Correlate spikes with deploy times. The AIOps for modern infrastructure piece discusses alert routing patterns that apply here.

For teams that prefer managed platforms but still need custom reconcile logic, enterprise application development engagements often include operator design alongside the app itself.

HA topology checklist

1. Run an odd number of control-plane nodes — three is the usual minimum for production.
2. Enable --leader-elect=true on every controller-manager instance.
3. Keep etcd latency under roughly 10 ms between peers; higher values delay writes controllers depend on.
4. Match controller-manager version to kube-apiserver within one minor release.
5. Back etcd before upgrades; controllers replay watches after restart but cannot reconstruct unsaved state.

Certificate rotation for the control plane ties into cert-manager automated TLS on workloads. The manager itself uses certificates kubeadm generates. Expired apiserver certs stop watches entirely — every controller stalls at once.

How do control loops differ from operators and custom controllers?

Built-in controllers ship inside kube-controller-manager. Operators and custom controllers use the same reconcile pattern but run as separate Deployments. They call the same APIs through client-go or controller-runtime.

The Kubernetes controller concept documentation describes both in one model. Custom controllers watch CRDs — for example a DatabaseBackup kind — and create Jobs or Secrets to fulfil spec.

Operators add domain knowledge. cert-manager, discussed in Kubernetes Ingress and TLS with cert-manager, watches Certificate objects and talks to ACME providers. That is a control loop, but not part of the core binary.

When to extend vs configure

Prefer built-in resources first. A Deployment plus ConfigMap beats a bespoke CRD when standard fields suffice. Reach for operator patterns when reconciliation spans external systems — DNS providers, payment gateways, legacy queues — that the core controllers cannot see.

controller-runtime abstracts informers, queues, and leader election. Your reconcile function returns RequeueAfter for timed retries. The library matches what kube-controller-manager does internally, which lowers the learning curve for teams already shipping REST API integrations.

GitOps tools like Argo CD or Flux add another layer. They reconcile Git commits to cluster manifests. Application controllers inside the cluster then reconcile those manifests to running Pods. You get nested loops — each level idempotent, each level able to drift independently. The Terraform import guide covers a related problem: bringing live infrastructure under declarative control after the fact.

Testing reconcile logic locally

envtest spins up a real apiserver and etcd without nodes. Unit tests call your Reconcile function against that API. Integration tests on kind or minikube catch RBAC and webhook interactions informers miss.

# Quick check: which controller owns this Pod?
kubectl get pod my-app-7d8f9c-abcde -o jsonpath='{.metadata.ownerReferences[*].kind}'
# → ReplicaSet

kubectl get rs my-app-7d8f9c -o jsonpath='{.metadata.ownerReferences[*].kind}'
# → Deployment

Paste complex ownerReference output into the JSON formatter tool when sharing traces with teammates. Readable JSON speeds incident reviews.

Projects like Adventure Third Pole Trek run Laravel plus Livewire on traditional VPS hosting today. When traffic warrants Kubernetes, the same reconcile mindset applies — desired booking capacity becomes replica count, health checks become probes, and controllers replace manual restart scripts.

Key Takeaways

• kube-controller-manager and Control Loops watch API objects, compare spec to status, and act until drift reaches zero.
• Shared informers and work queues keep apiserver load manageable while preserving level-triggered correctness.
• Deployment, Service, Node, and garbage-collection controllers share one binary — cloud-specific loops may live in cloud-controller-manager.
• Enable leader election on HA control planes and keep controller-manager within one minor version of kube-apiserver.
• Debug with object Events, ownerReference chains, and controller-manager logs before editing specs.
• Custom operators reuse the same pattern via controller-runtime; prefer built-in resources when they fit.

People Also Ask

Is kube-controller-manager the same as kube-scheduler?

No. kube-scheduler assigns Pods to nodes based on resources and constraints. kube-controller-manager runs loops that maintain Deployments, Services, ReplicaSets, and other cluster objects. Scheduling happens once per Pod; controllers run continuously afterward.

What happens if kube-controller-manager stops?

Existing Pods keep running — kubelet does not tear them down. New drift goes unrepaired. Deleted Deployments may leave orphaned ReplicaSets. Scale changes stall. Other control-plane components continue, but declared state slowly diverges from reality until the manager returns.

How often do control loops run?

Event-driven wakes happen on every relevant API change. Periodic resync re-queues cached objects at a configured interval — often minutes — even without new events. Failed reconciles retry with exponential backoff capped by the controller.

Can I write my own controller without operator-sdk?

Yes. client-go provides informers and work queues directly. controller-runtime and operator-sdk add scaffolding, metrics, and leader election defaults. Many production teams use controller-runtime alone for smaller CRDs.

Put control loops to work on your cluster

Understanding kube-controller-manager and Control Loops turns kubectl output from mystery into a traceable chain of intent and action. Start with owner references on the failing object. Read Events. Pull controller-manager logs from the leader Pod. Fix the root cause — quota, webhook, RBAC, or spec error — and let the loop heal the cluster.

If you are moving workloads to Kubernetes or need help hardening a control plane, see support and maintenance services or testing and optimization for production readiness reviews. For greenfield platforms that embed custom reconcile logic, custom software development covers operators and APIs together. Read more on the blog, browse the portfolio, or contact us to discuss your cluster architecture.