kubelet Explained: The Node Agent

By Kokil Thapa | Last reviewed: August 2026

If you are debugging why a pod stays in Pending, crashes immediately after starting, or fails to report metrics, you are almost always looking at a problem with kubelet explained: the node agent. While the control plane makes global scheduling decisions, the kubelet is the local enforcer on every worker node, translating API server specs into running containers. Understanding this component is non-negotiable for anyone operating production Kubernetes clusters, whether you are managing a simple Laravel application deployment or a complex microservices mesh. For developers accustomed to traditional VPS setups like those described in my guide on deploying Symfony on Ubuntu VPS, the shift from direct process management to an agent-driven declarative model requires a fundamental mental adjustment.

What is the core function of kubelet explained: the node agent?

The kubelet is essentially a systemd-managed daemon that runs on every single node in a Kubernetes cluster. Its primary mandate is simple but critical: ensure that the set of containers described in PodSpecs are running and healthy. Unlike higher-level controllers that run in the control plane, the kubelet operates locally with direct access to the node's resources, filesystem, and container runtime interface (CRI).

In modern Kubernetes (v1.32+), the kubelet no longer speaks directly to Docker. Instead, it communicates via the CRI standard using gRPC. This abstraction allows it to manage containerd, CRI-O, or any compliant runtime without code changes. When you deploy a PHP application or a Node.js service, the scheduler assigns the pod to a node, but it is the kubelet that actually pulls the image, creates the sandbox, mounts secrets, and starts the application process.

Beyond just starting containers, the kubelet handles several background responsibilities that keep the cluster functional:

• Node Registration: On startup, it registers itself with the API server and continuously updates its status including capacity, allocatable resources, and conditions.
• Volume Management: It mounts and unmounts volumes specified in the pod spec, handling CSI driver interactions when necessary.
• Secret and ConfigMap Injection: It projects sensitive data and configuration into the pod filesystem or environment variables securely.
• Garbage Collection: It cleans up unused images and terminated containers to prevent disk exhaustion, a common issue on nodes with limited storage.
• Metric Export: It exposes cAdvisor-based metrics via the /metrics/resource endpoint for HPA and monitoring systems.

How does kubelet handle pod lifecycle and health probes?

The most visible job of the kubelet is enforcing the pod lifecycle. When a pod is assigned to a node, the kubelet watches for changes via the API server (or static files). It then orchestrates a precise sequence of operations through the CRI. Understanding this sequence helps explain why pods sometimes hang in ContainerCreating or restart unexpectedly.

The Pod Startup Sequence

1. Sandbox Creation: The kubelet asks the runtime to create a pause container that holds the network namespace and IP address.
2. Image Pull: If the required image isn't present locally, it initiates a pull. Large images or slow registries block here.
3. Volume Mount: Persistent volumes, secrets, and configmaps are attached to the pod's directory structure.
4. Container Start: Application containers are created and started within the established sandbox.
5. Post-Start Hook: If defined, exec or HTTP hooks run inside the container before it's marked ready.

Probe Execution and Restart Logic

Once running, the kubelet becomes a watchdog. It executes three types of probes at configured intervals:

• Liveness Probe: Determines if the container is alive. Failure triggers a container restart by the kubelet.
• Readiness Probe: Determines if the container can serve traffic. Failure removes the pod from service endpoints without restarting it.
• Startup Probe: Protects slow-starting applications from premature liveness kills. Only after this succeeds do other probes activate.

A common mistake I see in production Laravel deployments is setting aggressive liveness probes without accounting for cache warming or database migrations during boot. The kubelet will faithfully kill and restart the container indefinitely if the probe fails, creating a crash loop. Always use startup probes for applications that take more than 30 seconds to initialize.

How do you configure and troubleshoot kubelet in production?

Configuration drift and misdiagnosis are the biggest operational risks with kubelet. Since Kubernetes 1.28+, the dynamic configuration feature has been removed entirely. All kubelet configuration must now be provided via the --config flag pointing to a local file, typically /var/lib/kubelet/config.yaml on Linux systems managed by kubeadm.

Essential Configuration Parameters

When tuning kubelet for production workloads, these parameters have the highest impact:

<!-- /var/lib/kubelet/config.yaml example -->
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
serializeImagePulls: false
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
containerLogMaxSize: "50Mi"
containerLogMaxFiles: 5
featureGates:
  SidecarContainers: true

• maxPods: Defaults to 110. Increase only if your node has sufficient IP addresses and memory. Each pod consumes kernel resources regardless of CPU usage.
• serializeImagePulls: Set to false to allow parallel image pulls. Critical for nodes running many different services where cold start latency matters.
• evictionHard: Defines thresholds where kubelet starts killing pods to reclaim resources. Tune based on your workload's tolerance for disruption versus OOM risk.
• containerLogMaxSize/Files: Prevents log files from filling the disk. Default values are often too generous for high-output applications.

Debugging Common Issues

When pods fail to start or nodes show NotReady, follow this diagnostic path:

1. Check kubelet logs: journalctl -u kubelet -f --no-pager reveals immediate errors like CRI failures, mount permission issues, or certificate expiration.
2. Verify CRI connectivity: Use crictl ps and crictl info to confirm the runtime is responsive independently of kubelet.
3. Inspect node conditions: kubectl describe node <name> shows MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable flags set by kubelet.
4. Validate resource limits: Check if eviction thresholds are triggering due to unexpected log growth or temporary file accumulation.

For teams managing infrastructure alongside application code, integrating these checks into automated pipelines saves hours. My article on DevOps automation practices covers patterns for embedding node health validation into CI/CD workflows.

Kubelet vs Kube-proxy vs Container Runtime: What's the difference?

Confusion between node-level components leads to wasted debugging time. Each has a distinct boundary of responsibility.

The key distinction: kubelet decides what should run and when; the runtime decides how to run it; kube-proxy decides how traffic reaches it. When a pod starts but receives no traffic, check kube-proxy and CNI. When a pod never starts at all, check kubelet and the runtime.

Why does kubelet matter for application reliability and security?

The kubelet is ultimately the gatekeeper of your application's runtime environment. Its configuration directly determines whether your services survive resource contention, whether sensitive data leaks between pods, and whether compromised containers can escape their boundaries. Security hardening at the kubelet level provides defense-in-depth that complements network policies and admission controllers.

Key security considerations include disabling read-only port access (--read-only-port=0), enforcing TLS authentication for all API endpoints, and restricting allowed volume types via allowedUnsafeSysctls only when absolutely necessary. For multi-tenant environments or legal-tech platforms handling sensitive documents, ensuring proper isolation at the kubelet level is as important as application-layer security. Teams building secure client portals should review server security fundamentals alongside Kubernetes-specific hardening guides.

Reliability also depends on understanding kubelet's garbage collection behavior. Without proper tuning, nodes accumulate dead containers and unused images until disk pressure triggers mass evictions. Setting imageGCHighThresholdPercent and imageGCLowThresholdPercent appropriately prevents this silent failure mode. Similarly, configuring maxOpenFiles and maxProcs in the systemd unit prevents resource exhaustion under high pod density.

Practical Takeaways for Operating Kubelet

Mastering kubelet explained: the node agent transforms how you operate Kubernetes. Stop treating nodes as black boxes and start viewing them as managed systems with observable, tunable agents. Monitor kubelet metrics proactively, version-lock your configurations, and test eviction scenarios before they hit production. Whether you're running e-commerce platforms, legal-tech portals, or internal tools, reliable kubelet operation is the foundation everything else depends on. If you need help designing resilient infrastructure or troubleshooting persistent node issues, reach out to discuss your specific requirements.