The Kubernetes Worker node is the engine of workloads in the Kubernetes cluster. If the Worker Node has an issue, it affects the pods running within it and every other workload, so it is a crucial part of the Kubernetes cluster.
In previous articles, I explained the meaning of Kubernetes and described the function of the control manager in the master node of a Kubernetes cluster.
In this article, we shall open a Kubernetes worker node and see what is inside. We shall zoom into the different components, their primary functions, and how they work together to ensure the smooth running of pods and other workloads in a Kubernetes cluster.
What is a Kubernetes Worker Node ?
A Kubernetes worker node is a machine—physical or virtual—responsible for running containerized workloads within a Kubernetes cluster. It is managed by the control plane, which schedules and distributes workloads across nodes. Each worker node contains essential components that allow it to communicate with the control plane, run containers, manage networking, and handle storage. These components include the Kubelet, Kube-Proxy, and a Container Runtime. Worker nodes are the backbone of a Kubernetes cluster, ensuring that applications run reliably and efficiently.
Components
1. Kubelet: The Node’s Primary Agent
The Kubelet is a critical component of every worker node in a Kubernetes cluster. It acts as an agent that continuously communicates with the Kubernetes control plane, receiving instructions about which pods should be scheduled on the node. The Kubelet ensures that containers are running according to the pod specifications defined in Kubernetes YAML manifests.
Functions of Kubelet:
Pod Management:
Pod Management is a primary responsibility of the Kubelet, the node agent that runs on every Kubernetes worker node. The Kubelet interacts with the Kubernetes API server to receive Pod definitions (specified in PodSpecs via YAML or JSON files) and ensures that the required containers are running on the node as defined. It continuously monitors the state of the Pods and their containers, comparing the actual state with the desired state specified by the control plane. If a container crashes or a Pod is deleted, the Kubelet takes corrective action, such as restarting the container or notifying the control plane to reschedule the Pod. The Kubelet also handles tasks like pulling container images from registries, mounting storage volumes, and applying resource limits to containers. Additionally, it reports the status of the node and its Pods back to the control plane, ensuring that the cluster remains in the desired state. By managing Pods effectively, the Kubelet plays a crucial role in maintaining the reliability, scalability, and performance of applications running on the worker node.
Container Health Monitoring:
Container Health Monitoring is a vital function performed by the Kubelet on a Kubernetes worker node to ensure the reliability and availability of applications running in containers. The Kubelet continuously monitors the health of running containers by checking their status against the liveness and readiness probes defined in the Pod specifications. Liveness probes determine whether a container is still running and functioning correctly, while readiness probes assess whether a container is ready to serve traffic. If a container fails a liveness probe, the Kubelet automatically restarts it to restore functionality. If a container fails a readiness probe, the Kubelet ensures it is removed from the service load balancer until it becomes healthy again. This proactive monitoring and remediation process helps maintain the desired state of the Pods and ensures that applications remain available and responsive. By enforcing these health checks, the Kubelet plays a critical role in minimizing downtime, improving application resilience, and ensuring that the Kubernetes cluster operates smoothly and efficiently.
Node Status Reporting:
The Kubelet is responsible for collecting and reporting the status and resource utilization of a node to the Kubernetes control plane. It continuously monitors key metrics such as CPU, memory, disk usage, and network health, ensuring that the node remains in an optimal state for scheduling and workload execution. This information is sent to the API Server, where it is used by the Scheduler and Controller Manager to make decisions regarding pod placement, scaling, and resource allocation. The Kubelet also checks the node’s health by running periodic status checks and reporting conditions such as Ready, MemoryPressure, DiskPressure, and NetworkUnavailable. If a node becomes unhealthy or unresponsive, Kubernetes can automatically reschedule workloads to healthy nodes to maintain high availability. This reporting mechanism ensures that cluster administrators have real-time insights into node performance, allowing them to optimize infrastructure usage and detect potential failures before they impact application workloads.
Interaction with the Container Runtime:
The Kubelet interacts directly with the container runtime (such as containerd, CRI-O, or Docker) to manage the lifecycle of containers running on a node. When a PodSpec is assigned to a node, the Kubelet communicates with the container runtime through the Container Runtime Interface (CRI) to pull the required container images, create containers, and ensure they run according to the pod definition. It continuously monitors the state of containers, ensuring they remain healthy and responsive. If a container crashes or stops unexpectedly, the Kubelet works with the runtime to automatically restart it based on the pod’s restart policy. Additionally, when a pod is scheduled for termination, the Kubelet gracefully stops its containers, ensuring any preStop hooks or cleanup operations are executed. This close interaction between the Kubelet and the container runtime is critical for maintaining the desired state of workloads, optimizing resource utilization, and ensuring high availability in a Kubernetes cluster.
Without the Kubelet, a worker node would not be able to receive workloads, making it an essential component in Kubernetes cluster operations.
2. Kube Proxy: The Network Manager
The Kube-Proxy is responsible for managing networking rules on each worker node. It ensures that applications running in different pods can communicate with each other and external services. Kubernetes uses a flat networking model, where each pod gets a unique IP address, and Kube-Proxy enables seamless routing of traffic between them.
Functions of Kube Proxy:
Service Discovery and Load Balancing:
These are critical functions managed by the Kube Proxy on a Kubernetes worker node. Kube Proxy maintains network rules that enable Kubernetes services to route requests to the appropriate Pods, ensuring seamless communication within the cluster. When a service is created in Kubernetes, it is assigned a virtual IP (ClusterIP) that acts as a stable endpoint for other services or external clients to access the Pods backing that service. Kube Proxy monitors changes to the cluster, such as the creation, deletion, or movement of Pods, and dynamically updates the network rules (using IP tables or IPVS) to reflect these changes. This ensures that incoming requests to the service are load-balanced across all healthy Pods, distributing traffic evenly and preventing any single Pod from becoming a bottleneck. Additionally, Kube Proxy enables service discovery by allowing Pods to locate and communicate with other services using their DNS names, abstracting away the complexity of managing individual Pod IPs. Together, these mechanisms ensure reliable and efficient communication between services and their underlying Pods, even in a highly dynamic and scalable environment.
Packet Forwarding:
Packet Forwarding is a fundamental function performed by Kube Proxy on a Kubernetes worker node to ensure seamless communication between Pods, both within the same node and across different nodes in the cluster. When a Pod sends a network request to another Pod, Kube Proxy intercepts the traffic and uses its configured network rules (via IP tables or IPVS) to determine the correct destination. For Pods on the same node, Kube Proxy efficiently routes the packets directly to the target Pod, minimizing latency. For Pods on different nodes, Kube Proxy forwards the packets to the appropriate node by encapsulating the traffic and ensuring it reaches the destination Pod. This process is transparent to the applications running in the Pods, as Kube Proxy abstracts away the complexity of network routing. By managing packet forwarding, Kube Proxy ensures that communication between Pods is reliable, secure, and efficient, regardless of their location in the cluster. This capability is essential for maintaining the performance and scalability of distributed applications in Kubernetes.
Network Policy Enforcement:
Network Policy Enforcement is a critical function managed by Kube Proxy and other network plugins (such as Calico, Cilium, or Weave Net) on a Kubernetes worker node. It ensures that only authorized network connections are established between Pods, based on the rules defined in Kubernetes Network Policies. Network Policies act as a firewall for Pods, specifying which Pods can communicate with each other and on which ports. When a Network Policy is applied, Kube Proxy and the network plugin work together to enforce these rules by configuring the underlying network infrastructure, such as IP tables or eBPF (extended Berkeley Packet Filter). For example, if a Network Policy restricts incoming traffic to a specific Pod, Kube Proxy ensures that only traffic from the allowed sources can reach that Pod, blocking all other connections. This granular control over network traffic enhances the security of the cluster by preventing unauthorized access and limiting the potential attack surface. By enforcing Network Policies, Kubernetes ensures that applications running in the cluster adhere to the principle of least privilege, reducing the risk of lateral movement by malicious actors and maintaining a secure and compliant environment.
Traffic Routing via iptables or IPVS:
Traffic Routing via iptables or IPVS is a core functionality of Kube Proxy that enables efficient and scalable network communication within a Kubernetes cluster. By default, Kube Proxy uses iptables, a Linux firewall utility, to manage network traffic routing. iptables works by creating and maintaining a set of rules that dictate how packets should be forwarded between Pods and services. While iptables is reliable and widely used, it can become inefficient in large-scale clusters due to the sheer number of rules that need to be processed, leading to increased latency and higher CPU usage. To address this, Kubernetes also supports IPVS (IP Virtual Server), a more advanced traffic routing mechanism built into the Linux kernel. IPVS is specifically designed for load balancing and offers superior performance for large-scale clusters. It uses hash tables to manage network rules, resulting in faster packet processing and lower overhead compared to iptables. Additionally, IPVS supports a variety of load-balancing algorithms, such as round-robin, least connections, and destination hashing, making it more flexible and efficient for handling high traffic volumes. By leveraging either iptables or IPVS, Kube Proxy ensures that network traffic is routed correctly and efficiently, enabling seamless communication between services and Pods while maintaining the scalability and performance of the Kubernetes cluster.
Kube-Proxy ensures that microservices running in different pods can communicate securely and efficiently across the cluster.
3. Container Runtime: Running Containers
The Container Runtime is the software responsible for executing and managing containers on the worker node. Kubernetes supports multiple container runtimes, such as:
Docker: One of the most widely used container runtimes, though Kubernetes now favors containerd and CRI-O.
containerd: A lightweight, high-performance container runtime optimized for Kubernetes.
CRI-O: A Kubernetes-native runtime designed for Open Container Initiative (OCI) images.
Functions of the Container Runtime:
Pulling Container Images:
The Kubelet is responsible for pulling container images from container registries such as Docker Hub, Amazon Elastic Container Registry (ECR), Google Container Registry (GCR), and private registries whenever a pod is scheduled to run on a node. When a PodSpec defines a container with a specific image, the Kubelet checks if the required image is already present on the node. If not, it communicates with the container runtime (e.g., containerd or Docker) to fetch the image from the specified registry. If authentication is required, Kubernetes uses image pull secrets to retrieve the necessary credentials. The image pull policy (Always, IfNotPresent, or Never) determines whether the image should be fetched from the registry or reused from the local cache. Efficient image pulling is crucial for fast container startup, scalability, and reducing deployment latency, ensuring that applications can be quickly launched and updated across the cluster.
Starting and Stopping Containers:
The Kubelet is responsible for managing the lifecycle of containers on a node by ensuring that they are started, stopped, and restarted as required based on the Kubernetes pod specifications. When a pod is scheduled to a node, the Kubelet communicates with the container runtime (such as containerd or CRI-O) to create and start the necessary containers. It continuously monitors the health and status of these containers, checking if they are running as expected. If a container crashes or stops unexpectedly, the Kubelet will restart it according to the pod’s restart policy (Always, OnFailure, or Never). Additionally, when a pod is deleted or scheduled for termination, the Kubelet ensures a graceful shutdown, allowing containers to complete any cleanup operations by executing preStop hooks before stopping them. This container lifecycle management is crucial for maintaining high availability, fault tolerance, and ensuring that workloads remain in the desired state as defined by Kubernetes.
Managing Container Lifecycle:
The Kubelet plays a critical role in managing the lifecycle of containers running on a Kubernetes node, ensuring they remain healthy, responsive, and persist data as needed. It continuously monitors each container’s health and status, leveraging liveness, readiness, and startup probes to detect failures and determine whether a container should be restarted or removed. If a container crashes, the Kubelet restarts it according to the pod’s restart policy. Additionally, it ensures that persistent storage volumes are correctly mounted and available for containers that require data persistence. When a pod is scheduled for termination, the Kubelet coordinates a graceful shutdown, executing any defined preStop hooks before stopping the container. This comprehensive lifecycle management ensures that applications remain highly available, resilient, and consistent across Kubernetes clusters.
Interfacing with Kubelet:
Interfacing with Kubelet is a critical function of the container runtime, which works under the Kubelet’s instructions to execute container operations seamlessly. The Kubelet communicates with the container runtime through the Container Runtime Interface (CRI), a standardized API that abstracts the details of container management. When the Kubelet receives Pod specifications from the Kubernetes control plane, it translates these into container runtime-specific commands, such as pulling container images, creating and starting containers, and stopping or deleting them when no longer needed. The container runtime, in turn, executes these commands and ensures that the containers are running as expected. It also provides the Kubelet with real-time information about the state of the containers, such as their resource usage and health status. This close collaboration between the Kubelet and the container runtime ensures that containerized applications are deployed, managed, and monitored efficiently, enabling Kubernetes to maintain the desired state of the cluster and deliver a reliable platform for running workloads.
Without a container runtime, worker nodes would not be able to execute containerized applications, making it a fundamental component of the Kubernetes architecture.
How Kubernetes Worker Node Components Work Together
The Kubelet, Kube-Proxy, and Container Runtime work in tandem to ensure Kubernetes worker nodes operate efficiently. When a pod is scheduled on a worker node, the Kubernetes scheduler assigns it based on resource availability and scheduling policies.
The Kubelet retrieves the pod specification from the API server and ensures the required containers are running, while the Container Runtime pulls the necessary container images and launches them. Simultaneously, Kube-Proxy configures network routing rules to enable communication between the new pod and other services or pods in the cluster.
The Kubelet continuously monitors the pod’s health and resource usage, restarting failed containers via the Container Runtime and reporting status updates to the control plane. Kube-Proxy enforces network policies and ensures traffic flows correctly.
Together, these components enable Kubernetes worker nodes to efficiently run containerized applications, maintain seamless network connectivity, and provide self-healing capabilities, ensuring the cluster remains robust, scalable, and resilient.
Conclusion
The coordination of the components of the Worker nodes in collaboration with the master node ensures the smooth running of services in a Kubernetes cluster.
Each worker node component has a unique functionality and ensuring they keep running is crucial to the reliability and availability of your workload.
The Kubelet ensures pods are running correctly, Kube-Proxy manages networking and service discovery, and the Container Runtime handles container execution.
