Kubernetes is a powerful container orchestration platform that automates containerized applications’ deployment, scaling, and management. At the heart of Kubernetes lies the Controller Manager, a critical component of the control plane that ensures the cluster operates in the desired state.
This article details the Kubernetes Controller Manager, exploring its role, architecture, and how it interacts with other components to maintain cluster stability.
I wrote an article previously about how to create Kubernetes Operators for an application to improve the deployment process and stability.
This article will take us a step backwards to the original controllers in Kubernetes, and how they work to ensure application stability in Kubernetes.
What is the Kubernetes Controller Manager?
The Controller Manager is a daemon that runs as part of the Kubernetes control plane (or master node). It embeds multiple controllers, each of which is a control loop responsible for managing specific aspects of the cluster.
These controllers continuously monitor the state of the cluster and take corrective actions to align the current state with the desired state defined by the user.
The Controller Manager is not a single monolithic component but rather a collection of independent control loops, each designed to handle a specific type of Kubernetes resource or functionality.
Key Responsibilities of the Controller Manager
The Controller Manager ensures the following:
1. Desired State Enforcement: It ensures that the cluster’s current state matches the desired state defined in Kubernetes objects like Pods, Deployments, Services, etc.
2. Self-Healing: It automatically detects and corrects issues, such as failed Pods or unreachable nodes.
3. Scalability: It manages the scaling of workloads to meet demand.
4. Resource Management: It ensures resources like CPU, memory, and storage are allocated efficiently.
Core Controllers in the Controller Manager
The Controller Manager includes several built-in controllers, each responsible for a specific aspect of cluster management. Below is a breakdown of the most important controllers:
Node Controller:
The role of monitoring the health of nodes in the cluster involves ensuring the stability and availability of the cluster by tracking the status of each node, such as whether it is in a “Ready” or “NotReady” state. It handles node failures by detecting unreachable nodes, marking them appropriately, and evicting Pods from the affected nodes to ensure workloads are rescheduled on healthy nodes. Additionally, it synchronizes node information with cloud providers when applicable, ensuring that the cluster’s view of node states aligns with the external infrastructure, enabling seamless integration and management in cloud environments.
Replication Controller:
The role of a ReplicationController or ReplicaSet in Kubernetes is to ensure that the correct number of Pod replicas are running at all times. It continuously watches the state of ReplicationController or ReplicaSet objects and automatically creates or deletes Pods as needed to match the desired replica count, maintaining application availability and stability.
Deployment Controller:
The role of managing the lifecycle of Deployments involves overseeing the creation, updates, and maintenance of Deployment objects within the cluster. It continuously watches for changes to Deployment specifications and ensures the desired state is achieved by creating and managing ReplicaSets, which in turn control the number of Pod replicas. This role also handles rolling updates, ensuring seamless transitions between application versions by incrementally replacing old Pods with new ones, and supports rollbacks to revert to a previous stable state in case of issues. By automating these processes, it ensures applications remain available and scalable while minimizing downtime during updates.
Endpoint Controller:
The role of the Endpoints Controller in Kubernetes is to dynamically manage and update Endpoints objects for Services, ensuring efficient service discovery and communication between Pods. It continuously watches Service and Pod objects, monitoring changes in their states. When a Pod matches a Service’s selector, the controller automatically updates the Endpoints object with the IP addresses and ports of those Pods. This process ensures that Services always route traffic to the correct set of healthy, running Pods, enabling seamless load balancing and reliable inter-Pod communication.
Service Account & Token Controllers:
The role of managing service accounts and API access tokens involves ensuring secure and controlled access to cluster resources. It automatically creates default service accounts for each namespace, providing a foundational identity for Pods and workloads running within that namespace. Additionally, it generates and manages tokens associated with service accounts, which are used to authenticate and authorize API requests. By handling these tasks, it enables workloads to interact securely with the Kubernetes API and other cluster resources, while maintaining proper access controls and adhering to the principle of least privilege.
Namespace Controller:
The role of managing the lifecycle of namespaces involves overseeing the creation, maintenance, and deletion of namespaces within the cluster. It handles the creation of namespaces, providing isolated environments for resources and workloads, and ensures proper cleanup of all resources within a namespace when it is deleted. This includes removing Pods, Services, ConfigMaps, and other associated objects to prevent resource leaks and maintain cluster efficiency. By managing namespaces effectively, it enables logical separation of workloads, supports multi-tenancy, and ensures a clean and organized cluster environment.
Job Controller:
The Role of the Job Controller in Kubernetes is to efficiently manage and oversee the execution of batch jobs, ensuring that they run to completion successfully. It continuously monitors Job objects, tracking their status and execution progress. When a new Job is created, the controller ensures that the necessary Pods are provisioned to carry out the assigned workload. If a Pod fails or encounters an issue during execution, the Job Controller automatically retries the job by creating replacement Pods as needed, ensuring that the task is completed successfully. By handling job scheduling, execution, and retry mechanisms, the Job Controller provides a robust and fault-tolerant way to run one-time or finite-duration tasks in a Kubernetes cluster, making it an essential component for batch processing and background workloads.
DaemonSet Controller:
The role of the DaemonSet Controller in Kubernetes is to ensure that a specific Pod runs on all or a defined subset of nodes within a cluster, making it crucial for deploying system-level workloads such as logging agents, monitoring services, and networking components. It continuously monitors DaemonSet objects and ensures that the designated Pods are scheduled according to the node selection criteria. When a new node is added to the cluster, the DaemonSet Controller automatically creates and deploys a Pod on that node to maintain uniform distribution. Conversely, if a node is removed or a Pod fails, the controller promptly adjusts by recreating the necessary Pods on the remaining eligible nodes. This mechanism guarantees that essential background services are consistently available across the cluster, optimizing resource utilization and maintaining operational stability. By automating Pod placement based on node specifications, the DaemonSet Controller plays a vital role in managing persistent infrastructure-level workloads within Kubernetes environments.
StatefulSet Controller:
The role of the StatefulSet Controller in Kubernetes is to manage the deployment and scaling of stateful applications, ensuring that Pods maintain a consistent identity and ordered execution. Unlike standard Deployments, which treat all replicas as identical and interchangeable, a StatefulSet assigns each Pod a unique, stable network identity and persistent storage, making it ideal for applications that require stable state management, such as databases or distributed systems. The controller continuously monitors StatefulSet objects, ensuring that Pods are created, updated, or deleted in a well-defined, sequential order. This guarantees that when a StatefulSet scales up or down, Pods are added or removed in a controlled manner, respecting dependencies between instances. Additionally, if a Pod fails, the StatefulSet Controller ensures that it is replaced while preserving its identity and persistent volume, preventing data inconsistencies. By handling the ordered deployment, scaling, and deletion of Pods, the StatefulSet Controller plays a critical role in maintaining data integrity and reliable communication in applications that require stable and predictable behavior across restarts.
How Controllers Work: The Control Loop
All controllers follow a similar pattern called the control loop. This loop consists of the following steps:
1. Watch: The controller watches the state of specific resources (e.g., Pods, Nodes, Services) via the Kubernetes API server.
2. Compare: It compares the current state of the resource with the desired state defined in the resource’s specification.
3. Reconcile: If there is a discrepancy, the controller takes corrective action to align the current state with the desired state.
4. Update: The controller updates the API server with the new state.
For example:
– If a Pod crashes, the Replication Controller creates a new Pod to replace it.
– If a Node goes down, the Node Controller marks it as unavailable and reschedules its workloads.
This is the same concept used in ArgoCD to maintain the state of applications deployed in Kubernetes. The difference here is; ArgoCD uses the Git repository as the source of tried to compare and reconcile and update, while the Kubernetes Control Manager uses etcd as its source of truth for comparison.
Interaction with Other Kubernetes Components
The Controller Manager does not operate in isolation. It interacts closely with other components of the Kubernetes control plane:
API Server:
– The Controller Manager communicates with the API server to:
– Fetch the desired state of resources.
– Update the current state of resources.
– Watch for changes to resources.
Scheduler:
– The Controller Manager works with the scheduler to ensure Pods are assigned to nodes.
– For example, when a new Pod is created, the Controller Manager ensures it is scheduled and running.
kubelet:
– The Controller Manager interacts with kubelets on nodes to manage Pods.
– For example, it instructs kubelets to start or stop Pods.
High Availability and Leader Election
In a highly available Kubernetes cluster, multiple instances of the Controller Manager can run simultaneously to provide redundancy and fault tolerance. However, to prevent conflicts and ensure consistency in managing the cluster’s control loops, only one instance is active at any given time. This is achieved through a critical mechanism known as leader election. The leader is the active instance that executes all control loop operations, managing resources such as Deployments, StatefulSets, DaemonSets, and other cluster components. Meanwhile, the remaining instances remain in a standby state, continuously monitoring the leader’s health and waiting for an opportunity to take over if necessary. If the current leader fails due to a crash, network partition, or other disruptions, one of the standby instances seamlessly takes over leadership, ensuring that the cluster’s core functions remain operational without manual intervention. This automatic failover mechanism enhances reliability and high availability, preventing downtime and maintaining the stability of Kubernetes workloads. By implementing leader election, Kubernetes ensures that even in the face of failures, the cluster remains resilient, self-healing, and efficiently managed.
Configuration and Tuning
The Controller Manager can be configured with various flags to adjust its behavior. Some common flags include:
– `–cluster-cidr`: Specifies the cluster CIDR for Pod IPs.
– `–node-monitor-period`: Sets the interval for node health checks.
– `–leader-elect`: Enables leader election for high availability.
– `–controllers`: Specifies which controllers to enable or disable.
Example Workflow
To illustrate how the Controller Manager works, consider the following example:
1. A user creates a Deployment with 3 replicas.
2. The Deployment Controller detects the new Deployment and creates a ReplicaSet.
3. The ReplicaSet Controller ensures 3 Pods are running.
4. If a Pod fails, the ReplicaSet Controller creates a new Pod to replace it.
5. The Node Controller monitors node health and reschedules Pods if a node fails.
Conclusion
The Kubernetes Controller Manager is a critical component of the Kubernetes control plane. It ensures the cluster operates in the desired state by continuously monitoring and reconciling the state of resources. By embedding multiple controllers, each responsible for a specific aspect of cluster management, the Controller Manager provides a robust and scalable mechanism for maintaining cluster stability and reliability.
Understanding how the Controller Manager works is essential for Kubernetes administrators and developers, as it forms the foundation of Kubernetes’ self-healing and automation capabilities. Whether you’re managing a small cluster or a large-scale production environment, the Controller Manager plays a vital role in ensuring your applications run smoothly.
