Build a Kubernetes Controller to Watch for Changes to CRD

The vast and intricate ecosystem of Kubernetes has fundamentally reshaped how applications are deployed, managed, and scaled in modern cloud-native environments. At its core, Kubernetes offers a powerful declarative model, where users define the desired state of their applications, and the system tirelessly works to achieve and maintain that state. This elegant paradigm is largely powered by a set of sophisticated control loops, known as controllers, which constantly monitor the cluster's actual state and reconcile it with the declared desired state. While Kubernetes provides a rich set of built-in resource types like Pods, Deployments, and Services, real-world applications often necessitate custom abstractions that extend beyond these defaults. This is precisely where Custom Resource Definitions (CRDs) emerge as a pivotal mechanism, allowing developers to define their own application-specific resources and integrate them seamlessly into the Kubernetes API.

The ability to define custom resources unlocks unparalleled flexibility, enabling Kubernetes to manage virtually any type of workload or infrastructure component. However, merely defining a custom resource is only half the battle. For these custom resources to become truly "native" and functional within the Kubernetes ecosystem, they require an active agent – a dedicated controller – to observe changes to their instances and react intelligently. This powerful combination of CRDs and custom controllers forms the bedrock of the Operator pattern, a design philosophy for packaging, deploying, and managing a Kubernetes-native application. Building such a controller involves a deep understanding of Kubernetes internals, its api interactions, and the mechanics of developing robust, fault-tolerant control loops.

This extensive guide embarks on a journey to demystify the process of building a Kubernetes controller specifically designed to watch for changes to instances of a Custom Resource Definition. We will delve into the foundational concepts, explore the essential tools and frameworks available to developers, walk through the architectural considerations, and provide a detailed blueprint for implementing a functional controller. Our exploration will cover the entire spectrum, from understanding the core components of the Kubernetes control plane and the role of the kube-apiserver, to the intricate details of client-go libraries, reconciliation loops, and ultimately, deploying and managing your custom controller in a production-like environment. By the end of this comprehensive article, you will possess a profound understanding of how to extend Kubernetes' capabilities with your own domain-specific logic, empowering you to automate complex operational tasks and manage your applications with unprecedented granularity and efficiency, all while interacting with the robust Kubernetes api framework.

Part 1: Understanding the Kubernetes Control Plane and Extension Mechanisms

To truly grasp the essence of building a Kubernetes controller, one must first comprehend the foundational architecture of Kubernetes itself, particularly its control plane. This collection of components is the brain of the cluster, responsible for maintaining the desired state and orchestrating all operations. Understanding how these pieces interact with the api server is crucial for developing any form of Kubernetes extension.

The Kubernetes Control Plane: The Cluster's Brain

The Kubernetes control plane is a set of core components that manage the cluster. They are responsible for global decisions about the cluster (e.g., scheduling, detecting and responding to cluster events), ensuring that the actual state of the cluster matches the desired state.

• kube-apiserver: This is the front-end for the Kubernetes control plane and the only control plane component that users and other components directly interact with via its RESTful api. It serves the Kubernetes api and processes REST requests, performing validation, and configuring data for api objects. The apiserver is the central communication hub, making it the most critical component. Every request to modify or query the cluster's state must go through the apiserver. When a controller watches for changes, it essentially establishes a connection with the apiserver to receive event notifications.
• etcd: A consistent and highly available key-value store used as Kubernetes' backing store for all cluster data. All configuration data, state data, and metadata are stored in etcd. The kube-apiserver is the only component that directly communicates with etcd. Controllers never interact with etcd directly; they always go through the apiserver, which then handles persistence in etcd. This separation is a crucial architectural decision, ensuring data consistency and simplifying client-side logic.
• kube-scheduler: This component watches for newly created Pods with no assigned node and selects a node for them to run on. The scheduler takes into account various factors like resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, and data locality. Its decision-making process is a sophisticated algorithm that optimizes resource utilization and ensures application availability.
• kube-controller-manager: This component runs controller processes. Logically, each controller is a separate process, but to reduce complexity, they are compiled into a single binary and run as a single process. Examples include:
  • Node Controller: Responsible for noticing and responding when nodes go down.
  • Replication Controller: Responsible for maintaining the correct number of Pods for every replication controller object.
  • Endpoints Controller: Populates the Endpoints object (which joins Services & Pods).
  • Service Account & Token Controllers: Create default Service Accounts and api access tokens for new Namespaces. These built-in controllers provide fundamental automation for Kubernetes' core resource types. Our custom controller will operate on the same principle but for our custom resource type, interacting with the same api mechanisms.
• cloud-controller-manager: This component embeds cloud-specific control logic. It allows you to link your cluster into your cloud provider's api and separates the cloud-specific controller logic from the generic controller logic. This component runs controllers that interact with the underlying cloud provider's infrastructure, such as node management, route management, and service load balancers.

The fundamental principle governing all these components is the "desired state" vs. "actual state." Users declare their desired state through api objects (YAML manifests), which are stored in etcd via the kube-apiserver. Controllers then continuously watch the actual state of the cluster (e.g., running Pods, allocated IPs) and take actions to bridge any gap with the desired state. This continuous reconciliation loop is the heart of Kubernetes' self-healing and automation capabilities.

Extending Kubernetes: Beyond Built-in Resources

While Kubernetes provides a robust set of built-in resource types, real-world applications often demand more specialized abstractions. Kubernetes offers powerful extension mechanisms to cater to these needs, allowing developers to integrate their custom logic and resources seamlessly.

• Admission Controllers: These are pieces of code that intercept requests to the Kubernetes api server prior to persistence of the object, but after the request is authenticated and authorized. They can either modify the object (Mutating Admission Controllers) or reject the request (Validating Admission Controllers). This mechanism is invaluable for enforcing policies, injecting sidecars, or performing data transformations before a resource is created or updated. For example, a Validating Admission Controller could ensure that all custom resources adhere to specific business rules, preventing malformed configurations from entering the system. Similarly, a Mutating Admission Controller could automatically inject default values or add labels to custom resources.
• Aggregated api Servers: This advanced extension method allows you to extend the Kubernetes api by serving your own api from another api server, which then aggregates with the main Kubernetes api server. It's suitable when you need deep integration, advanced validation, or entirely new api paths. While powerful, aggregated api servers are more complex to implement and maintain, often requiring a dedicated proxy service and certificate management. They are typically used for very specific use cases, such as providing a bespoke api for a complex domain or integrating with external systems that require their own api endpoints.
• Custom Resource Definitions (CRDs): CRDs are the most common and arguably the simplest way to extend the Kubernetes api. They allow users to define custom resource types directly within the Kubernetes api server without requiring modifications to the Kubernetes source code. Once a CRD is created, the Kubernetes api server starts serving the defined custom resource, allowing users to create, update, and delete instances of this resource using standard kubectl commands, just like built-in resources. This simplicity and native integration make CRDs the preferred method for most custom resource extensions.

Why CRDs are the Preferred Method for Most Custom Resources: CRDs strike an excellent balance between power and simplicity. They allow developers to define new resource types that are first-class citizens in the Kubernetes ecosystem. This means you can use kubectl to interact with them, they benefit from Kubernetes' RBAC (Role-Based Access Control), and their state is persisted in etcd through the kube-apiserver. Crucially, they enable the Operator pattern, where a custom controller watches instances of a CRD and acts upon them. This article focuses on CRDs because they represent the most practical and widely adopted approach for extending Kubernetes to manage custom applications and infrastructure components. The interaction of controllers with these CRD-defined resources heavily relies on the Kubernetes api, making it a central concept throughout our discussion.

Part 2: Deep Dive into Custom Resource Definitions (CRDs)

Custom Resource Definitions (CRDs) are the cornerstone of extending Kubernetes. They allow you to introduce your own object kinds into the Kubernetes api server, making them behave just like native Kubernetes objects. This section will explore what CRDs are, how they are structured, and how they relate to the custom resources (CRs) that your controller will be watching.

What is a CRD?

A Custom Resource Definition (CRD) is a definition file that tells the Kubernetes api server about a new, custom resource type. When you create a CRD, you're essentially registering a new api endpoint and schema with the apiserver. Once registered, the apiserver will begin to serve this new api, allowing clients (like kubectl or your custom controller) to create, read, update, and delete instances of your custom resource.

The primary purpose of a CRD is to: 1. Define a New api Endpoint: It tells the kube-apiserver to expose a new endpoint like /apis/<group>/<version>/<plural-kind>. 2. Define the Schema: It specifies the structure and validation rules for the custom resources that will be created under this endpoint. This schema is critical for ensuring data integrity and consistency. 3. Integrate with Kubernetes Ecosystem: Custom resources defined by CRDs seamlessly integrate with Kubernetes features such as kubectl commands, RBAC, and watches.

Let's dissect the YAML structure of a typical CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresources.example.com # Must be in the format <plural>.<group>
spec:
  group: example.com # The API group for the custom resource
  scope: Namespaced # Or 'Cluster' if the resource is not tied to a namespace
  names:
    plural: myresources
    singular: myresource
    kind: MyResource # The Kind of the custom resource instances
    shortNames:
      - mr
  versions:
    - name: v1 # Version of the API
      served: true
      storage: true # Only one version can be 'storage: true'
      schema:
        openAPIV3Schema:
          type: object
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec: # This is where your custom resource schema is defined
              type: object
              properties:
                image:
                  type: string
                  description: The container image to use.
                replicas:
                  type: integer
                  minimum: 1
                  description: Number of desired replicas.
              required:
                - image
                - replicas
            status: # Optional: for controller to report current status
              type: object
              properties:
                availableReplicas:
                  type: integer
                  description: Number of currently available replicas.
      subresources:
        status: {} # Enables /status subresource for atomic status updates

Key fields in a CRD spec:

• group: Defines the api group for your custom resource (e.g., example.com). This helps organize and distinguish your resources from built-in ones.
• scope: Can be Namespaced (default) or Cluster. Namespaced resources live within a Kubernetes namespace, while Cluster-scoped resources exist across the entire cluster.
• names: Specifies how the resource will be referred to:
  • plural: Used in kubectl commands (e.g., kubectl get myresources).
  • singular: The singular form.
  • kind: The Kind field used in YAML manifests for the custom resource (e.g., kind: MyResource).
  • shortNames: Optional, convenient aliases for kubectl.
• versions: A list of api versions supported for this CRD. Each version defines its schema and whether it's served (available via the api) and storage (the version used to persist the resource in etcd). It's common to start with v1.
  • schema.openAPIV3Schema: This is where the actual schema for your custom resource's spec and status fields is defined using OpenAPI v3 schema syntax. It allows for detailed validation, specifying data types, required fields, patterns, and more. This ensures that instances of your custom resource conform to your predefined structure.
• subresources.status: If present, it creates a /status subresource. This allows controllers to update the status field of a custom resource separately from its spec, preventing race conditions and ensuring efficient updates.

Examples of Common CRDs in the Ecosystem: Many popular Kubernetes projects leverage CRDs to define their operational model. For instance: * cert-manager: Uses Certificate and Issuer CRDs to manage TLS certificates. * Prometheus Operator: Defines Prometheus, ServiceMonitor, and Alertmanager CRDs to manage Prometheus monitoring stacks. * Knative: Utilizes Service, Configuration, and Revision CRDs for serverless deployments. These examples highlight the versatility and power of CRDs in extending Kubernetes to manage diverse applications and infrastructure components.

Custom Resources (CRs): Instances of Your CRD

A Custom Resource (CR) is an actual instance of a Custom Resource Definition. Once a CRD is applied to a cluster, you can create CRs just like you would create a Pod or Deployment. These CRs are api objects that conform to the schema defined in their respective CRD.

Let's consider an example Custom Resource (MyResource):

apiVersion: example.com/v1 # Matches the group and version from the CRD
kind: MyResource           # Matches the kind from the CRD
metadata:
  name: my-app-instance
  namespace: default
spec:
  image: "my-registry/my-app:1.2.3"
  replicas: 3

• apiVersion: Must match the group and version defined in the CRD.
• kind: Must match the kind defined in the CRD.
• metadata: Standard Kubernetes object metadata (name, namespace, labels, annotations).
• spec: This is the core of your custom resource. It contains the desired configuration that your controller will read and act upon. The structure of this spec must adhere to the openAPIV3Schema defined in the CRD. In our example, it includes image and replicas.
• status (optional, managed by controller): While not shown in the creation manifest above, once created, your controller would typically update the status field to reflect the current state of the resources it manages. For instance, after creating a Deployment for my-app-instance, the controller might update status.availableReplicas to reflect the number of running Pods.

How Users Interact with CRs: Users interact with custom resources using kubectl just like any built-in resource: * kubectl apply -f myresource.yaml: Creates or updates a MyResource instance. * kubectl get myresources: Lists all MyResource instances in the current namespace (or --all-namespaces). * kubectl get myresource my-app-instance -o yaml: Retrieves the details of a specific MyResource. * kubectl delete myresource my-app-instance: Deletes a MyResource.

This seamless integration into the kubectl workflow is a key benefit of using CRDs.

Lifecycle of a CRD/CR

The lifecycle of a CRD and its instances (CRs) is central to how a controller operates.

1. CRD Creation: An administrator or CI/CD pipeline applies the CRD definition to the cluster. The kube-apiserver then registers this new api type. kubectl apply -f crd.yaml
2. CR Creation/Update: Users (or automated systems) create instances of the custom resource. These requests go through the kube-apiserver, which validates them against the CRD's schema and persists them in etcd. kubectl apply -f myresource-instance.yaml
3. Controller Watching: Your custom controller, which is configured to watch MyResource objects, receives an "add" event from the kube-apiserver via its watch mechanism.
4. Reconciliation: The controller processes the event, reads the MyResource object, and performs the necessary actions to achieve the desired state (e.g., creates a Deployment, a Service, or other Kubernetes resources). It might also update the status field of the MyResource to reflect the outcome of its actions.
5. CR Update Event: If a user modifies my-app-instance (e.g., changes spec.replicas from 3 to 5), the kube-apiserver updates the resource in etcd, and the controller receives an "update" event.
6. Reconciliation (Update): The controller reads the updated MyResource, detects the change in spec.replicas, and modifies the associated Deployment to scale up to 5 replicas. It then updates the MyResource's status.
7. CR Deletion: If a user deletes my-app-instance, the kube-apiserver processes the deletion, and the controller receives a "delete" event.
8. Reconciliation (Deletion): The controller, upon receiving the delete event, performs cleanup actions (e.g., deletes the Deployment and Service it created for my-app-instance). If finalizers are used, the controller might prevent immediate deletion until cleanup is complete.

This continuous cycle of watching, reacting, and reconciling is the essence of a Kubernetes controller. The ability to watch for changes to these custom resources, defined by the CRD and managed through the Kubernetes api, is what empowers operators to automate complex application lifecycle management.

Part 3: The Anatomy of a Kubernetes Controller

The core of Kubernetes' automation capabilities lies in its controllers. A controller is a control loop that continuously watches for changes in the cluster's state, compares the observed actual state with the desired state (as declared in Kubernetes api objects), and takes actions to reconcile any discrepancies. For a custom resource defined by a CRD, a custom controller performs precisely this function, bringing your custom objects to life.

What is a Controller? The Reconciliation Loop

At its heart, a controller is a simple yet powerful concept: a never-ending loop that watches a specific set of resources. When an event pertaining to these resources occurs (creation, update, or deletion), the controller is notified. It then fetches the current state of the affected resource and any related resources, compares this actual state to the desired state specified in the resource's spec (and potentially other configurations), and performs operations to converge the actual state towards the desired state. This process is known as the reconciliation loop.

The reconciliation loop embodies the declarative nature of Kubernetes: * Desired State: Defined by the user in the spec of a Kubernetes api object (e.g., a Deployment's spec.replicas, or our MyResource's spec.image). * Actual State: The current condition of the cluster, observed by the controller (e.g., number of running Pods, current image of a Deployment). * Reconciliation: The process of taking actions (creating, updating, deleting resources) to make the actual state match the desired state.

If the controller finds a discrepancy, it takes corrective action. If the actual state already matches the desired state, the controller does nothing, gracefully exiting the reconciliation for that particular object until another event triggers it. This idempotency is a key characteristic of well-designed controllers.

Core Components of a Controller

While the concept of a reconciliation loop is straightforward, building a robust controller involves several sophisticated components that interact efficiently with the Kubernetes api. Modern controller development frameworks like controller-runtime (which Kubebuilder and Operator SDK are built upon) abstract away much of this complexity, but understanding the underlying components is crucial.

• Informer: The Kubernetes api server provides a Watch api call that allows clients to receive notifications about changes to resources. However, directly using Watch for every resource can be inefficient, especially for large clusters, as it would overwhelm the api server and require clients to maintain their own state. This is where the Informer pattern comes in. An Informer serves as a local, read-only cache of Kubernetes api objects. It reduces the load on the kube-apiserver by performing a ListAndWatch operation:
  1. Initial List: On startup, the Informer performs a full List operation to populate its cache with all existing resources of a certain type.
  2. Continuous Watch: After the initial List, it establishes a persistent Watch connection to the kube-apiserver. For every subsequent change (Add, Update, Delete), the apiserver sends an event, which the Informer uses to update its local cache. This pattern ensures that controllers always operate on a fresh, consistent view of the cluster state without constantly querying the api server directly. Event Handlers (AddFunc, UpdateFunc, DeleteFunc): Informers expose these callbacks, allowing your controller to register functions that will be executed whenever a resource is added, updated, or deleted in the cache. These functions typically enqueue the affected resource into a workqueue for processing.
• Indexer: The Informer's cache is usually backed by an Indexer. An Indexer provides methods for efficient lookups of objects within the cache, beyond just by name and namespace. It allows you to define custom indices (e.g., by a label, by a field selector) to quickly retrieve related objects. For example, if your MyResource controller needs to find all Deployments with a specific label indicating they are managed by a MyResource, an Indexer could provide a highly efficient way to do so from the local cache.
• Workqueue: The Informer's event handlers are typically asynchronous. When an event occurs, you don't want to block the Informer by directly executing the reconciliation logic. Instead, the event handler places the key of the affected object (e.g., <namespace>/<name>) into a queue. This queue is known as the Workqueue (or RateLimitingQueue in client-go). The Workqueue serves several critical purposes:
  1. Decoupling: It decouples event handling from the heavy reconciliation logic, allowing events to be processed quickly while reconciliation can take its time.
  2. Rate Limiting: It can be configured with exponential backoff to re-queue items that failed reconciliation, preventing a controller from constantly retrying an operation that is bound to fail immediately, thus reducing api server load.
  3. Deduplication: If multiple events for the same object occur in quick succession, the Workqueue often deduplicates them, ensuring the reconciliation function is only called once for the latest state.
  4. Concurrency: Multiple worker goroutines can pull items from the Workqueue concurrently, processing events in parallel.
• Reconciler: The Reconciler is the heart of your custom logic. It's the function that takes an object's key from the Workqueue, fetches the latest state of that object from the Informer's cache, and then applies the core business logic to bring the cluster's actual state into alignment with the desired state specified in the object. This function should be idempotent, meaning running it multiple times with the same input should produce the same result without adverse side effects. The Reconciler's tasks typically involve:
  • Fetching the target Custom Resource.
  • Fetching any secondary resources managed by the Custom Resource (e.g., a Deployment created by MyResource).
  • Comparing the spec of the Custom Resource with the actual state of secondary resources.
  • Creating, updating, or deleting secondary resources as needed.
  • Updating the status field of the Custom Resource to reflect the current operational state.
  • Handling potential errors and re-queuing the item if a retry is necessary.

Client Libraries: Tools for Building Controllers

Developing Kubernetes controllers from scratch, directly interacting with the RESTful api and implementing Informers, Indexers, and Workqueues, is a complex task. Fortunately, several powerful client libraries and frameworks abstract away much of this boilerplate, making controller development more approachable.

Feature / Tool	client-go	controller-runtime	Kubebuilder / Operator SDK
Abstraction Level	Low-level, direct api interaction	Mid-level, framework for controllers	High-level, code generation and opinionated structure
Core Components	Informers, Indexers, Workqueues, Typed Clients	Manager, Reconcilers, Webhooks	Manager, Reconcilers, Webhooks, CRD scaffolding
Primary Use Case	Building highly customized, performance-sensitive controllers; integrating with existing Go applications	Building robust controllers with less boilerplate	Rapidly developing Kubernetes Operators and CRDs
Learning Curve	Steep, requires deep understanding of Kubernetes api	Moderate, requires understanding of reconciliation	Moderate, requires understanding of the tools
Code Generation	No	Limited (CRD generation from Go types)	Extensive (CRDs, controller scaffold, Dockerfiles)
Recommended For	Experts, specific edge cases, integrating api clients	Most custom controllers, complex logic	New Operators, standard patterns, faster development
Opinionated Structure	Low	Moderate	High

• client-go: This is the official Go client library for the Kubernetes api. It provides the fundamental building blocks for interacting with a Kubernetes cluster programmatically. client-go offers:
  • Typed Clients: Go structs and methods for interacting with Kubernetes resources (e.g., corev1.Pods(), appsv1.Deployments()).
  • Informers and Listers: Implementations of the Informer pattern for caching resources and efficient lookups.
  • Workqueues: Rate-limiting workqueues for robust event processing. While client-go provides all the necessary primitives, writing a controller directly with it involves a significant amount of boilerplate code for managing caches, watches, and error handling. It offers maximum flexibility but also maximum responsibility.
• controller-runtime: This is a library built on top of client-go that significantly simplifies controller development. It provides a higher-level framework that abstracts away much of the boilerplate associated with Informers, Workqueues, and client setup. Key features include:
  • Manager: A single entry point that sets up shared caches, api clients, and starts all controllers.
  • Reconciler Interface: A simple interface (Reconcile(context.Context, ctrl.Request) (ctrl.Result, error)) that you implement with your core logic.
  • Watches: Easier declaration of what resources your controller should watch (your CR and any secondary resources). controller-runtime encourages best practices for controller development, making it the de-facto standard for building most custom controllers in Go.
• Operator SDK / Kubebuilder: These are command-line tools built on top of controller-runtime that provide an even higher level of abstraction and automation. They are designed to streamline the entire Operator development lifecycle, from project scaffolding to CRD generation, controller implementation, and deployment.
  • Scaffolding: Automatically generate project structure, Dockerfiles, Makefiles, and basic controller code.
  • CRD Generation: Generate CRD YAML from Go structs (defining your spec and status).
  • Webhooks: Easily add mutating and validating admission webhooks.
  • Testing Utilities: Provide frameworks for unit and integration testing. For most new controller projects, especially those following the Operator pattern, Kubebuilder (or Operator SDK, which shares much of the same underlying technology with Kubebuilder) is the recommended starting point due to its productivity enhancements and adherence to best practices.

Choosing the Right Tool: For this guide, we will leverage Kubebuilder/Operator SDK with controller-runtime. This combination offers the best balance of flexibility, robustness, and developer productivity for building a controller that watches CRD changes. It automates much of the setup and boilerplate, allowing us to focus on the crucial reconciliation logic, which directly interacts with the Kubernetes api to manage our custom resources.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Part 4: Setting up the Development Environment and Project Structure

Before we dive into writing the controller logic, we need to set up our development environment and scaffold a new Kubernetes controller project. We'll be using Kubebuilder, which leverages controller-runtime to simplify the entire process of building an Operator. This part will guide you through the prerequisites and the initial project setup.

Prerequisites

To follow along and build the controller, you'll need the following tools installed on your development machine:

• Go Language (1.19+): Kubernetes controllers are primarily written in Go. You can download it from golang.org/dl.
• Docker: Required for building the controller's container image. Install Docker Desktop or Docker Engine relevant to your OS from docker.com.
• kubectl: The Kubernetes command-line tool, essential for interacting with your cluster. Install it by following the official Kubernetes documentation.
• kind (Kubernetes in Docker) or minikube: A local Kubernetes cluster for development and testing. kind is generally preferred for controller development as it's lightweight and integrates well with testing frameworks. Install kind from its GitHub repository. If you prefer minikube, ensure it's installed and running.
  • To start a kind cluster: kind create cluster
• Kubebuilder: The command-line tool we'll use to scaffold our project. Install it by following the official Kubebuilder documentation: bash # Install controller-gen go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest # Install Kubebuilder os=$(go env GOOS) arch=$(go env GOARCH) curl -L https://go.kubebuilder.io/dl/latest/${os}/${arch} | tar -xz -C /tmp/ sudo mv /tmp/kubebuilder_*/bin /usr/local/bin/kubebuilder Verify installation: kubebuilder version

Using Kubebuilder (Recommended Approach)

Kubebuilder significantly accelerates the development of Kubernetes Operators by providing scaffolding, code generation, and adherence to best practices. It's the standard toolchain for building controllers with controller-runtime.

Why These Tools are Preferred for Rapid Development: Kubebuilder generates a complete project structure, including Go modules, Dockerfiles, Makefiles, and boilerplate code for the controller. This means you don't have to worry about setting up client-go informers, workqueues, or api clients manually. Instead, you can focus directly on the reconciliation logic.

kubebuilder init: First, create a new directory for your project and initialize a Go module within it. Then, initialize the Kubebuilder project.

mkdir my-controller
cd my-controller
go mod init example.com/my-controller # Replace with your module path
kubebuilder init --domain example.com --repo example.com/my-controller

• --domain example.com: This defines the domain for your api group (e.g., myresources.example.com).
• --repo example.com/my-controller: This specifies the Go module path for your project.

After kubebuilder init completes, you'll see a basic project structure: * main.go: The entry point for your controller manager. * go.mod, go.sum: Go module files. * Dockerfile: For building the controller's container image. * Makefile: Contains common targets for building, deploying, and testing. * config/: Contains YAML manifests for CRDs, RBAC, controller deployment, and webhooks.

kubebuilder create api --group <group> --version <version> --kind <kind>: Now, let's define our Custom Resource Definition and generate the corresponding api types and controller boilerplate. We'll create a MyResource CRD that manages a simple application.

kubebuilder create api --group app --version v1 --kind MyResource --namespaced=true

• --group app: Sets the api group to app.example.com (using the domain from kubebuilder init).
• --version v1: Defines the api version as v1.
• --kind MyResource: Sets the Kind of our custom resource to MyResource.
• --namespaced=true: Indicates that MyResource instances will be namespaced (not cluster-scoped).

This command generates several important files: * api/v1/myresource_types.go: Defines the Go structs for your MyResource's spec and status. This is where you'll define the fields of your custom resource. * controllers/myresource_controller.go: Contains the skeleton of your MyResource controller, including the Reconcile method and SetupWithManager function. * config/crd/bases/app.example.com_myresources.yaml: The generated CRD YAML definition.

Defining the Custom Resource (CRD api group/version/kind)

The most crucial step in defining your custom resource is to modify api/v1/myresource_types.go. This file contains the Go structs that represent the spec and status of your MyResource object. Kubebuilder uses tags (annotations) on these Go structs to generate the OpenAPI v3 schema within your CRD YAML.

Open api/v1/myresource_types.go and modify the MyResourceSpec and MyResourceStatus structs.

package v1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags the same way you have them here.

// MyResourceSpec defines the desired state of MyResource
type MyResourceSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make generate" to regenerate code after modifying this file

    // +kubebuilder:validation:Minimum=1
    // Replicas is the number of desired pods.
    Replicas int32 `json:"replicas,omitempty"`

    // Image is the container image to deploy.
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:Pattern=`^(.+)\/(.+):(.+)$`
    Image string `json:"image"`

    // Port is the container port to expose.
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=65535
    // +optional
    Port int32 `json:"port,omitempty"`
}

// MyResourceStatus defines the observed state of MyResource
type MyResourceStatus struct {
    // INSERT ADDITIONAL STATUS FIELDS - observed state of cluster
    // Important: Run "make generate" to regenerate code after modifying this file

    // +optional
    // AvailableReplicas is the number of currently available pods.
    AvailableReplicas int32 `json:"availableReplicas"`

    // +optional
    // Conditions represent the latest available observations of an object's state.
    Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Image",type="string",JSONPath=".spec.image",description="The container image"
// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas",description="Number of desired replicas"
// +kubebuilder:printcolumn:name="Available",type="integer",JSONPath=".status.availableReplicas",description="Number of available pods"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"

// MyResource is the Schema for the myresources API
type MyResource struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyResourceSpec   `json:"spec,omitempty"`
    Status MyResourceStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// MyResourceList contains a list of MyResource
type MyResourceList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []MyResource `json:"items"`
}

func init() {
    SchemeBuilder.Register(&MyResource{}, &MyResourceList{})
}

Annotations for CRD Generation: Notice the +kubebuilder: comments. These are special annotations that Kubebuilder's controller-gen tool uses to generate the CRD YAML and other boilerplate code. * +kubebuilder:object:root=true: Marks the MyResource struct as a root Kubernetes api object. * +kubebuilder:subresource:status: Enables the /status subresource for MyResource objects, allowing atomic updates to the status. * +kubebuilder:validation:Minimum=1, +kubebuilder:validation:Pattern: These generate validation rules in the CRD's OpenAPI schema, ensuring that replicas is at least 1 and image follows a specific pattern. * +kubebuilder:printcolumn: Defines custom columns for kubectl get myresources output, improving observability.

After modifying myresource_types.go, you must run make generate and make manifests to regenerate the api code and update the CRD YAML definition (config/crd/bases/app.example.com_myresources.yaml).

make generate
make manifests

Now, config/crd/bases/app.example.com_myresources.yaml will reflect your schema changes. For example, it will contain the openAPIV3Schema definitions for spec.replicas, spec.image, and status.availableReplicas. This CRD manifest is what you will apply to your Kubernetes cluster to register your custom resource type.

This initial setup provides a solid foundation. We've defined our custom resource's structure and generated the necessary api boilerplate. In the next part, we will implement the core controller logic that watches instances of this MyResource CRD and performs actions to reconcile their desired state, making full use of the Kubernetes api and the controller-runtime framework.

Part 5: Implementing the Controller Logic to Watch CRD Changes

With the project scaffolded and the Custom Resource Definition (CRD) schema defined, it's time to breathe life into our controller. This section focuses on implementing the core reconciliation logic within controllers/myresource_controller.go, which will watch for changes to our MyResource custom resources and manage associated standard Kubernetes objects like Deployments and Services.

Understanding the Reconcile Function

The Reconcile function is the heart of every controller-runtime based controller. It's an idempotent function that the controller manager calls whenever there's a change to a resource that the controller is watching, or when a periodic re-queue is triggered. Its goal is to ensure the cluster's actual state matches the desired state specified in the custom resource.

The signature of the Reconcile function is: func (r *MyResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

• ctx context.Context: A standard Go context, used for cancellation signals and passing request-scoped values.
• req ctrl.Request: Contains the NamespacedName (namespace and name) of the object that triggered the reconciliation.
• ctrl.Result: Indicates whether the reconciliation was successful, if the item should be re-queued, and for how long.
• error: If an error is returned, the item will be re-queued with exponential backoff.

Fetching the Custom Resource: The first step in any reconciliation loop is to fetch the custom resource that triggered the event.

package controllers

import (
    "context"
    "fmt"
    "time"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/intstr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    appv1 "example.com/my-controller/api/v1" // Our custom API group
)

// MyResourceReconciler reconciles a MyResource object
type MyResourceReconciler struct {
    client.Client
    Scheme *runtime.Scheme
    Log    logr.Logger // Inject logger
}

func (r *MyResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // Fetch the MyResource instance
    myresource := &appv1.MyResource{}
    err := r.Get(ctx, req.NamespacedName, myresource)
    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Owned objects are automatically garbage collected. For additional cleanup logic, use finalizers.
            // Return and don't requeue
            log.Log.Info("MyResource resource not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        log.Log.Error(err, "Failed to get MyResource")
        return ctrl.Result{}, err
    }

    // Logic starts here...
    // ...
}

• r.Get(ctx, req.NamespacedName, myresource): This uses the client from controller-runtime to fetch the MyResource object identified by req.NamespacedName. This client leverages the Informer's cache, so it's efficient.
• Handling NotFound errors: If errors.IsNotFound(err) is true, it means the MyResource object was deleted. In this case, we simply log it and return ctrl.Result{}, nil without re-queuing, as there's nothing left to reconcile for this object. If the controller created any child resources, these would typically be garbage-collected automatically if the owner reference is properly set, or handled by a finalizer.
• Other errors: Any other error typically indicates a transient issue (e.g., network problem talking to apiserver, etcd issue) or a configuration error. In such cases, returning the error causes controller-runtime to re-queue the request with exponential backoff, giving the system time to recover.

Watching for CRD (and Related) Changes

The SetupWithManager method, defined in controllers/myresource_controller.go, is where you configure what resources your controller should watch. This is crucial for the controller to receive notifications about api changes.

func (r *MyResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appv1.MyResource{}). // Watch for changes to MyResource objects
        Owns(&appsv1.Deployment{}). // Watch for changes to Deployments owned by MyResource
        Owns(&corev1.Service{}).    // Watch for changes to Services owned by MyResource
        Complete(r)
}

• For(&appv1.MyResource{}): This is the primary watch. It tells the controller to watch instances of our MyResource CRD. Any creation, update, or deletion of a MyResource object will trigger a reconciliation request for that specific MyResource instance. This is the direct implementation of "watching for changes to CRD" – specifically, watching for changes to the instances of our MyResource type.
• Owns(&appsv1.Deployment{}), Owns(&corev1.Service{}): This is a crucial feature for operators. It tells the controller to also watch for changes to Deployment and Service objects. If a Deployment or Service that is owned by a MyResource object changes (e.g., it's manually deleted, or its status changes), controller-runtime will enqueue a reconciliation request for the owner (MyResource) of that object. This ensures that our controller can react to external modifications or failures of the resources it manages, thereby enforcing the desired state. The owner reference is established when the controller creates these secondary resources.

Implementing the Reconciliation Loop for MyResource

Now, let's fill in the Reconcile function with the logic to manage a Deployment and a Service based on our MyResource.Spec.

func (r *MyResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    myresource := &appv1.MyResource{}
    if err := r.Get(ctx, req.NamespacedName, myresource); err != nil {
        if errors.IsNotFound(err) {
            log.Info("MyResource resource not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        log.Error(err, "Failed to get MyResource")
        return ctrl.Result{}, err
    }

    // Define a new Deployment object for the MyResource
    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myresource.Name + "-deployment",
            Namespace: myresource.Namespace,
        },
    }

    // Set MyResource instance as the owner and controller of the Deployment
    // This will enable garbage collection and allow the controller to react to changes in the Deployment
    if err := controllerutil.SetControllerReference(myresource, deployment, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for Deployment")
        return ctrl.Result{}, err
    }

    // Check if the Deployment already exists, if not, create a new one
    foundDeployment := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: deployment.Name, Namespace: deployment.Namespace}, foundDeployment)
    if err != nil && errors.IsNotFound(err) {
        log.Info("Creating a new Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
        deployment.Spec = appsv1.DeploymentSpec{
            Replicas: &myresource.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{
                    "app":        myresource.Name,
                    "controller": "myresource",
                },
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{
                        "app":        myresource.Name,
                        "controller": "myresource",
                    },
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "app",
                        Image: myresource.Spec.Image,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: myresource.Spec.Port,
                        }},
                    }},
                },
            },
        }
        err = r.Create(ctx, deployment)
        if err != nil {
            log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
            return ctrl.Result{}, err
        }
        // Deployment created successfully - return and requeue to check Service
        return ctrl.Result{Requeue: true}, nil // Requeue to ensure Service is created next, or status updated
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        return ctrl.Result{}, err
    }

    // Update the Deployment if needed
    desiredReplicas := myresource.Spec.Replicas
    desiredImage := myresource.Spec.Image
    desiredPort := myresource.Spec.Port

    if *foundDeployment.Spec.Replicas != desiredReplicas ||
        foundDeployment.Spec.Template.Spec.Containers[0].Image != desiredImage ||
        foundDeployment.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort != desiredPort {

        log.Info("Updating existing Deployment", "Deployment.Namespace", foundDeployment.Namespace, "Deployment.Name", foundDeployment.Name)
        foundDeployment.Spec.Replicas = &desiredReplicas
        foundDeployment.Spec.Template.Spec.Containers[0].Image = desiredImage
        foundDeployment.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort = desiredPort

        err = r.Update(ctx, foundDeployment)
        if err != nil {
            log.Error(err, "Failed to update Deployment", "Deployment.Namespace", foundDeployment.Namespace, "Deployment.Name", foundDeployment.Name)
            return ctrl.Result{}, err
        }
        // Deployment updated successfully - return and requeue to ensure Service is up-to-date and status is updated
        return ctrl.Result{Requeue: true}, nil
    }

    // Define a new Service object for the MyResource
    service := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myresource.Name + "-service",
            Namespace: myresource.Namespace,
        },
    }
    if err := controllerutil.SetControllerReference(myresource, service, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for Service")
        return ctrl.Result{}, err
    }

    // Check if the Service already exists, if not, create a new one
    foundService := &corev1.Service{}
    err = r.Get(ctx, types.NamespacedName{Name: service.Name, Namespace: service.Namespace}, foundService)
    if err != nil && errors.IsNotFound(err) {
        log.Info("Creating a new Service", "Service.Namespace", service.Namespace, "Service.Name", service.Name)
        service.Spec = corev1.ServiceSpec{
            Selector: map[string]string{
                "app":        myresource.Name,
                "controller": "myresource",
            },
            Ports: []corev1.ServicePort{{
                Protocol:   corev1.ProtocolTCP,
                Port:       80, // External port for the service
                TargetPort: intstr.FromInt(int(myresource.Spec.Port)), // Target container port
            }},
            Type: corev1.ServiceTypeClusterIP, // Or LoadBalancer, NodePort
        }
        err = r.Create(ctx, service)
        if err != nil {
            log.Error(err, "Failed to create new Service", "Service.Namespace", service.Namespace, "Service.Name", service.Name)
            return ctrl.Result{}, err
        }
        // Service created successfully - return and requeue to update status
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        log.Error(err, "Failed to get Service")
        return ctrl.Result{}, err
    }

    // Update the Service if needed (e.g., target port change)
    if foundService.Spec.Ports[0].TargetPort.IntVal != myresource.Spec.Port {
        log.Info("Updating existing Service", "Service.Namespace", foundService.Namespace, "Service.Name", foundService.Name)
        foundService.Spec.Ports[0].TargetPort = intstr.FromInt(int(myresource.Spec.Port))
        err = r.Update(ctx, foundService)
        if err != nil {
            log.Error(err, "Failed to update Service", "Service.Namespace", foundService.Namespace, "Service.Name", foundService.Name)
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

    // Update the MyResource status with the observed state
    // Get the latest Deployment status
    if err = r.Get(ctx, types.NamespacedName{Name: deployment.Name, Namespace: deployment.Namespace}, foundDeployment); err != nil {
        log.Error(err, "Failed to get latest Deployment for status update")
        return ctrl.Result{}, err
    }

    // Check if status needs to be updated
    if myresource.Status.AvailableReplicas != foundDeployment.Status.AvailableReplicas {
        myresource.Status.AvailableReplicas = foundDeployment.Status.AvailableReplicas
        log.Info("Updating MyResource status", "MyResource.Namespace", myresource.Namespace, "MyResource.Name", myresource.Name, "AvailableReplicas", myresource.Status.AvailableReplicas)
        err = r.Status().Update(ctx, myresource)
        if err != nil {
            log.Error(err, "Failed to update MyResource status")
            return ctrl.Result{}, err
        }
    }

    // No changes were made or nothing left to do. Reconcile in 5 minutes by default or if nothing new happens.
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Let's break down the reconciliation logic:

1. Fetch MyResource: Already covered, ensures we have the latest desired state.
2. Define and Manage Deployment:
   • A Deployment object is constructed based on the MyResource's spec. The name is derived from the MyResource name to ensure uniqueness and traceability.
   • controllerutil.SetControllerReference: This crucial function establishes an owner reference from the Deployment to the MyResource. This means:
     • When MyResource is deleted, the Deployment will be garbage-collected automatically.
     • If the Deployment changes or is deleted externally, the MyResource controller will be re-queued, thanks to Owns(&appsv1.Deployment{}) in SetupWithManager.
   • Create or Update: The controller attempts to Get the Deployment.
     • If NotFound, it creates a new Deployment with the specified replicas, image, and port from myresource.Spec.
     • If found, it compares the current Deployment's spec with the desired state from myresource.Spec. If there are discrepancies (e.g., replicas or image changed), it updates the existing Deployment.
     • After creation or update, return ctrl.Result{Requeue: true} ensures the reconciliation is immediately re-triggered. This is useful for sequential resource creation (e.g., create Deployment, then ensure Service exists).
3. Define and Manage Service:
   • Similar logic applies to the Service object. It's defined to expose the Deployment's Pods.
   • An owner reference is set to the MyResource.
   • The controller checks if the Service exists. If not, it creates one, mapping an external port (e.g., 80) to the container's myresource.Spec.Port.
   • If the Service exists, it checks for updates (e.g., if myresource.Spec.Port changed).
4. Update MyResource Status:
   • After ensuring the Deployment and Service are in the desired state, the controller fetches the latest Deployment to get its actual status (e.g., foundDeployment.Status.AvailableReplicas).
   • It then updates the MyResource.Status.AvailableReplicas field. Crucially, r.Status().Update(ctx, myresource) is used for status updates. This uses the /status subresource, preventing conflicts with spec updates. Updating the status provides valuable feedback to users about the operational state of their custom resource.
5. Requeue:
   • return ctrl.Result{RequeueAfter: 5 * time.Minute}: If no changes were made and no errors occurred, the controller will re-reconcile this MyResource after 5 minutes. This acts as a periodic sync to catch any external changes not caught by Owns watches or to re-verify state.

This reconciliation loop demonstrates how a Kubernetes controller observes custom resources defined by a CRD via the api, then creates, updates, and deletes standard Kubernetes resources to fulfill the custom resource's desired state. The use of controller-runtime and client.Client simplifies these interactions with the Kubernetes api, making the development process more efficient and less error-prone.

Part 6: Deployment, Testing, and Best Practices

Once the controller logic is implemented, the next crucial steps involve deploying it to a Kubernetes cluster, rigorously testing its behavior, and adhering to best practices to ensure robustness, security, and performance. This section will guide you through these essential phases.

Building and Deploying the CRD and Controller

Kubebuilder streamlines the entire deployment process with a set of convenient Makefile targets.

1. Install the CRD: Before deploying your controller, you must install the Custom Resource Definition into your Kubernetes cluster. This registers your MyResource type with the kube-apiserver, allowing it to understand and store instances of your custom resource. bash make install This command applies the CRD YAML definition located in config/crd/bases/app.example.com_myresources.yaml to your cluster. You can verify its installation: bash kubectl get crd myresources.app.example.com You should see myresources.app.example.com listed.
2. Build and Push the Controller Image: Your controller runs as a standard Kubernetes Pod, meaning it needs to be packaged into a Docker image. bash make docker-build docker-push IMG=example.com/my-controller:v0.0.1
   • make docker-build: This command builds the Docker image for your controller. It uses the Dockerfile generated by Kubebuilder.
   • make docker-push: This command pushes the built image to a container registry (e.g., Docker Hub, GCR, Quay.io). You'll need to be logged into your registry. Replace example.com/my-controller:v0.0.1 with your actual image name and tag. If you are using a local kind cluster, you might push the image directly to kind's internal registry or load it: kind load docker-image example.com/my-controller:v0.0.1.
3. Deploy the Controller to Kubernetes: After the image is pushed, you can deploy your controller to the cluster. Kubebuilder generates deployment manifests (Deployment, ServiceAccount, ClusterRole, ClusterRoleBinding) in the config/ directory. bash make deploy IMG=example.com/my-controller:v0.0.1 This command applies the deployment manifests, creating a Deployment for your controller (which will run your container image), an RBAC setup to grant it necessary permissions to interact with the Kubernetes api, and other required resources. You can check if your controller pod is running: bash kubectl get pods -n my-controller-system # (or your configured namespace) You should see a pod named similar to my-controller-controller-manager-... in a Running state.

Testing

Thorough testing is paramount for any robust software, especially for controllers that manage critical infrastructure.

1. Creating an Instance of MyResource: Now that the CRD and controller are deployed, let's create an actual instance of our custom resource. Create a file named myresource-test.yaml: yaml apiVersion: app.example.com/v1 kind: MyResource metadata: name: my-sample-app namespace: default spec: image: "nginx:latest" replicas: 2 port: 80 Apply this manifest: bash kubectl apply -f myresource-test.yaml
2. Verifying Controller Actions: Observe your controller in action:
   • Check MyResource status: bash kubectl get myresource my-sample-app -o yaml # Look for status.availableReplicas kubectl get myresources # Check custom columns: Image, Replicas, Available
   • Verify the created Deployment and Service: bash kubectl get deployment my-sample-app-deployment kubectl get service my-sample-app-service kubectl get pods -l app=my-sample-app
   • Check controller logs for reconciliation events: bash kubectl logs -f <my-controller-pod-name> -n my-controller-system
3. Modifying MyResource and Observing Updates: Edit myresource-test.yaml to change the replicas or image: yaml # ... spec: image: "httpd:latest" # Change image replicas: 3 # Change replicas port: 8080 # Change port Apply the change: kubectl apply -f myresource-test.yaml. Observe the controller updating the Deployment and Service (e.g., kubectl get deployment my-sample-app-deployment, kubectl get service my-sample-app-service). The Deployment will perform a rolling update, and the Service's target port should update.
4. Deleting MyResource and Observing Cleanup: Delete the custom resource: bash kubectl delete -f myresource-test.yaml Verify that the Deployment and Service created by the controller are also deleted due to the owner reference: bash kubectl get deployment my-sample-app-deployment kubectl get service my-sample-app-service They should report "not found."
5. Unit, Integration, and End-to-End Tests:
   • Unit Tests: Test individual functions and reconciliation logic components in isolation. Kubebuilder scaffolds a controllers/myresource_controller_test.go file with basic examples.
   • Integration Tests: Test the controller against a real (but isolated) apiserver instance (e.g., using envtest provided by controller-runtime). This tests the controller's interaction with the Kubernetes api without deploying a full cluster.
   • End-to-End Tests: Deploy the controller and custom resources to a full cluster (like kind or minikube) and verify the desired end-state.

Error Handling and Robustness

A production-ready controller must be robust and handle errors gracefully.

• Retries and Exponential Backoff: controller-runtime (via client-go's workqueue) automatically handles re-queuing reconciliation requests that return an error. It uses exponential backoff to avoid hammering the apiserver during transient failures. Design your Reconcile function to return an error for transient issues.
• Event Recording: Use k8s.io/client-go/tools/events to record Kubernetes events on your custom resources. These events (e.g., "DeploymentCreated", "FailedToCreateService") provide valuable user-facing feedback that can be viewed with kubectl describe myresource <name>.
• Metrics and Logging: Expose Prometheus metrics from your controller (Kubebuilder sets this up by default). Use structured logging (logr.Logger) to provide detailed, searchable logs that aid in debugging. Log significant events and errors.

Security Considerations

Controllers run with elevated permissions, making security a critical aspect.

• RBAC for the Controller: The config/rbac/ directory contains ClusterRole and ClusterRoleBinding manifests for your controller. These define the permissions (verbs like get, list, watch, create, update, delete) your controller's ServiceAccount has over various Kubernetes resources (e.g., myresources.app.example.com, deployments, services).
• Principle of Least Privilege: Grant your controller only the minimum necessary permissions. For example, if it only manages Deployments and Services, it shouldn't have permissions over Pods or Namespaces unless explicitly required. Review the generated ClusterRole carefully.
• Image Security: Use trusted base images for your controller's Dockerfile. Scan images for vulnerabilities.

Performance Considerations

Controllers should be efficient and avoid putting undue strain on the Kubernetes api server.

• Informers and Caching: controller-runtime and client-go are designed with performance in mind, using Informers to maintain local caches. This significantly reduces direct api server calls.
• Efficient Reconciliation Loops: Keep your Reconcile function lean. Avoid long-running operations. If complex, time-consuming tasks are needed, consider offloading them to separate goroutines or external systems, with the controller merely orchestrating their execution and monitoring their status.
• Consider the Implications of Large Numbers of CRs: If your controller needs to manage thousands of custom resources, optimize its reconciliation logic. Ensure indexers are used effectively for lookups, and consider sharding controllers if a single instance cannot handle the load. The less api calls a controller makes the better.

By meticulously following these deployment, testing, and best practices, you can ensure your Kubernetes controller is not only functional but also reliable, secure, and performant in a production environment, effectively extending the Kubernetes api to manage your custom applications.

Part 7: Advanced Controller Concepts and Ecosystem Integration

Building a basic controller to watch CRD changes is a significant achievement, but the Kubernetes ecosystem offers even more sophisticated patterns and tools for complex automation. This section explores some advanced controller concepts and integrates our discussion with the broader context of api management, naturally introducing APIPark.

Webhooks: Intercepting api Server Requests

Admission webhooks are an advanced form of Kubernetes extension that allows external HTTP callbacks to modify or validate api requests to the kube-apiserver. They provide a powerful mechanism to enforce policies or inject additional logic before objects are persisted in etcd.

• Mutating Admission Webhooks: These webhooks can intercept an api request and modify the object before it's stored. For instance, a mutating webhook could:
  • Automatically add labels or annotations to your MyResource objects upon creation.
  • Inject default values into fields if they are not specified by the user.
  • Add a sidecar container to a Pod created by your controller. This provides a central place to enforce consistency and apply common configurations.
• Validating Admission Webhooks: These webhooks can intercept an api request and reject it if the object violates certain rules or invariants. Unlike the basic OpenAPI schema validation in CRDs, validating webhooks can implement complex, dynamic, or cross-resource validation logic. For example:
  • Ensure that a MyResource's image field points to an approved registry.
  • Prevent deletion of a MyResource if it has active dependencies elsewhere in the cluster.
  • Implement custom business logic that cannot be expressed purely with JSON schema. Kubebuilder provides excellent support for scaffolding and implementing both mutating and validating webhooks, allowing them to be seamlessly integrated with your controller.

Finalizers: Ensuring Proper Cleanup

Kubernetes provides a mechanism called "finalizers" to control the deletion of resources. When an object has finalizers, its deletion request is blocked until all its finalizers are removed. This is incredibly useful for controllers that manage external resources or need to perform complex cleanup tasks.

If our MyResource controller, for example, were to create resources outside of Kubernetes (e.g., a database in a cloud provider, an entry in an external DNS system, or a bucket in object storage), simply relying on Kubernetes' garbage collection for owned resources wouldn't be enough. By adding a finalizer to the MyResource object: 1. When a user deletes the MyResource, its metadata.deletionTimestamp is set, but the object itself is not immediately removed. 2. Your controller observes this deletion timestamp, performs the necessary external cleanup (e.g., deleting the external database instance). 3. Once cleanup is complete, the controller removes its finalizer from the MyResource object. 4. Only then does the Kubernetes api server finally delete the MyResource object from etcd. This ensures that your controller has a guaranteed opportunity to clean up associated resources, preventing orphaned infrastructure.

Controller Composition: When Multiple Controllers Interact

In complex Kubernetes environments, it's common to have multiple controllers working together. For example, one controller might manage a custom database resource, while another manages applications that depend on that database. This raises challenges in coordination, dependencies, and ensuring consistent state across multiple resource types. Strategies for controller composition include: * Delegation: One controller creates a resource managed by another controller. For example, your MyResource controller creates a HelmRelease CRD, which is then managed by the Helm Operator. * Shared Information: Controllers might read from each other's custom resources to inform their own logic (e.g., a service mesh controller reads TrafficPolicy CRs created by a traffic management controller). * Event-Driven Communication: Controllers can emit Kubernetes events that other controllers listen to and react upon.

Cross-Resource Ownership: Managing Related Resources

While Owns in controller-runtime is excellent for managing resources that are directly children of your custom resource, sometimes a controller needs to manage resources that are merely related or referenced, but not strictly "owned" in the Kubernetes sense (i.e., not garbage-collected with the owner). For example, if your MyResource refers to a shared ConfigMap or Secret by name, your controller might need to watch those referenced resources for changes, even if it doesn't own them. This can be achieved using Watches with handler.EnqueueRequestsForObject() or by creating custom Predicates to filter events. This allows for more dynamic and complex relationships between resources.

Open Source AI Gateway & API Management - APIPark Integration

As developers build increasingly sophisticated custom resources and controllers to manage diverse workloads within Kubernetes, they often create services that expose their own APIs. Managing these APIs efficiently, securing them, and providing a developer-friendly portal becomes a subsequent, yet equally vital, challenge. This is where platforms like APIPark come into play.

APIPark, an open-source AI gateway and API management platform, offers a comprehensive solution for handling the lifecycle of APIs, from design and publication to monitoring and decommissioning. It seamlessly integrates a variety of AI models and standardizes API invocation formats, making it an invaluable tool for systems where custom controllers orchestrate AI workloads or complex microservices that ultimately expose an API to external consumers or internal teams.

Consider a scenario where our MyResource controller manages not just a generic application, but perhaps a custom AI model serving endpoint. This endpoint, once deployed by our controller, would expose an API for inference requests. To effectively manage this API – handling authentication, rate limiting, traffic routing, versioning, and providing a developer portal for consumers – a dedicated API gateway is indispensable. By leveraging APIPark, the APIs managed by our custom controllers, or any other service within our Kubernetes cluster, can benefit from unified authentication, cost tracking, prompt encapsulation into REST API, and end-to-end API lifecycle management, ensuring robust and scalable API governance. APIPark's ability to quickly integrate over 100 AI models and provide a unified API format for AI invocation makes it particularly relevant for controllers that manage machine learning inference services, offering a secure and performant layer for all API traffic. It ensures that the operational excellence achieved through custom Kubernetes controllers extends smoothly into the realm of API consumption and management, making the entire system more cohesive and manageable.

Conclusion

Building a Kubernetes controller to watch for changes to Custom Resource Definitions is a powerful way to extend the native capabilities of Kubernetes, enabling the automation of virtually any operational task. This comprehensive guide has walked through the fundamental components, from the intricate workings of the Kubernetes control plane and its api server, to the practical implementation of a controller using Kubebuilder and controller-runtime.

We began by dissecting the role of CRDs in defining custom api resources and how they integrate into the Kubernetes api ecosystem. We then delved into the anatomy of a controller, exploring the essential components like Informers, Workqueues, and the core reconciliation loop, which collectively form the brain of our automation logic. The practical walkthrough demonstrated how to scaffold a project, define the schema for our MyResource CRD, and implement the Reconcile function to create, update, and delete associated Kubernetes resources like Deployments and Services, all while meticulously observing changes to our custom resources.

Finally, we explored advanced concepts such as webhooks for enforcing policies, finalizers for robust cleanup, and the broader context of controller composition. We also saw how the services orchestrated by these controllers, often exposing their own APIs, can be effectively managed by platforms like APIPark, enhancing their security, discoverability, and overall lifecycle governance within a sophisticated cloud-native landscape.

The journey of building a Kubernetes controller is one of deep learning and practical application. It empowers developers and operators to transform complex, manual operational procedures into declarative, self-healing automation. By understanding and leveraging the Kubernetes api and its extension mechanisms, you gain the ability to tailor Kubernetes precisely to your application's unique needs, paving the way for truly autonomous and resilient systems. The power of Kubernetes lies not just in what it offers out-of-the-box, but in its boundless extensibility, and custom controllers are the key to unlocking that potential.

FAQ

1. What is the fundamental difference between a Custom Resource (CR) and a Custom Resource Definition (CRD)? A Custom Resource Definition (CRD) is the schema or blueprint that defines a new, custom resource type within Kubernetes. It tells the kube-apiserver what the resource is called, its scope (namespaced or cluster-wide), its api group, version, and the validation schema for its spec and status fields. A Custom Resource (CR), on the other hand, is an actual instance of that custom resource type. It's like the difference between a class definition (CRD) and an object created from that class (CR). You define the CRD once, and then you can create multiple CRs based on that definition using the Kubernetes api.

2. Why do I need a controller if I can define a Custom Resource Definition (CRD)? While a CRD allows you to extend the Kubernetes api with your own resource types, the CRD itself is merely a data schema; it doesn't do anything. A controller is the active component that watches for changes to instances of your custom resource (CRs) and takes actions to reconcile the actual state of the cluster with the desired state specified in those CRs. Without a controller, your custom resources would be inert data objects, incapable of automating or managing anything within the cluster. The controller is what brings your custom resource to life, interacting with the Kubernetes api to create, update, or delete other Kubernetes resources (like Deployments, Services) or even external infrastructure.

3. What is the role of client-go, controller-runtime, and Kubebuilder in controller development? * client-go: This is the lowest-level official Go client library for interacting with the Kubernetes api. It provides the raw building blocks like api clients, informers, and workqueues. Building a controller directly with client-go gives maximum control but requires a lot of boilerplate code. * controller-runtime: A library built on top of client-go that provides a higher-level framework. It abstracts away much of the boilerplate by offering a Manager to handle shared caches and api clients, and a simple Reconciler interface to implement your core logic. It's the recommended way to build most controllers. * Kubebuilder: A command-line tool built on controller-runtime. It scaffolds entire controller projects, generates CRD YAML from Go types, and includes makefiles and deployment manifests. It greatly speeds up development by adhering to best practices and automating common tasks, making it the ideal choice for starting new Operator projects.

4. How does a controller "watch for changes" to a CRD, and what happens when a change is detected? A controller primarily watches for changes to Custom Resources (CRs), which are instances defined by a CRD. It doesn't typically watch the CRD definition itself unless its logic needs to adapt to schema changes. This watching is done through controller-runtime's For() and Owns() methods in SetupWithManager. Behind the scenes, controller-runtime uses client-go Informers, which maintain a local cache of CRs by performing an initial List and then maintaining a continuous Watch connection to the kube-apiserver. When the apiserver notifies the Informer of an Add, Update, or Delete event for a CR, the Informer updates its cache and enqueues the CR's key into a workqueue. A worker goroutine then pulls the key from the workqueue and invokes the controller's Reconcile function for that specific CR. This reconciliation logic then compares the CR's desired state with the actual cluster state and takes appropriate actions, all through interactions with the Kubernetes api.

5. How does APIPark fit into a Kubernetes environment with custom controllers? While a custom Kubernetes controller excels at managing the lifecycle of applications and services within the Kubernetes cluster, the applications or microservices orchestrated by these controllers often expose their own APIs to be consumed by external users or other internal services. This is where APIPark, an open-source AI gateway and API management platform, becomes invaluable. APIPark provides a layer for managing these exposed APIs, offering features like unified authentication, rate limiting, traffic routing, versioning, and a developer portal. For example, if your controller deploys an API-driven AI inference service, APIPark can sit in front of that service, ensuring secure, performant, and well-governed API access. It complements the operational automation provided by custom controllers by handling the external facing aspects of API consumption and lifecycle, enhancing the overall system's manageability and security, especially for workloads involving AI models that present specific challenges in API standardization and integration.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.