How to Build a Controller to Watch for CRD Changes

Kubernetes has revolutionized how we deploy and manage containerized applications, offering an unparalleled level of abstraction and automation for complex distributed systems. At its core, Kubernetes operates on a declarative API, where users define the desired state of their applications and the system works to converge to that state. While Kubernetes provides a rich set of built-in resource types like Pods, Deployments, and Services, real-world applications often demand custom resource types to represent domain-specific concepts. This is where Custom Resource Definitions (CRDs) come into play, extending the Kubernetes API to allow users to define their own resource types.

However, merely defining a custom resource is only half the battle. To bring these custom resources to life and imbue them with operational intelligence, we need a mechanism to observe changes to them and react accordingly. This mechanism is known as a Kubernetes Controller. Controllers are the operational brains of Kubernetes, continuously watching for changes in the cluster's state and taking actions to reconcile the current state with the desired state. Building a controller to watch for CRD changes is a fundamental skill for anyone looking to extend Kubernetes' capabilities, automate complex workflows, or build Operators.

This comprehensive guide will delve deep into the intricacies of building such a controller. We'll start by laying the theoretical groundwork, understanding the core components of the Kubernetes control plane and the role of CRDs and controllers. We will then embark on a practical journey, step-by-step, to construct a robust and production-ready controller using the controller-runtime framework, a powerful set of libraries that simplifies controller development. By the end of this guide, you will possess a profound understanding of how to architect, implement, and deploy a Kubernetes controller capable of intelligently responding to changes in your custom resources.

The Foundation: Understanding Kubernetes' Control Plane and Extensibility

Before diving into controller development, it's crucial to grasp the fundamental architecture of Kubernetes, particularly its control plane, and how CRDs fit into this ecosystem. This foundational knowledge provides the necessary context for understanding why controllers are essential and how they interact with the broader Kubernetes environment.

The Kubernetes API Server: The Heart of the Cluster

At the very center of the Kubernetes control plane sits the API server. This component is the primary interface for users, external components, and other control plane components to interact with the cluster. All communications, whether creating a Pod, updating a Deployment, or querying the state of a Service, flow through the API server. It acts as a RESTful api endpoint, handling authentication, authorization, and validation for all incoming requests.

The API server's critical role is to persist the desired state of the cluster into a reliable, distributed key-value store, etcd. When you create a resource (e.g., a Deployment), you are essentially telling the API server, "I desire a Deployment with these specifications." The API server stores this desired state, and other components, including controllers, then work to make the cluster's actual state match this desired state. This declarative model is a cornerstone of Kubernetes' power and resilience.

Custom Resource Definitions (CRDs): Extending the Kubernetes API

While Kubernetes provides a rich set of built-in resource types, there will inevitably be scenarios where these generic resources are insufficient to model the specific domain concepts of your application or infrastructure. Imagine you are building a database-as-a-service platform on Kubernetes. You might want a resource type like DatabaseInstance to represent a provisioned database, specifying its type (e.g., PostgreSQL, MySQL), version, size, and replication strategy. Kubernetes doesn't have a built-in DatabaseInstance resource.

This is where CRDs come to the rescue. A CRD is itself a Kubernetes resource that defines a new, custom resource type. When you create a CRD, you are effectively telling the Kubernetes API server, "Hey, I'm introducing a new kind of object that can be stored and managed just like Pods or Deployments." Once a CRD is created, the API server begins to serve the new custom resource API endpoint. For instance, if you define a CRD named DatabaseInstance, you can then create, update, and delete DatabaseInstance objects using kubectl or any Kubernetes API client, just as you would with native resources.

A CRD definition includes: * apiVersion and kind: Standard Kubernetes metadata. * metadata: Name of the CRD. * spec: * group: The API group (e.g., database.example.com). * names: Defines singular, plural, shortnames, and the kind for your custom resource. * scope: Namespaced or Cluster (whether instances are confined to a namespace or cluster-wide). * versions: Defines the different versions of your custom resource's schema, including conversion strategies and schema validation rules.

The versions field, particularly the schema.openAPIV3Schema, is incredibly important. It allows you to define a robust JSON schema for your custom resource, ensuring that all created objects conform to your specified structure. This schema validation is performed by the API server, preventing malformed custom resources from ever being stored in etcd. This strict validation enhances the reliability and predictability of your custom resources.

Controllers and Operators: Bringing Custom Resources to Life

Defining a CRD merely creates a new data structure in Kubernetes. It doesn't automatically give that data structure any operational meaning or behavior. This is where controllers, particularly when paired with CRDs, evolve into what we often call "Operators." An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a user.

A controller's fundamental job is to watch a particular type of resource (or multiple types) in the Kubernetes API server. When it detects a change (creation, update, or deletion) to a resource it's watching, it performs a specific set of actions to reconcile the cluster's current state with the desired state defined by that resource.

For a CRD-based controller, this means: 1. Watching Custom Resources: The controller registers to receive notifications whenever a custom resource of a specific kind (defined by your CRD) is created, updated, or deleted. 2. Reconciliation: Upon receiving a notification, the controller fetches the latest state of the custom resource and compares it to the actual state of the cluster. Based on this comparison, it then takes corrective actions. * If a DatabaseInstance custom resource is created, the controller might provision a new database server, create a corresponding Kubernetes Service, and update the DatabaseInstance's status to reflect the server's endpoint. * If a DatabaseInstance is updated (e.g., its size increases), the controller might initiate a resizing operation on the underlying database server. * If a DatabaseInstance is deleted, the controller might de-provision the database server and clean up associated resources.

This continuous watch-and-reconcile loop is what gives Kubernetes its self-healing and automated capabilities. Building a controller to watch CRD changes is essentially building an automation engine for your domain-specific resources.

Prerequisites for Controller Development

To effectively build a Kubernetes controller, you'll need a development environment set up with specific tools and a basic understanding of key concepts.

Essential Tools and Environment

• Go Programming Language: Kubernetes components, including controllers, are predominantly written in Go. You'll need Go installed (version 1.16 or higher is recommended) and configured correctly (GOPATH, etc.).
• Kubernetes Cluster: Access to a Kubernetes cluster (local like Minikube, kind, or a cloud-managed cluster) is essential for testing your controller.
• kubectl: The Kubernetes command-line tool, used for interacting with your cluster, deploying CRDs, and observing your controller's behavior.
• git: For version control.
• controller-gen: A tool for generating Go code and YAML manifests (CRDs, RBAC) from Go types and markers. This is part of the controller-runtime ecosystem and vastly simplifies development.
• kustomize: A tool for customizing Kubernetes configurations, often used for packaging and deploying controller manifests.

Key Concepts and Libraries

• client-go: The official Go client library for interacting with the Kubernetes API. While controller-runtime abstracts much of client-go, understanding its core components (Clientset, Informers, Listers, Workqueues) is beneficial.
• controller-runtime: A high-level library built on top of client-go that provides common patterns and utilities for building controllers. It simplifies tasks like managing caches, handling events, and implementing reconcile loops. It's the recommended way to build controllers today.
• dep / go mod: Go's dependency management systems. go mod is the current standard.
• RBAC (Role-Based Access Control): Your controller will need specific permissions to interact with the Kubernetes API (e.g., to read/write custom resources, create/manage Pods). RBAC definitions are crucial for securing your controller.

Step-by-Step Guide: Building a CRD Controller with controller-runtime

We will build a controller that manages a hypothetical "Backup" resource. This Backup resource will define the desired state of a backup operation for a specific application. Our controller will then watch for these Backup resources and simulate a backup operation, updating the Backup resource's status accordingly.

Step 1: Initialize the Project and Scaffold the Controller

We'll start by initializing a new Go module and using the kubebuilder CLI (which leverages controller-runtime) to scaffold our project. kubebuilder is an excellent tool that automates much of the boilerplate.

# Install kubebuilder CLI
go install sigs.k8s.io/kubebuilder/cmd/kubebuilder@latest

# Initialize a new project
mkdir backup-controller
cd backup-controller
kubebuilder init --domain example.com --repo github.com/your-org/backup-controller

# Create the API (CRD definition)
kubebuilder create api --group backup --version v1 --kind Backup --resource=true --controller=true

This command generates several files: * api/v1/backup_types.go: Defines the Go structs for your Backup custom resource. * controllers/backup_controller.go: Contains the skeleton for your controller's reconciliation logic. * config/: Contains YAML manifests for CRDs, RBAC roles, and controller deployment.

Step 2: Define the Custom Resource (CRD) Schema

Open api/v1/backup_types.go. This file defines the Go struct for your Backup custom resource, specifically BackupSpec (the desired state) and BackupStatus (the actual observed state).

Let's define a Backup resource that includes: * Source: A string indicating what to back up (e.g., a PVC name, a deployment name). * Schedule: A cron string for recurring backups. * Destination: Where to store the backup (e.g., an S3 bucket name). * RetentionPolicy: How long to keep backups.

And for the Status: * Phase: Current phase of the backup (e.g., Pending, Running, Completed, Failed). * LastBackupTime: Timestamp of the last successful backup. * Message: Any relevant status message.

package v1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the Kubernetes API to work.

// BackupSpec defines the desired state of Backup
type BackupSpec struct {
    // Source specifies the application or data source to back up.
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`
    Source string `json:"source"`

    // Schedule for recurring backups (cron format).
    // +kubebuilder:validation:Pattern="^(@(yearly|annually|monthly|weekly|daily|hourly|reboot))|((((([0-5]?\\d)[,*-/])*([0-5]?\\d)+)|\\*)(\\s)+((([0-5]?\\d)[,*-/])*([0-5]?\\d)+)|\\*)(\\s)+(((([01]?\\d|2[0-3])[,*-/])*([01]?\\d|2[0-3])+)|\\*)(\\s)+((([1-9]|1\\d|2\\d|3[01])[,*-/])*([1-9]|1\\d|2\\d|3[01])+)|\\*)(\\s)+((([1-9]|1[012])[,*-/])*([1-9]|1[012])+)|\\*)(\\s)+((([0-6])[,*-/])*([0-6])+)|\\*)$"
    Schedule string `json:"schedule"`

    // Destination specifies where the backup should be stored (e.g., S3 bucket name).
    // +kubebuilder:validation:MinLength=1
    Destination string `json:"destination"`

    // RetentionPolicy defines how long to keep the backups (e.g., "7d", "30d").
    // +kubebuilder:validation:Pattern="^([0-9]+(s|m|h|d|w|y))+$"
    RetentionPolicy string `json:"retentionPolicy,omitempty"`
}

// BackupPhase defines the possible phases of a backup operation.
// +kubebuilder:validation:Enum=Pending;Running;Completed;Failed
type BackupPhase string

const (
    BackupPhasePending   BackupPhase = "Pending"
    BackupPhaseRunning   BackupPhase = "Running"
    BackupPhaseCompleted BackupPhase = "Completed"
    BackupPhaseFailed    BackupPhase = "Failed"
)

// BackupStatus defines the observed state of Backup
type BackupStatus struct {
    // Phase indicates the current phase of the backup operation.
    // +optional
    Phase BackupPhase `json:"phase,omitempty"`

    // LastBackupTime indicates the timestamp of the last successful backup.
    // +optional
    LastBackupTime *metav1.Time `json:"lastBackupTime,omitempty"`

    // Message provides a human-readable status message.
    // +optional
    Message string `json:"message,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Source",type="string",JSONPath=".spec.source",description="Source of the backup"
// +kubebuilder:printcolumn:name="Schedule",type="string",JSONPath=".spec.schedule",description="Backup schedule"
// +kubebuilder:printcolumn:name="Destination",type="string",JSONPath=".spec.destination",description="Backup destination"
// +kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase",description="Current phase of the backup"
// +kubebuilder:printcolumn:name="Last Backup",type="date",JSONPath=".status.lastBackupTime",description="Last successful backup time"

// Backup is the Schema for the backups API
type Backup struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   BackupSpec   `json:"spec,omitempty"`
    Status BackupStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// BackupList contains a list of Backup
type BackupList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Backup `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Backup{}, &BackupList{})
}

Important markers: * +kubebuilder:validation:*: These are crucial for defining OpenAPI schema validation rules, ensuring data integrity. * +kubebuilder:subresource:status: This tells controller-gen to enable the /status subresource for your CRD, allowing separate updates to the status field without requiring a full object update. * +kubebuilder:printcolumn: These define custom columns for kubectl get backups output, making your custom resources more user-friendly.

After modifying the types, regenerate the CRD manifests and client code:

make generate
make manifests

This updates config/crd/bases/backup.example.com_backups.yaml and other generated files.

Step 3: Implement the Controller Logic (Reconcile Loop)

The core logic of your controller resides in the controllers/backup_controller.go file, specifically within the Reconcile method of the BackupReconciler struct. This method is called by controller-runtime whenever a change is detected for a Backup resource that the controller is watching.

The Reconcile function's signature is Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error). * ctx: Standard Go context for cancellation and timeouts. * req: A reconcile.Request containing the NamespacedName (namespace and name) of the Backup object that triggered the reconciliation. * ctrl.Result: Tells controller-runtime what to do next (e.g., re-queue, with or without a delay). * error: If an error occurs, the resource might be re-queued.

Understanding the Reconcile Flow

1. Fetch the Custom Resource: The first step in any reconciliation is to retrieve the current state of the custom resource from the API server. If the resource no longer exists (e.g., it was deleted), the controller should usually clean up any external resources it created.
2. Compare Desired vs. Actual State: The controller then compares the Spec (desired state) of the Backup object with the actual state of the world (e.g., whether a backup job is running, when the last backup completed).
3. Take Action: Based on the comparison, the controller performs necessary actions to converge the actual state to the desired state. This could involve creating other Kubernetes resources (e.g., a CronJob, a Job, or even external calls to an object storage service) or updating the Backup object's Status.
4. Update Status: After performing actions, it's crucial to update the Status field of the Backup resource to reflect the current state of the world. This provides visibility to users and other controllers.
5. Handle Deletion (Finalizers): If the custom resource is being deleted, the controller might need to perform cleanup operations before the resource is fully removed from etcd. This is typically handled using finalizers.

Let's modify controllers/backup_controller.go:

package controllers

import (
    "context"
    "fmt"
    "time"

    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    backupv1 "github.com/your-org/backup-controller/api/v1"
)

// BackupReconciler reconciles a Backup object
type BackupReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=backup.example.com,resources=backups,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=backup.example.com,resources=backups/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=backup.example.com,resources=backups/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete // Example: if controller creates Pods
// +kubebuilder:rbac:groups="batch",resources=cronjobs,verbs=get;list;watch;create;update;patch;delete // Example: if controller creates CronJobs

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify Reconcile to be able to reconcile your objects via the client package
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.16.3/pkg/reconcile
func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // 1. Fetch the Backup instance
    backup := &backupv1.Backup{}
    err := r.Get(ctx, req.NamespacedName, backup)
    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Owned objects are automatically garbage collected. For additional cleanup logic, use finalizers.
            logger.Info("Backup resource not found. Ignoring since object must be deleted.")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        logger.Error(err, "Failed to get Backup")
        return ctrl.Result{}, err
    }

    // 2. Handle Deletion (Finalizers)
    // Our sample controller doesn't manage external resources that require explicit cleanup.
    // If it did (e.g., an external S3 bucket), we would add a finalizer here.
    // For demonstration, let's add a placeholder finalizer logic.
    backupFinalizer := "backup.example.com/finalizer"
    if backup.ObjectMeta.DeletionTimestamp.IsZero() {
        // The object is not being deleted, so if it does not have our finalizer,
        // then lets add it.
        if !containsString(backup.ObjectMeta.Finalizers, backupFinalizer) {
            backup.ObjectMeta.Finalizers = append(backup.ObjectMeta.Finalizers, backupFinalizer)
            if err := r.Update(ctx, backup); err != nil {
                logger.Error(err, "Failed to add finalizer to Backup resource")
                return ctrl.Result{}, err
            }
            logger.Info("Added finalizer to Backup resource", "backup", req.NamespacedName)
        }
    } else {
        // The object is being deleted
        if containsString(backup.ObjectMeta.Finalizers, backupFinalizer) {
            // Our finalizer is present, so let's handle any external dependency
            // that needs to be cleaned up.
            logger.Info("Performing finalizer cleanup for Backup resource", "backup", req.NamespacedName)
            // TODO(user): Add actual cleanup logic here (e.g., deleting S3 bucket, unregistering external service)

            // Remove our finalizer from the list and update it.
            backup.ObjectMeta.Finalizers = removeString(backup.ObjectMeta.Finalizers, backupFinalizer)
            if err := r.Update(ctx, backup); err != nil {
                logger.Error(err, "Failed to remove finalizer from Backup resource")
                return ctrl.Result{}, err
            }
            logger.Info("Removed finalizer from Backup resource", "backup", req.NamespacedName)
        }
        // Stop reconciliation as the item is being deleted
        return ctrl.Result{}, nil
    }

    // 3. Define the desired state (simplified for this example)
    // For a real backup, you might create a Kubernetes Job or CronJob here
    // and monitor its status. For this example, we'll simulate an immediate backup.

    // If the backup is in Pending phase or no phase, start processing
    if backup.Status.Phase == "" || backup.Status.Phase == backupv1.BackupPhasePending {
        logger.Info("Starting backup operation", "backup", backup.Name, "source", backup.Spec.Source)

        // Update status to Running
        backup.Status.Phase = backupv1.BackupPhaseRunning
        backup.Status.Message = fmt.Sprintf("Backup for '%s' is in progress to destination '%s'.", backup.Spec.Source, backup.Spec.Destination)
        if err := r.Status().Update(ctx, backup); err != nil {
            logger.Error(err, "Failed to update Backup status to Running")
            return ctrl.Result{}, err // Requeue
        }
        logger.Info("Backup status updated to Running", "backup", backup.Name)

        // Simulate backup process: In a real scenario, this would involve
        // interacting with external services or creating Kubernetes resources.
        // For now, we'll simply re-queue after a short delay to simulate work.
        logger.Info("Simulating backup work...", "backup", backup.Name)
        // We re-queue with a delay, so the controller will pick it up again
        // and check if the 'simulated' backup has completed.
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }

    // If the backup is Running, check if it's "done"
    if backup.Status.Phase == backupv1.BackupPhaseRunning {
        // In a real controller, you would check the status of a created Job/CronJob
        // or an external API call here.
        // For this example, we'll just assume it completes after a certain time/reconcile cycles.
        logger.Info("Backup is still in Running phase, checking progress...", "backup", backup.Name)

        // Assume it completes if enough time has passed since it started running,
        // or based on some other external condition. For this simple example,
        // we'll just transition it to Completed after a few reconcile loops.
        // This needs to be deterministic and based on actual external state in a real scenario.
        if time.Since(backup.CreationTimestamp.Time) > 20*time.Second && backup.Status.LastBackupTime == nil {
            logger.Info("Simulated backup operation completed successfully.", "backup", backup.Name)
            backup.Status.Phase = backupv1.BackupPhaseCompleted
            now := metav1.Now()
            backup.Status.LastBackupTime = &now
            backup.Status.Message = fmt.Sprintf("Backup for '%s' completed successfully at %s to destination '%s'.", backup.Spec.Source, now.Format(time.RFC3339), backup.Spec.Destination)
            if err := r.Status().Update(ctx, backup); err != nil {
                logger.Error(err, "Failed to update Backup status to Completed")
                return ctrl.Result{}, err // Requeue
            }
            logger.Info("Backup status updated to Completed", "backup", backup.Name)
            return ctrl.Result{}, nil // Do not re-queue immediately
        } else {
            // Still waiting for completion, re-queue to check again later.
            return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
        }
    }

    // If the backup is Completed or Failed, there's nothing more to do for now.
    // For schedules, a CronJob would handle recurring creation.
    logger.Info("Backup in stable phase, no action needed.", "backup", backup.Name, "phase", backup.Status.Phase)
    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *BackupReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&backupv1.Backup{}).
        Complete(r)
}

// Helper functions to manage finalizers
func containsString(slice []string, s string) bool {
    for _, item := range slice {
        if item == s {
            return true
        }
    }
    return false
}

func removeString(slice []string, s string) (result []string) {
    for _, item := range slice {
        if item == s {
            continue
        }
        result = append(result, item)
    }
    return
}

Explanation of the Reconcile Logic:

• Fetching the Resource: r.Get(ctx, req.NamespacedName, backup) retrieves the Backup object. errors.IsNotFound is crucial for handling deletions gracefully; if the object is gone, there's nothing to do.
• Finalizers: The finalizer logic ensures that even if a Backup resource is deleted, the controller gets a chance to perform cleanup (e.g., deleting a snapshot in an external storage system) before the resource is completely removed from Kubernetes. Without finalizers, the resource would be removed immediately, potentially leaving orphaned external resources.
• State Machine: The controller implements a simple state machine based on backup.Status.Phase:
  • Pending/Empty Phase: The controller starts the simulated backup process, updates the status to Running, and requests a re-queue.
  • Running Phase: The controller checks if the simulated backup is "complete." In a real scenario, it would query an external system or check a dependent Kubernetes Job's status. Once done, it updates the status to Completed.
  • Completed/Failed Phase: If the backup is in a terminal state, the controller typically does nothing until the Spec changes again or a new backup is explicitly requested (e.g., by a CronJob creating a new Backup resource).
• Status Updates: r.Status().Update(ctx, backup) is used specifically to update the Status subresource. This is generally preferred over r.Update(ctx, backup) for Status fields as it allows for concurrent updates to Spec and Status and prevents conflicts.
• ctrl.Result{RequeueAfter: ...}: This is used to tell controller-runtime to re-queue the current reconciliation request after a specified duration. This is useful for polling external systems or simulating asynchronous operations. If no error occurs and no re-queue is requested, controller-runtime will only re-trigger reconciliation if the watched object changes.

Step 4: Configure Main Function (main.go)

The main.go file sets up the controller manager and registers your controller. kubebuilder generates a good boilerplate here. The key part is mgr.Start(ctrl.SetupSignalHandler()) which starts all registered controllers and blocks until a termination signal is received.

No major changes are typically needed here, but it's where you would register multiple controllers or custom webhooks.

package main

import (
    "flag"
    "os"

    // Import all Kubernetes client auth plugins (e.g., Azure, GCP, OIDC, etc.)
    // to ensure that exec-entrypoint and run can make use of them.
    _ "k8s.io/client-go/plugin/pkg/client/auth"

    "k8s.io/apimachinery/pkg/runtime"
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/healthz"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"

    backupv1 "github.com/your-org/backup-controller/api/v1"
    "github.com/your-org/backup-controller/controllers"
    // +kubebuilder:scaffold:imports
)

var (
    scheme   = runtime.NewScheme()
    setupLog = ctrl.Log.WithName("setup")
)

func init() {
    utilruntime.Must(clientgoscheme.AddToScheme(scheme))

    utilruntime.Must(backupv1.AddToScheme(scheme))
    // +kubebuilder:scaffold:scheme
}

func main() {
    var metricsAddr string
    var enableLeaderElection bool
    var probeAddr string
    flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
    flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
    flag.BoolVar(&enableLeaderElection, "leader-elect", false,
        "Enable leader election for controller manager. "+
            "Enabling this will ensure there is only one active controller manager.")
    opts := zap.Options{
        Development: true,
    }
    opts.BindFlags(flag.CommandLine)
    flag.Parse()

    ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

    mgr, err := ctrl.NewManager(ctrl.Options{
        Scheme:                 scheme,
        MetricsBindAddress:     metricsAddr,
        Port:                   9443,
        HealthProbeBindAddress: probeAddr,
        LeaderElection:         enableLeaderElection,
        LeaderElectionID:       "a926a798.example.com",
        // LeaderElectionReleaseOnCancel: true, // Use this for faster failover in development
    })
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }

    if err = (&controllers.BackupReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "Backup")
        os.Exit(1)
    }
    // +kubebuilder:scaffold:builder

    if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
        setupLog.Error(err, "unable to set up health check")
        os.Exit(1)
    }
    if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
        setupLog.Error(err, "unable to set up ready check")
        os.Exit(1)
    }

    setupLog.Info("starting manager")
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

Step 5: Build and Deploy the Controller

Now that the code is complete, we need to build the controller binary and deploy it to our Kubernetes cluster.

Building the Docker Image

make docker-build IMG="your-docker-repo/backup-controller:v1.0.0"
docker push your-docker-repo/backup-controller:v1.0.0

Replace your-docker-repo with your actual Docker Hub username or image registry.

Deploying to Kubernetes

First, apply the CRD definition to your cluster. This step must be done before the controller starts, otherwise, the API server won't know about your Backup resource type.

kubectl apply -f config/crd/bases/backup.example.com_backups.yaml

Next, deploy the controller itself. kubebuilder generates all necessary manifests (Deployment, ServiceAccount, ClusterRole, ClusterRoleBinding) in the config/ directory.

# Apply RBAC and controller deployment
kubectl apply -f config/rbac/
kubectl apply -f config/samples/
kubectl apply -f config/manager/

The config/samples/ directory might contain example custom resources. For our Backup resource, we can create one manually or via kubebuilder:

# Create an example Backup custom resource
cat <<EOF | kubectl apply -f -
apiVersion: backup.example.com/v1
kind: Backup
metadata:
  name: my-app-backup
  namespace: default
spec:
  source: my-application-pvc
  schedule: "0 2 * * *" # Every day at 2 AM
  destination: s3://my-backup-bucket
  retentionPolicy: 30d
EOF

Verifying the Deployment

You can check if your controller pod is running:

kubectl get pods -n backup-controller-system

(The namespace might vary based on your kubebuilder setup, often default or system namespace for sample deployments, or your-project-name-system as generated by kubebuilder).

Watch the controller logs:

kubectl logs -f -n backup-controller-system <controller-pod-name>

You should see your controller logging messages as it reconciles the my-app-backup resource.

Observe the Backup resource:

kubectl get backup my-app-backup -o yaml

You should see its status.phase change from Pending to Running and then to Completed as your controller processes it. The LastBackupTime and Message fields in the status should also update.

kubectl get backup -o custom-columns=NAME:.metadata.name,SOURCE:.spec.source,SCHEDULE:.spec.schedule,DESTINATION:.spec.destination,PHASE:.status.phase,LAST_BACKUP:.status.lastBackupTime

This command will display the custom columns you defined in backup_types.go, providing a clear overview of your Backup resources.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Advanced Controller Concepts

While the above provides a solid foundation, real-world controllers often leverage more advanced features:

Owner References and Garbage Collection

Controllers frequently create other Kubernetes resources (e.g., Pods, Deployments, Services) to fulfill the desired state defined by a custom resource. To ensure these dependent resources are automatically cleaned up when the custom resource is deleted, controllers establish "owner references." When a custom resource is deleted, Kubernetes' garbage collector identifies and deletes all resources that list the custom resource as their owner. controller-runtime makes this easy with ctrl.SetControllerReference.

Webhooks: Mutating and Validating Admission Controllers

For more complex validation or modification of custom resources before they are stored, webhooks are invaluable. * Validating Admission Webhooks: Intercept API requests (create, update, delete) for your custom resource and can reject them if they don't meet specific criteria beyond what OpenAPI schema validation can provide (e.g., cross-field validation, validation against other cluster resources). * Mutating Admission Webhooks: Modify the custom resource object before it's persisted (e.g., injecting default values, adding labels/annotations). kubebuilder can also scaffold webhooks for your CRDs.

Leader Election

In production environments, you typically run multiple replicas of your controller for high availability. However, to prevent conflicting actions (e.g., multiple controllers trying to provision the same external resource), only one controller instance should be active at any given time. This is achieved through "leader election," where controller instances compete to become the leader, and only the leader performs the reconciliation. controller-runtime has built-in support for leader election.

Event Handlers and Predicates

Beyond simply watching a CRD, controllers can also watch other Kubernetes resources (e.g., Pods, Deployments) that they own or are interested in. When those dependent resources change, the controller might need to re-queue its primary custom resource for reconciliation. controller-runtime provides Watches and Owns functions in SetupWithManager to define these relationships. Predicates allow you to filter which events trigger a reconciliation, reducing unnecessary work.

Integrating with the Broader API Landscape: Beyond Kubernetes

While your controller meticulously manages custom resources within the Kubernetes ecosystem, the broader landscape of modern application development often involves a vast array of external APIs, AI models, and microservices that require robust management beyond the internal Kubernetes API server itself.

Consider a scenario where your Backup controller, once fully operational, needs to interact with an external object storage service like AWS S3 or Azure Blob Storage. These interactions rely on external APIs. Furthermore, many enterprise applications today incorporate complex AI models for tasks like data analysis or content generation, which also expose their functionalities through APIs. Managing the entire lifecycle of these application-level APIs—from discovery and consumption to security, rate limiting, and analytics—becomes a critical concern that transcends the scope of a single Kubernetes controller.

This is precisely where specialized API management platforms and AI gateways come into play. They act as a centralized control plane for all your external api interactions, providing a unified interface for developers and ensuring consistent governance across your entire API portfolio. For organizations looking to streamline the management of both traditional REST APIs and the increasingly prevalent AI model APIs, an open-source solution like APIPark offers a compelling advantage.

APIPark serves as an all-in-one AI gateway and API developer portal, designed to simplify the integration, deployment, and management of hundreds of AI models and REST services. Imagine your Backup controller needs to use an AI model to analyze backup health reports or predict storage needs. With APIPark, you can quickly integrate these AI models, standardize their invocation format, and encapsulate complex prompts into simple REST APIs. This means your controller, or any other application, can interact with diverse AI functionalities through a consistent and managed API endpoint, without needing to know the underlying complexities of each specific AI model.

Beyond AI integration, APIPark also provides comprehensive end-to-end API lifecycle management, assisting with design, publication, invocation, and decommissioning of all your application APIs. It enables robust features like traffic forwarding, load balancing, versioning, and detailed call logging, ensuring stability, security, and visibility into your API ecosystem. For teams, it facilitates API service sharing and allows for independent API and access permissions per tenant, enhancing collaboration while maintaining security. The high-performance capabilities, rivaling Nginx with over 20,000 TPS on modest hardware, ensure it can handle large-scale enterprise traffic efficiently. Thus, while your Kubernetes controller excels at managing internal cluster resources, platforms like APIPark perfectly complement this by offering a powerful solution for managing and securing the external api connections that drive modern, data-intensive applications.

Best Practices for Controller Development

Building a robust controller requires adherence to several best practices:

• Idempotency: Reconcile loops should be idempotent, meaning running the same reconciliation multiple times with the same input should produce the same result without unintended side effects. This is crucial because controllers might re-run reconciliation for various reasons (network issues, restarts, etc.).
• Declarative vs. Imperative: Always strive for a declarative approach. Define the desired state in your CRD's Spec, and let the controller work towards achieving that state, rather than prescribing a step-by-step imperative process.
• Error Handling and Re-queues: Properly handle errors in the Reconcile function. Returning an error will typically cause controller-runtime to re-queue the request, retrying later. Differentiate between transient errors (re-queue) and permanent errors (log and don't re-queue, perhaps update status to Failed).
• Status Updates: Always update the Status field of your custom resource to reflect the current state. This is how users and other controllers know what's happening. Use the Status subresource for this.
• Event Generation: Generate Kubernetes Events to provide human-readable messages about significant actions taken by your controller (e.g., "Backup created," "Backup failed"). This helps with debugging and auditing.
• Logging: Use structured logging (e.g., controller-runtime/pkg/log) to provide clear, actionable insights into your controller's operations. Include relevant identifiers like NamespacedName for easy filtering.
• Resource Limits: Define resource requests and limits for your controller's Deployment to ensure it doesn't consume excessive cluster resources and is scheduled appropriately.
• Testing: Implement unit tests for your reconciliation logic and integration tests using envtest (part of controller-runtime) to run your controller against a real (but in-memory) API server.
• Garbage Collection: Ensure proper owner references for all resources your controller creates to prevent orphaned resources upon deletion of your custom resource.
• Observability: Expose metrics (e.g., Prometheus metrics) from your controller to monitor its performance, reconciliation times, and error rates. controller-runtime integrates with Prometheus by default.

Conclusion

Building a controller to watch for CRD changes is a powerful way to extend Kubernetes and automate complex operational tasks. By understanding the interplay between the Kubernetes API server, Custom Resource Definitions, and the controller's reconciliation loop, developers can create sophisticated Operators that bring domain-specific intelligence directly into their clusters. The controller-runtime framework significantly streamlines this process, providing a robust and well-structured foundation for development.

From defining the desired state in your CRD's Spec to meticulously implementing the Reconcile function and handling lifecycle events, each step contributes to building an intelligent automation engine. This deep dive into a practical example, combined with an overview of advanced concepts and best practices, equips you with the knowledge to craft production-ready controllers. As you embrace the extensibility of Kubernetes, remember that while your controllers master the internal dynamics of your cluster, external API management tools like APIPark can complement this by providing a unified, secure, and performant gateway for all your application-level APIs, including those powered by cutting-edge AI models, thus forming a holistic approach to managing your entire digital landscape. The ability to create custom, self-healing, and self-managing systems within Kubernetes empowers organizations to build more resilient, scalable, and efficient applications.

---

5 Frequently Asked Questions (FAQs)

1. What is the fundamental difference between a CRD and a Custom Resource (CR)? A CRD (Custom Resource Definition) is the schema or blueprint that defines a new, custom resource type within Kubernetes. It tells the Kubernetes API server what fields a custom resource of that type can have, its validation rules, and how it should behave (e.g., namespaced or cluster-scoped). A Custom Resource (CR) is an actual instance of a resource created according to the rules defined by a CRD. For example, if you define a CRD for DatabaseInstance, then my-postgres-db is a Custom Resource of the DatabaseInstance type. Think of the CRD as the class definition and the CR as an object (instance) of that class.

2. Why do I need a controller if I can define CRDs? Defining a CRD merely extends the Kubernetes API to store custom data. It doesn't inherently give that data any operational meaning or behavior. A controller is the active component that "watches" for changes to instances of your custom resource (CRs) and takes specific actions to reconcile the desired state (defined in the CR's spec) with the actual state of the cluster or external systems. Without a controller, your custom resources would just be inert data objects in the Kubernetes API. The controller brings your custom resources to life by implementing the logic to fulfill their specified intent.

3. What is a "reconcile loop" and why is it important for controllers? A reconcile loop is the core operational pattern of a Kubernetes controller. It's a continuous process where the controller: 1. Observes a specific resource (e.g., your custom resource). 2. Compares the desired state (defined in the resource's spec) with the actual state of the cluster or external world. 3. Acts by taking necessary steps to converge the actual state to the desired state. 4. Updates the resource's status to reflect the current reality. This loop runs continuously, making controllers inherently self-healing and robust. If an external system goes down or a dependent resource is accidentally deleted, the controller will eventually detect the divergence and attempt to reconcile the state again.

4. How do I make my controller highly available and prevent multiple instances from acting simultaneously? To ensure high availability, you typically deploy multiple replicas of your controller. However, to prevent conflicts (e.g., two controllers trying to provision the same external resource), only one instance should be actively performing reconciliation at any given time. This is achieved through leader election. Kubernetes provides a built-in mechanism for leader election, often backed by etcd or by using ConfigMaps or Leases. Frameworks like controller-runtime (used in this guide) offer straightforward ways to enable leader election, ensuring that only the elected leader controller performs the actual work, while other replicas remain ready to take over if the leader fails.

5. What is the role of finalizers in controller development? Finalizers are special keys in a Kubernetes object's metadata.finalizers list that prevent an object from being fully deleted until those finalizers are removed. They are crucial for controllers that manage external resources (e.g., an S3 bucket, a cloud database instance) or need to perform cleanup actions before a custom resource is permanently removed from Kubernetes. When a user requests to delete an object with finalizers, Kubernetes sets its deletionTimestamp but doesn't remove it. The controller watching the object then detects the deletionTimestamp, performs its necessary cleanup, and finally removes the finalizer. Only when all finalizers are removed can Kubernetes proceed with the actual deletion of the object from etcd.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.