Write a simple Kubernetes Operator in Java using the Fabric8 Kubernetes Client

By Rohan Kumar October 7, 2019February 9, 2021

Write a simple Kubernetes Operator in Java using the Fabric8 Kubernetes Client

Kubernetes is becoming much more than just a platform for running container workloads. Its API can be extended with application-specific Custom Resource Definitions(CRDs), and you can implement your own logic adapting your applications dynamically to changes in the cluster. In this article, we’ll be writing a simple Kubernetes Operator in Java using the Fabric8 Kubernetes Client.

What is a Kubernetes Operator?

Kubernetes Operators are software extensions to Kubernetes that make use of Custom Resources to manage applications and their components. They let you extend the cluster’s behavior without modifying the code of Kubernetes itself. They are just clients of the Kubernetes API, which act as controllers of that Custom Resource.

In simple terms, a Kubernetes Operator is code that makes use of the Kubernetes API to execute some tasks; Custom Resources act as a configuration model on which that specific code acts.

Writing a simple PodSet Operator in Java

PodSet Custom Resource

We’ll be writing a very simple operator that tries to do something similar to a ReplicaSet. All the code is hosted on GitHub. It tries to main exactly x amount of pods as with it as a parent. For that, we’ll use a very simple Custom Resource called PodSet. Here is its custom resource definition:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: podsets.demo.fabric8.io
spec:
  group: demo.fabric8.io
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
      subresources:
        status: {}
  names:
    kind: PodSet
    plural: podsets
    singular: podset
    shortNames:
      - ps
  scope: Namespaced

With this Custom Resource Definition applied, A simple PodSet resource can be written as:

apiVersion: demo.fabric8.io/v1alpha1
kind: PodSet
metadata:
  name: example-podset
spec:
  replicas: 5

Writing Operator

Now let’s jump to writing the operator and have a look at the project structure:

podsetoperatorinjava : $ tree .    
.
├── license.txt
├── podset-operator-in-java.iml
├── pom.xml
├── README.md
└── src
    ├── main
    │   ├── java
    │   │   └── io
    │   │       └── fabric8
    │   │           └── podset
    │   │               └── operator
    │   │                   ├── controller
    │   │                   │   └── PodSetController.java
    │   │                   ├── crd
    │   │                   │   ├── PodSet.java
    │   │                   │   ├── PodSetList.java
    │   │                   │   ├── PodSetSpec.java
    │   │                   │   └── PodSetStatus.java
    │   │                   ├── PodSetOperatorMain.java
    │   │                   └── util
    │   │                       └── DeepCopy.java
    │   └── resources
    │       ├── crd.yaml
    │       ├── cr.yaml
    │       └── second-cr.yml

From the structure, we can see three things:

PodSet, PodSetList, PodSetSpec, PodSetStatus, DoneablePodSet as model classes for PodSet custom resources, which are required for deserializing Kubernetes API responses into objects.
PodSetOperatorMain, which is the main driver class of the project.
PodSetController Class, which contains main Kubernetes logic related to the operator.

Let’s look at each of these in detail.

First, to interact with Kubernetes API in Java, we need the Fabric8 Kubernetes Client. It’s one of the most popular Java APIs for interacting with Kubernetes. We need to add that in pom.xml.

Here is how our pom.xml would look after adding Fabric8 client as a dependency:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>io.fabric8</groupId>
    <artifactId>podset-operator-in-java</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <version.fabric8.client>5.0.1</version.fabric8.client>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven-assembly-plugin.version>3.3.0</maven-assembly-plugin.version>
        <junit-jupiter-engine.version>5.6.2</junit-jupiter-engine.version>
        <maven-surefire-plugin.version>3.0.0-M4</maven-surefire-plugin.version>
        <exec-maven-plugin.version>3.0.0</exec-maven-plugin.version>
        <jkube.version>1.1.0</jkube.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-client</artifactId>
            <version>${version.fabric8.client}</version>
        </dependency>

        <dependency>
            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-server-mock</artifactId>
            <version>${version.fabric8.client}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit-jupiter-engine.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit-jupiter-engine.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-migrationsupport</artifactId>
            <version>${junit-jupiter-engine.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>${maven-assembly-plugin.version}</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifest>
                                    <mainClass>
                                        io.fabric8.podset.operator.PodSetOperatorMain
                                    </mainClass>
                                </manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.eclipse.jkube</groupId>
                <artifactId>kubernetes-maven-plugin</artifactId>
                <version>${jkube.version}</version>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>${exec-maven-plugin.version}</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <!-- JUnit 5 requires Surefire version 2.22.0 or higher -->
                <version>${maven-surefire-plugin.version}</version>
            </plugin>
        </plugins>
    </build>

</project>

Once all the necessary dependencies are added, (which, in our case, is just one 🙂 ), we can then write POJOs, which would be needed by our Java client to create a client for our PodSet custom resource. Here’s how some of them would look after adding them to the project:

PodSet.java

package io.fabric8.podset.operator.model.v1alpha1;

import io.fabric8.kubernetes.api.model.Namespaced;
import io.fabric8.kubernetes.client.CustomResource;
import io.fabric8.kubernetes.model.annotation.Group;
import io.fabric8.kubernetes.model.annotation.Version;

@Version("v1alpha1")
@Group("demo.fabric8.io")
public class PodSet extends CustomResource<PodSetSpec, PodSetStatus> implements Namespaced { }

PodSetList.java:

package io.fabric8.podset.operator.model.v1alpha1;

import io.fabric8.kubernetes.client.CustomResourceList;

public class PodSetList extends CustomResourceList<PodSet> { }

PodSetSpec.java:

package io.fabric8.podset.operator.model.v1alpha1;

public class PodSetSpec {
    public int getReplicas() {
        return replicas;
    }

    @Override
    public String toString() {
        return "PodSetSpec{replicas=" + replicas + "}";
    }

    public void setReplicas(int replicas) {
        this.replicas = replicas;
    }

    private int replicas;
}

Once we have added the model classes, we can go ahead and begin writing our operator. Let’s start with the main driving class (i.e., PodSetOperatorMain), which would call all the necessary functions in the PodSetController.

First, we need to initialize informers for both Pod resource and PodSet resource, because we’ll be listening to events related to these resources and reacting upon them in our operator. To stay informed about when these events get triggered, we’ll use a primitive exposed by Kubernetes and the client-go (now added in Fabric8 Kubernetes Java Client) called SharedInformer. Let’s see how it works:

We can initialize the informer factory like this:

SharedInformerFactory informerFactory = client.informers();

To get informer for Pod resource, we need to pass classes of Pod, PodList and resync period (which is basically the interval after which informer should set up the connection again).

SharedIndexInformer<Pod> podSharedIndexInformer = informerFactory.sharedIndexInformerFor(Pod.class, PodList.class, 10 * 60 * 1000);

Because PodSet is a custom resource, we need to pass additional information while creating its informer. It’s not much, just small details related to its custom resource definition (CRD).

CustomResourceDefinitionContext podSetCustomResourceDefinitionContext = new CustomResourceDefinitionContext.Builder()
                    .withVersion("v1alpha1")
                    .withScope("Namespaced")
                    .withGroup("demo.fabric8.io")
                    .withPlural("podsets")
                    .build();
SharedIndexInformer<PodSet> podSetSharedIndexInformer = informerFactory.sharedIndexInformerForCustomResource(podSetCustomResourceDefinitionContext, PodSet.class, PodSetList.class, 10 * 60 * 1000);

Once we have informers for both Pod and PodSet resource, then we need to pass all these into PodSetController(would be discussed after this). After that, we would initialize the controller and run it. Now our PodSetOperatorMain would look like this:

try (KubernetesClient client = new DefaultKubernetesClient()) {
    String namespace = client.getNamespace();
    if (namespace == null) {
        logger.log(Level.INFO, "No namespace found via config, assuming default.");
        namespace = "default";
    }

    logger.log(Level.INFO, "Using namespace : " + namespace);
    CustomResourceDefinitionContext podSetCustomResourceDefinitionContext = new CustomResourceDefinitionContext.Builder()
            .withVersion("v1alpha1")
            .withScope("Namespaced")
            .withGroup("demo.fabric8.io")
            .withPlural("podsets")
            .build();

    SharedInformerFactory informerFactory = client.informers();

    MixedOperation<PodSet, PodSetList, Resource<PodSet>> podSetClient = client.customResources(PodSet.class, PodSetList.class);
    SharedIndexInformer<Pod> podSharedIndexInformer = informerFactory.sharedIndexInformerFor(Pod.class, PodList.class, 10 * 60 * 1000);
    SharedIndexInformer<PodSet> podSetSharedIndexInformer = informerFactory.sharedIndexInformerForCustomResource(podSetCustomResourceDefinitionContext, PodSet.class, PodSetList.class, 10 * 60 * 1000);
    PodSetController podSetController = new PodSetController(client, podSetClient, podSharedIndexInformer, podSetSharedIndexInformer, namespace);

    podSetController.create();
    informerFactory.startAllRegisteredInformers();
    informerFactory.addSharedInformerEventListener(exception -> logger.log(Level.SEVERE, "Exception occurred, but caught", exception));

    podSetController.run();
} catch (KubernetesClientException exception) {
    logger.log(Level.SEVERE, "Kubernetes Client Exception : " + exception.getMessage());
}

Now we need to take a look at PodSetController, which contains the main logic for interacting with Kubernetes APIs and making the desired changes as per the events received. Let’s first look at its create() method:

public void create() {
    podSetInformer.addEventHandler(new ResourceEventHandler<PodSet>() {
        @Override
        public void onAdd(PodSet podSet) {
            enqueuePodSet(podSet);
        }

        @Override
        public void onUpdate(PodSet podSet, PodSet newPodSet) {
            enqueuePodSet(newPodSet);
        }

        @Override
        public void onDelete(PodSet podSet, boolean b) {
            // Do nothing
        }
    });

    podInformer.addEventHandler(new ResourceEventHandler<Pod>() {
        @Override
        public void onAdd(Pod pod) {
            handlePodObject(pod);
        }

        @Override
        public void onUpdate(Pod oldPod, Pod newPod) {
            if (oldPod.getMetadata().getResourceVersion().equals(newPod.getMetadata().getResourceVersion())) {
                return;
            }
            handlePodObject(newPod);
        }

        @Override
        public void onDelete(Pod pod, boolean b) {
            // Do nothing
        }
    });
}

As we can see it’s adding event handlers for both informers: Pod and PodSet. In the case of PodSet, we’re adding it to the work queue of the operator to process it afterward. In the case of Pod, we’re checking whether that Pod resource is related to our PodSet custom resource or not, then only we’re trying to handle it (i.e., getting its owner from the metadata and checking the state of owner again).

Let’s look at run() method of our controller; it is just de-queuing item from the work queue and passing it to reconcile() method, which does the main work. Here is the body of reconcile() method:

/**
 * Tries to achieve the desired state for podset.
 *
 * @param podSet specified podset
 */
protected void reconcile(PodSet podSet) {
    List<String> pods = podCountByLabel(APP_LABEL, podSet.getMetadata().getName());
    if (pods.isEmpty()) {
        createPods(podSet.getSpec().getReplicas(), podSet);
        return;
    }
    int existingPods = pods.size();

    // Compare it with desired state i.e spec.replicas
    // if less then spin up pods
    if (existingPods < podSet.getSpec().getReplicas()) {
        createPods(podSet.getSpec().getReplicas() - existingPods, podSet);
    }

    // If more pods then delete the pods
    int diff = existingPods - podSet.getSpec().getReplicas();
    for (; diff > 0; diff--) {
        String podName = pods.remove(0);
        kubernetesClient.pods().inNamespace(podSet.getMetadata().getNamespace()).withName(podName).delete();
    }

    // Update PodSet status
    updateAvailableReplicasInPodSetStatus(podSet, podSet.getSpec().getReplicas());
}

This reconcile() method seems to be doing the following:

It receives a PodSet object as an argument for which it tries to do reconciliation. Then, it tries to list all the pods in the cluster with the label app=<name of PodSet>; let’s say it’s app=example-podset.
Once the list operation of all pods with the label app=example-podset is fetched, it checks whether the number of pods is equal to PodSet.spec.replicas. if the number of pods is less, it spins more pods into the cluster.
If the number of pods is more than the desired value, it tries to delete pods from the cluster.

Running your Operator

We have covered most of the important portions of PodSetController, so let’s try to run our operator on a Kubernetes cluster. I am using Minikube for running this, which you can get from their releases page.

# Make sure PodSet CustomResourceDefinition is already applied onto the cluster. If not, just apply it using this command
kubectl apply -f src/main/resources/crd.yaml

# Build project
mvn clean install

# Run project
mvn exec:java -Dexec.mainClass=io.fabric8.podset.operator.PodSetOperatorMain

When you run this, you should be able to see pods getting created upon creating a PodSet custom resource as shown in this gif below:

And that’s it! If the above steps worked for you, congratulations! You have successfully written an operator in Java.

Running your Operator as a Pod in Kubernetes:

We will be using Eclipse JKube‘s Kubernetes Maven Plugin for containerizing our operator into an image and deploying it to Kubernetes. Before deploying the operator to Kubernetes, we need to configure the permissions of ServiceAccount our application will be using. Since our application is just going to use the default ServiceAccount, I’ll just give Cluster Admin privileges to the default `ServiceAccount` using this command:

kubectl create clusterrolebinding default-pod --clusterrole cluster-admin --serviceaccount=default:default

For this blog, I’ll be demonstrating deployment to Kubernetes on minikube. You can easily build a docker image against minikube’s docker daemon and then deploy the operator like this:

podsetoperatorinjava : $ eval $(minikube -p minikube docker-env)
podsetoperatorinjava : $ mvn k8s:build k8s:resource k8s:apply

If you’re not trying this out on minikube and need to push the image to some external registry(Docker Hub or Quay.io). You would need to use the k8s:push goal from Eclipse JKube. Before that you would need to configure kubernetes-maven-plugin to create image according to registry name and your username. You can add these properties to the project:

<properties>
      <image.user>rohankanojia</image.user>
      <jkube.generator.name>quay.io/${image.user}/${project.artifactId}:${project.version}</jkube.generator.name>
       <!-- Rest of properties -->
</properties>

Then you can issue mvn k8s:build k8s:push to build and push the image to the specified registry.

See more

GitHub repository: https://github.com/rohanKanojia/podsetoperatorinjava

Blog Articles