Deploying an internal container registry with Minikube add-ons

Minikube has a feature called add-ons, which help in adding extra components and features to Minikube’s Kubernetes cluster.

The registry add-on will deploy an internal registry, which can then be used to push and pull Linux container images. But at times, we might wish to mimic push and pull to different registries (i.e., using aliases for container registry). In this article, I will walk you through the steps required to achieve the same.

What do we need?

• Minikube, preferred version is v1.1.1
• Git to clone the sources
• kubectl
• kubens
• tkn CLI

What we will do?

As part of this exercise we will:

• Enable registry via Minikube add-on.
• Update the Minikube node’s etc/hosts for the domains dev.local, example.com to resolve to the internal registry.
• Update CoreDNS to rules that will allow pods to push images( typical case of CI/CD) to the registry using aliases.

Deploy container registry

As described previously, we can use Minikube add-ons to deploy and enable the internal registry. The internal registry by default gets deployed in kube-system namespace.

minikube profile demo
minikube start -p demo --memory=8192 --cpus=6 --disk-size=50g

Note: If you like to use cri-o, then adjust the above command to be like:

minikube profile demo
minikube start -p demo --memory=8192 --cpus=6 --container-runtime=crio

With Minikube running, the next step is to deploy the registry.

minikube addons enable registry

Once the registry is enabled, you will see a registry pod kubectl -n kube-system get pod and a corresponding service kubectl -n kube-system get svc in the kube-system namespace.

The minkube-helper repo in GitHub has the sources along with the example application to test the configuration. Clone the sources and navigate to the registry sub-folder. For easier reference, we will call the sources folder as $PROJECT_HOME:

git clone https://github.com/kameshsampath/minikube-helpers && \
cd minikube-helpers/registry

Create aliases ConfigMap

The alias names that we want to use for the registry are configured via the ConfigMap, called registry-aliases:

apiVersion: v1
data:
  # Add additonal hosts seperated by new-line
  registryAliases: >-
    dev.local
    example.com
  # default registry address in minikube when enabled via minikube addons enable registry
  registrySvc: registry.kube-system.svc.cluster.local
kind: ConfigMap
metadata:
  name: registry-aliases
  namespace: kube-system

kubectl apply -f registry-aliases-config.yaml

Update Minikube /etc/hosts file

To make the aliases resolve to the registry service in kube-system namespace, we need to add the aliases entries in the Minkube VM’s /etc/hosts file. We will use DaemonSet to update the etc/hosts file inside the Minikube VM.

kubectl apply -f node-etc-hosts-update.yaml

As it will take few minutes for the DaemonSet to be running, you can watch the status using the command:

kubectl -n kube-system get pods --watch

Once the DaemonSet is successfully running, you can check the Minkube VM's /etc/hosts file, which will now be updated to point to CLUSTER-IP of the registry service.

minikube ssh -- cat /etc/hosts

Tips

• You can use CTRL+C to terminate the watch.
• You can check the CLUSTER-IP of the registry service using the command kubectl -n kube-system get svc registry -o jsonpath='{.spec.clusterIP}'.

Patch CoreDNS

The configurations and other settings we applied in the previous section are good enough for the container runtime to push and pull the images. A typical CI/CD scenario will be something like a Kubernetes pod doing a build, e.g., Jenkins, Tekton and pushing the container image to the registry post as part of the pipeline.

To make the pod resolve the aliases like dev.local, example.com, we need to have the CoreDNS rules configured. For our alias configuration to work, we will use the CoreDNS rewrite rules.

Running the following command will have the CoreDNS patched with the rewrite rules.

./patch-coredns.sh

The CoreDNS patch can be queried to see the updates. A successful update will show output for the command kubectl -n kube-sytem configmap coredns -oyaml as shown below:

apiVersion: v1
data:
  Corefile: |-
    .:53 {
        errors
        health
    rewrite name dev.local  registry.kube-system.svc.cluster.local
    rewrite name example.com  registry.kube-system.svc.cluster.local
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system

Test the configuration

I found the real need of this registry hack was when I was trying to deploy Tekton pipelines. Tekton is Kubernetes' native way of declaring CI/CD pipelines.

As part of my pipeline (see example), I want to build and deploy a simple Hello World application.

Deploy Tekton pipelines

kubectl apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml

The status of Tekton pipelines can be watched using:

kubectl get pods --namespace tekton-pipelines -w

Deploy application pipeline

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/build.yaml \
  --filename example/build.yaml

As it will take some time for the pipeline to complete, you can watch the status using the command:

tkn taskrun logs -f -a hello-world

A successful pipeline build will have the Hello World application deployed. You can use the following Minikube shortcut:

curl $(minikube service helloworld --url)

to call the service, which returns a “Hello World” as the response.

Note: When you see tkn logs -f -a hello-world showing a blank screen, it might be that it's pulling the required images. To know what's happening, you can use kubectl get events -w.

I hope this article will help with similar development environment use cases that you might have. Next time, we will take a deep dive into Tekton. Until then, happy Kubernetes hacking!