Minikube has a feature called add-ons, which help in adding extra components and features to Minikube’s Kubernetes cluster.
The registry add-on will deploy an internal registry, which can then be used to push and pull Linux container images. But at times, we might wish to mimic push and pull to different registries (i.e., using aliases for container registry). In this article, I will walk you through the steps required to achieve the same.
What do we need?
What we will do?
As part of this exercise we will:
- Enable registry via Minikube add-on.
- Update the Minikube node’s etc/hosts for the domains dev.local, example.com to resolve to the internal registry.
- Update CoreDNS to rules that will allow pods to push images( typical case of CI/CD) to the registry using aliases.
Deploy container registry
As described previously, we can use Minikube add-ons to deploy and enable the internal registry. The internal registry by default gets deployed in kube-system namespace.
minikube profile demo minikube start -p demo --memory=8192 --cpus=6 --disk-size=50g
Note: If you like to use cri-o, then adjust the above command to be like:
minikube profile demo minikube start -p demo --memory=8192 --cpus=6 --container-runtime=crio
With Minikube running, the next step is to deploy the registry.
minikube addons enable registry
Once the registry is enabled, you will see a registry pod kubectl -n kube-system get pod and a corresponding service kubectl -n kube-system get svc in the kube-system namespace.
The minkube-helper repo in GitHub has the sources along with the example application to test the configuration. Clone the sources and navigate to the registry sub-folder. For easier reference, we will call the sources folder as $PROJECT_HOME:
git clone https://github.com/kameshsampath/minikube-helpers && \ cd minikube-helpers/registry
Create aliases ConfigMap
The alias names that we want to use for the registry are configured via the ConfigMap, called registry-aliases:
apiVersion: v1
data:
# Add additonal hosts seperated by new-line
registryAliases: >-
dev.local
example.com
# default registry address in minikube when enabled via minikube addons enable registry
registrySvc: registry.kube-system.svc.cluster.local
kind: ConfigMap
metadata:
name: registry-aliases
namespace: kube-system
kubectl apply -f registry-aliases-config.yaml
Update Minikube /etc/hosts file
To make the aliases resolve to the registry service in kube-system namespace, we need to add the aliases entries in the Minkube VM’s /etc/hosts file. We will use DaemonSet to update the etc/hosts file inside the Minikube VM.
kubectl apply -f node-etc-hosts-update.yaml
As it will take few minutes for the DaemonSet to be running, you can watch the status using the command:
kubectl -n kube-system get pods --watch
Once the DaemonSet is successfully running, you can check the Minkube VM's /etc/hosts file, which will now be updated to point to CLUSTER-IP of the registry service.
minikube ssh -- cat /etc/hosts
Tips
- You can use CTRL+C to terminate the watch.
- You can check the CLUSTER-IP of the
registryservice using the commandkubectl -n kube-system get svc registry -o jsonpath='{.spec.clusterIP}'.
Patch CoreDNS
The configurations and other settings we applied in the previous section are good enough for the container runtime to push and pull the images. A typical CI/CD scenario will be something like a Kubernetes pod doing a build, e.g., Jenkins, Tekton and pushing the container image to the registry post as part of the pipeline.
To make the pod resolve the aliases like dev.local, example.com, we need to have the CoreDNS rules configured. For our alias configuration to work, we will use the CoreDNS rewrite rules.
Running the following command will have the CoreDNS patched with the rewrite rules.
./patch-coredns.sh
The CoreDNS patch can be queried to see the updates. A successful update will show output for the command kubectl -n kube-sytem configmap coredns -oyaml as shown below:
apiVersion: v1
data:
Corefile: |-
.:53 {
errors
health
rewrite name dev.local registry.kube-system.svc.cluster.local
rewrite name example.com registry.kube-system.svc.cluster.local
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
Test the configuration
I found the real need of this registry hack was when I was trying to deploy Tekton pipelines. Tekton is Kubernetes' native way of declaring CI/CD pipelines.
As part of my pipeline (see example), I want to build and deploy a simple Hello World application.
Deploy Tekton pipelines
kubectl apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml
The status of Tekton pipelines can be watched using:
kubectl get pods --namespace tekton-pipelines -w
Deploy application pipeline
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/build.yaml \ --filename example/build.yaml
As it will take some time for the pipeline to complete, you can watch the status using the command:
tkn taskrun logs -f -a hello-world
A successful pipeline build will have the Hello World application deployed. You can use the following Minikube shortcut:
curl $(minikube service helloworld --url)
to call the service, which returns a “Hello World” as the response.
Note: When you see tkn logs -f -a hello-world showing a blank screen, it might be that it's pulling the required images. To know what's happening, you can use kubectl get events -w.
I hope this article will help with similar development environment use cases that you might have. Next time, we will take a deep dive into Tekton. Until then, happy Kubernetes hacking!
Last updated: January 5, 2022