OpenShift Operator

In the first part of this series, we saw how effective a platform as a service (PaaS) such as Red Hat OpenShift is for developing IoT edge applications and distributing them to remote sites, thanks to containers and Red Hat Ansible Automation technologies.

Usually, we think about IoT applications as something specially designed for low power devices with limited capabilities.  IoT devices might use a different CPU architectures or platform. For this reason, we tend to use completely different technologies for IoT application development than for services that run in a data center.

In part two, we explore some techniques that allow you to build and test contains for alternate architectures such as ARM64 on an x86_64 host.  The goal we are working towards is to enable you to use the same language, framework, and development tools for code that runs in your datacenter or all the way out to IoT edge devices. In this article, I'll show building and running an AArch64 container image on an x86_64 host and then building an RPI3 image to run it on physical hardware using Fedora and Podman.

In the previous article, I assumed as a prerequisite that an IoT gateway may be capable of running x86_64 containers, but unfortunately, this is not a common use case due to the various type of IoT gateways available today on the market.

I discovered a very interesting project named “multiarch” with multiple repositories available on GitHub.

The aim of the project is to create a handy way for using a built-in Linux kernel feature named binfmt_misc, which is explained in Wikipedia as follows: “binfmt_misc is a capability of the Linux kernel which allows arbitrary executable file formats to be recognized and passed to certain user space applications, such as emulators and virtual machines. It is one of a number of binary format handlers in the kernel that are involved in preparing a user-space program to run.

The concepts behind the multiarch project are really simple: Imagine launching a privileged container that is able to interact with the containers’ host and use the binfmt_misc feature for informing the kernel that some other binary’s handlers are available somewhere in the PATH.

Are you guessing about the handlers? They are just QEMU x86_64 executables capable of running the specific architecture binaries: ARM, MIPS, Power, etc.

Keep in mind that QEMU is a generic and open source machine emulator and virtualizer, so it’s the right candidate for this job. It is also used for running KVM virtual machines with near-native performance.

At this point, the only thing you have to do is to place the right QEMU binaries in the containers with a different architecture. Why must they be placed in the containers?

The QEMU executables must be placed in the containers because when the container engine tries to execute some other ARCH binaries, it will trigger in the kernel the binfmt_misc feature, which will then redirect this execution to the binary in the path we specified. Due to the fact that we’re in the container's virtual root filesystem, the QEMU executable must reside in the same environment of the binary that just ran.

The feature activation is really simple. As the multiarch project page states, we need only ensure that multiarch/qemu-user-static:register with the --reset option is used.

The project page suggests using Docker for applying this action and, of course, this feature will be lost on the next machine’s restart, but we can set it as a one-shot systemd service.

For this article, we’re just using a Minishift installation. For this reason, due to the lack of persistence of Minishift’s base operating system, we’ll just run the Docker command once we have logged in:

[alex@lenny ~]$ cdk-minishift ssh
[docker@minishift ~]$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel

As you saw in the previous command, we just registered for the current x86_64 host a bunch of handlers that will receive specific requests for different architectures’ instructions from the host’s kernel.

We’re now ready to test a container build project with an architecture other than x86_64.

For this reason, I prepared a simple test project for building an ARM 64-bit container image and using it with a Raspberry Pi 3: it’s just a web server.

As you’ll see in the project page, the git repo contains just a Dockerfile:

FROM multiarch/debian-debootstrap:arm64-stretch-slim

RUN apt-get update
RUN apt-get install -y apache2
RUN sed -i 's/80/8080/g' /etc/apache2/ports.conf


CMD ["/usr/sbin/apache2ctl", "-DFOREGROUND"]

It starts from a Debian base image with an ARM64 architecture. It updates the APT repos and installs a web server. After that, it replaces the default listening port and then it sets the right init command.

I bet you’re asking, "Where is the magic?"

Well, the magic happens thanks to /usr/bin/qemu-aarch64-static, which is available in the container image itself. This binary is an x86_64 binary, different from all the others that are AArch64. The Linux Kernel will forward AArch64 binaries executions to this handler!

We’re now ready to create the project and the OpenShift resources for handling the container image’s build:

[alex@lenny ~]$ oc new-project test-arm-project
Now using project "test-arm-project" on server "".

[alex@lenny ~]$ oc new-app
--> Found Docker image 4734ae4 (3 days old) from Docker Hub for "multiarch/debian-debootstrap:arm64-stretch-slim"

    * An image stream tag will be created as "debian-debootstrap:arm64-stretch-slim" that will track the source image
    * A Docker build using source code from will be created
      * The resulting image will be pushed to image stream tag "test-arm-container:latest"
      * Every time "debian-debootstrap:arm64-stretch-slim" changes a new build will be triggered
    * This image will be deployed in deployment config "test-arm-container"
    * Port 8080/tcp will be load balanced by service "test-arm-container"
      * Other containers can access this service through the hostname "test-arm-container"
    * WARNING: Image "multiarch/debian-debootstrap:arm64-stretch-slim" runs as the 'root' user which may not be permitted by your cluster administrator

--> Creating resources ... "debian-debootstrap" created "test-arm-container" created "test-arm-container" created "test-arm-container" created
    service "test-arm-container" created
--> Success
    Build scheduled, use 'oc logs -f bc/test-arm-container' to track its progress.
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/test-arm-container' 
    Run 'oc status' to view your app.

Looking over the OpenShift web interface, we can see that the container image was successfully built and it’s also running:

The running container image

We can even access the running container and try to execute some commands:

Executing some commands

As you can see from the previous command, we attached to the running container and we verified that every command is proxied through /usr/bin/qemu-aarch64-static.

We also checked that the binaries are truly the AArch64 architecture.

We can now try the just-built container image on a Raspberry Pi 3. I chose as the base operating system Fedora ARM Linux.

First, I set up one SD card with an easy-to-use tool after I downloaded the image from Fedora’s website:

[alex@lenny Downloads]$ sudo arm-image-installer --addkey=/home/alex/.ssh/ --image=/home/alex/Downloads/Fedora-Minimal-29-1.2.aarch64.raw.xz --relabel --resizefs --norootpass --target=rpi3 --media=/dev/sda
[sudo] password for alex: 
 ** WARNING: You have requested the image be written to sda.
 ** /dev/sda is usually the root filesystem of the host. 
 ** Do you wish to continue? (type 'yes' to continue)
 = Continue? yes

= Selected Image:                                 
= /home/alex/Downloads/Fedora-Minimal-29-1.2.aarch64.raw.xz
= Selected Media : /dev/sda
= U-Boot Target : rpi3
= SELinux relabel will be completed on first boot.
= Root Password will be removed.
= Root partition will be resized
= SSH Public Key /home/alex/.ssh/ will be added.
= Raspberry Pi 3 Uboot is already in place, no changes needed.
= Removing the root password.
= Adding SSH key to authorized keys.
= Touch /.autorelabel on rootfs.

= Installation Complete! Insert into the rpi3 and boot.

While our brand new Raspberry Pi 3 operating system will boots, we can export the Docker image from the running Minishift virtual machine.

We’ll connect directly to the VM and run a docker save command. We can use this trick because we’re in a demo environment; in a real use-case scenario, we may export the internal OpenShift registry to let external devices connect.

[alex@lenny ~]$ cdk-minishift ssh
Last login: Thu Jan 24 19:15:36 2019 from

[docker@minishift ~]$ docker save -o test-arm-container.tar

[alex@lenny Downloads]$ scp -i ~/.minishift/machines/minishift/id_rsa docker@`cdk-minishift ip`/home/docker/test-arm-container.tar

We can then push the image to the Raspberry Pi and install Podman for running it!

Don’t know what Podman is? Read more about Podman, which is in Red Hat Enterprise Linux.

[alex@lenny Downloads]$ scp test-arm-container.tar root@                              100% 214MB 450.1KB/s 08:07 

[alex@lenny Downloads]$ ssh root@

[root@localhost ~]# cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.19.15-300.fc29.aarch64 #1 SMP Mon Jan 14 16:22:13 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

[root@localhost ~]# dnf install -y podman

Then we load the container image and finally run it:

[root@localhost ~]# podman load -i test-arm-container.tar 
Getting image source signatures
Copying blob 36a049148cc6: 104.31 MiB / 104.31 MiB [=====================] 1m34s
Copying blob d56ce20a3f9c: 15.20 MiB / 15.20 MiB [=======================] 1m34s
Copying blob cf01d69beeaf: 94.53 MiB / 94.53 MiB [=======================] 1m34s
Copying blob 115c696bd46d: 3.00 KiB / 3.00 KiB [=========================] 1m34s
Copying config 478a2361357e: 5.46 KiB / 5.46 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
Loaded image(s):

[root@localhost ~]# podman images
REPOSITORY                                       TAG IMAGE ID CREATED SIZE   latest 478a2361357e 4 hours ago 224 MB

[root@localhost ~]# podman run -d

[root@localhost ~]# podman ps
CONTAINER ID  IMAGE                                         COMMAND CREATED STATUS PORTS  NAMES
cdbf0ac43a2d  /usr/sbin/apache2... 6 seconds ago Up 5 seconds ago       pedantic_agnesi

We can then test the web server:

[root@localhost ~]# podman inspect cdbf0ac43a2d | grep -i address
            "LinkLocalIPv6Address": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "GlobalIPv6Address": "",
            "IPAddress": "",
            "MacAddress": "1a:c8:30:a4:be:2f"

[root@localhost ~]# curl 2>/dev/null | head
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Apache2 Debian Default Page: It works</title>
    <style type="text/css" media="screen">
  * {
    margin: 0px 0px 0px 0px;
    padding: 0px 0px 0px 0px;

Finally, let’s also check the content and compare the binaries in the container with ones in the Raspberry Pi:

[root@localhost ~]# podman run -ti /bin/bash

root@8e39d1c28259:/# id
uid=0(root) gid=0(root) groups=0(root)

root@8e39d1c28259:/# uname -a
Linux 8e39d1c28259 4.19.15-300.fc29.aarch64 #1 SMP Mon Jan 14 16:22:13 UTC 2019 aarch64 GNU/Linux

root@8e39d1c28259:/# cat /etc/debian_version 

root@8e39d1c28259:/# file /bin/bash

/bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, for GNU/Linux 3.7.0, BuildID[sha1]=29b2624b1e147904a979d91daebc60c27ac08dc6, stripped

root@8e39d1c28259:/# exit

[root@localhost ~]# file /bin/bash

/bin/bash: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, for GNU/Linux 3.7.0, BuildID[sha1]=25dee020ab8f6525bd244fb3f9082a47e940b1e6, stripped, too many notes (256)

We just saw that we can run this container with Podman. So why not apply some techniques we saw in past articles about Podman and its integration with systemd? :)

Read more about Podman and managing containerized system services.

Additional resources

That's all. I hope you enjoyed this IoT solution!

About Alessandro

Alessandro Arrichiello

Alessandro Arrichiello is a Solution Architect for Red Hat Inc. He has a passion for GNU/Linux systems, which began at age 14 and continues today. He has worked with tools for automating enterprise IT: configuration management and continuous integration through virtual platforms. He’s now working on distributed cloud environments involving PaaS (OpenShift), IaaS (OpenStack) and Processes Management (CloudForms), Containers building, instances creation, HA services management, and workflow builds.

Last updated: March 27, 2023