Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

IoT edge development and deployment with containers through OpenShift: Part 2

 

February 5, 2019
Alessandro Arrichiello
Related topics:
CI/CDKubernetes
Related products:
Red Hat OpenShift

Share:

    In the first part of this series, we saw how effective a platform as a service (PaaS) such as Red Hat OpenShift is for developing IoT edge applications and distributing them to remote sites, thanks to containers and Red Hat Ansible Automation technologies.

    Usually, we think about IoT applications as something specially designed for low power devices with limited capabilities.  IoT devices might use a different CPU architectures or platform. For this reason, we tend to use completely different technologies for IoT application development than for services that run in a data center.

    In part two, we explore some techniques that allow you to build and test contains for alternate architectures such as ARM64 on an x86_64 host.  The goal we are working towards is to enable you to use the same language, framework, and development tools for code that runs in your datacenter or all the way out to IoT edge devices. In this article, I'll show building and running an AArch64 container image on an x86_64 host and then building an RPI3 image to run it on physical hardware using Fedora and Podman.

    In the previous article, I assumed as a prerequisite that an IoT gateway may be capable of running x86_64 containers, but unfortunately, this is not a common use case due to the various type of IoT gateways available today on the market.

    I discovered a very interesting project named “multiarch” with multiple repositories available on GitHub.

    The aim of the project is to create a handy way for using a built-in Linux kernel feature named binfmt_misc, which is explained in Wikipedia as follows: “binfmt_misc is a capability of the Linux kernel which allows arbitrary executable file formats to be recognized and passed to certain user space applications, such as emulators and virtual machines. It is one of a number of binary format handlers in the kernel that are involved in preparing a user-space program to run.”

    The concepts behind the multiarch project are really simple: Imagine launching a privileged container that is able to interact with the containers’ host and use the binfmt_misc feature for informing the kernel that some other binary’s handlers are available somewhere in the PATH.

    Are you guessing about the handlers? They are just QEMU x86_64 executables capable of running the specific architecture binaries: ARM, MIPS, Power, etc.

    Keep in mind that QEMU is a generic and open source machine emulator and virtualizer, so it’s the right candidate for this job. It is also used for running KVM virtual machines with near-native performance.

    At this point, the only thing you have to do is to place the right QEMU binaries in the containers with a different architecture. Why must they be placed in the containers?

    The QEMU executables must be placed in the containers because when the container engine tries to execute some other ARCH binaries, it will trigger in the kernel the binfmt_misc feature, which will then redirect this execution to the binary in the path we specified. Due to the fact that we’re in the container's virtual root filesystem, the QEMU executable must reside in the same environment of the binary that just ran.

    The feature activation is really simple. As the multiarch project page states, we need only ensure that multiarch/qemu-user-static:register with the --reset option is used.

    The project page suggests using Docker for applying this action and, of course, this feature will be lost on the next machine’s restart, but we can set it as a one-shot systemd service.

    For this article, we’re just using a Minishift installation. For this reason, due to the lack of persistence of Minishift’s base operating system, we’ll just run the Docker command once we have logged in:

    [alex@lenny ~]$ cdk-minishift ssh
    [docker@minishift ~]$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
    Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
    Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
    Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
    Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
    Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
    Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
    Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
    Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
    Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
    Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
    Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
    Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
    Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
    Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
    Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
    Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
    Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
    Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
    Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
    Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
    Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
    Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
    Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
    Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
    Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
    Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel

    As you saw in the previous command, we just registered for the current x86_64 host a bunch of handlers that will receive specific requests for different architectures’ instructions from the host’s kernel.

    We’re now ready to test a container build project with an architecture other than x86_64.

    For this reason, I prepared a simple test project for building an ARM 64-bit container image and using it with a Raspberry Pi 3: it’s just a web server.

    As you’ll see in the project page, the git repo contains just a Dockerfile:

    FROM multiarch/debian-debootstrap:arm64-stretch-slim
    
    RUN apt-get update
    RUN apt-get install -y apache2
    RUN sed -i 's/80/8080/g' /etc/apache2/ports.conf
    
    EXPOSE 8080
    
    CMD ["/usr/sbin/apache2ctl", "-DFOREGROUND"]

    It starts from a Debian base image with an ARM64 architecture. It updates the APT repos and installs a web server. After that, it replaces the default listening port and then it sets the right init command.

    I bet you’re asking, "Where is the magic?"

    Well, the magic happens thanks to /usr/bin/qemu-aarch64-static, which is available in the container image itself. This binary is an x86_64 binary, different from all the others that are AArch64. The Linux Kernel will forward AArch64 binaries executions to this handler!

    We’re now ready to create the project and the OpenShift resources for handling the container image’s build:

    [alex@lenny ~]$ oc new-project test-arm-project
    Now using project "test-arm-project" on server "https://192.168.42.213:8443".
    
    [alex@lenny ~]$ oc new-app https://github.com/alezzandro/test-arm-container
    --> Found Docker image 4734ae4 (3 days old) from Docker Hub for "multiarch/debian-debootstrap:arm64-stretch-slim"
    
        * An image stream tag will be created as "debian-debootstrap:arm64-stretch-slim" that will track the source image
        * A Docker build using source code from https://github.com/alezzandro/test-arm-container will be created
          * The resulting image will be pushed to image stream tag "test-arm-container:latest"
          * Every time "debian-debootstrap:arm64-stretch-slim" changes a new build will be triggered
        * This image will be deployed in deployment config "test-arm-container"
        * Port 8080/tcp will be load balanced by service "test-arm-container"
          * Other containers can access this service through the hostname "test-arm-container"
        * WARNING: Image "multiarch/debian-debootstrap:arm64-stretch-slim" runs as the 'root' user which may not be permitted by your cluster administrator
    
    --> Creating resources ...
        imagestream.image.openshift.io "debian-debootstrap" created
        imagestream.image.openshift.io "test-arm-container" created
        buildconfig.build.openshift.io "test-arm-container" created
        deploymentconfig.apps.openshift.io "test-arm-container" created
        service "test-arm-container" created
    --> Success
        Build scheduled, use 'oc logs -f bc/test-arm-container' to track its progress.
        Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
         'oc expose svc/test-arm-container' 
        Run 'oc status' to view your app.

    Looking over the OpenShift web interface, we can see that the container image was successfully built and it’s also running:

    The running container image

    We can even access the running container and try to execute some commands:

    Executing some commands

    As you can see from the previous command, we attached to the running container and we verified that every command is proxied through /usr/bin/qemu-aarch64-static.

    We also checked that the binaries are truly the AArch64 architecture.

    We can now try the just-built container image on a Raspberry Pi 3. I chose as the base operating system Fedora ARM Linux.

    First, I set up one SD card with an easy-to-use tool after I downloaded the image from Fedora’s website:

    [alex@lenny Downloads]$ sudo arm-image-installer --addkey=/home/alex/.ssh/id_rsa.pub --image=/home/alex/Downloads/Fedora-Minimal-29-1.2.aarch64.raw.xz --relabel --resizefs --norootpass --target=rpi3 --media=/dev/sda
    [sudo] password for alex: 
     ***********************************************************
     ** WARNING: You have requested the image be written to sda.
     ** /dev/sda is usually the root filesystem of the host. 
     ***********************************************************
     ** Do you wish to continue? (type 'yes' to continue)
     ***********************************************************
     = Continue? yes
    
    =====================================================
    = Selected Image:                                 
    = /home/alex/Downloads/Fedora-Minimal-29-1.2.aarch64.raw.xz
    = Selected Media : /dev/sda
    = U-Boot Target : rpi3
    = SELinux relabel will be completed on first boot.
    = Root Password will be removed.
    = Root partition will be resized
    = SSH Public Key /home/alex/.ssh/id_rsa.pub will be added.
    =====================================================
    ...
    = Raspberry Pi 3 Uboot is already in place, no changes needed.
    = Removing the root password.
    = Adding SSH key to authorized keys.
    = Touch /.autorelabel on rootfs.
    
    = Installation Complete! Insert into the rpi3 and boot.

    While our brand new Raspberry Pi 3 operating system will boots, we can export the Docker image from the running Minishift virtual machine.

    We’ll connect directly to the VM and run a docker save command. We can use this trick because we’re in a demo environment; in a real use-case scenario, we may export the internal OpenShift registry to let external devices connect.

    [alex@lenny ~]$ cdk-minishift ssh
    Last login: Thu Jan 24 19:15:36 2019 from 192.168.42.1
    
    [docker@minishift ~]$ docker save -o test-arm-container.tar 172.30.1.1:5000/test-arm-project/test-arm-container:latest
    
    [alex@lenny Downloads]$ scp -i ~/.minishift/machines/minishift/id_rsa docker@`cdk-minishift ip`/home/docker/test-arm-container.tar

    We can then push the image to the Raspberry Pi and install Podman for running it!

    Don’t know what Podman is? Read more about Podman, which is in Red Hat Enterprise Linux.

    [alex@lenny Downloads]$ scp test-arm-container.tar root@192.168.1.52:/root/test-arm-container.tar                              100% 214MB 450.1KB/s 08:07 
    
    [alex@lenny Downloads]$ ssh root@192.168.1.52
    
    [root@localhost ~]# cat /etc/fedora-release 
    Fedora release 29 (Twenty Nine)
    
    [root@localhost ~]# uname -a
    Linux localhost.localdomain 4.19.15-300.fc29.aarch64 #1 SMP Mon Jan 14 16:22:13 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
    
    [root@localhost ~]# dnf install -y podman
    

    Then we load the container image and finally run it:

    [root@localhost ~]# podman load -i test-arm-container.tar 
    Getting image source signatures
    Copying blob 36a049148cc6: 104.31 MiB / 104.31 MiB [=====================] 1m34s
    Copying blob d56ce20a3f9c: 15.20 MiB / 15.20 MiB [=======================] 1m34s
    Copying blob cf01d69beeaf: 94.53 MiB / 94.53 MiB [=======================] 1m34s
    Copying blob 115c696bd46d: 3.00 KiB / 3.00 KiB [=========================] 1m34s
    Copying config 478a2361357e: 5.46 KiB / 5.46 KiB [==========================] 0s
    Writing manifest to image destination
    Storing signatures
    Loaded image(s): 172.30.1.1:5000/arm-project/test-arm-container:latest
    
    [root@localhost ~]# podman images
    REPOSITORY                                       TAG IMAGE ID CREATED SIZE
    172.30.1.1:5000/arm-project/test-arm-container   latest 478a2361357e 4 hours ago 224 MB
    
    [root@localhost ~]# podman run -d 172.30.1.1:5000/arm-project/test-arm-container:latest
    cdbf0ac43a2dd01afd73220d5756060665df0b72a43bd66bf865d1c6149f325f
    
    [root@localhost ~]# podman ps
    CONTAINER ID  IMAGE                                         COMMAND CREATED STATUS PORTS  NAMES
    cdbf0ac43a2d  172.30.1.1:5000/arm-project/test-arm-container:latest  /usr/sbin/apache2... 6 seconds ago Up 5 seconds ago       pedantic_agnesi
    
    

    We can then test the web server:

    [root@localhost ~]# podman inspect cdbf0ac43a2d | grep -i address
                "LinkLocalIPv6Address": "",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "GlobalIPv6Address": "",
                "IPAddress": "10.88.0.7",
                "MacAddress": "1a:c8:30:a4:be:2f"
    
    [root@localhost ~]# curl 10.88.0.7:8080 2>/dev/null | head
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml">
      <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>Apache2 Debian Default Page: It works</title>
        <style type="text/css" media="screen">
      * {
        margin: 0px 0px 0px 0px;
        padding: 0px 0px 0px 0px;

    Finally, let’s also check the content and compare the binaries in the container with ones in the Raspberry Pi:

    [root@localhost ~]# podman run -ti 172.30.1.1:5000/arm-project/test-arm-container:latest /bin/bash
    
    root@8e39d1c28259:/# 
    root@8e39d1c28259:/# id
    uid=0(root) gid=0(root) groups=0(root)
    
    root@8e39d1c28259:/# uname -a
    Linux 8e39d1c28259 4.19.15-300.fc29.aarch64 #1 SMP Mon Jan 14 16:22:13 UTC 2019 aarch64 GNU/Linux
    
    root@8e39d1c28259:/# cat /etc/debian_version 
    9.6
    
    root@8e39d1c28259:/# file /bin/bash
    
    /bin/bash: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=29b2624b1e147904a979d91daebc60c27ac08dc6, stripped
    
    root@8e39d1c28259:/# exit
    
    [root@localhost ~]# file /bin/bash
    
    /bin/bash: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=25dee020ab8f6525bd244fb3f9082a47e940b1e6, stripped, too many notes (256)

    We just saw that we can run this container with Podman. So why not apply some techniques we saw in past articles about Podman and its integration with systemd? :)

    Read more about Podman and managing containerized system services.

    Additional resources

    • IoT edge development and deployment with containers through OpenShift: Part 1
    • Podman: Managing pods and containers in a local container runtime
    • Managing containerized system services with Podman
    • Containers without daemons: Podman and Buildah available in RHEL 7.6 and RHEL 8 Beta
    • Podman – The next generation of Linux container tools
    • Intro to Podman (New in Red Hat Enterprise Linux 7.6)
    • Customizing an OpenShift Ansible Playbook Bundle

    That's all. I hope you enjoyed this IoT solution!

    About Alessandro

    Alessandro Arrichiello

    Alessandro Arrichiello is a Solution Architect for Red Hat Inc. He has a passion for GNU/Linux systems, which began at age 14 and continues today. He has worked with tools for automating enterprise IT: configuration management and continuous integration through virtual platforms. He’s now working on distributed cloud environments involving PaaS (OpenShift), IaaS (OpenStack) and Processes Management (CloudForms), Containers building, instances creation, HA services management, and workflow builds.

    Last updated: March 27, 2023

    Recent Posts

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    • How to integrate vLLM inference into your macOS and iOS apps

    What’s up next?

     

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue