New in MicroShift 4.20: The generic device plug-in

MicroShift is a small form factor Kubernetes distribution for edge devices that can be found as part of Red Hat Device Edge. We just released Red Hat build of MicroShift 4.20, which adds new features and capabilities like support for cert-manager, a component that simplifies certificate management for endpoints and APIs.

This blog focuses on a key new capability: support for a generic device plug-in. Let’s explore the benefits of a generic device plug-in and see how it works.

What is a generic device plug-in?

Edge computing solutions frequently need access to devices like serial ports, cameras, sound devices, and more. For example, in retail, video cameras require USB attachments for smart self-checkout solutions. But how do you get access from a Kubernetes pod to those devices, which are attached to the host/node? By default, security permissions deny access to any host devices.

Of course you could elevate the privileges of the pods, for example, by adding the hostaccess security context constraint. The problem with this approach is that it grants access to all host devices, such as the disks where the operating system is running from. This does not follow the least-privileges principle. This also opens an actual attack vector, especially if the pods expose a service to external via HTTP(S). If a vulnerability in the pod could be exploited externally, the system is compromised. The usual pod confinement or isolation measure would be undermined, posing a serious threat.

As a result, we need a better solution. Luckily, Kubernetes device plug-ins help address exact challenge.

How Kubernetes device plug-ins work

Kubernetes device plug-ins work by isolating the necessary privileges in a special pod: the device plug-in pod. The device plug-in pod discovers the devices using elevated privileges and announces them to the kubelet and the control plane. Discovered devices then become available in the form of resources on the node. In this approach, the device plug-in pod avoids unnecessary exposure, reducing the attack surface.

The workload pod requests a device declaratively using the usual Kubernetes resource request and limit fields, just as it requests CPU or memory resources. The device is just a special type of resource.

When such a pod is started, the scheduler identifies a node where a resource is available and schedules the pod to that node. The kubelet on that node maps the requested devices to the pod, and nothing more. The pod itself does not need any special privileges.

Kubernetes device plug-ins have been introduced for specialized devices like GPUs, field-programmable gate array (FPGAs), and so on. Special devices require specific drivers and initialization before they can be used. Usually, the device plug-in is provided by the device vendor or manufacturer, alongside the special drivers it needs.

With this latest version, MicroShift now provides a generic device plug-in. This can be used to access any generic device from a pod. Generic devices do not require special drivers or tools. The kernel recognizes and enables them without additional configurations.

Devices that should be made available to pods are declared or defined in the MicroShift generic device plug-in configuration. The config can be highly specific (for example, a device identified by ID like /dev/serial/by-id/xxxxx), or more generic using clobbing (for example, /dev/ttySerial* to make all serial USB devices available). See Figure 1.

Example: How to connect a pod to a serial port

These steps outline how to attach a /dev/ttyUSB0 device to a pod running inside a MicroShift cluster using the generic device plug-in.

1. Run an instance of MicroShift 4.20. See the docs on how to install it. Important: Make sure you have version 4.20. The generic device plug-in is a new feature available only in version 4.20 and later versions. You can use a physical device or a virtual machine to run MicroShift.
2. If your instance does not have a serial device, you can simulate one: run the command sudo mknod /dev/ttyUSB0 c 166 0 to create a dummy /dev/ttyUSB device.

3. Verify that your instance has a serial device available by running the command sudo ls -l /dev/ttyUSB0:

   [redhat@localhost ~]$ ls -l /dev/ttyUSB0
   crw-r--r--. 1 root root 166, 0 Oct  1 10:48 /dev/ttyUSB0

4. Now, configure the generic device plug-in in MicroShift to discover the device. Add the following content to the /etc/microshift/config.yaml file:

   genericDevicePlugin:
       status: Enabled
       domain: device.microshift.io
       devices:
         - name: my-serial
           groups:
             - paths:
                 - path: /dev/ttyUSB0

5. Restart MicroShift by running the command sudo systemctl restart microshift and make sure MicroShift restarts successfully.

6. Run oc describe node and verify my-serial is shown under Capacity, Allocatable, and Allocated resources as shown here:

   Capacity:
     cpu:                                2
     device.microshift.io/my-serial:  1
     ephemeral-storage:                  10176Mi
     hugepages-1Gi:                      0
     hugepages-2Mi:                      0
     memory:                             2776748Ki
     pods:                               250
   Allocatable:
     cpu:                                2
     device.microshift.io/my-serial:  1
     ephemeral-storage:                  10176Mi
     hugepages-1Gi:                      0
     hugepages-2Mi:                      0
     memory:                             2776748Ki
     pods:                               250
   Allocated resources:
     (Total limits may be over 100 percent, i.e., overcommitted.)
     Resource                           Requests     Limits
     --------                           --------     ------
     cpu                                245m (12%)   0 (0%)
     memory                             752Mi (27%)  0 (0%)
     ephemeral-storage                  0 (0%)       0 (0%)
     hugepages-1Gi                      0 (0%)       0 (0%)
     hugepages-2Mi                      0 (0%)       0 (0%)
     device.microshift.io/my-serial     0            0

7. Create a namespace using the command oc create namespace gdp-test.

8. Create a pod using the following spec:

   [redhat@localhost ~]$ cat /tmp/pod.yaml 
   apiVersion: v1
   kind: Pod
   metadata:
     name: gdp-test-pod
     namespace: gdp-test
   spec:
     containers:
     - name: gdp-container
       image: registry.access.redhat.com/ubi9/ubi:9.6
       command: ["sleep", "infinity"]
       resources:
         limits:
           # Request one instance of my-usb-stick device
           device.microshift.io/my-serial: 1 
       # No privileged: true needed. The container runtime injects the device.
       securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
          runAsNonRoot: true
          seccompProfile:
            type: "RuntimeDefault"

9. Verify that the pod gets created successfully and the serial device is shown in the pod:

   oc exec -it gdp-test-pod -- bash
   bash-5.1$ ls -l /dev/ttyUSB0 
   crw-rw-rw-. 1 root root 166, 0 Oct  1 10:54 /dev/ttyUSB0

10. Verify that the oc describe node shows allocated resources as 1:

    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      Resource                           Requests     Limits
      --------                           --------     ------
      cpu                                245m (12%)   0 (0%)
      memory                             752Mi (27%)  0 (0%)
      ephemeral-storage                  0 (0%)       0 (0%)
      hugepages-1Gi                      0 (0%)       0 (0%)
      hugepages-2Mi                      0 (0%)       0 (0%)
      device.microshift.io/my-serial     1            1

Conclusion

The new generic device plug-in, now supported with Red Hat’s build of MicroShift, simplifies secure access to generic devices like cameras, serial ports, etc., from inside a pod. To find out more, check the product documentation or reach out to a Red Hatter near you.