Aside from naming and versioning, managing sensitive assets, like credentials, is one of the more challenging aspects in technology. So, why is it so difficult? Well, to start off. What may be considered a sensitive asset to one individual or organization may not be the same as another. Also, given that there are so many different ways that sensitive assets can be managed, there is no universally accepted method available.
The challenges that encompass how sensitive assets are handled also apply to image mode, a new method that enables building and deploying Operating Systems using similar tools and approaches as any other traditional container. In this article, we will discuss the types of sensitive assets that apply to image mode for RHEL specifically and how to design appropriate workflows to incorporate secure practices within all phases, from build, deployment, to runtime.
Read more about image mode for RHEL
Content Types
Before looking into how sensitive assets can be used within the context of image mode for RHEL, it is important to understand several common types of content that are typically considered to contain resources that are sensitive in nature.
- SSH Keys - Public/private keypairs that enable communication between systems.
- Certificates - A signed document that represents the identity of an individual or system.
- User Details - Information associated with an individual, including their name, government and financial-related properties, and passwords
Careful consideration needs to be taken when handling any of these kinds of assets.
Considerations
With an understanding of mcommon types of content that are typically considered sensitive, let’s look at several approaches where they are used within image mode for RHEL and how they can be handled properly to support a secure operation.
Injection at Image Build Time
Since image mode for RHEL builds on top of standard container technologies, many of the same approaches for building containers apply, including the use of a Dockerfile/Containerfile or libraries/tools that are specifically designed for constructing container images. Examples of commonly used tools include Podman, Docker, Buildah, and Jib, just to name a few. By including the asset as part of the container build process, it must be handled appropriately both within the build tool, but also at the distribution point (Image registry).
The following examples demonstrate how an SSH public key can be included within a Dockerfile/Containerfile as a build argument and injected at build time by the build tool.
FROM registry.redhat.io/rhel9-beta/rhel-bootc:9.4
# You *must* specify this argument; it is a SSH public key
# (in general an authorized_keys formatted data)
# that will be used for the `root` user by default.
ARG SSHPUBKEY
# In this example, we add /usr/ssh to the search path OpenSSH uses for keys.
# The rationale for this is that `/usr` is always part of the immutable
# container state, as opposed to user home directories which are mutable.
# In this pattern, you can always have a "fallback" key available, but
# e.g. use an external system (such as cloud-init) to live-update
# the traditional authorized_keys in the user's home directories.
RUN set -eu; mkdir -p /usr/ssh && \
echo 'AuthorizedKeysFile /usr/ssh/%u.keys .ssh/authorized_keys .ssh/authorized_keys2' >> /etc/ssh/sshd_config.d/30-auth-system.conf && \
echo ${SSHPUBKEY} > /usr/ssh/root.keys && chmod 0600 /usr/ssh/root.keys
Recommendations
- At build time, utilize sensitive assets that are retrieved from a secret storage tool rather than storing content alongside the source code. The content is then passed to the image build process in an ephemeral fashion and only made available within the final image instead of at the source, such as a Git repository, or within the build tool.
- Control access at the distribution point (image registry) to restrict the ability to retrieve image content that includes sensitive assets. Assets may include the associated credentials for accessing protected content from image registries. Adhere to the processes for injecting sensitive assets as part of the build process as described previously.
- Consider using a multi-stage build process where a base bootc image is produced without any sensitive assets included. Subsequent downstream images can make use of the base image as their starting point and inject any necessary sensitive assets into the desired finalized image. This approach ensures that sensitive assets are not included for those consumers who do not need to make use of them, along with enabling an entire separate lifecycle of content based on each consumer. This will enable a separation of concerns between different producers and consumers.
Platform Level Injection
Most, if not all cloud providers support the injection of SSH keys into running instances at a minimum. By using this mechanism, the management/injection of sensitive content is offloaded from the end user to the platform. This is the preferred method when producing content for a specific target environment. Sensitive assets are not included within the image mode for RHEL build process and are instead offloaded either at instance provisioning time, or through an out-of-band process. Provisioned instances are then associated with the desired sensitive asset at runtime. Refer to the following resources that describe how to inject and utilize sensitive resources within deployed instances within cloud providers.
cloud-init
cloud-init is a facility to automate cloud instances at instantiation and provides an out of band method for injecting sensitive content for use at runtime. One of the common uses of cloud-init is the creation and management of user accounts within Virtual Machine instances. Of interest within this discussion in particular, is the ability to define the credentials associated with each account (whether it be a password or in most cases, SSH public keys).
The use of cloud-init requires the creation of two configuration files: properties related to the Virtual Machine instance (Metadata) and details that are either related or performed by users (User Data).
A minimalistic Metadata file takes the following form:
instance-id: ${VM_HOSTNAME}
local-hostname: ${VM_HOSTNAME}
The following represents a typical User Data file that creates a new user called cloud-user, grants the user elevated access (sudo), along with defining an SSH public key to enable remote access:
#cloud-config
users:
- default
- name: cloud-user
sudo: ALL=(ALL) NOPASSWD:ALL
groups: sudo, wheel
lock_passwd: true
ssh_authorized_keys:
- ${SSH_PUB_KEY}
These configurations are provided as baselines and can be customized as needed for use within the desired target environment.
VMware vSphere represents a common destination whereVirtual Machines created with image mode for RHEL are deployed. The aforementioned Metadata and User Data configurations can be used to customize a Virtual Machine by dynamically injecting cloud-init configurations at runtime.
These resources can be applied to existing Virtual Machines using govc, a CLI based utility for configuring vSphere environments in an automated fashion. The following command represents how cloud-init configurations, located in files called metadata.yaml and userdata.yaml, can be added to a deployed Virtual Machine. Be sure to replace the placeholder values with the appropriate values as necessary.
govc vm.change \
-dc="${DATACENTER}" \
-vm="${VM_NAME}" \
-e guestinfo.metadata="$(gzip -c9 <metadata.yaml | { base64 -w0 2>/dev/null || base64; })" \
-e guestinfo.metadata.encoding="gzip+base64" \
-e guestinfo.userdata="$(gzip -c9 <userdata.yaml | { base64 -w0 2>/dev/null || base64; })" \
-e guestinfo.userdata.encoding="gzip+base64"
Once the configurations have been applied, Virtual Machines can be accessed using the corresponding SSH private key. By leveraging cloud-init in this fashion, sensitive assets are not included within the bootc image, and the specific cloud-init configurations can differ depending on the desired use of the Virtual Machine.
Recommendation:
- When managing sensitive assets associated with cloud-init configurations in an automated fashion, ensure the associated tools that are integrating with target environments are employing proper security practices for both the sensitive cloud-init resources along with the credentials for accessing the target environment.
In this article, we described several common types of content that typically contain sensitive resources when working with image mode for RHEL. By employing one of the approaches described and adhering to the associated guidance, the odds of a compromise are reduced, and the overall security posture is increased, ensuring a safe operating environment.
Read more about image mode for RHEL.
