This article offers a quick overview of various Red Hat technologies that you can use to provide a composite solution for the inner and outer loops of the DevSecOps cycle. First, I'll introduce the concepts of the inner and outer loop in simple English, and then explain the use of specific products for realization of the platform engineer’s tasks within this cycle.
From DevOps to platform engineering
Since people started writing software, there has always been a divide between those who create and those who maintain. In the old days (and I’m old, so bear with me), teams were small and the job of maintaining the software in "production" was often part of the developer’s role. However, as software solutions truly took off, the scale became too large for the devs, and the sys admins who were used to owning the infrastructure started to get lumped in with maintaining the BAU (business as usual) applications as well as the hardware.
This led to certain, well, tensions. As the role of the dev effectively ended at the point the software was delivered, the responsibility for context of application moved to the sys admins, who were now adopting the role of operations (Ops). There were often communication breakdowns between the producers and the maintainers of the software (well, always, to be honest) . This led to the adoption of DevOps, which was meant to get rid of the old "chuck it over the wall" mentality. And that worked, to a point. But the Ops people found they were spending a lot of time on the BAU application maintenance, even more so when there were multiple releases, patches, etc.
This led to the concept of the software chain with automated build/test/deployment to ease the work of the Ops staff. In addition, Ops staff started to take on more responsibility for the provisioning of not only the underlying infrastructure, but the automation and development tooling provided to the software developers. And in this way, the role morphed into that of a platform engineer.
Introducing the inner and outer loop
As part of the evolution of the processes of automating development and deployment, the concepts of the inner and outer loop came into being.
The developer lives and works in the inner loop; their responsibility is to create code, build, debug, and finally push the code as the end product. Unlike the old days, where the dev would deliver a binary, the output is now code (and configuration as code).
The platform engineer exists in the outer loop; their responsibility is to run the automation processes that are driven by the arrival of end product from the developers. This entails code review, production build, compliance, and security checks, non-functional and functional testing, and deployment to target environments.
The two cycles work independently other than the crossover point (and instantiation point for the outer loop) of the delivery of code as the developer’s end product.
Figure 1 shows the concepts in essence.
This high-level approach defines a framework, but every organization is different in how it implements the individual points.
What we have done at Red Hat is to architect a set of products, backpinned by Red Hat Enterprise Linux (RHEL) and Red Hat OpenShift, to enable easier implementation of these critical points in such a way as to make the effort of creating the software right up to the point of production deployment as simple, controlled, and intuitive as possible.
The one thing to remember is that Red Hat products, by design, are not limiting by opinionation; we don’t provide a strict product that drives the way in which you work, we provide enterprise grade and highly configurable products that can be used together to realize your organization's requirements, not force an approach or methodology on you. This is the strength of these product sets.
Helping platform engineers realize the inner loop for developers
For a long time I was purely a developer, so I’m a little biased when it comes to the importance of making developer’s lives easier. Technology has advanced so quickly in the last twenty-five years that the developer’s job has become almost impossible. New languages, frameworks, and approaches appear all the time, especially in the open source realm, and developers must learn not only the technologies, but also the way in which they are consumed and exploited.
A typical developer has, on average, around five or so different tools they need to use during the process of writing software; their chosen IDE, build tools, testing tools, hosting platforms, team collaboration tools, etc. They have to know the nuances of these tools—I wasted literally days writing POM files back in the early phases of Maven; writing output such as POM files adds absolutely nothing to the end product of the software I was producing.
Red Hat Developer Hub
What developers have always needed is a one-stop shop with all their tools, knowledge bases, team interactions, shared source, and IDEs. A single pane of glass with helpful guidance and automation for the manual tasks a developer needs to do.
Red Hat Developer Hub, based on the upstream Backstage project, does just that. Not only does Red Hat Developer Hub benefit the consumer (the developer), it makes it easy for a platform engineer to quickly set up a portal configured to the needs of the organization, such that developers can just get on and create. Check out a video demo below.
Red Hat Developer Hub differs from Backstage in that it comes prebuilt with a number of plug-ins specifically around core developer components from Red Hat, ready to use. Plug-ins are extremely important in Backstage and Red Hat Developer Hub; as a standalone component Backstage is an empty framework with a huge amount of potential for providing an internal developer portal, but platform engineers need to put some effort in to get it to the point it helps with the developer’s inner loop. Red Hat Developer Hub comes ready to go out of the box; instead of having to add things such as authentication and additional plug-ins, these are preconfigured.
Plug-ins work by adding additional visual components for the singular portal provided to the end users, and by providing "actions" that can be combined in templates to automate manual processes for the developers—for instance, Red Hat Developer Hub provides visual components for examining the state of a deployed application on OpenShift, and also provides actions for interacting directly with Tekton for pipelines (Red Hat OpenShift Pipelines) execution and Argo CD for GitOps functionality (Red Hat OpenShift GitOps).
In addition, Red Hat Developer Hub is fully supported; once a platform engineer has deployed a portal for use by the organization’s developers, they can raise support requests if they have any issues.
Helping the platform engineer realize the outer loop
OpenShift works extremely well at abstracting the problems of deployment. The use of Kubernetes allows for application deployments to be defined as code; the use of Argo CD (OpenShift GitOps) allows for deployment and automated maintenance; and the use of Red Hat Developer Hub allows for the developers to be given all the tools they need. But the complexity, especially in the outer loop, comes about from the multiplicity of OpenShift clusters.
To make this part of a platform engineer’s job easier, Red Hat has created two additional components that provide single point control of multiple clusters, both from the application deployment perspective and from the securing the estate perspective.
Red Hat Advanced Cluster Management for Kubernetes
The first component is Red Hat Advanced Cluster Management for Kubernetes (ACM), an application that runs on OpenShift and provides a super-controller for multiple clusters. It enables a platform engineer to perform Day 2 operations, such as real-time monitoring, maintenance and patching, to remote clusters. It provides the ability to create new clusters with a single click (using the OpenShift Installer Provisioned Infrastructure approach), place applications on multiple clusters with cluster specific settings (using OpenShift GitOps and Kustomize), impose and maintain cluster level settings compliance (using an approach called profiles). It even works with non-OpenShift clusters (it uses the Kubernetes API to deploy workloads and impose compliance to non-OpenShift instances).
Think of the ACM as a single-pane-of-glass super controller for your entire OpenShift/Kubernetes estate. Similar to the way the Red Hat Developer Hub gives a developer a landing point for all of their actions, ACM gives the same thing to a platform engineer.
Red Hat Advanced Cluster Security for Kubernetes
The other component is Red Hat Advanced Cluster Security for Kubernetes (ACS). This is another deployable application that acts across many clusters, but this one provides real-time scanning, imposition of profiles and exploit detection within workloads. ACS works as a 24/7 watchdog which monitors the state of all clusters and applications within its remit.
The combination of ACM and ACS provides the platform engineer with a single set of tools for complete cluster control and security actions, providing full real-time visibility and tooling to ease the implementation of the outer loop component of the DevSecOps cycle.
Summary
The vast diversity and enterprise strength of the Red Hat product sets—specifically Red Hat Enterprise Linux, Red Hat OpenShift, Red Hat Developer Hub, Red Hat Advanced Cluster Manager for Kubernetes, and Red Hat Advanced Cluster Security for Kubernetes—provides next-generation tooling and components for safely and completely realizing the DevSecOps cycle.
