Over the years, the must-gather log archive has become the default method of supplying Red Hat Support with a comprehensive snapshot of an OpenShift cluster's health and state. The log archive contains all the necessary cluster resources, service logs, audit logs, pod states, and YAML manifests needed for a support engineer to debug a cluster issue without requiring access to the cluster itself. The support log gather operator is the most recent investment by Red Hat in improving the experience of troubleshooting and getting support for Red Hat OpenShift clusters, and it's now generally available.
The Red Hat community initially developed the must-gather command to enable cluster administrators to trigger the collection job on a cluster. However, as OpenShift has grown, the complexity of the requirements for these log bundles has also grown. This can occasionally result in massive (10-100 GB!) log archives that need to be collected quickly when an issue has been detected. Additionally, compliance and regulatory requirements have resulted in extension features like the must-gather clean command.
Today, we add another tool to the admin tool box: The support log gather operator. This operator brings with it additional features that make collection of crucial log bundles easier and faster than ever before, with compliance and automation features included by default. Think of the operator as an extension of the command-line interface, modernizing it for your support needs today. The support log gather operator is available with Red Hat OpenShift 4.22, Red Hat OpenShift 5, and all future versions.
Using the oc command to gather logs
Previously, the process of collecting logs for support cases had the following requirements and workflow:
Requirements
- Cluster administrator credentials
- The
occommand line tool installed
Collection steps
- Cluster administrator logs in to the cluster using the
occommand - Cluster administrator uses command
oc adm must-gatherwith appropriate parameter flags to collect log archive required by Red Hat Support teams - Command runs to completion and supplies a structured directory of all collected logs as a
.tar.gz("tarball") file - If the customer has strict compliance regulations, then cluster administrator runs the community-developed
must-gather-cleancommand on the tarball to obfuscate confidential or restricted information. - The cluster administrator must then re-archive the log bundle, typically as a
.tar.gzfile - The tarball is uploaded to the support ticket using the access.redhat.com support portal
- If more logs are required, then repeat steps 2-6.
Overall, the process is manual and, critically, requires cluster administrator credentials to execute. In most companies, those with cluster administrator permissions are vastly outnumbered by the number of developers within the organization. This often results in admins spending a significant amount of their time monitoring ongoing log collections rather than focusing on their other day to day tasks. This becomes exceptionally painful when admins are required to collect more than one must-gather archive, using custom must-gather images, or with different collection parameters. Cluster administrators' time becomes a bottleneck in troubleshooting a cluster issue where time is at a premium. They are also expected to remediate the issue, create necessary workarounds, and bring the workloads back online. Taking time to trigger the log collection, and monitor the job is something that cluster administrators don't want to have to worry about.
Introducing the support log gather operator
The goal of the support log gather operator is to offer an automated, secure, API-driven process to collect logs for a cluster when support from Red Hat is needed. When installed, the operator introduces a new custom resource (the MustGather CR), which can be deployed by standard OpenShift users or using GitOps to trigger the collection of a must-gather log archive.
When developing the operator, our primary goal was to meet feature parity with existing command-line tools. This meant ensuring that all parameters available in the CLI are supported for the operator. However, the operator framework also provided a good opportunity to develop additional features to improve the overall experience of collecting logs for a cluster.
Non-admin access
This is one of the biggest improvements over the CLI tool. The operator enables any developer, or automation tool, with credentials to deploy the MustGather CR to trigger collection of a must-gather log archive. The major implication of this is that, if an issue arises on a cluster, cluster administrators are no longer required to trigger a log collection job. This removes a significant bottleneck from the log collection process and eliminates a tedious job from cluster administrators so they can focus on critical tasks. Additionally, the RBAC for this is extremely tightly scoped, meaning that adding this capability to a non-privileged user doesn't grant them unexpected access to confidential information. See the example below:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: must-gather-trigger-role
namespace: must-gather-operator # Replace with your operator's namespace
rules:
- apiGroups:
- operator.openshift.io
- redhatcop.redhat.io
resources:
- mustgathers
verbs:
- create
- get
- list
- watch
Granting this role to a developer does not provide pathways for unexpected privilege escalation. Were a developer granted only this role, they wouldn't have access to any other part of the cluster, nor access to any of the information collected by the must-gather log collection job itself.
Must-gather clean
Many organizations have strict governance and compliance requirements with respect to the data they collect and share to third parties. Red Hat Support teams are no exception. The must-gather-clean tool developed by the Red Hat community helps obfuscate or delete common pieces of confidential information from a must-gather log archive prior to uploading it to a Red Hat support case. However, because it's maintained by the community, many customers who would like to use it are prohibited by the same types of policies that necessitate its use.
In the support log gather operator, the must-gather clean capabilities are included in the operator. For those who need the capabilities of the must-gather clean toolkit, but are restricted by company policy or regulation, this presents an opportunity to adopt the tooling through an offering supported by Red Hat.
This example YAML manifest shows a MustGather CR with several parameters filled in:
apiVersion: operator.openshift.io/v1
kind: MustGather
metadata:
name: obfuscated-custom-gather
spec:
serviceAccountName: must-gather-admin
# Reference to your custom ImageStream
imageStreamRef:
name: custom-must-gather # Name of the ImageStream resource
tag: latest # Tag within the ImageStream
obfuscate:
enabled: true
# Optional: Reference to custom must-gather clean configuration file
obfuscationConfigRef:
name: custom-obfuscation-config
uploadTarget:
type: SFTP
sftp:
caseID: '12345678'
caseManagementAccountSecretRef:
name: case-management-credsLook for the field obfuscate to see how to trigger must-gather clean to run on a collected must-gather log archive prior to uploading the archive to an existing Red Hat support case. Cluster administrators can even supply a custom ConfigMap containing specific obfuscation rules for the must-gather clean tool.
Custom images
Finally, as with the must-gather command, the support log gather operator comes with support for custom images. As OpenShift has evolved, custom images have become a crucial part of the must-gather ecosystem, allowing Red Hat developed products to ship an image containing the scripts needed to collect all the logs and data needed to troubleshoot their specific product. For example, there are custom images for Red Hat OpenShift Virtualization, Red Hat OpenStack for OpenShift, among many others. The support log gather operator offers a secure and well-defined manner to trigger "custom" must-gather collections. Cluster administrators can define an `ImageStream for any custom image they require, and the MustGather CR can reference that ImageStream when it is deployed.
The benefit of using the ImageStream architecture is that it abstracts away the management of the custom images from the usage of these images by the operator. A cluster administrator has complete control over ImageStreams, meaning they can create, delete, and modify them at any time. By comparison, the MustGather CR and the developers deploying them can request any ImageStream that exists within the support log gather operator namespace. As a result, only images that are whitelisted (configured by the cluster administrator as an ImageStream) can be invoked by the MustGather CR, minimizing the risk of running unexpected or illicit code in a relatively privileged manner. You can see an example of the image stream configuration block for the MustGather CR in the above code block under imageStreamRef.
The future for the operator is bright
This may be the first release for the support log gather operator, but we already have a full roadmap. Our goals for the operator extend well beyond bringing it to feature parity with the must-gather tool, and we view the operator as an opportunity to fundamentally improve the customer experience when collecting logs for Red Hat Support.
We're planning to implement rolling log collection in the future. This is significant for many reasons, but the primary goal of this feature is to minimize the amount of time between initially filing a support request and being able to provide logs for troubleshooting. Our goal is to provide the interface for scheduling collection of logs at regular intervals, to ensure that there's always an existing log archive. Meaning, if there is an issue, a developer only needs to trigger a log collection of the time between the last periodic log collection and the occurrence of the issue. This should significantly reduce the time spent waiting for a log collection to complete prior to sending the log archive to Red Hat.
We also expect to bring a number of quality of life improvements including improved SFTP host whitelists, improved error handling, and automatic PVC size management.
If you are interested in trying this operator out for yourself, feel free to check out the documentation, or head to the Red Hat operator catalog to install it.