How to forward OpenShift logs to Azure Monitor

September 19, 2024
Ahmed Bessifi
Related topics:
DevSecOpsGitOpsHelmObservability
Related products:
Red Hat OpenShift Container Platform

The logging for Red Hat OpenShift 5.9 release introduced support for forwarding logs to Azure Monitor using Vector. By supporting this functionality, Red Hat OpenShift users are able to seamlessly integrate their cluster infra services and application logs with Azure Monitor Logs, leveraging its powerful analytics and monitoring capabilities.

This integration empowers users to streamline their monitoring and troubleshooting processes, leading to better operational insights, faster issue resolution, and improved overall performance of their cluster services and applications.

Logging for OpenShift

As a cluster administrator, you can deploy logging for Red Hat OpenShift on a Red Hat OpenShift Container Platform cluster and use it to collect and aggregate node system audit logs, application container logs, and infrastructure logs. You can forward logs to your chosen log outputs, including on-cluster, Red Hat managed log storage. You can also visualize your log data in the OpenShift Container Platform web console, or the Kibana web console, depending on your deployed log storage solution.

With logging for Red Hat OpenShift 5.9 and later, you can forward logs to Azure Monitor Logs in addition to, or instead of, the default log store. This functionality is provided by the Vector Azure Monitor Logs sink.

Azure Monitor Logs is a comprehensive service provided by Microsoft Azure that enables the collection, analysis, and actioning of telemetry data across various Azure and on-premises resources.

Why forward logs to Azure Monitor?

Why use this?

  • As a product manager, you want to be able to forward logs from your OpenShift cluster to Azure Monitor Logs so that you can create dashboards and reports to track the performance and reliability of your applications and services.
  • As an administrator, you want to be able to forward logs from your OpenShift cluster to Azure Monitor Logs so that you can create alerts for security events or other critical incidents.
  • As a developer, you want to be able to forward logs from your OpenShift cluster to Azure Monitor Logs so that you can use Azure Log Analytics to search and filter logs, and to create custom visualizations of your log data.

Enable OpenShift logs forwarding to Azure Monitor

In this article, we'll be implementing this configuration using a Helm chart that:

  • Enables OpenShift LogForwarder.
  • Sets up LogForwarding to send cluster logs to Azure Monitor (via Vector).

Note

With a Red Hat OpenShift Kubernetes Engine subscription, the OpenShift Logging Operator could not be used to set up an aggregated logging stack for the cluster. Only the LogForwarding feature is available to help exporting logs to external systems like Azure Monitor service or another supported services. This feature is only available and supported with Vector.

The Red Hat OpenShift Logging Operator is only deployable to the openshift-logging namespace. To enable metrics service discovery, add the label openshift.io/cluster-monitoring: true to the namespace spec.

Requirements 

To enable OpenShift logs forwarding to Azure Monitor, you need:

  • A Log Analytics Workspace created.
  • OpenShift cluster-admin access.
  • The logging for Red Hat OpenShift operator 5.9 or later installed on your cluster.
  • Helm version 3.8 or greater.

Let's explore a few use cases for this technology.

Forwarding to Azure Monitor Logs via public network

The Workspace ID and the Shared Key are the two required parameters for this configuration. You can run the following commands to get these parameters if you have access to Azure APIs with the AZ command-line interface (CLI):

WORKSPACE_ID=$(az monitor log-analytics workspace show -g <AZ RESOURCE GROUP> -n <AZ LOG WORKSPACE NAME> -o tsv --query customerId) 
    SHARED_KEY=$(az monitor log-analytics workspace get-shared-keys -g <AZ RESOURCE GROUP> -n <AZ LOG WORKSPACE NAME> -o tsv --query primarySharedKey)

Note

This Helm Chart does not yet support integration with a Secret Manager. In this case the OpenShift Secret that holds the Azure Monitor Workspace Shared Key should be created manually.

Setup

The following procedure configures the OpenShift Logging Forwarder using Helm. This method supports a simplified installation path compared to a GitOps one that could rely on Argo CD, for instance. 

  1. Create a secret for your Azure workspace:

    apiVersion: v1
    kind: Secret
    metadata:
      name: azure-workspace-key
      namespace: openshift-logging
    type: Opaque
    data:
      shared_key: <base64-encoded-shared-key>

  2. Update your values.yaml file and run the helm install command:

    cd /some/where/
git clone https://github.com/abessifi/openshift-log-forwarding
cd openshift-log-forwarding/
helm install \
  --namespace openshift-logging \
  --values ./values.yaml \
  <release-name> \
  ./

    This will create a ClusterLogging instance with Vector as log collector.

  3. Verify the Operator installation. Check the contents of the specified namespace (openshift-logging) to ensure all pods and services have started successfully:

    oc get pods,ds,svc -n openshift-logging

    The response should resemble the following:

    NAME                                            READY   STATUS      RESTARTS   AGE
                    pod/collector-hw7dm                             1/1     Running     0          7m32s
                    [...]
                    NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
                    service/collector                          ClusterIP   172.30.37.74    <none>        24231/TCP   7m53s
                    [...]
                    NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
                    daemonset.apps/collector   1         1         1       1            1           kubernetes.io/os=linux   7m52s

Check for logs in Azure

Wait 5 to 15 minutes until the logs are pushed to Azure, then query your Azure Log Analytics Workspace as follows:

  1. Search and select the Monitor service from Azure Services catalog as shown in Figure 1.
Select Monitor from Azure Services Dashboard
Figure 1: Select Monitor from Azure Services Dashboard.
  1. Click Logs on the left side bar menu (Figure 2).
Select Logs option on the left side bar
Figure 2: Select Logs option on the left side bar.
  1. Define a Scope by selecting the target Log Analytics Workspace (Figure 3).
Define Scope by selecting the target Log Analytics Workspace
Figure 3: Define Scope by selecting the target Log Analytics Workspace.
  1. Query the Log Analytics (e.g., get OpenShift infrastructure logs) as shown in Figure 4.
Query Log Analytics
Figure 4: Query Log analytics.

 You can also use the CLI to check out the logs:

az monitor log-analytics query -w $WORKSPACE_ID --analytics-query "<your-log-type-string> | take 10" --output tsv

Use Azure Private Link to connect OpenShift on-premises to Azure Monitor

You can also leverage the same Helm chart with a existing/pre-configured Azure Private Link to achieve the following:

  • Connect privately to Azure Monitor without opening up any public network access.
  • Ensure your monitoring data is only accessed through authorized private networks.
  • Keep all traffic inside the Azure backbone network.

Requirements

  • Your on-premise network is connected to Azure VNET where Azure Monitor service is enabled.
  • Create an Azure Monitor Private Link.
  • Create a private endpoint on your network and connect it to the scope.
  • Configure the required access on your Azure Monitor resources.

Figure 5 depicts this process.

A flowchart depicting Azure Monitor Private Link Connection.
Figure 5: Azure Monitor Private Link connection.

Refer to the official Azure documentation for more details on how create a Private Link and implement the above requirements.

Info

With Azure Private Link, you can securely link Azure platform as a service (PaaS) resources to your virtual network by using private endpoints. Azure Monitor is a constellation of different interconnected services that work together to monitor your workloads. An Azure Monitor private link connects a private endpoint to a set of Azure Monitor resources to define the boundaries of your monitoring network. That set is called an Azure Monitor Private Link Scope (AMPLS).

References