How to classify Red Hat OpenShift audit logs

Key features of OpenShift audit logs include:

1. Granular tracking: Audit logs provide detailed information about each action taken within the platform, allowing for precise monitoring and analysis.
2. Security and compliance: By capturing all significant activities, audit logs help organizations meet security standards and regulatory compliance requirements.
3. Troubleshooting and forensics: In the event of an issue or security incident, audit logs serve as an invaluable resource for diagnosing problems and understanding the sequence of events leading up to the issue.
4. User accountability: By recording user actions, audit logs ensure accountability and help in identifying any unauthorized or malicious activities.

Understanding and effectively utilizing OpenShift audit logs is essential for maintaining the security, reliability, and compliance of your containerized applications and services. Audit works at the API server level, logging all requests coming to the server. 

You can view the logs for the OpenShift API server, Kubernetes API server, and OpenShift OAuth API server for each control plane node:

oc adm node-logs --role=master --path=openshift-apiserver/
oc adm node-logs --role=master --path=kube-apiserver/
oc adm node-logs --role=master --path=oauth-apiserver/

Red Hat OpenShift Container Platform provides the following predefined audit policy profiles:

• Default: Logs metadata for read and write requests (get, list, create, update, patch).
• WriteRequestBodies: Same as Default but it also logs request bodies for every write request (create, update, patch).
• AllRequestBodies: Same as WriteRequestBodies but it also logs the read requests (get, list).
• None: Disables logging.

Each audit log contains the following information:

• requestURI: The request URI as sent by the client to a server.
• verb: The Kubernetes verb associated with the request. For non-resource requests, this is the lowercase HTTP method.
• user: The authenticated user information.
• objectRef: Optional. The object reference this request is targeted at. This does not apply for List-type requests, or non-resource requests.
• annotations: Optional. An unstructured key value map stored with an audit event that may be set by plugins invoked in the request serving chain, including authentication, authorization and admission plugins. Note that these annotations are for the audit event, and do not correspond to the metadata.annotations of the submitted object. Keys should uniquely identify the informing component to avoid name collisions, for example podsecuritypolicy.admission.k8s.io/policy. Values should be short. Annotations are included in the metadata level.

This guide will help you classify OpenShift audit logs, distinguishing between operations performed by users and operators, and understanding the meaning of various fields in the logs. Keep in mind that there are some sensitive resources, such as Secrets, OAuthClients, or Routes, for which RequestBodies will never be logged. See Figure 1.

Example 1: Operator operation

{
  "@timestamp": "2024-06-27T16:24:19.367169194Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-logging.v5.9.3-78d9d68c5c\" of ClusterRole \"cluster-logging.v5.9.3-78d9d68c5c\" to ServiceAccount \"cluster-logging-operator/openshift-logging\""
  },
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "7d3a53e8-1a45-4fc0-a7c5-005d0f049442",
  "file": "/var/log/kube-apiserver/audit.log",
  "hostname": "ocp4-b4qtb-master-2",
  "k8s_audit_level": "Metadata",
  "kind": "Event",
  "kubernetes": {
    "container_name": "",
    "namespace_name": "",
    "pod_name": ""
  },
  "level": "Metadata",
  "log_type": "audit",
  "objectRef": {
    "apiVersion": "v1",
    "name": "grafana-dashboard-cluster-logging",
    "namespace": "openshift-config-managed",
    "resource": "configmaps"
  },
  "openshift": {
    "cluster_id": "6a934df4-cb47-4cf3-a01c-31dba9e9e35e",
    "sequence": 1719505459368005358
  },
  "requestReceivedTimestamp": "2024-06-27T16:24:19.363436Z",
  "requestURI": "/api/v1/namespaces/openshift-config-managed/configmaps/grafana-dashboard-cluster-logging",
  "responseStatus": {
    "code": 200,
    "metadata": {}
  },
  "sourceIPs": ["192.168.126.51"],
  "source_type": "file",
  "stage": "ResponseComplete",
  "stageTimestamp": "2024-06-27T16:24:19.366119Z",
  "user": {
    "extra": {
      "authentication.kubernetes.io/pod-name": ["cluster-logging-operator-7fb89478c6-7jn7j"],
      "authentication.kubernetes.io/pod-uid": ["80abd203-e809-4d89-a597-bbbbc1cdf87e"]
    },
    "groups": ["system:serviceaccounts", "system:serviceaccounts:openshift-logging", "system:authenticated"],
    "uid": "04951387-e4ce-48a1-ac2e-0e92271ac05a",
    "username": "system:serviceaccount:openshift-logging:cluster-logging-operator"
  },
  "userAgent": "cluster-logging-operator/v0.0.0 (linux/amd64) kubernetes/$Format",
  "verb": "get"
}

Structural analysis:

• file: /var/log/kube-apiserver/audit.log: path to file logs on the control plane node.
• Verb: Get Action of obtaining resources.
• Resource: ConfigMaps Resource type.
• Namespace: openshift-config-managed: Namespace of resource.
• User: system:serviceaccount:openshift-logging:cluster-logging-operator. This indicates that the action is performed by an operator.
• requestURI: /api/v1/namespaces/openshift-config-managed/configmaps/grafana-dashboard-cluster-logging
  • This field contains the address of the API endpoint to which the request was sent. The requestURI provides detailed information about the type of operation and the resource that was involved in the request. 
  • Structure: /{url-api}/namespaces/{Namespace-resource}/{Object}/{Name}
• Conclusion: ConfigMaps request graphana-dashboard-cluster-logging was executed from the ServiceAccount system:serviceaccount:openshift-logging:cluster-logging-operator in the openshift-config-managed namespace.

Example 2: User operation

{
  "@timestamp": "2024-06-27T16:25:30.423688906Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-admin-htpasswd\" of ClusterRole \"cluster-admin\" to User \"foo\""
  },
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "847b9ee8-8f90-4985-b2fd-564d39047c53",
  "file": "/var/log/kube-apiserver/audit.log",
  "hostname": "ocp4-b4qtb-master-0",
  "k8s_audit_level": "Metadata",
  "kind": "Event",
  "kubernetes": {
    "container_name": "",
    "namespace_name": "",
    "pod_name": ""
  },
  "level": "Metadata",
  "log_type": "audit",
  "objectRef": {
    "apiGroup": "helm.openshift.io",
    "apiVersion": "v1beta1",
    "resource": "helmchartrepositories"
  },
  "openshift": {
    "cluster_id": "6a934df4-cb47-4cf3-a01c-31dba9e9e35e",
    "sequence": 1719505530424537701
  },
  "requestReceivedTimestamp": "2024-06-27T16:25:30.325256Z",
  "requestURI": "/apis/helm.openshift.io/v1beta1/helmchartrepositories",
  "responseStatus": {
    "code": 200,
    "metadata": {}
  },
  "sourceIPs": ["192.168.126.13"],
  "source_type": "file",
  "stage": "ResponseComplete",
  "stageTimestamp": "2024-06-27T16:25:30.332028Z",
  "user": {
    "extra": {
      "scopes.authorization.openshift.io": ["user:full"]
    },
    "groups": ["system:authenticated:oauth", "system:authenticated"],
    "uid": "5c5eda6c-4ce1-48b6-bb99-a57abf98d2b1",
    "username": "foo"
  },
  "verb": "list"
}

Structural analysis:

• file: /var/log/kube-apiserver/audit.log: Path to file logs on the control plane node.
• Verb: list Action of listing resources.
• Resource: helmchartrepositories: Resource type.
• User: foo This indicates that the action is performed by a user.
• requestURI: /apis/helm.openshift.io/v1beta1/helmchartrepositories
  • This field contains the address of the API endpoint to which the request was sent. The requestURI provides detailed information about the type of operation and the resource that was involved in the request. 
  • Structure: /{url-api}/{api-group}/{version}/{Kind}
• Conclusion: The user foo made a request to list all Helm Chart repositories in the helm.openshift.io API version v1beta1.

A list of all API resources on the cluster can be obtained with the oc api-resources command:

[root@ocp ~]# oc api-resources
NAME                                       SHORTNAMES                                                                             APIVERSION                                            NAMESPACED   KIND
bindings                                                                                                                          v1                                                    true         Binding
componentstatuses                          cs                                                                                     v1                                                    false        ComponentStatus
configmaps                                 cm                                                                                     v1                                                    true         ConfigMap
endpoints                                  ep                                                                                     v1                                                    true         Endpoints
helmchartrepositories                                                                                                             helm.openshift.io/v1beta1                             false        HelmChartRepository
projecthelmchartrepositories                                                                                                      helm.openshift.io/v1beta1                             true         ProjectHelmChartRepository

To learn about the behavior of the related object in OpenShift you can look it up in our documentation or run the following command in order to get a brief description:

[root@ocp ~]# oc explain HelmChartRepository
KIND:     HelmChartRepository
VERSION:  helm.openshift.io/v1beta1

DESCRIPTION:
     HelmChartRepository holds cluster-wide configuration for proxied Helm chart
     repository Compatibility level 2: Stable within a major release for a
     minimum of 9 months or 3 minor releases (whichever is longer).

Use case

Let us look at some practical examples of how to classify audit logs and in particular those related to the following functions:

1. Login success
2. Login failed
3. Logout

{
  "@timestamp": "2024-06-25T10:46:10.359148847Z",
  "annotations": {
    "authentication.openshift.io/decision": "deny",
    "authentication.openshift.io/username": "bar",
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  },
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "e43b7d1a-6167-4c0f-84f3-a3a1fa6781db",
  "file": "/var/log/oauth-server/audit.log",
  "hostname": "ocp4-b4qtb-master-2",
  "kind": "Event",
  "kubernetes": {
    "container_name": "",
    "namespace_name": "",
    "pod_name": ""
  },
  "level": "Metadata",
  "log_type": "audit",
  "openshift": {
    "cluster_id": "6a934df4-cb47-4cf3-a01c-31dba9e9e35e",
    "sequence": 1719312370360536000
  },
  "openshift_audit_level": "Metadata",
  "requestReceivedTimestamp": "2024-06-25T10:46:04.570226Z",
  "requestURI": "/login",
  "responseStatus": {
    "code": 302,
    "metadata": {}
  },
  "sourceIPs": [
    "10.128.8.1"
  ],
  "source_type": "file",
  "stage": "ResponseComplete",
  "stageTimestamp": "2024-06-25T10:46:04.570667Z",
  "user": {
    "groups": [
      "system:unauthenticated"
    ],
    "username": "system:anonymous"
  },
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
  "verb": "post"
}

Structural analysis:

• file: /var/log/oauth-server/audit.log: Path to file logs on the control plane node.
• Verb: post Indicates an HTTP request of the POST type. This type of request is mainly used to create new resources within the cluster.
• Annotations: in this case it is really important to analyze the annotations to understand what was sent to the OAuth server:
  • authentication.openshift.io/username: bar: This indicates that the action was requested by a user.
  • authentication.openshift.io/decision: deny: This indicates that the action was rejected.
• requestURI: /login*: This field contains the address of the API endpoint to which the request was sent.
• Conclusion: Failed log in attempt for admin user, may be wrong password or user is nonexistent.

{
  "@timestamp": "2024-06-25T10:47:01.888804833Z",
  "annotations": {
    "authentication.openshift.io/decision": "allow",
    "authentication.openshift.io/username": "foo",
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  },
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "64aa17ed-568f-4b1a-8bee-19c82a9e991b",
  "file": "/var/log/oauth-server/audit.log",
  "hostname": "ocp4-b4qtb-master-2",
  "kind": "Event",
  "kubernetes": {
    "container_name": "",
    "namespace_name": "",
    "pod_name": ""
  },
  "level": "Metadata",
  "log_type": "audit",
  "openshift": {
    "cluster_id": "6a934df4-cb47-4cf3-a01c-31dba9e9e35e",
    "sequence": 1719312421978967800
  },
  "openshift_audit_level": "Metadata",
  "requestReceivedTimestamp": "2024-06-25T10:46:59.854523Z",
  "requestURI": "/login",
  "responseStatus": {
    "code": 302,
    "metadata": {}
  },
  "sourceIPs": [
    "10.128.8.1"
  ],
  "source_type": "file",
  "stage": "ResponseComplete",
  "stageTimestamp": "2024-06-25T10:46:59.895431Z",
  "user": {
    "groups": [
      "system:unauthenticated"
    ],
    "username": "system:anonymous"
  },
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
  "verb": "post"
}

Structural analysis:

• file: /var/log/oauth-server/audit.log: Path to file logs on the control plane node.
• Verb: post Indicates an HTTP request of the POST type. This type of request is mainly used to create new resources within the cluster.
• Annotations: in this case it is really important to analyze the annotations to understand what was sent to the OAuth server:
  • authentication.openshift.io/username: foo: This indicates that the action was requested by a user. 
  • authentication.openshift.io/decision: allow: This indicates that the action is permitted.
• requestURI: /login*: This field contains the address of the API endpoint to which the request was sent.
• Conclusion: Successful log in attempt for user foo.

{
  "@timestamp": "2024-06-25T10:47:30.454146125Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:oauth-token-deleters\" of ClusterRole \"system:oauth-token-deleter\" to Group \"system:unauthenticated\""
  },
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "898d4b30-da0c-4bbf-8bfe-ea9b925e00bc",
  "file": "/var/log/oauth-apiserver/audit.log",
  "hostname": "ocp4-b4qtb-master-1",
  "kind": "Event",
  "kubernetes": {
    "container_name": "",
    "namespace_name": "",
    "pod_name": ""
  },
  "level": "RequestResponse",
  "log_type": "audit",
  "objectRef": {
    "apiGroup": "oauth.openshift.io",
    "apiVersion": "v1",
    "name": "sha256~LCMSmy3S",
    "resource": "oauthaccesstokens"
  },
  "openshift": {
    "cluster_id": "6a934df4-cb47-4cf3-a01c-31dba9e9e35e",
    "sequence": 1719312450454908200
  },
  "openshift_audit_level": "RequestResponse",
  "requestReceivedTimestamp": "2024-06-25T10:47:30.122562Z",
  "requestURI": "/apis/oauth.openshift.io/v1/oauthaccesstokens/sha256~LCMSmy3S",
  "responseObject": {
    "apiVersion": "oauth.openshift.io/v1",
    "authorizeToken": "sha256~b9ZfGI9gP1BD4f2F7aLeV8Dai8hwCZ8Fbv-R9rrGx-Y",
    "clientName": "console",
    "expiresIn": 86400,
    "kind": "OAuthAccessToken",
    "metadata": {
      "creationTimestamp": "2024-06-25T10:47:00Z",
      "name": "sha256~LCMSmy3S",
      "resourceVersion": "907890964",
      "uid": "2ffcfcf1-89a3-4243-a1cd-ae7be090dfa7"
    },
    "redirectURI": "https://console-openshift-console.apps.ocp4.filice.eu/auth/callback",
    "scopes": [
      "user:full"
    ],
    "userName": "foo",
    "userUID": "5c5eda6c-4ce1-48b6-bb99-a57abf98d2b1"
  },
  "responseStatus": {
    "code": 200,
    "metadata": {}
  },
  "sourceIPs": [
    "192.168.126.11",
    "10.129.0.1"
  ],
  "source_type": "file",
  "stage": "ResponseComplete",
  "stageTimestamp": "2024-06-25T10:47:30.449656Z",
  "user": {
    "groups": [
      "system:unauthenticated"
    ],
    "username": "system:anonymous"
  },
  "userAgent": "Go-http-client/1.1",
  "verb": "delete"
}

Structural analysis:

• file: /var/log/oauth-server/audit.log Path to file logs on the control plane node.
• Verb: delete Indicates an HTTP request of the DELETE type. This type of request is mainly used to delete resources within the cluster.
• Resource: OAuthAccessToken describes an OAuth access token. The name of a token must be prefixed with a sha256~ string, must not contain / or % characters and must be at least 32 characters long.
  • https://docs.openshift.com/container-platform/4.14/rest_api/oauth_apis/oauthaccesstoken-oauth-openshift-io-v1.html
• responseObject: In this case it is really important to analyze the responseObject to understand what was sent to the OAuth server:
  • userName foo:  This indicates that the action was requested by a user.
• requestURI: /apis/oauth.openshift.io/v1/oauthaccesstokens/sha256~LCMSmy3S* 
  • This field contains the address of the API endpoint to which the request was sent.
  • /{api-url}/{api-group}/{version}/{kind}/{token}
• Conclusion:  Foo user logout and token deletion.

Conclusion

In conclusion, effectively classifying and interpreting OpenShift audit logs is integral to maintaining a secure, compliant, and smoothly operating environment. By categorizing audit logs, organizations can streamline their monitoring processes, swiftly detect anomalies, and ensure that all activities within the system are in accordance with security policies and regulatory standards.

Interpreting audit logs accurately allows administrators to gain valuable insights into the behavior and performance of their systems. This capability not only aids in early detection of potential security threats but also enhances the ability to conduct thorough analyses in the event of an incident. Furthermore, the accountability provided by audit logs fosters a culture of transparency and responsibility among users.

References, further information, and materials used as sources for this guide are available at the following links:

• Red Hat documentation: Viewing audit logs in OpenShift Container Platform
• Kubernetes documentation: Auditing
• OpenShift documentation: Viewing audit logs
• OpenShift documentation: Configuring log forwarding
• Red Hat Blog: OpenShift Logging and Kubernetes Auditing - The mighty Duo