External IP visibility in Red Hat Advanced Cluster Security

Red Hat Advanced Cluster Security for Kubernetes 4.8 introduces several new capabilities, including external IP visibility. This feature empowers security teams to identify the exact external IP addresses deployments communicate with, enhancing their understanding of external and internal connections and validating network policies. This article provides an overview of how to get started with external IP visibility in Red Hat Advanced Cluster Security.

New capabilities

Now Red Hat Advanced Cluster Security for Kubernetes 4.8 is available and includes several new capabilities:

• External IP visibility (GA)
• OpenShift infrastructure compliance (GA)
• Enhanced vulnerability advisories
• Support for keyless signing verification

These are exciting features with the support for external IP visibility in Red Hat Advanced Cluster Security 4.8. This enables security teams to understand the exact external IP addresses deployments communicate with and to better understand external (and internal) connections while validating network policies. This feature was developed with potential threat actors in mind, designed to combat the command and control (C2) techniques previously tried in Kubernetes environments. 

Our users understand the threat of malware, particularly malware associated with persistent threats designed to beacon out, or send outgoing packets, to contact C2 servers and receive additional payloads/capabilities, or simply advertise their status and where they have been deployed. Pinpointing the precise external IP addresses communicating with potential malware (possibly introduced into containers via supply-chain compromise) can offer security teams vital data. This information is vital to detect malware command-and-control (C2) activity and facilitate challenging investigations.

Demo requirements

Red Hat Advanced Cluster Security uses a hub-and-spoke architecture, shown in Figure 1. This diagram depicts the Red Hat Advanced Cluster Security hub-and-spoke architecture with central services handling manifests, scans, and alerts connected to secured cluster services, including sensors, admission controllers, and collectors on nodes across multiple clusters A, B, and C.

Central services manages the Red Hat Advanced Cluster Security policy manager, API, and user interface. It also contains the Red Hat Advanced Cluster Security vulnerability scanner. Secured cluster services which contain the sensor, collector, and enforcement (admission controller) deploy onto the spoke clusters. The sensor contains an extended Berkeley packet filter (eBPF) probe that interacts with the worker node kernel for Red Hat OpenShift and gathers information on container process execution and container network traffic flows.

Prerequisites for this demo:

• An OpenShift cluster with node requirements of 4vCPU, 8 Gb and 200 Gb of storage.
• Red Hat OpenShift administration access via the CLI or OpenShift console.
• Access to the demo repository.

Install Red Hat Advanced Cluster Security

For this example, you will deploy the central services and secured cluster services to a single OpenShift cluster. However, for multi-cluster use cases, we recommend creating a Red Hat Advanced Cluster Security central service as a hub, and install the Red Hat Advanced Cluster Security secured cluster services on spokes or separate clusters.

To start, install Red Hat Advanced Cluster Security for Kubernetes using the operator from the OpenShift operator hub. Either follow the documentation or the arcade to help with the installation process.

In the next step, we’ll assume that Red Hat Advanced Cluster Security deployed correctly. To verify this, go to Platform configuration -> Clusters. You should see the name of your cluster with a healthy status and all green checkmarks on the board, as shown in Figure 2. 

The collector service represents the eBPF probe running on OpenShift nodes and provides network traffic flow data we can use to identify external IP connections. We should verify it has a healthy green tick, shown in Figure 2.

Configuring external IP visibility

Now that we have Red Hat Advanced Cluster Security for Kubernetes correctly installed, and the collector service is healthy,  we can configure the external IP visibility. Fortunately, this is a really simple change. Using the CLI, we just need to create a new configmap by running the following command:

oc create -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: collector-config
  namespace: stackrox
data:
  runtime_config.yaml: |
    networking:
      externalIps:
        enabled: ENABLED
EOF

Alternatively, you can copy the configmap and paste it directly into the OpenShift console. The following demo will walk you through the process.

Deploy a sample application

Now that we have configured the external IP visibility, let’s see it in action.

Deploy a very simple .NET Core application to OpenShift. This application exposes an interface via an OpenShift route where users can see the current leaders of a space arcade game. 

The following commands clone the source repository, set a new project, and deploy the application to that project. Run these commands on the command line with admin access to your OpenShift environment:

git clone https://github.com/shaneboulden/tailspin-argocd-cd.git netcoreapp
oc new-project tailspin
oc apply -k netcoreapp/overlays/development

Once deployed, the OpenShift Networking -> Route page will show the OpenShift Route available for the web application (Figure 3).

Under Location, click the OpenShift Route to verify the page is up and running.

Figure 4 shows a web application interface entitled "Space Game," featuring space-themed cartoon illustrations like satellites, planets, and an alien, including a Download game button and empty leaderboard placeholders.

Investigate an external connection

Next, generate traffic to simulate normal application behavior. Run the following on the command line.

while true; do curl -k -s "https://$(oc get route web -n tailspin -o json | jq -r '.spec.host')" > /dev/null; sleep 1; done

Open the Red Hat Advanced Cluster Security central console and navigate to the Network tab. Select the tailspin namespace from the drop-down menus, and you should see a graph representing normal web traffic for this application (Figure 5).

The Red Hat Advanced Cluster Security network traffic graph shows that:

• The web deployment accepts incoming connections from the openshift-ingress namespace (user access to the web page).
• The web deployment makes DNS requests to the CoreDNS service running inside OpenShift in the openshiftdns namespace.
• The leaderboard service only accepts incoming requests from the web pods and doesn’t make any outbound connections.

Now that we understand what normal looks like, let’s generate external web traffic. Create a terminal session in the leaderboard pod’s sidecar container and make an outgoing connection to redhat.com. Run the following command on your command line:

while true; do oc exec deploy/leaderboard -n tailspin -c ubi -- curl -s -L https://redhat.com > /dev/null; sleep 3; done

Now observe the additional traffic shown in the Red Hat Advanced Cluster Security console (Figure 6). This network graph in the console illustrates anomalous external traffic from the leaderboard deployment in the tailspin namespace to an external entity with a sidebar listing external entities and flow details.

If you select the external networks in the network view (Figure 7), you can now see the external IPs, and running a simple who.is search shows that the Akamai CDN fronts redhat.com.

Figure 8 is the result from the who.is website for the IP address 23.198.52.225, showing that the IP address is associated with Akamai Technologies and represents the content delivery network (CDN) that supports redhat.com.

How to protect your applications

Setting up notifications and enforcement for anomalous traffic flows is straightforward with Red Hat Advanced Cluster Security for Kubernetes. Users can craft a policy that informs on anomalous network traffic and can even kill the running pod. This arcade illustrates the behavior.

Wrap up

In this article, we provided a brief overview of how to get started with external IP visibility using Red Hat Advanced Cluster Security for Kubernetes. This is a new capability in 4.8 that helps organizations better identify external connections for cloud-native workloads, and potentially identify workloads beaconing to external entities or attempting data exfiltration.