eBPF is a low-level technology for running sandboxed kernel programs. Its use in Kubernetes-based applications has grown rapidly due to the revolutionary capabilities it enables, as demonstrated by Cloud Native Computing Foundation (CNCF) projects such as Cilium, Datadog, Calico, and Pixie. However, eBPF in Kubernetes also poses a number of new challenges for developers and administrators alike. These challenges include program lifecycle management issues, the widespread use of privileged pods, the lack of eBPF subsystem visibility, and problems with program cooperation.
bpfman is an open source project dedicated to making eBPF easier to secure, manage, and use. The project includes an operator that allows eBPF application developers to package programs via OCI container images and deploy them via Kubernetes CustomResourceDefinitions (CRDs) such as TcPrograms and XdpPrograms. Additionally, it allows cluster administrators to tightly control which users can/can not deploy specific types of programs.
As of Red Hat OpenShift 4.16, bpfman will be available in Developer Preview status, meaning users will be able to install the bpfman operator from Operator Hub and start working and experimenting with the platform. This article will show exactly how to get up and running.
Traditional eBPF-enabled applications on OpenShift
Today across the OpenShift ecosystem there are numerous examples of internal and external applications utilizing eBPF. Core features such as the IngressNodeFirewall, NetworkObservability, and AdvancedClusterSecurity already make use of the technology in their implementations. The architectures for these applications have traditionally looked pretty similar, ultimately resembling the diagram in Figure 1.
In this model, eBPF programs are compiled and embedded into the userspace binaries that deploy them. Userspace applications traditionally make use of one of the provided eBPF loading libraries in order to load, attach, manage, and interact with their associated eBPF programs.
From a functional standpoint this works well enough; however, there are a few drawbacks to such a deployment model:
- All the applications need at least the
CAP_BPFLinux capability to interact with the eBPF subsystem, which should essentially be treated as root. In the worst case scenario, programs may need more broad-ranging capabilities that allow them to trace other processes, create network interfaces, and perform other potentially damaging tasks. - Every application needs to maintain duplicate logic for eBPF resource management such as program loading/attaching and eBPF file system management.
- eBPF has challenges with program cooperation. For example, some software using eBPF assumes exclusive use of an eBPF hook and can unintentionally eject existing programs when being attached, which is an even bigger issue in a platform like OpenShift which can facilitate running hundreds of different applications.
- No fine-grained versioning control of the eBPF program in relation to its accompanying userspace component.
eBPF-enabled applications on OpenShift with bpfman
When the bpfman-operator is deployed on OpenShift, applications can delegate eBPF program lifecycle and management responsibilities to bpfman, and the existing architecture can transform into the following (Figure 2).
With this new deployment model comes many benefits, including:
bpfmanacts as the central privileged access point, allowing applications to run as non-root.- Administrators can use traditional RBAC to tightly control which program types a given user/service account is allowed to deploy.
bpfmancan manage program cooperation at a cluster-wide level.- eBPF programs are packaged separately from the application in OCI container images according to the specification provided by the bpfman community allowing for fine-grained versioning and signing control.
- Applications can continue to use existing management libraries for interacting with the eBPF maps associated with their programs.
Now that we’ve gone over the current landscape, it’s time to get up and running with the bpfman-operator.
Install the bpfman-operator
Starting with an OpenShift cluster, users can easily deploy bpfman and its dependencies via the OperatorHub within OpenShift’s console. Start by navigating to the console’s OperatorHub page and finding the bpfman-operator in the listings, as shown in Figure 3.
After choosing the bpfman-operator from the listing page, make sure the installation namespace is correct by selecting "Create Project", and making a project named bpfman, as shown in Figure 4.
Finally, click Install to continue (Figure 5).
When everything has been installed correctly, the bpfman-operator, the bpfman-daemon, and the security-profiles-operator pods will be running in the bpfman namespace. The security profiles operator is deployed alongside bpfman to manage SELinux profiles which allow unprivileged userspace applications to work with their programs via eBPF Maps. See Figure 6.
Deploy an example eBPF-enabled application with bpfman
Once the operator has been successfully installed, bring up a terminal window that has the oc binary installed to deploy an application by simply using example manifests provided by the community. The example manifest below will deploy an eBPF application that makes use of an XDP program to count the number of packets flowing through a pod’s primary network interface. Other examples can be found in the Release Assets, load any of the *-install-selinx.yaml files.
# oc create -f https://github.com/bpfman/bpfman/releases/download/v0.4.2/go-xdp-counter-install-selinux.yaml
namespace/go-xdp-counter created
serviceaccount/bpfman-app-go-xdp-counter created
clusterrolebinding.rbac.authorization.k8s.io/xdp-binding created
daemonset.apps/go-xdp-counter-ds created
xdpprogram.bpfman.io/go-xdp-counter-example created
selinuxprofile.security-profiles-operator.x-k8s.io/bpfman-secure createdThis will create a full-fledged eBPF-enabled application which includes:
- An
XdpProgramCRD to specify the containerized bytecode image containing the program, the priority of the program on a given interface, the node(s) where the program should be deployed, and the function name of the program.
apiVersion: bpfman.io/v1alpha1
kind: XdpProgram
metadata:
labels:
app.kubernetes.io/name: xdpprogram
name: go-xdp-counter-example
spec:
bpffunctionname: xdp_stats
bytecode:
image:
url: quay.io/bpfman-bytecode/go-xdp-counter:v0.4.1
interfaceselector:
primarynodeinterface: true
nodeselector: {}
priority: 55- An application DaemonSet, which specifies the userspace application’s deployment for each node as well as what eBPF maps it needs access to.
apiVersion: apps/v1
kind: DaemonSet
metadata:
...
spec:
...
template:
...
spec:
containers:
- ...
image: quay.io/bpfman-userspace/go-xdp-counter:v0.4.1
imagePullPolicy: IfNotPresent
name: go-xdp-counter
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsGroup: 65534
runAsUser: 65534
seLinuxOptions:
type: bpfman-secure_go-xdp-counter.process
volumeMounts:
- mountPath: /run/xdp/maps
name: go-xdp-counter-maps
readOnly: true
nodeSelector: {}
securityContext:
fsGroup: 65534
runAsNonRoot: true
serviceAccountName: bpfman-app-go-xdp-counter
...
volumes:
- csi:
driver: csi.bpfman.io
volumeAttributes:
csi.bpfman.io/maps: xdp_stats_map
csi.bpfman.io/program: go-xdp-counter-example
name: go-xdp-counter-mapsThe example application uses bpfman to deploy its XDP eBPF program and to get unprivileged access to the generated maps via a CSI ephemeral volume. After the example is properly deployed, it’s simple to check on the status of the XdpProgram.
# oc get xdpprogram go-xdp-counter-example
NAME BPFFUNCTIONNAME NODESELECTOR STATUS
go-xdp-counter-example xdp_stats {} ReconcileSuccessFollowing verification that the program has in fact been reconciled successfully make sure all the application pods are running.
# oc get pods -n go-xdp-counter
NAME READY STATUS RESTARTS AGE
go-xdp-counter-ds-9ddgh 1/1 Running 0 36s
go-xdp-counter-ds-gbzsj 1/1 Running 0 36sThen simply dump the application’s pod logs to see how many packets have been counted by the XDP program:
# oc logs go-xdp-counter-ds-9ddgh -n go-xdp-counter
2024/06/24 02:45:29 3614 packets received
2024/06/24 02:45:29 23754428 bytes received
2024/06/24 02:45:32 3997 packets received
2024/06/24 02:45:32 23890444 bytes received
2024/06/24 02:45:35 4447 packets received
2024/06/24 02:45:35 24356469 bytes received
2024/06/24 02:45:38 4998 packets received
2024/06/24 02:45:38 24791765 bytes received
2024/06/24 02:45:41 5154 packets received
2024/06/24 02:45:41 24867116 bytes received
2024/06/24 02:45:44 5392 packets received
2024/06/24 02:45:44 25175031 bytes received
...Congratulations, you have now deployed your first eBPF-enabled application with bpfman on OpenShift!
Roadmap
bpfman is dedicated to constantly evolving based on upstream best practices, and you can view current and future work items in the upstream GitHub tracking project. Some proposed features include:
- First class multi-architecture support
- eBPF TCX attach type support
- Ability to attach networking programs via pod selectors
- Exportation of eBPF subsystem metrics and events
- Using the eBPF Token API to secure eBPF applications that either compile programs on the fly or require more fine-grained control over how their programs are loaded.
Additionally, the community has been accepted as a CNCF sandbox project, a larger community we look forward to working with.
For Red Hat Openshift 4.17, bpfman will be delivered as a Technical Preview within the new "eBPF manager" feature.
Get involved
To start learning more about the project please checkout the project website at bpfman.io. bpfman is completely open—feel free to open issues, start discussions, or more generally reach out to the reach out to the upstream community on Github or within Kubernetes Slack in the #bpfman channel. Any and all contributions are welcome and appreciated.
