Security remains a top concern for developers working with containers and Kubernetes. In a just-released paper from Red Hat, the 2022 State of Kubernetes security report, 93% of respondents experienced at least one security incident in their Kubernetes environments during the previous year. And 31% of respondents attributed revenue or customer loss to these security incidents.
Innumerable tools exist—including intrusion detection systems, code scanners, and more—to improve security in an increasingly at-risk software development environment. Unfortunately, few of these are built with the unique risks and needs of a Kubernetes environment in mind. But there are options available. In this article, you'll get a comparison of eight open source security tools designed for Kubernetes:
- KubeLinter
- Kube-bench
- Kube-hunter
- Falco
- Terrascan
- Checkov
- Open Policy Agent (OPA)
- Container Security Operator
Before we dive into the details of the tools, though, let's discuss some of the general principles behind them.
Requirements for security tools
Some of the protections needed by Kubernetes are the same as those of any other network-connected host. You don't want to open unneeded ports, which can happen if you run an image that happens to contain a service such as a database that you don't need. Similarly, you don't want processes accepting web requests to run as root.
But many vulnerabilities in Kubernetes applications are unique to containers or to Kubernetes orchestration itself. For instance, it's unsafe to let a container use the same IP address as the host operating system on which the container is running. That flaw potentially allows containers to gain access to each other, thus breaking the basic rule of containerization that calls for isolation of containers. Most of the tools in this article report a vulnerability if you use the host system's address.
Now, there could theoretically be times when you do want a container to use the host's IP address. If that's the case, you can tell most of the tools in this article to let you disable specific checks like this one.
As another example of a Kubernetes-specific vulnerability, some administrators enable profiling in the Controller Manager. This option can be useful for short periods of time to troubleshoot a slow cluster, but in general, profiling should be off in the Controller Manager. Many of the tools in this article report this problem.
Some vulnerabilities arise from a failure to update an old configuration to reflect the evolution of Kubernetes. For instance, Helm Tiller used to be installed on clusters, but it was found to be buggy and is now obsolete. Still, many configurations haven't changed their old ways.
Inattention or ignorance can lead to other vulnerabilities. For instance, several tools check to make sure you don't have a password or other vulnerable information in a ConfigMap. Kubernetes provides key-protected Secrets for such information.
A rough classification
The tools in this article can be classified in a few ways. Some are static, examining your manifests (configurations). Others are dynamic, running inside clusters to check their parameters or outside the clusters to look for vulnerabilities that are visible to the world.
Among the dynamic tools, some are passive, which means they do things such as port scans to observe the Kubernetes cluster's behavior. Others are active, performing penetration testing to see how the cluster reacts to an attack.
The tools do not try to remediate problems, but just report them. Remediation would require an entire extra layer of access to the system. Furthermore, it's safer to just report a problem to a human administrator, who can then judge whether and how to fix it.
The first six tools in this report operate on manifests in one of the ways described. The last two—OPA and Container Security Operator—don't examine manifests. They offer other forms of security.
There are also different ways to interact with the tools. Some offer just a command-line interface (CLI), whereas some also offer an API for integration into automated environments.
Now we'll hone in on each tool in turn.
KubeLinter
KubeLinter is a vulnerability checker in the tradition of the venerable Unix lint command. Some of the problems KubeLinter finds are explicit security vulnerabilities, such as specifying a Secret in an environment variable or allowing unsafe privilege escalation.
But like other linters, KubeLinter also looks for questionable practices that are probably errors and might indirectly lead to security weaknesses. The tool performs other checks that could help you eliminate waste and improve performance or availability, such as checking that you are running at least three replicas and including a liveness probe.
KubeLinter runs on the command line and has an easy hook to let you run it automatically on each commit to Git. In other words, KubeLinter can be part of your automated CI/CD pipelines. It can run on a single file or a directory. It is thus a static analysis tool, like other linters.
KubeLinter's web page says that the tool is at an early stage of development, and currently offers 46 checks.
kube-bench
kube-bench is an extensive static analysis tool that draws its ideas from a benchmark created by the Center for Internet Security (CIS). Although CIS is a non-profit membership organization and offers its benchmarks free for download and implementation, the benchmarks are not open source.
It's worth noting that the Kubernetes benchmark is just one of many dozens from CIS that provide secure practices for mobile devices, operating systems (including Red Hat Enterprise Linux and CentOS), cloud vendors, and much more.
Most of the benchmarks require running a simple command to check for a running process, permission on a file, and so on. But some CIS steps can't be fully automated.
Because each version of Kubernetes requires slightly different benchmarks, the relationship between kube-bench and the CIS Kubernetes Benchmarks is in flux. kube-bench tries to determine your Kubernetes version, then runs tests defined as commands in YAML files. If you pass the tests, you can be fairly confident that your system conforms to the benchmarks (although the CIS terms of service prohibit you from saying so).
kube-bench can run statically, like KubeLinter, but can also do its scan against a running cluster. kube-bench was developed and released as open source by Aqua. The tool consists of a mix of YAML and Go language files.
kube-hunter
kube-hunter is another Kubernetes security tool from Aqua, written in Python and released as open source. It runs dynamically, with a rich collection of 23 passive and 13 active tests. You can choose which tests to run and which IP address, domain names, or networks to run them on.
KubeLinter and kube-bench are somewhat abstract, showing you things that are considered bad practices. kube-hunter is more pragmatic, running alongside your Kubernetes cluster and hammering on it with the usual intrusion detection tools such as port scanners and penetration tests.
Falco
The Falco web page describes the tool as a "Kubernetes threat detection engine." Falco actually started as a static checker for the Linux kernel, looking for classic system-level hints that something is wrong: the creation of a symbolic link, a change of ownership to a file, and so on.
Now Falco offers a powerful analysis tool for Kubernetes, hooked into Kubernetes by the Kubernetes audit logging facility. A lot of Falco checks are for pods with incorrect privileges.
The Falco project was created by Sysdig and is now an incubation project at the Cloud Native Computing Foundation (CNCF), probably the most important open source organization in cloud computing.
Terrascan
Terrascan is a dynamic analysis tool that runs as its own process and interacts with its target. Kubernetes is just one of several environments that Terrascan can work with, which also include CI/CD pipelines on GitHub and GitLab.
Terrascan integrates with Kubernetes through admission webhooks, which allow an administrator to run some external script or tool when Kubernetes receives certain types of requests.
Terrascan offers both a CLI and an API, and allows a large degree of customization. For instance, Terrascan assigns severities (low, medium, or high) to its Kubernetes policies so you can tell it to ignore low-risk activities.
Checkov
Checkov can run hundreds of scans against a Kubernetes cluster. Many of its checks, as one could expect, are similar to those in kube-bench, as well as other tools in this article.
Although Checkov was originally static, it can operate against a running cluster too. It is written in Python.
Open Policy Agent
Open Policy Agent (OPA) is not a vulnerability checker like the previous tools profiled in this article. Instead, it is a set of policies with a runtime daemon to implement them. If you spent a long time learning how to configure iptables, get ready to do it all over again. Open Policy Agent is very similar to a firewall, enforcing all kinds of limitations on activities within Kubernetes.
OPA is a very busy creature, checking resources for compliance repeatedly. The documentation offers an example where someone tries to create a host with a name already taken by another running host. OPA can catch that problem (if you've asked it to) and prevent the duplicate name from being created.
OPA is also lavish in features. You can add policies and control OPA through either an API or a CLI. OPA even offers unit testing, where you submit a request to a policy and test whether the outcome matches your expected outcome.
OPA can interact with dynamic requests from outsiders. For instance, you could authorize a user to perform some task, such as deleting a file.
The language in which OPA expresses policies, Rego, calls for a bit of explanation. Policies are expressed declaratively: One statement might check whether a host exists, and another might check whether it is running the insecure HTTP protocol. As long as each statement returns true results, the daemon proceeds to the following statement. Administrators normally start with a set of requirements and then take action at the end.
Rego is complex, but it allows easy nesting of rules so that you don't have to repeat yourself. For instance, if one set of hosts is restricted to port 80 and others to port 5432, you can define the first restriction with the name web and the second with the name postgresql. Then you can quote the appropriate rule in the definition of each host. If you decide to switch the web port to 8080, you can do it in the rule instead of having to rewrite the definition of each host.
While you're grappling with Rego, treat your work as a walk-on part in a historical re-enactment. Rego's principles and syntax go back to the Prolog language of the early 1960s, often declared to be the first declarative language. Because both Kubernetes manifests and modern configuration tools (Ansible, Puppet, Chef, etc.) are declarative, take pleasure in returning to the foundations of a proud tradition.
Clair and the Container Security Operator
Clair falls under the category of static analysis tools. It creates an inventory of all dependencies used by a container image, scanning the image to make an inventory of all the applications, operating system components, and libraries installed. Like some of the other tools in this article, Clair checks each dependency against public databases that list software vulnerabilities. It also updates the inventory whenever a new vulnerability is introduced, so that you are informed of the new vulnerabilities not only when you build your image, but when a vulnerability is reported that could affect your image.
In short, once you run your container image through Clair, it keeps track of changes to both your code and the vulnerability listings applied to all of your dependencies.
Clair was created by the team that created Quay.io, and is therefore designed to work with container registries. It has been adopted by many organizations, who use it to check their own applications and libraries, storing its inventories on their own systems.
Kubernetes-related vulnerabilities can be checked through the Container Security Operator. Both Clair and the Container Security Operator are open source projects started by Red Hat.
Conclusion
Kubernetes is a complex environment with many vectors for attack. The tools in this article help you secure your applications and containers in different ways, helping to reduce your attack surface and overall risk.
Last updated: July 21, 2022