In a recent Red Hat survey of more than 300 production-level Kubernetes users, 93% of respondents admitted to experiencing at least one security incident in their Kubernetes environments during the previous year—and 31% of respondents say they experienced revenue or customer loss during that year as a consequence.
Our industry can do better. This article summarizes findings from the survey, reported in our 2022 State of Kubernetes security report, and highlights the weak points of Kubernetes security today along with a path forward involving DevSecOps.
Note: Most of the material in this article comes from the Red Hat report, 2022 State of Kubernetes security.
Security is one of the top concerns with container strategies
With security incidents in these environments being a nearly universal experience, it should come as no surprise that security remains a top concern among survey respondents when it comes to container strategies.
Taken together, concerns about security threats to containers (14%) and a lack of investment in container security (17%) show that security is the top concern with container strategies, at 31%. Another 22% of respondents cited the slow pace of progress as their biggest concern, while 20% consider skills gaps to be the thing that worries them the most (Figure 1).
Why are organizations suffering from so many production security problems?
Nearly 53% of respondents have experienced a misconfiguration incident in their environments over the previous year (Figure 2). Among all respondents, 38% discovered a major vulnerability, and 30% said they have suffered a runtime security incident. Lastly, 22% said they had failed an audit.
A combination of factors are likely behind the disruption, including:
- A lack of security knowledge about containers and Kubernetes
- Inadequate or unfit security tooling
- Inability of central security teams to keep up with fast-moving application development teams.
Runtime—sometimes also known as Day 2 operations or the post-deployment stage—is the container life cycle phase that organizations worry about the most: This phase is highlighted by 57% of respondents. This concern seems counterintuitive, given that an overwhelming majority of respondents identify misconfiguration as the source of the biggest security risk and have experienced a misconfiguration incident more often than any other type of security incident.
Behind the respondents' concern for the runtime phase lies the origin of bugs in the build or deploy stage. The negative impact of a security misstep at the build or deploy stages is likely to be felt only after an application is running in production. Incident response, a key aspect of security, is also more complicated at runtime. Lastly, security issues discovered at runtime are likely more costly to fix as well. Together, these factors make heightened runtime security worries more understandable.
Kubernetes and containers, while powerful, were designed for developer productivity, not necessarily security. Default pod-to-pod network settings, as an example, allow open communication to quickly get a cluster up and running, at the expense of security hardening.
Respondents worry about misconfiguration above all other security concerns
Kubernetes is a highly customizable container orchestrator, with various configuration options that affect an application’s security posture.
Consequently, respondents worry the most about exposures due to misconfigurations in their container and Kubernetes environments (46%)—nearly three times the level of concern over attacks (16%). Vulnerabilities constitute the second-leading cause of worry (28%). See Figure 3.
Configuration management poses a uniquely difficult challenge for security practitioners. The persons responsible for configuring workloads might not understand the security implications of various settings within Kubernetes. Although a host of tools are available for vulnerability scanning of container images, configuration management has no such automated help. Each configuration option requires consideration and will likely be unique to organizations and teams, depending on their risk tolerance and level of workload sensitivity.
People avoid deploying the Kubernetes dashboard because it indulges poor development practices: the dashboard rewards one-off activities instead of more robust automation. But security measures, such as configuring a pod’s security context or implementing Kubernetes role-based access control (RBAC), are challenging and tempt administrators to resort to the dashboard.
The best way to address this challenge is to automate configuration management as much as possible, so that security tools—rather than humans—provide the guardrails that help developers and DevOps teams configure containers and Kubernetes more securely.
Security concerns hinder innovation
Modern development practices (notably DevOps) are running head-on against traditional security practices, and both practices need to adapt.
Organizations are transforming how they build, run, and scale applications by adopting cloud-native technologies such as Kubernetes and microservices-based application architectures. Some are building all new applications as microservices, while others refactor existing applications alongside pre-existing monoliths. Organizations therefore must also transform their security across their cloud-native stack. The term DevSecOps represents the increasingly popular incorporation of security measures into DevOps pipelines.
More rapid release cycles, faster bug fixes, and greater flexibility to run and manage applications across hybrid environments are three of the most often cited benefits of containerization. However, when security becomes an afterthought, you could end up negating the greatest gain of containerization: agility. The majority of survey respondents (55%) have had to delay an application rollout because of security concerns over the last 12 months (Figure 4).
New technologies often create unforeseen security challenges. Some organizations are overwhelmed by security needs that stretch across all aspects of the application life cycle, from development through deployment and maintenance. They need a simplified way to protect their containerized applications without slowing development or increasing operational complexity.
Conclusion
A recent study revealed that human error was a major contributing factor in 95% of breaches. The best way to improve security is to automate best practices, but as we've seen in this article, automating security in a declarative, YAML-driven configuration is complex. Cooperation among developers, operators, and security teams is needed to get all the configuration operations in place. Working on a platform that provides a secure environment and tools can get you to your desired level of protection faster.