Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Best practices for successful DevSecOps

June 15, 2022
Ajmal Kohgadai Andy Oram
Related topics:
DevOpsDevSecOpsKubernetes
Related products:
Red Hat OpenShift

Share:

    In a recent Red Hat survey, summarized by our 2022 State of Kubernetes security report, a broad majority of the more than 300 correspondents reported that they are incorporating security into DevOps processes and pipelines, a practice called DevSecOps.

    This article draws on the report as well as a Red Hat white paper, How to automate DevSecOps in Red Hat OpenShift, to show the rate of DevSecOps adoption and the factors that we believe make it work well.

    The popularity of DevSecOps

    Securing containers and Kubernetes often requires multiple teams to collaborate across the application lifecycle. As applications move from dev and test to production runtime environments, new security use cases arise requiring different skill sets and responsibilities. At the build stage, for example, developers are often the primary guardians of security, ensuring that container images adhere to security best practices and catching fixable vulnerabilities early. The runtime phase introduces new attack vectors requiring a different set of security controls, such as identifying anomalous process execution indicative of a breach.

    To better understand the delineation of security responsibility, we asked survey respondents, "What role at your organization is most responsible for container and Kubernetes security?" Nearly two-thirds of the more than 300 respondents said that it was the role of DevOps (43%) or DevSecOps (19%). DevOps and DevSecOps are both roles that bridge different functions, which further underscores the need to collaborate across teams to properly protect containerized applications in Kubernetes.

    The responses show that modern organizations are well along the way to incorporating security into their DevOps pipelines. Only 16% of respondents assign primary responsibility to their security team, another 16% to operations, and 6% to developers.

    One surprise finding was how few respondents assign responsibility to the central IT security team for Kubernetes security. One way to explain the low percentage is by looking at what motivates container and Kubernetes adoption. Developers and DevOps teams are often the first to adopt container technologies, limiting them to the dev and test stages. This is done outside the purview of central IT governance (a workaround also known as shadow IT). To bridge the gap between the official security team and the developers, security tooling for container and Kubernetes must facilitate close collaboration among different teams—from developers to DevOps to operators to security—instead of perpetuating team isolation that can plague organizations with slow roll-outs and security vulnerabilities.

    Best practices for DevSecOps

    In the following subsections, we present suggestions for each of the three most common DevOps stages: build, deploy, and runtime.

    Build stage

    Here, your goal is to catch issues early so they don't become blockers later in the development life cycle:

    • Use a trusted, enterprise-grade private image registry and limit access.
    • Use minimalist base images.
    • Update base images and their dependencies when new versions are available.
    • Remove exploitable and nonessential software such as:
      • Package managers (apt, yum, apk)
      • Network tools and clients (curl, wget, netcat)
      • Unix shells, compilers, and debuggers
    • Scan the image registry regularly for known operating system, application, and language vulnerabilities.
    • Integrate image scanners with CI tools and build vulnerability awareness early, especially when there are severe policy violations.
    • Follow best practices for sensitive data:
      • Do not use secrets in a risky manner.
      • Do not store secrets unencrypted.
      • Do not use secrets as environmental variables.
      • Understand where and how secrets are being used to remove unnecessary exposure.
    • Make sure that builds break if they exceed the security risk threshold. This would include builds that:
      • Contain high severity, fixable vulnerabilities.
      • Have not been scanned recently, or at all.
      • Contain misconfigurations missed by a scanner, such as identity configurations or environment variables.

    Deploy stage

    This stage should automate a DevSecOps feedback loop and prioritize remediation:

    • Combine security-relevant data from the build stage with the deployment configuration to determine the security risk of each deployment, including:
      • Image vulnerabilities
      • Access to secrets, storage, etc.
      • Privileges and capabilities
      • Workload isolation, network exposure, and blast radius
    • Use a secrets management tool to protect sensitive data. Assess the privileges used by containers to keep them to a minimum viable set of capabilities.
    • Avoid deployments without resource limits unless absolutely necessary.
    • Annotate deployments with the name, email alias, or chat channel of the team responsible for the application.
    • Block risky deployments and alert the correct team for automated and streamlined remediation.

    Runtime stage

    By this stage, the pressing need is to prevent, detect, and contain runtime attacks:

    • Implement dynamic scanning to detect vulnerabilities in running containers.
    • Use behavioral baselining and process allow-listing to identify unusual runtime activity, such as:
      • Privilege escalation
      • Unauthorized network flows
      • Cryptomining
      • Malicious process execution or other exploits
    • Mitigate threats with Kubernetes-native controls:
      • Scaling to zero
      • Killing pods and restarting

    Further best practices

    We also recommend the following general practices:

    • Private, internal registries often offer greater security capabilities. Choose a registry that offers advanced access control and built-in vulnerability scanning.
    • Use trusted content. To start with, use resilient, minimalist base images to build your containers.
    • The security of your continuous integration (CI) build infrastructure is as important as your production environment. Implement security as code using CI/CD. Limit administrative access and allow only required network ingress.

    Conclusion

    Tools such as image scanners and registries can provide governance to projects and detect vulnerabilities. However, reducing risk from misconfigurations (such as when your containers run with root privileges) and runtime incidents requires processes and best practices that implement DevSecOps across the full application life cycle.

    Organizations experience security incidents across all phases: build, deploy, and runtime. Implementing and automating DevSecOps provides developer-friendly guardrails that can decrease user error at the build and deploy stages and protect workloads at runtime.

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    What’s up next?

    Getting GitOps e-book card

    Learn how to navigate the complex world of modern container-based software development and distribution with Getting GitOps: A Practical Platform with OpenShift, Argo CD, and Tekton.

    Download the e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue