Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Book Review - "Kubernetes Secrets Handbook"

July 8, 2024
Amy Fredj
Related topics:
Security
Related products:
Red Hat OpenShift Container Platform

Share:

    I’m an avid and quick reader. Except when it comes to text books, which are too reminiscent of my hectic undergrad and post-graduate days. So, I was a bit reluctant to embark on the book that Rom Adams shared with me (authored by him, Chen Xi and Emmanouil Gkatziouras), “Kubernetes Secrets Handbook” (available here: https://www.amazon.com/Kubernetes-Secrets-Handbook-production-grade-management/dp/180512322X). But, this book read like a thriller, addressing a broad and diverse audience: from novices in the world of Kubernetes and security to those who are more experienced; from technical to laymen; and from text-book-phobics like me to people who have these books on their nightstands for light nighttime reading.

    An Open Book

    First of all: the introduction of the authors and reviewers helped to assure me that this book would be a serious review of the state of the art, and not some marketing mumbo-jumbo. The authors and reviewers are all respected in their industry, with years of experience.

     I loved the opportunity to practice and implement real examples, but I equally appreciated that this is not mandatory, and that I was debriefed after every example.

     Finally, the chapters progress and build on each other, never assuming that the layman (or the novice) have a required background. This showed respect for my experience, without giving the feeling that I was wasting my time on subjects already familiar to me.

     The author gently led us through Kubernetes concepts (see page 10 for list of components), by showing the evolution from legacy through virtualization and mircroservices – culminating with the orchestrated containers, demonstrating all of these with fantastic diagrams and flows that explain what is happening (see Figure 1.2 on page 10 and Figure 1.3 on page 13).

    But Some Secrets Are Good 

    Having described the above, the basic security configuration is explained, as an opportunity to describe briefly some of the vulnerabilities. The authors point out that there are proposed solutions, primarily using some external vault. However, the authors admit that the vault solutions also have some problems:

    • These solutions address the application level, and not the ConfigMap or cross-application services.
    • Resilience

    The challenge of where and how to store secrets remains regardless of how the secrets are created. There are various tools to address the needs of secure storage of these secrets, such as Google GCP Secret Manager and AWS Secret Manager. Orthogonal to the question of *where* to store the secrets is the need to ensure that they are encrypted.

    The elusive threat of storing secrets in the etcd database (the key-value storage) is glossed over with a seemingly benign comment buried in Chapter 3: 

    “Data in payloads written in etcd is not encrypted but encoded in base64, which is almost equivalent to clear text. Encrypting the data contained in the payload will protect from the aforementioned protection mechanisms, but not replace them!”

    A Delicate Balance

    But now that we’ve been gently led to this discovery, in Chapter 3, the authors introduce us to the KMS framework available within the kube-apiserver.

    • Native encryption – enables local encryption, with a random encryption user-generated key. No additional software is required. However, this comes at the price of basic risks that a hacker could penetrate the system, retrieve the encryption key – and –voila – he has access to the etcd file.
    • And that’s why the authors suggest using native encryption with an external component, either on the control plane or externally. Kubernetes kms provider (currently based on KMSv2 as a KMS provider for kube-apiserver). The Kubernetes KMS provider plugin interconnects the kub-apiserver with external KMS(s) such as Hashicorp vault. In this solution, the kube-apiserver generates a Data Encryption Key (DEK), and sends it via the KMS plugin to the KMS server to encrypt with a Key Encryption Key (KEK). The flow is simply and neatly presented, as are all flows in the book.

    The authors then introduce us to a plugin which can be used to extend the kube-apiserver capabilities: Trousseau. From insider information, we also learned that other plugins are being developed in the community as we speak.

    The authors warn us that this is not enough! Once again – using a simple diagram (Figure 3.5), they show the threat surfaces where potential breaching could occur:

    • Payload data
    • Network (transport)
    • Operating system
    • File system
    • etcd

    As this is not the focus of the book, we are pointed to other sources for information, but a short discussion is shared in Chapter 3.

    Bringing It All Together

    Chapter 4 is a cookbook for designers looking to debug and troubleshoot Kubernetes secrets. The chapter is well-laid out, but – full disclosure – I only skimmed through it, as this was not my goal in reading the book.  Nonetheless, the tips and tricks are valuable. What I found most intriguing was the human element: it is often so easy to get lulled into a false sense of security when you troubleshoot: backdoors are opened, secrets are leaked, and, well, people are people. So – the authors wisely warn us to be cautious here, too, and remember that our goal is security!

    Part II of the book is an extremely practical section: how to apply the theoretical background we got to a real live production environment. We are invited to a discussion about security, auditing and compliance. The obvious that must be stated, to understand the environment in which we are practicing: 

    “Most organizations assign the security domain to their respective IT departments by designating a set of limited individuals almost working in isolation.” (To me – it felt like he was accurately describing some of my colleagues – and some of my customers!).” 

    The problem with this approach is that these individuals tend to have a deep understanding of security, but not as good a grasp on operational reality. The delicate balance between the two is achieved by studying risk management.

     We then revisit the diagram we had seen earlier, adding a dimension of discovery, assessment, analysis and execution: a DevOps approach on steroids – which grows to DevSecOps. Thus, the tools we had seen in earlier chapters are enhanced with tools that will assist us in implement this, including: Trivy, kube-bench, Compliance Operator and others.

     In Chapter 6, we learn about the relevance of Disaster Recovery in the context of security. The authors remind us (again – of the obvious): Loss of data, resulting from a physical or logical disaster, is always a nightmare. But the fear of losing secrets, keys and passwords, similar to Dorothy’s fears on the way to Oz (lions and tigers and bears, oh my!), is well founded. This could result in loss of functionality and possible loss of service! These are not fairy tales; they really happen in real life! Often, this kind of DR is managed by an external KMS provider.

    In the last chapter of Part II, we learn about some challenges and risks in secret management. Again – this was presented very methodologically, presenting the issues we would be likely to encounter in each phase. This is followed by a fascinating section about the real-world security risks (I told you, this book is a thriller!!). These risks include:

    • Secret zero – the paradox of the single password that protects all the other passwords, that the authors call the “key to the kingdom”. Protecting this single password comes with some significant challenges: Management complexity, compliance issues and enhanced risk of malicious insiders.
    • Secret access ballooning – access granted to too many users.
    • Secret sprawl – too many secrets n the system
    • Secret island – also known as shadow IT. As the other challenges present themselves (or as the solutions to those challenges become more complex), users may find alternative ways to access the system.

    But Real Life Happens

    In the last part of the book, the reader is introduced to alternatives for external secret stores, along with practical, hands-on examples of how to use them:

    • Cloud Secret Store on AWS
    • Cloud Secret Store on Azure
    • Cloud Secret Store on GCP
    • HashiCorp Vault
    • CyberArk Secrets Manager

    Continuing with the standards they set for themselves throughout the book, the authors present real world case studies.

     A Call to Action

    The book concludes with a discussion on the future of secret management in Kubernetes, and an invitation to join the conversation. This end is fitting on so many levels: as RedHat affiliates, the authors embody the culture of community as a strategy. While reading the book, I felt like I was being invited to a parlor meeting with friends, where we would strategize about the future of security, and our impact to this corner of the world. Indeed, that is what this is – and I invite you all to join this endeavor, too. 

    Last updated: July 10, 2024
    Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.

    Recent Posts

    • Our top 10 articles of 2025 (so far)

    • The benefits of auto-merging GitHub and GitLab repositories

    • Supercharging AI isolation: microVMs with RamaLama & libkrun

    • Simplify multi-VPC connectivity with amazon.aws 9.0.0

    • How HaProxy router settings affect middleware applications

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue