Automation and standardization are fundamental for any company. Zero-touch provisioning (ZTP) is one of the key solutions for deploying Red Hat OpenShift on bare metal. In this article, we analyze how ZTP revolutionizes the deployment and provisioning of OpenShift clusters, with particular attention to integration with Red Hat OpenShift Virtualization.
The value of ZTP
Day 0: Pre-installation automation
During the Day 0 phase, before the cluster installation, ZTP allows you to automate the preparation of the infrastructure through ClusterCurator and Red Hat Ansible Automation Platform:
This approach allows you to:
- Automatic networking configuration.
- Bare metal infrastructure preparation.
- Prerequisites verification.
- Pre-installation health checks performing.
Day 1: Automatic deployment
ZTP transforms the OpenShift deployment process from a manual and error-prone operation into a fully automated process. The main advantages include:
- Standardization: All clusters are deployed with identical configurations.
- Repeatability: The process can be infinitely replicated with consistent results.
- Speed: Significant reduction in time-to-market for new clusters.
- Reliability: Elimination of human errors in the installation process.
Day 2: Management and scalability
The true power of ZTP is manifested in the Day 2 phase, where:
- Scalability: Automatic extension of new nodes to the cluster.
- Policy management: Centralized implementation of safety and compliance policies.
- Consistency: Maintenance of consistency across multiple clusters.
- Automation: Automated updates and patches management.
GitOps architecture for managing distributed OpenShift edge clusters.
Figure 1 illustrates a GitOps-based deployment model using a central Hub cluster to manage OpenShift clusters across multiple network far edge sites, which can include varying configurations such as single-node OpenShift, three-node clusters, or standard clusters with multiple control and compute nodes.
Policy management
The Policy Generator is a fundamental tool of Red Hat Advanced Cluster Management for Kubernetes that allows us to define and manage policies in a declarative way. It works as a policy generator that converts YAML definitions into actual Kubernetes policies, simplifying the management of complex configurations across multiple clusters.
How the PolicyGenerator works
The PolicyGenerator operates through three main components:
- PolicySets: Assembling related policies and defining where they should be applied through placement rules.
- Placements: Determining on which clusters the policies should be applied.
- Policy definitions: Defining individual policies and their configurations.
Here is a basic example of how to use it:
apiVersion: policy.open-cluster-management.io/v1
kind: PolicyGenerator
metadata:
name: policies
policySets:
...
# array of policySets to define
policies:
...
# array of policies to defineRecommended organization policy
The following folder structure represents a recommended approach for organizing policies, but you can adapt it based on the specific needs of the organization. This organization is designed to scale well following the increase of policies and managed clusters and to improve the maintainability of the released code:
policies
├── Global # Global configurations for all clusters
│ ├── base-config # Essential base configurations
│ │ ├── policy-chrony.yaml # Time synchronization
│ │ ├── policy-custom-ca.yaml # Custom CA certificates
│ │ ├── policy-kubelet.yaml # Kubelet configuration
│ │ └── policy-ssh-key.yaml # SSH key management
│ ├── day2-config # Post-installation configurations
│ │ ├── policy-alertmanager-customrule.yaml # Alerting rules
│ │ ├── policy-file-integrity-operator.yaml # File integrity monitoring
│ │ ├── policy-ingress-certificate.yaml # Ingress certificates
│ │ └── policy-storagecluster.yaml # Storage configuration
│ ├── hub-config # Hub-specific configurations
│ │ ├── policy-clusterlogging.yaml # Centralized logging
│ │ ├── policy-enable-ceph-toolbox.yaml # Ceph tools
│ │ └── policy-storagecluster.yaml # Storage configuration
│ ├── kustomization.yaml # Kustomize file
│ ├── policy-generator.yaml # Policy generator
│ ├── registry # Registry configurations
│ │ └── policy-registry.yaml # Internal registry policy
│ ├── secrets-config # Secret management
│ │ ├── policy-external-secrets.yaml # Integration with external vaults
│ │ └── policy-secretstore.yaml # Secret store configuration
│ ├── security-auth # Security and authentication
│ │ ├── policy-authentication.yaml # Authentication methods
│ │ ├── policy-disable-self-provisioner.yaml # Self-service limitations
│ │ ├── policy-htpasswd.yaml # Htpasswd authentication
│ │ └── policy-remove-kubeadmin.yaml # Remove default admin
│ ├── storage # Storage configurations
│ │ ├── enable-multipath.yaml # Enable multipath
│ │ └── policy-storagecluster.yaml # Storage configuration
│ ├── testing # Test policies
│ │ ├── check-cluster-operator.yaml # Cluster operator check
│ │ └── virtualizations # Virtualization configurations
│ │ ├── policy-install-mtv.yaml # Migration Toolkit for Virtualization
│ │ ├── policy-install-nmstate.yaml # Network state management
│ │ └── policy-install-virtualization.yaml # OpenShift Virtualization
│ ├── CHANGELOG.md # Reference document for changes made
│ └── README.md # General documentationThis structure offers several advantages:
- Logical organization:
- Clear separation between global and specific policies.
- Easy identification of policies by type and purpose.
- Support for multi-region organizations.
- Simplified maintenance:
- Targeted updates by category.
- More efficient changes review.
- Reduced risk of conflicts.
- Governance and compliance:
- Policy tracking by region.
- Simplified configuration audit.
- Centralized management of security policies.
The following is an example of a policy-generator.yaml using this structure:
apiVersion: policy.open-cluster-management.io/v1
kind: PolicyGenerator
metadata:
name: policies
placementBindingDefaults:
# Set an explicit placement binding name to use rather than rely on the default.
name: "placementbinding"
policyDefaults:
categories:
- "CM Configuration Management"
controls:
- "CM-2 Baseline Configuration"
standards:
- "NIST SP 800-53"
namespaceSelector:
include: ["*"]
exclude: []
namespace: open-cluster-management-policies
remediationAction: inform
complianceType: "musthave"
severity: low
pruneObjectBehavior: "None"
metadataComplianceType: "musthave"
policySets:
- name: global
description: "Global configuration for all managed clusters"
placement:
name: "global-placement"
labelSelector:
matchExpressions:
- key: vendor
operator: In
values:
- OpenShift
policies:
- configure-alert-manager
- enable-registry
- ingress-certificate
- install-cert-manager
- install-external-secrets
- install-lvm
- oauth-configuration
- remove-kubeadmin
- service-account-aap
- ocp-tools
- kubelet-config
- check-cluster-operator
- check-job-failed
- file-integrity
- name: hub
description: "Dedicated configurations for the hub cluster"
placement:
name: "hub-placement"
labelSelector:
matchExpressions:
- key: cluster-name
operator: In
values:
- hub
policies:
- automation-template
- install-gitops
- name: sno
description: "Dedicated configurations for managed clusters of type sno"
placement:
name: "sno-placement"
labelSelector:
matchExpressions:
- key: cluster-name
operator: NotIn
values:
- sno
policies:
- install-smb-operator
policies:
- name: configure-alert-manager
description: "Configures AlertManager for notification management"
manifests:
- path: ../policies/alert-manager.yml
remediationAction: enforce
categories:
- "CM Configuration Management"
controls:
- "CM-2 Baseline Configuration"
standards:
- "NIST SP 800-53"
severity: low
- name: install-external-secrets
description: "Installs and configures External Secrets Operator for secret management"
manifests:
- path: ../policies/external-secrets.yml
remediationAction: enforce
categories:
- "CM Configuration Management"
controls:
- "CM-2 Baseline Configuration"
standards:
- "NIST SP 800-53"
severity: low
...This organization allows you to:
- Maintain a clear separation of responsibilities.
- Facilitate the addition of new policies.
- Support multi-tenant management.
- Simplify the review and approval processes.
- Improve change traceability.
Each folder can also contain a README.md that documents:
- The purpose of the policies in that folder.
- Dependencies with other policies.
- Specific requirements for implementation.
- Testing and validation procedures.
Each organization should evaluate and adapt this structure based on its specific needs. The main factors to consider:
- Number of managed clusters.
- Policy complexity.
- Compliance requirements.
- Organizational team structure.
OpenShift Virtualization and ZTP
Multi-cluster management
The integration between ZTP and OpenShift Virtualization brings significant benefits:
- Provisioning: Ease of installation and preparation of OpenShift clusters.
- Repeatability: Easily replicate new installations always starting from a well-organized template structure.
- Homogeneity: ZTP and Red Hat Advanced Cluster Management policies facilitate homogeneity and generalization of configurations applied to OpenShift clusters.
Best practices for implementation
Before implementing ZTP and GitOps, it is important to follow a few essential best practices to ensure security, scalability, and ease of management. Here are the main recommendations:
- GitOps as the single source of truth: All configurations should be versioned and managed through Git.
- Environment separation: Keep separate environments (dev, test, prod) to avoid mistakes and simplify changes.
- Secure secret management: Use operators like External Secrets to avoid storing sensitive data in the repository.
- Automated validation: Integrate automated tests and compliance checks in CI/CD pipelines.
- Essential documentation: Add brief documentation to configurations to support maintenance.
Argo CD application example:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: policy
namespace: openshift-gitops
spec:
destination:
namespace: open-cluster-management-policies
server: 'https://kubernetes.default.svc'
project: default
source:
path: policies
repoURL: 'git@github.com:albertofilice/rhacm-policies.git'
targetRevision: HEAD
syncPolicy:
automated:
selfHeal: true
syncOptions:
- CreateNamespace=true
- Validate=true
- RespectIgnoreDifferences=trueAutomation with Ansible Automation Platform
Automating cluster lifecycle operations is essential for consistency and efficiency in large-scale environments. The Ansible Automation Platform enables the orchestration of complex workflows, such as pre- and post-installation tasks, upgrades, and validations, ensuring that every step is repeatable and auditable. Below are some key concepts and examples of how automation can be integrated into the ZTP process.
Pre-hook and post-hook
The ClusterCurator allows you to define pre and post installation or upgrade hooks:
spec:
install:
towerAuthSecret: aap-integrations
prehook:
- name: Pre-Installation Check
extra_vars:
check_network: true
check_storage: true
posthook:
- name: Post-Installation Validation
extra_vars:
validate_operators: true
validate_network: true
upgrade:
towerAuthSecret: aap-integrations
prehook:
- name: Pre-Upgrade Check
extra_vars:
check_network: true
check_storage: true
posthook:
- name: Post-Upgrade Validation
extra_vars:
validate_operators: true
validate_network: true
type: Job
---
kind: Secret
apiVersion: v1
metadata:
name: aap-integrations
namespace: aap
data:
host: <base64-encoded-host>
token: <base64-encoded-token>
type: OpaqueKey benefits of the Git-centric approach
Key benefits include:
- Versioning and control
- Changes full tracking.
- Rollback possibility.
- Embedded documentation in the code.
- Automation and CI/CD
- Automated deployment pipelines.
- Configuration testing.
- Pre-deployment validation.
- Collaboration and standardization
- Sharing of best practices
- Structured review process
- Facilitated knowledge sharing
Day 1 structure and integration with Argo CD
The Day 1 phase represents the crucial moment of cluster installation. The file structure is organized to manage all aspects of the installation:
├── day1-agentclusterinstall.yaml # Cluster installation configuration
├── day1-bmh.yaml # Bare Metal Host definition
├── day1-clusterdeployment.yaml # Cluster deployment
├── day1-infraenv.yaml # Infrastructure environment
├── day1-klusterletaddonconfig.yaml # Managed cluster addon configuration
├── day1-managed-cluster-secret.yaml # Secret for the managed cluster
├── day1-managedcluster.yaml # Managed cluster definition
├── day1-namespace.yaml # Dedicated namespace
├── day1-nmstateconfig.yaml # Network configuration
├── day1-oidc-configmap.yaml # OIDC configuration
├── day1-pull-secret.yaml # Pull secret for images
├── day1-reports.md # Documentation and reports
├── day2-workers/ # Configurations for adding worker / Infra nodes
└── kustomization.yaml # Kustomize file for resource managementFor management convenience, in this example we preferred to allow the management of worker nodes directly in the installed IPI cluster and not to have them managed by Red Hat Advanced Cluster Management.
Therefore, in the Day 1 phase, a 3-node cluster will be configured and subsequently expanded through the dedicated ApplicationSet for cluster scaling.
Automation with Argo CD
The integration with Argo CD allows to automate the deployment process through ApplicationSet. Two main ApplicationSets manage the process:
- Environments ZTP: Manages the initial deployment of clusters
- Cluster Scale: Manages the scaling of worker nodes
ApplicationSet for Day 1:
kind: ApplicationSet
metadata:
name: environments-ztp
namespace: openshift-gitops
spec:
generators:
- git:
directories:
- path: '*'
repoURL: 'https://github.com/test/ztp-environments.git'
revision: develop
# ... existing code ...This ApplicationSet:
- Automatically scans directories in the Git repository
- Creates an application for each environment found
- Manages differences by ignoring specific fields to avoid conflicts
- Implements automated sync policies
ApplicationSet for scaling:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cluster-scale
namespace: openshift-gitops
spec:
generators:
- clusters:
selector:
matchLabels:
apps.open-cluster-management.io/acm-cluster: 'true'
# ... existing code ...This ApplicationSet:
- Selects managed clusters via label
- Applies scaling configurations from the day2-workers files
- Supports dynamic scaling of worker nodes
Structure benefits
Benefits of this structure include:
- Complete automation:
- Automated cluster deployment.
- Automatic scaling management.
- Self-healing of configurations.
- Flexibility:
- Support for multiple environments.
- Easy addition of new clusters.
- Customizable configurations per environment.
- Maintainability:
- Clear and organized structure.
- Separation of responsibilities.
- Integrated documentation.
- Security:
- Secret centralized management.
- Access control via OIDC.
- Changes audit trail.
Secret management with External Secrets Operator
Within our ZTP project, we implemented External Secrets Operator (ESO) to securely and centrally manage all secrets necessary for cluster operation. This approach allows to keep the Git repository clean from sensitive data, significantly improving the security of the infrastructure.
There are several certified products that allow you to integrate secrets via vault. We used a community-supported operator for convenience, but in a production environment, it's always better to use certified and supported products to ensure timely resolution of bugs.
Types of secrets
ESO manages various types of critical secrets for cluster operation:
- Access credentials:
- Htpasswd credentials for authentication.
- Access tokens for registry.
- Credentials for Bare Metal Host (BMH).
- Certificates:
- Certificates for ingress.
- Custom CA certificates.
- Certificates for inter-cluster communication.
- System secrets:
- Pull secret for OpenShift.
- Secret for communication with Red Hat Advanced Cluster Management.
- Token for integration with external services.
Implementation
Secret management is handled through two main components:
SecretStore: Defines the connection to the secret back end (e.g., HashiCorp Vault)ExternalSecret: Defines how to map secrets from the back end to Kubernetes
Example configuration:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: open-cluster-management-policies
spec:
provider:
vault:
server: "https://vault.example.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-role"
serviceAccountRef:
name: "eso-service-account"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: htpasswd-secret
namespace: open-cluster-management-policies
spec:
refreshInterval: "1h"
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: htpasswd-secret
data:
- secretKey: htpasswd
remoteRef:
key: clusters/htpasswdBenefits of this approach
GitOps provides significant benefits for cluster provisioning, including improved security, maintainability, and scalability. By using version control for infrastructure, it ensures consistency and auditability, making it a powerful solution.
- Security:
- No sensitive data in the Git repository.
- Automatic secret rotation.
- Centralized audit trail.
- Granular access control.
- Maintainability:
- Centralized secret management.
- Automatic updates.
- Reduced risk of hardcoded secrets.
- Simplified rotation process.
- Scalability:
- Support for multiple Vault instances.
- Efficient management of large quantities of secrets.
- Easy extension of new secrets.
- Support for multi-cluster environments.
Implemented best practices
Our approach incorporates several key principles to achieve these goals:
- Environment separation:
- Separate paths in Vault for each environment.
- Dedicated namespaces for secrets.
- Label for tracking secret origin.
- Lifecycle management:
- Automatic secret rotation.
- Secret versioning.
- Automatic backup of configurations.
- Monitoring and audit:
- Logging of secret operations.
- Alerting for unauthorized access.
- Periodic reports on usage.
Integration with the ZTP workflow
ESO integrates perfectly with the ZTP workflow:
- Day 0:
- Necessary secrets in Vault preparation.
- Access policies configuration.
SecretStoreSetup.
- Day 1:
- Automated provisioning during secret installation.
- Initial credentials Setup.
- Certificates Setup.
- Day 2:
- Automated secret rotation.
- User credentials management.
- Certificates update.
Security considerations
When designing and implementing a secure and scalable OpenShift environment, it is important to consider several key aspects. The following points highlight the most critical considerations to keep in mind to ensure a robust and compliant infrastructure:
- Secret access:
- PoLP (principle of least privilege) implementation.
- Authentication based on service account.
- All accesses audit logging.
- Encryption:
- Encrypted secrets at rest.
- TLS for communication with Vault.
- Secret key rotation.
- Compliance:
- Compliance with security policies.
- Tracking of changes.
- Documentation of procedures.
Final thoughts
The implementation of ZTP with a Git-centric approach represents a paradigm shift in the management of OpenShift infrastructure. The combination with OpenShift Virtualization allows to create a robust, scalable and easily manageable ecosystem, significantly reducing time-to-market and operational costs.
For more information, refer to these resources:
