Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

vSphere Cloud provider and VMDK dynamic provisioning

 

October 24, 2017
davis phillips
Related topics:
ContainersDeveloper ToolsDevOpsLinuxKubernetes
Related products:
Red Hat OpenShiftRed Hat OpenShift Container PlatformRed Hat Enterprise Linux

Share:

    The release of OpenShift Container Platform 3.6 brings support for vsphere cloud provider. This provides vsphere VMDK dynamic provisioning for persistent volumes for container workloads. The storage presented to vsphere virtual machines as a VMDK has ReadWriteOnce access mode.

    In the OCP 3.6 on vSphere reference architecture, much of this process is automated and can be implemented easily.

    Virtual Machine Disks or VMDKs exists in virtual machines. Configuring the OCP cluster for vsphere cloud provider support requires:

    • Master Node Configuration
    • Application Node Configuration
    • vCenter or vSphere Requirements

    The kubernetes docs do a great job of highlighting the requirements for vSphere.

    First, both master and nodes need the following parameters set and services restarted:

    • cloud-provider=vsphere
    • cloud-config=path_to_config.conf

    Master configuration

    $ vi /etc/origin/master/master-config.yaml
    
    kubernetesMasterConfig:
      apiServerArguments: 
        cloud-config:
        - /etc/vsphere/vsphere.conf
        cloud-provider:
        - vsphere
      controllerArguments: 
        cloud-config:
        - /etc/vsphere/vsphere.conf
        cloud-provider:
        - vsphere
    
    $ sudo systemctl restart atomic-openshift-master.service

    Node configuration

    $ vi /etc/origin/node/node-config.yaml
    
    kubeletArguments:
      cloud-config:
      - /etc/vsphere/vsphere.conf
      cloud-provider:
      - vsphere
    
    $ sudo systemctl restart atomic-openshift-node.service

    Next, the vsphere.conf file should loosely resemble this:

    $ cat /etc/vsphere/vsphere.conf 
    [Global]
    user = "administrator@vsphere.local"
    password = "vcenter_password"
    server = "10.*.*.25" 
    port = 443
    insecure-flag = 1
    datacenter = Boston
    datastore = ose3-vmware-prod
    working-dir = /Boston/vm/ocp36/
    [Disk]
    scsicontrollertype = pvscsi

    The variables are discussed more in the Kubernetes document. The working-dir variable is the folder that houses the OpenShift guest machines. In vSphere, the folder syntax will be /Datacenter/vm/foldername.

    Using GOVC

    The tool govc is a GO-based application for interacting with vSphere and vCenter.

    The documentation for govc is located in github:

    https://github.com/vmware/govmomi/tree/master/govc

    First, download govc then export the vars that govc needs to query vCenter:

    curl -LO https://github.com/vmware/govmomi/releases/download/v0.15.0/govc_linux_amd64.gz
    gunzip govc_linux_amd64.gz
    chmod +x govc_linux_amd64
    cp govc_linux_amd64 /usr/bin/
    
    export GOVC_URL='vCenter IP OR FQDN'
    export GOVC_USERNAME='administrator@vsphere.local'
    export GOVC_PASSWORD='vCenter Password'
    export GOVC_INSECURE=1
    
    $ govc ls
    /Boston/vm
    /Boston/network
    /Boston/host
    /Boston/datastore
    
    $ govc ls /Boston/vm
    /Boston/vm/ocp36
    
    $ govc ls /Boston/vm/ocp36
    /Boston/vm/ocp36/haproxy-0
    /Boston/vm/ocp36/app-0
    /Boston/vm/ocp36/infra-0
    /Boston/vm/ocp36/haproxy-1
    /Boston/vm/ocp36/master-0
    /Boston/vm/ocp36/nfs-0

    vSphere Pre-requisites

    • disk.enableUUID

    Next, the enableUUID parameter needs to be set on all virtual machines in the OCP cluster. The option is necessary so that the VMDK always presents a consistent UUID to the VM, this allows the new disk to be mounted properly.

    enableUUID

    The enableUUID option can be set at the template or VM in the vCenter client. Next, the template would then be used to deploy the OpenShift VMs.

    Additionally, govc can be used to set this as well:

    for each VM in `govc ls /Boston/vm/ocp36/`;do govc vm.change -e="disk.enableUUID=1" -vm="$VM"

    Lastly, with the cloud provider configuration in place a new storage class to deploy VMDKs can be created.

    Storage Classes and VMDK Provisioning

    The datastore storage class can be created with the following yaml file:

    $ vi cloud-provider-storage-class.yaml
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: "ose3-vmware-prod"
    provisioner: kubernetes.io/vsphere-volume
    parameters:
        diskformat: zeroedthick
        datastore: "ose3-vmware-prod"
    
    oc create -f cloud-provider-storage-class.yaml

    Now that the StorageClass object is created. The oc command can be used to verify the StorageClass exists.

    $ oc get sc
    NAME               TYPE
    ose3-vmware-prod   kubernetes.io/vsphere-volume
    
    $ oc describe sc ose3-vmware-prod
    Name:           ose3-vmware-prod
    IsDefaultClass: No
    Annotations:    <none>
    Provisioner:    kubernetes.io/vsphere-volume
    Parameters:     datastore=ose3-vmware-prod,diskformat=zeroedthick
    Events:         <none>

    OpenShift can now dynamically provision VMDKs for persistent container storage within the OpenShift environment.

    $ vi storage-class-vmware-claim.yaml
    
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: ose3-vmware-prod
      annotations:
        volume.beta.kubernetes.io/storage-class: ose3-vmware-prod
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
    
    $ oc create -f storage-class-vmware-claim.yaml
    
    $ oc describe pvc ose3-vmware-prod
    
    Name:           ose3-vmware-prod
    Namespace:      default
    StorageClass:   ose3-vmware-prod
    Status:         Bound
    Volume:         pvc-cc8a9970-7c76-11e7-ae86-005056a571ee
    Labels:         <none>
    Annotations:    pv.kubernetes.io/bind-completed=yes
                    pv.kubernetes.io/bound-by-controller=yes
                    volume.beta.kubernetes.io/storage-class=vmware-datastore-ssd
                    volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
    Capacity:       2Gi
    Access Modes:   RWO
    Events:
      FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason          Message
      --------- --------    -----   ----                -------------   --------    ------          -------
      19s       19s     1   persistentvolume-controller         Normal      ProvisioningSucceeded   Successfully provisioned volume pvc-cc8a9970-7c76-11e7-ae86-005056a571ee using kubernetes.io/vsphere-volume

    Now, in vCenter, a couple of changes are initiated:

    First, the new disk is created.

    ocp-new-disk

    Secondly, the disk is ready to be consumed by a VM to be attached to a POD.

    Lastly, while datastores are generally accessible via shared storage across a vCenter cluster, the VMDKs are tied to a specific machine. This explains the 'ReadWriteOnce' limitation of the persistent storage.


     

    Last updated: November 2, 2023

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue