How to configure persistent storage with OpenShift or Kubernetes for development environment

• We know that containers in Openshift or Kubernetes don't persist data. Every time we start an application, it is started in a new container with an immutable Docker image.
  Hence, any persisted data in the file systems is lost when the container stops. Hence if an application or container is rebuilt or restarted than we can't view previous logs or if we are using containers with mysql or any other database then schema, tables, and all data will be lost, if using any messaging broker than if there is journal file than it will also not persist.
  Hence, these ephemeral containers cannot be used in production environment. In a production environment, we must configure a shared storage.
• But what about the development environment, because we might not always have enough labs and VM's available. To rescue we have volume type hostPath, which can be easily set up with Minishift and Minikube.
• This article will provide details how to setup hostPath volume type.

1. PersistentVolumes: We have to configure PersistentVolume. We can create a pv.yaml file like:

apiVersion: v1
 kind: PersistentVolume
 metadata:
 name: pv0002
 spec:
 accessModes:
 - ReadWriteOnce
 capacity:
 storage: 500Mi
 hostPath:
 path: /data/pv0002/

•  Above we have created pv0002 Persistent Volume. With a specification like accessModes, capacity, and hostPath.
• In OpenShift we can easily create PersistentVolume with the command,

oc create -f pv.yaml

• To view all configured persistent volume, run the command below,

oc get pv

2. Check hostpath privileges: It needs elevated privileges to read and write from hostpath /data/pv0002. So we would have to first ssh to the host machine (which is a minishift virtual machine) then set the privileges so that all users can read/write.

minishift ssh
 
 docker@minishift:~$ id
 uid=1000(docker) gid=50(staff) groups=50(staff),100(docker)
 
 docker@minishift:~$ pwd
 /home/docker
 
 docker@minishift:~$ cd /data
 
 docker@minishift:~$ sudo chmod -R 777 pv0002
 
 docker@minishift:/mnt/sda1/data$ ls -ltr|grep pv0002
 drwxrwxrwx 5 root root 4096 Oct 18 07:41 pv0002/
 
 docker@minishift:/mnt/sda1/data$ pwd
 /data

3. PersistentVolumeClaims: To access this storage from Projects, PersistentVolumeClaims must be created that can access the PersistentVolume. PersistentVolumeClaims are created for each Project with customized claims for a certain amount of storage with certain access modes. We can create another yaml (mysql-deployment.yaml) file with below configuration. Using below configuration, we can have a mysql container, which would also persist the data. This is a reference from this link.

apiVersion: v1
 kind: Service
 metadata:
 name: mysql
 spec:
 ports:
 - port: 3306
 selector:
 app: mysql
 clusterIP: None
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
 name: mysql-pv-claim
 spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 300Mi
 ---
 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
 name: mysql
 spec:
 selector:
 matchLabels:
 app: mysql
 strategy:
 type: Recreate
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 # Use secret in real usage
 - name: MYSQL_ROOT_PASSWORD
 value: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pv-claim

Points to note above:

• We are creating a service mysql.
• We are creating PersistentVolumeClaim mysql-pv-claim, which should bind to volume pv0002.
• We are creating a Deployment mysql, with the environment variable MYSQL_ROOT_PASSWORD and mountPath /var/lib/mysql which mounts to persistentVolumeClaim mysql-pv-claim. We know persistentVolumeClaim is referring to /data/pv0002/, we have defined this path in persistent volume pv0002.

4. Deploy the yaml content: We can run this yaml as

oc create -f mysql-deployment.yaml

5.  Check if pv and pvc are bound: We can get details of persistentvolume and persistentvolumeclaim as

[cpandey@cpandey minishit_persistence_host]$ oc get pvc
 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
 mysql-pv-claim Bound pv0002 500Mi RWO 1d
 
 [cpandey@cpandey minishit_persistence_host]$ oc get pv
 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
 pv0002 500Mi RWO Retain Bound myproject/mysql-pv-claim 1d
 
 [cpandey@cpandey minishit_persistence_host]$ oc get service
 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
 mysql None none 3306/TCP 1d

• Above we find that pv0002 is bound and claimed by mysql-pv-claim.
• Also, we see that mysql service is up and running on port 3306 port.

6. Now if we again ssh to minishift. We will find files created within /data/pv0002 hostpath.

docker@minishift:~$ cd /data/pv0002
 
 docker@minishift:/mnt/sda1/data/pv0002$ ls -ltr
 total 110608
 -rw-rw---- 1 10000400 root 50331648 Oct 17 03:51 ib_logfile1
 drwx------ 2 10000400 root 4096 Oct 17 03:51 performance_schema/
 -rw-rw---- 1 10000400 root 56 Oct 17 03:51 auto.cnf
 drwx------ 2 10000400 root 4096 Oct 17 03:51 mysql/
 drwx------ 2 10000400 root 4096 Oct 17 03:55 csp/
 -rw-rw---- 1 10000400 root 12582912 Oct 18 07:41 ibdata1
 -rw-rw---- 1 10000400 root 50331648 Oct 18 07:42 ib_logfile0

7. Access mysql pod: Below we can access mysql pod. Here note that database csp and table testtable was created manually on the first run.

[cpandey@cpandey minishit_persistence_host]$ oc get pods
 NAME READY STATUS RESTARTS AGE
 mysql-63082529-66jyn 1/1 Running 1 1d
  
 [cpandey@cpandey minishit_persistence_host]$ oc rsh mysql-63082529-66jyn
 $ mysql -u root -ppassword
  
 mysql> show databases;
 +--------------------+
 | Database |
 +--------------------+
 | information_schema |
 | csp |
 | mysql |
 | performance_schema |
 +--------------------+
 4 rows in set (0.01 sec)

 mysql> use csp;
 Database changed

 mysql> show tables;
 +---------------+
 | Tables_in_csp |
 +---------------+
 | testtable |
 +---------------+
 1 row in set (0.00 sec)

 mysql> select * from testtable;
 Empty set (0.00 sec)

8. Verify if data persists: Now let us delete the current running pod and check if mysql db and tables still exist.

[cpandey@cpandey minishit_persistence_host]$ oc delete pod mysql-63082529-66jyn
 pod "mysql-63082529-66jyn" deleted
 
 [cpandey@cpandey minishit_persistence_host]$ oc get pods
 NAME READY STATUS RESTARTS AGE
 mysql-63082529-66jyn 1/1 Terminating 1 1d
 mysql-63082529-q7e5n 1/1 Running 0 4s
 
 [cpandey@cpandey minishit_persistence_host]$ oc get pods
 NAME READY STATUS RESTARTS AGE
 mysql-63082529-q7e5n 1/1 Running 0 14s
 
 [cpandey@cpandey minishit_persistence_host]$ oc rsh mysql-63082529-q7e5n
 $ mysql -u root -ppassword

 mysql> show databases;
 +--------------------+
 | Database |
 +--------------------+
 | information_schema |
 | csp |
 | mysql |
 | performance_schema |
 +--------------------+
 4 rows in set (0.00 sec)

 mysql> use csp;

 mysql> show tables;
 +---------------+
 | Tables_in_csp |
 +---------------+
 | testtable |
 +---------------+
 1 row in set (0.00 sec)

This way we can easily configure persistence even in our development environment, which is mostly minishift or minikube.