OCP Part2: GlusterFS Storage for OpenShift

In Kubernetes you assign (persistent) storage to container using PhysicalVolumeClaim telling how much space you want and which storage class. StorageClassName attribute is used to bound your request to appropriate provisioner.

In OpenShift cluster built by Ansible you can check available custom storage classes. One GlusterFS is provided directly by Kubernetes (glusterfs-storage), other one by Gluster maintainers — it is block storage. Well, sort of. Block storage is emulated via creation of big file on Gluster and the looping it as a block device.

[root@k0 ~]# oc get sc 
NAME PROVISIONER AGE 
glusterfs-storage kubernetes.io/glusterfs 1h 
glusterfs-storage-block gluster.org/glusterblock 1h

We can learn about these storage classes using standard kubectl notation get export json:

[root@k0 ~]# oc get sc glusterfs-storage — export -ojson 
{ 
 “apiVersion”: “storage.k8s.io/v1”, 
 “kind”: “StorageClass”, 
 “metadata”: { 
 “creationTimestamp”: null, 
 “name”: “glusterfs-storage”, 
 “selfLink”: “/apis/storage.k8s.io/v1/storageclasses/glusterfs-storage” 
 }, 
 “parameters”: { 
 “resturl”: “http://heketi-storage.app-storage.svc:8080”, 
 “restuser”: “admin”, 
 “secretName”: “heketi-storage-admin-secret”, 
 “secretNamespace”: “app-storage”, 
 “volumetype”: “none” 
 }, 
 “provisioner”: “kubernetes.io/glusterfs”, 
 “reclaimPolicy”: “Delete”, 
 “volumeBindingMode”: “Immediate” 
} 
[root@k0 ~]# oc get sc glusterfs-storage-block — export -ojson 
{ 
 “apiVersion”: “storage.k8s.io/v1”, 
 “kind”: “StorageClass”, 
 “metadata”: { 
 “creationTimestamp”: null, 
 “name”: “glusterfs-storage-block”, 
 “selfLink”: “/apis/storage.k8s.io/v1/storageclasses/glusterfs-storage-block” 
 }, 
 “parameters”: { 
 “chapauthenabled”: “true”,
 “restsecretname”: “heketi-storage-admin-secret-block”, 
 “restsecretnamespace”: “app-storage”, 
 “resturl”: “http://heketi-storage.app-storage.svc:8080”, 
 “restuser”: “admin” 
 }, 
 “provisioner”: “gluster.org/glusterblock”, 
 “reclaimPolicy”: “Delete”, 
 “volumeBindingMode”: “Immediate” 
}

OpenShift usually places fs provisioning in enclosed namespace:

[root@k0 ~]# oc project app-storage 
Now using project “app-storage” on server “https://k0.sys.one.pl:8443”. 
[root@k0 ~]# oc get pod 
NAME READY STATUS RESTARTS AGE 
glusterblock-storage-provisioner-dc-1-qrh2h 1/1 Running 0 1h 
glusterfs-storage-jt7dx 1/1 Running 0 1h 
heketi-storage-1-jrnf9 1/1 Running 0 1h

Let’s see what volumes we have:

[root@k0 ~]# oc get pv 
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 
pvc-9698a2b6–1a5d-11e9-bec1-b8aeed767836 10Gi RWO Delete Bound openshift-logging/logging-es-0 glusterfs-storage 1h 
registry-volume 4Gi RWX Retain Bound default/registry-claim 1h

and see inside Heketi provisioner (it exposes REST endpoint for volume management):

[root@k0 ~]# oc exec -it heketi-storage-1-jrnf9 /bin/bash 
[root@heketi-storage-1-jrnf9 /]# env | grep HEKETI 
HEKETI_DB_STORAGE_ENDPOINTS_SERVICE_HOST=172.30.86.57 
HEKETI_SNAPSHOT_LIMIT=14 
HEKETI_STORAGE_PORT_8080_TCP_ADDR=172.30.71.18 
HEKETI_STORAGE_PORT=tcp://172.30.71.18:8080 
HEKETI_DB_STORAGE_ENDPOINTS_PORT_1_TCP_ADDR=172.30.86.57 
HEKETI_IGNORE_STALE_OPERATIONS=true 
HEKETI_CLI_KEY=…
HEKETI_CLI_USER=admin 
HEKETI_EXECUTOR=kubernetes 
HEKETI_STORAGE_SERVICE_HOST=172.30.71.18 
HEKETI_DB_STORAGE_ENDPOINTS_PORT_1_TCP_PORT=1 
HEKETI_KUBE_GLUSTER_DAEMONSET=1 
HEKETI_STORAGE_SERVICE_PORT_HEKETI=8080 
HEKETI_DB_STORAGE_ENDPOINTS_SERVICE_PORT=1 
HEKETI_BRANCH=release/8 
HEKETI_ADMIN_KEY=…
HEKETI_DB_STORAGE_ENDPOINTS_PORT_1_TCP=tcp://172.30.86.57:1 
HEKETI_DB_STORAGE_ENDPOINTS_PORT=tcp://172.30.86.57:1 
HEKETI_FSTAB=/var/lib/heketi/fstab 
HEKETI_USER_KEY=…
HEKETI_STORAGE_PORT_8080_TCP=tcp://172.30.71.18:8080 
HEKETI_STORAGE_SERVICE_PORT=8080 
HEKETI_STORAGE_PORT_8080_TCP_PROTO=tcp 
HEKETI_DB_STORAGE_ENDPOINTS_PORT_1_TCP_PROTO=tcp 
HEKETI_STORAGE_PORT_8080_TCP_PORT=8080 
[root@heketi-storage-1-jrnf9 /]# HEKETI_KEY=$HEKETI_ADMIN_KEY heketi-cli — user=admin volume list 
Id:51b4d211a59a6f389e2c66fc952f5b8e Cluster:4c2f3e790c35569b7f167922a6d4b556 Name:heketidbstorage 
Id:58b443b2cdd4acc898568e1f6eaaee39 Cluster:4c2f3e790c35569b7f167922a6d4b556 Name:glusterfs-registry-volume 
Id:93e06615540552d2212dbac236526a56 Cluster:4c2f3e790c35569b7f167922a6d4b556 Name:vol_93e06615540552d2212dbac236526a56 
[root@heketi-storage-1-jrnf9 /]# export VOL_ID=`HEKETI_KEY=$HEKETI_ADMIN_KEY heketi-cli — user=admin volume list | tail -1 | awk ‘{print $1}’ | cut -d: -f2`
[root@heketi-storage-1-jrnf9 /]# HEKETI_KEY=$HEKETI_ADMIN_KEY heketi-cli — user=admin volume info $VOL_ID 
Name: vol_93e06615540552d2212dbac236526a56 
Size: 10 
Volume Id: 93e06615540552d2212dbac236526a56 
Cluster Id: 4c2f3e790c35569b7f167922a6d4b556 
Mount: 192.168.100.127:vol_93e06615540552d2212dbac236526a56 
Mount Options: backup-volfile-servers= 
Block: false 
Free Size: 0 
Reserved Size: 0 
Block Hosting Restriction: (none) 
Block Volumes: [] 
Durability Type: none 
Snapshot Factor: 1.00