Dynamic Provisioning in K8S using FlexVolume Plug-in

Unlike Static provisioning example there can be use cases where Dynamic provisioning is useful in cases where you do not want MapR and Kubernetes administrators to create storage manually to store the Pod storage state/data. The PersistentVolume is created automatically based on the parameters specified in the referenced StorageClass.

The following Blog uses a PersistentVolumeClaim that references a Storage Class. In this example, a Kubernetes Administrator has created a storage class called secure-maprfs for Pod creators to use when they want to create persistent storage for their Pods. The created Pod storage will survive the deletion of a Pod hence the reclaim policy is set to retain.

1) Below is sample yaml file to statically provision MapR Flex volume to K8S .

[root@tssperf09 abizerwork]# cat DynamicProvisioner.yaml 

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

   name: secure-maprfs

   namespace: mapr-system

provisioner: mapr.com/maprfs

parameters:

    restServers: "10.10.70.113:8443"

    cldbHosts: "10.10.70.113 10.10.70.114 10.10.70.115"

    cluster: "ObjectPool"

    securityType: "unsecure"

    ticketSecretNamespace: "mapr-system"

    maprSecretName: "mapr-provisioner-secrets"

    maprSecretNamespace: "mapr-system"

    namePrefix: "pv"

    mountPrefix: "/pv"

    readOnly: “true”

    reclaimPolicy: “Retain”

    advisoryquota: "100M"

---

kind: Pod

apiVersion: v1

metadata:

  name: test-secure-provisioner

  namespace: mapr-system

spec:

  containers:

  - name: busybox

    image: docker.io/maprtech/kdf-plugin:1.0.0_029_centos7 

    args:

    - sleep

    - "1000000"

    imagePullPolicy: Always

    volumeMounts:

      - name: maprfs-pvc

        mountPath: "/dynvolume"

  restartPolicy: "Never"

  volumes:

    - name: maprfs-pvc

      persistentVolumeClaim:

        claimName: maprfs-secure-pvc

---

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

  name: maprfs-secure-pvc

  namespace: mapr-system

spec:

  accessModes:

    - ReadWriteOnce

  storageClassName: secure-maprfs

  resources:

    requests:

      storage: 300M

---

apiVersion: v1

kind: Secret

metadata:

  name: mapr-provisioner-secrets

  namespace: mapr-system

type: Opaque

data:

  MAPR_CLUSTER_USER: cm9vdA==                       <--- Username in base64 format

  MAPR_CLUSTER_PASSWORD: YWJpemVy                   <--- Password in base64 format

[root@tssperf09 abizerwork]# 

Note :- Convert username and password to base64 format.

https://www.base64encode.org/

2)  Use the kubectl create command with the -f option to install Dynamic provisioner on  Kubernetes cluster .

[root@tssperf09 abizerwork]# kubectl create -f DynamicProvisioner.yaml 

storageclass "secure-maprfs" created

pod "test-secure-provisioner" created

persistentvolumeclaim "maprfs-secure-pvc" created

secret "mapr-provisioner-secrets" created

[root@tssperf09 abizerwork]# 

3) Now we can see "test-secure-provisioner" pod is up and running. This would give all the steps 

[root@tssperf09 abizerwork]# kubectl get pods -n mapr-system -o wide

NAME                                   READY     STATUS    RESTARTS   AGE       IP                NODE

mapr-kdfplugin-6l5n7                   1/1       Running   0          23d       192.168.61.65     tssperf10.lab

mapr-kdfplugin-crzhk                   1/1       Running   0          23d       192.168.217.129   tssperf11.lab

mapr-kdfplugin-srcln                   1/1       Running   0          23d       192.168.196.223   tssperf09.lab

mapr-kdfprovisioner-79b86f459d-hjkcn   1/1       Running   0          23d       192.168.217.130   tssperf11.lab

test-secure                            1/1       Running   0          2d        192.168.61.67     tssperf10.lab

test-secure-provisioner                1/1       Running   0          1m        192.168.217.135   tssperf11.lab

[root@tssperf09 abizerwork]# 

4) To review the status of the Pod being spun up review the "Events" para for the command output.

[root@tssperf09 abizerwork]# kubectl describe pod test-secure-provisioner  -n mapr-system

Name:         test-secure-provisioner

Namespace:    mapr-system

Node:         tssperf11.lab/10.10.72.251

Start Time:   Fri, 20 Apr 2018 14:57:41 -0600

Labels:       <none>

Annotations:  <none>

Status:       Running

IP:           192.168.217.135

Containers:

  busybox:

    Container ID:  docker://dce62da9f665d8362fa62af528fb2c2b005a3b23c505da3764e33d8cf7f2fa37

    Image:         docker.io/maprtech/kdf-plugin:1.0.0_029_centos7

    Image ID:      docker-pullable://docker.io/maprtech/kdf-plugin@sha256:eecb2d64ede9b9232b6eebf5d0cc59fe769d16aeb56467d0a00489ce7224278d

    Port:          <none>

    Args:

      sleep

      1000000

    State:          Running

      Started:      Fri, 20 Apr 2018 14:57:51 -0600

    Ready:          True

    Restart Count:  0

    Environment:    <none>

    Mounts:

      /dynvolume from maprfs-pvc (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9g8tq (ro)

Conditions:

  Type           Status

  Initialized    True 

  Ready          True 

  PodScheduled   True 

Volumes:

  maprfs-pvc:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  maprfs-secure-pvc

    ReadOnly:   false

  default-token-9g8tq:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  default-token-9g8tq

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type    Reason                 Age   From                    Message

  ----    ------                 ----  ----                    -------

  Normal  Scheduled              30s   default-scheduler       Successfully assigned test-secure-provisioner to tssperf11.lab

  Normal  SuccessfulMountVolume  30s   kubelet, tssperf11.lab  MountVolume.SetUp succeeded for volume "default-token-9g8tq"

  Normal  SuccessfulMountVolume  22s   kubelet, tssperf11.lab  MountVolume.SetUp succeeded for volume "pv-xplttvibrl"

  Normal  Pulling                21s   kubelet, tssperf11.lab  pulling image "docker.io/maprtech/kdf-plugin:1.0.0_029_centos7"

  Normal  Pulled                 20s   kubelet, tssperf11.lab  Successfully pulled image "docker.io/maprtech/kdf-plugin:1.0.0_029_centos7"

  Normal  Created                20s   kubelet, tssperf11.lab  Created container

  Normal  Started                20s   kubelet, tssperf11.lab  Started container

[root@tssperf09 abizerwork]# 

As seen above the POD is assigned to be spun up on node tssperf11 . Incase if there are any issues while provisioner is trying to Provision the POD we can review the logs where Provisioner is running in this case node tssperf11 and tail the logs to review the logs and errors they are running into to figure out the RC and fix the issue accordingly.

[root@tssperf11 logs]# tail -12 provisioner-k8s.log 

2018/04/19 19:18:33 main.go:443: INFO  === Starting volume provisioning ===

2018/04/19 19:18:33 main.go:444: INFO  options={Delete pvc-be766816-4438-11e8-b9b1-84b80208e1f2 &PersistentVolumeClaim{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:maprfs-secure-pvc,GenerateName:,Namespace:mapr-system,SelfLink:/api/v1/namespaces/mapr-system/persistentvolumeclaims/maprfs-secure-pvc,UID:be766816-4438-11e8-b9b1-84b80208e1f2,ResourceVersion:3036847,Generation:0,CreationTimestamp:2018-04-19 19:18:33 -0600 MDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{300 6} {<nil>} 300M DecimalSI},},},VolumeName:,Selector:nil,StorageClassName:*secure-maprfs,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},},} map[maprSecretName:mapr-provisioner-secrets mountPrefix:/pv namePrefix:pv readOnly:“true” restServers:10.10.70.113:8443 advisoryquota:100M cldbHosts:10.10.70.113 10.10.70.114 10.10.70.115 cluster:ObjectPool maprSecretNamespace:mapr-system reclaimPolicy:“Retain” securityType:unsecure ticketSecretNamespace:mapr-system]}

2018/04/19 19:18:33 main.go:445: INFO  Cleaning parameters...

2018/04/19 19:18:33 main.go:449: INFO  Parsing parameters...

2018/04/19 19:18:33 main.go:135: INFO  Getting admin secret: mapr-provisioner-secrets from namespace: mapr-system

2018/04/19 19:18:33 main.go:165: INFO  Got admin secret

2018/04/19 19:18:33 main.go:452: INFO  Constructed server info: (rest: 10.10.70.113:8443, cldb: 10.10.70.113 10.10.70.114 10.10.70.115, cluster: ObjectPool, securitytype: unsecure)

2018/04/19 19:18:33 main.go:105: INFO  Convert Kubernetes capacity: %!s(int64=300000000)

2018/04/19 19:18:33 main.go:108: INFO  Converted MapR capacity: 300M

2018/04/19 19:18:33 main.go:458: INFO  Generated Mapr volumename: pv.uclcmsgkxq mountpoint: /pv/pv-uclcmsgkxq

2018/04/19 19:18:33 main.go:319: INFO  Creating MapR query...

2018/04/19 19:18:33 main.go:326: INFO  Calling executeQuery with query string: /rest/volume/create?createparent=1&name=pv.uclcmsgkxq&advisoryquota=100M&path=%2Fpv%2Fpv-uclcmsgkxq&quota=300M&mount=1

2018/04/19 19:18:36 main.go:346: INFO  Response: {"timestamp":1524186116264,"timeofday":"2018-04-19 06:01:56.264 GMT-0700","status":"OK","total":0,"data":[],"messages":["Successfully created volume: 'pv.uclcmsgkxq'"]}

2018/04/19 19:18:36 main.go:467: INFO  Creating Kubernetes PersistentVolume: pv-uclcmsgkxq

2018/04/19 19:18:36 main.go:472: INFO  Reclaim Policy: Delete

2018/04/19 19:18:36 main.go:511: INFO  === Finished volume provisioning ===

[root@tssperf11 logs]# 

As seen in screenshot below Volume did get created with prefix pv-<RandomString>