Tuesday, May 30, 2017

Install MapR Cluster (Red-Hat/CentOS 7.x)

                             Install MapR Cluster


Quick and dirty way to install MapR cluster within few mins with just few commands if you don't want to script this process or use GUI installer.

In this post its assumed you have a system with Red-Hat/CentOS 7.x already then below steps required to get MapR Hadoop Cluster up and running.

1) Download and install EPEL repo

[root@node106rhel72 ~]# curl -O http://mirror.nodesdirect.com/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14704  100 14704    0     0  58477      0 --:--:-- --:--:-- --:--:-- 58581

[root@node106rhel72 ~]# rpm -ivh epel-release-7-9.noarch.rpm
warning: epel-release-7-9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:epel-release-7-9                 ################################# [100%]

2) Since MapR packages are signed install the MapR package key (maprgpg.key)

rpm --import http://package.mapr.com/releases/pub/maprgpg.key

3) Add user mapr within group mapr

groupadd -g 5000 mapr ; useradd -g 5000 -u 5000 mapr

4) Setup MapR repo for package installation.

[root@node106rhel72 ~]# cat /etc/yum.repos.d/maprtech.repo 
[maprtech]
name=MapR Technologies
baseurl=http://package.mapr.com/releases/v5.2.0/redhat/
enabled=1
gpgcheck=0
protect=1

[centos]
name=centos, Inc.
baseurl=http://mirror.centos.org/centos/7.3.1611/os/x86_64/
enabled=1
gpgcheck=0

5) Now creating list of disk to be given to MFS .

[root@node106rhel72 ~]# fdisk -l | awk '{print $2}' | cut -d: -f1 | grep dev | egrep -v "mapper|sda" >> /tmp/disks

6)  Install all required services with Java.


yum upgrade mapr-cldb mapr-core mapr-core-internal mapr-fileserver mapr-zookeeper mapr-zk-internal mapr-webserver -y

7) Setup "JAVA_HOME" in  /opt/mapr/conf/env.sh  file .

8) Run configure.sh for cluster to be aware of CLDB and ZK nodes.

   /opt/mapr/server/configure.sh -C Nodes -Z Nodes -N Cluster-Name


9) Now use disk setup to set up storage pools and get the disks formatted ready to be used for storage by MFS .

   /opt/mapr/server/disksetup -F /tmp/disks
   

10) Re-run configure.sh for autostart of ZK service and followed by warden.


[root@node106rhel72 logs]#  /opt/mapr/server/configure.sh -C 10.10.70.106 -Z 10.10.70.106 -N Container-cluster
Configuring Hadoop-2.7.0 at /opt/mapr/hadoop/hadoop-2.7.0
Done configuring Hadoop
CLDB node list: 10.10.70.106:7222
Zookeeper node list: 10.10.70.106:5181
Node setup configuration:  cldb fileserver webserver zookeeper
Log can be found at:  /opt/mapr/logs/configure.log
Zookeeper found on this node, and it is not running. Starting Zookeeper
Warning: mapr-zookeeper.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warden is not running. Starting mapr-warden. Warden will then start all other configured services on this node
... Starting cldb
... Starting fileserver
... Starting webserver
To further manage the system, use "maprcli", or connect browser to https://node106rhel72:8443/
To stop and start this node, use "systemctl start/stop mapr-warden "

11) Check MCS to Apply License and monitor the cluster.

Connect browser to https://{webserver host name}:8443




Wednesday, May 24, 2017

PACC ( Docker )


            PACC (Docker)


This Blog assumes you have RHEL 7.2 installed and has network connectivity with MapR cluster running 5.2.1 , For this blog we will create run PACC container on 1 node MapR cluster itself.

[root@node107rhel72 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)


1) Install and create a docker repo .

[root@node107rhel72 ~]# wget https://download.docker.com/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
--2017-05-20 17:52:08--  https://download.docker.com/linux/centos/docker-ce.repo
Resolving download.docker.com (download.docker.com)... 205.251.215.16, 205.251.215.214, 205.251.215.188, ...
Connecting to download.docker.com (download.docker.com)|205.251.215.16|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1802 (1.8K) [binary/octet-stream]
Saving to: ‘/etc/yum.repos.d/docker-ce.repo’

100%[===========================================================================================================================================>] 1,802       --.-K/s   in 0s      

2017-05-20 17:52:08 (209 MB/s) - ‘/etc/yum.repos.d/docker-ce.repo’ saved [1802/1802]



2) Install docker community edition via yum repo .

[root@node107rhel72 ~]# yum -y install docker-ce
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
docker-ce-stable                                                                                                                                              | 2.9 kB  00:00:00     
docker-ce-stable/x86_64/primary_db                                                                                                                            | 4.8 kB  00:00:00     

Installed:
  docker-ce.x86_64 0:17.03.1.ce-1.el7.centos                                                                                                                                         

Dependency Installed:
  audit-libs-python.x86_64 0:2.6.5-3.el7   checkpolicy.x86_64 0:2.5-4.el7          docker-ce-selinux.noarch 0:17.03.1.ce-1.el7.centos   libcgroup.x86_64 0:0.41-11.el7             
  libseccomp.x86_64 0:2.3.1-2.el7          libsemanage-python.x86_64 0:2.5-4.el7   libtool-ltdl.x86_64 0:2.4.2-21.el7_2                 policycoreutils-python.x86_64 0:2.5-8.el7  
  python-IPy.noarch 0:0.75-6.el7           setools-libs.x86_64 0:3.3.8-1.1.el7    

Updated:
  systemd.x86_64 0:219-30.el7                                                                                                                                                        

Dependency Updated:
  audit.x86_64 0:2.6.5-3.el7                audit-libs.x86_64 0:2.6.5-3.el7             libgudev1.x86_64 0:219-30.el7                        libselinux.x86_64 0:2.5-6.el7        
  libselinux-python.x86_64 0:2.5-6.el7      libselinux-utils.x86_64 0:2.5-6.el7         libsemanage.x86_64 0:2.5-4.el7                       libsepol.x86_64 0:2.5-6.el7          
  policycoreutils.x86_64 0:2.5-8.el7        selinux-policy.noarch 0:3.13.1-102.el7      selinux-policy-targeted.noarch 0:3.13.1-102.el7      systemd-libs.x86_64 0:219-30.el7     
  systemd-sysv.x86_64 0:219-30.el7         

Complete!



3) Start Docker service and check if it started correctly .

[root@node107rhel72 ~]#systemctl start docker.service
[root@node107rhel72 ~]# systemctl status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-05-20 18:07:50 PDT; 8s ago
     Docs: https://docs.docker.com
 Main PID: 6698 (dockerd)
   Memory: 17.4M
   CGroup: /system.slice/docker.service
           ├─6698 /usr/bin/dockerd
           └─6706 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontaine...

May 20 18:07:49 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:49.863150383-07:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
May 20 18:07:49 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:49.864332250-07:00" level=warning msg="mountpoint for pids not found"
May 20 18:07:49 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:49.864915554-07:00" level=info msg="Loading containers: start."
May 20 18:07:49 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:49.881350011-07:00" level=info msg="Firewalld running: false"
May 20 18:07:50 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:50.212516925-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17....IP address"
May 20 18:07:50 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:50.429830995-07:00" level=info msg="Loading containers: done."
May 20 18:07:50 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:50.443810965-07:00" level=info msg="Daemon has completed initialization"
May 20 18:07:50 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:50.443859449-07:00" level=info msg="Docker daemon" commit=c6d412e graphdriver=overlay version=17.03.1-ce
May 20 18:07:50 node107rhel72 dockerd[6698]: time="2017-05-20T18:07:50.460823980-07:00" level=info msg="API listen on /var/run/docker.sock"
May 20 18:07:50 node107rhel72 systemd[1]: Started Docker Application Container Engine.


Note :- When docker starts it creates a docker0 bridged network interface for the container to be able to send and receive all the traffic.

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:f3ff:feb9:f3da  prefixlen 64  scopeid 0x20<link>
        ether 02:42:f3:b9:f3:da  txqueuelen 0  (Ethernet)
        RX packets 70273  bytes 3818311 (3.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 88523  bytes 916959387 (874.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

4) Now download mapr-setup.sh script to install MapR PACC ( Persistent Application Client Container )


wget http://package.mapr.com/releases/installer/mapr-setup.sh -P /tmp

5) Make the script executable and give "docker client" arguments to the script for it to give you options on what kind of container image you are trying to built. In my case i am going to built an image for centos7 with yarn and fuse.

[root@node107rhel72 ~]# chmod 777 /tmp/mapr-setup.sh ; cd /tmp
[root@node107rhel72 tmp]# ./mapr-setup.sh docker client                                                                     MapR Distribution Initialization and Update
Copyright 2017 MapR Technologies, Inc., All Rights Reserved
                                                                                 http://www.mapr.com
                                                                                          
Build MapR client image? (y/n) [y]: 
Image OS class (centos6, centos7, ubuntu14, ubuntu16) [centos7]: 
Docker FROM base image name:tag [centos:centos7]: 
MapR core version [5.2.1]: 
MapR MEP version [3.0]: 
Install Hadoop YARN client (y/n) [n]: y
Add POSIX (FUSE) client to image? (y/n) [y]: y
Add HBase client to image? (y/n) [n]: n
Add Hive client to image? (y/n) [n]: n
Add Pig client to image? (y/n) [n]: n
Add MapR Streams clients to image? (y/n) [y]: n
MapR client image tag name [maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse]: 
Container network mode (bridge|host) [bridge]: 
Container memory: specify host XX[kmg] or 0 for no limit [0]: 
centos7: Pulling from library/centos
Digest: sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077
Status: Image is up to date for centos:centos7
Sending build context to Docker daemon 91.65 kB
Step 1/8 : FROM centos:centos7
 ---> 8140d0c64310
Step 2/8 : ENV container docker
 ---> Using cache
 ---> d622788657b5
Step 3/8 : RUN yum -y upgrade && yum install -y curl file net-tools openssl sudo syslinux wget which java-1.8.0-openjdk-devel && yum -q clean all
 ---> Using cache
 ---> 507ed868a51a
Step 4/8 : LABEL mapr.os centos7 mapr.version 5.2.1 mapr.mep_version 3.0
 ---> Using cache
 ---> 8cf58537d61a
Step 5/8 : COPY mapr-setup.sh /opt/mapr/installer/docker/
 ---> 527982533648
Removing intermediate container fe9abd05fcd2
Step 6/8 : RUN /opt/mapr/installer/docker/mapr-setup.sh -r http://package.mapr.com/releases container client 5.2.1 3.0 mapr-client mapr-posix-client-container
 ---> Running in f8e6bbc76039

Installing package dependencies (curl file net-tools openssl sudo syslinux 
wget which) 

Package curl-7.29.0-35.el7.centos.x86_64 already installed and latest version
Package file-5.11-33.el7.x86_64 already installed and latest version
Package net-tools-2.0-0.17.20131004git.el7.x86_64 already installed and latest version
Package 1:openssl-1.0.1e-60.el7_3.1.x86_64 already installed and latest version
Package sudo-1.8.6p7-21.el7_3.x86_64 already installed and latest version
Package syslinux-4.05-13.el7.x86_64 already installed and latest version
Package wget-1.14-13.el7.x86_64 already installed and latest version
Package which-2.20-7.el7.x86_64 already installed and latest version

...Success 

Testing for JDK 7 or higher ... 

Ensuring existing JDK 1.8 is up to date 

...Success 

Configuring MapR repositories... 

...Success 

Installing packages (mapr-client mapr-posix-client-container)... 

Resolving Dependencies
--> Running transaction check
---> Package mapr-client.x86_64 0:5.2.1.42646.GA-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package           Arch         Version                   Repository       Size
================================================================================
Installing:
 mapr-client       x86_64       5.2.1.42646.GA-1          MapR_Core       370 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 370 M
Installed size: 593 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mapr-client-5.2.1.42646.GA-1.x86_64                          1/1 

To configure default settings: run configure.sh (as root), using the -C (uppercase)
 option to specify the CLDB nodes of the default cluster, and the -c (lowercase) option
 to specify that it is a client configuration
  /opt/mapr/server/configure.sh [-N <cluster name>] -c -C <CLDB node>[:<port>][,<CLDB node>[:<port>]...]

To use this client with a secure cluster, add the -secure option to the configure.sh
  /opt/mapr/server/configure.sh [-N <cluster name>] -secure -c -C <CLDB node>[:<port>][,<CLDB node>[:<port>]...]
  Verifying  : mapr-client-5.2.1.42646.GA-1.x86_64                          1/1 

Installed:
  mapr-client.x86_64 0:5.2.1.42646.GA-1                                         

Complete!

Package mapr-client-5.2.1.42646.GA-1.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package mapr-posix-client-container.x86_64 0:5.2.1.42646.GA-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                       Arch     Version               Repository   Size
================================================================================
Installing:
 mapr-posix-client-container   x86_64   5.2.1.42646.GA-1      MapR_Core   223 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 223 k
Installed size: 770 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mapr-posix-client-container-5.2.1.42646.GA-1.x86_64          1/1 


Note: Forwarding request to 'systemctl enable mapr-posix-client-container.service'.
Created symlink /etc/systemd/system/default.target.wants/mapr-posix-client-container.service, pointing to /etc/systemd/system/mapr-posix-client-container.service.
  Verifying  : mapr-posix-client-container-5.2.1.42646.GA-1.x86_64          1/1 

Installed:
  mapr-posix-client-container.x86_64 0:5.2.1.42646.GA-1                         

Complete!

...Success 


...Success 

 ---> 04d446608ef2
Removing intermediate container f8e6bbc76039
Step 7/8 : ENTRYPOINT /opt/mapr/installer/docker/mapr-setup.sh container
 ---> Running in 554eea17594e
 ---> 41c385662bf6
Removing intermediate container 554eea17594e
Step 8/8 : CMD start
 ---> Running in 2970ba41549a
 ---> b2cba66ae741
Removing intermediate container 2970ba41549a
Successfully built b2cba66ae741

Edit '/tmp/docker_images/client/mapr-docker-client.sh' to set MAPR_CLUSTER and MAPR_CLDB_HOSTS and then execute it to start the container 
6) Now set cluster name and CLDB hostnames in below file ( Script to execute containers )

/tmp/docker_images/client/mapr-docker-client.sh

PS :- Below location has Dockerfile which has all the details of the image container should spin up with .

[root@node107rhel72 client]# ls /tmp/docker_images/client
Dockerfile  mapr-docker-client.sh  mapr-setup.sh

[root@node107rhel72 client]# cat Dockerfile 
FROM centos:centos7
ENV container docker
RUN yum -y upgrade && yum install -y curl file net-tools openssl sudo syslinux wget which java-1.8.0-openjdk-devel && yum -q clean all
LABEL mapr.os=centos7 mapr.version=5.2.1 mapr.mep_version=3.0
COPY mapr-setup.sh /opt/mapr/installer/docker/
RUN /opt/mapr/installer/docker/mapr-setup.sh -r http://package.mapr.com/releases container client 5.2.1 3.0 mapr-client mapr-posix-client-container
ENTRYPOINT ["/opt/mapr/installer/docker/mapr-setup.sh", "container"]
CMD ["start"]
[root@node107rhel72 client]#


7) Executing the script takes us into the container and we can communicate with the cluster via Hadoop and posix client as well .

[root@node107rhel72 tmp]# sh /tmp/docker_images/client/mapr-docker-client.sh

Testing for cluster user account... 

Enter MapR cluster user name: root
 ...Success 

Configuring MapR client ( -c -C 10.10.70.107 -N Container-cluster2)... 

create /opt/mapr/conf/conf.old
Configuring Hadoop-2.7.0 at /opt/mapr/hadoop/hadoop-2.7.0
Done configuring Hadoop
CLDB node list: 10.10.70.107:7222
Zookeeper node list: 

...Success 

[root@c469edf549d0 /]# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
 1162 ?        00:00:00 ps
[root@c469edf549d0 /]# hadoop fs -ls /
Found 6 items
drwxr-xr-x   - 5000 5000          0 2017-05-19 19:45 /apps
drwxr-xr-x   - 5000 5000          0 2017-05-19 19:44 /hbase
drwxr-xr-x   - 5000 5000          0 2017-05-19 19:46 /opt
drwxrwxrwx   - 5000 5000          0 2017-05-19 19:44 /tmp
drwxr-xr-x   - 5000 5000          0 2017-05-19 19:46 /user
drwxr-xr-x   - 5000 5000          1 2017-05-19 19:45 /var
[root@c469edf549d0 init.d]# mkdir /mapr

[root@c469edf549d0 init.d]# service mapr-posix-client-container start; cd /mapr

[root@c469edf549d0 mapr]# ls

Container-cluster2
[root@c469edf549d0 mapr]# cd Container-cluster2/
[root@c469edf549d0 Container-cluster2]# ls
apps  hbase  opt  tmp  user  var


Docker command examples which would be helpful in debugging issues:

1) Now from the node where the container is spun up we can see the container is running .

[root@node107rhel72 ~]# docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS               NAMES
c469edf549d0        maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse   "/opt/mapr/install..."   49 seconds ago      Up 48 seconds                           agitated_feynman
[

2) To get all the low level details of docker objects we can run "docker inspect" command on the container ID.

root@node107rhel72 ~]# docker inspect c469edf549d0
[
    {
        "Id": "c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473",
        "Created": "2017-05-21T02:02:13.744994026Z",
        "Path": "/opt/mapr/installer/docker/mapr-setup.sh",
        "Args": [
            "container",
            "start"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 30495,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-05-21T02:02:14.211984989Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:b2cba66ae741bf574cdc61e728d5dae21fc68a9918be35a134acf340fd7c8141",
        "ResolvConfPath": "/var/lib/docker/containers/c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473/hostname",
        "HostsPath": "/var/lib/docker/containers/c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473/hosts",
        "LogPath": "/var/lib/docker/containers/c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473/c469edf549d0e76d673493c5d1ca793bc1a4561c7a9708336dc55f88898ef473-json.log",
        "Name": "/agitated_feynman",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/sys/fs/cgroup:/sys/fs/cgroup:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "SYS_ADMIN",
                "SYS_RESOURCE"
            ],
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/fuse",
                    "PathInContainer": "/dev/fuse",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay/31725c13d270e155902a3de237c3a80ba1b1784df6a17fb06347bfa866aab4ec/root",
                "MergedDir": "/var/lib/docker/overlay/ad6c47f7636bc8c173d08f17323f947a3fafbe2c8f59d22665b9deec7a0c7ec1/merged",
                "UpperDir": "/var/lib/docker/overlay/ad6c47f7636bc8c173d08f17323f947a3fafbe2c8f59d22665b9deec7a0c7ec1/upper",
                "WorkDir": "/var/lib/docker/overlay/ad6c47f7636bc8c173d08f17323f947a3fafbe2c8f59d22665b9deec7a0c7ec1/work"
            }
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/sys/fs/cgroup",
                "Destination": "/sys/fs/cgroup",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "c469edf549d0",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "MAPR_DISKS=",
                "MAPR_CLUSTER=Container-cluster2",
                "MAPR_LICENSE_MODULES=",
                "MAPR_MEMORY=0",
                "MAPR_MOUNT_PATH=",
                "MAPR_SECURITY=",
                "MAPR_TZ=America/Los_Angeles",
                "MAPR_USER=",
                "MAPR_CONTAINER_USER=root",
                "MAPR_CONTAINER_UID=0",
                "MAPR_CONTAINER_GROUP=root",
                "MAPR_CONTAINER_GID=0",
                "MAPR_CONTAINER_PASSWORD=",
                "MAPR_CLDB_HOSTS=10.10.70.107",
                "MAPR_HS_HOST=",
                "MAPR_OT_HOSTS=",
                "MAPR_ZK_HOSTS=",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=docker"
            ],
            "Cmd": [
                "start"
            ],
            "ArgsEscaped": true,
            "Image": "maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/opt/mapr/installer/docker/mapr-setup.sh",
                "container"
            ],
            "OnBuild": null,
            "Labels": {
                "build-date": "20170510",
                "license": "GPLv2",
                "mapr.mep_version": "3.0",
                "mapr.os": "centos7",
                "mapr.version": "5.2.1",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "27175b0d7e504d9a33f09b681c8126bc00ec496cb45e0a54525bbcd6af8b3482",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/27175b0d7e50",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "fb5453cdbceae68bf8019cb891184a5e999a27f01aeb47b72dd530dab19bff88",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "f760a1067ed928e7333f6ff64160c38baec3b99998ac5b5d1a9de6397e083982",
                    "EndpointID": "fb5453cdbceae68bf8019cb891184a5e999a27f01aeb47b72dd530dab19bff88",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                }
            }
        }
    }
]


3) List all running and exited container.

[root@node107rhel72 ~]# docker ps -a
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS                        PORTS               NAMES
2ed28c9d3cb1        maprtech/pacc:5.2.1_3.0_centos7_yarn        "/opt/mapr/install..."   2 hours ago         Exited (137) 20 minutes ago                       confident_borg
da11037b9603        maprtech/pacc:5.2.1_3.0_centos7_yarn        "/opt/mapr/install..."   3 days ago          Exited (137) 20 minutes ago                       upbeat_tesla
c469edf549d0        maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse   "/opt/mapr/install..."   4 days ago          Exited (0) 4 days ago                             agitated_feynman
5d71c46be382        maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse   "/opt/mapr/install..."   4 days ago          Exited (127) 4 days ago                           practical_darwin


4) Review logs of exited container "docker logs -t Container-Name".

[root@node107rhel72 ~]# docker logs -t agitated_feynman  | tail -2
2017-05-21T02:43:37.826498971Z [root@c469edf549d0 Container-cluster2]# exit
2017-05-21T02:43:37.826551301Z logout

5) Restart an existing container after it exited , below are to commands to do so i.e start the container and then attach to the container.

[root@node107rhel72 ~]# docker start c469edf549d0
c469edf549d0
[root@node107rhel72 ~]# docker attach c469edf549d0
[root@c469edf549d0 /]#


6) Docker events command can help to log more detailed event details during container startup and troubleshooting . As seen below I can run "docker events" in background and then start container to log more details

[root@node107rhel72 ~]# docker events&
[1] 6062

[root@node107rhel72 ~]# docker start 5d71c46be382
2017-05-24T22:37:50.651821274-07:00 network connect 79009120c0e1776d9322f72697c6d2f9673ffd73652e0fa6e8a6114afa759c61 (container=5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702, name=bridge, type=bridge)
2017-05-24T22:37:50.911087073-07:00 container start 5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702 (build-date=20170510, image=maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse, license=GPLv2, mapr.mep_version=3.0, mapr.os=centos7, mapr.version=5.2.1, name=practical_darwin, vendor=CentOS)
5d71c46be382

[root@node107rhel72 ~]# docker stop 5d71c46be382
2017-05-24T22:42:39.052472873-07:00 container kill 5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702 (build-date=20170510, image=maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse, license=GPLv2, mapr.mep_version=3.0, mapr.os=centos7, mapr.version=5.2.1, name=practical_darwin, signal=15, vendor=CentOS)
2017-05-24T22:42:49.055051971-07:00 container kill 5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702 (build-date=20170510, image=maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse, license=GPLv2, mapr.mep_version=3.0, mapr.os=centos7, mapr.version=5.2.1, name=practical_darwin, signal=9, vendor=CentOS)
2017-05-24T22:42:49.101832392-07:00 container die 5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702 (build-date=20170510, exitCode=137, image=maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse, license=GPLv2, mapr.mep_version=3.0, mapr.os=centos7, mapr.version=5.2.1, name=practical_darwin, vendor=CentOS)
2017-05-24T22:42:49.272937972-07:00 network disconnect 79009120c0e1776d9322f72697c6d2f9673ffd73652e0fa6e8a6114afa759c61 (container=5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702, name=bridge, type=bridge)
5d71c46be382
2017-05-24T22:42:49.317249816-07:00 container stop 5d71c46be382761c4af055c951e6b815b76782f14a731f12a73a958c0029e702 (build-date=20170510, image=maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse, license=GPLv2, mapr.mep_version=3.0, mapr.os=centos7, mapr.version=5.2.1, name=practical_darwin, vendor=CentOS)
[root@node107rhel72 ~]#


7) Delete all containers then Delete all images

[root@noderhel73 ~]# docker rm $(docker ps -a -q)
7389f897234b
5c582b6bba46
83e1c2d29be1
592fec61d139
efbdb75dfdba
dfcfe9705b78
961722aa2680
72d2021e3b4a
9fa14c6c9721
2ffed37c61ac
45ba36812c72
cf1c23d67f16
48621dcb6a68
5ccb73612be7
4d3441c1a2ae
facbe962ee55

[root@noderhel73 ~]# docker rmi $(docker images -q)
Untagged: maprtech/pacc:5.2.1_3.0_centos7_yarn_fuse
Deleted: sha256:86aa95a5262cbce049b7c46e359b0d70122ece6b16bbe6a4ba442911a84cd034
Deleted: sha256:e28f39f9c92fafb20dbaf2613c24f7684b3a1e2cc6a5e3db2740dbad93f0f172
Deleted: sha256:a66aa39fa4e279c4ceac91fdedf14a501dd3e2c7af17cda471cd04a9bfbadf6a
Deleted: sha256:8113731069455ec29f04bc9d77d438f4fb62c460d91a694ea021203b16de9106
Deleted: sha256:99b28d9413e47df267df84318788754dd39d4c8e0dc2c72c9e8922b3a91f21fa
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:382452f82a8bbd34443b2c727650af46aced0f94a44463c62a9848133ecb1aa8
Deleted: sha256:ebcd9d4fca80e9e8afc525d8a38e7c56825dfb4a220ed77156f9fb13b14d4ab7
Deleted: sha256:ef5b99eed7c2ed19ef39f72ac19bb66e16ed6c0868053daae60306a73858fbd4
Deleted: sha256:257e51479af1e9d2e0c9b958e68f6b992329904df24d81efa191cef515a9bf8b
Deleted: sha256:6e1d2d371500e2fe6df75f5755d0b9f2a3b69a42fe88100d514212bbba7ad23f
Deleted: sha256:afa9e7a5e3f3b006942d128c562a3273947c7ab50cdac33fea7213890072a5b6

Deleted: sha256:2df9b8def18a090592bf1cbd1079e1ac2274435c53f027ee5ce0a8faaa5d6d4b