Setting up Databricks Instance/Shard (AWS)

To set up Databricks you must grant Databricks permission to access your AWS account in which it will create and manage compute and VPC resources. Databricks can use either a cross-account role or access keys. This Blog describes how to configure Databricks via cross-account role.

AWS Account :

Step 1: Configure Databricks to use a cross-account role

2. Click the AWS Account tab.

3. Select the Deploy to AWS using Cross Account Role radio button, in our case i will use Oregon.

4. Make a note of External ID, you will need this in step 2 .

Step 2: Create a cross-account role and an access policy

https://us-west-2.console.aws.amazon.com/console/home?region=us-west-2#

2. Click the Roles tab in the sidebar.

3. Click Create role.

a. In Select type of trusted entity, click the Another AWS account tile.

b. In the Account ID field, enter the Databricks account ID 414351767826.

c. Select the Require external ID checkbox.

d. In the External ID field, paste the Databricks External ID you copied in Step 1.

e. Click the Next: Permissions button.

f. Click the Next: Review button.

g. In the Role name field, enter a role name as captured in below screenshot .

h. Click Create role .

4) In the list of roles, click the role you created.

a. On the Permissions tab, click Add inline policy in lower right corner.

b. In the policy editor, click the JSON tab and copy below access policy.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "Stmt1403287045000",
           "Effect": "Allow",
           "Action": [
               "ec2:AssociateDhcpOptions",
               "ec2:AssociateRouteTable",
               "ec2:AttachInternetGateway",
               "ec2:AttachVolume",
               "ec2:AuthorizeSecurityGroupEgress",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:CancelSpotInstanceRequests",
               "ec2:CreateDhcpOptions",
               "ec2:CreateInternetGateway",
               "ec2:CreateKeyPair",
               "ec2:CreatePlacementGroup",
               "ec2:CreateRoute",
               "ec2:CreateSecurityGroup",
               "ec2:CreateSubnet",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:CreateVpc",
               "ec2:CreateVpcPeeringConnection",
               "ec2:DeleteInternetGateway",
               "ec2:DeleteKeyPair",
               "ec2:DeletePlacementGroup",
               "ec2:DeleteRoute",
               "ec2:DeleteRouteTable",
               "ec2:DeleteSecurityGroup",
               "ec2:DeleteSubnet",
               "ec2:DeleteVolume",
               "ec2:DeleteVpc",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeInstanceStatus",
               "ec2:DescribeInstances",
               "ec2:DescribePlacementGroups",
               "ec2:DescribePrefixLists",
               "ec2:DescribeReservedInstancesOfferings",
               "ec2:DescribeRouteTables",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeSpotInstanceRequests",
               "ec2:DescribeSpotPriceHistory",
               "ec2:DescribeSubnets",
               "ec2:DescribeVolumes",
               "ec2:DescribeVpcs",
               "ec2:DetachInternetGateway",
               "ec2:ModifyVpcAttribute",
               "ec2:RequestSpotInstances",
               "ec2:RevokeSecurityGroupEgress",
               "ec2:RevokeSecurityGroupIngress",
               "ec2:RunInstances",
               "ec2:TerminateInstances"
           ],
           "Resource": [
               "*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "iam:CreateServiceLinkedRole",
               "iam:PutRolePolicy"
           ],
           "Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
           "Condition": {
               "StringLike": {
                   "iam:AWSServiceName": "spot.amazonaws.com"
               }
           }
       }
   ]
}

c. Click review policy on the lower right corner .

d. In Name field , enter policy name and click create policy

6. In the role summary , make a note of Role ARN you will need in step 3 .

Step 3: Configure the cross-account role in your Databricks account

In the Databricks Account Console, return to the AWS Account tab.
Click Next step .

Troubleshooting :

Incase the validation failed . In my case around 9:40 am PST ( 05/02/2018 ) it failed for my account abizer.adenwala+1@databricks.com. Below were 2 messages which but didn't give me enough clue why the validation failed .

Something went wrong, see above.

Failed to perform validation due to a server error. Please try again later or contact help@databricks.com if the problem persists.

Just without looking at web-app logs , I assumed maybe my Databricks login screen expired so i re-logged in and Boom Validation went through.

AWS Storage :

Databricks stores your account-wide assets, such as libraries, in an Amazon Web Services S3 bucket, below steps walks you through the steps to configure your bucket to complete Databricks deployment.

Step 1: Generate S3 bucket policy

1. In the Databricks Account Console you would landing in the AWS Storage tab.

2. Enter the name of your S3 bucket and click on Generate Policy .

Note : Make sure bucket name does not contain uppercase characters or underscores

3. Make a note of this generated policy .

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Grant Databricks Access",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::414351767826:root"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::abizersdeploymenttesting/*",
                "arn:aws:s3:::abizersdeploymenttesting"
            ]
        }
    ]
}

Step 2: Configure the S3 bucket :

In the AWS Console, go to the S3 service and create a bucket.

Step 2a: Apply bucket policy

1) Once the bucket is created click on the bucket name and go to Permissions tab and click on Bucket Policy button.

2) Paste the policy which was noted in Step 1 and click Save .

Step 2b: Enable bucket versioning

For information on versioning, see Using Versioning in the AWS documentation.

Click the Properties tab.

Click the Versioning tile.

Click Enable versioning and click Save.

After versioning is enabled, you can also optionally configure lifecycle rules by following the instructions in How Do I Create a Lifecycle Policy for an S3 Bucket? in the AWS documentation.

Step 3: Apply the change to your Databricks account

Now go back to the Databricks Account Console, go to the AWS Storage tab.
Click Apply Change.

This should take you below screen which means all the configuration is done and Databricks instance is ready to be launched once clicked Launch .

Once the lauch is successful you would see below message about deployment being successful .

Cheers Instance/Shard is up and running !! Below is quick peek at the instance to get me started using DB platform .

Ref links :

https://docs.databricks.com/administration-guide/account-settings/aws-accounts.html

https://docs.databricks.com/administration-guide/account-settings/index.html#aws-storage

AWS/Azure(Cloud)/Spark/Hadoop / Linux

Wednesday, May 2, 2018

Setting up Databricks Instance/Shard (AWS)