VPC peering between two VPC's in same account

                             VPC peering between two VPC's in same account

VPC peering allows your Databricks clusters to connect to your other AWS infrastructure (RDS, Redshift, Kafka, Cassandra, and so on) using private IP addresses within the internal AWS network. In order to establish a peering connection, both the Databricks VPC and the VPC hosting your other infrastructure must exist in the same AWS region. Also the VPC hosting the other infrastructure must have a CIDR range distinct from the Databricks VPC and any other CIDR range included as a destination in the Databricks VPC main route table.

In this Blog we will see how to create VPC peering between Databricks VPC where databricks shard is setup/running and a VPC where I will run Mysql DB.

Below table is ideal to keep up with details needed for this peering.

VPC	Databricks VPC	vpc-7f4c0d18	10.205.0.0/16
VPC	mysql VPC	vpc-09848ffe9f7781e96	172.30.0.0/16
Route Table	Databricks Main Route Table	rtb-c1c629a7

1) Databricks VPC where databricks shard is setup/running.

2) Create Mysql instance from Amazon RDS in VPC with no overlapping IP's.

Mysql VPC details :

Once Mysql is up we will get the endpoint (externalmetastore.cj11tymkwz5w.us-west-2.rds.amazonaws.com).

3)   Create a peering connection

1. Navigate to the VPC Dashboard.
2. Select Peering Connections.
3. Click Create Peering Connection (Fill details as in screenshot)

Once peering request succeeds you should see as below

4)  Record the ID of the peering connection and update our table.

VPC	Databricks VPC	vpc-7f4c0d18	10.205.0.0/16
VPC	mysql VPC	vpc-09848ffe9f7781e96	172.30.0.0/16
Route Table	Databricks Main Route Table	rtb-c1c629a7
Peering Connection	Databricks VPC <> Mysql VPC	pcx-0f503173dab903f9d

5) Accept the peering connection request as seen in fig below.

6) Add DNS resolution to peering connection

7) Add destination to Databricks VPC main route table

1. Select Route Tables in the VPC Dashboard.
2. Search for the Databricks VPC ID.
3. Click the Edit button under the Routes tab.
4. Click Add another route.
5. Enter the CIDR range of the Aurora VPC for the Destination.
6. Enter the ID of the peering connection for the Target.
   
   8) Add destination to Aurora VPC main route table
   1. Select Route Tables in the VPC Dashboard.
   2. Search for the Mysql VPC ID.
   3. Click the Edit button under the Routes tab.
   4. Click Add another route.
   5. Enter the CIDR range of the Databricks VPC for the Destination.
   6. Enter the ID of the peering connection for the Target.
      
   
   
   
   
   
   
   
   
   
   
   
   
   
   

  9) Test connectivity :

1. Create a Databricks cluster and attach a notebook.
2. Check to see if you can connect to the database with the following netcat command:
   
   
   
   Yay !! the connection was successful .