Kubenetes 101

                                                                 Kubenetes 101 

Architecture :

Master :

- Node has 4 process - API server (Client, command line eg kubectl etc), Scheduler (Decides which worker node pod will be scheduled on), Controller Manager (Detect state changes of Pod and recover) and etcd (state of cluster in form of Kv store)

Worker Nodes (Multiple):

- App Pods where work is done.

- Node has 3 processes on every node :  Contrainer Runtime, Kubelet (Schedules and tracks Pod on local node) and Kube-Proxy(Networking related decisions)

 

Resources in K8S :

Pod - Smallest unit of K8s and each pod has an IP (Non static across restarts)

Service - Perm IP to Pod's (Even if service terminates IP stays)

   Internal Service -  Hostname:Port  (Risk to expose Hostname)  usually ClusterIP

   External Service - LB Service

Ingress : Service which talks to external word and passes traffic internally to services. ( https://your-app.com )

ConfigMap -  To have configs externally. 

Secrets -  To have credentials stored and stored in base64 encoded  (Passwd/Cert).

Volume : Local/External storage to persist data across Pod restart 

Deployments : Blue print for most application deployments.  Abstraction level over Pods .

StatefulSet - For DB app's to make sure writes are synchronized (Avoid corruption).  

Writing Functions and Importing Custom/System Functions

 

                    Writing Functions and Importing Custom/System Functions

# writing custom function to find index 

print('Imported testp...')

def find_index(to_search, target):
    '''Find the index of a value in a sequence'''
    for i, value in enumerate(to_search):
        if value == target:
            return i

    return -1



# Import sys module and printing all System Path.

import sys
print(sys.path)

# Import OS module and print location of file where the function is defined.

import os
print(os.__file__)

Comparisons and Loops with break/Continue

# Comparisons:


# Equal:            ==
# Not Equal:        !=
# Greater Than:     >
# Less Than:        <
# Greater or Equal: >=
# Less or Equal:    <=
# Object Identity:  is



Break statement :

nums = [1,2,3,4,5] for num in nums: if num == 2: ### Here we are breaking out of while loop when num is equal to 2, output will only be 1 break print(num) num += 1
Continue statement : nums = [1,2,3,4,5] for num in nums: if num == 2: ### Here we are passing off when num is equal to 2, so output is 1,3,4,5 continue print(num) num += 1

List, Tuples, Set and Dictionary

                                          List, Tuples, Set and Dictionary

# Empty Lists - Mutable
empty_list = []
empty_list = list()

# Empty Tuples - Immutable
empty_tuple = ()
empty_tuple = tuple()

# Empty Sets - throw away duplicates or membership test
empty_set = set()

# Empty Dict - Key Value pairs

empty_dict = {} # This isn't right! It's a dict

Examples  
list_1 = ['History', 'Math', 'Physics', 'CompSci']
print(list_1)

Update list
list_1[0] = 'Art'


#tuple
tuple_1 = ('History', 'Math', 'Physics', 'CompSci')

print(tuple_1)
# Sets
cs_courses = {'History', 'Math', 'Physics', 'CompSci'}

print(cs_courses)

# Dictionary
student = {'name': 'John', 'age': 25, 'courses': ['Math', 'CompSci']}

for key, value in student.items():
    print(key, value)

Python String formatting

                                       Python String formatting

                                             

  Formatting string in python can be achieved via different ways as below.

Declaring variables :

greeting = 'Hello'

name = 'Abizer'

morninggreeting = 'Morning !'

Concatenating string (Default and easy when not many strings need to be Concatenated)

message =  greeting + ", " + name + "." + morninggreeting

Formatting strings via below method to make it more readable.

message =  '{}, {}.{}'.format(greeting, name, morninggreeting)

If your using 3.6+ version of Python we can use f strings to achieve same thing

message = f'{greeting}, {name}.{morninggreeting}'

Finally print the output.

print(message)

Output :

Hello, Abizer.Morning !

Git Basics and commands Part 1

                                 Git based add, delete, modify and commit 

First create a dir where you are working on with files and you would finally be syncing with Git repo .  All commands are run on local Mac till final modifications are done to be ready to commit to repo.

1) Initialize Git to track files under that dir .

C02WG59KHTD5:GitProjects abizeradenwala$ git initInitialized empty Git repository in /Users/abizeradenwala/Desktop/Learning/Git/GitProjects/.git/
.git file indicates the directory is being tracked by GIT .
C02WG59KHTD5:GitProjects abizeradenwala$ ls -ltatotal 8drwxr-xr-x  9 abizeradenwala  staff  288 Jun 13 15:44 .gitdrwxr-xr-x  5 abizeradenwala  staff  160 Jun 13 15:44 .-rw-r--r--  1 abizeradenwala  staff    0 Jun 13 15:43 Git_cheatsheet-rw-r--r--  1 abizeradenwala  staff   13 Jun 13 15:43 git_testdrwxr-xr-x  5 abizeradenwala  staff  160 Jun 13 15:41 ..

2) Make an update to "git_test" file and add the commits to staging before it can be committed.

C02WG59KHTD5:GitProjects abizeradenwala$ git add git_test C02WG59KHTD5:GitProjects abizeradenwala$ git status 

On branch master
No commits yet
Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
new file:   git_test
Untracked files:
  (use "git add <file>..." to include in what will be committed)
Git_cheatsheet

Add everything to the staging in present dir .

git add .

C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch master
No commits yet
Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
new file:   Git_cheatsheet
new file:   git_test

3) Finally commit the changes .

C02WG59KHTD5:GitProjects abizeradenwala$ git commit -m "revision 1"
[master (root-commit) 493ff9a] revision 1 2 files changed, 1 insertion(+) create mode 100644 Git_cheatsheet create mode 100644 git_testC02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masternothing to commit, working tree clean

Check all the past commit logs.

C02WG59KHTD5:GitProjects abizeradenwala$ git log 

commit 493ff9a38d285fb9dc0582c92091aaf1369526d2 (HEAD -> master)Author: abizeradenwala <abizer.adenwala@databricks.com>Date:   Sat Jun 13 15:47:17 2020 -0500
    revision 1
C02WG59KHTD5:GitProjects abizeradenwala$

4) Modify the file and discard changes .

C02WG59KHTD5:GitProjects abizeradenwala$ echo "Appending some data" >> git_test C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges not staged for commit:  (use "git add <file>..." to update what will be committed)  (use "git checkout -- <file>..." to discard changes in working directory)
modified:   git_test
no changes added to commit (use "git add" and/or "git commit -a")

Discard changes

C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending some dataC02WG59KHTD5:GitProjects abizeradenwala$ git checkout git_testC02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is test

5) Delete and retrieve the deleted files back tracked by GIT

C02WG59KHTD5:GitProjects abizeradenwala$ rm git_test C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges to be committed:  (use "git reset HEAD <file>..." to unstage)
modified:   git_test
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
deleted:    git_test
C02WG59KHTD5:GitProjects abizeradenwala$ git checkout git_test
C02WG59KHTD5:GitProjects abizeradenwala$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
modified:   git_test
C02WG59KHTD5:GitProjects abizeradenwala$ ls
Git_cheatsheet git_test
C02WG59KHTD5:GitProjects abizeradenwala$

                                Or

C02WG59KHTD5:GitProjects abizeradenwala$ git rm git_test rm 'git_test'C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges to be committed:  (use "git reset HEAD <file>..." to unstage)
deleted:    git_test
C02WG59KHTD5:GitProjects abizeradenwala$ ls
Git_cheatsheet
C02WG59KHTD5:GitProjects abizeradenwala$ 

Undo deletes : 

C02WG59KHTD5:GitProjects abizeradenwala$ git reset HEAD git_test 

Unstaged changes after reset:

D git_test

C02WG59KHTD5:GitProjects abizeradenwala$ git checkout -- git_test 

C02WG59KHTD5:GitProjects abizeradenwala$ git status

On branch master

nothing to commit, working tree clean

C02WG59KHTD5:GitProjects abizeradenwala$ ls

Git_cheatsheet git_test

C02WG59KHTD5:GitProjects abizeradenwala$ 

6) Do multiple commits and track those .

C02WG59KHTD5:GitProjects abizeradenwala$ echo "Appending data 2st time" >> git_test C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending data 1st timeAppending data 2st timeC02WG59KHTD5:GitProjects abizeradenwala$ git add git_test C02WG59KHTD5:GitProjects abizeradenwala$  git commit -m "revision 3"[master f096b1b] revision 3 1 file changed, 1 insertion(+)C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masternothing to commit, working tree cleanC02WG59KHTD5:GitProjects abizeradenwala$ git logcommit f096b1ba79695110f36693e048d20c7164893eed (HEAD -> master)Author: abizeradenwala <abizer.adenwala@databricks.com>Date:   Wed Jun 17 15:41:52 2020 -0500
    revision 3
commit e2c4c00104c751f8de592d28c4dac0805762fa0d
Author: abizeradenwala <abizer.adenwala@databricks.com>
Date:   Wed Jun 17 15:40:21 2020 -0500
    revision 2
commit 493ff9a38d285fb9dc0582c92091aaf1369526d2
Author: abizeradenwala <abizer.adenwala@databricks.com>
Date:   Sat Jun 13 15:47:17 2020 -0500
    revision 1
C02WG59KHTD5:GitProjects abizeradenwala$ 

Roll back to first commit version :

C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending data 1st timeAppending data 2st timeC02WG59KHTD5:GitProjects abizeradenwala$ git checkout 493ff9a38d285fb9dc0582c92091aaf1369526d2Note: checking out '493ff9a38d285fb9dc0582c92091aaf1369526d2'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at 493ff9a revision 1
C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test 
this is test
C02WG59KHTD5:GitProjects abizeradenwala$ git status
HEAD detached at 493ff9a
nothing to commit, working tree clean
C02WG59KHTD5:GitProjects abizeradenwala$ 

Access Databricks workspace via Azure Active Directory tokens (Non-admin)

This article describes how a service principal can be used to access DB workspace via rest api . SP will be given contributor role but its added to workspace as non-admin.

Service principal defined in Azure Active Directory (Azure AD) can also act as a principal on which authentication and authorization policies can be enforced in Azure Databricks. Service principals in an Azure Databricks workspace can have different fine-grained access control than regular users (user principals) and be used to access ADB.

1) Provision a service principal in Azure portal as below

2) Click Certificates & secrets and generate a new client secret.

Note :- Secrets used in this blog have been invalidated for security reasons.

3) Assign the SP contributor role.

4) Get an Azure Active Directory access token

curl -X GET -H 'Content-Type: application/x-www-form-urlencoded' \

-d 'grant_type=client_credentials&client_id=<client-id>&resource=<azure_databricks_resource_id>&client_secret=<application-secret>' \

https://login.microsoftonline.com/<tenant-id>/oauth2/token

5) Decide group to which SP will be added and fire rest API to add SP to group .

i) Group 7 is empty 

ii) Add SP to the group

iii) Confirm it gets added .

6) Finally use a Azure Active Directory access token to access the Databricks REST API

Access Databricks workspace via Azure Active Directory tokens (Admin)

This article describes how a service principal can be used to access DB workspace via rest api . SP will be given contributor role but it doesn't belong to workspace .

Service principal defined in Azure Active Directory (Azure AD) can also act as a principal on which authentication and authorization policies can be enforced in Azure Databricks. Service principals in an Azure Databricks workspace can have different fine-grained access control than regular users (user principals) and be used to access ADB.

1) Provision a service principal in Azure portal as below

2) Click Certificates & secrets and generate a new client secret.

Note :- Secrets used in this blog have been invalidated for security reasons.

3) Assign the SP contributor role.

4) Get an Azure Active Directory access token

curl -X GET -H 'Content-Type: application/x-www-form-urlencoded' \

-d 'grant_type=client_credentials&client_id=<client-id>&resource=<azure_databricks_resource_id>&client_secret=<application-secret>' \

https://login.microsoftonline.com/<tenant-id>/oauth2/token

5) Get the Azure Management Resource endpoint token

6) Finally use a management endpoint access token to access the Databricks REST API