Git Basics and commands Part 1

                                 Git based add, delete, modify and commit 

First create a dir where you are working on with files and you would finally be syncing with Git repo .  All commands are run on local Mac till final modifications are done to be ready to commit to repo.

1) Initialize Git to track files under that dir .

C02WG59KHTD5:GitProjects abizeradenwala$ git initInitialized empty Git repository in /Users/abizeradenwala/Desktop/Learning/Git/GitProjects/.git/
.git file indicates the directory is being tracked by GIT .
C02WG59KHTD5:GitProjects abizeradenwala$ ls -ltatotal 8drwxr-xr-x  9 abizeradenwala  staff  288 Jun 13 15:44 .gitdrwxr-xr-x  5 abizeradenwala  staff  160 Jun 13 15:44 .-rw-r--r--  1 abizeradenwala  staff    0 Jun 13 15:43 Git_cheatsheet-rw-r--r--  1 abizeradenwala  staff   13 Jun 13 15:43 git_testdrwxr-xr-x  5 abizeradenwala  staff  160 Jun 13 15:41 ..

2) Make an update to "git_test" file and add the commits to staging before it can be committed.

C02WG59KHTD5:GitProjects abizeradenwala$ git add git_test C02WG59KHTD5:GitProjects abizeradenwala$ git status 

On branch master
No commits yet
Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
new file:   git_test
Untracked files:
  (use "git add <file>..." to include in what will be committed)
Git_cheatsheet

Add everything to the staging in present dir .

git add .

C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch master
No commits yet
Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
new file:   Git_cheatsheet
new file:   git_test

3) Finally commit the changes .

C02WG59KHTD5:GitProjects abizeradenwala$ git commit -m "revision 1"
[master (root-commit) 493ff9a] revision 1 2 files changed, 1 insertion(+) create mode 100644 Git_cheatsheet create mode 100644 git_testC02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masternothing to commit, working tree clean

Check all the past commit logs.

C02WG59KHTD5:GitProjects abizeradenwala$ git log 

commit 493ff9a38d285fb9dc0582c92091aaf1369526d2 (HEAD -> master)Author: abizeradenwala <abizer.adenwala@databricks.com>Date:   Sat Jun 13 15:47:17 2020 -0500
    revision 1
C02WG59KHTD5:GitProjects abizeradenwala$

4) Modify the file and discard changes .

C02WG59KHTD5:GitProjects abizeradenwala$ echo "Appending some data" >> git_test C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges not staged for commit:  (use "git add <file>..." to update what will be committed)  (use "git checkout -- <file>..." to discard changes in working directory)
modified:   git_test
no changes added to commit (use "git add" and/or "git commit -a")

Discard changes

C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending some dataC02WG59KHTD5:GitProjects abizeradenwala$ git checkout git_testC02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is test

5) Delete and retrieve the deleted files back tracked by GIT

C02WG59KHTD5:GitProjects abizeradenwala$ rm git_test C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges to be committed:  (use "git reset HEAD <file>..." to unstage)
modified:   git_test
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
deleted:    git_test
C02WG59KHTD5:GitProjects abizeradenwala$ git checkout git_test
C02WG59KHTD5:GitProjects abizeradenwala$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
modified:   git_test
C02WG59KHTD5:GitProjects abizeradenwala$ ls
Git_cheatsheet git_test
C02WG59KHTD5:GitProjects abizeradenwala$

                                Or

C02WG59KHTD5:GitProjects abizeradenwala$ git rm git_test rm 'git_test'C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masterChanges to be committed:  (use "git reset HEAD <file>..." to unstage)
deleted:    git_test
C02WG59KHTD5:GitProjects abizeradenwala$ ls
Git_cheatsheet
C02WG59KHTD5:GitProjects abizeradenwala$ 

Undo deletes : 

C02WG59KHTD5:GitProjects abizeradenwala$ git reset HEAD git_test 

Unstaged changes after reset:

D git_test

C02WG59KHTD5:GitProjects abizeradenwala$ git checkout -- git_test 

C02WG59KHTD5:GitProjects abizeradenwala$ git status

On branch master

nothing to commit, working tree clean

C02WG59KHTD5:GitProjects abizeradenwala$ ls

Git_cheatsheet git_test

C02WG59KHTD5:GitProjects abizeradenwala$ 

6) Do multiple commits and track those .

C02WG59KHTD5:GitProjects abizeradenwala$ echo "Appending data 2st time" >> git_test C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending data 1st timeAppending data 2st timeC02WG59KHTD5:GitProjects abizeradenwala$ git add git_test C02WG59KHTD5:GitProjects abizeradenwala$  git commit -m "revision 3"[master f096b1b] revision 3 1 file changed, 1 insertion(+)C02WG59KHTD5:GitProjects abizeradenwala$ git statusOn branch masternothing to commit, working tree cleanC02WG59KHTD5:GitProjects abizeradenwala$ git logcommit f096b1ba79695110f36693e048d20c7164893eed (HEAD -> master)Author: abizeradenwala <abizer.adenwala@databricks.com>Date:   Wed Jun 17 15:41:52 2020 -0500
    revision 3
commit e2c4c00104c751f8de592d28c4dac0805762fa0d
Author: abizeradenwala <abizer.adenwala@databricks.com>
Date:   Wed Jun 17 15:40:21 2020 -0500
    revision 2
commit 493ff9a38d285fb9dc0582c92091aaf1369526d2
Author: abizeradenwala <abizer.adenwala@databricks.com>
Date:   Sat Jun 13 15:47:17 2020 -0500
    revision 1
C02WG59KHTD5:GitProjects abizeradenwala$ 

Roll back to first commit version :

C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test this is testAppending data 1st timeAppending data 2st timeC02WG59KHTD5:GitProjects abizeradenwala$ git checkout 493ff9a38d285fb9dc0582c92091aaf1369526d2Note: checking out '493ff9a38d285fb9dc0582c92091aaf1369526d2'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at 493ff9a revision 1
C02WG59KHTD5:GitProjects abizeradenwala$ cat git_test 
this is test
C02WG59KHTD5:GitProjects abizeradenwala$ git status
HEAD detached at 493ff9a
nothing to commit, working tree clean
C02WG59KHTD5:GitProjects abizeradenwala$ 

Connecting to Databricks from PyCharm (On Mac) using Databricks Connect

Databricks Connect allows you to connect your favorite IDE (IntelliJ, Eclipse, PyCharm, RStudio, Visual Studio), notebook server (Zeppelin, Jupyter), and other custom applications to Azure Databricks clusters and run Apache Spark code.

I will walk you through steps to connect your PyCharm installed on Mac book to DB clusters to run jobs against and get results back in stdout of PyCharm .

Note :  Here I will be connecting to cluster with Databricks Runtime version 6.3  and  Python 3.7 . It is assumed you have PyCharm and python 3.7 already setup on your Mac .

      Step 1 : Install the client

1. Uninstall PySpark If installed (In my case it was not installed)
   
   
   C02WG59KHTD5:Downloads abizeradenwala$ pip uninstall pyspark
   Skipping pyspark as it is not installed.
   C02WG59KHTD5:Downloads abizeradenwala$
   
   
   Step 2 : Install the Databricks Connect client (I had older client which is removed automatically)
   
   

C02WG59KHTD5:Downloads abizeradenwala$ /Library/Frameworks/Python.framework/Versions/3.7/bin/pip3 install -U databricks-connect==6.3.*

Collecting databricks-connect==6.3.*

  Downloading https://files.pythonhosted.org/packages/fd/b4/3a1a1e45f24bde2a2986bb6e8096d545a5b24374f2cfe2b36ac5c7f30f4b/databricks-connect-6.3.1.tar.gz (246.4MB)

    100% |████████████████████████████████| 246.4MB 174kB/s 

Requirement already satisfied, skipping upgrade: py4j==0.10.7 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from databricks-connect==6.3.*) (0.10.7)

Requirement already satisfied, skipping upgrade: six in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from databricks-connect==6.3.*) (1.14.0)

Installing collected packages: databricks-connect

  Found existing installation: databricks-connect 5.5.3

    Uninstalling databricks-connect-5.5.3:

      Successfully uninstalled databricks-connect-5.5.3

  Running setup.py install for databricks-connect ... done

Successfully installed databricks-connect-6.3.1

C02WG59KHTD5:Downloads abizeradenwala$ 

        Step 3 : Gather connection properties

-Azure workspace URL(Has orgID)

-Token (PAT)

-Cluster ID

       Step 4 : Configure the connection. You will have to interactively answer and provide details collected above.

C02WG59KHTD5:Downloads abizeradenwala$ databricks-connect configure

Copyright (2018) Databricks, Inc.

...

...

Databricks Platform Services: the Databricks services or the Databricks

Community Edition services, according to where the Software is used.

Licensee: the user of the Software, or, if the Software is being used on

behalf of a company, the company.

Do you accept the above agreement? [y/N] y

Set new config values (leave input empty to accept default):

Databricks Host [no current value, must start with https://]: https://westus2.azuredatabricks.net                                           - Databricks Token [no current value]: XYZZZZZZZZZZZ

IMPORTANT: please ensure that your cluster has:

- Databricks Runtime version of DBR 5.1+

- Python version same as your local Python (i.e., 2.7 or 3.5)

- the Spark conf `spark.databricks.service.server.enabled true` set

Cluster ID (e.g., 0921-001415-jelly628) [no current value]: 0317-213025-tarry631

Org ID (Azure-only, see ?o=orgId in URL) [0]: 6935536957980197

Port [15001]: 

Updated configuration in /Users/abizeradenwala/.databricks-connect

* Spark jar dir: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark/jars

* Spark home: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark

* Run `pip install -U databricks-connect` to install updates

* Run `pyspark` to launch a Python shell

* Run `spark-shell` to launch a Scala shell

* Run `databricks-connect test` to test connectivity

Databricks Connect User Survey: https://forms.gle/V2indnHHfrjGWyQ4A

C02WG59KHTD5:Downloads abizeradenwala$

          Step 5 : Setup spark home via running below command on command line or save it in  ~/.bash_profile

export SPARK_HOME=/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark

         Step 6 : Test connectivity to Azure Databricks.

C02WG59KHTD5:bin abizeradenwala$ databricks-connect test

* PySpark is installed at /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyspark

* Checking SPARK_HOME

* Checking java version

java version "1.8.0_181"

Java(TM) SE Runtime Environment (build 1.8.0_181-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

* Testing scala command

20/03/21 00:55:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

20/03/21 00:55:45 WARN MetricsSystem: Using default name SparkStatusTracker for source because neither spark.metrics.namespace nor spark.app.id is set.

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5-SNAPSHOT

      /_/

         

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)

Type in expressions to have them evaluated.

Type :help for more information.

scala> spark.range(100).reduce(_ + _)

Spark context Web UI available at http://c02wg59khtd5.attlocal.net:4040

Spark context available as 'sc' (master = local[*], app id = local-1584770145654).

Spark session available as 'spark'.

View job details at https://westus2.azuredatabricks.net/?o=6935536957980197#/setting/clusters/0317-213025-tarry631/sparkUi

View job details at https://westus2.azuredatabricks.net/?o=6935536957980197#/setting/clusters/0317-213025-tarry631/sparkUi

res0: Long = 4950

scala> :quit

* Testing python command

20/03/21 00:56:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

20/03/21 00:56:07 WARN MetricsSystem: Using default name SparkStatusTracker for source because neither spark.metrics.namespace nor spark.app.id is set.

View job details at https://westus2.azuredatabricks.net/?o=6935536957980197#/setting/clusters/0317-213025-tarry631/sparkUi

[Stage 6:>                                                          (0 + 4) / 8]* Testing dbutils.fs

[FileInfo(path='dbfs:/FileStore/', name='FileStore/', size=0), FileInfo(path='dbfs:/Knox/', name='Knox/', size=0), FileInfo(path='dbfs:/PradeepKumar/', name='PradeepKumar/', size=0), FileInfo(path='dbfs:/Users/', name='Users/', size=0), FileInfo(path='dbfs:/abc.sh', name='abc.sh', size=20), FileInfo(path='dbfs:/bank-full.csv', name='bank-full.csv', size=4610348), FileInfo(path='dbfs:/bogdan/', name='bogdan/', size=0), FileInfo(path='dbfs:/checkpoint/', name='checkpoint/', size=0), FileInfo(path='dbfs:/cluster-logs/', name='cluster-logs/', size=0), FileInfo(path='dbfs:/databricks/', name='databricks/', size=0), FileInfo(path='dbfs:/databricks-datasets/', name='databricks-datasets/', size=0), FileInfo(path='dbfs:/databricks-results/', name='databricks-results/', size=0), FileInfo(path='dbfs:/dbfs/', name='dbfs/', size=0), FileInfo(path='dbfs:/delta/', name='delta/', size=0), FileInfo(path='dbfs:/foobar', name='foobar', size=876), FileInfo(path='dbfs:/gauarav/', name='gauarav/', size=0), FileInfo(path='dbfs:/gaurav_poc/', name='gaurav_poc/', size=0), FileInfo(path='dbfs:/gaurav_rupnar/', name='gaurav_rupnar/', size=0), FileInfo(path='dbfs:/glm_data.csv', name='glm_data.csv', size=9836), FileInfo(path='dbfs:/glm_model/', name='glm_model/', size=0), FileInfo(path='dbfs:/jordan/', name='jordan/', size=0), FileInfo(path='dbfs:/jose/', name='jose/', size=0), FileInfo(path='dbfs:/jose.gonzalezmunoz@databricks.com/', name='jose.gonzalezmunoz@databricks.com/', size=0), FileInfo(path='dbfs:/knox/', name='knox/', size=0), FileInfo(path='dbfs:/local_disk0/', name='local_disk0/', size=0), FileInfo(path='dbfs:/matt/', name='matt/', size=0), FileInfo(path='dbfs:/ml/', name='ml/', size=0), FileInfo(path='dbfs:/mlflow/', name='mlflow/', size=0), FileInfo(path='dbfs:/mnt/', name='mnt/', size=0), FileInfo(path='dbfs:/piyushmnt/', name='piyushmnt/', size=0), FileInfo(path='dbfs:/pradeepkumar/', name='pradeepkumar/', size=0), FileInfo(path='dbfs:/rdd1-1562366996207/', name='rdd1-1562366996207/', size=0), FileInfo(path='dbfs:/scripts/', name='scripts/', size=0), FileInfo(path='dbfs:/takeshi/', name='takeshi/', size=0), FileInfo(path='dbfs:/te', name='te', size=36), FileInfo(path='dbfs:/test/', name='test/', size=0), FileInfo(path='dbfs:/test1/', name='test1/', size=0), FileInfo(path='dbfs:/testing/', name='testing/', size=0), FileInfo(path='dbfs:/testing1/', name='testing1/', size=0), FileInfo(path='dbfs:/testing2', name='testing2', size=3717), FileInfo(path='dbfs:/tmp/', name='tmp/', size=0), FileInfo(path='dbfs:/tmp1/', name='tmp1/', size=0), FileInfo(path='dbfs:/user/', name='user/', size=0), FileInfo(path='dbfs:/xin/', name='xin/', size=0), FileInfo(path='dbfs:/xyz.sh', name='xyz.sh', size=20), FileInfo(path='dbfs:/{workingDir}/', name='{workingDir}/', size=0)]

* All tests passed.

C02WG59KHTD5:bin abizeradenwala$

This confirms Mac can connect to Databricks cluster remotely.  

Configuring PyCharm 

• Create New Project → give it a name (dbconnectabizer)

• Specify interpreter → File → Preference for new projects → expand your user folder  → find python3.7 → OK → Create

- Also install Databricks-connect package as show below and click ok.

• Select your project → New → Python File
• Create dbctest (will create a .py file)
• Run → Edit Configurations 
• Click “+” icon (top left) → Python → Script path (.py file created earlier) → Open 
• Add Environment variables → Add new → PYSPARK_PYTHON, python3 → Apply & OK
  
  
  
  
  
  
  
  
  
• Apply
• Type your code (execute something from PyCharm to Databricks) → Run

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

from pyspark.sql.functions import col

song_df = spark.read \

    .option('sep','\t') \

    .option("inferSchema","true") \

    .csv("/databricks-datasets/songs/data-001/part-0000*")

tempo_df = song_df.select(

                    col('_c4').alias('artist_name'),

                    col('_c14').alias('tempo'),

                   )

avg_tempo_df = tempo_df \

    .groupBy('artist_name') \

    .avg('tempo') \

    .orderBy('avg(tempo)',ascending=False)

print("Calling show command which will trigger Spark processing")

avg_tempo_df.show(truncate=False)

• You can see it’s executing against the cluster
  
  
  
  
  

Bingo !!! we got can now execute spark code from PyCharm to Databricks and vie the results in stdout .

Network debugging from Databricks notebook

               Network debugging from Databricks notebook

Network setup issues:

Any time you messages as below in init script logs or when you do simple Apt-get install it usually means our notebook cannot reach archive.ubuntu.com on port 80 .

Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) [IP: 2001:67c:1560:8001::14 80]

or if you see messages with stack trace in driver logs.

Caused by: java.util.concurrent.TimeoutException: Timeout during file system operation after 1200 seconds. Please configure databricks.data.rpcTimeout to change this timeout.

Above can be due to number of reasons .

1) Outgoing traffic to internet is blocked and hence it cannot reach to archive.ubuntu.com hence you see "Network is unreachable"

%sh ping -c 2 google.com

PING google.com (172.217.9.238): 56 data bytes--- google.com ping statistics ---2 packets transmitted, 0 packets received, 100% packet loss

Mostlike here traffic is blocked by SG or NACL . Below blog has all details on what ports need to be open for communication. If not route setup for the instances are incorrect .

https://abizeradenwala.blogspot.com/2018/09/nacl-and-security-group-settings-in.html

2) Another test for this could be with Telnet to specific hostname and port  . You should see message as below any other message like timeout/ connection refused etc means driver is not able to reach to specific host on specific port .

%sh telnet archive.ubuntu.com 80

Trying 91.189.88.24...Connected to archive.ubuntu.com.Escape character is '^]'.

Or use NC to see if you can get to port and is connection can be opened.

%sh nc -zv archive.ubuntu.com 80archive.ubuntu.com [91.189.88.149] 80 (http) open

If you don't see output as above Issues could be due to incorrect VPC peering or ports not opening or firewall . Below blog can help debug .

https://abizeradenwala.blogspot.com/search?q=vpc+peering

3) Also its possible your driver is not able to resolve the hostname, you can quickly check if hostname is getting resolved to IP address you expect to else you can try step 2 with IP instead of hostname to confirm .

%sh nslookup archive.ubuntu.com

Server: 10.177.0.2 Address: 10.177.0.2#53 Non-authoritative answer: Name: archive.ubuntu.com Address: 91.189.91.23 Name: archive.ubuntu.com Address: 91.189.91.26 Name: archive.ubuntu.com Address: 91.189.88.24 Name: archive.ubuntu.com Address: 91.189.88.31 Name: archive.ubuntu.com Address: 91.189.88.149 Name: archive.ubuntu.com Address: 91.189.88.162

Note : -  In unlikely event where server name nslookup is incorrect then it could be due to DNS caching old record info and server IP has changed .

%sh  nslookup -type=SOA novartis-prod.cloud.databricks.com

Server: 10.177.0.2 Address: 10.177.0.2#53 Non-authoritative answer: novartis-prod.cloud.databricks.com canonical name = dbc-67a72554-618d.cloud.databricks.com. dbc-67a72554-618d.cloud.databricks.com canonical name = ec2-3-90-35-224.compute-1.amazonaws.com. Authoritative answers can be found from: compute-1.amazonaws.com origin = dns-external-master.amazon.com mail addr = root.amazon.com serial = 2013233418 refresh = 28800 retry = 900 expire = 2592000 minimum = 7211 -> Default TTL for record

Flaky Network issues:

Most issues would be solved here but there can be cases where connection Randomly fails . This usually points to problem where network is flaky and not necessarily blocked by Firewall or incorrect configuration.

Messages as below

Caused by: java.io.IOException: SQL Server did not return a response. The connection has been closed. ClientConnectionId:XXX-XXX-XXXX

In above case we need to capture TCP dump from driver for all the traffic going to 3306 i.e Mysql DB .

%scala dbutils.fs.put("dbfs:/databricks/init_scripts/take_tcpdump.sh", """#!/bin/bash echo "initiating tcp dump" sudo tcpdump -w /dbfs/databricks/tcpdump/trace_%Y_%m_%d_%H_%M_%S.pcap -W 1000 -G 1800 -K -n port 3306 > /dbfs/databricks/tcpdump/tcpdump.log 2>&1 & """,true)

Once the issue is hit, download the pcap file of interest and use Wireshark to understand when and how the connection is getting closed .

Below is good example where connection was getting closed abruptly and TCPdump helped to get good understanding on the problem .

https://abizeradenwala.blogspot.com/2018/02/analyze-broken-pipe-error-in-hive.html

Network Latency issues:

Network latency issues is either due to bad node or network choke caused by bad/slow network .

 To troubleshoot Network latency, one thing that can be checked is the send queue sizes for open TCP connections (e.g. the third column in output of "netstat -pan").  On a normal operating network, where the source and destination machines have free memory/CPU and there is no network bottleneck on the interfaces of the source and dest machines, the send queue size should not be more than a few thousand bytes at most.  If you see a lot of connections with 10K+ bytes in the send queue then that generally indicates some sort of problem.


Collect "netstat -pan" every 30 seconds with timestamps during the time of the problem to review .

i)  If you see lots of connections with large send queue sizes but only on one particular node, that would typically indicate that one node is having trouble sending data out onto the network.

ii) If you see connections on lots of different nodes with large send queue sizes and the destinations of those connections are all to one particular node then that indicates that one particular node is having trouble receiving data.

iii) If you see connections on many different nodes with large send queue sizes, and the destinations of those connections are also to a wide variety of different nodes then that indicates a cluster wide issue such as a faulty switch.