Wednesday, April 23, 2025

SQL Server - Get all Tables and Columns from ALL Databases

 This query below gives you All Tables with Columns in All the database in SQL Server:

SET NOCOUNT ON
DECLARE @AllTables table (DbName sysname,SchemaName sysname, TableName sysname,
ColumnID sysname, ColumnName sysname, ColumnType sysname, IsNullable sysname)
DECLARE
@SearchDb nvarchar(200)
,@SearchSchema nvarchar(200)
,@SearchTable nvarchar(200)
,@SQL nvarchar(4000)
SET @SearchDb='%'
SET @SearchSchema='%'
SET @SearchTable='%'
SET @SQL='select ''?'' as DbName, s.name as SchemaName, t.name as TableName,
c.column_id, c.name ColumnName, tp.name ColumnType, c.is_nullable
from [?].sys.tables t
inner join [?].sys.schemas s
on t.schema_id=s.schema_id
inner join [?].sys.columns c
on t.object_id = c.object_id
inner join [?].sys.types tp
on c.system_type_id = tp.system_type_id
where ''?'' like '''+@SearchDb+'''
and s.name like '''+@SearchSchema+'''
and t.name like '''+@SearchTable+''''

INSERT INTO @AllTables (DbName, SchemaName, TableName, ColumnID, ColumnName, ColumnType, IsNullable)
EXEC sp_msforeachdb @SQL
SET NOCOUNT OFF
--SELECT DbName, SchemaName, count(1) FROM @AllTables
select DbName, SchemaName, TableName, ColumnName, ColumnType,
case when IsNullable = 0 then 'Not Null' else '' end "IsNullable"
from @AllTables
where DbName in ('list of dbs interested in or comment out this filter')
--group by DbName, SchemaName
ORDER BY DbName, SchemaName, TableName, ColumnID;

Wednesday, June 15, 2022

Microsoft Teams Takes forever to upload a file

Got annoyed with Teams taking forever to upload a simple file. 

Found that clearing Teams Cache would help and here are the steps to do so in Mac

1] Quit Teams

2] Go to Finder and get to ~/Library/Application Support/Microsoft

3] Delete the folder Teams from there

4] Go to Keychain Access App and search for Microsoft Teams and delete the entry.

5] Restart Teams 

Credit goes to: https://www.uvm.edu/it/kb/article/clearing-teams-cache/

Wednesday, March 23, 2022

Snowflake Constraint Column Details

 At the time of this post, Snowflake did not have any dictionary view that gives the column names for constraints that are defined for a table. 

So, here is what I found to get that list:

show primary keys; --and then get the query id from this and plug it in below query

select "database_name" db_name, "schema_name" schema_name, "table_name" table_name, "column_name" column_name, "key_sequence" key_seq
  from TABLE(RESULT_SCAN('query id from show command above'))
 where not "table_name" like any ('SAN_%', '%_STAGE', '%_STAGE_CLONE', 'BACKUP_%', 'raw_%', 't_raw_%', '_202%')
 order by 1,2,3,5;

Repeat same for unique keys and imported keys(foreign) 


Tuesday, June 1, 2021

awk - Exclude or Include Records from a fixed-width file

The below command gets the records that have a value "new" in the given position (in this case the file was fixed-width). All other records which do not have the value "new" are ignored.

 awk '{if (substr($0, 7, 3) == "new") print $0}' file1.txt

substr 7, 3 is the starting position 7 with 3 characters

sed - Add a line to all files


#below inserts a line at 1st row with the content "Added new line using sed " in all the files in the path

for i in *; do sed -i '1s/^/Added new line using sed \n/' $i; done


#below inserts a line at 3rd row with the content "Added new line using sed " in all the files in the path

for i in *; do sed -i '3s/^/Added new line using sed \n/' $i; done

Friday, April 2, 2021

DBeaver Not Responding in macOS

Not quite often but sometimes when either DBeaver or your macOS decides to frustrate you (particularly when you are already frustrated with the work 😉) then you will see DBeaver not starting up with just a spinning rainbow wheel forever. 

In The Activity Monitor, it says DBeaver not responding!!!

Force Quit DBeaver and navigate to this folder:

/Users/schanna/Library/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench

and remove the file: workbench.xmi 

schanna@schanna org.eclipse.e4.workbench % rm workbench.xmi 

Now, you should be back to business.

Tuesday, August 25, 2020

Split a word based on case sensitive...

Received data like "AetnaMercyCareRBHA" in one of our records and showing this data in the reports is quite doesn't readable. 

So, the ask is to display the value like "Aetna Mercy Care RBHA" instead.

This is in Snowflake:

select regexp_replace(regexp_replace('AetnaMercyCareRBHA', '([A-Z])', ' \\1'), '([A-Z])( )', '\\1') ;

Note that there is space before escape character in the inner regexp like ' \\1' and space within the paranethesis in the outer regexp like ( ).

The above simple regexp_replace does the trick. 

Inner regexp_replace divides the word with space whenever there is an uppercase letter.

Outer regexp_replace removes the space between upper case letters eg., "R B H A" to "RBHA"


Wednesday, July 31, 2019

Model-based and Model-free in Reinforcement Learning (RL)

I was going thru some predictive analytics reading and came across this interesting terms "Model-Based and Model-Free" and found the below excerpt from this pdf and thought of blogging it over here for my own future reference as this explained so well and easy to understand.

Reinforcement learning methods can broadly be divided into two classes, model-based and model-free.

Consider the problem illustrated in the figure below, of deciding which route to take on the way home from work on Friday evening.

We can abstract this task as having states (in this case, locations, notably of junctions), actions (e.g. going straight on or turning left or right at every intersection), probabilities of transitioning from one state to another when a certain action is taken (these transitions are not necessarily deterministic, e.g. due to road works and bypasses), and positive or negative outcomes (i.e. rewards or costs) at each transition from scenery, traffic jams, fuel consumed, etc. (which are again probabilistic).

Model-based computation, illustrated in the left ‘thought bubble’, is akin to searching a mental map (a forward model of the task) that has been learned based on previous experience. This forward model comprises knowledge of the characteristics of the task, notably, the probabilities of different transitions and different immediate outcomes. Model-based action selection proceeds by searching the mental map to work out the longrun value of each action at the current state in terms of the expected reward of the whole route home, and chooses the action that has the highest value.

Model-free action selection, by contrast, is based on learning these long-run values of actions (or a preference order between actions) without either building or searching through a model. RL provides a number of methods for doing this, in which learning is based on momentary inconsistencies between successive estimates of these values along sample trajectories. These values, sometimes called cached values because of the way they store experience, encompass all future probabilistic transitions and rewards in a single scalar number that denotes the overall future worth of an action (or its attractiveness compared with other actions). For instance, as illustrated in the right ‘thought bubble’, experience may have taught the commuter that on Friday evenings the best action at this intersection is to continue straight and avoid the freeway.

Model-free methods are clearly easier to use in terms of online decision-making; however, much trial-and-error experience is required to make the values be good estimates of future consequences. Moreover, the cached values are inherently inflexible: although hearing about an unexpected traffic jam on the radio can immediately affect action selection that is based on a forward model, the effect of the traffic jam on a cached propensity such as ‘avoid the freeway on Friday evening’ cannot be calculated without further trial-and-error learning on days in which this traffic jam occurs. Changes in the goal of behavior, as when moving to a new house, also expose the differences between the methods: whereas model-based decision making can be immediately sensitive to such a goal-shift, cached values are again slow to change appropriately. Indeed, many of us have experienced this directly in daily life after moving house. We clearly know the location of our new home, and can make our way to it by concentrating on the new route; but we can occasionally take an habitual wrong turn toward the old address if our minds wander. Such introspection, and a wealth of rigorous behavioral studies (see [15], for a review) suggests that the brain employs both model-free and model-based decision-making strategies in parallel, with each dominating in different circumstances [14]. Indeed, somewhat different neural substrates underlie each one [17].


Thursday, June 27, 2019

Connecting DBeaver with Spark Databricks Cluster


Got the Simba JDBC drivers from databricks.

Extracted the zip and then SimbaSparkJDBC41-2.6.3.1003.zip

Adding Simba driver to DBeaver:

In the DBeaver:
  1. Driver Manager
    1. Select New
                                                               i.      Give some name to Driver Name – Label only
                                                             ii.      Click on Add file and select the SimbaSparkJDBC41-2.6.3.1003.jar file
                                                           iii.      Add com.simba.spark.jdbc41.Driver to the Class Name: (Class name is as of 06/27/2019)
Getting JDBC URL from Databricks:
  1. Goto your Cluster from Databricks
  2. Click on Advanced Options in the Configuration tab
  3. Click on JDBC/ODBC tab
  4. Grab the JDBC URL provided which will look like below:
jdbc:spark://<server-name-info>:<port>/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/0/cluster-name;AuthMech=3;UID=token;PWD=<personal-access-token>

Getting the Token from Databricks:
  1. Click on the user Icon on the top right most corner and click on User Settings.
  2. Select Access Tokens and create a token. NOTE: Pay attention to the dialogue box as this token is only showed once so you save it first.

Connecting Spark thru DBeaver:
  1. Click New Database connection in DBeaver
  2. Select the driver you just added – check for the label you provided for Driver Name when you added Simba driver
  3. Copy the URL you obtained from databricks into JDBC URL:
  4. Provide the token you obtained in PWD= in the URL
jdbc:spark://<server-name-info>:<port>/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/0/cluster-name;AuthMech=3;UID=token;PWD= ##############
  1. Test Connection and you should be connected to your cluster and should be able to see all databases.

Added UseNativeQuery=1 at the end of url as I was getting errors in DBeaver:

jdbc:spark://<server-name-info>:<port>/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/0/cluster-name;AuthMech=3;UID=token;PWD=##############;UseNativeQuery=1

Thursday, May 30, 2019

Concatenated string to individual rows in Spark SQL, PG and Snowflake

I had this column named age_band which will have values like "55-64|65-74|75+"
As you can see it contains age groups stored in as a string concatenated with '|' and each age group needs to be compared separately.

I had this taken care in PostgreSQL using:


select unnest(string_to_array('55-64|65-74|75+', '|'));


which results:

# unnest
1 55-64
2 65-74
3 75+

Now, I have to perform the same in Spark SQL and here it is:


select explode(split('55-64|65-74|75+', '[|]'));


and here is the result:

col
55-64
65-74
75+

Update: 11/27/2019

And in SnowFlake:

select col1, c.value::string as age_band
  from san_test,
     lateral flatten(input=>split(col2, '|')) c;

col2 is the column where the age_band has concatenated values with | delimited. 

select c.value::string as age_band from lateral flatten(input=>split('55-64|65-74|75+', '|')) c;

AGE_BAND 55-64 65-74 75+

Monday, March 18, 2019

AWS Storage Systems Comparison - S3 vs EBS vs EFS

The table below shows the major differences (not all) between available AWS storage systems.
Note I am not including Glacier here as Glacier is purely meant for archiving and it doesn't make sense to compare with other storage systems -at least as of this day :)


Feature/Product S3 (Simple Storage Service) EBS (Elastic Block Storage) EFS (Elastic File Storage)
Accessibility Publicly accessible with key-id Only thru the associated EC2 instance Can be accessed thru more than one EC2 and AWS services
Access Control IAM
Bucket policies and User policies
IAM
Security Groups
IAM
Security Groups
Interface Web Interface File System Interface Web and File System Interface
Storage Type Object Storage Block Storage Object Storage
Cost $$ Cheaper than EBS and EFS > S3 and < EFS More expensive than EBS
Scalable Scalable Manual Scalable
Performance Slower than EBS and EFS Faster than S3 and EFS Faster than S3 and slower than EBS
Best used for Storing backups meant to be EC2 drive Sharable apps and workloads.

Wednesday, December 12, 2018

How to Add a shell script to Mac Applications...

Came across with this situation where I keep forgetting in which folder "that" particular shell script is at which I used a while ago... Damn, I should have placed it under "Applications" so that I don't have to waste my time looking for a script.....

Anyways, found an easy way how I can make a shell script to place under "Applications" folder and use my Launchpad to find it in there.

Step#1: Type "automator" in the Spotlight Search (command + space bar)
Step#2: Choose "Application" when asked for "Choose a type for your document"
Step#3: Type "shell" in the search box right next to "Variables"
Step#4: Doble click on "Run Shell Script"
Step#5: You will get a "Run Shell Script" window on the side as shown below



Step#6: You can simply drag and drop the shell script from your folder or type in the box as shown below



Step#7: Now, save the file with File Format as "Application" under "Applications" and now you can see your named application in Launchpad.




Wednesday, August 8, 2018

AWS Athena Vs Redshift...

In recent years data warehouse architecture has a huge shift towards Cloud-based.

Here are the few benefits of cloud-based data warehousing comparing to traditional on-premise: Scalability, Cost, Performance and Time to market.

I will try to cover the concepts/considerations between cloud vs on-premise in another blog post.

I am already working on Google BigQuery and thought of spending some time in AWS offerings in data warehousing arena and here is what I have learned so far.

AWS offers Redshift as their main data warehousing capability. But also comes Athena.

Athena is a serverless service and does not need any infrastructure to create, manage, or scale data sets. It works directly on top of Amazon S3 data sets. It creates external tables and therefore does not manipulate S3 data sources, working as a read-only service from an S3 perspective. Athena uses Presto and ANSI SQL to query on the data sets. It also uses HiveQL for DDL statements.

On the other hand, Redshift is a petabyte-scale data warehouse used together with business intelligence tools for modern analytical solutions. Unlike Athena, Redshift requires a cluster for which we need to upload the data extracts and build tables before we can query. Redshift is based on PostgreSQL 8.0.2.

Comparing Athena to Redshift is not simple. Athena has an edge in terms of portability and cost, whereas Redshift stands tall in terms of performance and scale.


Component Athena Redshift
Serverless  Yes. Athena works with the data hosted on S3. No. Need to setup AWS Cluster(s)
Query  Athena uses Presto and ANSI SQL to query on the data sets. It also uses HiveQL for DDL. Redshift is based on Postgres 8.0.2
Datafile Format CSV, JSON (Both simple and nested),
Columnar Storage, ORC and Parquet.
CSV, JSON (Both simple and nested), TSV and Apache logs.
Compression Supports all compressed formats except LZO (Snappy can be used instead) Supports All compression methods
Data Types Supports complex data types like:
Arrays, maps, and structures.
No Support for Complex data types.
Partitioning Limit on Singly key or multi-column partitioning is available.
20,000 per Table
Distribution Style. There are 3 styles available when creating a table:
1] Even - This is the Default Distribution style.
The leader node distributes the rows across the slices in a round-robin fashion, regardless of the values in any particular column.
EVEN distribution is appropriate when a table does not participate in joins or when there is not a clear choice between KEY distribution and ALL distribution.
2] Key - The rows are distributed according to the values in one column.
The leader node will attempt to place matching values on the same node slice. If you distribute a pair of tables on the joining keys, the leader node collocates the rows on the slices according to the values in the joining columns so that matching values from the common columns are physically stored together.
3] All - A copy of the entire table is distributed to every node.
Where EVEN distribution or KEY distribution place only a portion of a table's rows on each node, ALL distribution ensures that every row is collocated for every join that the table participates in.
ALL distribution is appropriate only for relatively slow moving tables; that is, tables that are not updated frequently or extensively.
UDF
(User Defined Functions)
No Yes (but not with network calls using UDF's)
Primary Key Yes but only as logical as it depends on the data loaded on S3. NO
Pricing Charges for the amount data scanned during Query Execution. Scanned data is rounded to the nearest 10MB. No charge for Failed Queries. Charges depends on the cluster(s) hosted.
Row Size No Limit 4 MB is the max size of any row in a data source
Max Number of Tables One Database can contain max of 100 tables.
Max number of Database is capped at 100.
Max number of tables per cluster is 9900 (including temporary tables).
Max Number of Schemas   9900 per cluster
SerDe (Serializer and Deserializer) Yes No.
External Table Only External Table is allowed when creating the table as the data is only referred from S3 as a table. Allowed in Redshift Spectrum.
A table can be either External (from S3) or local to Redshift.
Integration with BI Tools Using JDBC it can integrate with BI Tools or SQL Clients. Or with QuickSight for easy visualizations. Redshift can be integrated with Tableau, Informatica, Microstrategy, Pentaho, SAS and other BI Tools.
QuickSight to connect to Redshift instance, you must create a new security group for that instance.