Data Talks and Musings

Understanding True Couple Goals: Beyond Social Media Portrayals

2024-06-20T05:52:00.000-07:00

In today’s social media-driven world, hashtags like #CoupleGoals and #RelationshipGoals are constantly trending. While these portrayals can be inspiring, they often create unrealistic expectations about relationships. Popular posts under #CoupleGoals typically showcase couples on exotic vacations or exchanging expensive gifts. However, these posts only capture moments rather than the complete picture of a relationship.

I recently came across an Instagram post where a couple was celebrating their three-month anniversary on a private beach. The caption read, "Thanks for always being there for me, thanks for always understanding me bla bla..." Initially, I thought they had been dating for years and had finally gotten married three months ago. To my surprise, they had met on a matrimonial site just four months back. While it’s great that they have so much understanding in such a short time, what surprised me even more were the comments from their friends, calling them “inspiring” and labeling them as #CoupleGoals. I mean, seriously? Couple goals at the honeymoon stage?

It takes years to truly understand each other. While trips together can be enjoyable and part of a relationship, they aren’t what true couple goals are about. True couple goals mean standing up for each other and being there through thick and thin. They involve deep understanding, mutual motivation, planning a future together, and working as a team. There is so much more to a relationship than what is often portrayed on social media.

True couple goals are less about grand gestures and more about the day-to-day efforts that partners put into their relationship. These include mutual support, shared responsibilities, quality time, open and honest communication, respect for each other, trust, and patience.

It’s particularly important for Generation Z to understand this. Life is not a bed of roses and can get really tough sometimes. You may try to be the best partner and still have a failed relationship, but the journey is important. Trying and giving your 100 percent is what truly matters—everything else will follow.

True couple goals go beyond the superficial portrayals seen on social media. Remember, real relationships require effort and dedication. Building a meaningful connection with each other takes years, not just months. It’s about the small, everyday actions that contribute to a strong, lasting partnership. So, the next time you see a #CoupleGoals post, remember that the real goals are built on a foundation of mutual respect, understanding, and unwavering support.

And there you have it, folks—#AnotherGyanPost!

Image courtesy of Bing Chat Copilot

Parenthood: Embracing Joy Over Expectations

2024-06-09T00:03:00.000-07:00

I recently read a perspective on parenting from the co-founder of India's biggest stock brokerage and a well-known influencer. He said, 'I don't want to ruin 20 years of my life babysitting. What if they leave?' This was his reasoning for not wanting children. Thankfully, his parents didn't think this way, or we wouldn't have many of the significant contributions he's made today. This entire discussion reminded me of terms like DINK (Double Income No Kids) and DISK (Double Income Single Kid), which a friend of mine recently introduced to me.

Choosing to be DINK or DISK is a deeply personal decision. However, should the decision to have children really be driven by the expectation that they will take care of us in our old age? I personally believe this mindset is fundamentally flawed. A lot of previous generations thought this way, but it is really sad that someone from the new generation and an influential person is echoing this sentiment. Many young minds reading his post might start thinking along similar lines, which is concerning.

Parenting is undeniably challenging and a lifelong commitment. For the first 20 years, you're responsible for their upbringing and education, but your concerns for them will last a lifetime.

I believe we should have children because we genuinely want to experience parenthood and the joy it brings, not because we expect them to support us in our old age. The idea that "children are a safety net for old age" is something I strongly disagree with.

Children will eventually leave home; you can't expect them to stay with you forever. You just have probably 18 to 22 summers with them. All you can do is give them your best, spend quality time with them, and help them become good humans. This generation of kids is very different. They may choose to stay in touch with you in your old age, or they may not. All we can do is wish them well and hope for the best.

Parenting should be about nurturing and guiding another human being, and finding fulfillment in the growth and development of your child. It's about the love, connection, and personal growth that come from raising a child, not about ensuring you have someone to depend on later in life.

I’d love to hear your thoughts on this topic. Do you believe that having children should be a choice driven by joy and the desire for parenthood, rather than expectations of support in old age? Share your opinions and experiences in the comments below!

Image Courtesy : Bing Chat Copilot

The Reality of Remote Working Moms

2024-06-08T09:39:00.000-07:00

Working from home means you can focus on family instead of traffic and office drama.

Yes, working remotely allows for better planning and quality family time, but it doesn't mean uninterrupted hours with your children. Meetings, deadlines, and tasks, make it challenging to be there for every school pickup or bedtime story.

That's where a support system comes in handy. Grandparents, daycare, or a nanny help moms juggle work and family.

But even with support, the challenges persist. Kids have impeccable timing, demanding attention during meetings, craving snacks at the most inconvenient moments, or seeking a lap just as a crucial presentation begins.

For moms in technical roles, the demands are even more demanding. Juggling code and childcare requires a level of multitasking that truly showcases their strength.

So, the next time you see a mom working from home and assume she has it easy, remember the unseen battles she fights daily. Don't forget the mom guilt. And to all the women doing it alone, know that your dedication is truly admirable.

Image Courtesy : Bing Chat Copilot

Mainframe Modernization

2024-05-17T21:27:00.000-07:00

Are Mainframes still relevant ?

Mainframe is still as relevant as ever for many enterprises around the world. Here’s why. Mainframes continue to shine at traditional tasks Mainframes are still hard at work doing the jobs they have traditionally done. 67 of Fortune 100 enterprises continue to use mainframes for their most crucial business functions. They’re at the centre of banking, manufacturing, retail, energy, the public sector and many other industries. Over these years, mainfram ehas evovled to meet the changing market needs.

What is modernization?

Modernization is the act of updating organizational processes, systems, and tools to the most current versions or best practices. In the context of cloud computing, modernization is the process of transitioning an organization's applications, processes, and data management to a cloud-first approach. The goal is to improve organizational and technological performance, enhance the quality of customer and employee experiences, and accelerate time to market for new offerings and updates.

What is legacy modernization ?

Application modernization is the process of updating an organization's existing apps to a cloud-first model—sometimes called legacy modernization. If an organization uses on-premises apps, modernization may involve migrating those apps to a public, private, or hybrid cloud. Commonly modernized apps and data include: .NET apps ,Linux web apps , Java apps , SAP apps, SQL databases

What are the benefits of application modernization ?

What are the challenges ?

Modernization/Digital Transformation approaches :

Rehost. Sometimes called "lift-and-shift," this modernization strategy emphasizes speed because it requires practically no code changes. Organizations take apps from their previous environment and transition them as-is to their current environment.

Replatform. This approach lies in between rehosting and refactoring. With replatforming, organizations make code changes so that apps can be used with cloud technologies.

Refactor (or repackage). Refactoring prioritizes productivity as well as speed. With this strategy, apps need only minimal code changes so that they can connect easily to and make the most of a cloud-first environment.

Rearchitect. If an organization needs cloud scalability, rearchitecting might be the right approach. With rearchitecting, apps functionality and code get modified and extended to scale better in the cloud.

Rebuild (or rewrite). For recreating an app using cloud solutions, rebuilding is sometimes the right option. It's a heavier lift, but it may be essential if existing apps have limited functionality or lifespan.

Replace. If an app won't meet current or future business needs even after rebuilding, replacing it with a ready-made solution may be necessary. This approach can be faster than rebuilding and free up valuable development resources. But replacing apps could pose challenges including interruptions to business processes and limitations to future modernization initiatives.

Encapsulating : This involves accessing the mainframe resources by using API's

Encapsulating :

Let us understand how to expose mainframe applications and data through API with the help of above architecture diagram. Consider a cloud native application hosted on the cloud. The cloud be AWS, Azure, GCP or the IBM cloud. This particular application is invoking an API to access customer information which is turn invoking a mainframe application which could be a CICS application which is exposed as an API.

A cloud native application calls a managed API to get customer information. With the help of an API gateway, the cloud native application invokes an API endpoint. It secures the API and invokes the corresponding API via IBM Z/OS connect EE.

Now the API gateway and IBM Z/OS connect EE are the tools that modernize the legacy applications.

An API gateway is an API management tool that resides between client and collection of backend services. It acts as reverse proxy to accept all API calls, aggregate the various services required to fulfill them and return the result with appropriate response.

IBM Z/OS connect EE : This is used to provide RESTful API access to Z/OS applications and data hosted in subsystems such as CICS, IMS, IBM MQ etc. Respresentational state transfer REST APIs are an application programming interface that conforms to the constraints of REST architectural style and allows for interaction with Restful webservices. Ibm zos connect facilities calling zos target applications by using REST calls. It provides standard way to identify assets that are on Zos and reach these assets by using REST. Zos connect enterprise edition allows to serve API from common zos middleware like ibm cics, ibm ims and ibm DB2. Zos connect translates incoming REST calls to the format and codepage understood by the configured middleware . Zos connect gives the ability to easily consume api from applications written on zos where in many cases these applications were written before REST was conceived. Main purpose of using REST enablement for zos is the ability to communicate with Z/OS applications in a secure way.

Going back to the diagram above, an API on IBM Z/OS will transform the http request into the input format of a CICS application and this format specified in a COBOL copybook and after that it will invoke the CICS application. The CICS app would perform the required operation and return the result in the output format which is specified in a cobol copybook. After recieving the response, the output is converted into JSON format. The response is then returned to the API gateway which in turn returns this JSON based response to the calling cloud native app.

Understanding DevOps

2024-05-17T21:26:00.000-07:00

DevOps :

The contraction of "Dev" and "Ops" refers to replacing siloed Development and Operations. The idea is to create multidisciplinary teams that now work together with shared and efficient practices and tools. Essential DevOps practices include agile planning, continuous integration, continuous delivery, and monitoring of applications. DevOps is a constant journey.

By adopting DevOps practices :

You shorten your cycle time by working in smaller batches.
Using more automation.
Hardening your release pipeline.
Improving your telemetry.
Deploying more frequently.

Continuous Integration drives the ongoing merging and testing of code, leading to an early finding of defects. Other benefits include less time wasted fighting merge issues and rapid feedback for development teams.
Continuous Delivery of software solutions to production and testing environments helps organizations quickly fix bugs and respond to ever-changing business requirements.
Version Control, usually with a Git-based Repository, enables teams worldwide to communicate effectively during daily development activities. Also, integrate with software development tools for monitoring activities such as deployments.
Use Agile planning and lean project management techniques to:
- Plan and isolate work into sprints.
- Manage team capacity and help teams quickly adapt to changing business needs.
- A DevOps Definition of Done is working software collecting telemetry against the intended business goals.

Slowly Changing Dimensions

2023-01-16T06:07:00.000-08:00

A slowly changing dimension (SCD) is one that appropriately manages the change of dimension members over time. It applies when the values of a business entity change over time, and not on a set schedule.

a set of fields to track when the record was added and modified.

Types of SCD -

Type 0 – Fixed Dimension
- No changes allowed, dimension never changes
Type 1 – No History
- Update record directly, there is no record of historical values, only current state
Type 2 – Row Versioning
- Track changes as version records with current flag & active dates and other metadata
Type 3 – Previous Value column
- Track change to a specific attribute, add a column to show the previous value, which is updated as further changes occur
Type 4 – History Table
- Show current value in dimension table but track all changes in separate table
Type 6 – Hybrid SCD
- Utilise techniques from SCD Types 1, 2 and 3 to track change

including type 3 and type 6.

Confusingly, there is no SCD type 5 in commonly agreed definitions.

After you have implemented your chosen dimension type, you can then point your fact records at the relevant business or surrogate key. Surrogate keys in these examples relate to a specific historical version of the record, removing join complexity from later data structures.

A type 0 SCD the table remains the same. This means our existing reports will continue to show the same figures, maybe it is a business requirement that each customer is always allocated to the country they signed up from.

In order to support type 2 changes, we need to add four columns to our table:

· Surrogate Key – the original ID will no longer be sufficient to identify the specific record we require, we therefore need to create a new ID that the fact records can join to specifically.

· Current Flag – A quick method of returning only the current version of each record

· Start Date – The date from which the specific historical version is active

· End Date – The date to which the specific historical version record is active

A type 4 SCD , has no change to existing table here, we simply update the record as if a Type 1 change had occurred. However, we simultaneously maintain a history table to keep track of these changes:

Our Dimension table reads:

Whilst Type 4 historical table is created as:

Depending on requirements, place both ID and Surrogate Key onto the fact record so as to optimise performance whilst maintaining functionality.

Separating the historical data makesthe dimensions smaller and therefore reduces complexity and improves performance if the majority of uses only need the current value.

However, if you do require historical values, this structure adds complexity and data redundancy overheads. It is generally assumed that the system will use Type 1 or Type 2 rather than Type 4.

Understanding Azure Data Factory -2

2023-01-16T02:32:00.001-08:00

Azure custom roles

If the Azure built-in roles don't meet the specific needs of your organization, you can create your own custom roles.

Click this link to see an overview of custom roles.

Understanding Azure Data Factory

2023-01-16T02:19:00.006-08:00

A data factory might have one or more pipelines. A pipeline is a logical grouping of activities that performs a unit of work. Together, the activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingests data from an Azure blob, and then runs a Hive query on an HDInsight cluster to partition the data.

Activity

Activities represent a processing step in a pipeline. For example, you might use a copy activity to copy data from one data store to another data store. Similarly, you might use a Hive activity, which runs a Hive query on an Azure HDInsight cluster, to transform or analyze your data. Data Factory supports three types of activities: data movement activities, data transformation activities, and control activities.

Datasets

Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your activities as inputs or outputs.

Linked services

Linked services are much like connection strings, which define the connection information that's needed for Data Factory to connect to external resources. For example, an Azure Storage-linked service specifies a connection string to connect to the Azure Storage account.

Integration Runtime

n integration runtime provides the bridge between the activity and linked Services. It's referenced by the linked service or activity, and provides the compute environment where the activity either runs on or gets dispatched from. This way, the activity can be performed in the region closest possible to the target data store or compute service in the most performant way while meeting security and compliance needs.

Azure Integration runtime

a data source in a private network,

Azure - SSIS IR

Data Factory Triggers

Pipeline runs are typically instantiated by passing arguments to parameters that you define in the pipeline. You can execute a pipeline either manually or by using a trigger.

1. Pipelines and triggers have many to many relationships(except for tumbling window trigger). Multiple triggers can kick off a single pipeline or a single trigger can kick off multiple pipelines.

2. Different types of triggers -

Schedule Trigger : A trigger that invokes a pipeline on a wall-clock schedule. It supports periodic and advanced calendar options.
Event based trigger - A trigger that responds to an event.
Tumbling window trigger - A trigger that operates on a periodic interval, while also retaining state. Creates windows, Works well for historical dates Self Dependency Trigger - Using of offsets becomes compulsory

Data flows

These are one of the features inside the Azure Data Factory which allows data engineers to develop data transformation logic in a graphical approach without writing code. The resulting data flows can then be executed as activities within Azure Data Factory pipelines that use scaled-out Spark clusters. Data flows will run on your own execution cluster for scaled-out data processing. ADF internally handles all the code translation, spark optimization and execution of transformation. Data flow activities can be operationalized via the existing Data Factory scheduling, control, flow, and monitoring capabilities.

There are two types of Data flows:

Mapping Data Flow
Wrangling Data Flow

Pipeline runs

Variables

ADF security

Data can be copied from more than 80 connectors or in between 80 different applications in Azure. These applications may be Software as a Service kind of applications like Dynamic 365 or Salesforce. Or these applications may be on-premises data stores like SQL Server or Oracle. Or these applications may be cloud data stores like Azure SQL Database or maybe Amazon S3.

Thus it enables to copy data from application in any environment to different application in any other environment. While copying, it can convert file formats, like zip or unzip file , map the columns between the source and the destination.

In addition to copying data, transformation of the data is also possible. Previously the only way to do this was using the external services like HDInsight, or Databricks, or maybe SQL Server stored procedures. But very recently in 2019, Azure Data Factory added the Data Flow task. Now this Data Flow task is very much like a Data Flow in the SSIS and it has all possible kind of transformations.

Now both copy as well as transforming the data is possible right from the same interface in Data Factory. This makes Data Factory a complete ETL tool or Data ingestion tool.

Different ways of working with Azure Data Factory -

1. Azure Portal UI

2. Azure PowerShell

3. .NET

4. Python

5. REST

6. Resource Manager Template

Summary :

1. ADF allows to create workflows for transforming and orchestrating the data movement.

2. Data movement triggers can be timed or event driven.

3. The management of these strategic events is called as orchestration. The better to say is the management of these events, triggers, workflows, data pipelines dependencies is called as orchestration.

4. ADF is a cloud based software as a service (ETL) tool. Also called as code free ETL service.

5. Operations associated with ADF are : Ingest, Control, Data flow, Schedule and Monitor.

6. Parameterize Datasets in ADF : You can parameterize a dataset and pass dynamic values at run time.

7. System Variables in ADF : Available at three scopes (Pipeline scope, tumbling window trigger scope and schedule trigger scope)

8.Connectors in ADF :

Data can be copied from more than 80 connectors or in between 80 different applications in Azure.

9.Supported formats for datasets :

Avro, Binary (.txt) , Delimited text format (csv), JSON format, ORC format, Parquet format

ORC, Parquet and Avro format are all part of Apache Hadoop eco-system. They work on compression algorithm , data is stored in compression and hence query results are faster

Operational Needs - How will you use the Data ?

2023-01-15T06:21:00.001-08:00

Azure Stream Analytics

2023-01-15T01:34:00.004-08:00

Azure Cosmos DB

2023-01-14T22:05:00.006-08:00

Azure Cosmos DB is a globally distributed multi model database.

Deploy it using anyone of these -

Consider this example where Azure Cosmos DB helps resolve a business problem,

Data storage in Azure Storage

2023-01-14T21:38:00.016-08:00

Microsoft Azure storage is a managed service that provides durable, secure and scalable storage in the cloud.

Durable, this means that the redundancy ensures that your data is safe in the event of transient hardware failures and also replicate data across data centers or geographical regions for extra protection from local catastrophe or natural disaster. Data replicated in this way remains highly available in the event of an unexpected outage.

Security, all data written to Azure storage is encrypted by the service as your storage provides you with fine grained control over who has access to your data.

Scalability, Azure storage is designed to be massively scalable to meet the data storage and performance needs of today's applications.

Management, Microsoft Azure handles maintenance and any critical problems for you.

A single Azure subscription can host up to 200 storage accounts, each of which can hold 500 terabytes of data.

Azure storage offers four configuration options.

Azure blob, a scalable object store for text and binary data.
Azure files, managed file shares for cloud or on premises deployments(for traditional SMB file shares)
Azure queue, a messaging store for reliable messaging between application components(persistent messaging)
Azure table and NoSQL store for no schema storage of structured data (for structured storage of key/values)

Azure blob storage is an object storage solution optimized for storing massive amounts of unstructured data, such as text or binary data.

Ideal for serving images or documents directly to a browser, including full static websites, storing files for distributed access, streaming video and audio, storing data for backup and restoration. Disaster recovery and archiving and storing data for analysis by on premises or Azure hosted service.

Block blobs are used to hold text or binary files up to 5 terabytes, 50,000 blocks of 100 megabytes in size. The primary use case for block blobs is the storage of files that are read from beginning to end, such as media files or image files for websites. And they are named block blobs because files larger than 100 megabytes must be uploaded a small blocks. These blocks are then consolidated or committed into the final blob.

Page blobs are used to hold random access files up to 8 terabytes in size. Page Blobs are used primarily as the backing storage for the virtual hard disks or VHDs used to provide durable discs for Azure virtual machines or Azure VMs. They are named page blobs because they provide random read write access to 512-byte pages. These blobs are frequently used for logging information from one or more sources into the same blob, and to give an example, you might write all your trace logging to the same append blob for an application running on multiple VMs.

Append blobs are made up of blocks like block blobs, but they are optimized for append operations. A single append blob can be up to 195 gigabyte

Azure files enables you to set up highly available network file shares that can be accessed using the standard server message block or SMB protocol. This means that multiple VMs can share the same files with both read and write access. You can also read the files using the REST interface and the storage client libraries. Finally, you can also associate a unique URL to any file to allow fine grained access to a private file for a set period. File shares can be used for many common scenarios, storing shared configuration files for VMs, tools or utilities so that everyone is using the same version. Log files such as diagnostics, metrics and crash dumps and shared data between on premises applications and Azure VMs to allow migration of apps to the cloud over a period.

Azure Queue service is used to store and retrieve messages. Queue messages can be up to 64 kilobytes in size and a queue can contain millions of messages. Finally, queue are used to store lists of messages to be processed asynchronously. You can use queue to loosely connect different parts of your application together. For example, we could perform image processing on the photos uploaded by our users. Perhaps we want to provide some sort of face detection or tagging capability so people can search through all the images they have stored in our service. And we could use queues to pass messages to our image processing service to let it know that new images have been uploaded and are ready for processing. This sort of architecture would allow you to develop and update each part of the service independently.

Azure storage offers several types of storage accounts. Each type supports different features and has its own pricing model.

Standard General Purpose v2 is the standard storage account type for blobs, file shares, queues and tables, it is recommended from most scenarios using Azure storage. And the premium storage account type is for block blobs and a pen blobs, this is recommended for scenarios with high transaction rates, scenarios that use smaller objects or those that require consistently low storage latency.

Premium storage account type is for file shares only. It is recommended for enterprise or high performance scale applications. This is useful if you require both SMB and NFS, file share support. Finally, the premium storage account type is for page blobs only.

Azure Storage provides a REST API to work with the containers and data stored in each account. Each type of data you can store has its own independent API. Recall that we have four specific data types: Blobs, Queues, Tables, Files.

SAS

Reference and Credits : Microsoft.com

Microsoft Azure - A case study

2023-01-14T21:23:00.001-08:00

Contoso Health Network recently deployed IoT devices to its Intensive Care Unit, ICU. Here are the goals of the project -

This is the provisioning workflow. Provision Azure Data Lake Storage Gen 2,Provision Azure Synapse Analytics, Provision Azure IoT Hub, Provision Azure Stream Analytics, Provision Azure Machine Learning, Provision Azure Data Factory, and Provision Power BI.
The plan for the holistic workflow is to set up Azure IoT Hub to capture data from the ICU IoT devices. Connect Azure IoT Hub to Azure Stream Analytics. Set up window creation functions for the ICU data. The functions will aggregate the data for each window. At the same time, set up the IoT Hub to move the streaming data to Azure Data Lake Storage by using Azure Functions. Set up Azure Functions to store the Azure Stream Analytics aggregates in Azure Data Lake Storage Gen 2.Use Azure Data Factory to load data from the Data Lake into Azure Synapse Analytics to support the Chief Medical Officer's needs.
After the data is loaded, transformations can occur within Azure Synapse Analytics.
In parallel, connect the Azure Machine Learning Service to Azure Data Lake Storage to perform predictive analytics. Connect Power BI to stream analytics to pull the real-time aggregates for the patient data. Connect Azure Synapse Analytics to pull the historical data to create a combined dashboard. Contoso Health Network's project was to deploy IoT devices to its ICU.

Databricks Repos

2023-01-13T22:49:00.004-08:00

Databricks Repos is a visual Git client in Databricks. It supports common Git operations such a cloning a repository, committing and pushing, pulling, branch management, and visual comparison of diffs when committing.

Summary :

1. Clone the target repository

2. Create a new development branch or select an existing branch agreed upon by the team in which development work is conducted.

3. It is always good to create feature branch and not develop directly on the main branch.

4. Make changes(add items, modify code) and test.

5. Commit and push the changes to your branch

6. Finally merge the changes into main(in accordance with organization's code review and release process)

Databricks supports the following Git providers: GitHub, Bitbucket Cloud,GitLab, Azure DevOps, AWS CodeCommit, GitHub AE

For following tasks, work in your Git provider:

Create a pull request.
Resolve merge conflicts.
Merge or delete branches.
Rebase a branch.

On the Github page :

1. Configure Git Integration in our workspace and have a repo ready.

2. Connect databricks to Git using username and PAT which is the personal access token

3. Go to the settings page on the Github and click on the developer settings, click PAT, generate new token

4. Mention in the Note and Expiration period, select scope as Repo and create it by clicking Generate token.

5. Copy the alphanumeric text and save in a temporary secure folder.

On the Databricks workspace :

1. Go to User setting-- GIT Integration tab, provide git provider and credentials and save.

On the Github landing page:

1. create a repo

2. Copy the URL from Code

On the Databricks workspace :

1. Go to repos in the sidebar

2. Add repo -- specify the URL copied earlier and create

3. The repo is now cloned and new repo will now contain a local copy of the upstream repo, navigate the repo and start developing before committing the changes.

4. To create a branch called dev click on the main branch , create the branch by specifying the name

5. Now start making changes in the dev branch by creating folder or whatever is necessary and then add a notebook (create or import notebook is possible)

6. commit all changes at once or any number of changes, provide summary and then commit and push

Note that pulling regularly is important to avoid merge conflicts specially when multiple developers are working on the same kind of changes.

Reference and Credit - Databricks.com

Navigating the Databricks Workspace

2023-01-13T21:22:00.004-08:00

Navigate the workspace

A Databricks workspace is an environment for accessing all of your Databricks assets. The workspace organizes objects such as notebooks, libraries, experiments, queries, and dashboards into folders, and provides access to data and computational resources such as clusters and jobs.

Use the sidebar

Access all of your Databricks assets using the sidebar. The sidebar’s contents depend on the selected persona: Data Science & Engineering, Machine Learning, or SQL.

By default, the sidebar appears in a collapsed state and only the icons are visible. Move cursor over the sidebar to expand to the full view.
To change the persona, click the icon below the Databricks logo , and select a persona.
To pin a persona so that it appears the next time you log in, click next to the persona. Click it again to remove the pin.
Use Menu options at the bottom of the sidebar to set the sidebar mode to Auto (default behavior), Expand, or Collapse.
When you open a machine learning-related page, the persona automatically switches to Machine Learning.

Switch to a different workspace

If you have access to more than one workspace in the same account, you can quickly switch among them.

Click the workspace name in the top bar of the Databricks workspace.
Select a workspace from the drop down to switch to it.

What can you do in this workspace ?

Attach a notebook to a cluster - Deploying a cluster can take several minutes. A solid green circle will appear to the left of the cluster name once resources have been deployed. If your cluster has an empty gray circle to the left, you will need to follow instructions to start a cluster. (Refer chapter 1 )
Execute a cell in a notebook - Notebooks provide cell-by-cell execution of code. Multiple languages can be mixed in a notebook. Users can add plots, images, and markdown text to enhance their code.
Set the language for a notebook - The cell above executes a Python command, because our current default language for the notebook is set to Python. Databricks notebooks support Python, SQL, Scala, and R. A language can be selected when a notebook is created, but this can be changed at any time.
Describe and use magic commands - A single percent (%) symbol at the start of a cell identifies a magic command, these are specific to the Databricks notebooks. The magic command %md allows us to render Markdown in a cell. Language magic commands allow for the execution of code in languages other than the notebook's default eg %sql %python
Export a Databricks notebook - Steps:
- Click the File option to the right of the cluster selection at the top of the notebook
- From the menu that appears, hover over Export and then select Source File
Export a collection of Databricks notebooks - Steps:
- Click the Repos on the left sidebar
  - This should give you a preview of the parent directories for this notebook
- On the left side of the directory preview around the middle of the screen, there should be a left arrow. Click this to move up in your file hierarchy.
- You should see a directory called Data Engineering with Databricks. Click the the down arrow/chevron to bring up a menu
- From the menu, hover over Export and select DBC Archive

Reference and Credits - Databricks.com

Databricks Architecture

2023-01-13T21:14:00.012-08:00

Databricks Architecture Overview

High-level architecture

Databricks operates out of a control plane and a data plane as follows -

The control plane includes the backend services that Databricks manages in its own cloud account such as Azure, AWS and GCP. Notebook commands and many other workspace configurations are stored in the control plane and encrypted at rest.
The cloud account(AWS, Azure or GCP) that is in use manages the data plane, and this is where the data resides and data is also processed here.
Considering Azure, Azure Databricks connectors are used to connect clusters to external data sources outside of the Azure account to ingest data, or for storage. Ingesting data is also possible from external streaming data sources, such as events data, streaming data, IoT data, and more.
Your data is stored at rest in your Azure account or any cloud account in the data plane and in your own data sources, not the control plane, so you maintain control and ownership of your data.
Job results reside in storage in your account.
Interactive notebook results are stored in a combination of the control plane (partial results for presentation in the UI) and your Azure/AWS storage. If you want interactive notebook results stored only in your cloud account storage(AZure/AWS), ask your Databricks representative to enable interactive notebook results in the customer account for your workspace.

Clusters :

A cluster is basically a collection of virtual machines in a cluster that is usually a driver node,

which orchestrates the tasks performed by one or more worker nodes.

Clusters allow us to treat this group of computers as a single compute engine via the driver node.

A Databricks cluster is a set of computation resources and configurations on which you run data engineering, data science, and data analytics workloads, such as production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning.

You run these workloads as a set of commands in a notebook or as an automated job. Databricks makes a distinction between all-purpose clusters and job clusters.

All-purpose clusters vs Job Clusters -

To create the cluster, click on the compute icon on the sidebar which shows list of cluster like below -

Then click on Create cluster and select the option required -

Depending on the workspace being used, one may or may not have privileges to create the clusters. Check with the platform admin to see if there is already a cluster created which can be connected for use or you need to create a new cluster.

Cluster Configuration :

1. Single Node/Multi Node

Multi node cluster will have one driver node and one or more worker nodes.

When you run a spark job against a multi node cluster, the driver node will distribute the tasks to

run on the worker nodes in parallel and returns the result.

They give us the ability to horizontally scale the cluster depending on your workload.

We can basically keep adding worker nodes as we need. These are the default type of clusters used for spark jobs and suitable for large workloads.

Single node cluster will have only one node- the driver node, and there are no worker nodes.

Even though there are no worker nodes, single node clusters also supports park workloads.

When you run a spark job, the driver node acts as both the driver and the worker as there are no worker nodes. The single node clusters are not horizontally scalable, so they're not suitable for large ETL workloads. They're mainly targeted for lightweight machine learning and data analysis workloads which don't require any distributed compute capacity.

2. Access Mode

3. Databricks Run time

4. Auto Termination

Terminates the cluster after X minutes of inactivity
Default value for single node and standard clusters is 120 minutes
Users can specify a value between 10 and 10000 mins as the duration

5. Auto Scaling

User specifies the min and max work nodes
Auto scales between min and max based on the workloads
Not recommended for streaming workloads

6. Cluster VM Type/size

Memory optimized - Recommended for memory intensive applications

Compute optimized - Structured streaming applications where processing rate is above the input rate at peak times of the day.

Storage optimized - For use cases requiring high disk throughput and I/O

General purpose - Recommended for enterprise grade applications and analytics in-memory caching

GPU accelerated Instance types - Recommended for deep learning models that are data and compute intensive.

7. Cluster Policy

Users could accidentally create clusters which are oversized and too expensive to run. Cluster policies help us avoid these common issues. Administrators can create cluster policies with restrictions and assign them to users or groups.

cluster policies simplify the user interface, thus enabling standard users to create clusters
No need for administrators to be involved in every decision.
Achieves cost control by limiting the maximum size of the clusters.
Only available on premium tier.

Types

Job compute
Personal compute
Power user compute
Shared compute

Cluster Pool

Clusters usually take time to start up and alter scale.
In order to minimize that time, we can use pools. A cluster pool is basically a set of idle ready to use virtual machines that allow us to reduce the cluster start and auto scaling times.

Azure Databricks Cluster Pricing

Depends on workload (All purpose/Job/SQL/Photon)
Tier(Standard/premium)
VM Type (General Purpose/GPU/Optimised)
Purchase plan(Pay as you go /pre-purchased)

A Databricks Units (DBU) is a normalized unit of processing power on the Databricks Lakehouse Platform used for measurement and pricing purposes.

Chapter 2 - https://rashdesai.blogspot.com/2023/01/navigating-databricks-workspace-chapter.html

Reference and credits - Databricks.com

Computer Programming : The Basics

2022-12-30T04:30:00.006-08:00

Computer Program :

A sequence or set of instructions in a programming language for a computer to execute.
A computer program in its human-readable form is called source code. Source code needs another computer program to execute because computers can only execute their native machine instructions. Therefore, source code may be translated to machine instructions using the language's compiler. (Assembly language programs are translated using an assembler.) The resulting file is called an executable. Alternatively, source code may execute within the language's interpreter

Computer programming is the act of composing the selected programming language's elements in the order that will cause the desired effect. The effect could be different in every specific case – it's up to the programmer's imagination, knowledge and experience.
The composition has to be correct in many senses:

alphabetically – a program needs to be written in a recognizable script, such as Roman, Cyrillic, etc.
lexically – each programming language has its dictionary and needs to be mastered
syntactically – each language has its rules and they must be followed
semantically – the program has to make sense

Difference between High-level vs Low-level programming language

Machine code is a program written in machine language, which consists of sequences of digital binary numbers: 0s and 1s. A low-level programming language consisting of binary digits/bits that the computer reads and understands.

Difference between a compiler and Interpreter

Due to historical reasons, languages designed to be utilized in the interpretation manner are often called scripting languages, while the source programs encoded using them are called scripts.

Azure Fundamentals 1 : Cloud concepts and Azure Architecture, Services

2022-12-28T01:48:00.020-08:00

Cloud Computing : It is the delivery of computing services over the internet, which is otherwise known as the cloud by using pay-as-you-go pricing model. These services include servers, storage, databases, networking, software, analytics, and intelligence. Cloud computing offers faster innovation, flexible resources, and economies of scale.

Cloud Computing benefits, use GASHED:

Geo-distribution
Agility - ability to react quickly , i.e ability to allocate or deallocate (scale) resources quickly.
Scalability

Scale vertically to increase compute capacity by adding RAM or CPUs to a virtual machine(scaling up or scaling down)
Scaling horizontally increases compute capacity by adding instances of resources, such as adding VMs to the configuration (scaling in (decrease ) / scaling out (increase))
Thus, scaling is the process of allocating(adding) or deallocating (removing) resources.

High Availability - Uptime/(Uptime + Downtime) , ability to keep services running for extended periods of time with very less downtime
Elasticity - Automatic scaling, i.e to adjust its processing power to match the demand or the ability to scale dynamically
Disaster Recovery - Ability to recover from an event that has taken down the service (disaster)
Fault Tolerance - Ability of the system to remain up and running during component and service failures.

In other words,

Lower your operating costs.
Run your infrastructure more efficiently.
Scale as your business needs change.

Why cloud?

The cloud provides on-demand access to:

A nearly limitless pool of raw compute, storage, and networking components.
Speech recognition and other cognitive services that help make your application stand out from the crowd.
Analytics services that deliver telemetry data from your software and devices.

Azure Portal : Web-based unified console that allows you to build, manage, and monitor everything in Azure.

Azure Marketplace : Connects users to partners and vendors that offer solutions/services for Azure

Accounts : Azure Account -> Many subscriptions -> Many Resource groups -> Resources

Azure Services Big Picture :

Azure Services :

Types of Cloud :

Expenses :

Consumption Based model :

No up-front costs
Pay for what you use or require
Pay for additional resources only when they are in use
Stop paying for resources not used

Cloud Service models :

IaaS - Infrastructure-as-a-Service
PaaS - Platform-as-a-Service
SaaS - Software-as-a-Service

Serverless computing

Like PaaS, serverless computing enables developers to build applications faster by eliminating the need for them to manage infrastructure. With serverless applications, the cloud service provider automatically provisions, scales, and manages the infrastructure required to run the code. Serverless architectures are highly scalable and event-driven, only using resources when a specific function or trigger occurs.

Subscriptions, Management Groups, and Resources :

Azure Region : A geographical area with one or more datacenters nearby and networked together.

Special Regions: US DoD, US Gov: physically+logically isolated with additional compliance certifications. China is operated by 21Vianet.

Azure Availability Zones : One or more physically separate datacenters within an Azure region. A.K.A. Isolation Boundary (HA/redundancy).

Availability Zones are interconnected with ultra high-speed, private, fiber network.

Not all regions have AZs.

Services that support AZs have these categories:

Zonal service: Pins to a zone
Zone-redundant: Auto-replication across zones
Non-regional: HA in an Azure geography.

Azure Region Pairs

Each Azure region is always paired with another region within the same geography.

AZs have one or more datacenters, and a Region has at least 3 zones.

Helps protect against natural disasters or civil unrest. Separated at least 300 miles.

Replication resides always within the same Geography as the pair except for Brazil South.

Azure resources and Azure resource Manager

Resource: A manageable item within Azure. Like a database or a VM Resource group: A grouping of resources you want to manage as a group.

Azure Resource Groups

Can contain anything you create in Azure to form a logical grouping of services (resources). Helps provide organization.

Life cycle: If you delete a resource group, all contained resources are deleted as well. Makes it easier to get rid of.
Authorization: A resource group is a scope for applying RBAC

Azure Resource Manager

Deployment and management service for Azure. CRUD for Azure resources

Manage infrastructure with templates
Deploy, manage, and monitor
Define dependencies between resources for correct ordering
Apply RBAC and tags

The following image shows the role Resource Manager plays in handling Azure requests.

Azure subscriptions

Provides authenticated and authorized access to products and services. Always linked back to an account.

An account can have one or many subscriptions.

Types of subscription boundaries:

Billing boundary: Determines how an Azure account is billed. You can create multiple subscriptions for different billing requirements.
Access Control boundary: Access-management policies happen at the subscription level. You can control access+resources for specific subscriptions.

Additional subscription helps with:

Environments: Separate environments via subscriptions. E.g. development and testing
Org structure: Marketing and IT, helping manage access and limit resources
Billing: Make it easier to track billing better.

The following diagram shows an overview of how billing is structured

Azure management groups

Provides a level of scope above subscriptions. Helps organize subscriptions into groups.

Helps provide user access to multiple subscriptions with a single RBAC that gets inherited.

Source : https://learn.microsoft.com/en-us/certifications/exams/az-900

Scheduling jobs using Cron

2022-12-09T00:14:00.000-08:00

Cron

Cron is the general name of the tool that runs scheduled jobs consisting of shell commands
or shell scripts.
Crond is the service that interprets “crontab files” every minute and submits
the corresponding jobs to cron at scheduled times.
A crontab, short for “cron table,” is a file containing jobs and schedule data.
Crontab is also a command that invokes a text editor to allow you to edit a crontab
file.

Job scheduling using Crontab

The five time-and-date fields cannot contain spaces. The five time-and-date fields and their allowed values are as follows:

minute (0-59)

hour(0-23, 0 = midnight)

day (1-31)

month (1-12)

weekday (0-6, 0 = Sunday)

Bash Scripting Advanced

2022-12-08T22:49:00.000-08:00

Filters :

These are shell commands, which :

Take input from standard input
Send output to standard output
Transform input data to output data
Examples are wc, cat, more, head, sort
Filters can be chained together

Pipes :

Pipe command - |

For chaining filter commands : commands1 | commands2
Here the Output of commands1 is the input of commands2

Summary :

Shell variables are variables that are limited in scope to the shell in which they were created. Accordingly, shells cannot “see” each other’s shell variables.
Filters are shell commands or programs such as wc, cat, sort , etc. that transform the input provided to them.
The pipe operator can be used to create a pipeline of chained filter commands, whereby output of one command becomes the input of another command.
Shell variables can be assigned values with a simple equal sign, and listed using the ‘set’ command.
Environment variables are shell variables with scope extended to all child processes of the shell.  They are created using the ‘export’ command and listed using the ‘env’ command.

Metacharacters :

Metacharacters are special characters that have meaning to the shell.

Quoting :

Quoting is used to specify whether the shell should interpret special characters as metacharacters,
or ‘escape’ them

Refers to a set of features used for redirecting either the standard input, which is the keyboard, or the standard output, which is the terminal.

Store the output of the command hostname -i in a variable named $myip


myip=$(hostname -i)
echo $myip

Print the following message on screen:

"Running on host : host_name" ,

Where 'host_name' should be your current hostname.


echo "Running on host: $(hostname)"

Command substitution can be done using the backquote syntax also.


ls -l `which cat`

The output of command which cat is the path to the command cat. This path is sent to ls -l as an input.

Batch vs Concurrent mode:

Shell Scripting basics

2022-12-08T22:06:00.000-08:00

Shell Scripting basics

A script is a list of commands that can be interpreted and run by a program called scripting language.
Commands can be entered interactively at the command line or listed line by line in a text file.
Scripting languages are usually not compiled. They are interpreted at runtime.
Scripts are generally slower to run than compiled languages, but they are also much easier and faster to develop.
Scripts are widely used to automate processes, such as ETL jobs, file backups and archiving, and general system administration tasks.
Scripts can be used for nearly any computational task including, application integration, and plug-in and web application development.
A shell script is an executable text file in which the first line usually has the form of an interpreter directive.
The interpreter directive is also known as a ‘shebang’ directive, and has the following form: 'pound, bang, interpreter' plus an optional argument.
Interpreter is an absolute path to an executable program, and the optional argument is a string representing a single argument.

Shell scripts are scripts that invoke a shell program.
For example:

‘#!/bin/sh’ invokes the Bourne shell or other compatible shell program, from the bin directory.
‘#!/bin/bash’ ‘shebang’ invokes the Bash shell.
‘Shebang’ directives aren't limited to shell programs.

The ‘.sh’ extension is a convention used to indicate that the file is a shell script.
Change text file into a bash script by echoing the bash shebang, and appending that echoed text to file using the ‘double greater than symbol,’ which is the Bash ‘output redirection’ operator used for appending output to file.
Then use the echo command to print the statement ‘echo Hello World’ and again, redirect that output to your bash script.

To make it executable, first, check the current permission settings of your script by using the ‘ls’ command with the ‘-l’ option.
The R and W entries indicate that the script is readable and writable, but the lack of an X means it is not executable.
The R, W, and X entries apply to three user-based permission groups: the owner, which is you, the group, and all users.
Make it executable for all users by invoking the ‘chmod’ command with the 'plus X' option.
Checking the permission settings and show that the script is executable for all users, due to the ‘X’ entries for all three per mission groups.

‘./hello_world.sh’ and pressing enter, and the text ‘hello world’ appears on the command line.

Linux Commands and Shell Scripting

2022-12-08T21:02:00.000-08:00

Shell Script :

A shell script is a computer program designed to be run by the Unix/Linux shell which could be one of the following:

The Bourne Shell

The C Shell

The Korn Shell

The GNU Bourne-Again Shell

Shell :

A shell is a command-line interpreter / user interface for running commands.

A shell, which enables access to files, utilities, and applications, is also an interactive language.

A shell is also a scripting language. And it can be used to automate tasks and typical operations performed by shell scripts include file manipulation, program execution, and printing text.

The default shell on linux systems is the bash(‘bourne again shell').

To find out what the default is, enter ‘printenv SHELL’ on the command line.

This returns the path to the default shell program, which in this case is bash.

If your default shell is not Bash, you can always switch to it,   simply by entering

'bash' on the command line.

Shell command applications :

Getting Information
Navigating and working with files and directories
Printing file and string contents
Compression and archiving
Performing network operations
Monitoring performance and status
Running batch jobs

Linux and Bash Command Cheat Sheet: The Basics

Getting information

# return your user name
whoami

# return your user and group id
id

# return operating system name, username, and other info
uname -a

# display reference manual for a command
man top

# get help on a command
curl --help

# return the current date and time
date

The following command displays current date in mm/dd/yy format.

date "+%D"

Here are some of the popular format specifiers that you can try out.

Specifier	Explaination
%d	Display the day of the month (01 to 31)
%h	Displays abbreviated month name (Jan to Dec)
%m	Displays the month of year (01 to 12)
%Y	Display four-digit year
%T	Display the time in 24 hour format as HH:MM:SS
%H	Display the hour

Monitoring performance and status

# list selection of or all running processes and their PIDs
ps
ps -e

# display resource usage
top

When top started for the first time, you'll be presented with the following elements on the main top screen.

Summary Area. - shows information like system uptime, number of users, load average, memory usage
Fields/Columns Header.
Task Area.

The output keeps refreshing until you press 'q' or Ctrl+c

Using 'top' we can find out which process is consuming the most resources, can press the following keys while 'top' is running to sort the list :

M - sort by memory usage
P - sort by CPU usage
N - sort by process ID
T - sort by the running time

# list mounted file systems and usage
df

Working with files

# copy a file
cp file.txt new_path/new_name.txt

# change file name or path
mv this_file.txt that_path/that_file.txt

# remove a file verbosely
rm this_old_file.txt -v

# create an empty file, or update existing file's timestamp
touch a_new_file.txt

# change/modify file permissions to 'execute' for all users
chmod +x my_script.sh

Option	Description.
r, w and x	representing read, write and execute permissions respectively
u,g and o	representing user categories owner, group and others respectively
+, -	representing grant and revoke operations respectively

# get count of lines, words, or characters in file
wc -l table_of_data.csv
wc -w my_essay.txt
wc -m some_document.txt

# return lines matching a pattern from files matching a filename pattern - case insensitive and whole words only
grep -iw hello \*.txt

# return file names with lines matching the pattern 'hello' from files matching a filename pattern
grep -l hello \*.txt

Navigating and working with directories

# list files and directories by date, newest last
ls -lrt

Here are some popular options to try with the ls command.

Option	Description
-a	list all the files including hidden files
-d	list directories themselves, not their contents
-h	with -l and -s, print sizes like 1K, 234M, 2G etc
-l	long listing of files which include information about permission, owner, size etc
-F	classify files by appending type indcator like *,/ etc. to file names
-r	reverse order while sorting
-S	sort by file size, largest first
-t	sort by time, newest first

# find files in directory tree with suffix 'sh'
find -name '\*.sh'

# return present working directory
pwd

# make a new directory
mkdir new_folder

# change the current directory: up one level, home, or some other path
cd ../
cd ~ or cd
cd another_directory

# remove directory, verbosely
rmdir temp_directory -v

Printing file and string contents

# print file contents
cat my_shell_script.sh

# print file contents page-by-page
more ReadMe.txt

# print first N lines of file
head -10 data_table.csv

# print last N lines of file
tail -10 data_table.csv

# print string or variable value
echo "I am not a robot"
echo "I am $USERNAME"

Compression and archiving

# archive a set of files
tar -cvf my_archive.tar.gz file1 file2 file3

# compress a set of files
zip my_zipped_files.zip file1 file2
zip my_zipped_folders.zip directory1 directory2

# extract files from a compressed zip archive
unzip my_zipped_file.zip
unzip my_zipped_file.zip -d extract_to_this_direcory

Performing network operations

# print hostname
hostname

# send packets to URL and print response
ping www.google.com

# display or configure system network interfaces
ifconfig
ip

# display contents of file at a URL
curl <url>

# download file from a URL
wget <url>

Bash shebang

#!/bin/bash

Pipes and Filters

# chain filter commands using the pipe operator
ls | sort -r

# pipe the output of manual page for ls to head to display the first 20 lines
man ls | head -20

Shell and Environment Variables

# list all shell variables
set

# define a shell variable called my_planet and assign value Earth to it
my_planet=Earth

# display shell variable
echo $my_planet

# list all environment variables
env

# environment vars: define/extend variable scope to child processes
export my_planet
export my_galaxy='Milky Way'

Metacharacters

# comments
# The shell will not respond to this message

# command separator
echo 'here are some files and folders'; ls

# file name expansion wildcard
ls *.json

# single character wildcard
ls file_2021-06-??.json

Quoting

# single quotes - interpret literally
echo 'My home directory can be accessed by entering: echo $HOME'

# double quotes - interpret literally, but evaluate metacharacters
echo "My home directory is $HOME"

# backslash - escape metacharacter interpretation
echo "This dollar sign should render: \$"

Special Character	Meaning
\n	Represents a newline character
\t	A tab character

I/O Redirection

# redirect output to file
echo 'Write this text to file x' > x

# append output to file
echo 'Add this line to file x' >> x

# redirect standard error to file
bad_command_1 2> error.log

# append standard error to file
bad_command_2 2>> error.log

# redirect file contents to standard input
$ tr “[a-z]” “[A-Z]” < a_text_file.txt

# the input redirection above is equivalent to
$cat a_text_file.txt | tr “[a-z]” “[A-Z]”

Command Substitution

# capture output of a command and echo its value
THE_PRESENT=$(date)
echo "There is no time like $THE_PRESENT"

Command line arguments

./My_Bash_Script.sh arg1 arg2 arg3

Batch vs. concurrent modes

# run commands sequentially
start=$(date); ./MyBigScript.sh ; end=$(date)

# run commands in parallel
./ETL_chunk_one_on_these_nodes.sh & ./ETL_chunk_two_on_those_nodes.sh

Scheduling jobs with Cron

# open crontab editor
crontab -e

# job scheduling syntax
m h dom mon dow command
minute, hour, day of month, month, day of week
* means any

# append the date/time to file every Sunday at 6:15 pm
15 18 * * 0 date >> sundays.txt

# run a shell script on the first minute of the first day of each month
1 0 1 * * ./My_Shell_Script.sh

# back up your home directory every Monday at 3 am
0 3 * * 1 tar -cvf my_backup_path\my_archive.tar.gz $HOME\

# deploy your cron job
Close the crontab editor and save the file

# list all cron jobs
crontab -l

MongoDB

2022-12-08T03:33:00.000-08:00

MongoDB

MongoDB is a document and NoSQL database
Documents are associative arrays like JSON objects or python dictionaries
Collections are group of stored documents
Support different data types
Model data as you read/write
Allows structured and unstructured data
High availability
Documents provide flexible way to store data
Flexible schema like seen below -

In relational databases it is important to design and code, however in MongoDB, no complex table definition is required and we can write code as soon as we connect to DB
Querying is done using MQL
For complex analysis use aggregate pipelines
High availabilty - resilience through redundancy, no system maintenance downtime

Typical MongoDB setups are three-node replica sets, where one of them is a primary member and the others are secondary members.
Replication keeps a copy of your data on other data bearing nodes in the cluster.
If one system fails, the other one takes over and you do not see any downtime.
This also works for system maintenance situations where you might take away a node for making updates to software, operating systems, file systems, security patches, or your MongoDB version.

Use cases :

MongoDB can be used with IoT devices. These devices generate vast amounts of data. A weather station will send a temperature and wind speed reading every minute. With scaling capabilities, MongoDB can easily store all of this data distributed globally.

MongoDB is a good fit when you want real-time analytics.
Most organizations want to make better decisions based on their data.
Doing historical analysis is easy, but only a few can respond to changes happening minute by minute. And most of the time, this is due to complex Extract, Transform, and Load (or ETL) processes.With MongoDB, you can do most of the analysis where the data is stored. That data can be semi-structured or completely unstructured. All of this can be done in real time.

MongoDB has a part to play in the gaming world, too. With more than ever multiplayer games being played globally, getting the data across is hugely important. With native scalability, also known as Sharding, MongoDB makes it easier to reach users around the world.

MongoDB can be used with e-commerce solutions.

          Products sold on an e-commerce website have different attributes.
          For example,A book will have attributes such as publisher, writer, and number of pages.
          Products also have attributes such as top reviews, pricing, inventory, and other metadata.
          With the help of documents, sub-documents, and list properties in MongoDB, storing the information together becomes easy so it is optimized for reads.

MongoDB has usage scenarios in the finance industry too.
And again, with the flexible schema it is easier to support ever-changing data needs.
With MongoDB, perform thousands of operations on the database per second.
And all the information is encrypted while in transfer and when stored on disk.
Additionally encrypt individual fields to avoid any data incidents.
And finally, like every industry, financial institutions have much higher requirements for
reliability and services being available around the clock. We expect our finances
to be available without any downtime. This makes MongoDB a good choice for banks,
trading companies, and the financial industry in general.

Azure Data Redundancy

2022-12-05T04:23:00.000-08:00

Data Redundancy ensures multiple copies of data is available for restoring in case of any hardware failures.

Redundancy gives us the higher availability and durability for your data. So the trade off is between the cost and the availability.

Every region has multiple Availability Zones, and every Availability Zones have multiple Data Center. And every Data Center has different racks in it.

Redundancy in the primary region

Locally-redundant storage LRS, that means making a copy within the same Data Center.
Zone - redundant storage ZRS, copies in three availability zones (AZ)

Redundancy in secondary region

Geo-redundant storage GRS - LRS +Asynchronous copy to secondary region
Geo-zone redundant storage

For read access to the secondary region, configure storage account -

Read access Geo-redundant storage (RA- GRS)
Read access Geo-zone redundant storage (RA- GZRS)

NoSQL Databases

2022-12-05T04:05:00.000-08:00

NoSQL, which stands for Not Only SQL or No Structured Query Language is a database system which is employed for managing the massive collection of unstructured data. NoSQL solved two problems of scalability and flexibility. It is important to note that NoSQL and non relational data are not the same thing. Historically, large databases have run on expensive machines or mainframes. Modern enterprises are employing cloud architectures to support their applications and the distributed nature of these NoSQL databases helps in deployed and on operating on clusters of servers in cloud which reduces the cost.

Can be used for both relational and non relational data
Provides new ways of storing and querying the data
Designed to handle 'Big Data'
Types : Key-Value, Document, Graph and Column

History of how it all started :

Source : IBM , skills network

Many customers of Azure Cosmos DB use this NoSQL database to store relational data. Cosmos DB SQL API also support SQL like Queries, and as far as scaling is concerned, scaling is much cheaper for NoSQL database because in NoSQL we can scale horizontally when relational database had to scale vertically.

Note : Vertically scaling means that you are increasing the load on a single server by adding more and more RAM and CPU. Horizontally scalable involves adding more server to process request. Vertical scalable has a limit. You can't just keep adding more and more RAM and CPU in a single machine. But scaling is much cheaper for NoSQL database because you don't need to scale up. You can have many cheap machines and scale your database horizontally.

NoSQL Use cases :

1. Big data and real time web applications

2. Relationship between data is not important

3. Data changes frequently

NoSQL Limitations:

1. Schema-less data means data is inconsistent

2. Denormalized data means redundant data

3. Redundant data means inaccuracies and conflicts

4.Does not support many good features of Relational DB.

SQL vs NoSQL :

Types of NoSQL Databases :

Images : Google images

Document Store :

A document store database (also known as a document-oriented database, aggregate database, or simply document store or document database) is a database that uses a document-oriented model to store data.
Document store databases store each record and its associated data within a single document. Each document contains semi-structured data that can be queried against using various query and analytics tools of the DBMS.

Key characteristics:

Each document offers flexible schema
Each piece of data is considered as document(XML/JSON)
Values are visible and can be queried
Content of the document databases can be indexed and queried - keys and value range lookups and search, Analytical queries with MapReduce
Horizontal scalable
Allow sharding across multiple nodes

Suitable Use cases:

Event logging for apps and processes - each event instance is represented by a new document
Online blogs - Each user, post, comment, like is represented by a document

Unsuitable Use cases:

When ACID transactions are required
If data naturally falls into normalized tabular model

Here are examples of some of the leading document store DBMSs.

Web Applications - Content management systems, Blogging platforms, eCommerce applications, Web analytics, User preferences data

User Generated Content - Chat sessions, Tweets, Blog posts, Ratings , Comments
Catalog Data - User accounts, Product catalogs, Device registries for Internet of Things , Bill of materials systems

Gaming - In-game stats, Social media integration, High-score leaderboards , In-game chat messages

Networking/computing - Sensor data from mobile devices, Log files,Realtime analytics, Various other data from Internet of Things

Key Value store :

A key-value store, or key-value database is a simple database that uses an associative array (think of a map or dictionary) as the fundamental data model where each key is associated with one and only one value in a collection. This relationship is referred to as a key-value pair. In each key-value pair the key is represented by an arbitrary string such as a filename, URI or hash. The value can be any kind of data like an image, user preference file or document. The value is stored as a blob requiring no upfront data modeling or schema definition. In general, key-value stores have no query language. They provide a way to store, retrieve and update data using simple get, put and delete commands; the path to retrieve data is a direct request to the object in memory or on disk. The simplicity of this model makes a key-value store fast, easy to use, scalable, portable and flexible.

Key characteristics :

Least complex, data is stored with a key and corresponding value blob
Represented as hashmap
Ideal for basic CRUD operations
Scale well
Shard easily
Not meant for complex queries

Some popular key-value stores are:

Amazon DynamoDB
Aerospike
Apache Cassandra
Cosmos DB Table API
Oracle NoSQL Database
Voldemorte
Berkeley DB
Couchbase Server
Redis
Riak

Image and further reading - hazelcast.com

Suitable Use cases:

1. For quick basic CRUD operations on non-interconnected data
Storing and retrieving session information for web applications
2. Shopping cart data for online stores
3. Storing in-app user profiles and preferences

Unsuitable Use cases:

1. For data that is interconnected with many-to-many relationships such as social networks and recommendation engines , go for graph in these cases

2. High level consistency is required for multi-operation transactions with mutiple keys

3. When apps run queries based on value vs key, consider document catergory of NoSQL.

Document Store vs Key-Value Databases

Document databases are similar to key-value databases in that, there’s a key and a value. Data is stored as a value. Its associated key is the unique identifier for that value.

The difference is that, in a document database, the value contains structured or semi-structured data. This structured/semi-structured value is referred to as a document.

The structured/semi-structured data that makes up the document can be encoded using one of any number of methods, including XML, JSON, YAML, BSON, etc. It could also be encoded using binary, such as PDFs, MS Office documents, etc.

Column Store

A column store database is a type of database that stores data using a column oriented model.

Extremly quick to load and query.

Also refered as -

Column database
Column family database
Column oriented database
Wide column store database
Wide column store
Columnar database
Columnar store

A column family consists of multiple rows.
Each row can contain a different number of columns to the other rows. And the columns don’t have to match the columns in the other rows (i.e. they can have different column names, data types, etc).
Each column is contained to its row. It doesn’t span all rows like in a relational database. Each column contains a name/value pair, along with a timestamp. Note that this example uses Unix/Epoch time for the timestamp.

Here’s a breakdown of each element in the row:

Row Key. Each row has a unique key, which is a unique identifier for that row.
Column. Each column contains a name, a value, and timestamp.
Name. This is the name of the name/value pair.
Value. This is the value of the name/value pair.
Timestamp. This provides the date and time that the data was inserted. This can be used to determine the most recent version of data.

Benefits - Compression, Aggregation queries, scalabilty,fast to load and query

Suitable use cases :

Can be used for event logging and blogs, counters, data with expiration values

Examples of Column Store DBMSs

Source : https://database.guide/what-is-a-column-store-database/

Graph Store

A graph database stores nodes(entities) and egdes (relationships) instead of tables, or documents. Data is stored just like you might sketch ideas on a whiteboard. Data is stored without restricting it to a pre-defined model, allowing a very flexible way of thinking about and using it.

They are ACID compliant but dont shard well

Suitable use cases

Highly connected and related data like social networking sites

Routing, spatial and maps

Recommendation engines

UnSuitable use cases

When application needs to scale horizontally

Graph Database use cases - Credit card fraud, social media analysis, money laundering

Examples - CosmosDB Gremlin API, Neo4j, Blazegraph, OrientDB, JanusGraph, DGraph

Summary :

NoSQL means Not only SQL.
NoSQL databases have their roots in the open source community.
NoSQL database implementations are technically different from each other.
There are several benefits of adopting NoSQL databases including storing and retrieving session information, and event logging for apps.
The four main categories of NoSQL database are Key-Value, Document, Wide Column, and Graph.
Key-Value NoSQL databases are the least complex architecturally.
Document-based NoSQL databases use documents to make values visible for queries.
In document-based NoSQL databases, each piece of data is considered a document, which is typically stored in either JSON or XML format.
Column-based databases spawned from the architecture of Google’s Bigtable storage system.
The primary use cases for column-based NoSQL databases are event logging and blogs, counters, and data with expiration values.
Graph databases store information in entities (or nodes) and relationships (or edges).