Technology

Whether to build software or to buy?

2022-07-27T20:05:00.009-07:00

Many organizations struggle with the question of when to build a software in-house and when to buy off the shelf commercial product. There is no easy answer to it.

A number of criteria's come into play and need careful evaluation of each of them. From in house software product management and development capabilities to long term sustenance to the perceived complexity of the Software.

I have compiled various levers in the video linked below.

Intuition and System Design

2020-04-15T02:00:00.002-07:00

What is intuition? We often hear that "This is very intuitive." Intuition in simple terms is the ability to comprehend in a natural way. Intuition is also a function of one's background and experience. Intuitive for one may not be the same for others. For example, a painter knows how with a couple of strokes and adjustments of colours he or she can change the mood of the painting. But that capability has been built through years of practice. That is why I consider intuition as a relative term and not an absolute one. Intuitive for one may not be so intuitive for others. That also leads to another interesting thing that people interpret things based on their experience and background. Confirmation bias has a role to play here but it also is a function of the tendency to put lesser mental load while comprehending a thing. We are driven by our intuition and our intuition is a function of our experience and background.

Back of the envelop calculation

2020-04-11T09:40:00.002-07:00

Most of you would have heard about "Back of the envelope calculation", which stands for what is a rough estimate of a certain thing. It can be the time a certain task will take or how many people or other resources will be needed to accomplish something. We do many kinds of estimation in our daily life both in professional and personal life. For example, how many years I should work and at what level so that I can get my retirement. Unfortunately, that never happens. If you really want to retire it's today or you will be made to retire for some other reason.

Git Branching

2019-11-30T23:45:00.001-08:00

Red dots denote commits. The model can be tuned based on the size of the team. For example, a single developer might just use one dev branch from the master and work on it and keep merging back in the master. However, it's suggested to still cut release branches.

Larger teams would want to use a more elaborate mechanism of tagging and cutting release branches. Versioning of the build can be done as per the blog.

The basic theme that should always be followed is to integrate the code as much as possible to the parent branches and maintain the stability of every code base across all branches. Pull frequently, push early and run integrations as much as needed. Have integration environments where any commit can trigger integration tests.

Migrating from DynamoDB to Postgres

2019-02-17T09:01:00.002-08:00

The mass popularity of NoSQL databases has also resulted in it being used in all use cases without understanding the suitability for the use case. One fundamental rule that is usually forgotten is that the NoSQL databases are designed around queries. Initially, the schema is evolved based on the need for initial business use cases. However, the business use cases evolve and change in multiple ways and soon the new needs of interacting with the database become unwieldy. This is the fundamental problem that people usually hit wiht NoSQL databases.

Where DynamoDB gets into trouble:

As the business use cases evolve and change the need to query the database in multiple ways arise. DynamoDB is not easy to query if it is not queried based on the partition key. One can build indexes but there is a cost associated with it. Filters can be used to query the data but that involves scan and as the data grows it starts becoming costly both in terms of money and time.
DynamoDB is schemaless so with time the data evolve in multiple ways. In older data records, it is quite possible that the fields might be missing or might have a different interpretation. Developers keep handling them in the deep layers of code to keep the world moving. However, soon it results in too many if-else statements. Migration is a pain to handle such cases, however, one has to be ready for missing fields and handle them with suitable defaults.
There is no relationship integrity so it's easy to put wrong data in relationships and it's very difficult to figure out even if something like that has happened. In SQL also it's possible to put a wrong key with a valid foreign relationship but still in terms of integrity SQL provides much better primitives.
This will be a repetition of the above points but a different perspective. As it's fine to add any kind of data in the table, people start putting all kind of data in it. Imagine that everyone in the world is given all kind of freedom. Sounds romantic. However, soon ti will be chaos as everyone is living in all different ways.

More discussion is available at https://tech.lalitbhatt.net/2018/09/sql-vs-nosql.html

Sample code for migration

import boto3

import psycopg2

# Create a connection to DynamoDB. Please fill the required keys. A better way to do is

# to put it in config file and pass it through. In AWS environment it's better to use Roles

dynamo = boto3.client('dynamodb',aws_access_key_id='<access-key>', \

aws_secret_access_key='<access-secrte>', \

region_name='<region>')

# Create the database connection

db = psycopg2.connect(host="<db_host>,database="<db>", user="<user>", password="<pass>")

dbCurr = db.cursor()

#Assume we have a user table in dynamoDB and insert it into the postgres Users table

user = dynamo.query(TableName="User", KeyConditionExpression ="Email = :email", \

ExpressionAttributeValues = {":email": { 'S': email }})

email = user.Items[0]['Email']

#Note the returning id so that we can use the id of the newly persisted record.

#This can be used to create foreign key relationships for further table

userSql = 'INSERT INTO users(email) VALUES(%s) returning id'

userValues=(email)

dbCurr.execute(userSql,userValues)

dbUserId = int(dbCurr.fetchone()[0])

db.commit()

dbCurr.close()

Some more stories from the web:

Learning a programming language

2018-12-27T03:30:00.003-08:00

There are tons of programming language and new ones keep popping and old ones going out. There are some who withstand the test of time and continue to flourish. Being in the information technology industry, it's imperative to keep learning new languages and keep sharpening the skills. Learning a new language is not always easy as many times there are paradigm shifts and it takes time to grasp them.

In this blog, I will delve on the art of learning a language and what has helped me while learning nuances of Java, Python, Javascript and couple of more. I still would not claim myself as an expert in any of this but have done a fair amount of coding in those languages and couple of more. Whenever I had to pick a new language, I try to understand the language on the following dimensions to get a good understanding of what the language offers. And be careful, the languages are not sometimes plain programs, they are ecosystems in themselves.

AWS pricing model

2018-10-07T09:27:00.005-07:00

Many organizations have adopted cloud services as an inherent part of their IT infrastructure. With the cloud infrastructure, the upfront capital cost is not needed. One can hire services on a need basis. However, one still has to be careful about the cost that gets incurred as part of hosting the IT infrastructure on the cloud. Every second service is up and running, it makes the dollar meter to rotate.

SQL vs NoSQL

2018-09-11T04:47:00.005-07:00

Image Source: Pixabay

SQL and NoSQL are two important choices for application data storage needs. There is a lot of confusion about which one to choose and what is a good fit. There are buzzwords on both sides which make the choices more confusing. How to choose one over another?

Meetup - Pune Develope community - IoT and AI

2018-09-01T20:29:00.000-07:00

Spoke about IoT and AI with a focus on the technical architecture and various elements of it. IoT and AI are bound to change the world in multiple ways. IoT is a perfect setup to generate a lot of data about various things and AI is data hungry to build better insights.

Slides at https://www.slideshare.net/LalitMohanChandraBha/iot-and-ai-112716904
Meetup Link: https://www.meetup.com/Pune-Developers-Community/events/252807697/

Thanks to a wonderful and engaging audience.

AWS IoT - Registering CA certificate

2018-08-22T01:48:00.000-07:00

Use openSSL to generate the root key

Generate the key:

2048 - Encryption strength. AWS needs minimum 2048

openssl genrsa -out rootCA.key 2048

Generate the pem file

Put the appropriate days for the certificate to be valid

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 10000 -out rootCA.pem

This will ask a set of questions. Answer them appropriately. An example is
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields, there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:MH
Locality Name (eg, city) []:Pune
Organization Name (eg, company) [Internet Widgits Pty Ltd]: My company
Organizational Unit Name (eg, section) []: mycompany
Common Name (e.g. server FQDN or YOUR name) []:admin.mycompany
Email Address []:admin.mycompany@whatever

Now go to AWS IoT Service

Navigate to Secure -> CA

Click on Register on the right-hand side. This will open a page. Click on Register CA and follow the instructions. Make sure that in Step 3 in details for FQDN you have to put the key as mentioned in Step 2.

At Step 5 and 6 upload the required files.

Check "Activate CA certificate"

Check "Enable auto-registration of device certificates"

DynamoDB best practices for scalability

2018-08-20T05:04:00.001-07:00

DynamoDB can be taken as a glorified hash map. It's essentially a key-value pair. This brings an important difference in terms of how to design an RDBMS schema and a DynamoDB schema. DynamoDB schema has to be approached via a query mindset. How you are going to query to the system should be basically driving the structure of your schema.

Think about queries and not about the data model and design around queries.

Facebook not updating shared image after changing it at source

2018-07-13T20:54:00.000-07:00

You have shared a link on Facebook from your Video or webpage. Facebook shows the image or thumbnail as a part of the Facebook feed. Now you go and change your content however the Facebook refuses to show the updated content/

What happens?
Facebook crawls the link and caches the image/thumbnails in the Facebook DB. It will refresh only when Facebook will crawl/scrape the link again.

How to Fix?

Go to the link https://developers.facebook.com/tools/debug
If you have multiple URL's use Batch invalidator and enter URLs with space between each URL
Enter your Url and hit debug

It will take you to another screen which will show when the scraping was done last time.
Click on "Scrape Again"
Your new content should be visible
Sometimes it still shows the old content for example with YouTube thumbnails. That happens because after updating on YouTube it also takes time to propagate the change to all its servers. It may take up to 24 hours for the same.

Forking and maintaining a git repository

2018-03-18T07:15:00.001-07:00

There are many public repositories which at times we want to fork and continue our path. However, at the same time, we want to keep pulling the changes from the original repository also. In this post, we will look into how to fork a repository and how to pull the changes in the future.

Forking

Let's say we want to fork a repository

Application versioning with Git and Gradle

2018-03-18T02:00:00.002-07:00

Please follow Rest API Server to see the big picture and GitHub repo details.

Application versioning in Git is tricky compared to subversion. Subversion commits are in the increasing order of number so it's easy to put them as part of the application version. Being numbers they are easy to comprehend. However, Git commit id's are hash which would frighten any faint-hearted. A typical git commit id may look like

Apache Tika Tutorial

2018-03-04T06:13:00.000-08:00

Apache Tika is a powerful library to detect and extract text and metadata from thousands of file formats. This is very useful when you are more interested in the content of the file and building your logic on top of it.

Let's see a simple code in terms of how to use Tika

Hibernate Envers

2018-02-27T07:49:00.003-08:00

Please follow Rest API Server to see the big picture and GitHub repo details.

Hibernate Envers helps in creating an audit trail of the records in the database. Hibernate Envers audits all the attribute changes in an audit table. To enable the support for Hibernate Envers, add the dependency

compile('org.hibernate:hibernate-envers')

Add the appropriate version. If you are using Spring boot dependency management then the version will be automatically taken care by Spring boot.

Paging and Sorting using Spring Boot

2018-02-27T04:34:00.004-08:00

Picture Credit : Pixabay

Please follow Rest API Server to see the big picture and GitHub repo details.

Paging and Sorting is an important use case for any application. Any web or mobile frontend, a listing will usually need such a capability.

To support Paging and sorting in a Spring boot application, we need to do the following.

The Spring data repository interface

Liquibase Tutorial

2018-02-25T07:46:00.001-08:00

Photo credit: Pixabay.com

Please follow Rest API Server to see the big picture and GitHub repo details.

Liquibase helps in database schema management. The schema can be defined in XML, YAML, JSON and SQL format. It supports multiple databases. Also, the evolution of the database can be managed through liquibase using the concept of changesets.

In build.gradle add

compile('org.liquibase:liquibase-core')

In application.properties add

Swagger 2 Support for Rest API Documentation

2018-02-17T21:26:00.000-08:00

Please follow Rest API Server to see the big picture and GitHub repo details.

Swagger 2 helps in building the documentation for Rest API's. Springfox provides a suite of libraries to support this. To support, Rest API documentation in Spring boot, follow the steps below

Add to Gradle dependencies

compile('io.springfox:springfox-swagger2:2.7.0')
compile('io.springfox:springfox-swagger-ui:2.7.0')

The Swagger dependency helps in documenting the Rest API. The second dependency helps in providing a UI to present the set of API's on browser.

Now define the configuration bean

Rest based API server

2018-02-13T21:49:00.003-08:00

Photo credit: Pixabay.com

Rest based API servers are very common these days. This has also led to the logical separation of front-end and back-end. New front-end techniques like Single page applications nicely fit with these concepts and also is a logical separation of ownership. Backend deals with providing the information in the form of API and front-end deals with presenting them in an appropriate way. This is also true for mobile applications that connect to the backend via Rest-based API's.

Data science - An Engineering Implementation perspective.

2017-12-17T20:35:00.002-08:00

Presented the talk in Expert talks 2017.

numpy ndarray

2017-11-02T10:38:00.000-07:00

numpy ndarray is a n dimensional array. The array is a table of elements of same type. Let's go through some basic operations that we can do with ndarray

Pandas dataframe data manipulation

2017-11-01T06:40:00.001-07:00

Python dataframe is a tabular structure and can be think like a spreadsheet. It provides a lot of functionality to play with data at column, row or individual data point level.

Let's look into a jupyter python notebook about how the different data can be manipulated

Pandas Dataframe data selection

2017-10-31T09:27:00.003-07:00

Python dataframe is a tabular structure and can be think like a spreadsheet. It provides a lot of functionality to play with data at column, row or individual data point level.

Let's look into a jupyter python notebook about how the different data can be displayed

Reading csv and excel files in pandas dataframe

2017-10-30T19:39:00.002-07:00

Dataframe can be used to load the data from various files. Let's look into how we can load the data from excel and csv.

Make sure the data file that you are loading is present in the given location. In my case, I will be assuming that the data file is present at the same location where python code is.

Code to load different files