For Developers

OpenSearch

2024-04-12T11:41:00.000-07:00

2: Snapshot Stored location: We choose snapshot frequently 1 hour in AWS in which location snapshot are getting stored ?

To get list of automated snapshots run the below API:

If your domain encrypts data at rest—>GET _snapshot/cs-automated-enc/_all
If your domain doesn’t encrypt data at rest—>GET _snapshot/cs-automated/_all

Snapshot Restore from Kibana console

Syntax:

POST /_snapshot/location/snapshot_name/_restore?wait_for_completion=false

POST /_snapshot/instaclustr-production/es-a4f4eb4a-271c-4300-84f4-0794bb498a50-snapshot-5ff78b07-1d94-4df0-866b-d821f5a3a812/_restore?wait_for_completion=false

{

"indices": "index_name",

"ignore_unavailable": true,

"include_global_state": false,

"include_aliases": true

}

Data Engineering

2023-09-08T12:02:00.006-07:00

What is PySpark?

Python API for Apache spark

It's openSource

Distributed computing framework

set of libraries for real-time and large-scale data processing

Apache Spark is written in Scala

Key Features:

Rapid processing

Effectiveness with RDD -> Resilient Distributed DataSet

Objects are immutable and it runs in distributed

Fault tolerance, as a result of node failure it re-process

PySpark is needed

pyspark can be done in python or scala

Python is superior in readability, comprehensive, and maintainable, compare to scala which gives complex code

It also provides machine-learning libraries

Ease of learning python with simple syntax compare to scala

PySpark vs Scala

Courtesy link: https://www.youtube.com/watch?v=6F2doPE0-vc

Docker

2021-03-11T19:22:00.002-08:00

Install MySQL 5.7 in Docker


bash> docker pull mysql/mysql-server:5.7
bash> docker images
bash> docker run --name=mysql57 -d mysql/mysql-server:5.7
bash> docker logs mysql57
bash> docker exec -it mysql57 mysql -uroot -p
 mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'newpassword';
 mysql> update mysql.user set host = '%' where user='root';
Quit from mysql client
docker restart mysql57

Connect from MySQL WorkBench

jacoco plugin usage with maven

2020-12-08T16:09:00.006-08:00

From the place where you have the pom.xml run the below line to get the site > jacoco under target for index.html to see the report on the package.

mvn -B org.jacoco:jacoco-maven-plugin:prepare-agent clean verify org.jacoco:jacoco-maven-plugin:report

ElasticSearch Commands and Support

2020-10-15T13:32:00.006-07:00

Curl to GET index data includes header and replicas with sort by replica.

curl -X GET "http://IP:9200/_cat/indices?h=index,rep&s=rep"

Check the cluster health

curl -X GET "http://IP:9200/_cluster/health?pretty=json"

Turn of cluster settings for wildCard index execution

curl -XPUT "http://IP:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"persistent":{"action.destructive_requires_name":"false"}}'

curl -XDELETE "http://IP:9200/1.9.2__idx_test__y*"

curl -XPUT "http://IP:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{"persistent":{"action.destructive_requires_name":"true"}}'

Update index replication

curl -X PUT "http://IP:9200/index_name/_settings" -H 'Content-Type: application/json' -d '{"index":{"refresh_interval":"30s","number_of_replicas":1}}'

RED ElasticSearch Cluster (Debug with explain to see the current allocation).

curl -XGET IP:9200/_cluster/allocation/explain?pretty

Find Red Indices:
health can be red, green, yellow

curl -XGET 'http://IP:9200/_cat/indices?health=red'

GET Indices and its store size

curl -XGET "http://IP:9200/_cat/indices?format=json&s=store.size:desc" > cat_indices_feb9.json

Steps to Simple Docker File creation with centos 7 images includes jdk, node, maven and push to AWS ECS

2020-07-09T16:20:00.001-07:00

Steps to Simple Docker File to create centos 7 install node, maven

$ vi Dockerfile

FROM centos:7

RUN yum -y update ; yum -y install java-1.8.0-openjdk-devel wget which

RUN yum install -y gcc gcc-c++ \

                   libtool libtool-ltdl \

                   make cmake \

                   git \

                   pkgconfig \

                   sudo \

                   automake autoconf \

                   yum-utils rpm-build && \

    yum clean all

RUN curl -sL https://rpm.nodesource.com/setup_10.x | bash - 

RUN yum -y install nodejs

RUN npm install -g @angular/cli

RUN wget https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

RUN tar xvf apache-maven-3.6.3-bin.tar.gz -C /usr/lib/

RUN echo 'M2_HOME="/usr/lib/apache-maven-3.6.3"' >> /etc/profile

RUN echo 'export M2_HOME' >> /etc/profile

RUN echo 'M2="$M2_HOME/bin"' >> /etc/profile

RUN echo 'MAVEN_OPTS="-Xms512m -Xmx1024m"' >> /etc/profile

RUN echo 'export M2 MAVEN_OPTS' >> /etc/profile

RUN echo 'PATH=$M2:$PATH' >> /etc/profile

RUN echo 'export PATH' >> /etc/profile

RUN echo "source /etc/profile" >> /root/.bashrc

ENV FLAVOR=rpmbuild OS=centos DIST=el7

CMD ["/bin/bash"]

Build docker image and push to AWS. Make sure docker and AWS CLI installed

1. Create repository in AWS ECS. 

2. In AWS select repository (view push commands) to see the commands

Retrieve an authentication token and authenticate your Docker client to your registry.

aws ecr get-login-password --region us-west-2 --profile <profile> | docker login --username AWS --password-stdin gubendran.dkr.ecr.us-west-2.amazonaws.com

Note: If you receive an error using the AWS CLI, make sure that you have the latest version of the AWS CLI and Docker installed.

Build your Docker image using the following command. For information on building a Docker file from scratch see the instructions here . You can skip this step if your image is already built:

docker build -t test_build:2.5.0 .

After the build completes, tag your image so you can push the image to this repository:

docker tag test_build:2.5.0 gubendran.dkr.ecr.us-west-2.amazonaws.com/test_build:2.5.0

Run the following command to push this image to your newly created AWS repository:

docker push gubendran.dkr.ecr.us-west-2.amazonaws.com/test_build:2.5.0

Describe AWS ECR image from the repository created

aws ecr describe-images --repository-name test_build

Install NPM HighCharts, Elastic Dump

2020-07-09T16:19:00.001-07:00

Install NPM HighCharts

Verify node and npm. If missing then install

node -v

npm -v

Install

curl -sL https://rpm.nodesource.com/setup_10.x | bash - 

yum -y install nodejs

node -v

npm -v

yum install bzip2

yum install fontconfig freetype freetype-devel fontconfig-devel libstdc++

npm install -g highcharts-export-server --ignore-scripts

cd /usr/lib/node_modules/highcharts-export-server/node_modules/phantomjs-prebuilt/

node install.js

cd /usr/lib/node_modules/highcharts-export-server

node build.js

To Start highcharts-export-server

highcharts-export-server -enableServer 1 &

ElasticDump

npm install -g elasticdump

Extract from x to y lines from a file in Linux

2020-01-22T11:17:00.001-08:00

$X=5
$Y=10
sed -n -e "$X,$Y p" -e "$Y q" FILE_NAME

Install Elasticsearch, Kibana & Cerebro

2019-10-15T15:31:00.001-07:00

Install Kibana

yum remove kibana (This will remove the kibana and kibana config which exist earlier)

Download Kibana RPM from the below link
https://www.elastic.co/guide/en/kibana/5.4/rpm.html
> touch /var/log/kibana.log
> chown -R kibana:kibana /var/log/kibana.log
> vi /etc/kibana/kibana.yml
server.port: 5601
server.host: "HOST_NAME"
server.name: "SERVER_NAME"
elasticsearch.url: "http://ES_URL:9200"

logging.dest: /var/log/kibana.log

Start Kibana & Status on CentOS 7
> systemctl start kibana
> systemctl status kibana

Check the log
> journalctl -u kibana

Check the Kibana port open
nc -z localhost 5601

Install Cerebro
> wget https://github.com/lmenezes/cerebro/releases/download/v0.8.3/cerebro-0.8.3.tgz
> tar -xvf cerebro-0.8.3.tgz

Edit cerebro service file in centOS 7 and update the location where cerebro is extracted
> vim /etc/systemd/system/cerebro.service

Edit cerebro configuration and update the Elasticsearch config which it needs to have lookup
> vim cerebro-0.8.3/conf/application.conf
Update the ES host server, port and ES Cluster name by enabling

> systemctl start cerebro
> systemctl status cerebro

Or manually start
> ./bin/cerebro -Dhttp.port=1234 -Dhttp.address=127.0.0.1

Check the log
> journalctl -u cerebro

Check the Cerebro port open
nc -z localhost  9000

Install Elasticsearch
Download from the link:
  https://www.elastic.co/guide/en/elasticsearch/reference/5.4/gs-installation.html
Increase the file max in your linux
> sysctl -w fs.file-max=500000
Linux Scheduler (Cron) to purge the ES logs based on the cluster name
> 0 * * * * /bin/bash -c "/bin/find /var/log/elasticsearch -type f | /bin/grep -Pi '(ent\-stage1\-es)(\d{1,4}\-?){1,3}' | /bin/xargs rm -f"

Increase the JVM for ES (Min and Max memory)
> vim /etc/elasticsearch/jvm.options
-Xms8GB
-Xmx8GB

Modify the ES Configuration
> vim /etc/elasticsearch/elasticsearch.yml
cluster.name: ent-stage1-es
node.master: true
node.data: false (For Data Nodes keep this flag to true)
node.ingest: false (For Ingest Nodes keep this flag to true)
  path.data: /mnt/elasticsearch-data
  path.logs: /var/log/elasticsearch
  network.host: 0.0.0.0
  http.port: 9200
transport.tcp.port: 9300

Configure only the master, datanodes will be linked based on the cluster name we keep across the servers
discovery.zen.ping.unicast.hosts: ["ES_MASTER_NODE_IP1", "ES_MASTER_NODE_IP2", "ES_MASTER_NODE_IP3"]
Configure 1 or 3 or 5 Master nodes in Odd numbers to avoid split brain
discovery.zen.minimum_master_nodes: 1

Note : a) Cluster name should be same in Master Node, DataNodes and other nodes. Keep master true for master node and others false. Keep the master either 1 or 3 to avoid split brain. Refer ES documentation to know more about it.

Master Node: Keep only node.master: true if you want master node
Data Node:  Keep only node.data: true if you want data node
Ingest Node: Keep only node.ingest: true if you want ingest node
Co-ordinate Node: Keep node.master: false, node.data: false and node.ingest: false and dedicated node will be available for co-ordination as co-ordinate node.

> systemctl start elasticsearch
> systemctl status elasticsearch

Check the log
> journalctl -u elasticsearch

Check the ElasticSearch port open
nc -z localhost 9200

Add or Remove x-pack in ES
> cd /usr/share/elasticsearch/bin/
./elasticsearch-plugin <install|remove> x-pack

Exclude / Decommission a node
First run the below query in kibana. Once the shards gets moved to other data nodes than shutdown the elasticsearch on that node. You can use cerebro overview to click the node and see the docs availability in the instance if any.

PUT _cluster/settings
{
"transient" :{
"cluster.routing.allocation.exclude._ip" : "<IP_ADDRESS>"
}

}

Note: If master and data been the same node still you can run the above settings to move the shards from master node to data node.

Adding a node (Data)
Install ES with the same version on the node same as other nodes in the cluster and keep the cluster.name in the elasticsearch.yml file same as other nodes, other fields node.data: true and bring the ES up in new node. So, based on the cluster name new node attach to the existing cluster.

Adding a node (Master):
Install ES with the same version on the node same as other nodes in the cluster and keep the cluster.name in the elasticsearch.yml file same as other nodes, other fields node.master: true and bring the ES up in new node. So, based on the cluster name new node attach to the existing cluster.

You may need to edit the elasticsearch.yml file add the new node (master) ip in the discovery.zen.ping.unicast.hosts and increment the discovery.zen.minimum_master_nodes.

System Configuration Suggestion
1) Try to have r5 instance types for Master nodes and i3 for data nodes. As master node need more memory and i3 for data has more IOPS.

Git Bash

2019-02-22T11:55:00.000-08:00

Git Bash
----------
- Git Clone project (Get the project into your local)
git clone <git_url>

- Command will show the git master (origin)
git branch

- Command will fetch everything from the master
git fetch

- Checkout the branch / switch to existing branch
git checkout <branch_name>

- Revert a single file (Complete revert like SVN)
git checkout <file_name_includes_path>

- Creating a new branch 'branch_name'
git checkout -b <branch_name>

- Merge specific commit id (Advantage it pushes the same commit version over new branch)
git cherry-pick <218c4e40654a2d53fdaf3f1b47a58867f432b8ed>

- Changes which is in local and not in remote
git log --branches --not --remotes

- Pull the changes into local from git
git pull origin <branch_name>

- Hard reset from local cahanges
git reset --hard HEAD^

- Unstagged a staged file
git reset HEAD .

- Add .gitignore file
touch .gitignore

- Push local changes file into staged
git add <FileName>

- Commit the file into git (From unstagged to stagged)
git commit -m 'commit message' <filename>

- Push the changes into remote branch from local
git push -u origin <branch_name>

- Show the differences in local
git diff .

- Git stash of switching between branches (store the files temperoarty storage and switch to branch (new)). It will help you to store and switch between branches
git stash

- Stashing untracked files (save)
$ git stash -u or $ git stash --include-untracked

- save specific file file specific messages. -u untracked files to push untracked files.
git stash push -u -m "<your message>" <filename_includes_path> [<filename_includes_path>]

- Git show the stash files
git stash show stash@{1}

- Git stash show the stash files diff
git stash show -p stash@{1}

- see the `git stash list` once you see choose and apply specific. --index will stage the file in the same location where you kept earlier
git stash apply stash@{1} --index

- Popping your stash removes the changes from your stash and reapplies the last saved state
git stash pop

- Delete from git stash (storage) the latest
git stash drop [stash_id]

- Remove all from stash
git stash clear

How to Merge:
--------------
git checkout <working_branch>
git pull origin <working_branch>
git checkout <master_branch_source_branch>
git pull origin <master_branch_source_branch>
git merge <working_branch_which_you_want_to_merge_in_source>
(Resolve the conflicts if any)
vi <file_name>
git add <resolved_conflict_files>
git commit -m "" <resolved_conflict_files>
git push -u origin <master_branch_source_branch>

Export / Import Putty All Settings :

2017-10-02T17:11:00.001-07:00

Open cmd.exe (Command prompt). Execute below command in command prompt

regedit /e "%USERPROFILE%\Desktop\putty.reg" HKEY_CURRENT_USER\Software\SimonTatham

In Desktop you will see putty.reg (Exported)

To Import putty settings

Double-click on the *.reg file and accept the import.

Or from command prompt execute below command :

regedit /i putty.reg

https://stackoverflow.com/questions/13023920/how-to-export-putty-sessions-list

Apache Configuration with Proxy in RHEL 7 / CentOS 7

2017-09-21T16:12:00.000-07:00

Apache Web Server (httpd) in RHEL and CentOS are similar.

$ yum install httpd mod_proxy mod_ssl mod_proxy_html mod_rewrite -y

httpd : httpd daemon install http server (web) apache

mod_proxy : This library usually comes by default. This helps for proxy setup (Forward / reverse)

mod_ssl : This library helps for HTTPS connection support in mod_proxy (proxy) to do for HTTPS as well

mod_proxy_html : This library supports for rewrite rule to support html, images and other content loading in apache

mod_rewrite : This library helps for rewrite rule in apache configuration.

$ grep 'mod_proxy' /etc/httpd/conf.modules.d/00-proxy.conf

Above command will show you already the mod proxy libaries loaded (LoadModule) in apache (httpd) and in the same configuration you will see for ssl (LoadModule configuration) in different *.conf

In order for apache to achieve forward and reverse proxy create below configuration.

Create reverse-proxy.conf file or your own file naming file and have the below configuration.

$ vi /etc/httpd/conf.d/reverse-proxy.conf

# Below fields helps on forward proxy request and proxy time out

ProxyRequests On

ProxyVia On

ProxyTimeout 60

# Below field helps to turn on HTTPS configurations (reverse / forward)

SSLProxyEngine On

# Proxy servers permissions

Require all granted

# Specify Forward proxy remote servers

ProxyRemote "http" "http://proxy.ebiz.abccompany.com:80"

ProxyRemote "https" "http://proxy.ebiz.abccompany.com:80"

# Specify no proxy servers

NoProxy ".abccompany.com"

# Reverse proxy urls and connectiontimeout happens when there is no response from the url

ProxyPass /foo http://www.webreference.com/html/tutorial2/2.html connectiontimeout=5

ProxyPassReverse /foo http://www.webreference.com/html/tutorial2/2.html

ProxyPass /bar https://vzweb2.abccompany.com connectiontimeout=5

ProxyPassReverse /bar https://vzweb2.abccompany.com

ProxyPass /geek https://www.ssllabs.com/ connectiontimeout=5

ProxyPassReverse /geek https://www.ssllabs.com/

Restart 'service httpd restart' after configuration changes.

You can add environment variables in file '/etc/sysconfig/httpd' if required.

Test it with : curl http://localhost/foo

curl http://localhost/bar

Best Article to know about forward-reverse proxy :

https://geekpeek.net/forward-reverse-apache-proxy-centos/ (The best forward and reverse proxy)

Reverse proxy well explained video link : https://www.youtube.com/watch?v=MQMIhAbPzb4

AWS Tutorial: How to quickly set up an SSL certificate for use with Amazon Web Services ELB

2017-07-22T11:44:00.003-07:00

Refer Easy Steps : https://www.thorntech.com/2015/08/setting-up-an-ssl-certificate-for-use-with-amazon-web-services-elb/

View Java keystore information with "keytool list"

keytool -list -v -keystore privateKey.store

keytool executable comes in JDK/bin. You must configure $JAVA_HOME/bin in $PATH to use the command. Check the CN (CommonName)

Article about Certificates Extensions and Encodings : http://info.ssl.com/article.aspx?id=12149

PKCS7 Decoder

https://certlogik.com/decoder/

https://www.sslshopper.com/certificate-decoder.html

Installing a SSL certificate on Amazon Web Services (AWS)

https://www.namecheap.com/support/knowledgebase/article.aspx/9593//installing-a-ssl-certificate-on-amazon-web-services-aws

Use AWS Certificate Manager to upload SSL certificate (Import).

Certificate body : Provide the public certificate (PEM) provided by ssl team by converting .cer to pem

Certificate private key : Provide the private key (PEM) you used to generate CSR file

Certificate Chain : Provide the intermediate and Root certificate (PEM)

In order to verify the healthy of the file and port in https://

wget https://ec2-52-2-59-63.compute-1.amazonaws.com:8443/index.html --no-check-certificate

In order to verify the healthy of the file and port in http://

wget http://ec2-52-2-59-63.compute-1.amazonaws.com:8080/index.html

You also need to create keystore file and drop in tomcat 8443 port by combining keys.

Step by step instructions on self-signed certificate and configure Tomcat over SSL

2017-07-19T15:51:00.000-07:00

Note : If you create self signed certificate from openssl then create keystore file also using openssl

1.1 Create a pair of PKI keys

openssl genrsa -out public-dns-name-private.pem 2048

Create a self-signed X509 certificate:

openssl req -new -x509 -key public-dns-name-private.pem -out public-dns-name-certificate.pem -days 365

Create a PKCS12 keystore and import (or export depending on how you look at it) the host certificate we just created:

openssl pkcs12 -export -out humkeystore.pkcs12 -in public-dns-name-certificate.pem -inkey public-dns-name-private.pem

2.2 Convert the PKCS12 keystore to Java keystore using Java keytool.

keytool -importkeystore -srckeystore humkeystore.pkcs12 -srcstoretype PKCS12 -destkeystore humkeystore -deststoretype JKS

Update Tomcat : In $CATALINA_HOME/CONF/server.xml

In Amazon Alexa Skill : "I will upload a self-signed certificate in X.509 format" open generated x509 certificate file and provide content : public-dns-name-certificate.pem

Thanks to https://yimingsun.wordpress.com/2012/03/12/step-by-step-instructions-on-self-signed-certificate-and-tomcat-over-ssl/

AWS Account Migration - Steps for some of the services to migrate from one AWS Account to other

2017-07-19T15:50:00.000-07:00

1) Login into AWS account and go to BillingDashboard and find out list of services used.

Note : There will be charge either 0$ or x$ for used services.

2) Go to VPC dashboard and understand VPC (custom), ElasticIps, Security Group, subnets and create the same in another account (target account you want to migrate)

3) EC2 Instance Account Transfer : https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-ec2-instance/

4) RDS account Transfer : https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-rds/

Note : Only manual snapshot backup can be shared and restored.

5) Acount transfer S3 Objects : https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-s3/

Note : aws s3 sync s3://sourceBucket s3://DestinationBucket --profile AccountAAdmin --profile AccountBAdmin

Please make sure the destinationBucket name is different as its unique. Sometimes we try to provide same assuming copy.

Followed example from AWS : http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html

Once copied delete bucket commands : http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html

6) ELB : Create LoadBalancer target Group and other details in new account from the existing account.

7) Lambda, Lex, elasticBeanStack you need to configure the same in new account and copy the code from the old aws account.

Amazon - Alexa - Tips to start your own Skill Kit

2017-02-22T11:44:00.001-08:00

Why Alexa ?

Alexa, the voice service that powers Echo, provides capabilities, or skills, that enable customers to interact with devices in a more intuitive way using voice. (Coolest IoT as well to integrate with lights and thermostat control through Amazon Echo)

Examples of these skills include the ability to play music, answer general questions, set an alarm or timer and more. Alexa is built in the cloud, so it is always getting smarter.

Official Amazon Pages about
Alexa : https://developer.amazon.com/alexa
Alexa Skill Kit : https://developer.amazon.com/alexa-skills-kit

Alexa Skill Kit (ASK) develop in an hour quick tour to get your hands dirty (JS) - Note : Alexa Skill Interaction Model creation was not updating in IE. Better use firefox / chrome.
6 Steps to build first Alexa Skill : https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/content/fact-skill-1

Steps Detailed to develop : https://developer.amazon.com/blogs/post/Tx3DVGG0K0TPUGQ/New-Alexa-Skills-Kit-Template:-Step-by-Step-Guide-to-Build-a-Fact-Skill

Alexa Skill kit Samples : https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/using-the-alexa-skills-kit-samples

Alexa Web Browser Testing tool (Works only in webRTC (RTC-RealTimeCommunication Browser)) : https://echosim.io/welcome?next=%2F

Alexa Skill Kit (ASK) forum : https://forums.developer.amazon.com/spaces/23/index.html

ASK Certification Requirements for Custom Skills : https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-submission-checklist

ASK custom skill using Web-Service - https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/deploying-a-sample-skill-as-a-web-service

Amazon 10 Minutes tutorial (Try tutorials in AWS) - https://aws.amazon.com/getting-started/tutorials/

Social Pages (YouTube) about Alexa to watch and learn :

ASK 101 by Amazon Developer : https://www.youtube.com/watch?v=31K4vse9Jmc

ASK Custom skill easy implementation step in more understandable with intents and slots includes nice examples and github sourceCode reference: https://www.youtube.com/watch?v=zt9WdE5kR6g&t=1s

Developing Alexa Skill (Explained in 1 to 6 chapter in more detail) : https://www.youtube.com/watch?v=QxgdPI1B7rg&list=PL2KJmkHeYQTO6ci5KF08mvHYdAZu2jgkJ

Find your icon for your custom app : https://www.iconfinder.com/icons/10930/badge_reddit_icon#size=128

In MSPaint open the image 512 and click -> re-size and re-size the image to size you need.

Client / Server Authentication using openssl

2017-02-22T09:02:00.000-08:00

Steps to Create CSR (Certificate Signing Request) using OpenSSL and create JKS file to interact with Server

What is client authentication?
Client Authentication is the process by which users securely access a server or remote computer by exchanging a Digital Certificate.

#To create CSR and Private Key
openssl req -new -newkey rsa:2048 -nodes -out star_yourdomainname_com.csr -keyout star_yourdomainname_com.key -subj "/C=US/ST=New Jersey/L=Bedminster/O=Organization Name/OU=Organization Unit/CN=*.yourdomainname.com"

# Combine private key and public key file received from 3rd party
cat star_yourdomainname_com.key star_yourdomainname_com.pem_SIGNED.pem > projectName.pem

# Generate pkcs12 file from private + public key file
openssl pkcs12 -export -in projectName.pem -out projectNamekeystore.pkcs12 -name projectName -noiter -nomaciter

# Create jks File from pkcs12 file generated
keytool -importkeystore -srckeystore projectNamekeystore.pkcs12 -srcstoretype pkcs12 -srcalias projectName -destkeystore projectNameKeyStore.jks -deststoretype jks -deststorepass pass@123 -destalias projectName

# To list the keystore stored
keytool -v -list -keystore keystore

Write Java program using HttpClient to test this JKS file and Get / Post the data from 3rd party server

Curl : Through curl also same works and testing can be done

# Combine private key and public key file received from 3rd party
cat star_yourdomainname_com.key star_yourdomainname_com.pem_SIGNED.pem > projectName.pem

Curl Command :
curl -k -vvvv --request GET -H "Accept: application/json" -H "Content-Type: application/json" -a --cert projectName.pem "https://thirdpartydomainname.com/WebGateway/rest/Location/query/location?userId=1234"

MySQL 5.7 supports JSON dataType field

2016-05-16T12:13:00.000-07:00

MySQL 5.7 supports JSON dataType field

Below is the quick working syntax to play-around with JSON data Type field.

#Create a table with JSON Field.
#You can have primary key, foreign key and indexes with json type field.



CREATE TABLE `book` (

 `id` MEDIUMINT(8) UNSIGNED NOT NULL AUTO_INCREMENT,

  `title` VARCHAR(200) NOT NULL,

  `tags` json DEFAULT NULL,

  PRIMARY KEY (`id`)

 )ENGINE=INNODB;

# Insert JSON Array



 INSERT INTO `book` (title, tags) VALUES

    ('JavaScript',

     '["JS", "E12", "JSON"]'

  ); 

  

 # Insert JSON Object

 INSERT INTO `book` (title, tags) VALUES

     ('JSP',

     '{"c" : 3, "d" : 4}'

     );

# Select Queries



SELECT id, title, tags FROM book;



SELECT id, title, JSON_TYPE(tags) FROM book;



SELECT id, title, JSON_ARRAY(tags) FROM book WHERE id = 1;



SELECT id, title, JSON_VALID(tags) FROM book WHERE id = 2;



SELECT * FROM book WHERE JSON_SEARCH(tags, 'one', 'JSON%') IS NOT NULL;



# Return available field in 0th index from JSON

SELECT title, tags->"$[0]" AS tag1 FROM book;



#Returns null on non-availability

SELECT title, tags->"$.a" AS tag1 FROM book;

Java Quiz

2015-02-24T09:01:00.000-08:00

1) What will be the output when you execute below code ?

for (int i = 0; i >= (-1) * 5; i--) {
          System.out.println(i);
}

Output : 0 -1 -2 -3 -4 -5

for loop repetitive control structure flow :

for (initialize; boolean_expression; update) {
}

Initialize will be called only once in the for loop to be initialize for the first time loop starts. You can leave blank by providing semicolon(;) placeholder.
boolean_expression will be called next to initialize to validate expression and if its true body of the for loop exucutes. False loop exits
update will be called upon completion of the body execution of for loop.
Once update complete it calls the boolean expression to validate to proceeds body and body completion will call update and moves on back to booleanExpression

Note : There is no difference between i-- or --i or i++ or ++i in the for loop update.

2) What will be the output when you execute below code ?

int i = 1;
int j = ++i;
System.out.println("i: " + i + " j: " + j);

and

int i = 1;
int j = i++;
System.out.println("i: " + i + " j: " + j);

Ouput : 
i: 2 j: 2
i: 2 j: 1

Why Difference?
++i increments the value first and then return it
i++ return the value first and then increments it

Usually, try to use i = i + 1 for clear understanding in the code.

3) What will be the output when you execute below code ?

String testSplit = "hello,,world";
String[] testSplits = testSplit.split(",");
System.out.println(testSplits[2]);

Output : world

String testSplit = "hello,world,";
String[] testSplits = testSplit.split(",");
System.out.println(testSplits[2]);

Output : java.lang.ArrayIndexOutOfBoundsException: 2

String testSplit = "hello,world,";
String[] testSplits = testSplit.split(",", -2);
System.out.println(testSplits[2]);

Output :      (empty)

Note : By default, split drops all empty trailing columns, so any attempt to access the final column will result ArrayIndexOutOfBoundsException. Passing negative number (limit) as the second arg to split causes it to retain the trailing empty columns.

4) What will be the output when you execute below code ?

Pattern p = Pattern.compile("^([\"']?)\\d\\d:\\d\\d\\1,([\"']?)[A-Z]\\w+\\2,.*$");
  String regexpInput = "1:23,Logout Now";
  if (p.matcher(regexpInput.toString()).matches()) {
   System.out.println("Good");
  } else {
   System.out.println("Bad");
  }
Ouput : Bad

Why Bad ?
1. hour has only 1 digit 1 instead 2 digit
2. 2nd column has space in regexp which is not mentioned
3. Regexp expects 3rd column with comma(,) after some text 1 or more which is missing

\\1 and \\2 is backreference to repeat the grouping element
group starts from 1, 2, 3 Backreference also considered as group in regexp.

Difference between iBatis and Hibernate ? (Both are persistence framework)
iBatis / myBatis is sql driven model. It means based on sql you want to control application changes (SqlMap XML file)
Hibernate is object driven model. It means you design your object and create fields in database. (hbm - hibernate mapping xml file)
Pros and Cons :
iBatis is database dependent due to SQL usage, but faster development and lighter with cache support.
Hibernate is database independent due to HQL based approach. Its more heavy compare to JPA / iBatis where as highly scalable with advance cache support.

iBATOR - Code generator for iBatis.

castor - Castor is open source java data binding framework. Moving data from XML to Java programming language objects and from Java to database. same as JAXB.

Difference between JPA and Hibernate ?
JPA is specification / interface based on JSR
Hibernate / iBatis is implementation using JPA.

Oozie Examples

2015-02-21T15:13:00.000-08:00

Setting Up the Examples
The examples/ directory must be extracted from oozie and copied to the user HOME directory in HDFS:

cd /usr/local/oozie
tar -xvzf oozie-examples.tar.gz
chown -R hduser:hadoop examples/
hadoop fs -put /usr/local/oozie/examples/ /user/hduser/examples/

NOTE: If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be copied.

Running the Examples
Add Oozie bin/ to the environment PATH in .bashrc or /etc/profiles

export OOZIE_HOME=/usr/local/oozie
export PATH=$PATH:$HADOOP_HOME/bin:$HIVE_HOME/bin:$PIG_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin

Execute job from Terminal (hduser)

oozie job -oozie http://localhost:11000/oozie -config /usr/local/oozie/examples/apps/map-reduce/job.properties -run

NOTE: The job.properties file needs to be a local file during submissions, and not a HDFS path. Modify job.properties for namenode & jobtracker url.

Check Oozie URL to track the oozie workflow job status : http://localhost:11000/oozie

Note : The example applications are under the $OOZIE_HOME/examples/app directory, one directory per example. The directory contains the application XML file (workflow, or worklfow and coordinator), the job.properties file to submit the job and any JAR files the example may need. Go through each workflow to run hive, java, sqoop, streaming examples and try

Reference : Oozie Example
Oozie workflow with Pig and Hive

Big Data Growth Trends and Events

2015-02-20T16:23:00.000-08:00

Hadoop is Growing faster than expected..

Click to Zoom the image : Hadoop Growth

US President speaks about importance of BigData Trend.

Apache Oozie - Schedule your job using powerful workflow engine

2015-02-20T16:13:00.001-08:00

Apache Oozie
Oozie is an extensible, scalable and reliable system to define, manage, schedule, and execute complex Hadoop workloads via web services. More specifically, this includes:

* XML-based declarative framework to specify a job or a complex workflow of dependent jobs.
* Support different types of job such as Hadoop Map-Reduce, Pipe, Streaming, Pig, Hive and custom java applications.
* Workflow scheduling based on frequency and/or data availability.
* Monitoring capability, automatic retry and failure handing of jobs.
* Extensible and pluggable architecture to allow arbitrary grid programming paradigms.
* Authentication, authorization, and capacity-aware load throttling to allow multi-tenant software as a service.

Oozie Engines :
Oozie has Workflow Engine, Coordinator Engine and Bundle Engine

PreRequiste :
JAVA_HOME (Java)
M2_HOME (Install Maven)
HADOOP_HOME (hadoop)
Pig (PIG_HOME)

Download Oozie

Make sure below command works

$ java -version
$ javac -version
$ mvn -version

Extract Oozie archieve

$ sudo cp ~/Downloads/oozie-*.tar.gz /usr/local/
$ sudo su -
$ cd /usr/local
$ tar -xzf oozie-3.3.2.tar.gz

Building Oozie
The simplest way to build Oozie is to run the mkdistro.sh script:

$ cd oozie-3.3.2
$ ./bin/mkdistro.sh -DskipTests

Oozie Server Setup
Copy the built binaries to the home directory as ‘oozie’

$ cd ..
$ cp -R oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/ oozie

Create the required libext directory

$ cd oozie
$ mkdir libext

Copy all the required jars from hadooplibs to the libext directory using the following command:

$ cp ../oozie-3.3.2/hadooplibs/target/oozie-3.3.2-hadooplibs.tar.gz .
$ tar xzvf oozie-3.3.2-hadooplibs.tar.gz
$ cp oozie-3.3.2/hadooplibs/hadooplib-1.1.1.oozie-3.3.2/* libext/

Get Ext2Js – This library is not bundled with Oozie and needs to be downloaded separately. This library is used for the Oozie Web Console:

$ cd libext
$ wget http://extjs.com/deploy/ext-2.2.zip
$ cd ..

Update ../hadoop/conf/core-site.xml as follows. Hadoop Version 1.2.x:

<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>localhost</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>hadoop</value>
</property>

Note : Here, ‘hduser’ is the username and it belongs to ‘hadoop’ group.

Prepare the WAR file

$../bin/oozie-setup.sh prepare-war

INFO: Oozie is ready to be started

Provide permission to oozie directory

$ chown -R hduser:hadoop oozie

Create sharelib on HDFS

$ su hduser
$ /usr/local/oozie/
$ ./bin/oozie-setup.sh sharelib create -fs hdfs://localhost:54310

Create the OoozieDB

$ ./bin/ooziedb.sh create -sqlfile oozie.sql -run

The SQL commands have been written to: oozie.sql

Start Oozie as a daemon process run / start Oozie as a foreground process run:
oozie-start.sh, oozie-run.sh , and oozie-stop.sh

$ ./bin/oozie-start.sh
or
$ ./bin/oozie-run.sh
or
$ ./bin/oozie-stop.sh

Note : oozie log will be in /usr/local/oozie/logs/oozie.log

URL for Oozie Web Console is http://localhost:11000/oozie

Check Oozie status, should be NORMAL.

$ bin/oozie admin -oozie http://localhost:11000/oozie -status

Try Oozie Examples : Oozie Examples which i tried from my same blog

Oozie Client Setup may required in the remote machine

$ cd ..
$ cp oozie/oozie-client-3.3.2.tar.gz .
$ tar xvzf oozie-client-3.3.2.tar.gz
$ mv oozie-client-3.3.2 oozie-client
$ cd bin

Add the /home/hduser/oozie-client/bin to PATH in .bashrc or /etc/profiles and restart your terminal.

References : Oozie Installation : (Apache Oozie, Rohit Blog, CloudBlog)

Install Maven from archive

2015-02-17T16:20:00.002-08:00

Why Maven ?

Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.

Quick project set-up, no complicated build.xml files, just a POM and go
All developers in a project use the same jar dependencies due to centralized POM.
Getting a number of reports and metrics for a project "for free"
Reduce the size of source distributions, because jars can be pulled from a central location

Make sure you installed Java and exported JAVA_HOME in your PATH variable.

Verify java installation on your machine

$java -version 
$javac -version

Download maven

Extract Maven archive

$sudo cp ~/Downloads/apache-maven-*.tar.gz /usr/local/
$sudo su -
$cd /usr/local
$tar -xzf apache-maven-3.1.1-bin.tar.gz

Set maven environment variables (vi ~/.bashrc or /etc/profiles)

export M2_HOME=/usr/local/apache-maven-x.x.x
export MAVEN_OPTS="-Xms256m -Xmx512m"

Add maven bin directory to system path

export PATH=$PATH:$M2_HOME/bin

Verify Maven installed

$mvn -version

References : Maven

Install Apache Sqoop - Sql to hadoop (HDFS) and inverse the same

2015-02-16T20:12:00.002-08:00

sqoop - Sql to hadoop

Efficient tool to transfer bulk data from structured (relational db's) to hadoop (hdfs / hive / hbase). You can also export data from hdfs to import into other datawarehouses.

Multiple ways to install sqoop :
To install sqoop in Debian distributions (Ubuntu / Debian)

$ sudo apt-get install sqoop

Advantage installing with debian packages than Tar ball. Still to handle yourself efficiently the package use tar ball.

Handle dependencies
Provide for easy upgrades
Automatically install resources to conventional locations

Download Sqoop - Download sqoop version based on the hadoop you installed in your box. If you hadoop 1.x then download sqoop with hadoop 1.x version to avoid incompatible error upon import / export.

$ sudo su
$ (cp ~/Downloads/sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz /usr/local && cd /usr/local/
 && tar -zxvf path_to_sqoop.tar.gz)
$ mv sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz sqoop-1.4.5
$ chown -R hduser:hadoop sqoop-1.4.5

Configure sqoop wrapper with hadoop

$cd /usr/local/sqoop-1.4.5/conf/
$mv sqoop-env-template.sh sqoop-env.sh

Enable HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME by providing hadoop available path. If you want to sqoop data to HBASE or Hive enable those variables as well and provide path in sqoop-env.sh

$vi sqoop-env.sh
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop

In /home/hduser/.bashrc or /etc/profiles

export SQOOP_HOME=/usr/local/sqoop-1.4.5
export PATH=$PATH:$HADOOP_HOME/bin:$HIVE_HOME/bin:$PIG_HOME/bin:$SQOOP_HOME/bin

Installing JDBC Driver for Sqoop

sudo apt-get install libmysql-java

(This will install the jar for mysql connector through java, cd /usr/share/java/ to see the mysql-connector-java.jar). mysql-connector-java.jar must be symlink to mysql.jar. If so follow below

sudo ln -s /usr/share/java/mysql.jar /usr/local/sqoop-1.4.5/lib/

Check your sqoop version

$sqoop-version

Create table movies in your mysql
Refer : Download sqoopSample.sql from github for mysql schema.

Import data (table or all tables) from MySQL into Hadoop (HDFS)
Make sure you started your hadoop. (cd $HADOOP_HOME/bin; ./start-all.sh)
-m Use n map tasks to import in parallel

$cd /home/hduser
$sqoop import --connect jdbc:mysql://hostName/dbName --username userName --password password --table tableName --target-dir /samples/movies -m 1

$sqoop import-all-tables --connect jdbc:mysql://hostName/dbName --username userName --password password

Important :

If you see error "Exception in thread "main" java.lang.IncompatibleClassChangeError Found class org.apache.hadoop.mapreduce.JobContext, but interface was expected" then hadoop version between you installed and sqoop hadoop has mismatched. Hadoop 1 and hadoop 2 has major changes. So, have 1.x or 2.x on hadoop and sqoop-hadoop.
Make sure table has primary key otherwise you need to mention map tasks.

Export data from hadoop (hdfs) into MySQL

$sqoop export --connect jdbc:mysql://localhost/sqoop_test --table movies_export --username root --password root --export-dir '/samples/movies/' -m 1;

sqoop command to create hive tables matches the database table. If already exist it will throw error

hduser$ sqoop create-hive-table --connect jdbc:mysql://localhost/sqoop_test --username root --password root --table movies

Note : Impala uses the same metadata as hive, So you can use create-hive-table to import and query in Impala

sqoop command to import the entire content from the database table to hive, that uses commas(,) to separate the files in data files

hduser$ sqoop import --connect jdbc:mysql://localhost/sqoop_test --username root --password root --table movies --fields-terminated-by ',' --hive-import

hduser$

--hive-overwrite -> This will overwrite the existing content and write the data into hive table
--hive-import -> To import the table into hive we need --hive-import when there is no table in hive.
--hive-table -> If this is not specified by default the database table name will be created in the hive

Default delimiter using sqoop into hive is ^a. You can specify --fields-terminated-by ',' for the specific separator.

Refer : Download sqoopSample.sql from github for mysql schema.

References : Tutorial Reference,
GoodPlaceToStart

Fomat your blog

2015-02-16T09:43:00.002-08:00

How to add CSS to blogger to be available in all pages
Blogger -> Posts -> Layout -> Template Designer -> Advanced -> Add CSS

Add below CSS

.preTagCodeStyle {
font-family:arial;font-size:12px;border:1px dashed #CCCCCC;width:99%;height:auto;overflow:auto;background:#f0f0f0;;
background-image:URL(https://github.com/gubs4u/DownloadSamples/blob/master/codebg.gif);
padding:0px;color:#000000;text-align:left;line-height:20px;
}
.codeTagStyle {
 color:#000000;word-wrap:normal;
}

div.note {
    background-color: #fefaee;
    padding: 10pt;
    margin-top: 0.6em;
    margin-bottom: 0.6em;
    margin-right: 0.6em;
    border: 1px dashed;
    border-radius: 3px;
    border-color: #CCCCCC;
}
div.configuration {
    padding: 1em;
    border: 1px dashed #2f6fab;
    color: black;
    background-color: #f9f9f9;
    line-height: 1.1em;
    font-family: Courier New, Courier, mono;
    font-size: 12px;
    font-style: italic;
    display: block;
}

To format your code add below:

<pre class="preTagCodeStyle"><code class="codeTagStyle">
Your code here
</code></pre>

To add a note use below:


<div class="note">
Your note here
</div>

To add a configuration use below:


<div class="configuration">
Your configuration here
</div>

Note : Go to HTML tab and add this tag. In 'Compose' have option 'Show HTML Literally'