Satya's BigData/Cloud Blog

Comparison between AWS and GCP resources

2022-12-04T20:39:00.000+05:30

GCP (Google Cloud Platform) & AWS (Amazon Web Services) Cloud services comparison

AWS	GCP
Availability Zone	Zone
Virtual Private Cloud (VPC)	Virtual Private Cloud (VPC)
Identity Access Management (IAM)	Identity Access Management (IAM)
AWS CloudShell	Cloud Shell
Market Place	Market Place
EC2	Compute Engine (VM)
S3	Cloud Storage
EFS	Filestore
EBS	Persistent Disk
ElastiCache / Cloud Front	Cloud CDN
Device Farm	Cloud Test Lab
Elastic Beanstalk	App Engine
RDS	Cloud SQL
Redshift	BigQuery
DynamoDB	Bigtable
Elastic Container Service	Kubernetes Engine
Lambda	Cloud Functions
API Gateway	Cloud Endpoints
Elastic Transcoder	Cloud Video
Fargate	Cloud Run/Anthos
Elastic Load Balancer	Cloud Load Balancing
Direct Connect	Cloud Interconnect/Cloud VPN
Route 53	Domains, Cloud DNS
EMR	Cloud DataProc
Kinesis/Data Pipeline	Cloud Dataflow
SNS/Kinesis Data Steams	Pub/Sub
Data Pipeline	Cloud Composer
Code Pipeline	Cloud Build
Auto Scaling	Autoscaler
CloudWatch	Operations Suite (Stackdriver)
Cloud Trail	Cloud Logging
AWS Snowball	Transfer Appliance
Tags	Labels
Reserved instances	VM with committed use discounts
Spot instances	Preemptible VM
Sagemaker	Cloud AutoML
AWS Cloud9	Cloud Code
AWS Glue Data Catalog	Cloud Data Catalog

Other Softwares/Platforms	GCP
Docker Hub	Container Registry
Kafka	Cloud Pub/sub)

Hadoop Online Quiz [ 2022 ]

2022-05-01T20:19:00.000+05:30

Apache Hadoop Interview Questions And Answers

Related Articles: List of NoSQL Databases Hadoop Distributions

Apache Spark Online Quiz

2022-03-13T21:01:00.000+05:30

Spark Interview Questions and Answers [2022]

Loading...

Related Articles: Why Apache Spark? Hadoop Online Mock Test

Hive Date functions

2021-11-10T18:57:00.000+05:30

Date functions in Apache Hive

hive> select current_date();
current_timestamp()

TO_DATE('2018-04-01 10:25:30')
YEAR('2018-03-01 11:20:50')
MONTH('2018-03-01 10:30:50')
DAY('2018-03-01 04:20:30')
DAYOFMONTH('2018-03-01 10:20:30')
HOUR('2018-03-01 10:20:30')
MINUTE('2018-03-01 12:20:30')
SECOND('2018-03-01 11:20:30')
WEEKOFYEAR('2018-05-01 6:20:30')

add_months(timestamp date, int months)
DATE_ADD('2018-03-01', 5)
select date_add(current_date(), 1);
DATE_SUB('2018-03-01', 5)
DATEDIFF('2018-03-01', '2018-02-10') -- difference in number of DAYS

UNIX_TIMESTAMP('2018-01-01 10:20:30','yyyy-MM-dd')
FROM_UNIXTIME( UNIX_TIMESTAMP() )
select year(from_unixtime(unixreviewtime)) as yr, count(*) as review_count from electronics group by year(from_unixtime(unixreviewtime));
SELECT from_unixtime(unix_timestamp('2013-01-01 10:10:10') + 10 * 60) AS result
from_utc_timestamp(timestamp, string timezone)
to_utc_timestamp(timestamp, string timezone)
SELECT (unix_timestamp('2017-10-10 10:10:10') - unix_timestamp('1970-01-01 00:00:00'))/60 -- difference in number of minutes
select unix_timestamp('17:22:38','HH:mm:ss') - unix_timestamp('14:53:19','HH:mm:ss'); -- difference in number of seconds

trunc('2015-03-17', 'MM')
select trunc(current_timestamp(), 'MONTH');
select extract(hour from "2016-10-20 05:06:07")
select extract(minute from interval '3 12:20:30' day to second)
select minute(CAST('2017-09-22 17:22:38' as timestamp) – CAST('2017-09-12 14:53:19' as timestamp ));
select CAST('2017-09-22' as DATE) - CAST('2017-09-12' as DATE );

Related Articles: Sqoop commands awscli commands

Sqoop Commands

2021-09-15T20:00:00.000+05:30

Sqoop command line (CLI)

sqoop help
usage: sqoop COMMAND [ARGS]

Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information

sqoop help import
sqoop import --help

sqoop version

sqoop-version

sqoop import --connect jdbc:mysql://database.example.com/employees --username satya --password sat345
$ sqoop import --connect jdbc:mysql://localhost:3306/hadoop --driver com.mysql.jdbc.Driver --username root --password r00t --table bse
$ sqoop import --connect jdbc:mysql://localhost:3306/hadoop --driver com.mysql.jdbc.Driver --username root --P --table bsmcg --hbase-row-key date --hbase-create-table --column-family firstdecade --hbase-table bsmcg
sqoop import --query 'SELECT a.*, b.* FROM a JOIN b on (a.id == b.id) WHERE $CONDITIONS' --split-by a.id --target-dir /user/foo/joinresults
sqoop import --connect <connect-str> --table SomeTable --package-name com.foocorp
sqoop import --connect jdbc:mysql://db.foo.com/corp --table EMPLOYEES --username SomeUser -P
sqoop import --connect jdbc:mysql://db.foo.com/corp --table EMPLOYEES --fields-terminated-by '\t' --lines-terminated-by '\n' --optionally-enclosed-by '\"'
sqoop import --connect jdbc:mysql://db.foo.com/corp --table EMPLOYEES --hive-import

sqoop import-all-tables --connect jdbc:mysql://db.foo.com/corp

$ sqoop export --connect jdbc:mysql://localhost:3306/hadoop --driver com.mysql.jdbc.Driver --username root --password root --table bsmcg --export-dir /user/satya/bsmcg
$ sqoop export --connect jdbc:mysql://localhost:3306/hadoop --driver com.mysql.jdbc.Driver --username root --password root --table bse --export-dir /user/hive/warehouse/bse
sqoop export --connect jdbc:mysql://db.example.com/foo --table bar --export-dir /results/bar_data

$ sqoop create-hive-table --connect jdbc:mysql://localhost:3306/hadoop --driver com.mysql.jdbc.Driver --username root --P --table bse
sqoop create-hive-table --connect jdbc:mysql://db.example.com/corp --table employees --hive-table emp

sqoop list-databases --connect jdbc:mysql://database.example.com/
sqoop list-databases --connect jdbc:mysql://sqoop......rds.amazonaws.com:3306 --username admin -P

sqoop list-tables --connect jdbc:postgresql://database.example.com/corp
sqoop list-tables --connect jdbc:mysql://sqoop......rds.amazonaws.com:3306/ssDB --username admin -P

sqoop job --list
sqoop job --show myjob
sqoop job --create myjob -- import --connect jdbc:mysql://example.com/db --table mytable
sqoop job --exec myjob -- --username someuser -P

sqoop merge --new-data newer --onto older --target-dir merged --jar-file datatypes.jar --class-name Foo --merge-key id

sqoop codegen --connect jdbc:mysql://db.example.com/corp --table employees

sqoop eval --connect jdbc:mysql://db.example.com/corp --query "SELECT * FROM employees LIMIT 10"
sqoop eval --connect jdbc:mysql://db.example.com/corp -e "INSERT INTO foo VALUES(42, 'bar')"

Related Articles: Spark Interview Questions

awscli commands

2021-06-06T13:52:00.000+05:30

Amazon Web Services Command Line Interface (AWSCLI)

Installing aws cli in Python:
pip install awscli
pip install awscli --upgrade --user

aws [options] <command> <subcommand> [parameters]

aws --version
aws-cli/1.15.4 Python/3.6.3 Darwin/16.7.0 botocore/1.10.4

aws help
aws ec2 help
aws autoscaling create-auto-scaling-group help

aws configure
$ aws configure --profile user2

output format
json
text
table
export AWS_DEFAULT_OUTPUT="text"

$ aws s3 ls
aws s3 ls s3://satya-sparks/NYC_Parking_Tickets/
aws s3 ls s3://satya-sparks --recursive --human-readable --summarize
aws s3 ls s3://mybucket --recursive
aws s3 cp mnist.csv s3://thirumani-bucket
aws s3 cp /tmp/dir/ s3://thirumani-bucket/ --recursive
aws s3 cp /tmp/my_dir/ s3://thirumani-bucket/ --recursive --exclude "*" --include "*.jpg"
sudo aws s3 cp $OUTDIR.tar.gz s3://sr-cache/perfResults/$DATE_DIR/$RUN_ID/
aws s3 cp -R <dir_name>
time aws s3 cp --recursive --quiet . s3://test_bucket/test_smallfiles/
aws s3 cp s3://mybucket/test.txt test_local.txt
aws s3 cp s3://srcbucket/ s3://destbucket/ --recursive --exclude "a*" --exclude "b*"
aws s3 cp s3://mybucket/test.txt s3://mybucket/test2.txt --acl public-read-write
aws s3 cp file.txt s3://mybucket/ --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=emailaddress=user@example.com
aws s3 mv test.txt s3://mybucket/test2.txt
aws s3 mv s3://mybucket/test.txt s3://mybucket/test2.txt
aws s3 mv s3://mybucket/ s3://mybucket2/ --recursive --exclude "mybucket/another/*"
aws s3 rm s3://mybucket/test2.txt
aws s3 rm s3://mybucket --recursive
aws s3 mb s3://mybucket # make bucket
aws s3 mb s3://mybucket --region us-west-1
aws s3 rb s3://mybucket # remove bucket
aws s3 rb s3://bucket-name --force
aws s3 sync . s3://mybucket
time aws s3 sync --quiet . s3://test-bucket/test_randfiles/
aws s3 sync s3://mybucket s3://mybucket2
aws s3 website s3://my-bucket/ --index-document index.html --error-document error.html
aws s3api head-object --bucket test-bucket --key test_bigfiles/bigfile

$ aws ec2 start-instances --instance-ids i-X13836c
aws ec2 stop-instances --instance-ids i-X13836c
aws ec2 terminate-instances --instance-ids i-X13836c
$ aws ec2 describe-instances
aws ec2 describe-instances --filter Name=instance-type,Values=t2.nano
aws ec2 describe-instances --filters Name=instance-state-name,Values=stopped --region eu-west-1 --output json | jq -r .Reservations[].Instances[] .StateReason.Message
aws ec2 describe-instances --filters "Name=tag:Type,Values=Build" \
--query "Reservations[0].Instances[0].PublicDnsName" | sed 's/"$.*$"/http:\/\/\1\/manage/'

aws ec2 run-instances --image-id ami-785bae10 --count 1 --instance-type t2.micro --key-name newpair.pem --security-group new-sg
aws ec2 delete-vpc --vpc-id vpc-a01106c2
aws ec2 describe-key-pairs
aws ec2 describe-key-pairs --key-name MyKeyPair
aws ec2 create-key-pair --key-name MyKeyPair --query 'KeyMaterial' --output text > MyKeyPair.pem
aws ec2 create-key-pair --key-name newpair.pem
aws ec2 delete-key-pair --key-name MyKeyPair
aws ec2 describe-tags --output table
aws ec2 describe-spot-price-history help
aws ec2 create-security-group --group-name my-sg --description "My security group"
aws ec2 describe-vpcs --region us-west-2
aws ec2 describe-vpcs --region us-west-2 --filter "Name=tag:Name,Values=Web VPC"
aws ec2 describe-vpc-attribute --region us-west-2 --attribute enableDnsHostnames --vpc-id vpc-24be405c

aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./step.json aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=IMPALA,Name='Impala program',ActionOnFailure=CONTINUE,Args=--impala-script,s3://myimpala/input,--console-output-path,s3://myimpala/output
aws emr list-steps --cluster-id j-3SD91U2E1L2QX
aws emr create-cluster --ami-version=3.3.0 --applications Name=Hue Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-groups \
InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=m1.large

aws iam create-user --user-name Satya
aws iam create-group --group-name SysAdmin
aws iam add-user-to-group --group-name SysAdmin --user-name Satya
aws iam list-users
aws iam list-users --output table
aws iam list-users --query 'Users[*].[UserName,CreateDate]'
aws iam list-users --query 'Users[*].{Name:UserName,CreateDate:CreateDate}'
aws iam list-groups-for-user --user-name Satya
aws iam list-groups
aws iam get-user-policy --user-name myuser --policy-name mypolicy
aws iam put-group-policy --group-name SysAdmin --policy-name admin-policy --policy-document file://sytem_admin_policy_doc.json
aws iam list-group-policies --group-name SysAdmin
aws iam delete-group-policy --group-name SysAdmin --policy-name admin-policy
aws iam remove-user-from-group --user-name myuser --group-name mygroup
aws iam delete-group --group-name SysAdmin

aws rekognition detect-labels --image "S3Object={Bucket=photo-collection,Name=photo.jpg}" --region us-west-2

aws dynamodb list-tables
aws dynamodb scan --table-name clients_table
aws dynamodb delete-table --table-name clients_table

aws rds create-db-instance \
--engine MySQL \
--db-instance-identifier mydbinstance \
--allocated-storage 20 \
--db-instance-class db.m1.small \
--db-security-groups mydbsecuritygroup \
--db-subnet-group mydbsubnetgroup \
--master-username masterawsuser \
--master-user-password masteruserpassword \
--backup-retention-period 3
aws rds create-db-instance ^
--engine oracle-se2 ^
--db-instance-identifier oradbinstance ^
--allocated-storage 40 ^
--db-instance-class db.t2.small ^
--db-security-groups mydbsecuritygroup ^
--db-subnet-group mydbsubnetgroup ^
--master-username masterawsuser ^
--master-user-password masteruserpassword ^
--backup-retention-period 4
aws rds describe-db-instances
aws rds modify-db-instance \
--db-instance-identifier mydbinstance \
--backup-retention-period 7 \
--no-auto-minor-version-upgrade \
--no-apply-immediately
aws rds modify-db-instance ^
--db-instance-identifier mydbinstance ^
--backup-retention-period 7 ^
--auto-minor-version-upgrade ^
--apply-immediately
aws rds download-db-log-file-portion --db-instance-identifier mydbinstance --log-file-name trace/sqlnet-parameters --output text
aws rds modify-db-instance \
--db-instance-identifier <oradbinstance> \
--engine-version <12.1.0.2.v10> \
--option-group-name <default:oracle-ee-12-1> \
--db-parameter-group-name <default.oracle-ee-12.1> \
--allow-major-version-upgrade \
--no-apply-immediately
aws rds modify-db-snapshot --db-snapshot-identifier <mydbsnapshot> --engine-version <11.2.0.4.v12> --option-group-name <default:oracle-se1-11-2>

Related Articles: AWS Certified Cloud Practitioner Practice Exam

Google Associate Cloud Engineer Certification Practice Exam

2021-03-24T19:41:00.000+05:30

GCP Associate Cloud Engineer (ACE) Certification - Sample Questions

Related Articles: Google Cloud CLI commands

Difference between NACL and Security Groups in AWS

2021-02-04T13:09:00.000+05:30

What are the differences between Security Groups and NACL in Amazon Web Services ?

Security Groups (SG)
Security groups, act as a firewall for associated Amazon EC2 instances, controlling both inbound and outbound traffic at the instance level.

Security groups can also be used by AWS services such as Amazon RDS, Amazon Redshift, Amazon EMR, and Amazon ElastiCache.

By default, Security Groups allow only outbound traffic and block all incoming traffic. To enable inbound traffic, we have to specify the protocol, port and source.

VPC is like a house and subnets are like rooms, security group is like (fire)wall.

Network Access Controls Lists (NACL)
Network access controls lists (ACLs) act as a firewall for associated subnets, controlling both inbound and outbound traffic at the subnet level.

NACL is at subnet level.

In a VPC, the default NACL is set to allow all inbound and outbound traffic. If you deploy a custom NACL, then all inbound and outbound traffic is blocked.

Security Groups control traffic flow at the instance level whereas NACLs control traffic at the subnet level.

Security Groups can thus be used to enable traffic flow between different types of instances within a subnet itself which NACLs cannot control.

NACL and Security Groups work in collaboration with each other to offer a more complete security setup.

Differences between Security Groups and Network ACL

Security Groups	NACL
Security groups are stateful (changes applied to an incoming rule will be automatically applied to the outgoing rule)	NACLs are stateless (changes applied to an incoming rule will not be applied to the outgoing rule)
By default, Security Groups allow only outbound traffic and block all incoming traffic	By default, NACL is set to allow all inbound and outbound traffic
Security groups are tied to an instance	Network ACLs are tied to the subnet
Firewall of EC2 Instances	Firewall of Subnet
Security group support ALLOW rules only (by default all rules are denied)	Network ACL support ALLOW and DENY rules
All rules in a security group are applied	NACLs apply rules in number/priority order, from top to bottom
Applies to an instance only if specified the security group	Automatically applies to all instances in the subnets it's associated with
Can not block specific IP address	Can block specific IP address

Related AWS Articles: AWS Certified Cloud Practitioner Practice Exam 2

AWS Cloud Practitioner Exam 1

2021-02-01T11:34:00.000+05:30

AWS Certified Cloud Practitioner Practice Exam 1

Loading...

Related Articles: AWS Certified Cloud Practitioner Practice Exam 2

AWS Certified Cloud Practitioner Practice Exam 2

2020-11-26T13:50:00.000+05:30

AWS Certified Cloud Practitioner Practice Exam 2

Loading...

Related Articles: AWS Certified Cloud Practitioner Practice Exam 1

Terraform commands

2020-08-10T14:16:00.000+05:30

Terraform Command Line Interface (Terraform CLI)

$ terraform
Usage: terraform [--version] [--help] <command> [args]

Common commands:
apply Builds or changes infrastructure
console Interactive console for Terraform interpolations
destroy Destroy Terraform-managed infrastructure
fmt Rewrites config files to canonical format
get Download and install modules for the configuration
graph Create a visual graph of Terraform resources
import Import existing infrastructure into Terraform
init Initialize a new or existing Terraform configuration
output Read an output from a state file
plan Generate and show an execution plan
providers Prints a tree of the providers used in the configuration
push Upload this Terraform module to Terraform Enterprise to run
refresh Update local state file against real resources
show Inspect Terraform state or plan
taint Manually mark a resource for recreation
untaint Manually unmark a resource as tainted
validate Validates the Terraform files
version Prints the Terraform version
workspace Workspace management

All other commands:
debug Debug output management (experimental)
force-unlock Manually unlock the terraform state
state Advanced state management

terraform -version
terraform taint -help

terraform init # will install provider plugins
terraform init -upgrade commands_exec/
terraform get

terraform plan
terraform plan -out=demo.tfplan
terraform plan --destroy -out destroy-plan
terraform apply
terraform apply demo.tfplan
terraform apply -var 'zones=["us-east-1b", "us-east-1d"]'
terraform apply -var-file=sample.tfvars
terraform destroy
terraform destroy -target=aws_security_group.elb_sg

terraform workspace list
terraform workspace new dev
terraform workspace select
$ terraform import aws_instance.example i-abcd1234 # import existing infrastructure
$ terraform import aws_key_pair.deployer deployer-key
terraform import aws_vpc.web_vpc vpc-25b7495d

terraform refresh
terraform show # to show the attributes in the state file
terraform state list
terraform state show aws_vpc.web_vpc
terraform graph

terraform output
terraform output ips_var
terraform output -module=network
site_address=$(terraform output site_address)
web_server_ip=$(terraform output -module=vm | cut -d " " -f 3)

terraform console
echo "2 + 8" | terraform console
terraform providers
terraform fmt
terraform fmt -check
terraform validate # performs syntax checks

Google Cloud CLI commands

2020-03-17T20:20:00.000+05:30

gcloud commands

gcloud help
gcloud help compute instances create
gcloud -h
gcloud --help

gcloud info

gcloud init
gcloud init --console-only

gcloud config set
gcloud config set compute/zone asia-south1-c
gcloud config set compute/region us-central1
gcloud config configurations create

gcloud config list
gcloud config list project
gcloud config list --all

gcloud auth list # list the active account name
gcloud auth list --limit=1
gcloud auth configure-docker

gcloud projects list
gcloud projects delete

gcloud iam service-accounts list
gcloud iam service-accounts create proxy-user --display-name "[SERVICE_ACCOUNT_NAME]"
gcloud iam service-accounts keys create key.json --iam-account [SERVICE_ACCOUNT_EMAIL]

gcloud compute
gcloud compute regions list

gcloud compute instances list
gcloud compute instances list --filter='labels.production:fin'
gcloud compute instances
gcloud compute instances create --help
gcloud compute instances create test-instance
gcloud compute instances create vm2 --machine-type n1-standard-2 --zone asia-south1-c
gcloud compute instances create celab2 --machine-type n1-standard-2 --zone $ZONE
gcloud compute instances describe <vm-name>

gcloud compute instance-templates create nginx-template --metadata-from-file startup-script=startup.sh
gcloud compute target-pools create nginx-pool
gcloud compute instance-groups managed create nginx-group --base-instance-name nginx --size 2 --template nginx-template --target-pool nginx-pool
gcloud compute instance-groups managed set-named-ports nginx-group --named-ports http:80

gcloud compute disks create mydisk --size=200GB --zone us-central1-c
gcloud compute instances attach-disk celab --disk mydisk --zone us-central1-c

gcloud compute ssh celab2 --zone [YOUR_ZONE]
gcloud compute ssh instance-1 --zone asia-south1-c

gcloud compute addresses list # to list Public IPs

gcloud compute firewall-rules list
gcloud compute firewall-rules create www-firewall --allow tcp:80

gcloud compute routes list

gcloud compute forwarding-rules list
gcloud compute forwarding-rules create nginx-lb --region us-central1 --ports=80 --target-pool nginx-pool
gcloud compute forwarding-rules create http-content-rule --global --target-http-proxy http-lb-proxy --ports 80

gcloud compute http-health-checks create http-basic-check

gcloud compute backend-services create nginx-backend --protocol HTTP --http-health-checks http-basic-check --global
gcloud compute backend-services add-backend nginx-backend --instance-group nginx-group --instance-group-zone us-central1-a --global
gcloud compute url-maps create web-map --default-service nginx-backend
gcloud compute target-http-proxies create http-lb-proxy --url-map web-map

gcloud compute project-info describe --project <project_ID>
gcloud compute project-info describe --project labs-gcp

gcloud components list
gcloud components install beta
gcloud components install kubectl
gcloud components remove cbt
gcloud components update

gcloud app deploy

gcloud beta interactive

gcloud functions delete get --trigger-http
gcloud functions deploy kgSearch --runtime nodejs8 --trigger-http
gcloud functions logs read --limit 100
gcloud functions delete kgSearch

gcloud sql instances list
gcloud sql instances describe [INSTANCE_NAME]| grep connectionName
gcloud sql connect demo
gcloud sql connect labs-demo --user=root

gcloud spanner databases create <db_id> --instance=<instance_id>
gcloud spanner databases execute-sql <db_id> --instance=<instance_id> --sql='select * from table where key=1'
gcloud spanner databases list --instance=<instance_id>
gcloud spanner instances list
gcloud spanner instances delete test-instance

gcloud container clusters create [CLUSTER-NAME]
gcloud container clusters create satya-ke-cluster
gcloud container clusters get-credentials [CLUSTER-NAME]
gcloud container clusters get-credentials satya-ke-cluster
gcloud container clusters delete [CLUSTER-NAME]
gcloud container clusters delete my-cluster

gcloud pubsub subscriptions pull --auto-ack MySub

gcloud ai-platform local train \
--module-name trainer.task \
--package-path trainer/ \
--job-dir $MODEL_DIR \
--train-files $TRAIN_DATA \
--eval-files $EVAL_DATA \
--train-steps 1000 \
--eval-steps 100
gcloud ai-platform local predict --model-dir output/export/census/<timestamp> --json-instances ../test.json
gcloud ai-platform jobs stream-logs $JOB_NAME
gcloud ai-platform models create $MODEL_NAME --regions=$REGION
gcloud ai-platform models list

gcloud services enable cloudkms.googleapis.com
gcloud kms keyrings create $KEYRING_NAME --location global
gcloud kms keyrings add-iam-policy-binding $KEYRING_NAME --location global --member user:$USER_EMAIL --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Apache Hadoop Terms/Abbreviations

2019-08-28T13:25:00.000+05:30

HDFS - Hadoop Distributed File System
GFS - Google File System
JSON - Java Script Object Notation

NN - NameNode
DN - Data Node
SNN - Secondary NameNode
JT - Job Tracker
TT - Task Tracker
HA NN - Highly Available NameNode (or NN HA - NameNode Highly Available)

REST - Representational State Transfer
HiveQL or HQL - Hive Query Langauge or Hive SQL
CDH - Cloudera’s Distribution Including Apache Hadoop
ZKFC - ZooKeeper Failover Controller
HVE - Hadoop Virtual Extensions
FUSE - Filesystem In Userspace
YARN - Yet Another Resource Negotiator
SerDe - Serialiser and Deserialiser
Hue - Hadoop User Experience

JRE - Java Runtime Engine
JNA - Java Native Access
JVM - Java Virtual Machine
JMX - Java Management Extensions
JAR - Java ARchive

AWS - Amazon Web Services
S3 - Simple Storage Service
EC2 - Elastic Compute Cloud
VPC - Virtual Private Cloud
EMR - Elastic Map Reduce
IAM - Identity Access Management
RDS - Relational Database Service

GCP - Google Cloud Platform
GCE - Google Compute Engine

User Defined Functions (UDFs)
User Defined Aggregates Functions (UDAFs)
User Defined Table Functions (UDTFs)

Related Hadoop Articles: Apache Hadoop Hadoop Commercial Distributions

Apache Hadoop Versions

2019-04-20T15:04:00.000+05:30

Hadoop Versions

Hadoop 3
6 Apr 2018: Release 3.1.0 available
25 March 2018: Release 3.0.1 available
13 December 2017: Release 3.0.0 generally available
03 October 2017: Release 3.0.0-beta1 available
07 July 2017: Release 3.0.0-alpha4 available
26 May 2017: Release 3.0.0-alpha3 available
25 January, 2017: Release 3.0.0-alpha2 available
03 September, 2016: Release 3.0.0-alpha1 available

Hadoop 2
14 December, 2017: Release 2.7.5 available
12 December 2017: Release 2.8.3 available
17 November 2017: Release 2.9.0 available
24 October 2017: Release 2.8.2 available
04 August, 2017: Release 2.7.4 available
08 June, 2017: Release 2.8.1 available
22 March 2017: Release 2.8.0 available
08 October, 2016: Release 2.6.5 available
25 August, 2016: Release 2.7.3 available
11 February, 2016: Release 2.6.4 available
25 January, 2016: Release 2.7.2 (stable) available
17 December, 2015: Release 2.6.3 available
28 October, 2015: Release 2.6.2 available
23 September, 2015: Release 2.6.1 available
06 July, 2015: Release 2.7.1 (stable) available
21 April 2015: Release 2.7.0 available
18 November, 2014: Release 2.6.0 available
19 November, 2014: Release 2.5.2 available
12 September, 2014: Release 2.5.1 available
11 August, 2014: Release 2.5.0 available
30 June, 2014: Release 2.4.1 available
07 April, 2014: Release 2.4.0 available
20 February, 2014: Release 2.3.0 available
15 October, 2013: Release 2.2.0 available
23 September, 2013: Release 2.1.1-beta available
25 August, 2013: Release 2.1.0-beta available
23 August, 2013: Release 2.0.6-alpha available
6 June, 2013: Release 2.0.5-alpha available
25 April, 2013: Release 2.0.4-alpha available
Hadoop 2.0.3-alpha (released on 14 February, 2013)
Hadoop 2.0.2-alpha (released on 9 October, 2012)
Hadoop 2.0.1-alpha (released on 26 July, 2012)
Hadoop 2.0.0-alpha (released on 23 May, 2012)

Hadoop 1
1 Aug, 2013: Release 1.2.1 (stable) available
13 May, 2013: Release 1.2.0 available
Hadoop 1.1.2 (released on 15 February, 2013)
Hadoop 1.1.1 (released on 1 December, 2012)
Hadoop 1.1.0 (released on 13 October, 2012)
Hadoop 1.0.4 (released on 12 October, 2012)
Hadoop 1.0.3 (released on 16 May, 2012)
Hadoop 1.0.2 (released on 3 Apr, 2012)
Hadoop 1.0.1 (released on 10 Mar, 2012)
Hadoop 1.0.0 (released on 27 December, 2011)

Hadoop 0
27 June, 2014: Release 0.23.11 available
11 December, 2013: Release 0.23.10 available
8 July, 2013: Release 0.23.9 available
5 June, 2013: Release 0.23.8 available
18 April, 2013: Release 0.23.7 available
Hadoop 0.23.6 (released on 7 February, 2013)
Hadoop 0.23.5 (released on 28 November, 2012)
Hadoop 0.23.4 (released on 15 October, 2012)
Hadoop 0.23.3 (released on 17 September, 2012)
Hadoop 0.23.1 (released on 27 Feb, 2012)
Hadoop 0.22.0 (released on 10 December, 2011)
Hadoop 0.23.0 (released on 11 Nov, 2011)
Hadoop 0.20.205.0 (released on 17 Oct, 2011)
Hadoop 0.20.204.0 (released on 5 Sep, 2011)
Hadoop 0.20.203.0 (released on 11 May, 2011)
Hadoop 0.21.0 (released on 23 August, 2010)
Hadoop 0.20.2 (released on 26 February, 2010)
Hadoop 0.20.1 (released on 14 September, 2009)
Hadoop 0.19.2 (released on 23 July, 2009)
Hadoop 0.20.0 (released on 22 April, 2009)
Hadoop 0.19.1 (released on 24 February, 2009)
Hadoop 0.18.3 (released on 29 January, 2009)
Hadoop 0.19.0 (released on 21 November, 2008)
Hadoop 0.18.2 (released on 3 November, 2008)
Hadoop 0.18.1 (released on 17 September, 2008)
Hadoop 0.18.0 (released on 22 August, 2008)
Hadoop 0.17.2 (released on 19 August, 2008)
Hadoop 0.17.1 (released on 23 June, 2008)
Hadoop 0.17.0 (released on 20 May, 2008)
Hadoop 0.16.4 (released on 5 May, 2008)
Hadoop 0.16.3 (released on 16 April, 2008)
Hadoop 0.16.2 (released on 2 April, 2008)
Hadoop 0.16.1 (released on 13 March, 2008)
Hadoop 0.16.0 (released on 7 February, 2008)
Hadoop 0.15.3 (released on 18 January, 2008)
Hadoop 0.15.2 (released on 2 January, 2008)
Hadoop 0.15.1 (released on 27 November, 2007)
Hadoop 0.14.4 (released on 26 November, 2007)
Hadoop 0.15.0 (released on 29 October 2007)
Hadoop 0.14.3 (released on 19 October, 2007)
Hadoop 0.14.1 (released on 4 September, 2007)

Related Hadoop Articles: Hadoop Commercial Distributions Hadoop Commands

Hadoop Commands

2019-04-02T16:39:00.000+05:30

Hadoop CLI Commands

hadoop command [genericOptions] [commandOptions]

hadoop fs
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] <path> ...]
[-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] [-x] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getfattr [-R] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
[-help [cmd ...]]
[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]

hadoop fs -ls / -- Get a directory listing of the HDFS root directory
hadoop fs -ls /user/hive
hadoop fs -ls s3a://satya-hive/trip_data_dec17
hadoop fs -cat /test1/foo.txt -- Display contents of the file residing in HDFS
hadoop fs -cat /test1/foo.txt | wc -l -- Count the number of lines of file in HDFS
hadoop fs -rm /test1/foo.txt -- Remove the file in HDFS
hadoop fs -rm -r hadoop-test2
hadoop fs -mkdir /test6/ -- Create a directory under the HDFS root directory
hadoop fs -put foo.txt /test6/ -- Copy file foo.txt from local disk to the directory
hadoop fs -put /etc/note.txt /test2/note_fs.txt
hadoop fs -get /user/satya/passwd ./ -- Copy the file back to local disk
hadoop fs -cp hadoop-test1/dwp-payments-april10.csv hadoop-test2
hadoop fs -cp /user/satya/reviews_Home_and_Kitchen_5.json s3a://satya-sparks/reviews_HomeKitchen
hadoop fs -Ddfs.replication=2 -cp hadoop-test2/dwp-payments-april10.csv hadoop-test2/test_with_rep2.csv
hadoop fs -mv hadoop-test1/dwp-payments-april10.csv hadoop-test3
hadoop fs -setrep 5 -R /user/satya/tmp/
hadoop fs -chmod 1777 /tmp
hadoop fs -touch /user/satya/test/foo
hadoop fs -rmr /user/satya/test/foo
hadoop fs -touchz /user/satya/test/bar
hadoop fs -count -q /user/satya
hadoop fs -copyFromLocal /hirw-starterkit/hdfs/commands/dwp-payments-april10.csv hadoop-test1
hadoop fs -copyToLocal hadoop-test1/dwp-payments-april10.csv .
hadoop -execute start-all.sh

hadoop job -list
hadoop job -kill jobID
hadoop job -list-attempt-ids jobID taskType taskState
hadoop job -kill-task taskAttemptId

hadoop namenode -format

hadoop jar <jar_file> wordcount <output_file>
hadoop jar /opt/hadoop/hadoop-examples-1.0.4.jar wordcount /out/wc_output

hadoop dfsadmin -report
hadoop dfsadmin -setSpaceQuota 10737418240 /user/esammer
hadoop dfsadmin -refreshNodes
hadoop dfsadmin -upgradeProgress status
hadoop dfsadmin -finalizeUpgrade

hadoop fsck
Usage: DFSck <path> [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]
<path> -- start checking from this path
-move -- move corrupted files to /lost+found
-delete -- delete corrupted files
-files -- print out files being checked
-openforwrite -- print out files opened for write
-blocks -- print out block report
-locations -- print out locations for every block
-racks -- print out network topology for data-node locations
By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status.
hadoop fsck / -files -blocks -locations
hadoop fsck /user/satya -files -blocks -locations

hadoop distcp -- Distributed Copy (distcp)
distcp [OPTIONS] <srcurl>* <desturl>
OPTIONS:
-p[rbugp] Preserve status
r: replication number
b: block size
u: user
g: group
p: permission
-p alone is equivalent to -prbugp
-i Ignore failures
-log <logdir> Write logs to <logdir>
-m <num_maps> Maximum number of simultaneous copies
-overwrite Overwrite destination
-update Overwrite if src size different from dst size
-skipcrccheck Do not use CRC check to determine if src is different from dest. Relevant only if -update is specified
-f <urilist_uri> Use list at <urilist_uri> as src list
-filelimit <n> Limit the total number of files to be <= n
-sizelimit <n> Limit the total size to be <= n bytes
-delete Delete the files existing in the dst but not in src
-mapredSslConf <f> Filename of SSL configuration for mapper task

NOTE 1: if -overwrite or -update are set, each source URI is interpreted as an isomorphic update to an existing directory.
For example:
hadoop distcp -p -update "hdfs://A:8020/user/foo/bar" "hdfs://B:8020/user/foo/baz"
would update all descendants of 'baz' also in 'bar'; it would *not* update /user/foo/baz/bar

NOTE 2: The parameter <n> in -filelimit and -sizelimit can be specified with symbolic representation. For examples,
1230k = 1230 * 1024 = 1259520
891g = 891 * 1024^3 = 956703965184

hadoop distcp hdfs://A:8020/path/one hdfs://B:8020/path/two
hadoop distcp /path/one /path/two

Related Hadoop Articles: Apache Hadoop Hadoop Training in India (Bangalore/Hyderabad)

Hadoop Training Institutes in India

2018-10-07T17:37:00.000+05:30

Apache Hadoop Online/Offline Training Institutes in India (Bangalore/Hyderabad)

Many institutes/companies are providing Hadoop training in India. And it's available like any other course (like Java, .NET) in Ameerpet (Hyderabad), but it's costly.

BigData/Hadoop Training in Hyderabad:

I had personally enquiried all the below Hyderabad Hadoop training institutes.
Big Data/Hadoop training course fees - Cost will be around Rs. 12,000/- to Rs. 18,000/-

Hadoop Training Institutes @ Hyderabad	Contact Numbers	Website/Mail
Aswin DWH Technologies, Ameerpet, Hyderabad	040 65226522, 9295555929	www.aswindwhtechnologies.com
Ctrl-A Technologes, Ameerpet, Hyderabad	040 40142647, 9505675670
eNexus Software Technologies, SR Nagar, Hyderabad	040 64622229, 8096374447
FIFO Technologies, SR Nagar, Hyderabad, India	040 6565 6090, 08121389698	www.fifotechnologies.com
iFocus IT Solutions, Ameerpet, Hyderabad	040 65444455, 9849388104
Inuomsoft Pvt. Ltd, Ameerpet, Hyderabad	040 66633775, 9642101010
Kelly Technologies, Ameerpet, Hyderabad	040 6462 6789, 9985706789	www.kellytechno.com
Mind Links IT R&D Labs, Ameerpet, Hyderabad	040 64568575, 9951081013	www.themindlinks.com
Nihitha IT, Ameerpet, Hyderabad, India	040 66413341, 9000415176
Sree Aditya Software Solutions, Ameerpet, Hyderabad	040 60606011	www.sreeaaditya.com
Sree Nipuna Software Solutions, Ameerpet, Hyderabad	040 66849695, 9247052622
Surya Software Solutions, Ameerpet, Hyderabad	9966661188, 8686536306
Version IT, Ameerpet, Hyderabad, India	040 66738677, 040 66738766
Visual Path, Ameerpet, Hyderabad	9618245689, 9704455959	www.visualpath.in
Orient IT, Ameerpet, Hyderabad	,

BigData/Hadoop Training in Bangalore:

Hadoop Training Institutes @ Bengaluru	Contact Numbers	Website/Mail
A.P. Software Solutions, Madivala, Bangalore	080 - 32021688, 9844621700	enquiry@apsoftsolutions.com
Ahana Systems & Solutions, Hanumantha Nagar, Bangalore	080 - 26675891, 9686113314	www.ahana.co.in
ANOVA IT SERVICES, Bengaluru, India	7411120120, 7411121121	www.techinformatic.com
Astrid Solutions, Jaya Nagar, Bangalore	080 - 41100370, 64520268, 8123673106	www.astridsolutions.in
Avaram Technologies, Bengaluru, India	7829415003, 07829415003	www.AvaramTechnologies.com
Business Intelligence Solutions Provider	7552410354, 9977997254	www.bispsolutions.com
Cloud Soft Solutions, Marathahalli, Bangalore	080 - 65659992, 9916224915	www.cloudsoftsol.com
CMS Computer Institute, Rajaji Nagar, Bangalore	080 - 65681433, 080 - 64557745, 8861421441	www.cmsinstitute.co.in
Colossal Software Technologies	080 40636168, 9741913113	www.verticaldivers.com
Dolphin Institute of Computers, Konanakunte, Bangalore	080 - 31904040
e-Care technologies, Bangalore, India	9845642721, 9844752189	www.ecaretechnologies.info
eCloud Solutions, Bangalore, India	9845988299	www.ecloudsol.com
Elegant IT Services, Marathahalli, Bangalore	080 - 41523314, 9886541264	www.elegantitservices.com
Ematic Technologies, Bangalore, India	8971571117,080-65316888	www.ematictechnologies.com
IGROW TECHNOLOGIES, Bangalore, India	080-41724999, 9738025666	www.igrowtechnologies.com
ISAC Software Academy	080 4093 3246	www.isacsoftware.com
KnowledgeWorks IT Consulting Pvt. Ltd.	80 26630622, 22459941, 9886221314	www.knowledgeworks.co.in
Lorven IT Pvt. Ltd., BTM Layout 1st Stage, Bangalore	9611205205	www.lorvenit.net
MeritForge, Bengaluru, India	080 2529 7815, 9611778865	www.meritforge.com
NovelVista IT Learning Solutions	8411050011, 09422540480	www.novelvista.com
QSIT, Bengaluru, India	080 4081 8888	www.qsitglobal.com
RGS IT Solutions, Jaya Nagar, Bangalore	080 - 41307782, 9663914467	suresh9135@gmail.com
RK Software Solutions, Madivala, Bangalore	080 - 32401030, 9740609586	rksoftwaresolutions11@gmail.com
Rrootshell Technologies Pvt. Ltd., BTM 2nd Stage, Bangalore	080 - 32972711, 080 - 26789818, 8105845978	www.rrootshell.com
Sancentre IT Solutions Pvt Ltd, Bengaluru, India	9611836161	www.sancentre.com
SAPTEQ GLOBAL CONSULTING SERVICES, Bengaluru, India	8711046553, 9535160010	WWW.SAPTEQ.COM
Shine Softwares, Bengaluru, India	9739231171, 9739231181	www.shinesoftwares.com
Simplilearn Solutions Pvt. Ltd., Bengaluru, India	080 6547 6230	www.simplilearn.com
Sure Step Solutions Pvt Ltd, Bengaluru, India	080-26721156	www.surestep.co.in
Techbricks software training center, Bengaluru, India	8065467957, 9243022443
Trendwise Analytics, White Field, Bangalore	080 - 40949600, 9845815003, 9886768879	www.trendwiseanalytics.com
United Global Soft-Online Training Institute	080-66444122, 8099902123	www.unitedglobalsoft.com
Varnaaz Academy, Banashankari 2nd Stage, Bangalore	9845562620	www.varnaaz.com
cloudwick technologies, Bangalore	xxxxx	www.cloudwick.com

Related Hadoop Articles: Apache Hadoop Hadoop Commands

Hadoop Certifications

2018-04-15T12:28:00.000+05:30

Hadoop Certifications Cloudera/Hortonworks/IBM

Cloudera - Cloudera University

CCAH (Administrator) Exams
Cloudera Certified Administrator for Apache Hadoop (CCA-410)
(or)
Cloudera Certified Administrator for Apache Hadoop CDH3 (CCA-332) -- Not available
Cloudera Certified Administrator for Apache Hadoop CDH4 Upgrade Exam (CCA-470) -- Aplicable only if u have cleared CCA-332

CCDH (Developer) Exams
Cloudera Certified Developer for Apache Hadoop (CCD-410)
(or)
Cloudera Certified Developer for Apache Hadoop CDH3 (CCD-333) -- Not available
Cloudera Certified Developer for Apache Hadoop CDH4 Upgrade Exam (CCD-470) -- Aplicable only if u have cleared CCD-333

Test Name: Cloudera Certified Administrator for Apache Hadoop CDH4 (CCA-410)
Number of Questions: 60
Time Limit: 90 minutes
Passing Score: 70%
Languages: English, Japanese
Price: USD $295, AUD285, EUR225, GBP185, JPY25,500

Test Name: Cloudera Certified Developer for Apache Hadoop CDH4 (CCD-410)
Number of Questions: 60
Time Limit: 90 minutes
Passing Score: 67%
Languages: English, Japanese
Price: USD$295, AUD285, EUR225, GBP185, JPY25,500

HortonWorks - Hortonworks University

Hortonworks Certified Apache Hadoop Administrator (HCAHA) - HWX-0011
Hortonworks Certified Apache Hadoop Developer (HCAHD) - HWX-0012
Cost: $150.00
Passing score: 75%

IBM - BigData University

BD001EN - Hadoop Fundamentals I
BD005EN - Hadoop & Amazon Cloud
BD006EN - Hadoop & IBM Cloud

Related Hadoop Articles: Hadoop Training Hadoop Versions

What is Spark ? Why Apache Spark?

2018-02-28T14:10:00.000+05:30

Spark is one of Hadoop’s sub project developed in 2009 in UC Berkeley’s AMPLab by Matei Zaharia, as an alternative to the MapReduce paradigm.

It was Open Sourced in 2010 under a BSD license.

It was donated to Apache software foundation in 2013.

Now Apache Spark has become a top level Apache project from Feb-2014.

Spark is known mainly for its high-speed cluster computing capabilities and support for programming languages such as Python, R, Java, SQL, Scala etc.

Spark has the capability to connect to a variety of data sources like S3, Cassandra, HDFS, HBase, Avro, Parquet, etc.

Related Articles: Spark Interview Questions and Answers

Hadoop Distributions

2017-04-19T13:23:00.000+05:30

Below are the companies offering commercial implementations and/or providing support for Apache Hadoop, which is the base for all the below.

Cloudera offers CDH (Cloudera's Distribution including Apache Hadoop) and Cloudera Enterprise.
Hortonworks (formed by Yahoo and Benchmark Capital), whose focus is on making Hadoop more robust and easier to install, manage and use for enterprise users. Hortonworks provides Hortonworks Data Platform (HDP).
MapR Technologies offers distributed filesystem and MapReduce engine, the MapR Distribution for Apache Hadoop.
Oracle announced the Big Data Appliance, which integrates Cloudera's Distribution Including Apache Hadoop (CDH).
IBM offers InfoSphere BigInsights based on Hadoop in both a basic and enterprise edition.
Greenplum, A Division of EMC, offers Hadoop in Community and Enterprise editions.
Intel - the Intel Distribution for Apache Hadoop is the product includes the Intel Manager for Apache Hadoop for managing a cluster.
Amazon Web Services - Amazon offers a version of Apache Hadoop on their EC2 infrastructure, sold as Amazon Elastic MapReduce.
VMware - Initiate Open Source project and product to enable easily and efficiently deploy and use Hadoop on virtual infrastructure.
Bigtop - project for the development of packaging and tests of the Apache Hadoop ecosystem.
DataStax - DataStax provides a product of Hadoop which fully integrates Apache Hadoop with Apache Cassandra and Apache Solr in its DataStax Enterprise platform.
Cascading - A popular feature-rich API for defining and executing complex and fault tolerant data processing workflows on a Apache Hadoop cluster.
Mahout - Apache project using Hadoop to build scalable machine learning algorithms like canopy clustering, k-means and many more.
Cloudspace - uses Apache Hadoop to scale client and internal projects on Amazon's EC2 and bare metal architectures.
Datameer - Datameer Analytics Solution (DAS) is a Hadoop-based solution for big data analytics that includes data source integration, storage, an analytics engine and visualization.
Data Mine Lab - Developing solutions based on Hadoop, Mahout, HBase and Amazon Web Services.
Debian - A Debian package of Apache Hadoop is available.
HStreaming - offers real-time stream processing and continuous advanced analytics built into Hadoop, available as free community edition, enterprise edition, and cloud service.
Impetus
Karmasphere - Distributes Karmasphere Studio for Hadoop, which allows cross-version development and management of Apache Hadoop jobs.
Nutch - Apache Nutch, flexible web search engine software.
NGDATA - Makes available Lily Open Source that builds upon Hadoop, HBase and SOLR. Distributes Lily Enterprise.
Pentaho – Pentaho provides a complete, end-to-end open-source BI and offers an easy-to-use, graphical ETL tool that is integrated with Apache Hadoop for managing data and coordinating Hadoop related tasks in the broader context of ETL and Business Intelligence workflow.
Pervasive Software - Provides Pervasive DataRush, a parallel dataflow framework which improves performance of Apache Hadoop and MapReduce jobs by exploiting fine-grained parallelism on multicore servers.
Platform Computing - Provides an Enterprise Class MapReduce solution for Big Data Analytics with high scalability and fault tolerance. Platform MapReduce provides unique scheduling capabilities and its architecture is based on almost two decades of distributed computing research and development.
Sematext International - Provides consulting services around Apache Hadoop and Apache HBase, along with large-scale search using Apache Lucene, Apache Solr, and Elastic Search.
Talend - Talend Platform for Big Data includes support and management tools for all the major Apache Hadoop distributions. Talend Open Studio for Big Data is an Apache License Eclipse IDE, which provides a set of graphical components for HDFS, HBase, Pig, Sqoop and Hive.
Think Big Analytics - Offers expert consulting services specializing in Apache Hadoop, MapReduce and related data processing architectures.
Tresata - Financial Industry's first software platform architected from the ground up on Hadoop. Data storage, processing, analytics and visualization all done on Hadoop.
WANdisco is a committed member & sponsor of the Apache Software community and has active committers on several projects including Apache Hadoop.

Related Hadoop Articles: Apache Hadoop Commands Hadoop Training in India

What is Hadoop?

2017-03-25T17:13:00.000+05:30

Apache Hadoop is, an open-source software framework, written in Java, by Doug Cutting and Michael J. Cafarella, that supports data-intensive distributed applications, licensed under the Apache v2 license. It supports the running of applications on large clusters of commodity hardware. Hadoop was derived from Google's MapReduce and Google File System (GFS) papers.

The Hadoop framework transparently provides both reliability and data motion to applications. Hadoop implements a computational paradigm named MapReduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster. It provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster.

Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework. It enables applications to work with thousands of computation-independent computers and petabytes of data.

The entire Apache Hadoop platform is commonly considered to consist of the Hadoop kernel, MapReduce and Hadoop Distributed File System (HDFS), and number of related projects including Apache Hive, Apache HBase, Apache Pig, Zookeeper etc.

Related Articles: NoSQL Databases What is Apache Cassandra