Pin Your Interest

mayankthapa1994@gmail.com (Mayank) — Wed, 25 Jan 2017 12:31:00 +0000

Which IT Job Is the Best?

All computer science jobs are not created equal. That degree might qualify you for a number of different jobs in technology. But how do you know which one is the best for you?

Software Developer/Computer Programmer

What you’ll do

Research, design, implement and test software and systems

How much you’ll work

Generally, a 40-hour workweek, though you should expect long hours when project deadlines approach

Where you’ll work

Office, much of time spent alone writing code

What you’ll make

Median salary, $92,660

Biggest benefits

Job is in high demand, as the world increasingly relies on technology

Biggest drawbacks

Job is generally low-stress, though project deadlines can become intense

Database Administrator

What you’ll do

Design, code and maintain database systems to store and manage data

How much you’ll work

Generally, a 40-hour workweek, though project deadlines and after-hours crises can spring up

Where you’ll work

Office, generally for large organizations

What you’ll make

Median salary, $77,080

Biggest benefits

Generally have ability to work with lots of different types of people and of projects

Biggest drawbacks

Risk of database failure can make for high-stress job

Hardware Engineer

What you’ll do

Configure hardware of computers; improve layout and design of devices for increased efficiency

How much you’ll work

Generally, a 40-hour workweek, though project deadlines and crises can lead to night and weekend work

Where you’ll work

Factory, research facility or lab

What you’ll make

Median salary, $100,920

Biggest benefits

Typically get to work with newest technology

Biggest drawbacks

Narrow, project-based focus can make for monotonous days

Systems Analyst

What you’ll do

Design IT solutions; adapt existing systems to improve business efficiency and productivity

How much you’ll work

Generally, 40-hour workweek, though extended hours aren’t uncommon, particularly at end of projects or for consultants working with clients

Where you’ll work

Office; freelancers and consultants travel to client offices

What you’ll make

Median salary, $79,680

Biggest benefits

Good deal of flexibility to work on variety of projects

Biggest drawbacks

Can be high-stress when systems fail

Network Architect

What you’ll do

Design, build and test communication networks

How much you’ll work

Generally, 40-hour work week

Where you’ll work

Office, but often must work on-site at different buildings and offices

What you’ll make

Median salary, $91,000

Biggest benefits

Likely to see increase in job openings as companies expand use of wireless and mobile technology

Biggest drawbacks

Dozens of possible certifications can be daunting

Web developer

What you’ll do

Use software languages and other tools to create, implement and test Web applications

How much you’ll work

Full time, but hours vary widely; a quarter of Web developers are self-employed

Where you’ll work

Good chance of working from home; otherwise, office setting

What you’ll make

Median salary, $62,500

Biggest benefits

Flexible schedule

Biggest drawbacks

Project deadlines can create stress

Information Security Analyst

What you’ll do

Develop and implement security measures to protect organizations’ computer networks and systems

How much you’ll work

40-hour workweek is typical, though after-hours crises are not uncommon

Where you’ll work

Office, most likely for large company

What you’ll make

Median salary, $86,170

Biggest benefits

Job security thanks to ever-increasing cyber threats

Biggest drawbacks

Must keep on top of constantly changing attack methods

Systems Manager

What you’ll do

Plan, coordinate and direct technology-related activities in organizations

How much you’ll work

Generally, 40-hour workweek, though overtime is common

Where you’ll work

Office, for small or large companies; limited ability to work from home as manager

What you’ll make

Median salary, $120,950

Biggest benefits

Chance to influence entire company

Biggest drawbacks

After-hours crises common

IT Support

What you’ll do

Provide technical help and support to organizations

How much you’ll work

Most likely full-time, but not necessarily 9-to-5

Where you’ll work

Call center, could even work from home

What you’ll make

Median salary, $48,900

Biggest benefits

Flexible schedule

Biggest drawbacks

Must be constantly available for users

Introduction To R Language

mayankthapa1994@gmail.com (Mayank) — Tue, 07 Jun 2016 06:13:00 +0000

INTRODUCTION TO R

R is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R’s strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formula where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

R is available as Free Software under the terms of the Free Software Foundation’s GNU General Public License in source code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows and Mac OS.

The R environment

R is an integrated suite of software facilities for data manipulation, calculation and graphical display. It includes

an effective data handling and storage facility,

a suite of operators for calculations on arrays, in particular matrices,

a large, coherent, integrated collection of intermediate tools for data analysis,

graphical facilities for data analysis and display either on-screen or on hard copy, and

a well-developed, simple and effective programming language which includes conditionals, loops, user-defined recursive functions and input and output facilities.

The term “environment” is intended to characterize it as a fully planned and coherent system, rather than an incremental accretion of very specific and inflexible tools, as is frequently the case with other data analysis software.
R, like S, is designed around a true computer language, and it allows users to add additional functionality by defining new functions. Much of the system is itself written in the R dialect of S, which makes it easy for users to follow the algorithmic choices made. For computationally-intensive tasks, C, C++ and Fortran code can be linked and called at run time. Advanced users can write C code to manipulate R objects directly.
Many users think of R as a statistics system. We prefer to think of it of an environment within which statistical techniques are implemented. R can be extended (easily) via packages. There are about eight packages supplied with the R distribution and many more are available through the CRAN family of Internet sites covering a very wide range of modern statistics.
R has its own LaTeX-like documentation format, which is used to supply comprehensive documentation, both on-line in a number of formats and in hard copy.

Every data analysis technique at your fingertips : R includes virtually every data manipulation, statistical model, and chart that the modern data scientist could ever need. You can easily find, download and use cutting-edge community-reviewed methods in statistics and predictive modeling from leading researchers in data science, free of charge.

Create beautiful and unique data visualizations : Representing complex data with charts and graphs is an essential part of the data analysis process, and R goes far beyond the traditional bar chart and line plot. Heavily influenced by thought leaders in data visualization like Bill Cleveland and Edward Tufte, R makes it easy to draw meaning from multidimensional data with multi-panel charts, 3-D surfaces and more. The custom charting capabilities of R are featured in many of the stunning infographics seen in the New York Times, The Economist, and the Flowing Data blog.

Get better results faster : Instead of using point-and-click menus or inflexible "black-box" procedures, R is a programming language designed expressly for data analysis. Intermediate level R programmers create data analyses faster than users of legacy statistical software, with the flexibility to mix-and-match models for the best results. And R scripts are easily automated, promoting both reproducible research and production deployments.

Draw on the talents of data scientists worldwide : As a thriving open-source project, R is supported by a community of more than 2 million users and thousands of developers worldwide. Whether you're using R to optimize portfolios, analyze genomic sequences, or to predict component failure times, experts in every domain have made resources, applications and code available for free, online.

Learn More : https://www.r-project.org/about.html

Top IOT Platforms

mayankthapa1994@gmail.com (Mayank) — Mon, 30 May 2016 11:13:00 +0000

Top IOT Platforms

What is an IOT Platform ?

In simple words the purpose of any IoT device is to connect with other IoT devices and applications (cloud-based mostly) to relay information using internet transfer protocols.
The gap between the device sensors and data networks is filled by an IoT Platform. Such a platform connects the data network to the sensor arrangement and provides insights using backend applications to make sense of plethora of data generated by hundreds of sensors.
In light of the possibilities that internet of things is offering tech companies has started capitalizing it. There are many IoT platforms available now that provide option to deploy internet of things applications on the go.
While there are hundreds of companies and a few startups venturing into IoT platform development, players like Amazon and Microsoft are way ahead of others in the competition. Read on to know about top 10 IoT platforms you can use for your applications.

1) Amazon Web Services (AWS) IOT

Last year Amazon announced the AWS IoT platform at it s Re:Invent conference. Main features of AWS IoT platform are:

Registry for recognizing devices

Software Development Kit for devices

Device Shadows

Secure Device Gateway

Rules engine for inbound message evaluation

According to Amazon, their IoT platform will make it a lot easier for developers to connect sensors for multiple applications ranging from automobiles to turbines to smart home light bulbs.
Taking the scope of AWS IoT to the next level, the vendor has partnered with hardware manufacturers like Intel, Texas Instruments, Broadcom and Qualcomm to create starter kits compatible with their platform.

2) Microsoft Azure IOT

Microsoft is very much interested in bringing up products for internet of things. For the initiative the Microsoft Azure cloud services compatible IoT platform, the Azure IoT suite is on the offer. Features included in this platform are:

Device shadowing

A rules engine

Identity registry

Information monitoring

For processing the massive amount of information generated by sensors Azure IoT suite comes with Azure Stream Analytics to process massive amounts of information in real-time.

3) ThingWorx IOT Platform

In vendor’s own words, “ThingWorx is the industry’s leading Internet of Things (IoT) technology platform. It enables innovators to rapidly create and deploy game-changing applications, solutions and experiences for today’s smart, connected world.”
Thingsworx is an IoT platform which is designed for enterprise application development. It offers features like:

Easy connectivity of devices to the platform

Remove complexity from IoT application development

Sharing platform among developers for rapid development

Integrated machine learning for automating complex big data analytics

Deploy cloud, embedded or on-premise IoT solutions on the go.

4) IBM Watson

We can never expect the Big Blue to miss on the opportunity to making a mark in the internet of things segment. IBM Watson is an IoT platform which is pretty much taken among developers already. Backed by IBM’s hybrid cloud PaaS (platform as a service) development platform, the Bluemix, Watson IoT enables developers to easily deploy IoT applications.
Users of IBM Watson get:

Device Management

Secure Communications

Real Time Data Exchange

Data Storage

Recently added data sensor and weather data service.

5) Cisco IOT Cloud

Cisco’s IoT platform is for mobile operators offering:

Voice and data connectivity

SIM lifecycle management

IP session control

Customizable billing and reporting

Lately Cisco also partnered with National Farmers’ Federation in Australia to provide internet of things in agriculture solutions.

6) Salesforce IOT Cloud

Salesforce’s IoT platform is powered by Thunder and is focused entirely on customer engagement. According to the official announcement the platform following features:

Sales-Creating sales orders and capturing potential opportunities

Services-Request and order repairs automatically

Marketing-Notifies customers through texts directly on their devices

Apps-Automatic inspections of inventory.

7) Carriots Cloud

Helping to build and host internet of things applications, Carriots is a PaaS platform. It is slowly gaining popularity among the users due to its ease of integration with other applications. It also widely used for Machine to Machine development.
It offers following functionalities:

Device management

SDK application engine

Debug and logs

API key management

Data export feature

Custom alarms

Customer hierarchy levels

User management

Custom control panel

8) Oracle Integrated Cloud

Offering big data analysis for real-time IoT data, device virtualization, endpoint management and high speed messaging, Oracle’s Integrated Cloud platform is offering an analytics platform for IoT applications. Users can receive the notifications directly on their devices.

According to the vendor their IoT platform is:

Faster to market

Integrated

Real time insight

Secure and scalable for real-time IoT data, device virtualization, endpoint management and high speed messaging, Oracle’s Integrated Cloud platform is offering an analytics platform for IoT applications. Users can receive the notifications directly on their devices.

According to the vendor their IoT platform is:

Faster to market

Integrated

Real time insight

Secure and scalable

Hadoop Interview Questions and Answers

mayankthapa1994@gmail.com (Mayank) — Wed, 25 May 2016 03:47:00 +0000

Spark Interview Questions

mayankthapa1994@gmail.com (Mayank) — Mon, 23 May 2016 08:09:00 +0000

Interview Questions For Spark

1.What is Apache Spark?

Spark is a fast, easy-to-use and flexible data processing framework. It has an advanced execution engine supporting cyclic data flow and in-memory computing. Spark can run on Hadoop, standalone or in the cloud and is capable of accessing diverse data sources including HDFS, HBase, Cassandra and others.

2.Explain key features of Spark.

• Allows Integration with Hadoop and files included in HDFS.
• Spark has an interactive language shell as it has an independent Scala (the language in which Spark is written) interpreter
• Spark consists of RDD’s (Resilient Distributed Datasets), which can be cached across computing nodes in a cluster.
• Spark supports multiple analytic tools that are used for interactive query analysis , real-time analysis and graph processing

3.Define RDD.

RDD is the acronym for Resilient Distribution Datasets – a fault-tolerant collection of operational elements that run parallel. The partitioned data in RDD is immutable and distributed. There are primarily two types of RDD:
• Parallelized Collections : The existing RDD’s running parallel with one another
• Hadoop datasets: perform function on each file record in HDFS or other storage system

4.What does a Spark Engine do?

Spark Engine is responsible for scheduling, distributing and monitoring the data application across the cluster.

5.Define Partitions?

As the name suggests, partition is a smaller and logical division of data similar to ‘split’ in MapReduce. Partitioning is the process to derive logical units of data to speed up the processing process. Everything in Spark is a partitioned RDD.

6.What operations RDD support?

• Transformations
• Actions

7.What do you understand by Transformations in Spark?

Transformations are functions applied on RDD, resulting into another RDD. It does not execute until an action occurs. map() and filer() are examples of transformations, where the former applies the function passed to it on each element of RDD and results into another RDD. The filter() creates a new RDD by selecting elements form current RDD that pass function argument.

8. Define Actions.

An action helps in bringing back the data from RDD to the local machine. An action’s execution is the result of all previously created transformations. reduce() is an action that implements the function passed again and again until one value if left. take() action takes all the values from RDD to local node.

9.Define functions of SparkCore.

Serving as the base engine, SparkCore performs various important functions like memory management, monitoring jobs, fault-tolerance, job scheduling and interaction with storage systems.

10.What is RDD Lineage?

Spark does not support data replication in the memory and thus, if any data is lost, it is rebuild using RDD lineage. RDD lineage is a process that reconstructs lost data partitions. The best is that RDD always remembers how to build from other datasets.

11.What is Spark Driver?

Spark Driver is the program that runs on the master node of the machine and declares transformations and actions on data RDDs. In simple terms, driver in Spark creates SparkContext, connected to a given Spark Master.
The driver also delivers the RDD graphs to Master, where the standalone cluster manager runs.

12.What is Hive on Spark?

Hive contains significant support for Apache Spark, wherein Hive execution is configured to Spark:
hive> set spark.home=/location/to/sparkHome;
hive> set hive.execution.engine=spark;
Hive on Spark supports Spark on yarn mode by default.

13.Name commonly-used Spark Ecosystems.

• Spark SQL (Shark)- for developers
• Spark Streaming for processing live data streams
• GraphX for generating and computing graphs
• MLlib (Machine Learning Algorithms)
• SparkR to promote R Programming in Spark engine.

14.Define Spark Streaming.

Spark supports stream processing – an extension to the Spark API , allowing stream processing of live data streams. The data from different sources like Flume, HDFS is streamed and finally processed to file systems, live dashboards and databases. It is similar to batch processing as the input data is divided into streams like batches.

15.What is GraphX?

Spark uses GraphX for graph processing to build and transform interactive graphs. The GraphX component enables programmers to reason about structured data at scale.

16.What does MLlib do?

MLlib is scalable machine learning library provided by Spark. It aims at making machine learning easy and scalable with common learning algorithms and use cases like clustering, regression filtering, dimensional reduction, and alike.

17.What is Spark SQL?

SQL Spark, better known as Shark is a novel module introduced in Spark to work with structured data and perform structured data processing. Through this module, Spark executes relational SQL queries on the data. The core of the component supports an altogether different RDD called SchemaRDD, composed of rows objects and schema objects defining data type of each column in the row. It is similar to a table in relational database.

18.What is a Parquet file?

Parquet is a columnar format file supported by many other data processing systems. Spark SQL performs both read and write operations with Parquet file and consider it be one of the best big data analytics format so far.

19.What file systems Spark support?

• Hadoop Distributed File System (HDFS)
• Local File system
• S3

20.What is Yarn?

Similar to Hadoop, Yarn is one of the key features in Spark, providing a central and resource management platform to deliver scalable operations across the cluster . Running Spark on Yarn necessitates a binary distribution of Spar as built on Yarn support.

21.List the functions of Spark SQL.

Spark SQL is capable of:
• Loading data from a variety of structured sources
• Querying data using SQL statements, both inside a Spark program and from external tools that connect to Spark SQL through standard database connectors (JDBC/ODBC). For instance, using business intelligence tools like Tableau
• Providing rich integration between SQL and regular Python/Java/Scala code, including the ability to join RDDs and SQL tables, expose custom functions in SQL, and more

22.What are benefits of Spark over MapReduce?

• Due to the availability of in-memory processing, Spark implements the processing around 10-100x faster than Hadoop MapReduce. MapReduce makes use of persistence storage for any of the data processing tasks.
• Unlike Hadoop, Spark provides in-built libraries to perform multiple tasks form the same core like batch processing, Steaming, Machine learning, Interactive SQL queries. However, Hadoop only supports batch processing.
• Hadoop is highly disk-dependent whereas Spark promotes caching and in-memory data storage
• Spark is capable of performing computations multiple times on the same dataset. This is called iterative computation while there is no iterative computing implemented by Hadoop.

23.Is there any benefit of learning MapReduce, then?

Yes, MapReduce is a paradigm used by many big data tools including Spark as well. It is extremely relevant to use MapReduce when the data grows bigger and bigger. Most tools like Pig and Hive convert their queries into MapReduce phases to optimize them better.

24.What is Spark Executor?

When SparkContext connect to a cluster manager, it acquires an Executor on nodes in the cluster. Executors are Spark processes that run computations and store the data on the worker node. The final tasks by SparkContext are transferred to executors for their execution.

25.Name types of Cluster Managers in Spark.

The Spark framework supports three major types of Cluster Managers:
• Standalone: a basic manager to set up a cluster
• Apache Mesos: generalized/commonly-used cluster manager, also runs Hadoop MapReduce and other applications
• Yarn: responsible for resource management in Hadoop

26.What do you understand by worker node?

Worker node refers to any node that can run the application code in a cluster.

27.What is PageRank?

A unique feature and algorithm in graph, PageRank is the measure of each vertex in the graph. For instance, an edge from u to v represents endorsement of v’s importance by u. In simple terms, if a user at Instagram is followed massively, it will rank high on that platform.

28.Do you need to install Spark on all nodes of Yarn cluster while running Spark on Yarn?

No because Spark runs on top of Yarn.

29.Illustrate some demerits of using Spark.

Since Spark utilizes more storage space compared to Hadoop and MapReduce, there may arise certain problems. Developers need to be careful while running their applications in Spark. Instead of running everything on a single node, the work must be distributed over multiple clusters.

30.How to create RDD?

Spark provides two methods to create RDD:

• By parallelizing a collection in your Driver program. This makes use of SparkContext’s ‘parallelize’ method

val Data = Array(2,4,6,8,10)
val distData = sc.parallelize(Data)

• By loading an external dataset from external storage like HDFS, HBase, shared file system

Start With Scala

mayankthapa1994@gmail.com (Mayank) — Fri, 20 May 2016 09:31:00 +0000

What is Scala ?

Scala is a programming language for general software applications. The design of Scala started in 2001, and the first public release was in 2004.

The name Scala is a combination of the words “Scalable” and “Language,” which was chosen to indicate its design goal of growing with the demands of the user base.

Many of the design decisions of Scala are inspired by perceived shortcomings of the Java language with Scala source code compiling to Java Byte Code, allowing it to run on a Java virtual machine.

One of the biggest attractions of Scala is that it has full support for functional programming and has a very strong static type system.

This allows for very concise code, requiring fewer lines of code to achieve the same functionality than many other languages, including Java.

Scala runs on the JVM

Scala gets compiled into Java Byte Code (Java) and is executed by the Java Virtual Machine (JVM). Scala and Java share a common runtime, allowing you to freely mix Java and Scala code if you so desire.

Scala executes Java Code

Scala gets compiled into Byte Code, Scala can call Java code from a Scala program. Hence, you can thus use all the classes of the Java SDK in Scala, as well as your customized Java classes, or any Java open source projects. Scala has a compiler, interpreter and runtime.

Scala has both a compiler and an interpreter which can execute Scala code

Scala code is compiled into Byte Code by the Scala compiler, which can then be executed by the scala command. The scala has similarity with the Java command, in that it executes your compiled Scala code.

Scala code can be directly executed by the Scala interpreter, without you needing to compile it. The Scala interpreter may come in handy as a Scala script interpreter. It is somewhat like a shell script interpreter on a Unix platform.

Scala runs on the JVM. Java and Scala classes, whether they reside in different projects or in the same one, can be freely intermixed and even execute each other’s code.

Getting Started

There are a number of ways to use Scala. Perhaps the easiest is with an IDE (Integrated Development Environment).

This tutorial will use the IntelliJ IDE, which is an IDE that supports a number of languages, including Scala, and has a great free edition, plus a great interactive environment that is perfect for developers new to the language and those with more experience.

In addition, IntelliJ IDE has Scala Worksheet feature support. Worksheets are a REPL (Read-Evaluate-Print Loop) feature that gives you the best of both worlds: compiled code and interactive testing.

Effectively, you get an interactive interpreter for a statically typed language.

You get to enjoy a best-of-class editor support with code completion, hyperlinking, auto-format, etc. A Scala worksheet in IntelliJ IDEA is a Scala file with an .sc extension, which you can run and get evaluation results in a special view appeared in the editor.

Visit: http://www.scala-lang.org/

Five things you need to know about Hadoop vs Apache Spark

mayankthapa1994@gmail.com (Mayank) — Tue, 17 May 2016 13:18:00 +0000

Things to know about Hadoop vs Apache Spark

1) They do different things: Hadoop and Apache Spark are both big-data frameworks, but they don't really serve the same purposes. Hadoop is essentially a distributed data infrastructure: It distributes massive data collections across multiple nodes within a cluster of commodity servers, which means you don't need to buy and maintain expensive custom hardware. It also indexes and keeps track of that data, enabling big-data processing and analytics far more effectively than was possible previously. Spark, on the other hand, is a data-processing tool that operates on those distributed data collections; it doesn't do distributed storage.

2) You can use one without the other: Hadoop includes not just a storage component, known as the Hadoop Distributed File System, but also a processing component called MapReduce, so you don't need Spark to get your processing done. Conversely, you can also use Spark without Hadoop. Spark does not come with its own file management system, though, so it needs to be integrated with one -- if not HDFS, then another cloud-based data platform. Spark was designed for Hadoop, however, so many agree they're better together.

3) Spark is speedier: Spark is generally a lot faster than MapReduce because of the way it processes data. While MapReduce operates in steps, Spark operates on the whole data set in one fell swoop. "The MapReduce workflow looks like this: read data from the cluster, perform an operation, write results to the cluster, read updated data from the cluster, perform next operation, write next results to the cluster, etc.," explained Kirk Borne, principal data scientist at Booz Allen Hamilton. Spark, on the other hand, completes the full data analytics operations in-memory and in near real-time: "Read data from the cluster, perform all of the requisite analytic operations, write results to the cluster, done," Borne said. Spark can be as much as 10 times faster than MapReduce for batch processing and up to 100 times faster for in-memory analytics.

4) You may not need Spark's speed: MapReduce's processing style can be just fine if your data operations and reporting requirements are mostly static and you can wait for batch-mode processing. But if you need to do analytics on streaming data, like from sensors on a factory floor, or have applications that require multiple operations, you probably want to go with Spark. Most machine-learning algorithms, for example, require multiple operations. Common applications for Spark include real-time marketing campaigns, online product recommendations, cybersecurity analytics and machine log monitoring.

5) Failure recovery: different, but still good: Hadoop is naturally resilient to system faults or failures since data are written to disk after every operation, but Spark has similar built-in resiliency by virtue of the fact that its data objects are stored in something called resilient distributed datasets distributed across the data cluster. "These data objects can be stored in memory or on disks, and RDD provides full recovery from faults or failures," Borne pointed out.

Skills Needed For Machine Learning

mayankthapa1994@gmail.com (Mayank) — Mon, 16 May 2016 12:39:00 +0000

Skills Needed For Machine Learning

1. Computer Science Fundamentals and Programming:

Computer science fundamentals important for Machine Learning engineers include data structures (stacks, queues, multi-dimensional arrays, trees, graphs, etc.), algorithms (searching, sorting, optimization, dynamic programming, etc.), computability and complexity (P vs. NP, NP-complete problems, big-O notation, approximate algorithms, etc.), and computer architecture (memory, cache, bandwidth, deadlocks, distributed processing, etc.).

You must be able to apply, implement, adapt or address them (as appropriate) when programming. Practice problems, coding competitions and hackathons are a great way to hone your skills.

2. Probability and Statistics:

A formal characterization of probability (conditional probability, Bayes rule, likelihood, independence, etc.) and techniques derived from it (Bayes Nets, Markov Decision Processes, Hidden Markov Models, etc.) are at the heart of many Machine Learning algorithms; these are a means to deal with uncertainty in the real world. Closely related to this is the field of statistics, which provides various measures (mean, median, variance, etc.), distributions (uniform, normal, binomial, Poisson, etc.) and analysis methods (ANOVA, hypothesis testing, etc.) that are necessary for building and validating models from observed data. Many Machine Learning algorithms are essentially extensions of statistical modeling procedures.

3. Data Modeling and Evaluation:

Data modeling is the process of estimating the underlying structure of a given dataset, with the goal of finding useful patterns (correlations, clusters, eigenvectors, etc.) and/or predicting properties of previously unseen instances (classification, regression, anomaly detection, etc.). A key part of this estimation process is continually evaluating how good a given model is. Depending on the task at hand, you will need to choose an appropriate accuracy/error measure (e.g. log-loss for classification, sum-of-squared-errors for regression, etc.) and an evaluation strategy (training-testing split, sequential vs. randomized cross-validation, etc.). Iterative learning algorithms often directly utilize resulting errors to tweak the model (e.g. backpropagation for neural networks), so understanding these measures is very important even for just applying standard algorithms.

4. Applying Machine Learning Algorithms and Libraries:

Standard implementations of Machine Learning algorithms are widely available through libraries/packages/APIs (e.g. scikit-learn, Theano, Spark MLlib, H2O, TensorFlow etc.), but applying them effectively involves choosing a suitable model (decision tree, nearest neighbor, neural net, support vector machine, ensemble of multiple models, etc.), a learning procedure to fit the data (linear regression, gradient descent, genetic algorithms, bagging, boosting, and other model-specific methods), as well as understanding how hyperparameters affect learning. You also need to be aware of the relative advantages and disadvantages of different approaches, and the numerous gotchas that can trip you (bias and variance, overfitting and underfitting, missing data, data leakage, etc.). Data science and Machine Learning challenges such as those on Kaggle are a great way to get exposed to different kinds of problems and their nuances.

5. Software Engineering and System Design:

At the end of the day, a Machine Learning engineer’s typical output or deliverable is software. And often it is a small component that fits into a larger ecosystem of products and services. You need to understand how these different pieces work together, communicate with them (using library calls, REST APIs, database queries, etc.) and build appropriate interfaces for your component that others will depend on. Careful system design may be necessary to avoid bottlenecks and let your algorithms scale well with increasing volumes of data. Software engineering best practices (including requirements analysis, system design, modularity, version control, testing, documentation, etc.) are invaluable for productivity, collaboration, quality and maintainability.

Machine Learning Job Roles

Jobs related to Machine Learning are growing rapidly as companies try to get the most out of emerging technologies. The chart below depicts the relative importance of core skills for these general types of roles, with a typical Data Analyst role for comparison.

The Future of Machine Learning

What is perhaps most compelling about Machine Learning is its seemingly limitless applicability. There are already so many fields being impacted by Machine Learning, including education, finance, computer science, and more. There are also virtually NO fields to which Machine Learning doesn’t apply. In some cases, Machine Learning techniques are in fact desperately needed. Healthcare is an obvious example. Machine Learning techniques are already being applied to critical arenas within the Healthcare sphere, impacting everything from care variation reduction efforts to medical scan analysis. David Sontag, an assistant professor at New York University’s Courant Institute of Mathematical Sciences and NYU’s Center for Data Science, recently gave a talkon Machine Learning and the Healthcare system, in which he discussed “how machine learning has the potential to change health care across the industry, from enabling the next-generation electronic health record to population-level risk stratification from health insurance claims.”

The world is unquestionably changing in rapid and dramatic ways, and the demand for Machine Learning engineers is going to keep increasing exponentially. The world’s challenges are complex, and they will require complex systems to solve them. Machine Learning engineers are building these systems. If this is YOUR future, then there’s no time like the present to start mastering the skills and developing the mindset you’re going to need to succeed.

Link : https://en.wikipedia.org/wiki/Machine_learning

8 Skills You Need To Be A Data Scientist

mayankthapa1994@gmail.com (Mayank) — Mon, 16 May 2016 10:36:00 +0000

8 Skills You Need To Be A Data Scientist

1) Basic Tools: No matter what type of company you’re interviewing for, you’re likely going to be expected to know how to use the tools of the trade. This means a statistical programming language, like R or Python, and a database querying language like SQL.

2) Basic Statistics: At least a basic understanding of statistics is vital as a data scientist. An interviewer once told me that many of the people he interviewed couldn’t even provide the correct definition of a p-value. You should be familiar with statistical tests, distributions, maximum likelihood estimators, etc. Think back to your basic stats class! This will also be the case for machine learning, but one of the more important aspects of your statistics knowledge will be understanding when different techniques are (or aren’t) a valid approach. Statistics is important at all company types, but especially data-driven companies where the product is not data-focused and product stakeholders will depend on your help to make decisions and design / evaluate experiments.

3) Machine Learning: If you’re at a large company with huge amounts of data, or working at a company where the product itself is especially data-driven, it may be the case that you’ll want to be familiar with machine learning methods. This can mean things like k-nearest neighbors, random forests, ensemble methods – all of the machine learning buzzwords. It’s true that a lot of these techniques can be implemented using R or Python libraries – because of this, it’s not necessarily a dealbreaker if you’re not the world’s leading expert on how the algorithms work. More important is to understand the broadstrokes and really understand when it is appropriate to use different techniques.

4) Multivariable Calculus and Linear Algebra: You may in fact be asked to derive some of the machine learning or statistics results you employ elsewhere in your interview. Even if you’re not, your interviewer may ask you some basic multivariable calculus or linear algebra questions, since they form the basis of a lot of these techniques. You may wonder why a data scientist would need to understand this stuff if there are a bunch of out of the box implementations in sklearn or R. The answer is that at a certain point, it can become worth it for a data science team to build out their own implementations in house. Understanding these concepts is most important at companies where the product is defined by the data and small improvements in predictive performance or algorithm optimization can lead to huge wins for the company.

5) Data Munging: Often times, the data you’re analyzing is going to be messy and difficult to work with. Because of this, it’s really important to know how to deal with imperfections in data. Some examples of data imperfections include missing values, inconsistent string formatting (e.g., ‘New York’ versus ‘new york’ versus ‘ny’), and date formatting (‘2014-01-01’ vs. ‘01/01/2014’, unix time vs. timestamps, etc.). This will be most important at small companies where you’re an early data hire, or data-driven companies where the product is not data-related (particularly because the latter has often grown quickly with not much attention to data cleanliness), but this skill is important for everyone to have.

6) Data Visualization & Communication: Visualizing and communicating data is incredibly important, especially at young companies who are making data-driven decisions for the first time or companies where data scientists are viewed as people who help others make data-driven decisions. When it comes to communicating, this means describing your findings or the way techniques work to audiences, both technical and non-technical. Visualization wise, it can be immensely helpful to be familiar with data visualization tools like ggplot and d3.js. It is important to not just be familiar with the tools necessary to visualize data, but also the principles behind visually encoding data and communicating information.

7)Software Engineering: If you’re interviewing at a smaller company and are one of the first data science hires, it can be important to have a strong software engineering background. You’ll be responsible for handling a lot of data logging, and potentially the development of data-driven products.

8) Thinking Like A Data Scientist: Companies want to see that you’re a (data-driven) problem solver. That is, at some point during your interview process, you’ll probably be asked about some high level problem – for example, about a test the company may want to run or a data-driven product it may want to develop. It’s important to think about what things are important, and what things aren’t. How should you, as the data scientist, interact with the engineers and product managers? What methods should you use? When do approximations make sense?

Data science is still nascent and ill-defined as a field. Getting a job is as much about finding a company whose needs match your skills as it is developing those skills. This writing is based on my own firsthand experiences – I’d love to hear if you’ve had similar (or contrasting) experiences during your own process.

5 Reasons to Learn Apache Spark

mayankthapa1994@gmail.com (Mayank) — Mon, 16 May 2016 06:32:00 +0000

5 Reasons to Learn Apache Spark

1) Learn Apache Spark to have Increased Access to Big Data

Apache Spark is opening up various opportunities for big data exploration and making it easier for organizations to solve different kinds of big data problems. Spark is the hottest technology now, not just among the data engineers but even majority of data scientists prefer to work with Spark. Apache Spark is a fascinating platform for data scientists with use cases spanning across investigative and operational analytics.

Data scientists are exhibiting interest in working with Spark because of its ability to store data resident in memory that helps speed up machine learning workloads unlike Hadoop MapReduce. Apache Spark has witnessed continuous upward trajectory in the big data ecosystem. With IBM’s recent announcement that it will educate more than 1 million data engineers and data scientists on Apache Spark – 2016 is definitely THE year to learn Spark and pursue a lucrative career.

2) Learn Apache Spark to Make Use of Existing Big Data Investments

After the inception of Hadoop, several organizations invested in novel computing clusters to make use of the technology. However, Apache Spark does not pose any limitations on investing in new computing clusters as organizations can use Spark on top of the existing Hadoop clusters.

Spark can run on Hadoop MapReduce as it can run on YARN and on HDFS. With high compatibility of Spark with Hadoop, companies are on the verge of hiring increased number of Spark developers as they do not have to re-invest on computing clusters because it can be integrated well with Hadoop. This also makes learning spark an added advantage for professionals with expertise in Hadoop skills.

3) Learn Apache Spark to pace up with Growing Enterprise Adoption

Spark will reinvigorate Hadoop, and in 2016, nine out of every 10 projects on Hadoop will be Spark-related projects. — said Monte Zweben, CEO of Splice Machine

With companies embracing the adoption of various adjacent big data technologies that complement Hadoop-Spark adoption rate is increasing exponentially. Spark is no more just a component of the big data Hadoop ecosystem but has become the go-to big data technology for enterprises across various verticals.

“Spark provides dramatically increased data processing speed compared to Hadoop and is now the largest big data open-source project.” said Apache Spark originator Matei Zaharia.

A recent survey on Spark adoption revealed that Spark community has had most of the contributions compared to other open source projects managed by Apache foundation. There is an increasing demand to support BI workloads using a combination of the two big data tools - Hadoop and Spark SQL.

The survey findings show that among Apache Spark adopters 68% of the companies are using Spark to render support for BI workloads. Spark’s clear value proposition is leading to an increased adoption rate by enterprises opening up lucrative opportunities for big data developers with Spark and Hadoop skills.

Big data predictions for 2016 expect Apache Spark to go its own way, creating a novel, vibrant ecosystem with popular cloud vendors releasing their individual Spark PaaS offerings.

4) Learn Apache Spark as 2016 is set to witness an increasing demand for Spark Developers

Spark’s enterprise adoption is rising because of its potential to eclipse Hadoop as it is the best alternative to MapReduce - within the Hadoop framework or outside it. Similar to Hadoop, Apache Spark also requires technical expertise in object oriented programming concepts to program and run- thus opening up job opportunities for those who have hands-on working experience in Spark. Industry-wide Spark skills shortage is leading to a number open jobs and contracting opportunities for big data professionals.

For people who want to make a career on the forefront of big data technology, learning apache spark now will open up a lot of opportunities. There are several ways to bridge the skills gap for getting a data related jobs and finding a position as a Spark developer. The best way is to take a formal training that provides hands-on working experience and helps learning through hands on projects.

According to the popular IT job portal, Dice.com, a keyword search for the term “Spark Developer” showed 34617 listings as of 16th December, 2015.

5) Learn Apache Spark to make big money

Spark developers are so in-demand that companies are agreeing to bend the recruitment rules, offer attractive benefits and provide flexible work timings just to hire experts skilled in Apache Spark. According to indeed.com, the average salary for a Spark Developer in San Francisco is $128, 000 as of December 16, 2015. Indeed.com statistics reveal that the average salary for spark developers in San Francisco is 35% more than the average salaries for Spark developers in US.

According to O’Reilly, data engineers who have experience with Apache Spark and Storm earn the highest average salaries. Several recent salary surveys found that data engineers and data analysts having big data skills like Hadoop are earning close to $120,000/year, when compared to the average IT tech salary of $89,450. Apache Spark and Storm skilled professionals are pulling close to $150,000 in yearly salaries, when compared to the total average salary of data engineers which is $98,000. People with a keen desire to grow their big data career and earn high salaries - must learn Apache Spark online now.

Get Started with Learning Apache Spark

http://spark.apache.org/documentation.html

As Spark continues to be used for interactive scale out data processing requirements and batch oriented needs, it is expected to play a vital role in the next generation scale out BI applications. Professionals need to undertake comprehensive hands-on training in Spark to become productive especially if they are newbies to Scala programming. It requires professionals to get comfortable with a new programming paradigm like Scala. However, one can also use Shark i.e. SQL on Shark to get started with learning Apache Spark.

Internet of Things (IoT)

mayankthapa1994@gmail.com (Mayank) — Sat, 14 May 2016 09:59:00 +0000

Internet of Things (IoT)

What it is and Why it matters

The Internet of Things is the concept of everyday objects – from industrial machines to wearable devices – using built-in sensors to gather data and take action on that data across a network. So it’s a building that uses sensors to automatically adjust heating and lighting. Or production equipment alerting maintenance personnel to an impending failure. Simply put, the Internet of Things is the future of technology that can make our lives more efficient.

History Of the Internet of Things

We’ve been fascinated with gadgets that function on a grander scale for decades (think spy movie-type stuff) – but it’s only been in the past several years that we’ve seen the IoT’s true potential. The concept evolved as wireless Internet became more pervasive, embedded sensors grew in sophistication and people began understanding that technology could be a personal tool as well as a professional one. The term “Internet of Things” was coined in the late 1990s by entrepreneur Kevin Ashton. Ashton, who’s one of the founders of the Auto-ID Center at MIT, was part of a team that discovered how to link objects to the Internet through an RFID tag. He said he first used the phrase “Internet of Things” in a presentation he made in 1999 – and the term has stuck around ever since.

Why is the Internet of Things important ?

You might be surprised to learn how many things are connected to the Internet, and how much economic benefit we can derive from analyzing the resulting data streams. Here are some examples of the impact the IoT has on industries:

Intelligent transport solutions speed up traffic flows, reduce fuel consumption, prioritize vehicle repair schedules and save lives.

Smart electric grids more efficiently connect renewable resources, improve system reliability and charge customers based on smaller usage increments.

Machine monitoring sensors diagnose – and predict – pending maintenance issues, near-term part stockouts, and even prioritize maintenance crew schedules for repair equipment and regional needs.

Data-driven systems are being built into the infrastructure of "smart cities," making it easier for municipalities to run waste management, law enforcement and other programs more efficiently.

You’re low on milk. When you’re on your way home from work, you get an alert from your refrigerator reminding you to stop by the store.

Your home security system, which already enables you to remotely control your locks and thermostats, can cool down your home and open your windows, based on your preferences.

But also consider the IoT on a more personal level. Connected devices are making their way from business and industry to the mass market. Consider these possibilities:

Who's Using It ?

The IoT is more than just a convenience for consumers. It offers new sources of data and business operating models that can boost productivity in a variety of industries.

1) Health Care :

Many people have already adopted wearable devices to help monitor exercise, sleep and other health habits – and these items are only scratching the surface of how IoT impacts health care. Patient monitoring devices, electronic records and other smart accessories can help save lives.

2) Manufacturing :

This is one of the industries that benefits from IoT the most. Data-collecting sensors embedded in factory machinery or warehouse shelves can communicate problems or track resources in real time, making it easy to work more efficiently and keep costs down.

3) Retail :

Both consumers and stores can benefit from IoT. Stores, for example, might use IoT for inventory tracking or security purposes. Consumers may end up with personalized shopping experiences through data collected by sensors or cameras.

4) Telecommunications :

The telecommunications industry will be significantly impacted by the IoT since it will be charged with keeping all the data the IoT uses. Smart phones and other personal devices must be able to maintain a reliable connection to the Internet for the IoT to work effectively.

5) Transportation

While cars aren’t at the point of driving themselves, they’re undoubtedly more technologically advanced than ever. The IoT also impacts transportation on a larger scale: delivery companies can track their fleet using GPS solutions. And roadways can be monitored via sensors to keep them as safe as possible.

6) Utilities :

Smart meters not only collect data automatically, they make it possible to apply analytics that can track and manage energy use. Likewise, sensors in devices such as windmills can track data and use predictive modeling to schedule downtime for more efficient energy use.

How It Works ?

In IoT discussions, it’s recognized from the onset that analytics technologies are critical for turning this tide of streaming source data into informative, aware and useful knowledge. But how do we analyze data as it streams nonstop from sensors and devices? How does the process differ from other analytical methods that are common today?

In traditional analysis, data is stored and then analyzed. However, with streaming data, the models and algorithms are stored and the data passes through them for analysis. This type of analysis makes it possible to identify and examine patterns of interest as data is being created – in real time.

So before the data is stored, in the cloud or in any high-performance repository, you process it automatically. Then, you use analytics to decipher the data, all while your devices continue to emit and receive data.

With advanced analytics techniques, data stream analytics can move beyond monitoring existing conditions and evaluating thresholds to predicting future scenarios and examining complex questions.

To assess the future using these data streams, you need high-performance technologies that identify patterns in your data as they occur. Once a pattern is recognized, metrics embedded into the data stream drive automatic adjustments in connected systems or initiate alerts for immediate actions and better decisions.

Essentially, this means you can move beyond monitoring conditions and thresholds to assessing likely future events and planning for countless what-if scenarios.

BIG DATA – FINDING NEW AREAS OF APPLICATIONS

mayankthapa1994@gmail.com (Mayank) — Sat, 14 May 2016 07:12:00 +0000

There is no denying the fact that today the buzz around big data is earsplitting. Organizations (both private and public) are reconsidering their data management approaches in the light of rapid advancements in emerging technologies such as Apache Hadoop, an open source big data technology.

Today, these data-driven decisions are helping businesses revamp their product lines, overhaul their service offerings, find new revenue opportunities and spur innovation. Even the public sector is entering the arena to leverage available data. The key point here is that it’s not enough just to accumulate vast amounts of information. You also have to deploy the right tools to analyze information effectively and glean insights from it.

Let us cut to the chase and discuss some innovative application areas of Big Data that have come to the fore:

Forecasting Disease Outbreak – The healthcare sector has already been the beneficiary of this new wave of big data integration. Enterprise health analytics is being driven primarily due to the need to cut down costs and improve the quality of care. For instance, researchers at the John Hopkins University showed how the Twitter universe can be analyzed extensively (5000 Tweets/minute) to get early warning signals on any surge in flu related conversations.

Protecting Economic Initiatives – Data-driven analytics are no longer restricted to the use of the private sector alone. Global Pulse is a United Nations initiative to track in real-time any anomalies that might hamper development goals both at local and global levels. Timely, actionable information extracted via big data analysis is turning out to be a potent tool in the hands of decision makers in bridging the gaps before the situation escalates and hampers any significant social development gains.

Helping in Talent Management – Big-data driven talent analytics will soon swarm the recruitment industry and help it in attracting and retaining talent for long. Sears, with more than 2500 retail outlets in the US and Canada, is harnessing the power of Big Data to make informed decisions when it comes to talent acquisition.

Aiding the Insurance Industry – The insurance industry is ripe for disruption, and there is no better example than the insurance products sold by The Climate Corporation. The company’s platform manages to generate 10 trillion weather simulation data points, which are in turn used to ascertain risk and pricing for the insurance products. This voluminous amount of weather-related data helps in reducing risks and safeguarding agricultural interests in the event of any unforeseen calamity.

THE POPULAR GRAPH DATABASES

mayankthapa1994@gmail.com (Mayank) — Sat, 14 May 2016 04:05:00 +0000

1) NEO4J

Neo4j is the market leader in GraphDB. It is highly performant, scalable, and flexible. It is equipped with a rich UI, called the Neo4j browser, that is used for queries, visualization, and data interaction. It has compatibility with most programming platforms like Java, Nodejs, PHP, Python, .NET, etc. It supports the Cypher query language, whose syntax naturally depicts data and relationships. It has a good integration support with Apache Spark, Elasticsearch, MongoDB, Cassandra, and Docker. The community edition is free under GPL v3 license.

2) ORIENT DB

Orient DB is a high speed, scalable, and reliable GraphDB. It has a high and constant traversing speed that is not affected by database size. User, role, and record level security is built-in. It has a good compatibility with most programming platforms like Java, Nodejs, PHP, Python, .NET, etc. It has a strong community support and is licensed under Apache 2.

3) TiTAN DB

Titan DB is highly scalable and can store and query graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. Titan DB is transactional and supports various backend stores like Cassandra, HBase, and BerkeleyDB. It supports integration with Elastic Search, Lucene, and Solr, and also supports the Gremlin query language. It is licensed under Apache 2.

4) Apache Giraph

It is an Iterative Graph processing framework backed by Apache Hadoop. Giraph jobs are highly scalable as they run on Hadoop Cluster. Giraph can greatly increase the efficiency of graph computation on a huge dataset. It is licensed under Apache 2. We wanted to take a deep dive into these graph databases to understand how they stacked against each other. At the outset, we eliminated TitanDB because a stable 1.0 version had not been released at the time. We also skipped Apache Giraph because it is backed up only by Hadoop. This left Neo4j and OrientDB for more exploration.

TEST ENVIRONMENT

CPU- Intel quad core i7 2.2 GHz
RAM – 16 GB
Hard disk type – Flash
Database on same machine as test program (no network latency)

TEST METHODOLOGY

We used the sample Spring projects and Java drivers of each of the databases. The intent is to programmatically

Add 100k users,
Randomly search users across 100k edges in total, and
Come up with our average read/write speeds.

We then searched users with similar preferences to each other. We ran the suite as Java unit-tests and captured the times using timestamps. Neo4j allowed indexing capability on one of the attributes - we indexed the userID to give us indexed and unindexed speeds. Read queries were not identical in each DB, so the closest comparison was run.

RESULTS

The results scale from 1 to 5, where the higher value is better.

Dimensions	Neo4j	Orient DB	Comments
Stable Java Driver? (Yes/No)	Yes	Yes
Maturity of Client Framework (1-5)	5	2
Ease of query language (1-5)	5	3	Gremlin relates to SQL. Cypher relates more to graph queries.
Ease of setup (1-5)	3	5
# of Reads per Second	0.161ms (no index) 0.139 (indexed)	0.955ms (search)	Single node retrieval; the gap decreased when searching for multiple nodes.
# of Writes per Second	0.023ms/edge	0.31ms/edge
Storage Footprint (MB)	103mb	86mb
Average Memory Footprint (MB)	172mb – Average	256mb – Average
License Cost	Community edition is free under GPL v3 license.	Community edition is free under Apache 2 license.	Enterprise version cost is unknown and can be figured out with the vendor directly.
Community Support	5	2

Whitepaper: http://www.stingergraph.com/data/uploads/papers/ppaa2014.pdf

Steps For Installing & Configuring Hadoop in Standalone Mode

mayankthapa1994@gmail.com (Mayank) — Fri, 13 May 2016 03:54:00 +0000

Steps For Installing & Configuring Hadoop in Standalone Mode

You might want to create a dedicated user for running Apache Hadoop but it is not a prerequisite. In our demonstration, we will be using a default user for running Hadoop.

Environment

Ubuntu 10.10

JDK 6 or above

Hadoop-1.1.2 (Any stable release)

Follow these steps for installing and configuring Hadoop on a single node:

Step-1. Install Java

In this tutorial, we will use Java 1.6 therefore describing the installation of Java 1.6 in detail.

Use the below command to begin the installation of Java

$ sudo apt-get install openjdk-6-jdk

or

$ sudo apt-get install sun-java6-jdk

This will install the full JDK under /usr/lib/jvm/java-6-sundirectory.

Step-2. Verify Java installation

You can verify java installation using the following command

$ java -version

On executing this command, you should see output similar to the following:

java version “1.6.0_27″

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

Step-3. SSH configuration

Install SSH using the command.

sudo apt-get install ssh
Generate ssh key
ssh -keygen -t rsa -P “” (press enter when asked for a file name; this will generate a passwordless ssh file)

Now copy the public key (id_rsa.pub) of current machine to authorized_keysBelow command copies the generated public key in the .ssh/authorized_keys file.

cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Verify ssh configuration using the comman

ssh localhost

Pressing yes will add localhost to known hosts

Step-4. Download Hadoop

Download the latest stable release of Apache Hadoop from http://hadoop.apache.org/releases.html.

Unpack the release tar – zxvf hadoop-1.0.3.tar.gz

Save the extracted folder to an appropriate location, HADOOP_HOME will be pointing to this directory.

Step-5. Verify Hadoop

Check if the following directories exist under HADOOP_HOME: bin, conf, lib, bin

Use the following command to create an environment variable that points to the Hadoop installation directory (HADOOP_HOME)

export HADOOP_HOME=/home/user/hadoop

Now place the Hadoop binary directory on your command-line path by executing the command

export PATH=$PATH:$HADOOP_HOME/bin

Use this command to verify your Hadoop installation:

hadoop version

The o/p should be similar to below one

Hadoop 1.1.2

Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r911707

Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010

Step-6. Configure JAVA_HOME

Hadoop requires Java installation path to work on, for this we will be setting JAVA_HOME environment variable and this will point to our Java installation dir.

Java_Home can be configured in ~/.bash_profile or ~/.bashrc file. Alternatively you can also let hadoop know this by setting Java_Home in hadoop conf/hadoop-env.sh file.

Use the below command to set JAVA_HOME on Ubuntu

export JAVA_HOME=/usr/lib/jvm/java-6-sun

JAVA_HOME can be verified by command

echo $JAVA_HOME

Step-7. Create Data Directory for Hadoop

An advantage of using Hadoop is that with just a limited number of directories you can set it up to work correctly. Let us create a directory with the name hdfs and three sub-directories name, data and tmp.

Since a Hadoop user would require to read-write to these directories you would need to change the permissions of above directories to 755 or 777 for Hadoop user.

Step-8. Configure Hadoop XML files

Next, we will configure Hadoop XML file. Hadoop configuration files are in the HADOOP_HOME/conf dir.

conf/core-site.xml

<!--?xml version="1.0"-->>

<!--?xml -stylesheet type="text/xsl" href="configuration.xsl"?-->

<! -- Putting site-specific property overrides the file. -->

fs.default.name

hdfs://localhost:9000

hadoop.temp.dir

/home/girish/hdfs/temp<span style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: 13px; line-height: 19px;"> </span>

conf/hdfs-site.xml

<! -- Putting site specific property overrides in the file. -->

dfs.name.dir

/home/girish/hdfs/name

dfs.data.dir

/home/girish/hdfs/data

dfs.replication

1

<strong style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: 13px; line-height: 19px;">conf/mapred-site.xml</strong>

<! -- Putting site-specific property overrides this file. -->

mapred.job.tracker

localhost:9001

conf/masters

Not required in single node cluster.
conf/slaves
Not required in single node cluster.
Step-9. Format Hadoop Name Node-

Execute the below command from hadoop home directory

$ ~/hadoop/bin/hadoop namenode -format

The following image gives an overview of a Hadoop Distributed File System Architecture.

Step-10. Start Hadoop daemons

$ ~/hadoop/bin/start-all.sh

Step-11. Verify the daemons are running

$ jps  (if jps is not in path, try  /usr/java/latest/bin/jps)

output will look similar to this

9316 SecondaryNameNode

9203 DataNode

9521 TaskTracker

9403 JobTracker

9089 NameNode

Now we have all the daemons running:

Note: If your master server fails to start due to the dfs safe mode issue, execute this on the Hadoop command line:

hadoop dfsadmin -safemode leave

Also make sure to format the namenode again if you make changes to your configuration.

Step-12. Verify UIs by namenode & job tracker

Open a browser window and type the following URLs:

namenode UI: http://machine_host_name:50070

job tracker UI: http://machine_host_name:50030

substitute ‘machine host name’ with the public IP of your node e.g: http://localhost:50070

Now you have successfully installed and configured Hadoop on a single node.

BASIC HADOOP ADMIN COMMANDS

(Source: Getting Started with Hadoop):

The ~/hadoop/bin directory contains some scripts used to launch Hadoop DFS and Hadoop Map/Reduce daemons. These are:

start-all.sh – Starts all Hadoop daemons, the namenode, datanodes, the jobtracker and tasktrackers.
stop-all.sh – Stops all Hadoop daemons.
start-mapred.sh – Starts the Hadoop Map/Reduce daemons, the jobtracker and tasktrackers.
stop-mapred.sh – Stops the Hadoop Map/Reduce daemons.
start-dfs.sh – Starts the Hadoop DFS daemons, the namenode and datanodes.
stop-dfs.sh – Stops the Hadoop DFS daemons.

BigData Facts and Stats

mayankthapa1994@gmail.com (Mayank) — Fri, 28 Aug 2015 12:09:00 +0000

Pin Your Interest

Which IT Job Is the Best?

Software Developer/Computer Programmer

Database Administrator

Hardware Engineer

Systems Analyst

Network Architect

Web developer

Information Security Analyst

Systems Manager

IT Support

Introduction To R Language

INTRODUCTION TO R

The R environment

R is an integrated suite of software facilities for data manipulation, calculation and graphical display. It includes

Top IOT Platforms

Top IOT Platforms

What is an IOT Platform ?

1) Amazon Web Services (AWS) IOT

Voice and data connectivity SIM lifecycle management IP session control Customizable billing and reporting Lately Cisco also partnered with National Farmers’ Federation in Australia to provide internet of things in agriculture solutions. 6) Salesforce IOT Cloud

7) Carriots Cloud

8) Oracle Integrated Cloud

Offering big data analysis for real-time IoT data, device virtualization, endpoint management and high speed messaging, Oracle’s Integrated Cloud platform is offering an analytics platform for IoT applications. Users can receive the notifications directly on their devices.

According to the vendor their IoT platform is: Faster to market Integrated Real time insight Secure and scalable

Hadoop Interview Questions and Answers

Top Hadoop Interview Questions

Spark Interview Questions

Interview Questions For Spark

Start With Scala

What is Scala ?

Getting Started

Visit: http://www.scala-lang.org/

Five things you need to know about Hadoop vs Apache Spark

Things to know about Hadoop vs Apache Spark

Skills Needed For Machine Learning

Skills Needed For Machine Learning

8 Skills You Need To Be A Data Scientist

8 Skills You Need To Be A Data Scientist

5 Reasons to Learn Apache Spark

5 Reasons to Learn Apache Spark

1) Learn Apache Spark to have Increased Access to Big Data

2) Learn Apache Spark to Make Use of Existing Big Data Investments

3) Learn Apache Spark to pace up with Growing Enterprise Adoption

4) Learn Apache Spark as 2016 is set to witness an increasing demand for Spark Developers

5) Learn Apache Spark to make big money

Get Started with Learning Apache Spark

Internet of Things (IoT)

Internet of Things (IoT)

What it is and Why it matters

History Of the Internet of Things

Why is the Internet of Things important ?

Who's Using It ?

1) Health Care :

2) Manufacturing :

3) Retail :

4) Telecommunications :

5) Transportation

6) Utilities :

BIG DATA – FINDING NEW AREAS OF APPLICATIONS

THE POPULAR GRAPH DATABASES

1) NEO4J

3) TiTAN DB

4) Apache Giraph

TEST ENVIRONMENT

TEST METHODOLOGY

RESULTS

Steps For Installing & Configuring Hadoop in Standalone Mode

BASIC HADOOP ADMIN COMMANDS

BigData Facts and Stats

Here are the facts about the BigData that u should know . . . . .

Voice and data connectivity

SIM lifecycle management

IP session control

Customizable billing and reporting

Lately Cisco also partnered with National Farmers’ Federation in Australia to provide internet of things in agriculture solutions.

6) Salesforce IOT Cloud

According to the vendor their IoT platform is:

Faster to market

Integrated

Real time insight

Secure and scalable