ML Hero from zero

NEW TREND OF DATA SCIENCE: REINFORCEMENT LEARNING

2020-11-03T23:49:00.002-08:00

Reinforcement Learning (RL) is a machine learning method that empowers a specialist to learn in an intuitive environment by performing trial and error utilizing observations from its very own activities and encounters. In spite of the fact that both direct and reinforcement learning use mapping among input and output, not at all like supervised learning where input gave to the specialist is basically the right set of activities for playing out a task, reinforcement learning utilizes prizes and discipline as signs for positive and negative conduct.

When compared with unsupervised learning, reinforcement learning is distinctive as far as objectives are taken into consideration. While the objective in unsupervised learning is to discover synonymities and contrasts between data points, in reinforcement learning the objective is to locate a reasonable activity model that would boost the aggregate total reward of the specialist.

Reinforcement learning will be a huge thing in Data science in 2019. While RL has been around for quite a while in the scholarly world, it has barely observed any industry adoption whatsoever. Halfway on the grounds that there have been a lot of low-draping organic products to pick in predictive analytics, yet for the most part as a result of the hindrances in execution, learning and accessible tools. The potential worth of utilizing RL in proactive analytics and AI is huge, yet it additionally demands a more noteworthy range of abilities to ace. In addition to the fact that it involves more complicated algorithms and less advanced tools, it likewise requires precise recreations of real-life conditions. An expanding number of individuals in the industry think about the boundless capabilities of RL, however not many have been willing to make real investments. It’s, for the most part, considered excessively uncertain.

Reinforcement learning is the way DeepMind developed the AlphaGo framework that beat a high-positioning Go player and has of the late been winning online Go matches namelessly. It’s the means by which University of California Berkeley’s BRETT robot figures out how to move its hands and arms to perform physical undertakings like stacking squares or screwing the top onto a container, in only three hours or even in ten minutes if it’s told where the objects are kept that it will work with, and where they have to wind up. Engineers at a hackathon assembled a smart trash that can also be considered as AutoTrash which deployed reinforcement learning for sorting compostable and recyclable waste into the correct compartments. Reinforcement learning is the reason Microsoft just purchased Maluuba, which the company intends to utilize it to help in understanding day-to-day language for inquiry and chatbots, as a springboard to general intelligence.

However, business deployments are far rarer. In 2016, Google began utilizing DeepMind’s reinforcement learning to understand how to spare power in some of its data centers by figuring out how to enhance around 120 unique settings like how the fans and cooling frameworks run, signifying a 15% enhancement in power utilization proficiency. Further, hardly people noticed, back in January 2016, Microsoft begun utilizing a very particular subset of reinforcement learning called contextual bandits to pick up customized features for MSN.com; something various machine learning frameworks had not been able to do. The contextual bandit framework expanded clickthrough by 25% and a couple of months after the fact, Microsoft transformed it into an open source Multiworld Testing Decision Service based on the Vowpal Wabbit machine learning framework, that you can keep running on Azure.

In contrast to contextual bandits, there isn’t only one procedure for reinforcement learning. There isn’t an institutionalized stage like the Multiworld Decision Service that you can use on your own issues. But in a lot of places, stages are being developed which can be used by researchers to carry out their experiments.

When we talk so much about AI, reinforcement learning isn’t new; the principal reading material covering it dates to 1998. What’s new currently, is that we have involvement with a few issues that are surely known, especially in the two territories of contextual bandits and imitation learning. In any case, we additionally require these new experimentation stages like Universe, Project Malmo and DeepMind Lab to give more scientists certain access and to assess solutions in a similar situation with benchmark progress.

A static dataset isn’t valuable for assessing more general reinforcement learning. Two unique operators will take two distinct directions through an environment. Rather, analysts require an expansive, different set of conditions that is likewise institutionalized so everybody in the field neutralizes them. Adaptable, diverse stages can serve the similar capacity as a depository for reinforcement learning tasks where we can assess and emphasize on thoughts coming out of research a lot quicker than before when we needed to confine the algorithms to easy assessment problems since more perplexing ones weren’t accessible. In the current scenario, we can take ideas to the stages and see regardless of whether they work to perfection.

In reality, each individual has distinctive knowledge, abilities and wants. It’s vital over the long haul to see how an AI professional might find out about those objectives and about the quirks and capabilities of the individual it’s working with and have the capacity to customize its assistance and activities to enable that specific individual to accomplish their objectives.

What do DATA SCIENTIST do?

2020-11-02T06:46:00.002-08:00

In this blog, you will discover what are the things that data scientists do?

Data science is not about making complicated models. It's not about making awesome visualizations. It's not about writing code. Data science is about using data to create as much impact as possible for your company.

Now, the impact can be in the form of multiple things. It could be:

In the form of insights.
In the form of data products.
In the form of product recommendations for a company.

Now to do those things, then you need tools like making complicated models or data visualizations or writing code. But essentially as a data scientist, your job is to solve real company problems using data, and what kind of tools you use we don't care.

Now there's a lot of misconception about data science, especially on the Web and I think the reason for this is because there's a huge misalignment between what's popular to talk about and what's needed in the industry. So because of that, I want to make things clear. I am a data scientist working for a reputated company and those companies really emphasize using data to improve their products.

So this is my take on what is data science. Before data science, we popularized the term data mining in an article called from data mining to knowledge discovery in databases. In 1996 in which it referred to the overall process of discovering useful information from data.

In 2001, William S. Cleveland wanted to bring data mining to another level. He did that by combining computer science with data mining. Basically, he made statistics a lot more technical which he believed would expand the possibilities of data mining and produce a powerful force for innovation. Now you can take advantage of computing power for statistics and he called this combo data science. Around this time

this is also when web 2.0 emerged where websites are no longer just a digital pamphlet, but a medium for a shared experience amongst millions and millions of users. These are web sites like MySpace in 2003, Facebook in 2004, and YouTube in 2005. We can now interact with these websites, meaning we can contribute post comments like upload share leaving our footprint in the digital landscape we call the Internet and help create and shape the ecosystem. we now know and love today. And guess what?

That's a lot of data so much data, it became too much to handle using traditional technologies. So we call this Big Data. That opened a world of possibilities in finding insights using data. But it also meant that the simplest questions require sophisticated data infrastructure just to support the handling of the data. We needed parallel computing technology like MapReduce, Hadoop, and Spark.

so the rise of big data in 2010 sparked the rise of data science to support the needs of the businesses to draw insights from their massive unstructured data sets. Then the journal of data science described data science as almost everything that has something to do with data, collecting analyzing modeling. Yet the most important part is its applications. All sorts of applications. Yes, all sorts of applications like machine learning

So in 2010 with the new abundance of data, it made it possible to train machines with a data-driven approach rather than a knowledge-driven approach. All the theoretical papers about recurring neural networks support vector machines became feasible something that can change the way we live and how we experience things in the world. Deep learning is no longer an academic concept in this thesis paper. It became a tangible useful class of machine learning that would affect our everyday lives. So machine learning and AI dominated the media overshadowing every other aspect of data science like exploratory analysis, experimentation, ... And skills we traditionally called business intelligence. So now the general public thinks of data science as researchers focused on machine learning and AI but the industry is hiring data scientists as analysts so there's a misalignment there.

The reason for the misalignment is that yes, most of these data scientists can probably work on more technical problems, but big companies like Google Facebook Netflix have so many low-hanging fruits to improve their products that they don't require any advanced machine learning or statistical knowledge to find these impacts in their analysis. Being a good data scientist isn't about how advanced your models are, it's about how much impact you can have with your work. You're not a data cruncher, You're a problem solver, You're strategists. Companies will give you the most ambiguous and hard problems. And we expect you to guide the company in the right direction.

So this is a very useful chart that tells you the needs of data science. Now, it's pretty obvious but sometimes we kind of forget about it now. At the bottom of the pyramid, we have collected. you obviously have to collect some sort of data to be able to use that data. So collecting, storing, transforming all of these data engineering efforts is pretty important and it's actually quite captured pretty well in media because of big data we talked about how difficult it is to manage all this data, we talked about parallel computing which means like Hadoop and Spark, stuff like that. We know about this. Now the thing that's less known is the stuff in between which is above cleaning. Surprisingly this is actually one of the most important things for companies because you're trying to tell the company, what to do with your product. So what do I mean by that?

So I'm analytics that tells you using the data, what kind of insights can tell me, what is happening to my users and then metrics this is important because what's going on with my product? You know, these metrics will tell you if you're successful or not.

So these things are actually really important but they're not so covered in media. What's covered in media

is this part. AI, deep learning. We've heard it on and on about it, you know but when you think about it for a company, for the industry. It's actually not the highest priority or at least it's not the thing that yields the most result for the lowest amount of effort. That's why AI deep learning is on top of the hierarchy of needs and these things may be testing analytics. They're actually way more important for industry. So that's why we're hiring a lot of data scientists that does that. So what do data scientists actually do?

Well, that depends on the company because of them as of the size. So for a start-up you kind of lack of resources, so you can only kind of have one DS. So that one data scientist, he has to do everything. So you might be seeing all this being data scientists. Maybe you won't be doing AI or deep learning because that's not a priority right now, but you might be doing all of these. You have to set up the whole data infrastructure, you might even have to write some software code to add logging and then you have to do the analytics yourself, then you have to build the metrics yourself, and you have to do A/B testing yourself. That's why for startups if they need a data scientist this whole thing is data science, so that means you have to do everything.

But let's look at medium-sized companies. Finally, they have a lot more resources. They can separate the data engineers and the data scientists, so usually in the collection, this is probably software engineering. And then here, you're gonna have data engineers doing this. And then depending on if you're medium-sized company does a lot of recommendation models or stuff that requires AI, then DS will do all these

Right. So as a data scientist, you have to be a lot more technical. That's why they only hire people with PhDs or masters because they want you to be able to do the more complicated things

So let's talk about a large company now. Because you're getting a lot bigger you probably have a lot more money and then you can spend it more on employees

So you can have a lot of different employees working on different things. That way

the employee does not need to think about this stuff that they don't want to do and they could focus on the things that they're best at. For example, me and my untitled large company, I would be in analytics so I could just focus my work on analytics and metrics and stuff like that. So I don't need to worry about data engineering or AI deep learning stuff. So here's how it looks for a large company Instrumental logging sensors. This is all handled by software engineers

Right? And then here, cleaning and building data pipelines. This is for data engineers. Now here, between these two things, we have Data Science Analytics. That's what it's called. But then once we go to AI and deep learning, this is where we have research scientists or we call it data science core, and they are backed by and now engineers which are machine learning engineers.

Anyways, so in summary, as you can see, data science can be all of this and it depends on what company you are in and the definition will vary. So please let me know what you would like to learn more about AI deep learning, or A/B testing, experimentation,..........

Depending on what you want to learn about, leave a comment down below so I could talk about it or I could find someone who knows about this and I can share the insights with you. So yeah, if you like this blog, don't forget to like, and subscribe. Hope you have a wonderful day. Hope this was helpful. But yeah, thanks for reading, Peace.

Implementation KNN- Classifier in Python

2020-11-01T01:15:00.003-07:00

As we previously examine the KNN that how it works and how to select the K for better outcomes and no overfitting. In this article, we will be going to code the python version for KNN and we will find the most immeasurable value of K to use for better outcomes.

Find the below code with explanation:

Import the necessary libraries which we will need for the future.

Now load the dataset from the local and find how rows look in it.

To download the dataset please refer to the following link.

Now select all the features and the target_class from the set and divide the set into test and train by 33% and 67%.

Now for a range values of K fit on the training dataset and then test on the test dataset and find the accuracy and then store it into some array.

Now plot the K-values with their corresponding accuracy and see which value is best.

Now we have our model and we can predict any given unknown value with more accuracy because now we know the best value of k.

Thanks for browsing my pattern, and I hope it benefits you in theory and in practice!!!!

KNN(K-Nearest Neighbour) algorithm, maths behind it and how to find the best value for K

2020-11-01T01:14:00.001-07:00

KNN is a powerful classifier and a regressor. yes, you got it right we can do both regression or classification by this algorithm. For its implementation in python please visit this link.

What is KNN and how it works:

Let’s head by setting some definitions and notations. We will take x to denote a feature and y to denote the target.

KNN falls in the supervised learning algorithms. This means that we have a dataset with labels training measurements (x,y) and would want to find the link between x and y. Our goal is to discover a function h:X→Y so that having an unknown observation x, h(x) can positively predict the identical output y.

Working

First, we will talk about the working of the KNN classification algorithm. In the classification problem, the K-nearest neighbor algorithm essentially said that for a given value of K algorithm will find the K nearest neighbor of unseen data point and then it will assign the class to unseen data point by having the class which has the highest number of data points out of all classes of K neighbors.

For distance metrics, we will use the Euclidean metric.

Finally, the input x gets assigned to the class with the largest probability.

For Regression the technique will be the same, instead of the classes of the neighbors we will take the value of the target and to find the target value for the unseen datapoint by taking an average, mean or any suitable function you want.

Ideal Value for K

Now most probably, you are wondering how to decide the value for variable K and how it will affect your classifier. Well, like most machine learning algorithms, the K in KNN is a hyperparameter that you, as a data scientist, must decide in place to get the most suitable fit for the data set.

When K is small, we are holding the region of a given prediction and pushing our classifier to be “more blind” to the overall distribution. A small value for K provides the most adjustable fit, which will have low bias but high variance. Graphically, our decision boundary will be more irregular. On the other hand, a higher K averages more voters in each prediction and hence is more flexible to outliers. Larger values of K will have smoother decision boundaries which means lower variance but increased bias.

Improvements

An easy and mild approach to change skewed class distributions is by implementing weighted voting.
Changing the distance metric (i.e. Hamming distance for text classification)
Dimensionality reduction techniques like PCA should be executed prior to applying KNN and help make the distance metric more meaningful.

Thanks for browsing my pattern, and I hope it benefits you in theory and in practice!!!!

Logistic Regression, Maths behind it and where to use it

2020-11-01T01:12:00.002-07:00

Logistic Regression:

Logistic regression is a classification algorithm used to determine the values for the discrete classes. Unlike the linear regression which usually outputs the continuous value, the logistic regression uses sigmoid function and it will output the probability which will then mapped to the two or more classes.

There can be two types of logistic regression possible which are:

Binary: (having two classes)
Multi-variable: (having more than two)

First of all, let us understand the binary logistic regression and then it will be too easy to grasp what is multi-variable logistic regression.

Binary Logistic Regression:

As notified that name binary, so we will trade with the two classes primarily, we will just determine the probabilities and based upon some rule we will determine which class it belongs to mostly we will do like if p(x) > 0.5 then 1st class else 0th class. So we will decide the class like this simply. For predicting the probability we will use sigmoid function in the logistic regression which looks like

s(z) = output between 0 and 1 (probability estimate)
z = input to the function (your algorithm’s prediction e.g. MX + b)
e = base of natural log

Multiclass Logistic regression:

Instead of only two classes, we will extend our boundary so that now we have n classes from 0 to n-1 Essentially we run binary classification multiple times, once for each class.

Method:

Divide the problem into n binary classification problems
For each class…
Predict the probability.
prediction = <math>max(probability of the classes

For each sub-problem, we select one class and lump all the others into a second class. Then we take the class with the highest predicted value.

Future Learning:

Tolles, Juliana; Meurer, William J (2016). “Logistic Regression Relating Patient Characteristics to Outcomes”.
Walker, SH; Duncan, DB (1967). “Estimation of the probability of an event as a function of several independent variables”.

Linear Regression, Maths behind it and Best Reason for using least squares

2020-11-01T01:10:00.001-07:00

Linear Regression

Linear Regression is quite a way that will attempt to fit a line through observed variables (x,y). If we look into the graphs it will try to find the line which is more touching to the point (x^i,y^i) for all I in the dataset, y as a linear function of x and it will seem like

As it will try to prophesy a line most possible it will do by the least-squares method which is going to decrease the squares of the distance from the line for all the points.

Catholic rule of Least Squares

As the method will try to find the line Y = MX+ b and for solving it by math just find the distance from the line by putting the point in the equation and all the distances and just minimize it by taking derivative and assigning it to the zero. The general equation looks like

and to view it on the graph it will look like

Why we use least squares

One purpose is that the equations required in solving for the best-fit line are easy to solve. The only satisfying reason for using it will be proved by the gauss theorem which states

Suppose y=f(x). We measure values x^i,y^i and compute errors y^i-f(x^i). If these errors are independent and normally distributed, then consider, for any possible linear function f the probability P^f of getting the measurements (x^i,y^i) if y=f(x) were the correct model. The least-squares line is the line for which P^f is maximized.

This is the only best reason why we used it over the other methods.

Future Readings

Björck, Å. (1996). Numerical Methods for Least Squares Problems. SIAM.
Kariya, T.; Kurata, H. (2004). Generalized Least Squares. Hoboken

I hope this helps you in understanding you, linear regression easy way.

Naive Bayes Algorithm: Easy way understand like expert

2020-11-01T01:08:00.001-07:00

Bird’s eye view

In this article, you will be going to learn the Naive Bayes algorithm easily and after this, you can also talk to anyone about this like an expert.

Algorithm

It is the algorithm that is working on the Bayes theorem which describes the method to find the probability of an action or event based on prior knowledge.

For example, a ball may be considered to be a ball if it is round, and about 5 inches in diameter. As we can see these features may describe the ball but these features are independent while contributing that ball is a ball. That’s why we call it a NAIVE.

How it works

All you have to do is you just have to convert the data you have into the frequencies for all the classes and have to compute the probability of each case possible then you have to apply the Bayes theorem for prediction as simple as this. As you can see you just have to apply the formulae.

When you should use this

Suppose a situation comes, where you have to classify the data points into some classes and you know that there are too many points then easy way or you can say at sudden I will prefer you to do it by naive Bayes algorithm because it is too fast then others classification algorithm.

Decision Tree for classification with example and why or why not we use them.

2020-10-31T23:50:00.007-07:00

What is a Decision Tree

A Decision Tree is a type of supervising algorithm which indicates that in the training set you know the input variable ( features ) and their corresponding target. Decision Tree can operate as both classifiers or regressors. In this method, we split the dataset into two or more uniform sets based on the most significant splitter/differentiator in the dataset features.

Example:-

Let’s say we have a sample of 50 students with three variables Gender (Boy/ Girl), Class( X/ XI) and Height (5 to 6 ft). 20 out of these 50 play cricket in rest time. Suppose you want to find on unknown dataset which contains all the features(Gender, class, height) that he/she will play or not in rest time.

This is where decision tree supports, it will separate the students based on all values of three variable and identify the variable, which creates the best uniform sets of students like below:

Terms you should know

Root Node: It describes the complete dataset and this additional gets split into two or more uniform sets.
Splitting: It is a process of dividing a node into two or more sub-nodes.
Decision Node: When a sub-node splits into further sub-nodes, then it is called the decision node.
Leaf/ Terminal Node: Nodes do not split is called Leaf or Terminal node.

Why we should use

Clear to Understand quickly.
Beneficial in Data exploration.
Less data cleaning required.
Data type is not a constraint.

Why not

It may lead you to an overfitting problem.
Not a useful approach for continuous variables.

Feel free to share your knowledge, suggestions, and opinions in the comments section below.

Random Forest and how it works

2020-10-31T23:04:00.004-07:00

Random Forest

Random Forest is a Machine Learning Algorithm based on Decision Trees. Random forest works on the ensemble method which is very common these days. The ensemble method means that to make a decision collectively based on the decision trees. Actually, we make a prediction, not simply based on One Decision Tree, but by an unanimous Prediction, made by ‘K’ Decision Trees.

Why should we use

There are four reasons why should we use the random forest algorithm. The one is that it can be used for both classification and regression businesses. Overfitting is one critical problem that may make the results worse, but for the Random Forest algorithm, if there are enough trees in the forest, the classifier won’t overfit the model. The third reason is the classifier of Random Forest can handle missing values, and the last advantage is that the Random Forest classifier can be modeled for categorical values.

How does the Random Forest algorithm work

There are two stages in the Random Forest algorithm, one is random forest creation, the other is to make a prediction from the random forest classifier created in the first stage. The whole process is shown below, and it’s easy to understand using the figure.

In the first stage, we will build the random forest:

Randomly select “K” features from total “m” features where k << m
Among the “K” features, calculate the node “d” using the best split point
Split the node into daughter nodes using the best split
Repeat the a to c steps until “l” number of nodes has been reached
Build forest by repeating steps a to d for “n” number times to create “n” number of trees

In the next stage, with the random forest created, we will make the prediction. The random forest prediction pseudocode is shown below:

Takes the test features and use the rules of each randomly created decision tree to predict the outcome and stores the predicted outcome (target)
Calculate the votes for each predicted target
Consider the high voted predicted target as the final prediction from the random forest algorithm

The process is easy to understand, but it’s somehow efficient.

Where should we use

The random forest can be used in the: Banking, Medicine, Stock Market and E-commerce:

For the application in banking, the Random Forest algorithm is used to find loyal customers.
For the application in medicine, the Random Forest algorithm can be used to both identify the correct combination of components in medicine.
For the application in the stock market, the Random Forest algorithm can be used to identify a stock’s behavior and the expected loss or profit.
For the application in e-commerce, the Random Forest algorithm can be used for predicting whether the customer will like the recommend products, based on the experience of similar customers.

Feel free to share your knowledge, suggestions, and opinions in the comments section below.

How to learn Data Science / Machine Learning / Artificial Intelligence — (Schedule for you to become expert)

2020-10-31T22:56:00.005-07:00

Bird’s eye view

To become good at something we need to start from the beginning so if you want to expert in Machine learning / Data Science / Artificial Intelligence you will have to start from the beginning. Some words from the experts: “Everyone is learner so don't panic go with flow”.

For newcomers, programming languages like python, statistics, machine learning algorithms.
For transitional, advanced machine learning algorithms are required to be understood.
Practicing with datasets and an online kaggle profile are helpful in practicing your skills.

Learning Path — Schedule for you

Getting started →Statistics → Programming language →Basic of ML →Practice.

It will take you 6–7 months to be an expert or as an expert then you can compete anywhere in the competition. Let’s talk about the steps of how you should proceed

Getting Started: For this, you have to know why do you want to learn ML or data science, for doing that just connect with experienced people in the field of ML or data science on some social platforms like Linkedin, Facebook. I preferred you to connect on Linkedin.
Statistics: For this learn some statistics course from Stanford or MIT online course they are the best.
Programming language: You should know how to code in python or R.I code on python so I prefer python over anything
Basics of ML: Read about algorithms used in machine learning very often.
Practice: After all the steps above now you are ready for going out and show your talent or what you had learned. For that just practice on Kaggle or Hacker earth also hosts some machine learning contest.

Just do above 5 steps and you will be pro :)

Top Algorithms for Machine learning Geeks: Only for experts :)

2020-10-31T22:55:00.006-07:00

So in my opinion to be an expert in machine learning like the other expert geeks one should know the listed below algorithms :

Naive Bayes Algorithm
Linear Regression Algorithm
Logistics Regression Algorithm
KNN algorithm
Decision Tree
Random Forest
SVM
K means Algorithm
Artificial Neural Networks

Learn about the above algorithms and you can also be an expert in machine learning like others in a few days.

How To Get Data Science Job/Internship

2020-10-31T22:38:00.002-07:00

If you all guys out there looking for tips and tricks about how to get a data science job or internship you want to apply for?

Before telling you how to get you should know how to be an expert in it so for that please read my article.

You have come to the right place! Nowadays reports and publications consistently name ‘data scientist’ as one of the preferable jobs. While there are many articles about the set of skills you need to get the data scientist position, we wanted to focus on the students who crave working in this prosperous field.

The data science will be a good job if you want to get chill in your life because according to many articles it will be the best job in 2020.

Just follow the below steps for getting jobs/ internship

Learn about basic stuff you do not need to research topics. so read-only about regression or classification and from top neural network. That will be enough study for fresher to know, another thing you can learn on the job.
After knowing the basics now you have to do some project to practice those skills go to Kaggle and do basic problems that will give you an idea of how to tackle the problem in real life.
Now you are ready to give an interview for getting the job, now for this, you have to show to the recruiter that you know the basics and you also did the problems based on real-life on kaggle.
Last thing I want to say don't forget to revise DATA STRUCTURES AND ALGORITHM because it will help in everything.

Four Points just join them.

Now you know the four points that will help now you just have to solve the puzzle( for getting the job ) step by step as I told.

How to be expert in Machine Learning

2020-10-31T22:37:00.004-07:00

One should know how to play with data

First of all the main thing, we need to be focused on before stepping into machine learning is that we should know how to play with data. Because believe all the geeks sitting out there doing machine learning and other cool stuff is all because they know how to play with data.

Note: Hereby playing with data means is you should know how to manipulate the data or reshape the data into something that model can understand and make sense from it and able to predict the new value.

So what one fresher or maybe who know something but not pro should need for playing with data is only some tools like

Pandas — For Python

By pandas, one can do anything that he/she can do with data. literally anything.

Applying ML Models

After you need the data you wanted to feed the model you will have to do only 2% of the work you did for reshaping the data. Because now you have to use 2 golden words of machine learning FP

.Fit()

.Predict()

It will be going to some for any model you use just import and use these two functions for train and predict.

So to become an ML expert you need to know data manipulation only rest is just 2 golden words.

Airflow for Beginners — II

2020-10-31T22:36:00.009-07:00

As in the previous article Airflow for beginners — I we learn about the airflow and how to install the airflow on local. In this article, we will learn about the DAG and how to use the web airflow for scheduling the work and managing it.

Let’s start with the DAG and then we will learn about the web airflow.

DAG (Directed Acyclic Graph)

DAG is a directed graph without a cycle in it. It has a finite number of edges and nodes. In which we can know which thing(node) will come first and we will know the sequence of the nodes. so just think as each node is a work( that you want to execute ) so we can represent the sequencing of the work by the DAG and that’s why DAG is an essential part of the airflow.

WEB AIRFLOW

So now as you saw that on

localhost:8080

if you open it will look like:

now, these are the example DAG or work that is added already so for adding your own script(DAG) just follow simple steps mentioned below:

Just go to the airflow directory in terminal (it will be in the root) then just make the new folder by the name “dags” because in the config file the path is set for dags for taking the dags.
Now only left is how to create the dag that we will now go through by taking the example of executing two files one after another. We are doing this example because usually in your workspace you have to execute some files at some scheduled interval.

2.a. create two files in my case I created two python scripts for executing.

2.b. now you have to write a python script for creating DAG

so let's go in details

First import the libraries we need

import airflowfrom airflow.models import DAGfrom airflow.operators.bash_operator import BashOperatorfrom airflow.operators.dummy_operator import DummyOperator

2. Now initialize the dag argument that we need to pass and initialize the dag.

def_arg = {'owner': 'RD_TYGA','start_date': airflow.utils.dates.days_ago(2),}dag = DAG(dag_id='rahul_dhawan_dag',default_args=def_arg,schedule_interval='0 0 * * *',) # run every minute

3. Now create tasks you want to execute.

t1 = BashOperator(task_id='python_task_1',bash_command='python /Users/rahul/Desktop/goal.py ',dag=dag)t2 = BashOperator(task_id='python_task_2',bash_command='python /Users/rahul/Desktop/goal2.py ',dag=dag)

4. Now add the order for the run.

t1 >> t2

now just save it and run this command

python your_file_name.py

and after 1 or 2 minutes you will see that dag in the webserver.

Here you are ready with scheduling now just see how it's running.

Thank you for your support.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

DBSCAN Clustering Algorithm-with maths

2020-10-31T22:36:00.008-07:00

DBSCAN is a short-form of Density-Based Spatial Clustering of Applications with Noise. It is an unsupervised algorithm that will take the set of points and make them into some sets which have the same properties. It is based on the density-based clustering and it will mark the outliers also which do not lie in any of the cluster or set.

There are some terms that we need to know before we proceed further for algorithm:

Density Reachability

A point “p” is said to be density reachable from a point “q” if point “p” is within ε distance from point “q” and “q” has a sufficient number of points in its neighbors which are within distance ε.

Density Connectivity

A point “p” and “q” are said to be density connected if there exists a point “r” which has a sufficient number of points in its neighbors and both the points “p” and “q” is within the ε distance. This is a chaining process. So, if “q” is neighbor of “r”, “r” is neighbor of “s”, “s” is neighbor of “t” which in turn is neighbor of “p” implies that “q” is neighbor of “p”.

Algorithm

Let X = {x1, x2, x3, …, xn} be the set of data points. DBSCAN requires two parameters: ε (eps) and the minimum number of points required to form a cluster (minPts).

1) Start with an arbitrary starting point that has not been visited.

2) Extract the neighborhood of this point using ε (All points which are within the ε distance are neighborhood).

3) If there are sufficient neighborhoods around this point then the clustering process starts and the point is marked as visited else this point is labeled as noise (Later this point can become the part of the cluster).

4) If a point is found to be a part of the cluster then its ε neighborhood is also the part of the cluster and the above procedure from step 2 is repeated for all ε neighborhood points. This is repeated until all points in the cluster is determined.

5) A new unvisited point is retrieved and processed, leading to the discovery of a further cluster or noise.

6) This process continues until all points are marked as visited.

Advantages

1. It does not require a-priori specification of the number of clusters.
2. Able to identify noise data while clustering.
3. DBSCAN algorithm is able to find arbitrarily size and arbitrarily shaped clusters.

Disadvantages

DBSCAN algorithm fails in case of varying density clusters.
Fails in case of neck type of dataset.

Airflow for Beginners — I

2020-10-31T22:35:00.003-07:00

PHASE:- 1 ( Introduction and Installation )

Introduction

Apache Airflow is a platform for programmatically author schedule and monitor workflows. It is one of the best workflow management system.

Airflow was originally developed by Airbnb to manage its data-based operations with a fast-growing data set. Airflow is undergoing incubation at Apache Software foundation as Airbnb has decided to open source it under Apache certification.

Apache airflow makes your workflow a little bit simple and organized by allowing you to divide it into small independent task units. Easy to organize and easy to schedule. Your entire workflow can be converted into a DAG with Airflow.

Once your workflows are defined by your code it becomes more maintainable. With the feature-rich user interface your workflow pipelines can be easily visualized, monitored and troubleshot.

Airflow also provides a rich set of command-line utilities that can be used to perform complex operations on DAG.

In short, airflow is a platform where you can see and schedule your run for the script.

Installation

you can simply instaled the airflow by PIP.

pip install apache-airflow

so this command will installed your airflow local.

To start the airflow run the following command and you are set up with airflow.

airflow initdb   # initating the database for storing the DAGsairflow scheduler  # initating the scheduler for triggering on scheduled timeairflow webserver  # initating the webserver for UI

after this execution, your airflow is fully setup. You can find it in:

localhost:8080

so now you have fully set up with airflow and about execution and creating DAG, we will see in the next article.

THANKS FOR YOUR SUPPORT

Overfitting and Underfitting ( BUG IN ML MODELS )

2020-10-31T22:34:00.006-07:00

The worst performance of the ML( Machine Learning ) Models is due to the Overfitting and Underfitting largely. As in the past, we discovered that the generalization is the idea that every model should do but the overfitting and underfitting will go along with them so we generally have to more aware that the model should not do the overfitting and underfitting. A vital factor in determining the objective function from the training data is how well the model generalizes to new data. Generalization is important because the data we receive is only a sample, it is incomplete and noisy.

Overfitting:-

Overfitting is the bug in the ML which means that the model trains itself very much on the training data set.

Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively influences the appearance of the model on brand-new data. This actually gives us the idea that the noise in the training data set is picked up and learned as ideas by the model. The dilemma is that these ideas do not apply to the test data set and negatively impact the model’s ability to generalize.

Overfitting is more likely with nonparametric and nonlinear models that have more flexibility when learning a target function. There are many models that can face this problem for e.g. Random forest.

For example, decision trees are a nonparametric machine learning algorithm that is very flexible and is subject to overfitting training data. This problem can be approached by clipping a tree after it has learned in order to remove some of the detail it has picked up. This problem can be seen many times in the decision trees.

Underfitting in Machine Learning

Overfitting is the bug in the ML which means that the model can’t generalize to the new data and can not fit the training data set also.

An underfit machine learning model is not the best model and will be clear as it will have poor performance on the training data and will also perform worst on the test data set.

Underfitting is often not discussed as it is easy to detect given a good performance metric. The solution is to move on and try alternate machine learning algorithms. Nevertheless, it does provide a good contrast to the problem of overfitting. So my only suggestion for you is to try some new model and it will help many times.

As we saw the defination of both underfitting and overfitting and we are sure that the underfit is not that big threat it is solvable by just trying the new model but the overfit is major bug in the machine learning models so to resolve this bug there are some few tips from my side to yours.

Overfitting — Solution for this bug

Overfitting is such a problem because the evaluation of machine learning algorithms on training data is different from the evaluation of the data set that we actually want the model to perform well.

There are two powerful procedures that one can use when evaluating machine learning algorithms to smash overfitting:

Resampling technique.
Validation dataset.

The common hot resampling technique is k-fold cross-validation. It allows us to train and test your model k-times on different subsets of training data and build up an estimate of the performance of a machine learning model on unseen data.

A validation dataset is simply a subset of your training data that you hold back from your machine learning algorithms until the very end of your project. After you have selected and tuned your machine learning algorithms on your training dataset you can evaluate the learned models on the validation dataset to get a final objective idea of how the models might perform on unseen data.

I hope this will help you in understanding the biggest bug in the machine learning model and also help you in to smash it. #bug_smash

Fuzzy sets & Fuzzy C-Means Clustering Algorithm

2020-10-31T22:33:00.009-07:00

All the classification algorithms that we studied in the past, most of them are based on soft-based computation. Soft based computation means as we are giving the prediction from algorithm that point j belongs to which set i ( where i can take any value from 0 to number of sets) but there is category also existed which we called fuzzy-based computation which basically means that the if there is point i and n sets it is not compulsory that it will belong to only one set but there is something called membership function which says that the point can belong to multiple sets. In nutshell, the output is basically the number which indicated the probability of the point to belong to a particular set.

Actually, if you can feel the difference between the simple set and the fuzzy set then you are on the right track otherwise in short note you can say simple set includes the elements which will belong to only one class but the fuzzy sets can contain the element which may belong to multiple sets. This rule is called fuzzy logic.

For e.g. let us take the example of the height of the person you can classify into two classes TALL or SHORT but now think in the terms of the

Soft based computation: You will say that if the input is greater than some threshold than TALL otherwise short.
Fuzzy based computation: If the input is very far from the threshold then we are very sure about the output But think like if there is input which is very near to threshold than practically you can not say it as TALL or SHORT then their fuzzy logic comes in the picture which will tell you about the probabilities. So the membership function will look like:

So by now, I think you must be very clear about the fuzzy sets.

Before getting into the FCM algorithm let us understand one more thing called Fuzzy-c partition. The fuzzy-c partition is like to partition the set S into C classes. Where the fuzzy-c partition is defined by tuple (µ, S) where µ represents the matrix of size (i*j) i is the number of points and j is the number of classes. The µ(i,j) represents the membership value of point i for the jth class. There are some rules that each class has to follow to be the class of the fuzzy-c partition.

The rules go likes:

Every row in the matrix sums should to one.
Every column in the matrix sums should less than the total number of points in the set.

So as we are done with setting up the base for going fully in the fuzzy c means algorithm. A little bit interesting fact about it is Fuzzy c-means (FCM) clustering was developed by J.C. Dunn in 1973, and improved by J.C. Bezdek in 1981.

Now the algorithm is similar to the k-means but it works differently.

Choose a number of clusters.

Assign coefficients randomly to each data point for being in the clusters.
Repeat until the algorithm has converged.
Compute the centroid for each cluster.

So the fuzzy-c means algorithm will not overfit the data for clustering like the k-means algorithm it will mark the data point to multiple clusters instead of the one cluster which will be more helpful than giving the point to the one cluster.

Why we should use Fuzzy C instead of k means:

The main reason it works equal to the k means algorithm but the only thing that differentiates it from the others is that it will be the best to differentiate the point which can be assigned to multiple clusters.

I hope you guys learned from this and if not, please feel free to ask questions.

Thanks.

Feature Selection — (Exploration Study)

2020-10-31T22:33:00.007-07:00

These days there is a hype in the world that if one has more features then we have better discrimination in the model but this hypothesis will not hold in all the situations. Actually the performance of the model vs no of features will decrease when the number of features increases and it will look like

As in Instance-Based learning methods (like k-means) where features are very important in the case of k-means extra features will add noise in the distance calculation. To handle this problem we should have to avoid irrelevant features while selecting the features because they will add the noise to the model learning. These extra features will generally affect the performance when we have a limited training dataset. In the data science world, we call this a curse of dimensionality

Curse of dimensionality in simple words is when we have training dataset with large number of training features and more computational resource to learn.

The only solution to this big threat is feature reduction. There are two ways to do feature reduction

Feature Selection
Feature Extraction

In this module, we are going to explore the feature selection method of feature reduction. The feature selection is the way in which we have to do select the feature set from all the features such that it will be a subset of the bigger one.

There are 2^n possibilities of the feature reduction set. If n is the size of the features set. For starting this procedure you have to first create the dataset in which all the features are highly uncorrelated.

Feature Selection:

To do the process there are two ways exists:

Forward Selection: In this, we will start with the empty feature list and we will try each feature combination and select the best combination.
Backward Selection: In this, we will start with the full feature list and we will try each feature combination and select the best combination.

As till now, we are working with the multiple combinations of the features and trying for optimization of the model. we can use the feature by feature and give them score then select those feature which will clear the cutoff those feature will be the best for the optimum solution/model.

There are many ways to give score when we are trying the single wise features which will help to select the best features which are :

Pearson correlation coefficient
F-score
Chi-square
Signal to noise ration
mutual information

These methods will help you to select the features which are really useful.

I hope all this will help you to select the feature space and built the finest model.

How to be a HERO in Machine Learning/Data Science Competitions

2020-10-31T22:32:00.005-07:00

At present to master machine learning models one has to participate in the competition which is appearing in various platforms. So how somebody who is new to ml can become a hero from zero. The guideline is in this article.

The idea for this is not too hard. Just patience and some hard work are required. I will take an example of a Competition that is just finished within top 10. So the competition generally gives you the problem in which some of the features are hidden because they want you to explore the data and come up with the feature that explains the target value. By exploring I mean to say the few things:

Look at the data. Get the sense of the data.
Find the correlation of all features with a target value.
Try new features made up of existing features.

Exploration needs some cleaning of the data also. Because in general, the host will add the noise into the data so that it becomes a trouble for us to achieve good accuracy. By cleaning I mean

Trading with NaN values.
Find and remove outliers from training.

One of the most essential steps is to select the features for training. To select the features one should know the correlation and how to generate the new features which are extremely impactful. These features can be mean of some features, result by adding some features, etc. There are many ways people can look at this.

Then the last part but not the least is to select the model. By selecting the model not means just take some model and train it. But the most important part is to train the model with some good value of its parameters. Because I saw that a little tweak in parameters will let you achieve high accuracy.

For beginners:
1- Start learning the concepts of math and statistics.
2- Learn programming tools for data science either R or Python.
3- Learn Machine learning algorithms (part 1)

And here are the resources for the previous list:
Descriptive Statistics from Udacity: https://www.udacity.com/course/intro-to-descriptive-statistics--ud827
A good statistics book (Optional): http://onlinestatbook.com/2/index.html
Introduction to Probabilities course from EDx: https://www.edx.org/course/introduction-probability-science-mitx-6-041x-2
Introduction to Probability book: https://www.stat.berkeley.edu/~aldous/134/grinstead.pdf
Intro to Inferential Statistics course from Udacity: https://www.udacity.com/course/intro-to-inferential-statistics--ud201

For professionals:
1- Learn the concepts of Deep learning and build at least one project.
2- Learn Data Visualization

Recommended Resources

Documentation! Pandas and Scikit-Learn both have extensive documentation with many interesting discussions.
For more information about Kaggle's success stories, I recommend the Kaggle blog, where they frequently interview competition winners regarding their approach and methods.
For detailed summaries of DataFrames, I recommend checking out pandas-summary and pandas-profiling.
For an excellent explanation of more advanced Random Forest usage, I recommend the Intuitive Interpretation of Random Forest.
Reddit for everything related: r/machinelearning, r/learnmachinelearning, r/datascience.

Here is a glimpse of the result that I achieved after following these steps.

I hope these procedures will help you. Let me get a clap if this helps you in some way.

Neural Network theory and implementation for Regression

2020-10-31T22:31:00.002-07:00

Introduction and background

Inthis article, we are going to build the regression model from neural networks for predicting the price of a house based on the features. Here is the implementation and the theory behind it. The neural network is basically if you see is derived from the logistic regression, as we know that in the logistic regression:

Formulae for Logistic Regression: y = ax+b

so for every node in each layer, we will apply it and after this output is from the activation function which will have the input from logistic regression and the output is output from the activation function.

So now we will implement the neural network with 5 hidden layers.

Implementation

1. Import the libraries which we will going to use

2. Import the dataset and check the types of the columns

3. Now build your training and test set from the dataset.

4. Now we have our data we will now make the model and I will describe to you how it will predict the price.

Here we are making the neural network 5 hidden layer with activation layer as `relu` and with their nodes in them. The final output is the house price.

5. Now we will fit our dataset and then predict the value.

Here we are ready with the neural network and we can use it to predict the house price.

Points to remember for future

Here you can select the number of the layer and then you can see the variance in prediction.
You can also select the activation function which will result in the variance in the accuracy.

Thanks for browsing my pattern, and I hope it benefits you in theory and in practice!!!!

How can I prepare for big companies as data scientist

2020-10-31T22:03:00.002-07:00

Here are some top companies in India that hire data scientists and pay cost effective salaries:

Analytics Giants

IBM- offers 14 lacs per annum
Deloitte- offers 13 lacs per annum
ZS Associates- 10 lacs per annum
Accenture- 11 lacs per annum

IT Giants

Amazon- 15 lacs per annum
Flipkart- 16 lacs per annum
Google- 20 lacs per annum
Paytm- 12 lacs per annum
Myntra- 13 lacs per annum

First of all, these companies are way too suspicious when it comes to hiring data scientists for the best teams that they have. You already know the thing that only a few can make it to select in their teams. Especially for beginners, it is very difficult to start their careers with these giants. But as dispirit as it sounds, this is not something that is impossible to achieve. Following the right way, you guys too can land your dream job as a data scientist. Here’s how you should go with the preparation:

Master dominant skills

Your top priority strategy should be to get perfectly experienced with all the needed and in-demand technologies. A possible candidate is expected to be well-skilled with multiple skills and technologies which covers:

Probability
Statistics
Statistical models
Programming Languages
Machine Learning
Understanding of Databases
Visualization Tools

All these are some of the skills that are principally plowed under job postings for these Giants. Possessing a broad knowledge of them is the most fundamental necessity.

All the best