All about Data Science

Skillset, topics, projects and virtual internships for DS

2020-08-27T20:36:00.004+05:30

This post is for those who are beginner and do not have any idea about topics that they need as a beginner DATA SCIENCE/ DATA ANALYST.

I am also facing the same problem before a year ago and till date I have some relevant knowledge about data science and also have some projects.

People are saying the we need so many skills like Mathematics, Programming language, some cloud concepts too. Actually they are right. Being a Data Scientist is not like being a web developer or a front-end developer that have limited skill set.

In this post I will tell you the exact topics that you need to learn at beginner level.

MATHEMATICS

Descriptive Statistics, distributions, hypothesis testing and regression analysis.

Bayesian Thinking, conditional probability, priors, maximum likely hood.

Vectors and matrices

Matrices operations

Eigenvalues and eigenvectors

Linear and non linear functions

Multivariable calculus

PROGRAMMING LANGUAGE(Python or R)

Data types, String operations, Expressions and variables

Tuple, List, Dictionaries and set.

Conditioning and branching( if, if-else, if-elif-else)

Loops(for, while)

Functions

Object and classes

File operations

Libraries - Pandas , Numpy, matplotlib and sklearn.

DATA ANALYSIS AND MACHINE LEARNING -

Importing and exporting of data.

Identifying and dealing with missing values.

Data normalization

Binning

Co - relation

Linear regression with single and multiple variables(prediction).

R - Squared and Mean Squared error.

Model Evaluation parameters.

Supervised and unsupervised learning.

Classification and algorithms( SVM, Logistics regression, Decision tree, K- nearest neighbours )

Clustering( k mean cluster, hierarchical cluster, and density based clustering).

These are the core skills that everyone should need. You can learn all skills on internet ( YOUTUBE IS THE BEST SOURCE) if any one want to go for paid courses. IBM DATA SCIENCE is one of the best.

There are more skills like MS - EXCEL , handling with database queries, story telling and presentation skills.

VIRTUAL DATA SCIENCE INTERNSHIPS -

KPMG DATA ANALYTICS INTERNSHIP

QUANTIUM DATA ANALYTICS

DATA@ANZ

I try to covered many resources and skillset that are required for an entry level Data Science / Data Analyst profile.

If anyone has query regarding post, do comments.

For projects - kavyansh.pandey

Thanking You.

KPMG Data Science Virtual Internship Task 2 | KPMG virtual internship

2020-08-03T02:44:00.003+05:30

I'm sharing the content of my PPT that I submitted as Task 2 solution for KPMG Virtual Internship.

Let me add some snapshots of my Presentation, ( It's not very good looking but consize all meaningful words)

Now what you have to do is just copy the below content and paste in a ms PowerPoint file.

In the place of Powerpoint, you can also make a document file.

-------------------------------
Sections

1.Data exploration
2.Model development
3.Interpretation and report

Data Exploration

Understand the characteristics of given fields in the underlying data such as variable distributions, whether the dataset is skewed towards a certain demographic and the data validity of the fields. For example, a training dataset may be highly skewed towards the younger age bracket. If so, how will this impact your results when using it to predict over the remaining customer base.

There are some limitations in the given datasets like some values are missing and some data types are different according to their value.

Furthermore, transformation of required data so that it is in an appropriate format for analysis. This may include steps such as ensuring that the data types are appropriate and rolling data up to an aggregated level. Or, joining in already aggregated ABS data at a geographic level to create additional variables.

Document assumptions, limitations and exclusions for the data; as well as how you would further improve in the next stage if there was additional time to address assumptions and remove limitations.

Model Development

1.First of all, we have to determine a hypothesis related to the business question that can be answered with the help of existing data. Perform statistical testing to determine if the hypothesis is valid or not.

2.Create calculated fields based on existing data, for example, convert the D.O.B into an age bracket

3. Test the performance of the model using factors like residual deviance, AIC, ROC curves, R Squared). Appropriately according to the model performance, assumptions and limitations.

Interpretation and Report

Visualization and presentation of findings. This may involve interpreting the significant variables and co-efficient from a business perspective.

With the help of this slide, we get an idea around the business issue and support our case with quantitative and qualitative observations.

_______________________________

Thats how you will make your presentation.
Pleanse Note - Sorry for the inconvenience, I'm receiving number of emails regarding ppt and this is not possible for me to send ppt each and every one.

Thank you.

Computer Graphics in Artificial Intelligence | Image Processing and Computer Vision

2020-08-01T02:13:00.002+05:30

If we look back, when there was only CLI( command line interface), number of computer users are less beacuse working in command line was quite typical.

Then Microsoft released Microsoft Windows and working on computer become very easier due to graphical user interface. Things became quite easier and people can perform any operation by just pointing specifics places, like work on specific applications etc.

That is about the beginning of GUI and from there, a new term came out and it was Computer Graphics.

Till today, there are number of Applications of Computer Graphics in various fields.

1. Medical field

2. Analytics and Visualization

3. Gaming

4. Artificial Intelligence (image proccesing and computer Vision)

5. Movies

And so on..

AI is the technology of present and future as well. Let's take a deep look, how computer graphics are useful in AI.

Two terms are there i.e; image processing and Computer vision.

Computer vision

It is the field of AI that trains computer to interpret and understand tha visual (graphical worlds).

Eg. Facial recognition, object detection etc.

These two procces can work in static as well as in real time.

Image processing

Process of performing some operations on an digital image is know as image processing. Operations like enhanced version of a given image, extracting some useful information, re - coloring of a given image etc.

Do you know, there is website "thispersondoesnotexist" has the million faces and all faces are totally artificial and generated by Image Processing.

Working of a Image Processing Model -

1. working on any image is totally based on working with pixels and vertices.

2. We need GPUs for this. GPUs has the number of precessing cores.

3. Some cores work with vertices and processing on vertices is done by using Co - ordinate systems.

4. Working on pixels take more Computation as the compare of vertices.

Thank You for reading.

What Data Scientist do | Jobs and Scope | Necessary Skills for Data Scientist

2020-05-22T04:14:00.000+05:30

1. Who is Data Scientist?
2. What is the job of Data Scientist?
3. Skillset required for Data Scientist.

Data Scientist

In a nutshell, Data Scientist is the person having statistical knowledge with some coding skills.

Data Scientist are those who analyze the previous records and fetch out the future insights. And that is the difference between Data Scientist and Data Analyst.

Analyst are those who tells about past insights of data and data Scientist are those, who can tell about the future related with those insights.

Let's take an simple example, suppose we have a some records of person salary related to their yearly experience. By having records, we can perform number of operations. Analysts can tell about the annually growth in salary, can give the different different visualization with clarity.

What will happen if we will want to predict the salary of new employees who was not working with company previously, here the role of Data Scientist comes as he is going to tell about the future insights.

Big tech companies like amazon or google , they all are doing their daily activities oj the basis of data they stored in their Information system.

And these work are done by the Data Scientist/ Data Analyst and Machine learning engineers.

Scope

Data is everywhere so scope of data Scientist in every fields like medical, information technology, finance and so on..

"According to Linked In Data Scientist job is the highest paying job from 2019."

Skills

If we talk about skills, there will not be any end As much you can get, will good for you. But some of important skills that Data Scientist should have are listed below.

1. Programming skills

2. Statistical knowledge

3. Multivariable calculus

4. Linear algebra

5. Knowledge of Data bases and cloud based architecture

6. Ms Excel

If person from non mathematical background are wanted to be Data Scientist, things will be quite challenging for them.

But programming languages like python, R are number of libraries that makes things quite easier.

Thank you for reading this post.

If you have any query, let me know in comment section.

Recommendation System | Types of recommendation system | Data sparsity | Scalability | Cold start

2020-05-17T19:39:00.000+05:30

Introduction

It is the system that recommends something to the persons or the machine. For example, suppose a person purchase two books i.e; machine learning and python. Most probably that person also interested in visualization or analysis. Book seller recommend these books to that person.

Now a days we are surrounded with the lots of applicantions. And most of them are worked on recommendation system. Applications on -

E- commerce platforms
Amazon and other e commerce websites are using recommendations system to recommend their products to the users.

Social platforms
Linked in, facebook and other social media platform are also used recommendation system to recommend new connections , products etc.

Suppose according to your linked in profile, you are a software engineer and have number of connections related to similar domain. So linked in automatically reccomend profiles that have occupation as software engineer.

Types of recommendation system

1. Content based

2. Collaborative filtering

Content based -

Content based recommendations system tries to recommend things to the user according to their profile, taste, preferences and their past activities.

Suppose we have dataset of number of movies and all movies have different diffrent genre like action, horrar, comics etc.

And we have a user profile with his interest and rating for different different movies.

We want to recommend new movies to the users.

But which movie user like most. That can be done by recommendation system.

Suppose we have four movies and user already liked first and second movies. So among third and fourth which movie user like most? Movie fourth is as same genre as movie first, and movie third is completely different genre. Most probably will user like movie fourth.

Collaborative filtering -

1. User based

2. Item based

In user based collaborative filtering, we took different user profiles having apporx same interest and preferences. And according to that preferences, It recommends products or movie to the target users.

In item baesd, we check the number of user records and check their previous reviews related to product, then compare their profile to the target user( for whom, we are going to recommend products) and then recommend products to the customer.

Problems related to collaborative filtering

1. Data sparsity - It refers to the difficulty during finidng sufficient active users for recommend a product to the target users.

2. Cold starts - It is not possible to recommend the best things to the new users.

3. Scalability - Due to large anount of past preferences and reviews, it would be a challange to recommend most appropriate products to the target user.

Advantage of recommendation system

1. Broader exposure to thr users.

2. Tends to more selling and increase revenues.

3. Too many options for selecting a product.

I hope, now you have a better understanding about recommendation system and it's working.

If you want a project on how to make recommendation system, write Project in comment section.

If you have any doubts, let me know in comment section or connect with me from the following link -

www.instagram.com/kavyansh.pandey

Thank you.

GRADIENT DESCENT IN MACHINE LEARNING | WORKING OF GRADIENT DESCENT ALGORITHM

2020-05-06T19:02:00.006+05:30

Gradient descent is the most popular algorithm and its concept is used in so many machine learning topics like regression etc.

After reading this post, you have an idea about the following topics.
1. What is gradient descent ?
2. How does it work ?
3. How gradient descent are used in machine learning.

Gradient Descent

Gradient descent is an optimization algorithm that works iteratively and use for finding local minimum or global minimum of an differentiable function.

Working of gradient descent

Suppose a person is on the top of the mountain(let A in above figure) and he want to come down to the ground.

From top of the mountain, he will go to that point which will less then his previous height.

After that again, he will search for that point which will less than his current height.

After the number of jumps, he will come down to the ground, that's how gradient descent algorithm works iteratively.

Let's see formally,

This is the mathematical formula for calculating gradient descent.

:= is the assignment operator that is assigning new point to the previous point.

If the value of 'alpha' should not be very large otherwise our model never reach on convergence.

Here is the 3d graph that is showing, how this algorithm works.

"The above showing picture is the snapshot from Andrew ng Machine learning"

In vertical axes, we have our cost function J(Theta 0, Tbeeta 1).

How Gradient Descent use in machine learning

As we know this algorithm is use for getting local mnimum, let's take an example of linear regression. We want to draw a best fit line in for a dataset... then gradient descent work for coming points near the line by applying above mentioned algorithm.

In this figure, we will minimize the distance of the red circle points ( the same apply for all points) from the best fit line.

I hope, now you have better understanding regarding gradient descent algorithm.

If you have any doubt, let me know in comment section.

Thank you for reading this post.

What is GPU and how does it work | GPU vs CPU

2020-05-02T04:28:00.000+05:30

GPU is the term that is using in so many fields but most of the persons are not very familier with this term.

1. What is GPU?
2. How it is differ from CPU?
3. How to make AI/ML models without having GPU inside your machine?

GPU

GPU is an acronym and it stands for GRAPHIC PROCESSING UNIT. It is use for doing complex computation task that CPU can not do with ease. Nvidia and AMD are the two main gpu manufacturer of the world. It is a hardware device that is used to run applications having heavy graphics or 3D gaming etc.

GPU is made for doing efficient calculation on graphical information.

Working of a GPU

GPU has number of processing cores running on slow speed. This slow speed is actual very fast because every cores deals with two functions, pixels and vertices. That is how we do parallel processing by using GPU.

Vertex processing is done by co-ordinate system(x,y) and GPU does geometrical calculation to reproduce dimensional space on screen.

Pixel processing requires more computational power than vertices processing.

How GPU is differ from CPU

CPU is made up with few number of cores like dual, quad, octa core and these core is the power to do any calculation or processing like start operating system, do any calculations, run any program or applications and other all task are done by cpu.

On the other hand GPU has a hundreds of cores. So it works parallel means number of operations at a time while CPU does serial processing.

Suppose we have code having 1000 of instructions.

All instructions send to CPU for processing.

Instructions from 600 to 1000 are very complex and require some additional support. So CPU can't process that instructions.

Then those instructions will send to the GPU for further processing.

USE of GPU

1. VDI ( Virtual Desktop Infrastructure) - GPUs are used to support high intensive graphic application.

examples - Movie Animation, Rendering, High resolution games etc.

2. Artificial intelligence - Two main subset of AI are machine learning and deep learning and GPUs are specifically created for these two purpose. Some GPUs are created to run Machine learning model ex - self driving car. Some GPUs are created to run neural networks model.

3. HPC ( High Performance Computing)

How to get GPU in your Machine

We can use the facility of GPU in our machine by using cloud.

Cloud is a platform where any one can store thier data and can do intensive computational work that they can't do in their machine. There are the following well known cloud Plateform -

AWS cloud

IBM cloud

GCP (Google Cloud platform)

Reasons for using GPU on cloud.

Every year new generation GPUs are coming and we can't change GPUs yearly as it will not very efficient. GPUs on cloud are maintained and time to time updated by the service provider.

It is also very cost effective and load on our system is also reduced.

Thank you for reading this post.

If you have any doubts, let me know in the comment section.

Continuous Vs Categorical Variables | Neccessary tools for Visualization of variables

2020-04-30T17:10:00.004+05:30

In a dataset, we have two type of variables.
1. Qualitative Variable
2. Quantitative Variable

Qualitative Variable

Qualitative variable are the group of values that represent any category that can be related to any aspects. It is also known as Categorical Variable.

Ex. Suppose we have a dataset having numbers of person having different different races( face colour) for example fair, white, Asia, dark and other.

So we assign and integer number for every categories like

Fair = 1

White = 2

Asian = 3

Dark = 4

Othet = 5

So we assign different - different integers with different different categories. And that's how we deals with categorical varible in our dataset.

Advantage of converting categories in to integers

Easy to apply mathematical functionality on thr categories.

We can easily summarise our dataset with the help of frequency distribution.

In frequency distribution, we havr two options.

1. Count

2. Percentage

Just take a look of following pictures.

In this table every categories are shown with count as well as proportion.

And It is convenient when and only we convert every categories in to interger.

Visualization for Categorical data.

We can use Barchart and Pie chart for categorical data, for for better undertanding, go for bar chart.

Quantitative Variable

Those variable in which, we can easily perform mathematical operation called quantitative variable.

There are two types of quantitative variable.

1. Continuous variable

2. Discrete variable

Continuous variables are those that contain all types of numbers including rational numbers.

Ex 1, 1.1, 3.4 etc.

Discrete varibles are those that don't contain rational numbers, only intergers are included in it.

Ex 1, 2, 3 .....and so on.

And for Visualization of quantitative variable, we use histograms, boxplot, frequency ploygon etc.

Histogram example.

Go for histrogram description.

Tnank you.

If you have any doubt, feel free to comment.

Show you support and subscribe our blog.

We also have a YouTube channel-

https://www.youtube.com/channel/UCAVf2yJz-K8LV7dJKES38fw

Univariate Analysis

2020-04-22T20:24:00.001+05:30

Analysis of Data is one of the most important activity for any organization. Big techie's like Amazon, Google are doing their daily activities on the basis of data.

There ard two reasons for analysis of data.
1. By analyzing data, we have an idea or stats of anything related to any prospectives.
2. Some times these analysis are used for taking future decisions.(For predicting outcomes)

Univariate analysis is the very first analysis technique that should be done in every analysis scenario.

What is Univariate Analysis ?

It is the simplest form of analyzing data. 'Uni' means one , so this is analysis by taking only one variable or we can say when data has only one variable, it is called univariate analysis.

It's main purpose is to give description about data.

How it's work ?

Step.1 It takes data.
Step.2 summarize data.
Step.3 Find different different patterns in the data.

Patterns like min value,max value, average, IQR etc.

Univariate variable

Variable can be any condition or categorical representation like age, height, number of goods etc that given into your data.

There are so many options by which you can describe your data like Barchart, Histogram, Frequency polygon, Pie charts etc.

example-

This is Visualization of systolic blood pressure of persons with frequencies.
From the above histograms, we can get the following stats .
Histogram is bellshaped (unimodal), mean or center is about 115 approx. Range is approx 169 (max -min)
And it has some outliers as distribution is going till 244 on x axis.

If you want to know more about Univariate Analysis, or have any doubts just drop a comment or connect with us from the following link.

- www.instagram.com/kavyansh.pandey

Thank you for reading this post.

Decision Tree | Machine learning | Supervised Learning Algorithm

2020-04-20T20:18:00.000+05:30

Decision tree is a supervised algorithm and use for solving classification and regression problem means it uses for prediction and forming clusters.

As we know modeling of machine learning model involves so many steps like collection of data, pre-processing on data, model training (In this phase, we give data to our model) and at last, we have testing phase where we test our model on number of input values.

It represents a function that takes input in the form of vector and return a decision as an output, so the goal of decision tree based machine learning model that can use to predict the class or value of the target variable by learning simple decision rules which is based on the previous data.

Target variable are those features, which a model want to predict or classify according to that feature.

Types of Decision Tree

There are two type of decision tree.

1. Categorical variable decision tree.

2. Continuous variable decision tree.

We can understand above mentioned two terms with the help of following example.

Let's take a student and based of his previous academic record, we want to predict whether he/she will pass in the next examination or not means (YES/NO), this is the example of categorical because no numeric terms are there for target feature.

But in the same example, if we want to predict the percentage will be achieved by that student, it's the example of continuous variable decision tree.

Algorithm

There are so many approach for decision tree algorithm but ID 3 is one of the most common decision tree algorithm.

ID 3 stands of Iterative Dichtomisers, means dividing a things (features) into two completely opposite things.

Algorithm iteratively divides attributes into two groups.

Calculate the entropy and information gain for each attribute.(this is use for selecting which attribute should divided next into subgroup and work as tree or decision node)

Terminologies in Decision Tree

Root node

Parent node

Child node

Splitting

Branch/Subgroup

leaf nodes

Formula's

Information gain = ( -P /(P+N)) ㏒₂(P / (P+N)) - (N /(P+N))㏒₂(N/(P+N))

Entropy = -p㏒₂p - Q㏒₂Q

where q = 1-p (probability)

P = positive outcomes

N = negative outcomes

P+N = total number of outcomes.

If the sample is completely homogeneous then entropy will be zero.

I hope, now you have better understanding of Decision Tree Algorithm and how's it work.

If you have any doubt, let me know in comment section.

connect with us from the following link-

www.instagram.com/datasciencewithkp27

Thanking You.

Precision | Recall | Accuracy | F1 score | ML model evaluation parameters

2020-04-12T19:40:00.002+05:30

Precision | Recall | Accuracy | F1 score

Evaluation of a machine learning model is based on four parameters and that are -

1. Accuracy
2. Recall
3. precision
4. F1 score

For understanding of above mentioned four parameters, firstly we need to understand what is confusion metrics?

Confusion metrics -

It is a two dimensional metrics and these two parameters are actual results and prediction results. It is used for describing the performance of a machine learning classification model with the help of True and False values.

It contain four type of values.

For understanding of below mentioned terminologies, let's take an example - "suppose a person is in the doctor clinic and want to know whether he is suffering from fever or his health is normal. so what are the possibilities of results."

TN (True Negative) Neither person actually suffering from fever nor doctor predict ,he is suffering.

TP (True Positive) person actually suffering from fever as well as doctor predict ,he is suffering

FP (False Positive) person is actually not suffering from fever but doctor predict, he is suffering.

FN (False Negative) person is actually suffering from fever but doctor predict, he is not suffering.

With the help of above mentioned (TN, TP, FP, FN), we evaluate all parameters.

Precision -

Precision is the ration of 'True Positives' with predicted 'Positives'. As clear from the above figure i.e; predicted positives contains True Positives and False Positives.

formula - TP / (TP + FP)

Recall -

Recall is the ratio of 'True Positives' with Actual 'Positives'. And actual positives contains True Positives and False Negatives.

formula - TP/ (TP + FN)

Accuracy -

It is the ratio of 'True Positives' and 'True Negatives' with Total observations.

formula - (TP + TN) / (TP + TN + FN + FP)

F1 score -

F1 score is the weighted average of Precision and Recall and can be calculated by following formula.

F1 score = 2*(Recall * Precision) / (Recall + Precision)

Accuracy and F1 score both are quite similar but F1 score is more useful than accuracy because accuracy works better when False Negative and False Positive have similar costs(value) so on that condition we should take F1 score for model selection.

I hope that now you have better understanding regarding terminologies for machine learning model evaluation.

If you found this post helpful , share with your friend. If you have any doubts, just drop a comment.

Connect with us -

www.instagram.com/kavyansh.pandey

Thank You for reading this.

Support Vector Machine with implementation | supervised learning Algorithm

2020-04-07T18:41:00.003+05:30

Support vector machine is basically used for supervised learning means it deals with classification and regression and the dataset which is used with this, is labeled. It is one of the popular algorithm that is used for classification problem.This algorithm is powerful as well as clear for complex task.

Definition
SVM is a supervised algorithm that classifies cases by finding a separator and it involves two steps that are following.
1.Mapping data into a high dimensional feature space.
2.Finding a separator.

Above figure have three blocks. In first block model is taking dataset ,after then model will be trained on that dataset, in second phase, it will be tested for some inputs and at last we will get the result.

Support vector machine generally deals with two types of data, first is linear separable and next one is non separable data.

It is an example of non linear separable data.
We use kernalling to solve non linear problem.

It is nothing but conversion of non linear into linear by using polynomials, sigmoid function etc.

eg. non linear data with one variable can be easily converted in to linear separable by using following function.
f(x) = [ x : x^2 ]

In this figure data can be separated by a single straight line, that's why it is linear separable data.

some terms that are related with SVM are given below.
1. Hyperplane - Straight line which divides datasset into two clusters and separate the whole dataset in to two parts is know as hyperplane. It is also known as decision boundary.
2. Margin- two straight lines are drawn parallel to hyperplane. The distance between hyperplane and each parallel lines is known as margin. It is denoted by D- and D+ respectively. It is the distance of parallel lines.
3. Support vectors - two data points that are nearest to the parallel lines is known as support vectors.

Significance of Margin

We only take that model which have greater margin because models having large margin are more accurate and it will be good for future prediction.
distance between margin is directly proportional to accuracy.

Advantage
1. Accurate in high dimensional space
2. memory efficient.

Disadvantage
1. leads to over-fitting.
2. No probability estimation.
3. Useful for small datasets.

Applications
1. Image classification.
2. text mining.
3. hand writing recognition.
4. genes expression classification.

Implementation of SVM algorithm

here I am going to upload snapshots of code , if you guys want the whole project based on Support vector machine, just drop a comment I'll explain the whole project to you.

After running this script you'll get the value 0.9635036496350365 which is almost 1, means our model has approx 96 % accuracy that will be better for future prediction.

If you have any doubt, just drop a comment.
connect with us on -
https://www.instagram.com/kavyansh.pandey

Thank You.

Types of Machine Learning | Supervised, Unsupervised and Reinforcement learning | Post 2

2020-03-25T17:10:00.000+05:30

Machine learning is classified into two types.

1. Supervised learning

2. Unsupervised learning

Most of the persons think that machine learning has one more field that is reinforcement learning. yes there are two more terms that is reinforcement learning and recommendation but these two terms are totally belong to deep learning and neural networks. As I told in the Post 1 that deep learning is the subset of machine learning that is why we consider three types.

1. Supervised learning

2. Unsupervised learning

3. Reinforcement learning

1. Supervised Learning

In this field of machine learning we have enough information about the data sets or labels and we take these data sets as an input into an algorithm, test and train data set and predict the outcomes.

labelled dataset means we have enough information about dataset and it has both input and output parameters.

"when the model is getting trained over the labelled dataset is know as supervised learning."

Regression and classification are the two approaches that are used in supervised learning.

In Regression, we give the datasets to the algorithm and predict the result and our result contains continuous values.

for example- any of your friend wants to sell his house so in what amount he should sell his house.

this type of problems can be solved using regression.

On right hand side, there is a figure in which a straight line and a quadratic function is set over data points.

Classification is similar to regression but in the place of continuous value , we get discrete values as an output.

example- we want to predict or check whether any person has breast cancer or not. for this we have tumor size as our dataset. and we make two assumptions for this case that is 0 for benign( normal) case and 1 for malignant case .These types of problem are stated as classification problem.

generally we use test and train method to solve supervised learning problem.

In above mentioned two examples there are limited number of attributes . In case that there are infinite number of attributes so we use SVM that is support vector machine.

2. Unsupervised learning

In this type, data doesn't have any labels or really no labels, so we are given dataset and we're not told what to do with it and we're not told what each data point is.

"Unsupervised learning allows us to approach problems with little or no idea what our results should look like. We can derive structure from data where we don't necessarily known the effect of variables."

we can derive this structure by clustering the data based on relationships among the variables in the data.

Clustering is the example of unsupervised learning.

for example- Take a collection of 1000000 different genes, and find a way automatically group these genes into groups that are somehow similar or related by different variables.

In clustering, unlabeled data provided but in classification, labeled data provided that is why these two put in unsupervised and supervised learning respectively. In both cases, we make group or cluster with coherent groups of data points that's why we can use k mean cluster algorithm in both cases.

3. Reinforcement learning

learning in neural networks means continuously updating the weights between the neurons of any specific network.

Reinforcement learning is quite similar to supervised learning as we make decisions same as we make prediction in supervised learning.

"Train neural networks model by the actions that agent perform and update set of actions in dataset."

In reinforcement learning, policy networks are present that can be implemented by policy gradients.

Agent calculate the bunch of experience.

After performing every action agent get a reward , if the value of reward is positive , we add that experience in dataset. In case we receive a neg gradient means that action is not very good for our final outcome.

Actions having negative record slowly filtered out from the program so probability of getting failure is very less so only positive actions are there. That's how program run by itself.

A term that generally used in reinforcement learning is CAP (credit assignment problem). It is the process of identifying among the set of actions chosen in an event that ones which are responsible for the final outcome.And moreover it is the best and worst set of decisions chosen over an event so that the best decisions are reinforced and worst penalized.

Now I hope that you have clear view about the types of machine learning.

If have any doubt, just drop a comment.

Connect with our Data Science community from the below mentioned link.

www.instagram.com/datasciencewithkp27

Thank you.

K - mean cluster algorithm | classification | machine learning

2020-03-23T00:50:00.000+05:30

As we know in classification problem, we deal with unlabeled dataset. Unlabeled data means we don't have enough information about the dataset. In this type of problems we have to make cluster with similar type of values(data).

We need an algorithm that can be automatically grouped the data into coherent subset and these groups are known as clusters.

What is k-mean cluster ?

It is an algorithm that is use for classification problem for grouping or clustering of coherent data points. It is divided into two steps.

1. Assignment of centriods.

2. Movement of centriods.

where centriods are the random points or dataset from given dataset.

Suppose we have unlabeled dataset in this visualization and our task is to grouped data into coherent cluster.

first of all we select two random points that is known as cluster as shown in figure.

after then we apply euclidean distance formula that is

√(Xo - Xc)^２+ (Yo-Yc)^２

where

Xo = observed value

Xc = centriod value

Yo = oberved value

Yc = centriod value , here all values is in the form of x,y pair because every point in figure is in two dimension.

We calculate values for every data point and put that value( data point) with minimum valued cluster as we take initially.

Replace that cluster value with new cluster value by mean of recently got value and initially taking value.

This algorithm is repeatedly work so after number of iterations we will get following clusters.

This two circle (clusters) will get after number of iterations and these clusters will be coherent groups.

Algorithm:

Input :-

k ( number of clusters , it can be one, two, three and so on... )

training set ( x1, x2 ,x3.......data points)

for i=1 to m

c(i) = index (from 1 to k) of cluster centriod closest to x(i)

for k=1 to K

𝜇k = average of points assign to cluster k.

suppose we have four cluster then

𝜇k = (𝜇1 + 𝜇2 + 𝜇3 + 𝜇4)/4

If in any case that centriod don't have any points, we just eliminate that centriod.

Why use k - mean cluster ?

k - mean clustering algorithm is one of the most popular algorithm for solving classification problem and it is also easy to implement as it use only two formulas euclidean distance and normal mean.

This algorithm is also used for solving non separated clusters problem.

suppose we have a large data set of height and width. our task is to separate small, ,medium and large size shirts from that data points. This problem can be solved with the help of k - mean cluster algorithm.

So this is all about k - mean cluster algorithm.

If you have any query, just drop a comment.

Connect with our Data Science community -

www.instagram.com/datasciencewithkp27

Thank You.

Starting with Machine learning | Introduction | Post 1

2020-03-20T16:19:00.000+05:30

In this series of articles, we are to going to have basic understanding of machine learning. We will start from introduction then we will move forward towards the types of machine learning. After then we will go in depth in every topic in easiest way.

We will also discuss some tools for implementing the ideas behind concepts.

WHAT IS MACHINE LEARNING ?

In this time ,there are several terms that we hear in our daily life i.e; AI , ML, DEEP LEARNING and so on..
Some person says that these terms are totally different but some where they are related to each other. Machine learning and deep learning is the subset of AI that is nothing but Artificial Intelligence.

We involve with Machine Learning in our daily life . Let's take some examples.. Whenever we type or search something in google or bing then google or bing search engines use machine learning algorithms to rank up the websites.
Whenever you see some tags (typically with square shape ) on you and your friends face in the pictures that save in phone or post on social media, it is also done by machine learning.
There are number of activities in daily life like using google map, using any e commerce website,scrolling instagram or using Netflix and so on....we involve with machine learning.

It is the picture showing facial recognition of several persons, done by machine learning algorithm.

DEFINITION

According to Arthur Samuel(1959):

"Machine learning is the filed of study that gives computers the ability to learn without being explicitly programmed."

According to Tom Mitchell(1998):

"Well-posed learning, A computer program is said to learn from experience E with respect to some task T and some performance measure P, if it's performance on T, as measured by P, improves with experience E."

there are numbers of definition given by the different authors but above mentioned definition are well known and standard.

SOME OTHER EXAMPLES

have you ever wonder why amazon always show products that you frequently searched, how our email system automatically put some number of mails in side spam folder which are not very useful to us..how every person has different google feed inside their phones...

Well it all are caused by MACHINE LEARNING algorithms.

I hope that now you have basic understanding of what machine learning exactly is...

Thank You.

If you have any doubt, feel free to drop comment or connect-

https:www.instagram.com/kavyansh.pandey

Difference between DFA and NDFA | Automata | Theory of computation

2020-03-12T11:57:00.000+05:30

What are DFA and NDFA ?

What are the major differences between DFA and NDFA?

Which one is more powerful?

Examples

DFA is acronym of deterministic finite automata and it is subpart of finite automata.

Definition of DFA

A deterministic finite automata can be defined with the help of 5 tuples(signatures).

That are Σ ,δ, q0 ,Q , F.
where

Σ = set of inputs / alphabets

δ = transition function ( Σ × Q = Q )

q0 = initial state Q = set of states

F = set of final states

Definition of NDFA

A non deterministic finite automata can be defined with the help of following 5 tuples.
i.e; Σ ,δ, q0 ,Q , F.
where
Σ = set of inputs / alphabets

δ = transition function ( Σ × Q = 2^Q )
"here transition function is in the form subsets.
and 2^Q showing for single input symbol , there will be multiple output or states."

q0 = initial state

Q = set of states

F = set of final states

Difference between DFA and NDFA

In DFA there is a single output state for a single input but in the case of NDFA there is multiple output for single input.
DFA is more complex to construct in the comparison of NDFA.
NDFA is more powerful in the comparison of DFA.
NDFA is less efficient in the comparison of DFA.

Figures

In this figure, q0 is initial state and here a and b are inputs and qf is final state.
so on 'a' inputs we have single output state i.e; qf.
so this is DFA.

In this figure, q0 is initial state and here a is input and qf is final state.
so on 'a' inputs we have multiple output states i.e; q0 and qf.
so this is NDFA.

Both DFA and NDFA is the part of finite automata.

If have some doubts feel free to connect.
https://www.instagram.com/kavyansh.pandey

Thanking you.

Knapsack & Backtracking | Design & Analysis of Algorithms | post 2

2019-10-21T15:05:00.000+05:30

Some important applications of dynamic programming.

0/1 knapsack problem-

In this problem , knapsack is just like a bag and we have to fill items in this bag by making sure two things -

1) Value of the bag should be maximum.

2) Wight of the bag should be minimum.

" In numerical problems, weight is given in numeric form so be sure that the weight of knapscak should not be exceeded from given Total weight. "

Differences between knapsack and 0/1 knapsack problem is that we can split items in parts for the first one but we cant split item in 0/1 knapsack problem. 0 stands for item is not picked and 1 stands for item is picked inside the knapsack.

This is a formula for putting item in knapsack with values -

Max { ([i].value with given weight + j-n) , [i-1].value from knapsack table) }

Where ,

i = row number in knapsack table

j = values in columns of knapsack table

n = number of backward steps to the knapsack table( column wise)

Example

Total weight = 7

Set of weights are given = { 1, 3, 4, 5 }

Set of values respective wts = { 3, 4, 7, 9 }

So you have to fill all the items in knapsack like that the weight should be minimum and the value of knapsack should be maximum.

Backtracking -

It is used to solve a problem in which a sequence of objects is chosen from a specified set so that sequence satisfy some criteria.

When is it used ?

1. Difference choices are given and from these choices we have to select a decision.

2. When a sufficient information is not avialable on best choice.

3. Each decision leads to new set of choices.

4. Some sequences of choices may be the solution.

How is it performed ?

Backtracking is a systematic method of trying out various sequences of decisions until you find out that works (solution).

Block diagram (Backtracking) -

There are two type of constraints in Backtracking.

1. Implicit - means how each element in tuple should be related.

2. Explicit - means rules which are restricted choosen from a given set.

Applications of Backtracking -

1. N queens problem
2. Sum of subsets
3. Graph colouring
4. Hemiltonian problem

These all applications are dictate in further posts.

If have any queries, just drop a comment or feel free to connect at -
https://www.instagram.com/kavyansh.pandey

Thanking you.

Dynamic Programming | Design & Analysis of Algorithms

2019-10-16T02:18:00.003+05:30

In this post I'm going to give you a brief description about what the Dyanmic programming actually is !!

Dynamic programming is a strategy for designing of algorithms with attaining optimal solution. Don't confuse like this is a programming language like C , C++ etcetera.

It is used in that place where problem breaks down into recurring small subproblems.

Because In this technique, problem divides in many subproblems so there will be sequences of decisions (solutions) and with the help of these sequences, we get an optimal solution.

Elements and characteristics of Dynamic programming

1. Simple subproblem - A problem breaks in small subproblems which have same structure ture as main problem.

2. Optimal Sub structure problem - Optimal solution to the problem contain with inoptimal solution to its subproblems.

3. Overlapping sub problem - Some times there are some cases for which we have to solve subproblems more than one time, this is called overlapping in subproblems.

Steps for Designing Dynamic Algorithms

1. Characterize optimal sub structure.

2. Recursively defined the value of an optimal solution.

3. Compute the value by using bottom up approach.

4. Construct an Optimal solution (if needed).

Block Diagram

We can understand Dynamic Programming with the help of following diagram.

Problem breaks into many subproblems and evry subproblems have decision's (solutions) and have some value and we select the optimal (minimum) value.

Applications of Dynamic Programming

1. Matrix chain multiplication

2. Optimal binary search tree

3. All pairs shortest path problems

4. Travelling salesman problem

5. 0/1 knapsack problem

6. Reliable designing problme

We will cover all these applications in further posts.

Thanking you.

If you have any query, just drop a comment or connect -

https://www.instagram.com/kavyansh.pandey

Topics and Algorithms in Data Science

2019-09-27T03:56:00.001+05:30

In my previous posts, I’ve told about What is Data Science, different fields of Data Science like medical field, industrial field, education field is also involve in it as data is everywhere so we can say that “Data Science is in every place where data exist.”

I have also told about different types of data that we deal in Data Science field like structured, unstructured, continuous, discrete. It is very convenient to deal with structured data (Data generally given in tabular format) but in case of unstructured data, it takes some efforts (because its may be video, audio or text or the combination of these).

Take an example of a webpage as may be it contains image, texts, audio or video files etcetera. So from this data exploration of insights will be a quite complex task.
These are some of things which I discussed on my previous posts.

Now let’s take a look of Data Science topics and some of Algorithms.

Regression

It is one of the simplest topic of Data Science. It works on structured data. By knowing some past information, telling about the future insights or telling about the insights between any fraction.

The value of one variable is depend upon another variable. And applying some algorithms we find the value of our desired (targeted) variable.

Ex . suppose any person hire a cab and cost for every mile is 10. Here miles(distance) is independent variable and cost is dependent variable.
So
Cost = 5 x distance,
with the help of this equation we can predict the value for different distances (In further post we will learn how to make a regression model using Data Science libraries and Python programming language:-)

Data Visualization-

It is the best way for understanding of insights . It is an art of representing data in the front of others with clarity and it is also easy to understand. It is the key element to telling the people what you really want to express them.

Besides this there are also so many topics like neural networks, deep learning (advance version of neural networks) , machine learning including supervised, unsupervised and reinforcement, classification, clusters, recommendation and Algorithms like KNN (nearest neighbour), SVM (support vector mechanism), Decision tree and many more.

These all mentioned topic will be surely cover in upcoming posts.

Thanking you.

If have any query, feel free to connect -
https://www.instagram.com/kavyansh.pandey

Types of Data in Data Science

2019-09-18T00:38:00.000+05:30

In my last post INTRODUCTION TO DATA SCIENCE, I used two terms for data that is structured data and unstructured data. In this post I'll give you the brief introduction about data generally we talked in Data Science.

There are basically 4 types of data.
1. Structured
2. Unstructured
3. Continuous
4. Discrete

Structured Data

The data which is given in well formed as you can easily access it and easily perform all algorithm(searching algorithm or anything else) is called structured data.
We can say like that
"Structured data is just like a form of formatted repository just like a database so that its elements are more addressable for further procces or analysis."

Example
Name of person, age, number of books etc all data given in the form of rows and columns.

Unstructured Data

It is opposite of structured data as in case of unstructured data we can't perform any algorithms easily it will take some time and more complexity there...
We can say like that
"Information that doesn't have predefine model or is not organized in pre-defined manner. Unstructured data typically text heavy, but may contain data such as dates , numbers and facts as well."

Example
Image, videos , mp3 files etc..

Continuous Data

Data which have infinite number of possible values and when we plot of graph with these data there is not any separation or split between curve...

Discrete Data

Data which have finite number of possible values and during graph represent there is a space between curves or line...

With this I'm concluding this post and I'm trying my best to be consize and be clear what are these data and we have to interact with these data in Data Science field...

Thanking you.

If any query, feel free to connect
https://www.instagram.com/kavyansh.pandey

Introduction to Data Science

2019-09-11T01:35:00.000+05:30

As we all know data is the thing which is very useful and important for any organization. And the amount of data is very large that no one can handle it manually....

Initally in 90's there is also data but the amount of data is very less in the comparison of present. In past we can handle data manually or we have some storage devices like card reader, magnetic tape etc where we can store data.

And now in this era, the amount of data is very huge and we can store it in different places like different types of storage devices, cloud platforms...

But the task is how to use this data in convenient way ...how to use this data for future insights , how to do predictions on the behalf of this data etc, it's all come in Data Science.

There is not any standard definition of Data Science but in general we can define it as -

"It is the study of data for future insights "

Data Science is a field about procceses and system to extract data from various form whether it is structured or unstructured.

"Data Science involves data and some signs"

Data is real and it has some properties and we need to study them if we are going to work on them...

DATA SCIENCE IS THE ART OF UNCOVERING THE INSIGHTS AND TRENDS WHICH ARE HIDDEN BEHIND DATA.

Thanking you....

If any query, feel free to connect at.
https://www.instagram.com/kavyansh.pandey