InfinityCodeX

From Complexity to Clarity: Simplifying Principal Component Analysis (PCA)

2024-12-14T22:25:00.004+05:30

From Complexity to Clarity: Simplifying Principal Component Analysis (PCA)

"Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. It is a statistical procedure which is also used for finding patterns in high dimension data. It uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called as principal components."

I know after reading the above paragraph you are like...

Oh, really? Why it couldn’t possibly be any easier?

Then answer is... well, of course, it is simple! and, don’t stress—I’m here to save the day, because clearly, that’s what was missing all along

Principal Component Analysis: Because Your Data Has Too Many Opinions, Just Like Your Ex Used To 🤭

Lets forget you ex and discuss the agenda, what all things we are going to cover in this article?

1.) What even is PCA?

2.) Why PCA? or, "Why Shrink Your Data?"

3.) How does PCA actually works?

4.) Application of PCA (Yes people actually use it)

5.) Math Stuff

6.) Python Coding Example

7.) Conclusion: Do you really need it?

1. What Even Is PCA?

* Definition: Principal Component Analysis — or, as we like to call it, "Magic Math that Makes Your Data Less Annoying."
* Simple Explanation: It’s where we take a perfectly fine dataset, perform a mathy shuffle dance, and end up with fewer columns that apparently contain the same enough information. Simple, right?
* Behind the Scenes: Eigenvalues, eigenvectors, matrices… you know, all the good stuff you happily forgot after that one linear algebra class.

2. Why PCA? Or, "Why Shrink Your Data?"

* To Stop Crashing Your Laptop: Imagine actually trying to work with a 100-dimensional dataset. PCA helps avoid the horror of infinite loading circles.
* Because Interpretability Is for Amateurs: With PCA, you can have the thrill of knowing your data just got simpler without needing to explain why.
* Optimal Confusion Guarantee: Impress your friends and baffle your audience with reduced dimensions that convey almost the same story. Who doesn’t love a good riddle in data form?

3. How Does PCA Actually Work?

* Step-by-Step Guide: A 15-step process of complex calculations, conveniently glossed over in most blog posts. But don’t worry — you can just use a library function! I have covered that in 6 steps.
* Scaling Your Data Like a Pro: Normalize first, or you’ll end up with results that don’t make sense (not that they ever really do).
* Covariance and Variance Fun: Rejoice in knowing how much variation you can squeeze out of your data without it screaming back in protest.

4. Applications of PCA (Yes, People Actually Use It)

* Dimensionality Reduction (Obviously): Because who needs all those columns when you can turn them into just a few "principal components"? Image Compression, a.k.a. “Why’s My Picture Blurry Now?” Ever wonder how to shrink images into an indecipherable mosaic? PCA’s got you covered.
* Pattern Recognition and ML Models: You’ll get to say you improved your model’s performance by 0.02% and act like it’s groundbreaking.
* Noise Reduction (Or Data Disguising): PCA will kindly help you ignore all the pesky little outliers you didn't want to deal with.

5. Math Stuff

This section is for nerds, who are interested in the math behind this algorithm.

Steps to perform PCA:

(i) Standardize the dataset.
(ii) Calculate the covariance matrix for the feature in data.
(iii) Calculate the eigen values and eigen vectors for covariance matrix.
(iv) Sort eigen values and their corresponding eigen vectors.
(v) Pick k eigen values and form a matrix of eigen vector.
(vi) Transform original matrix.

(i) Standardize the dataset.

Let's assume we have data with 4 features & 5 rows of training data

Training Data Table

Feature 1	Feature 2	Feature 3	Feature 4
1	2	3	4
5	5	6	7
1	4	2	3
5	3	2	1
8	1	2	2

Standardize the data :

Where:

$z$ z: The standardized value (z-score).
$x$ x: The original data point.
$\mu$ μ: The mean of the dataset.
$\sigma$ σ: The standard deviation of the dataset.

	Feature 1	Feature 2	Feature 3	Feature 4
μ =	4	3	3	3.4
σ =	3	1.58	1.73	2.30

(Standardize Data) Apply formula in each feature:

Feature 1	Feature 2	Feature 3	Feature 4
-1	-0.63	0	0.26
0.33	1.26	1.73	1.56
-1	0.63	-0.57	-0.17
0.33	0	-0.57	-1.06
1.33	-1.26	-0.57	-0.60

(ii) Calculate the covariance matrix for the feature in data.

For Population:

For Sample:

Calculate covariance matrix of given data.

	Feature 1	Feature 2	Feature 3	Feature 4
Feature 1	var(f1)	cov(f1,f2)	cov(f1,f3)	cov(f1,f4)
Feature 2	cov(f2,f1)	var(f2)	cov(f2,f3)	cov(f2,f4)
Feature 3	cov(f3,f1)	cov(f3,f2)	var(f2)	cov(f3,f4)
Feature 4	cov(f4,f1)	cov(f4,f2)	cov(f4,f3)	var(f4)

μ of each feature is 1 & σ is 0. Since we are standardizing.

Var(f1) = 0.8

Cov(f1,f2) = -0.2529

- Like this calculation the other covariance is:

	Feature 1	Feature 2	Feature 3	Feature 4
Feature 1	0.8	-0.25	0.03	-0.14
Feature 2	-0.25	0.8	0.51	0.49
Feature 3	0.03	0.51	0.8	0.75
Feature 4	-0.14	0.49	0.75	0.8

(iii) Calculate the eigen values and eigen vectors for covariance matrix.

- A eigen vector is an non-zero vector that changes at most by a scalar factor when that linear transformation is applied to it.

- The corresponding eigen value is the factor by which the eigen vector is scaled.

- Let 'A' be a square matrix (in our case the covariance matrix), 'v' a vector & 'λ' a scalar that satisfies Av = λv , then λ is called eigen value associated with eigen vector v of A.

Rearranging the above equation:

Av - λv = 0; (A-λ1)v = 0

Since we know v is non-zero, only way this equation can be equal to 0, if

det(A-λ1)=0

	Feature 1	Feature 2	Feature 3	Feature 4
Feature 1	0.8-λ	-0.25	0.03	-0.14
Feature 2	-0.25	0.8-λ	0.51	0.49
Feature 3	0.03	0.51	0.8-λ	0.75
Feature 4	-0.14	0.49	0.75	0.8-λ

Solving the above equation = 0

Eigen Vectors:

solving the (A-λI)v = 0 equation for vector with the different λ values:

0.8-λ	-0.25	0.03	-0.14
-0.25	0.8-λ	0.51	0.49
0.03	0.51	0.8-λ	0.75
-0.14	0.49	0.75	0.8-λ

v₁

v₂

v₃

v₄

= 0

for λ = 2.51, solving the above equation using Cramer's rule, the values for vectors "v" are

v₁ = 0.16

v₂ = -0.52

v₃ = -0.58

v₄ = -0.59

Going by the same approach we can calculate the eigen vectors for the other eigen values.

We can form matrix using eigen vectors.

e1	e2	e3	e4
0.16	-0.91	-0.30	0.19
-0.52	0.20	-0.81	0.12
-0.58	-0.32	0.18	-0.72
-0.59	-0.11	0.44	0.65

(iv) Sort eigen values & their corresponding eigen vectors

Since eigen values are already sorted so no need to sort them again.

(v) Pick eigen values and from matrix of eigen vectors

If we choose the top 2 eigen vectors, the matrix will look like this

(Top 2 eigen vectors 4 * 2 matrix)

e1	e2
0.16	-0.91
-0.52	0.20
-0.58	-0.32
-0.59	-0.11

(vi) Transform the original matrix

Feature_matrix * Top_k_eigen_vectors = Transformed_data

Feature 1	Feature 2	Feature 3	Feature 4
-1.00	-0.63	0.00	0.26
0.33	1.26	1.73	1.56
-1.00	0.63	-0.57	-0.17
0.33	0.00	-0.57	-1.04
1.33	-1.26	-0.57	-0.60

e1	e2
0.16	-0.91
-0.52	0.20
-0.58	-0.32
-0.59	-0.11

nf1	nf2
0.01	0.75
-2.55	-0.78
-0.05	1.25
1.01	0.00
1.57	-1.22

6. Python Coding Example

# Code Example:

import pandas as pd
import numpy as np

# Input data
A = np.matrix([[1, 2, 3, 4], [5, 5, 6, 7], [1, 4, 2, 3], [5, 3, 2, 1], [8, 1, 2, 2]])

# Create DataFrame
df = pd.DataFrame(A, columns=["f1", "f2", "f3", "f4"])

# Standardize the data
df_std = (df - df.mean()) / df.std()

# Number of Principal Components
n_components = 2

# Apply PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=n_components)
pc = pca.fit_transform(df_std)

# Convert PCA results to DataFrame
principalDf = pd.DataFrame(data=pc, columns=["nf" + str(i + 1) for i in range(n_components)])

# Display output
print(principalDf)

Output

Index	nf1	nf2
0	-0.014003	-0.755975
1	2.556534	0.780432
2	0.051480	-1.253135
3	-1.014150	-0.000239
4	-1.579861	1.228917

7. Conclusion: Do You Really Need PCA?

Probably not, but it sure sounds fancy when you tell people at parties, right? If you love mysterious statistical transformations, you’re going to adore PCA!

I hope this helped you. If it did then please share it with your friends and spread this knowledge.

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

The Rise of BERT AI: The Model Redefining Natural Language Processing

2023-02-08T22:32:00.006+05:30

BERT Model: Revolutionizing Natural Language Processing

BERT

BERT, which stands for Bidirectional Encoder Representations from Transformers, is a deep learning-based language model developed by Google that has been making waves in the field of natural language processing (NLP). BERT is unique in its ability to understand the context of words in a sentence and make predictions about the likelihood of a word being in a particular context. This has made BERT a highly versatile and powerful tool for solving a wide range of NLP tasks, such as sentiment analysis, question answering, and text classification.

BERT's Architecture

BERT's architecture is based on the transformer network, which was introduced in 2017 by Vaswani et al. The transformer network is a neural network architecture designed specifically for NLP tasks and has since been used in many state-of-the-art NLP models.

BERT's architecture consists of two main components: an encoder and a decoder. The encoder is responsible for encoding the input text data into a hidden representation, while the decoder is responsible for predicting the likelihood of a word being in a particular context based on the encoded representation.

The input to BERT is a sequence of tokens, where each token represents a word in the sentence. The tokens are fed into the encoder, which is composed of multiple layers of attention mechanisms and feed-forward networks. The attention mechanism allows the model to focus on specific parts of the input sequence and weigh their importance in determining the encoded representation.

The encoded representation is then fed into the decoder, which is composed of a single feed-forward network. The decoder outputs a probability distribution over the vocabulary for each token in the input sequence. The decoder's predictions are then compared to the actual labels (i.e., the words in the sentence) to calculate the loss, which is used to update the model's parameters during training.

BERT's Unique Features

BERT is unique among NLP models in several ways. First, BERT is bidirectional, meaning that it considers both the left and right context of each word in the sentence. This allows BERT to capture the context of words more accurately and outperforms unidirectional models.

Second, BERT is pre-trained on a massive amount of text data, which allows it to perform well on a wide range of NLP tasks without the need for task-specific training data. This makes BERT highly versatile and cost-effective, as it can be fine-tuned for specific NLP tasks with relatively small amounts of task-specific data.

Finally, BERT uses masked language modeling as its pre-training task, where a percentage of the tokens in the input sequence is masked and the model is trained to predict the masked tokens. This pre-training task helps BERT capture the context of words in a sentence more effectively and outperforms models trained with traditional pre-training tasks.

BERT in Action

BERT has been applied to a wide range of NLP tasks and has consistently outperformed previous state-of-the-art models. For example, in the GLUE benchmark, which evaluates the performance of NLP models on a range of benchmark datasets, BERT outperformed previous state-of-the-art models by a substantial margin.

BERT has also been used to solve more complex NLP tasks, such as question answering and text summarization. For example, BERT has been fine-tuned to answer questions based on a large corpus of text data, such as the SQuAD dataset. BERT has also been used to generate summaries of long pieces of text, such as news articles.

Transformers in BERT: Technical Math and Steps to Create the Model

Transformer

The transformer architecture is at the heart of the BERT model and is responsible for its impressive performance in natural language processing (NLP) tasks. In this section, we will provide a detailed explanation of the math behind transformers and the steps involved in creating a BERT model.

The Transformer architecture is a deep neural network that is used for sequence processing tasks such as language translation and text generation. The key idea behind the Transformer architecture is the use of self-attention mechanisms to dynamically weight the importance of different inputs in a sequence.

A Transformer consists of the following components:

1. Multi-head attention mechanism

2. Pointwise feedforward network

3. Residual connections and layer normalization

The following equations explain each component in detail:

1. Multi-head attention mechanism: The multi-head attention mechanism calculates attention scores between a query matrix Q, a key matrix K, and a value matrix V. The attention scores are calculated using the following equation:

Attention(Q,K,V) = softmax(QK⊤/√d_k)V

Where d_k is the dimensionality of the key vectors and softmax is applied row-wise to the scores. The attention scores are then used to weigh the importance of the values.

2. Pointwise feedforward network: The Pointwise feedforward network is a simple two-layer feedforward neural network that is applied to each position in the sequence independently. The following equation represents the pointwise feedforward network:

FFN(x) = max(0,xW1 + b1)W2 + b2

Where W1 and W2 are the weights of the two layers and b1 and b2 are the biases. The activation function used is the ReLU activation.

3. Residual connections and layer normalization: Residual connections are used to add the input to the output of each layer to improve the flow of information through the network. The following equation represents the residual connection:

Output = Layer_Output + Input

Layer normalization is used to normalize the activations of each layer, which helps to stabilize the training process and improve the performance of the network. The following equation represents the layer normalization:

LayerNorm(x) = (x - mean(x)) / variance(x)

These components of the Transformer architecture are combined to form the final network, where the multi-head attention mechanism and pointwise feedforward network are applied alternately, followed by residual connections and layer normalization. The resulting network is capable of processing sequences in parallel, which greatly reduces the training time compared to traditional RNN-based models.

Self-Attention Mechanisms

Self-attention mechanisms are the key component of transformers and allow the model to weight the importance of different parts of the input data. This is done by computing attention scores, which represent the importance of each input element relative to all other elements.

The attention scores are computed using a set of learnable parameters and are used to compute a weighted sum of the input elements. This weighted sum is then passed through a feedforward neural network to produce the final representation of the input data.

Mathematically, the attention mechanism can be represented as follows:

$$Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$$

Where Q, K and V are the query, key and value matrices, respectively, and $d_k$ is the dimension of the keys.

Creating a BERT Model

The steps involved in creating a BERT model are as follows:

1.) Pre-processing: BERT requires the input data to be pre-processed and tokenized into subwords. This is done using a tokenizer that splits words into subwords and adds special tokens to represent the start and end of sentences.

2.) Encoding: The input data is then passed through a series of transformer layers to produce an encoded representation. Each transformer layer consists of a self-attention mechanism and a feedforward neural network.

3.) Fine-tuning: The pre-trained BERT model can be fine-tuned on a specific NLP task, such as sentiment analysis or question answering, by adding task-specific layers on top of the encoded representation.

4.) Training: The fine-tuned model is then trained on a labeled dataset to adjust the parameters of the self-attention mechanism and feedforward neural network.

5.) Evaluation: Finally, the model is evaluated on a held-out dataset to measure its performance on the task.

In conclusion, the use of transformers and self-attention mechanisms in BERT has revolutionized NLP by allowing models to understand the relationships between words and sentence structure in a way that was not possible with traditional models. The combination of pre-training and fine-tuning has made it possible to create models that achieve state-of-the-art performance on a wide range of NLP tasks.

BERT Model Simple Sentiment Analysis Example

 

# pip install ktrain

# git clone https://github.com/laxmimerit/IMDB-Movie-Reviews-Large-Dataset-50k.git

import tensorflow as tf
import pandas as pd
import numpy as np
import ktrain
from ktrain import text
import tensorflow as tf
     

data_train = pd.read_excel('IMDB-Movie-Reviews-Large-Dataset-50k/train.xlsx', dtype = str)
data_test = pd.read_excel('IMDB-Movie-Reviews-Large-Dataset-50k/test.xlsx', dtype = str)
     
(x_train,y_train),(x_test,y_test),preprocess=text.texts_from_df(data_train,text_column='Reviews'
                                             ,label_columns='Sentiment',val_df=data_test
                                            ,maxlen=400,preprocess_mode='bert')

model = text.text_classifier(name='bert',train_data=(x_train,y_train)
                             ,preproc=preprocess)

learner = ktrain.get_learner(model=model, train_data=(x_train, y_train),
                   val_data = (x_test, y_test),
                   batch_size = 6)

learner.fit_onecycle(lr = 2e-5, epochs = 1)

     

predictor = ktrain.get_predictor(learner.model, preprocess)
predictor.save('/content/drive/My Drive/bert')
     

data = ['this movie was horrible, the plot was really boring. acting was okay',
        'the fild is really sucked. there is not plot and acting was bad',
        'what a beautiful movie. great plot. acting was good. will see it again']
     

predictor.predict(data)

>>> ['neg', 'neg', 'pos']

So we hope that you enjoyed this project. If you did then please share it with your friends and spread this knowledge.

Instagram: https://www.instagram.com/infinitycode_x/

Facebook: https://www.facebook.com/InfinitycodeX/

Pinterest: https://in.pinterest.com/vegadhardik/

Twitter: https://twitter.com/InfinityCodeX1

EVERY POSSIBLE MYSQL INTERVIEW QUESTION FOR DATA SCIENCE, DATA ANALYST, DATA ENGINEER, AND DATABASE MANAGER.

2022-08-14T18:22:00.006+05:30

EVERY POSSIBLE MYSQL INTERVIEW QUESTION FOR DATA SCIENCE, DATA ANALYST, DATA ENGINEER, AND DATABASE MANAGER.

MySQL

If you want to be a successful data scientist or data analyst or data engineer or database manager MySQL is must to know and most of the time we ignore it and fail to crack the interview due to lack of knowledge or we couldn't find a place where we can find solutions of almost all the MySQL questions. Here are some of the best MySQL / SSMS interview questions which are guaranteed to be asked in an interview.

So without wasting any time let's get started.

See All DataBases:

show databases;

See All Tables:

show tables;

Use Database:

use company; /* company is database name */

See Table Content:

select * from Employee; /* Employee table name */

Create Table:

 
CREATE TABLE Employees(e_id int, 
                       e_name varchar(20), 
                       join_date date, 
                       e_gender varchar(5), 
                       e_age int(10),
                       e_dept varchar(20), 
                       e_sal int(20));


CREATE TABLE Departement(d_id int, 
                         d_name varchar(20), 
                         d_loc varchar(20));

Insert Into Table Values:

 
INSERT INTO Employees VALUES
(101, 'Naruto', '20-12-2020', 'M', 22, 'IT', 100000),
(102, 'Priya', '22-02-2021', 'F', 23, 'MARKET', 50000),
(103, 'Jay', '25-10-2021', 'M', 21, 'FINANCE', 30000);


INSERT INTO Departement VALUES
(1, 'IT', 'MUMBAI'),
(2, 'ACCOUNTS', 'DELHI'),
(3, 'MARKET', 'PUNE');

Create Copy Of Table:


SELECT *
INTO Copy_Employees
FROM Employees;

Where Clause:


/* Search Record */

SELECT * FROM Employees
WHERE e_name = 'Priya';

/* ------------------------------ */

/* Delete Record */

DELETE FROM Employees
WHERE e_name = 'Priya';

AND | OR | NOT Operators:


/* AND */

SELECT * FROM Employees
WHERE e_name = 'Priya' AND e_age = 23;

/* --------------------------------------- */

/* OR */

SELECT * FROM Employees
WHERE e_age = 22 OR e_dept = 'SALES';

/* --------------------------------------- */

SELECT * FROM Employees
WHERE e_age = 22 OR e_age = 23;

/* --------------------------------------- */

/* NOT */

SELECT * FROM Employees
WHERE NOT e_dept = 'IT';

Distinct Operator:


/* (GIVES UNIQUE VALUES) */

SELECT DISTINCT e_gender FROM Employees;

Between Operator:


SELECT * FROM Employees
WHERE e_age
BETWEEN 20 AND 30;

Basic Functions For Data Interpretation:

MIN()
MAX()
COUNT()
SUM()
AVG()


/* MIN */

SELECT MIN(e_age) FROM Employees;

/* MAX */

SELECT MAX(e_age) FROM Employees;

/* COUNT */

SELECT COUNT(*) FROM Employees WHERE e_gender='M';

/* SUM */

SELECT SUM(e_sal) FROM Employees;

/* AVG */

SELECT AVG(e_sal) FROM Employees;

Like Operator:


/* Stats With } 'value%' */

SELECT * FROM Employees WHERE e_name LIKE 'M%';     /* Name Starts With M */

/* Ends With } '%value' */

SELECT * FROM Employees WHERE e_name LIKE '%m';     /* Name Ends With m */

/* In a specfic value rage } 'value_' */

SELECT * FROM Employees WHERE e_age LIKE '2_';      /* Age In 20's */

String Functions:

LTRIM()
UPPER()
LOWER()
REVERSE()
SUBSTRING()


/* LTRIM */

SELECT LTRIM('    Naruto');

/* LOWER */

SELECT LOWER('NARUTO');

/* UPPER */

SELECT UPPER('naruto');

/* REVERSE */

SELECT REVERSE('naruto');

/* SUBSTRING */

SELECT SUBSTRING('This is sparta',9,6);

Group By:


/* Get AVG sal with respect to gender */

SELECT AVG(e_sal) FROM Employee
GROUP BY e_gender;

/* Get AVG age with respect to gender */

SELECT AVG(e_age) FROM Employee
GROUP BY e_gender;

Order By & Top:


/* Order By } Sort The Data by ASC or DESC */

SELECT * FROM Employees ORDER BY e_age;

/* ---------------------------------------- */

SELECT * FROM Employees ORDER BY e_age DESC;

/* ---------------------------------------- */

/* Top x */

SELECT TOP 3 * FROM Employees ORDER BY e_age DESC;

Having Clause:


/* Departement Name Whose AVG(e_sal) is Greater than 80000 */

SELECT e_dept, AVG(e_sal) AS dep_avg_sal
FROM Employees
GROUP BY e_dept
HAVING AVG(e_sal) > 80000
ORDER BY dep_avg_sal DESC;

/* ------------------------------------------------------------ */

/* The following SQL statement lists the number of Emp in each Departement. 
Only include Departement with more than 5 Emp: */

SELECT COUNT(e_id), e_dept
FROM Employees
GROUP BY e_dept
HAVING COUNT(e_id)>5;

Alter Table:


/* Add Column To a Table */

ALTER TABLE Employees
ADD e_height int;

/* ---------------------------------------- */

/* Delete Column To a Table */

ALTER TABLE Employees
DROP COLUMN e_height;

Update Statement:


UPDATE Employees
SET e_age = 24
WHERE e_name = 'Priya';

Delete Statement:


/* Delete specific record */

DELETE Employees
WHERE e_name = 'Priya';

Truncate Statement:


/* Delete all the value from table but structure remain same */

TRUNCATE TABLE Employees;

Drop Statement:


/* Delete table including structure */

DROP TABLE Employees;

Union | Union All | Except | Intersect Operators:


/* Union (A + B) Drop Duplicates */

SELECT * FROM Employees
UNION
SELECT * FROM Customers

/* ----------------------------------------- */

/* Union All (A + B) Include Duplicates */

SELECT * FROM Employees
UNION ALL
SELECT * FROM Customers

/* ----------------------------------------- */

/* Except (A - B) Drop Duplicates */

SELECT * FROM Employees
EXCEPT
SELECT * FROM Customers

/* ----------------------------------------- */

/* Intersect (A n B) Common to both */

SELECT * FROM Employees
INTERSECT
SELECT * FROM Customers

Joins:

INNER
LEFT
RIGHT
FULL


/* INNER Join } Return records which has matching values */

SELECT 
Employees.e_name, Employees.e_dept, 
Departement.d_name, Departement.d_loc
FROM Employees
INNER JOIN Departement
ON Employees.e_dept = Departement.d_name;

/* ----------------------------------------- */

/* LEFT Join } Return all records from left tables & 
   matching records from right */

SELECT 
Employees.e_name, Employees.e_dept, 
Departement.d_name, Departement.d_loc
FROM Employees
LEFT JOIN Departement
ON Employees.e_dept = Departement.d_name;

/* ----------------------------------------- */

/* RIGHT Join } Return all records from right tables & 
   matching records from left */

SELECT 
Employees.e_name, Employees.e_dept, 
Departement.d_name, Departement.d_loc
FROM Employees
RIGHT JOIN Departement
ON Employees.e_dept = Departement.d_name;

/* ----------------------------------------- */

/* FULL Join } Return all records from right tables & left table */

SELECT 
Employees.e_name, Employees.e_dept, 
Departement.d_name, Departement.d_loc
FROM Employees
FULL JOIN Departement
ON Employees.e_dept = Departement.d_name;

Update & Delete Using Joins:


/* Update e_sal value in Employee table where ever
 the departement location is Mumbai */

UPDATE Employees
SET e_age = e_age+10
FROM Employees 
JOIN Departement ON Employees.e_dept = Departement.d_name
WHERE d_loc = 'Mumbai';

/* ---------------------------------------------------------- */

/* Delete value in Employee table where ever 
   the departement location is Mumbai */

DELETE Employees
FROM Employees 
JOIN Departement ON Employees.e_dept = Departement.d_name
WHERE d_loc = 'Mumbai';

Views:


/* Create view of only female employees */

CREATE VIEW Female_emps AS
SELECT * FROM Employees
WHERE e_gender = 'F';

/* ---------------------------------------------------------- */

/* See view */

SELECT * FROM Female_emps;

/* ---------------------------------------------------------- */

/* Drop view */

DROP VIEW Female_emps;

Merge:

Perform INSERT, UPDATE, and DELETE all at once.

When rows match we want to perform an update.

When rows don't match but the data is present in the SOURCE table but not in the TARGET table we need to INSERT all such rows into the TARGET table.

When records are present only in the TARGET table but not in the SOURCE table, we need to DELETE all such rows from the TARGET table.


MERGE [TARGET] AS T

USING [SOURCE] AS S

ON [JOIN_CONDITION]

WHEN MATCHED THEN

UPDATE STATEMENT
WHEN NOT MATCHED BY TARGET THEN

INSERT STATEMENT
WHEN NOT MATCHED BY SOURCE THEN

DELETE STATEMENT;

/* CODE: */

MERGE Students AS S
USING students_new AS SN
ON S.s_id = SN.s_id
WHEN MATCHED THEN
    UPDATE SET S.s_name = SN.s_name
WHEN NOT MATCHED BY TARGET THEN
    INSERT(s_id, s_name) VALUES(SN.s_id, SN.s_name)
WHEN NOT MATCHED BY SOURCE THEN
    DELETE;

Function Types In SQL:

(1) Scalar Valued Function

(2) Table Valued Function


/* 1.) Scalar Valued Function 
  (Returns Scalar Values such as int, varchar...etc) */

CREATE FUNCTION add_five(@num as int)

RETURNS int
AS
BEGIN
RETURN
(@num + 5)
END

SELECT dbo.add_five(100);

/* ----------------------------------------------------------------- */

/* 2.) Table Valued Funtion (Returns Tabular Values) */

CREATE FUNCTION select_gender(@gender AS VARCHAR(20))
RETURNS TABLE
AS 
RETURN
(SELECT * FROM Employees WHERE e_gender = @gender)


SELECT * FROM DBO.select_gender('M');

Temporary Table:

The table which gets deleted as soon as terminated.


CREATE TABLE /* NEW_STUDENTS */
(ns_id int, ns_name varchar(20));

INSERT INTO /* NEW_STUDENTS */
VALUES 
(101, 'Ram'),
(102, 'Jay');

SELECT * FROM NEW_STUDENTS;

Case Statement:


/* e_sal_cmt will be a new column which will 
  tell if employee is rich or poor */

SELECT *, e_sal_cmt =                     
CASE
    WHEN e_sal < 90000 THEN 'POOR'
    WHEN e_sal > 90000 THEN 'RICH'
    ELSE 'MIDDLE CLASS'
END
FROM Employees
GO

IIF() Function:


SELECT e_id, e_name, e_age,
IIF (e_age>35, 'Old Employee', 'New Employee')
AS e_generation
FROM Employees;

Stored Procedure:

Functions Which Call Be Called Anytime We Want


/* Normal Stored Procedure */

CREATE PROCEDURE access_age
AS
SELECT e_id, e_name, e_age
FROM Employees
WHERE e_age>35
GO

EXEC access_age;

/* --------------------------------------------- */

/* Stored Procedure With Parameters (Get Male & Female Employee Seperately) */

CREATE PROCEDURE emp_gender @gender varchar(20)
AS
SELECT *
FROM Employees
WHERE e_gen = @gender
GO

EXEC emp_gender @gender = 'M';

Try Catch:


DECLARE @val1 int;
DECLARE @val2 int;

BEGIN TRY
    SET @val1 = 5;
    SET @val2 = @val1/0;
END TRY

BEGIN CATCH
PRINT ERROR_MESSAGE()
END CATCH;

/* --------------------------------------------- */

BEGIN TRY
    SELECT e_name + e_sal FROM Employees
END TRY

BEGIN CATCH
    PRINT 'CANNOT ADD NUM VALUE WTIH STRING'
END CATCH
GO

Commit & RollBack Using Transcation:


/* Roll Back */

BEGIN TRANSACTION
    UPDATE Employees
    SET e_age = 30
    WHERE e_name = 'Sam';

SELECT * FROM Employees;

ROLLBACK TRANSACTION;

SELECT * FROM Employees;

/* --------------------------------------------- */

/* Commit */

BEGIN TRANSACTION
    UPDATE Employees
    SET e_age = 30
    WHERE e_name = 'Sam';

SELECT * FROM Employees;

COMMIT TRANSACTION;

SELECT * FROM Employees;

Transaction (If Error In Code End Whole Transaction & RollBack):


SELECT *
INTO Copy_Employees
FROM Employees;

/* ------------------------------------------------------------ */

/* This Code Will RollBack Since 120/0 gives ZeroDivisionError */

SELECT * FROM Copy_Employees;



BEGIN TRY
    BEGIN TRANSACTION
        UPDATE Copy_Employees SET e_sal = 500 WHERE e_gender = 'M'
        UPDATE Copy_Employees SET e_sal = 120/0 WHERE e_gender = 'F'
    COMMIT TRANSACTION
    PRINT 'Transaction Commited'
END TRY

BEGIN CATCH
    ROLLBACK TRANSACTION
    PRINT 'Transaction RolledBack'
END CATCH



SELECT * FROM Copy_Employees;

/* --------------------------------------------------------------------- */

/* This Code Will Get Commited */

SELECT * FROM Copy_Employees;



BEGIN TRY
    BEGIN TRANSACTION
        UPDATE Copy_Employees SET e_sal = 500 WHERE e_gender = 'M'
        UPDATE Copy_Employees SET e_sal = 120 WHERE e_gender = 'F'
    COMMIT TRANSACTION
    PRINT 'Transaction Commited'
END TRY

BEGIN CATCH
    ROLLBACK TRANSACTION
    PRINT 'Transaction RolledBack'
END CATCH



SELECT * FROM Copy_Employees;

SQL With Python:

Connect MySQL to Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost',user='root',password='mousepad',port=3306)

print(mydb.connection_id)

Create Data Base In MySQL with Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad',port=3306)

cur = mydb.cursor()

cur.execute("CREATE DATABASE my_database1")

Create Table In Database using Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad', port=3306, database='my_database1')

cur = mydb.cursor()

str1 = "CREATE TABLE SCHOOL(Stu_id int(10), Stu_fname varchar(10), Stu_lname varchar(10))"

cur.execute(str1)

Insert Into School Value Using Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad', port=3306, database='my_database1')

cur = mydb.cursor()

str2 = "INSERT INTO SCHOOL VALUES(%s,%s,%s)"

vals = [(101,'Naruto','Uzumaki'),(102,'Rock','Lee'),(103,'Rin','Nohara')]

cur.execute(str2,vals)

mydb.commit()

Extract Data From Table Using Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad', port=3306, database='my_database1')

cur = mydb.cursor()

str1 = "select * from school"

cur.execute(str1)
result = cur.fetchall()

for i in result:
    print(i)

Update Table With Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad', port=3306, database='my_database1')

cur = mydb.cursor()

s = "UPDATE SCHOOL SET Stu_id=Stu_id+1 WHERE Stu_id>102"

cur.execute(s)
mydb.commit()

Delete Record With Python


import mysql.connector

mydb = mysql.connector.connect(host='localhost', user='root', password='mousepad', port=3306, database='my_database1')

cur = mydb.cursor()

s = "DELETE FROM school WHERE Stu_id>102"

cur.execute(s)
mydb.commit()

Interview Questions:

Q.) Change the Data Type Of a column


ALTER TABLE Employees
ALTER COLUMN Age float;

Q.) Get the Average Age Of Females


SELECT AVG(e_age) AS avg_female_Age 
FROM Employees
WHERE e_gender = 'F';

Q.) Second Highest Salary In Employee Table


SELECT MAX(e_sal) FROM Employees
WHERE e_sal < (SELECT MAX(e_sal) FROM Employees);

Q.) Top N=2 Salary From Employees


SELECT TOP 2 * FROM 
(SELECT DISTINCT TOP 2 * 
 FROM Employees 
 ORDER BY e_sal DESC) AS RESULT
ORDER BY e_sal DESC;

Q.) Get Nth Highest Salary


# 3rd highest salary

WITH RESULT AS
(
    SELECT e_sal, DENSE_RANK() OVER (ORDER BY e_sal DESC) as D
    FROM Employees
)
SELECT TOP 1 e_sal
FROM RESULT
WHERE RESULT.D = 3;

Q.) Department wise highest salary


SELECT MAX(e_sal) AS Max_Dep_sal, e_dept FROM Employees
GROUP BY e_dept
ORDER BY Max_Dep_sal DESC;

Q.) Number of employees in each department


SELECT COUNT(*) AS EMP_CNT, e_dept FROM Employees
GROUP BY e_dept;

Q.) Displayed Duplicated Records


SELECT COUNT(*) AS Number_of_times_Duplicated, e_id FROM Copy_Employees
GROUP BY e_id
HAVING COUNT(*)>1;

Q.) Delete Duplicate Records in SQL


WITH EmployeesCTE AS
(
    SELECT *, ROW_NUMBER() OVER (PARTITION BY e_id ORDER BY e_id) as Row_Num
    FROM Copy_Employees
)
DELETE FROM EmployeesCTE WHERE Row_Num>1;

Q.) Top 3 Employees With Highest Salary


Select TOP 3 * from Employees
ORDER BY e_sal DESC;

Q.) Concat First Name & Last Name Of Employee


SELECT CONCAT(f_name,'_',l_name) as Full_Name FROM Employees;

Q.) Departement Wise Count Of Employee


SELECT COUNT(*) AS cnt_of_emp_as_per_dep, e_dept FROM Employees
GROUP BY e_dept
ORDER BY COUNT(*) ASC;

Q.) Show All Even Records


SELECT * FROM Employees
WHERE e_id%2=0;

Q.) Last 2 Rows


SELECT TOP(2) e_id, e_name, e_sal    
FROM Employees    
Order By e_id DESC;

Q.) Show Employee's location and date of joining which is in the EmpPos table


SELECT
Employees.e_id, Employees.e_name, Employees.sal,
EmpPos.e_loc, EmpPos.e_join
FROM Employees
INNER JOIN EmpPos
ON Employees.e_id = EmpPos.e_id;

Q.) Recently join employees in a given range (MONTH)


SELECT *, DATEDIFF (MONTH, e_join, GETDATE()) as Diff
FROM EmpPos
WHERE DATEDIFF(MONTH, e_join, GETDATE()) BETWEEN 1 AND 25
ORDER BY e_join DESC;

Q.) Recently join employees in a given range (DAY)


SELECT *, DATEDIFF (DAY, e_join, GETDATE()) as Diff
FROM EmpPos
WHERE DATEDIFF(DAY, e_join, GETDATE()) BETWEEN 200 AND 600
ORDER BY e_join DESC;

Q.) Recently join employees in a given range (YEAR)

SELECT *, DATEDIFF (YEAR, e_join, GETDATE()) as Diff FROM EmpPos WHERE DATEDIFF(YEAR, e_join, GETDATE()) BETWEEN 1 AND 5 ORDER BY e_join DESC;

Q.) Dept with the highest number of employees


SELECT TOP 1 COUNT(*) AS cnt_of_emp_as_per_dep, e_dept FROM Employees
GROUP BY e_dept
ORDER BY COUNT(*) DESC;

Q.) Join 3 tables in SQL server


SELECT ENAME, DNAME, Gender
FROM EmployeeD
JOIN Departements ON EmployeeD.DID = Departements.DID
JOIN Genders ON Genders.GID = EmployeeD.GID;

Q.) If we have 2 different datatypes we can do that too.


select T1.T1Column1,T2.T2Column1
from T1
JOIN T2 on T1.ID = T2.ID;


Alter Table T2 Alter column Id nvarchar(3); /*Change datatype*/

Q.) Employee who join in December


select * from EmpPos where DATEPART(Month,e_join)='12';

Q.) Average Cost Of Individual Departement


SELECT e_dept, AVG(e_sal) AS DEPT_COST_AVERAGE FROM Employees
WHERE e_dept = 'ALY'
GROUP BY e_dept;

Q.) Give e_name and how many happen he/she has month has joined the organization


SELECT 
Employees.e_name,
EmpPos.e_join, DATEDIFF(MM,CONVERT(DATETIME, e_join),GETDATE()) AS Months_Of_Joining
FROM Employees
INNER JOIN EmpPos 
ON Employees.e_id = EmpPos.e_id;

Q.) Display all the e_name and their specific month of joining


SELECT Employees.e_name,
EmpPos.e_join
FROM Employees
INNER JOIN EmpPos
ON Employees.e_id = EmpPos.e_id
WHERE DATEPART(Month,e_join)='12';

MySQL Prac

Create table Employee


create table Employee(
    empid int Not null primary key auto_increment,
    first_name varchar(25), 
    last_name varchar(25), 
    salary int, 
    joining_date datetime,
    department_name varchar(25)
);

insert into Employee values
(0001,'Krish','Naik',50000,'14-12-11 09:00:00','Development'),
(0002,'Sudhanshu','Kumar',60000,'14-12-11 09:00:00','Development'),
(0003,'Sanket','Kumar',70000,'14-12-12 09:00:00','HR'),
(0004,'Darius','Bengali',70000,'14-12-13 09:00:00','HR'),
(0005,'Satish','Bhansal',30000,'15-10-21 09:00:00','Accountant'),
(0006,'Shaktiman','Hero',50000,'15-10-15 09:00:00','Accountant');

Check Schema


desc Employee;

Creating Bonus Table


create table Bonus(
    emp_ref_id int,
    bonus_amt int,
    bonus_date datetime,
    foreign key(emp_ref_id) references Employee(empid)  # foreign key example
    on Delete cascade                                   # change in employee table then changes will also happen in Bonus table
);

insert into bonus values
(0001,5000,'16-03-14'),
(0002,5000,'16-03-13'),
(0003,5000,'17-03-15'),
(0001,5000,'18-03-17'),
(0004,3500,'16-03-19'),
(0005,7000,'16-03-20'),
(0001,8000,'20-03-21'),
(0002,8500,'20-03-21');

Creating Emp Designation


create table Designation(
    emp_ref_id int,
    designation varchar(25),
    designation_date datetime,
    foreign key(emp_ref_id) references Employee(empid)
    on delete cascade
);

INSERT INTO designation VALUES
 (0001,'Manager','2016-02-5 00:00:00'),
 (0002,'Executive','2016-06-11 00:00:00'),
 (0003,'Executive','2016-06-11 00:00:00');

1. Write an SQL query to receive all details where the first name from the employee table which starts with 'k'



select * from Employee where first_name like 'k%';

2. Write an Sq1 query to print all details of the employees whose salaries between 10000 to 35000 fetch the Employee name



select * from Employee where salary between 10000 and 35000;

/* using subquery */

select concat(first_name,' ',last_name) as Fullname, salary from Employee
where empid in
(select empid from Employee where salary between 10000 and 55000);

3. SQL query to retrieve details of the employees who have joined on a date



select * from Employee where year(joining_date)=2014 and month(joining_date)=12 and day(joining_date)=13;

4. SQL query to fetch the number of employees in every department



select department_name, count(empid) as cnt 
from Employee Group by department_name;

5. SQL query to print details of the employee who are also Executives



/* Executives is in Designation */

select employee.empid, employee.first_name,
designation.emp_ref_id, designation.designation
from Employee
right Join designation
on Employee.empid = designation.emp_ref_id
where designation.designation = 'Executive';

6. SQL query to clone a new table from another table



/* Copy only schema / structure */

create table clone_emp like Employee;

/* Copy whole table */

create table clone_table as select * from Employee;

7. Top n salary of the employee



select * from Employee
order by salary
desc limit 2,1;

8. Nth highest salary



select first_name, salary from employee
order by salary 
desc limit 3,1;                               # N --> 4 } 4th highest salary

/* Without Limit */

SELECT * FROM Employee Emp1
WHERE (3-1) = 
(SELECT COUNT(DISTINCT(Emp2.Salary))
 FROM Employee Emp2
 WHERE Emp2.Salary > Emp1.Salary);

9. Arrange



SELECT * FROM tablename WHERE age>25 GROUP BY id HAVING something > 5 ORDER BY date_from LIMIT 10;

10. First 5 char of employee name



select LEFT(first_name, 5) as First_five from Employee;

11. Mask a column from credit_card_details such last few chars are converted to '*'



SELECT CONCAT(LEFT(credit_card_no,4),'*****') as masked_cred_card_details from CREDIT_CARD_DETAILS;

12. Get Credit_card_no from 3rd digit to 7th



select substr(credit_card_no,3,7) from credit_card_details;

13. Get all the employee name who has salary greater than Krish's salary



select * from employee where salary > (select salary from Employee where first_name='Krish');

14. Get all the employee name who has salary greater than Krish's salary and create a view



create view new_view as
select * from employee where salary > (select salary from Employee where first_name='Krish');

select * from new_view;

/* Let say you have wifi_id , wifi_speed , wifi_latency , date */


/* Question 1 : Find the average wifi speed for each wifi */
    
select avg(wifi_speed), wifi_id from wifi_details group by wifi_id;    
    
    
/* Question 2 : Find the average wifi speed for each wifi in the last 2 days */
    
select avg(wifi_speed), wifi_id, dt from wifi_details
where dt>=date_add(SYSDATE(), interval -2 day)
group by wifi_id, dt
order by dt DESC;

create table WIFI_DETAILS (
wifi_id varchar(100),
wifi_speed varchar(100),
wifi_latency int,
dt date )


INSERT INTO WIFI_DETAILS VALUES ( '1' , '120' , 12 , sysdate ( ) ) ;
INSERT INTO WIFI_DETAILS VALUES ( '2' , '110' , 12 , sysdate ( ) ) ;
INSERT INTO WIFI_DETAILS VALUES ( '1' , '111' , 12 , sysdate ( ) - 1 ) ;
INSERT INTO WIFI_DETAILS VALUES ( '2' , '122' , 12 , sysdate ( ) - 1 ) ;
INSERT INTO WIFI_DETAILS VALUES ( '1' , '89' , 12 , sysdate ( ) - 2 ) ;
INSERT INTO WIFI_DETAILS VALUES ( '2' , '56' , 12 , sysdate ( ) - 3 ) ;

15. Show the first and last values of the table



(select * from Employee order by empid ASC Limit 1)        /* First */
UNION
(select * from Employee order by empid DESC Limit 1);      /* Last */

16. First 4 char of name column string



SELECT SUBSTRING(first_name,1,4) FROM employee;

17. Get all the records that are not working in "HR" and "Development"



select * from employee
where department_name NOT in ("HR" , "Development");

18. Top 2 and last 2 salary



(select * from Employee order by salary asc limit 2)
UNION
(select * from Employee order by salary desc limit 2);

19. Employee Id in Ascending and Salary in descending



select * from Employee order by Department_name ASC, salary DESC;

20. Joining date in December



select * from employee
where month(joining_date) = 12;

21. Change datatype of a column



ALTER TABLE employee
ALTER COLUMN DateOfBirth date;

22. Remove Duplicate Rows



DELETE t1 FROM Employee t1
INNER JOIN Employee t2 
WHERE t1.empid < t2.empid;

23. Position of "a" in each first_name



select first_name, instr(first_name, 'a') from worker;

24. Unique value of the department and its length



select distinct length(department) from worker;

25. details of the workers whose names contain "a"



select * from worker
where instr(first_name, "a");

26. details of a worker whose name ends with 'h' and contains 6 alphabets



select * from worker
where first_name like "_____h"

# ---------------------------------------------------------------

select * from worker
where lenght(first_name) = 6 and first_name like "%h";

27. number of workers in each department



select department, count(*) as numOfWorker from worker
group by department
order by numOfWorker DESC;

28. Print details of the workers who are also managers



select worker_id, first_name, department, salary, worker_ref_id, worker_title
from worker
left join title
on worker.worker_id = title.worker_ref_id
where title.worker_title = 'Manager';

29. Show only odd rows from the table worker



select * from worker
where (worker_id%2) != 0;

30. Show top n records of the table



select * from worker limit 2;

31. 3rd highest salary from the table



select * from worker as w1
where (3-1) = (select count(distinct(w2.salary)) from worker as w2
               where w1.salary < w2.salary);

# -------------------------------------------------------------------------

select salary from worker
order by salary DESC
LIMIT N-1,1;

32. List of workers with the same salary



select department, count(worker_id) as numOfEmp from worker
group by department having count(worker_id) < 4 ;

34. department with less than 5 people



SELECT t.DEPARTMENT,t.FIRST_NAME,t.Salary 
from(SELECT max(Salary) as TotalSalary,DEPARTMENT from Worker group by DEPARTMENT) as TempNew 
Inner Join Worker t on TempNew.DEPARTMENT=t.DEPARTMENT 
and TempNew.TotalSalary=t.Salary;

36. create a view of only admin, show, drop



/* create view */

CREATE VIEW department_only_admin AS
SELECT * FROM worker
where department = 'Admin';

/* show view */

SELECT * FROM department_only_admin;

/* drop view */

DROP VIEW department_only_admin;

37. Rich or Poor Employee



SELECT empid, first_name, salary,
IF (salary>55000, 'Rich Employee', 'Poor Employee')
AS emp_Status
FROM Employee;

38. Rich, Poor, Middle Employee Status using CASE



SELECT empid, first_name, salary,
(CASE
    WHEN salary > 50000 THEN "RICH EMP"
    WHEN salary = 50000 THEN "MIDDLE EMP"
    ELSE "POOR EMP"
END) AS emp_status
FROM Employee;

39. Create a Stored Procedure



/ *Schemas ==> Stored Procedures (Right Click) ==> Create Stored Procedure ==> {write code} ==> Apply ==> Apply */

/* call strored_procedure_name */


CREATE DEFINER=`root`@`localhost` PROCEDURE `get_emp_info`()
BEGIN
    select * from Employee;
END

/* ---------------------------------------------------------------- */

call get_emp_info;

/* IN */

CREATE DEFINER=`root`@`localhost` PROCEDURE `get_rich_emp_info`(IN amt int)
BEGIN
    select * from Employee where salary > amt;
END

/* ---------------------------------------------------------------- */

call get_emp_info(50000);

/* OUT */

CREATE DEFINER=`root`@`localhost` PROCEDURE `get_rich_emp_info`(OUT records int)
BEGIN
    select count(*) into records from Employee where salary = 50000;
END

/* ---------------------------------------------------------------- */

call get_rich_emp_info(@records);

select @records as TotalRecords;

40. Commit and Rollback



SET SQL_SAFE_UPDATES = 0;

# your code SQL here

SET SQL_SAFE_UPDATES = 1;

/* example: */
    
SET autocommit = 0;

update employee set salary = 60000 where first_name = 'Krish';

SET autocommit = 1;

rollback;

41. Export in CSV



SELECT Id, Name, Email, Phone, City FROM employee
INTO OUTFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/employee_backup.csv'   
FIELDS ENCLOSED BY '"'   
TERMINATED BY ';'   
ESCAPED BY '"'   
LINES TERMINATED BY '\r\n';

42. CSV in MySQL



LOAD DATA LOCAL INFILE '/path/to/your/csv/file/model.csv'
INTO TABLE test.dummy FIELDS TERMINATED BY ','
ENCLOSED BY '"' LINES TERMINATED BY '\n';

So these are the question which is asked the most.

How To Deal With Joint Words In Hindi Text (Devanagari Words) And Successfully Put Them On Images Using Python

2022-03-06T14:40:00.009+05:30

How To Deal With Joint Words In Hindi Text And Successfully Put Them On Images Using Python

Result Image

So few days ago I was working on a task which was related to Image manipulation... Where I just have to put some random English text on an image with adjusting their parameters such as text's font, size, color... etc, and I was very much successful in doing it using python library such as pillow. So I though that it would be great that I do that thing for a different language. So I chose a completely different language that is devanagari script, which includes languages such as sanskrit, hindi, marathi..etc. Well, believe me, it wasn't as easy as I thought😅.

So I used pillow library as my tool to put some hindi text on image same as I put English text on image... But then I noticed that the spelling of the text which I had wrote is incorrect and completely change the meaning of the sentence which means that python or i can specifically say pillow library is not able to work with joint words in devanagari script.

Mistake in marathi text

So I googled each and every page and sites just to find a completed detailed solution for this problem... I googled every possible questions such as

How can I print Hindi sentence (unicode) on an image in Python?

How to put correct Hindi/Marathi text on image using PIL in python?

Devanagari font not working in python.

How do I add Hindi text on an image in windows?

After so much research and trial and error I finally came up with a step by step guide to Put Hindi Text On Image Using Python or we can also say Put Devanagari Text On Image Using Python.

Requirerments:

- Python 3.6 or greater

- pip install numpy

- pip install pandas

Let's dive into the solution for this problem.

Step1: Install Pyvips

Pyvips don’t manipulate images directly, instead they create pipelines of image processing operations building on a source image. When the end of the pipe is connected to a destination, the whole pipeline executes at once, streaming the image in parallel from source to destination a section at a time.

Because pyvips is parallel, it’s quick, and because it doesn’t need to keep entire images in memory, it’s light.

You can check the documentations for pyvips for more details:

https://libvips.github.io/pyvips/README.html

Now go to your jupyter notebook and type:

 !pip install pyvips

You can also go in command prompt and type:

 pip install pyvips

Now if you go to your jupyter notebook and try to install it you will get an error from pyvips:

error while importing pyvips

Step2: Download & Locate

As you are done with installation pyvips, its done hurrah we are done... Haha, just kidding I wish life would have been so easy.

Lets deal with this error OSError: cannot load library 'libgobject-2.0-0.dll': error 0x7e.

Basically this is the step-by-step guide to install pyvips and using it for image manipulation task.

Go to the following link:

https://github.com/libvips/libvips/releases

which will look some thing like this:

click on the link

Now based on your following system download the suitable file, mine is windows10 64-bits so I downloaded that file. After click it will look some thing like this:

https://vips-dev-w64-all-8.12.2.zip/ ...Download

After the downloading the file go any location and save it ==> extract the files which will give you folder name vips-dev-8.12.
vips-dev-8.12 folder

Step3: Setup path in environment variables

Now go to vips-dev-8.12 ==> bin
copy it's path in my system it is in
C:\vips-dev-8.12\bin

Now go to your environment variables & paste it.

For user variables:

Righ click in ThisPc ==> Advanced System settings ==> Environment Variables ==> User variabes ==> Path ==> Edit ==> New ==> Paste ==> Ok

pasting path in user variables

For System variables:

Righ click in ThisPc ==> Advanced System settings ==> Environment Variables ==> Sytem variabes ==> Path ==> Edit ==> New ==> Paste ==> Ok

pasting path in System variable

Step4: Almost done with the process

* Restart your system *

Now go to your Jupyter Notebook or python file and type import pyvips and yess we did it. So we are half way though.

Now Let's start with our main contain:

 # IMPORTS  
 
 import os  
 import pyvips  
 import textwrap  
 from PIL import *  
 import pandas as pd  
 from PIL import Image  
 from PIL import ImageFont  
 from PIL import ImageDraw   
 

 # MAKE A FUNCTION  
 
 def bgoutput(filename, text):  
 
   rendered_text, feedback = pyvips.Image.text(text,   
                         font='Mangal', fontfile='Mangal Regular.ttf',   
                         width=900, height=900,   
                         autofit_dpi=True) 
 
   rendered_text = rendered_text.gravity('centre', 1500, 1500)  

   image = rendered_text.new_from_image([0, 0, 0]).bandjoin(rendered_text)  
   image.write_to_file(f'{filename}.png') 
 
 # GENERATE OUTPUT 1   

 text = "परिस्थितियां विपरीत हो तो कुछ लोग टूट जाते हैं, और कुछ लोग रिकॉर्ड तोड़ देते हैं..!!"  
 bgoutput('new', text)  

 # COMBINE OUTPUT 1 WITH BACKGROUND IMAGE  

 img = Image.open('new.png')   
 b1 = Image.open('bg_img.png')  
 img = img.resize((1000,1000))  
 b1.paste(img,(50,50), mask=img)  

 # GENERATE FINAL OUTPUT  

 b1.save("final.png")

Output:

Final Output

Understand the parameters here you can tune some values and make changes based on your requirements.

Example:

out = pyvips.Image.text(text, font=str, width=int, height=int, align=Union[str, Align], rgba=bool, dpi=int, justify=bool, spacing=int, fontfile=str, autofit_dpi=bool)

Returns:

out (Image): Output image

Args:

text (str): Text to render

Keyword args:

font (str): Font to render with

width (int): Maximum image width in pixels

height (int): Maximum image height in pixels

align (Union[str, Align]): Align on the low, centre or high edge

rgba (bool): Enable RGBA output

dpi (int): DPI to render at

justify (bool): Justify lines

spacing (int): Line spacing

fontfile (str): Load this font file

Other Parameters:

autofit_dpi (int): DPI selected by autofit

Finally finally did it😀.

So we hope that you enjoyed this session. If you did then please share it with your friends and spread this knowledge.

Support us by following our social media:

Instagram:

https://www.instagram.com/infinitycode_x/

Facebook:

https://www.facebook.com/InfinitycodeX/

Twitter:

https://twitter.com/InfinityCodeX1

Youtube:

https://www.youtube.com/channel/UC9F_A8HF2PISaQnP3TUOKWg

TOP 10 JUTSUS OF FEATURE ENGINEERING EVERY DATA SCIENTISTS NINJA SHOULD KNOW RIGHT NOW

2020-12-02T17:03:00.021+05:30

TOP 10 JUTSUS OF FEATURE ENGINEERING EVERY DATA SCIENTISTS NINJA SHOULD KNOW RIGHT NOW

If you want to be a successful data scientist and to make your model predict the most accurate result, then the following article is for you.

What is Feature Engineering?

As per Wikipedia, Feature engineering is the process of using domain knowledge to extract features from raw data via data mining techniques. These features can be used to improve the performance of machine learning algorithms. Feature engineering can be considered as applied machine learning itself.

Okay, so we saw what Wikipedia said about it but still, we want to know more about it. So let’s divide this term into 2 parts for better understanding i.e Feature and Engineering.

What are the Features in a dataset?

Basically, all machine learning algorithms use some input data (independent data) to generate output. These input data are features, which are usually in the form of structured columns.

So, why do we engineer it?

To get the most accurate or I can say to get the most precise output from our machine learning the algorithm we need to give it cleanest data possible which should be compatible with our machine learning algorithm.

So the process of preparing the proper input dataset for our machine learning model is known as Feature Engineering.

Now you will ask how important is this process of feature engineering to generate any machine-learning project?

According to Forbs survey, data scientist spends around 80% of their time on data preparation.

Link of the complete survey by Forbes.

Data scientists spend 60% of their time cleaning and organizing data. Collecting data sets come second at 19% of their time, meaning data scientists spend around 80% of their time on preparing and managing data for analysis.

76% of data scientists view data preparation as the least enjoyable part of their work

57% of data scientists regard cleaning and organizing data as the least enjoyable part of their work and 19% say this about collecting data sets.

List of Techniques you can find in this blog:

1.) Techniques of Imputation of numerical and categorical data

2.) Dealing with outliers

3.) Binning

4.) Techniques of dealing with Gaussian-Distribution / Skewness

5.) OneHotEncoding and OrdinalEncoder

6.) Feature splitting & extraction

7.) Group By

8.) Concat, Merge, Join

9.) Scaling

10.) Extracting Date

1.) Techniques of Imputation of numerical and categorical data

In this we deal with missing values that are present in our dataset.

The most simple solution to deal with missing values is to drop that row or column, suppose that you have more than 75% of values missing that you can definitely drop that row or column.


 threshold = 0.75  
 # Columns:  
 data = data[data.columns[data.isnull().mean() < threshold]]  
 
 #------------------------------------------------------------------------
 
 # Rows:  
 data = data.loc[data.columns[data.isnull().mean(axis=1) < threshold]]

The most common way to deal with missing values is through fill 0, Mean, Mean, or Mode value in that missing space.


 # fill NaN values with 0  
 data = data.fillna(0)  
 
 #------------------------------------------------
 
 # fill NaN values with median  
 data = data.fillna(data.mean())  
 
 #------------------------------------------------
 
 # fill NaN values with median: Better than Mean  
 data = data.fillna(data.median())  
 
 #------------------------------------------------
 
 # fill NaN values with mode  
 data = data.fillna(data.mode())

To deal with numerical data specifically you can use imputers they are very useful to get the perfect value to fill in that missing space.

Above mention is too naïve way of imputation, but we are ninjas we have some of our own ninja techniques we can use

(i) Simple Imputer

-A simple imputer is univariate which means it will take only a single feature in the count.


 # using simple imputer  
 
 import numpy as np  
 from sklearn.impute import SimpleImputer  
 imputer = SimpleImputer(missing_values=np.nan, strategy='median') # can be mean, median, mode  
 imputer = imputer.fit(x)  
 data = imputer.transform(x)  
 print('Imputed Data:',data)

-This is quite a blunt approach.

-What if the feature “x” which has NaN values is very well co-correlated with the features such as “y” or “z”. Let’s say, people with higher “Age” give more “Rent” and people with lower “Age” give less “Rent”. So that suggests a multivariate approach. Which just means it takes multiple features into count.

-So the Iterative Imputer and KNN Imputer comes into the concentration.

(ii) Iterative Imputer

-In iterative imputer for all the rows in which “Age” is not missing sklearn trains a regression model.

-Where other features such as “Gender” and “Rent” are considered as independent input features and “Age” is considered as the target output feature.

-Now for all rows in which “Age” is missing it makes predictions for “Age” by using features “Gender” and “Rent” to trained model.

 
 # IterativeImputer  
 
 from sklearn.experimental import enable_iterative_imputer  
 from sklearn.impute import IterativeImputer  
 imputer_it = IterativeImputer()  
 imputer_it.fit_transform(x)

NOTE: This model used by iterative imputer is totally independent of the model you are using this dataset as training data.

(iii) KNN Imputer

-Let’s pretend we have a row in which “Age” is missing.

-Then sklearn finds the 2 most similar rows measured by how close the “Gender” and “Fare” values are to this row.

 
  # KNNImputer   
  
  from sklearn.experimental import enable_iterative_imputer   
  from sklearn.impute import KNNImputer   
  imputer_knn = KNNImputer(n_neighbors=2)   
  imputer_knn.fit_transform(x)

Dealing with the categorical and nominal data with the help of mode


 data['categorical_col'] = data['categorical_col'].fillna(data['categorical_col'].mode()[0])

2.) Dealing with outliers

Detecting outliers:

Using Z-score:

Data that fall outside of the 3^rd standard deviation is considered as an outlier. We can use a Z-score if Z-score falls outside of μ+3 or μ-3 then we will consider it as an outlier.

Z-Score

 
 import numpy as np  
 
 box = [10,20,15,25,11,12,16,26,19,29,3000]  
 
 outliers = []  
 def detect_outliers(data):  
   Threshold = 3  
   mean = np.mean(data)  
   std = np.std(data)  
   for i in data:  
     z_score = (i-mean)/std  
     if np.abs(z_score) > Threshold:  
       outliers.append(i)  
   return outliers  
 out = detect_outliers(box)  
 out

Using IQR:

Datapoint that fall outside of 1.5 times od Inter Quartile Range above 1^st the quartile and the 3^rd quartile.

 
 import numpy as np  
 
 box = [10,20,15,25,11,12,16,26,19,29,3000]  
 
 box = sorted(box)  
 Q1,Q3 = np.percentile(box, [25,75])  
 print(Q1,Q3)  
 iqr = Q3 - Q1  
 print(iqr)  
 
 # Lower bound and higher bound values  
 lower_bound_val = Q1 - (1.5*iqr)  
 higher_bound_val = Q3 + (1.5*iqr)  
 print(lower_bound_val,higher_bound_val)

Using Box-plot:

The values which are beyond the Max or Min are considered as the outliers.

 
   for feature in data:  
   dataset = data.copy()  
   if 0 in dataset[feature].unique():  
     pass  
   else:  
     dataset[feature] = np.log(dataset[feature])  
     dataset.boxplot(column=feature)  
     plt.ylabel(feature)  
     plt.title(feature)  
     plt.show()

Dropping outliers with the Standard Deviation:

 
 fact = 3  
 upper_bound_val = data['column'].mean() + data['column'].std() * fact  
 lower_bound_val = data['column'].mean() - data['column'].std() * fact  
 data = data[(data['column'] < upper_bound_val) & (data['column'] > lower_bound_val)]

 
 data = pd.DataFrame(np.random.randn(500,4))

 from scipy import stats

 data[(np.abs(stats.zscore(data)) < 3).all(axis=1)]

Dropping the outliers rows with percentile:

  
 upper_bound_val = data['column'].quantile(0.95)  
 lower_bound_val = data['column'].quantile(0.05)  
 data = data[(data['column'] < upper_bound_val) & (data['column'] > lower_bound_val)]

3.) Binning

Image Credit: https://wisdomschema.com/data-binning/

-Binning can be applied to both categorical and numerical data.

- The main reason behind binning is to make the model more robust and prevent overfitting, however, it has a performance cost.

#Numerical Binning Example

Value Bin

0-30 => Fail

31-70 => Average

71-100 => Excelent

#Categorical Binning Example

Value Bin

Mumbai => Maharashtra

Pune => Maharashtra

Bikaner => Rajasthan

Jaipur => Rajasthan

Numerical Binning Example:


data['bin'] = pd.cut(data['value'], bins=[0,30,70,100], labels=["Fail", "Average", "Excelent"])

value | bin

0 | 2 | Fail
1 | 45 | Average
2 | 7 | Fail
3 | 85 | Excellent
4 | 28 | Fail

Categorical Binning Example:

 
  conditions = [  
   data['State'].str.contains('Mumbai'),  
   data['State'].str.contains('Pune'),  
   data['State'].str.contains('Bikaner'),  
   data['State'].str.contains('Jaipur')]  
  choices = ['Maharashtra', 'Maharashtra', 'Rajasthan', 'Rajasthan']  
  data['Continent'] = np.select(conditions, choices, default='Other')

  value    | bin

4.)Techniques of dealing with Gaussian-Distribution / Skewness

-Helps to handles skewed data and after transformation, the distribution becomes more approximate to the normal.

-Decreases the effect of the outliers to the normalization of magnitude difference and the model becomes more robust.

(i)Log Transformation:

Log transformation is a data transformation method in which it replaces by log(x) with base 10, base 2, or natural log.

 
 data['log_column'] = np.log(data['column']+1)

(ii)Reciprocal Transformation:

In Reciprocal Transformation, x will replace by the inverse of X(1/X). The reciprocal transformation will give little effect on the shape of the distribution. This transformation can be only used for non-zero values. The skewness for the transformed data is increased.

 
 data['Reciprocal_column'] = 1/(data['column']+1)

(iii)Square-Root Transformation:

This transformation will give a moderate effect on distribution. The main advantage of square root transformation is, it can be applied to zero values. Here the x will replace by the square root(x). It is weaker than the Log Transformation.

 
  data['sqr_column'] = data['column']**(1/2)

5.)OneHotEncoding and OrdinalEncoder

image credit: https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding

Since our computer systems only understand numerical data. So categorical data makes no sense for them unless they are converted into the numerical data and for that we use techniques such as OneHotEncoding and OrdinalEncoders.

# For single column


  encoded_col = pd.get_dummies(data['column'], drop_first = True)

# For multi column

 
  encoded_multi_col = pd.get_dummies(data,columns = ['column1', 'column2', 'columnN'], drop_first=True)

The above mentioned is too naïve way of doing this, but we are ninjas we have a secret technique to get the optimum result.

For unordered (Nominal) data we will be using OneHotEncoding

Example: Male, Female

 
 # OneHotEncoding is for unordered (Nominal) data  
 # Example: Male, Female  
 
 from sklearn.preprocessing import OneHotEncoder  
 ohe = OneHotEncoder(sparse=False)  
 ohe.fit_transform(data[['column']])

For ordered (Ordinal) data we will be using OrdinalEncoder

Example: First, Second, Third

 
 # OrdinalEncoder is for ordered (Ordinal) data  
 # Example: First, Second, Third  
 from sklearn.preprocessing import OrdinalEncoder  
 oe = OrdinalEncoder(categories=[['First', 'Second', 'Third'],  
                 ['O','A','B','C']])  
 oe.fit_transform(data[['Rank', 'Grade']])

6.)Feature splitting & extraction

-By extracting the utilizable part of a column into a new feature:

*We enable the machine-learning algorithms to comprehend them.

*Make possible to bin and group them.

*Improve model performance by uncovering potential information.

Splitting a function is a good option, but however, there is one way of splitting features.

It depends on the characteristics of the column, how to split it.

Feature Splitting:

 
 import pandas as pd  
 import numpy as np  
 data = [('Arnold Schwarzenegger','M'),   
     ('Natasha Romanova','F'),   
     ('Sylvester Stallone','M'),   
     ('Gal Gadot','F'),   
     ('Dwayne Johnson','M')]  
 df = pd.DataFrame(data, columns=['name', 'gender'])  
 df


 # Extracting First Name  
 df.name.str.split(" ").map(lambda X : X[0])


 # Extracting Last Name  
 df.name.str.split(" ").map(lambda X : X[-1])

Feature Extraction:

 
 weather_data = [('1/1/2017', 32, 6, 'Rain'),  
         ('1/2/2017', 30, 7, 'Sunny'),  
         ('1/3/2017', 32, 2, 'Snow'),  
         ('1/4/2017', 34, 6, 'Snow'),  
         ('1/5/2017', 32, 4, 'Rain'),  
         ('1/6/2017', 32, 2, 'Sunny')  
         ]  
 df = pd.DataFrame(weather_data, columns=['day', 'temp', 'windspeed', 'weather'])  
 df


 df['day'][df['weather']=='Rain']


  df.day[df.temp == df.temp.max()]

7.) Group By

groupby() function is used to split the data into groups based on some criteria. pandas objects can be split on any of their axes. The abstract definition of grouping is to provide a mapping of labels to group names.

 
  weather_data = [('1/1/2017', 'Mumbai' , 32, 6, 'Rain'),  
         ('1/2/2017', 'Pune' , 30, 7, 'Sunny'),  
         ('1/3/2017', 'Mumbai' , 32, 2, 'Snow'),  
         ('1/4/2017', 'Pune' , 34, 6, 'Snow'),  
         ('1/5/2017', 'Mumbai' , 32, 4, 'Rain'),  
         ('1/6/2017', 'Delhi' , 32, 2, 'Sunny')  
         ]  
 data = pd.DataFrame(weather_data, columns=['day', 'city', 'temp', 'windspeed', 'weather'])  
 data

 
 grp_city = data.groupby('city')  
 for city, city_data in grp_city:  
   print(city)  
   print(city_data)


 # Get specific group  
 grp_city.get_group('Mumbai')

 
  print(grp_city.max())


  print(grp_city.mean())

 
  print(grp_city.describe())

8.) Concat, Merge, Join

Works like how we had worked on SQL.

 
  Maharashtra_weather_data = pd.DataFrame({  
   'city': ['Mumbai', 'Pune', 'Thane'],  
   'temperature': [32, 45, 30],  
   'humidity': [80, 60, 78]  
 })  
 Maharashtra_weather_data

 
 Gujarat_weather_data = pd.DataFrame({  
   'city': ['Surat', 'Rajkot', 'Mehsana'],  
   'temperature': [21, 24, 35],  
   'humidity': [68, 65, 75]  
 })  
 
 Gujarat_weather_data

(i) Concat:

-Column Wise Concatenation


 df = pd.concat([Maharashtra_weather_data, Gujarat_weather_data])  
 df

-Row Wise Concatenation

 
 df = pd.concat([Maharashtra_weather_data, Gujarat_weather_data], axis=1)  
 df

(ii) Merge

 
  temp_data = pd.DataFrame({  
   'city':['Mumbai','Delhi','Banglore','Hydrabad'],  
   'temp':[32,45,30,40]  
 })  
 temp_data


 humidity_data = pd.DataFrame({  
   'city':['Mumbai','Delhi','Banglore'],  
   'humidity':[62,65,70]  
 })  
 humidity_data


 # Merge 2 dataframe without explicitly mentioning the index  
 
 df = pd.merge(temp_data, humidity_data, on='city')  
 df

(iii) Join

 
 df = pd.merge(temp_data, humidity_data, on='city',how='outer')  
 df

9.) Scaling

Scaling are done basically in 2 ways first is Normalization and the second is Standardization.

 
 data = pd.DataFrame({  
   'name':['Ram', 'Lakhan', 'Shiva', 'Ria', 'Lucy', 'Suraj', 'Rohan', 'Anny', 'Priya', 'Niraj'],  
   'age':[25,40,33,26,30,35,28,43,36,50],  
   'Salary':[25000, 32000, 50000, 33000, 20000, 29000, 22000, 52000, 23000, 26000],  
   'purchased':[1,0,1,1,0,0,0,1,0,1]  
 })  
 data

(i) Normalization (MinMaxScaler)

* X_norm = X – X_min / X_max – X_min

* Range 0 to 1

* Due to the decrease in standard deviation the effect of outliers increases.

*Before Normalization, it is recommended to handle the outliers.


 from sklearn.model_selection import train_test_split  
 X = data.drop(['name','purchased'], axis=1)  
 y = data['purchased']  
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)


 from sklearn.preprocessing import MinMaxScaler  

 Min_Max_scaler = MinMaxScaler()  
 
 Min_Max_X_train = Min_Max_scaler.fit_transform(X_train)  
 Min_Max_X_test = Min_Max_scaler.transform(X_test)  
 
 Min_Max_X_train

 
  Min_Max_X_test

(ii) Standardization (StandardScaler)

*Z = X - μ / σ

*μ = 0 and σ = 1

*Range -1 to 1

*If the standard deviation of feature is different, their range also would differ from each other.

 
 from sklearn.preprocessing import StandardScaler  
 
 StandardScaler_scaler = StandardScaler()  
 
 StandardScaler_X_train = StandardScaler_scaler.fit_transform(X_train)  
 StandardScaler_X_test = StandardScaler_scaler.transform(X_test)  
 
 StandardScaler_X_train

 
 StandardScaler_X_test

10.) Extracting Date

There are 3 ways we can preprocessing date

*Extracting the parts of the date into the different columns: YEAR, MONTH, DAY.

*Extracting the time period between the current date and column in terms of YEAR, MONTH, DAY.

*Extracting some specific features from the date such as the name of the weekday, weekend, etc.

*With dealing with date features like that our machine learning model can easily understand data and deal with the data.

 
 from datetime import date  
 data = pd.DataFrame({  
   'date': ['01/01/2017', '04/12/2000', '23/04/2011', '11/02/2008', '08/08/2018']  
 })  
 
 data

 
  data.info()


 # Transform String to Date  
 
 data['date'] = pd.to_datetime(data.date, format='%d/%m/%Y')  
 data.info()

 
 # Extracat year  
 
 data['year'] = data['date'].dt.year  
 data


 # Extract month  
 
 data['month'] = data['date'].dt.month  
 data


 # Extract day  
 
 data['day'] = data['date'].dt.day  
 data

 
 # Extracting passed year since the date   
 
 data['passed_years'] = date.today().year - data['date'].dt.year  
 data

 
 # Extracting passed month since the date   
 
 data['passed_months'] = (date.today().year - data['date'].dt.year)*12 + (date.today().month - data['date'].dt.month)  
 data

Here is the Github link of all the code which is mentioned here:

Github Link of the Notebook: Link

Use all the tips and tricks which are mentioned above and you will surely get amazing data to train your model on.

So we hope that you enjoyed this session. If you did then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :
https://www.facebook.com/InfinitycodeX/

Twitter :
https://twitter.com/InfinityCodeX1

Unable to render code block error in Github Solution

2020-10-26T16:54:00.010+05:30

Unable to render code block Github Error for .ipynb Solution

I am sure most of you Data science enthusiasts are facing this kind of error while you open your any .ipynb on GitHub via a link.

Today I’m going to give you a very simple solution to fix this issue... No, it’s not to keep clicking on the reload button.

Step 1:

First, go to your .ipynb file which is showing an error and copy the URL of that .ipynb file.

Step 2:

Visit the following link:-

https://nbviewer.jupyter.org/

Step 3:

Paste your copied link and hit enter.

Result:

I hope this helped you. If it did then please share it with your friends and spread this knowledge.

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Heart Disease Prediction End to End Machine Learning Project (Interface Creation)

2020-10-14T11:49:00.003+05:30

Heart Disease Prediction End to End

Machine Learning Project

(Interface Creation)

Part2: Creating a Web Interface for our Machine Learning Model.

In this part, we are going to create a simple web application using HTML and CSS which will make our model more interactive with the user.

If you haven’t checked our Part1 where we had created our machine learning model to predict heart disease, I will highly recommend you to check that part and then dive into this.

Part1:

Heart Disease Prediction End to End Machine Learning Project

(Model Creation)

Here is the app.py code:

 from flask import Flask, request, render_template  
 import joblib  
 import sklearn  
 import pickle, gzip  
 import pandas as pd  
 import numpy as np  
 app = Flask(__name__)  
 model = joblib.load('Heart_Disease_Prediction.pkl')  
 @app.route('/')  
 def home():  
   return render_template("home.html")  
 @app.route("/predict", methods=["POST"])  
 def predict():  
   age = request.form["age"]  
   sex = request.form["sex"]  
   trestbps = request.form["trestbps"]  
   chol = request.form["chol"]  
   oldpeak = request.form["oldpeak"]  
   thalach = request.form["thalach"]  
   fbs = request.form["fbs"]  
   exang = request.form["exang"]  
   slope = request.form["slope"]  
   cp = request.form["cp"]  
   thal = request.form["thal"]  
   ca = request.form["ca"]  
   restecg = request.form["restecg"]  
   arr = np.array([[age, sex, cp, trestbps,  
            chol, fbs, restecg, thalach,  
            exang, oldpeak, slope, ca,  
            thal]])  
   pred = model.predict(arr)  
   if pred == 0:  
     res_val = "NO HEART PROBLEM"  
   else:  
     res_val = "HEART PROBLEM"  
   return render_template('home.html', prediction_text='PATIENT HAS {}'.format(res_val))  
 if __name__ == "__main__":  
   app.run(debug=True)

Here is the HTML & CSS code:

 <html lang="en">  
 <head>  
   <meta charset="UTF-8">  
   <meta name="viewport" content="width=device-width, initial-scale=1.0">  
   <meta http-equiv="X-UA-Compatible" content="ie=edge">  
 <!--  <link rel="stylesheet" href="styles.css">-->  
   <title>Heart Disease Prediction</title>  
 </head>  
 <style>  
   @import url('https://fonts.googleapis.com/css2?family=Montserrat:wght@500&display=swap');  
 *{  
   margin: 0;  
   padding: 0;  
   box-sizing: border-box;  
   font-family: 'Montserrat', sans-serif;  
   color:black;  
 }  
 body{  
   background: url("https://cdn.pixabay.com/photo/2016/08/10/20/26/stethoscope-1584223_1280.jpg") no-repeat top center;  
   padding: 0 10px;  
   background-size: cover;  
 }  
 .wrapper{  
   max-width: 500px;  
   width: 100%;  
   background: rgba(0, 0, 0, 0.7);  
   margin: 20px auto;  
   padding: 30px;  
   box-shadow: 1px 1px 2px rgba(0, 0, 0, 1.25);  
 }  
 .wrapper .title{  
   font-size: 24px;  
   font-weight: 700;  
   margin-bottom: 25px;  
   color: #fec107;  
   text-transform: uppercase;  
   text-align: center;  
 }  
 .wrapper .form{  
   width: 100%;  
 }  
 .wrapper .form .input_field{  
   margin-bottom: 15px;  
   display: flex;  
   align-items: center;  
 }  
 .wrapper .form .input_field label{  
   width:80px;  
   font: bold;  
   color: wheat;  
   margin-right: 10px;  
   font-size: 14px;  
 }  
 .wrapper .form .input_field input{  
   width: 100px;  
 }  
 .wrapper .form .input_field .textarea{  
   resize: none;  
   height: 250px;  
   width: 500px;  
 }  
 </style>  
 <body>  
 <form action="{{ url_for('predict')}}" method="post">  
   <div class="wrapper">  
     <div class="title">  
       <h1 style="color: white;">Heart Disease Prediction</h1>  
     </div>  
     <div class="form">  
       <div class="input_field">  
         <textarea style="color: white; background: rgba(0, 0, 0, 0.3);" class="textarea" readonly>  
 It's a clean, easy to understand set of data. However, the meaning of some of the column headers are not obvious. Here's what they mean,  
 Age: displays the age of the individual.  
 Sex: displays the gender of the individual using the following format :  
 1 = male  
 0 = female  
 Chest-pain type: displays the type of chest-pain experienced by the individual using the following format :  
 0 = typical angina  
 1 = atypical angina  
 2 = non — anginal pain  
 3 = asymptotic  
 Resting Blood Pressure: displays the resting blood pressure value of an individual in mmHg (unit)  
 Serum Cholestrol: displays the serum cholesterol in mg/dl (unit)  
 Fasting Blood Sugar: compares the fasting blood sugar value of an individual with 120mg/dl.  
 If fasting blood sugar > 120mg/dl then : 1 (true) else : 0 (false)  
 Resting ECG : displays resting electrocardiographic results  
 0 = normal  
 1 = having ST-T wave abnormality  
 2 = left ventricular hyperthrophy  
 Max heart rate achieved : displays the max heart rate achieved by an individual.  
 Exercise induced angina :  
 1 = yes  
 0 = no  
 ST depression induced by exercise relative to rest: displays the value which is an integer or float.  
 Peak exercise ST segment :  
 1 = upsloping  
 2 = flat  
 3 = downsloping  
 Number of major vessels (0–3) colored by flourosopy : displays the value as integer or float.  
 Thal : displays the thalassemia :  
 0 = normal  
 1 = fixed defect  
 2 = reversible defect  
 Diagnosis of heart disease : Displays whether the individual is suffering from heart disease or not :  
 0 = absence  
 1, 2, 3, 4 = present.  
           </textarea>  
        </div>  
       <div class="input_field">  
         <label>AGE</label>  
         <input type="number" id="age" min="0" max="150" name="age" class="input_text">  
         <br/>  <i style="font-size: 10px; color: white;">(Age: 1 - 150)</i>  
       </div>  
       <div class="input_field">  
         <label>SEX</label>  
          <input type="number" min="0" max="1" name="sex" id="sex" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Male: 1 & Female: 0)</i>  
        </div>  
        <div class="input_field">  
         <label>CP</label>  
          <input type="number" name="cp" id="cp" min="0" max="3" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Single Value From Range 0-3)</i>  
        </div>  
        <div class="input_field">  
         <label>TRESTBPS</label>  
          <input name="trestbps" id="trestbps" type="number" min="0" step="1" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Non-Decimal Value)</i>  
        </div>  
        <div class="input_field">  
         <label>CHOL</label>  
          <input name="chol" id="chol" type="number" min="0" step="1" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Non-Decimal Value)</i>  
        </div>  
        <div class="input_field">  
         <label>FBS</label>  
          <input type="number" min="0" max="1" name="fbs" id="fbs" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(1 = True; 0 = False)</i>  
        </div>  
        <div class="input_field">  
         <label>RESTECG</label>  
          <input type="number" name="restecg" min="0" max="2" id="restecg" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Single Value From Range 0-2)</i>  
        </div>  
        <div class="input_field">  
         <label>THALACH</label>  
          <input name="thalach" id="thalach" type="number" min="0" step="1" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Non-Decimal Value)</i>  
        </div>  
        <div class="input_field">  
         <label>EXANG</label>  
          <input type="number" min="0" max="1" name="exang" id="exang" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Exercise: 1 = YES; 0 = NO)</i>  
        </div>  
        <div class="input_field">  
         <label>OLDPEAK</label>  
          <input name="oldpeak" id="oldpeak" type="number" min="0" step="0.01" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Decimal Value)</i>  
        </div>  
        <div class="input_field">  
         <label>SLOPE</label>  
          <input type="number" min="0" max="2" name="slope" id="slope" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Single Value From Range 0-2)</i>  
        </div>  
        <div class="input_field">  
         <label>CA</label>  
          <input type="number" name="ca" min="0" max="4" id="ca" class="input_text">  
          <br/>  <i style="font-size: 10px; color: white;">(Enter Single Value From Range 0-4)</i>  
        </div>  
        <div class="input_field">  
         <label>THAL</label>  
          <input type="number" name="thal" min="0" max="3" id="thal" class="input_text">  
          <br>  <i style="font-size: 10px; color: white;">(Enter Single Value From Range 0-3)</i>  
        </div>  
        <div class="input_field">  
         <input type="submit" style="color:black; background: skyblue;" id="submit" value="SUBMIT">  
       </div>  
       <div>  
         <center>  
           <h2 style="color: white; margin-bottom:10px;">  
             {{prediction_text}}  
           </h2>  
         </center>  
       </div>  
     </div>  
   </div>  
 </form>  
 </body>  
 </html>

Go to the visual code or notepad and create an HTML file name as home.html then paste this above code into it.

Create a new file named requirements.txt and paste the below text into that file.

Flask==1.1.2
pandas==1.0.4
numpy==1.18.4
joblib==0.15.1
scikit_learn==0.23.2
gunicorn==20.0.4
Werkzeug==0.16.0
Jinja2==2.10.1
itsdangerous==1.1.0
MarkupSafe==1.1.1

Go to your jupyter notebook folder and create a file and put the below text in that file and remember DO NOT PUT ANY EXTENSIONS TO THAT FILE LIKE .txt, .css,...

web: gunicorn app: app

Now we are can bring all the files together and save them to the same location, create a git repository, and push that file into that repositories. Now create an account in Heroku and launch our we application.

So we hope that you enjoyed this project. If you did then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Heart Disease Prediction End to End Machine Learning Project (Model Creation)

2020-10-13T17:49:00.009+05:30

Heart Disease Prediction End to End

Machine Learning Project

(Model Creation)

Part1: Creating a Machine Learning Model

In this part we are going to write a code to create a machine-learning model which will help us to predict is that person is suffering from heart disease or not.

Data Set For Heart Disease Prediction Project: DATASET

Requirements: Python >= 3.6

Jupyter Notebook

It's a clean, easy to understand set of data. However, the meaning of some of the column headers is not obvious. Here's what they mean,

1.age: The person's age in years

2.sex: The person's sex (1 = male, 0 = female)

3.cp: The chest pain experienced (Value 1: typical angina, Value 2: atypical angina, Value 3: non-anginal pain, Value 4: asymptomatic)

4.trestbps: The person's resting blood pressure (mm Hg on admission to the hospital)

5.chol: The person's cholesterol measurement in mg/dl

6.fbs: The person's fasting blood sugar (> 120 mg/dl, 1 = true; 0 = false)

7.restecg: Resting electrocardiographic measurement (0 = normal, 1 = having ST-T wave abnormality, 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria)

8.thalach: The person's maximum heart rate achieved

9.exang: Exercise-induced angina (1 = yes; 0 = no)

10.oldpeak: ST depression induced by exercise relative to rest ('ST' relates to positions on the ECG plot. See more here)

11.slope: the slope of the peak exercise ST segment (Value 1: upsloping, Value 2: flat, Value 3: downsloping)

12.ca: The number of major vessels (0-3)

13.thal: A blood disorder called thalassemia (3 = normal; 6 = fixed defect; 7 = reversable defect)

14.target: Heart disease (0 = no, 1 = yes)

Code:

# Importing Essential Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# calling our CSV file (Your Dataset should be of CSV format)
# To create your CSV file go to Excel file and save as CSV : Ctrl + S } Format==>CSV

data = pd.read_csv("D:\corona\heart_Disease\heart.csv")
data.head()

# Gives Complete information of the data values such as Number of Data present, Null or Not-Null, Data type

data.info()

# Counting the target values

data.target.value_counts()

1 165
0 138
Name: target, dtype: int64

# Give a complete statistical description of the Data

data.describe().T

# This will show us people who suffer from heart disease with respect to the sex i.e Male and Female

pd.crosstab(data.sex, data.target).plot(kind="bar",figsize=(10,5),color=['blue','red' ])

plt.xlabel('Sex (0 = Female, 1 = Male)') # X-Label

plt.xticks(rotation=0) # Get or set the current tick locations and labels of the x-axis.

plt.legend(["Haven't Disease", "Have Disease"]) # legend = Index

plt.ylabel('Frequency') # X-Label

plt.show() # Help to show our diagram

# Finding the numbers of outliers which are present in each feature

Q1 = data.quantile(0.25)
Q3 = data.quantile(0.75)

IQR = Q3 - Q1

((data < (Q1 - 1.5 * IQR)) | (data < (Q3 - 1.5 * IQR))).sum()

# Let's Find out the Distribution of Continuous values

for feature in data:
dataset = data.copy()
dataset[feature].hist(bins=25)
plt.xlabel(feature)
plt.ylabel("Count")
plt.title(feature)
plt.show()

# Analyzing and treating outliers with a box plot

for feature in data:
dataset = data.copy()

if 0 in dataset[feature].unique():
pass
else:
dataset[feature] = np.log(dataset[feature])
dataset.boxplot(column=feature)
plt.ylabel(feature)
plt.title(feature)
plt.show()

# Detecting outliers using z_score

# z = (x-μ)/σ

outliers = []
def detect_outliers(values):
Threshold = 3
mean_val = np.mean(values)
std_val = np.std(values)

for i in values:
z_score = (i-mean_val)/std_val
if np.abs(z_score) > Threshold:
outliers.append(i)
return outliers

out = detect_outliers(data['age'])
out

[]

[200,192]

[417, 564, 407, 409]

[71]

# From the above analysis we can say that we lack the domain knowledge, so it would be best if we just keep all the dataset as it is.

# Finding the correlation between variables

data.corr()['chol'].sort_values().plot(kind='bar')

# This will give the correlation between that features

# Example: age in Y-axis and age in X-axis will have the maximum correlation that's why it's dark in color

# annot: Help to see the value inbox

# cmap: The mapping from data values to color space.

# fmt: String formatting code to use when adding annotations.

plt.figure(figsize=(14,10)) # Help to set figure size

sns.heatmap(data.corr(),annot=True,cmap='hsv',fmt='.3f',linewidths=2)

plt.ylim(15,0) # show us the exact number of values we want

plt.show()

# This will show us Heart Disease Frequency for Ages

# Target: YES/NO

# color: Green / Yellow

pd.crosstab(data.age,data.target).plot(kind="bar",figsize=(25,8),color=['green','yellow' ])
plt.title('Heart Disease Frequency for Ages')
plt.xlabel('Age')
plt.ylabel('Frequency')
plt.show()

# The train_test_split function is for splitting a single dataset for two different purposes: training and testing. The testing subset is for building your model. The testing subset is for using the model on unknown data to evaluate the performance of the model.

# train_test_split is a function in Sklearn model selection for splitting data arrays into two subsets: for training data and for testing data. With this function, you don't need to divide the dataset manually. By default, Sklearn train_test_split will make random partitions for the two subsets. However, you can also specify a random state for the operation.

# .values: This returns back the numpy array

# X, y: The first parameter is the dataset you're selecting to use.

# train_size: This parameter sets the size of the training dataset. There are three options: None, which is the default, Int, which requires the exact number of samples, and float, which ranges from 0.1 to 1.0.

# test_size: This parameter specifies the size of the testing dataset. The default state suits the training size. It will be set to 0.25 if the training size is set to default.

# random_state: The default mode performs a random split using np.random. Alternatively, you can add an integer using an exact number.

from sklearn.model_selection import train_test_split

X = data.drop('target', axis=1).values
y = data['target'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)

# Database normalization is the process of structuring a relational database in accordance with a series of so-called normal forms in order to reduce data redundancy and improve data integrity.

# Transform features by scaling each feature to a given range.

# MinMaxScaler: This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one.

# By fit the imputer calculates the means of columns from some data, and by transforming it applies those means to some data (which is just replacing missing values with the means). If both these data are the same (i.e. the data for calculating the means and the data that means are applied to) you can use fit_transform which is basically a fit followed by a transform.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Creating Machine Learning Model.

1)Logistic Regression:

If you want to know more about Logistic Regression, you can definitely check out these blogs which are written in very simple language.

Logistic Regression Blogs:

7 most frequently asked Logistic Regression Questions Answered in 1 blog.

How To Find Optimal Threshold Value And Change Threshold Value In Logistic Regression?

from sklearn.linear_model import LogisticRegression

lr_model = LogisticRegression()

lr_model.fit(X_train,y_train)

lr_pred = lr_model.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test,lr_pred))

print(confusion_matrix(y_test,lr_pred))

[[25 7]
[ 1 28]]

2)Support Vector Machine:

If you want to know more about Support Vector Machine, you can definitely check out our blog which is written in very simple language.

Support Vector Machine Blog:

Support Vector Machine in Machine Learning

from sklearn import svm

clf_model = svm.SVC(kernel='rbf')

clf_model.fit(X_train,y_train)

clf_pred = clf_model.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test,clf_pred))

print(confusion_matrix(y_test,clf_pred))

[[26 6]
[ 1 28]]

3)Random Forest:

If you want to know more about Random Forest, you can definitely check out our blog which is written in very simple language.

Random Forest Blog:

Top 10 Strategies Which Will Make You King Of RANDOM FOREST [2020]

from sklearn.ensemble import RandomForestClassifier

rf_model = RandomForestClassifier(n_estimators=200)

rf_model.fit(X_train,y_train)

rf_pred = rf_model.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test,rf_pred))

print(confusion_matrix(y_test,rf_pred))

[[26 6]
[ 0 29]]

# RANDOM FOREST GAVE US THE BEST RESULT

import joblib
joblib.dump(rf_model,"Heart_Disease_Prediction.pkl")

['Heart_Disease_Prediction.pkl']

# Testing our model performance
m = joblib.load('Heart_Disease_Prediction.pkl')
m.predict([[55,1,3,145,233,0,0,150,0,2.2,0,0,1]])

array([1], dtype=int64)

So From these, we can conclude that our model is performing great and we are ready to deploy it with the help of flask, which we will see at our next blog.

Click this link for part 2:

Heart Disease Prediction End to End Machine Learning Project (Interface Creation)

So we hope that you enjoyed this project. If you did then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

How To Find Optimal Threshold Value And Change Threshold Value In Logistic Regression

2020-08-22T17:22:00.006+05:30

How To Find Optimal Threshold Value And Change Threshold Value In Logistic Regression

Whenever we learn Logistic Regression, we always encounter the Question that How to find the optimal Threshold value for our model? or How we can change the Threshold value as per user’s requirement?

If someone who doesn’t know what is a Threshold value in Logistic Regression then you must check out this link

https://www.infinitycodex.in/logistic-regression-in-machine-learning

And for those who know what threshold value is can keep on reading.

Now why some users/client want to change there threshold value & at which problem we should decide that we have to change the threshold value.

Now the threshold value can be increased or decreased based on the problem or dataset we are dealing with. By the way, the default threshold value in Logistic Regression is 0.5. Now you must be thinking why the hell we want to increase or decrease the threshold value?

Let’s understand this with 2 simple scenarios one for decreasing and others for increasing the threshold value.

1.) Decreasing the Threshold value:

Let’s say you want to predict that the students who score less than 30% failed in there examination. So at this problem statement, you have to reduce the threshold value at 30%.

2.) Increasing the Threshold value:

Let’s say you got a cancer dataset and doctor’s told you that you have to predict the person is having cancer or not. If the person is above 80% infected than that person should be considered as worst infected by cancer and need immediate chemotherapy if that person is having less than 80% the chance that we can consider that person can be treated with normal medication.

I know most of them to know all this thing already by the main question is how do we code this entire process from finding optimal Threshold value till the changing the threshold value.

We will be using the ROC Curve which will help us to predict the optimal threshold value. For those who don't know what the ROC Curve is... ROC Curve is known as Receiver Operating Characteristic.

* ROC Curve is used in Binary Classification.

* It is a plot of True Positive Rate(1) on Y-Axis against False Positive Rate(0) on X-Axis.

* Once you get the output in probability. You can use the different cut-off to distinguish what is going to be the True Case and what will be the False Case.

* ROC Curve say when your curve is closer to the Y-Axis that is True Positive Rate than it is a very good model and your model is in between that is 0.5 than it's an average model and if your curve is towards the False Positive Rate than it's the worst model.

ROC Curve looks something like this:

CODE:

import pandas as pd
import numpy as np

data = pd.read_csv("D:\corona\heart_Disease\heart.csv")
data.head()

data = pd.get_dummies(data, columns=['cp','slope','thal','restecg'], drop_first=True)
data.head()

from sklearn.model_selection import train_test_split

X = data.drop('target', axis=1).values
y = data['target'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Normalizing the values using MinMaxScaler.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Find Threshold Value

Here you can use Decision Tree, Random Forest, Bagging, Boosting.. etc.

from sklearn.metrics import roc_auc_score, roc_curve

# Random Forest
from sklearn.ensemble import RandomForestClassifier

rf_model = RandomForestClassifier()
rf_model.fit(X_train, y_train)

rf_ytrain_pred = rf_model.predict_proba(X_train)
print("RF Train roc-auc:{}".format(roc_auc_score(y_train, rf_ytrain_pred[:,1])))

rf_y_test_pred = rf_model.predict_proba(X_test)
print("RF Test roc-acc:{}".format(roc_auc_score(y_test, rf_y_test_pred[:,1])))

#----------------------------------------------------------------------------------

# Logistic Regression
from sklearn.linear_model import LogisticRegression

lg_model = LogisticRegression()
lg_model.fit(X_train, y_train)

lg_ytrain_pred = lg_model.predict_proba(X_train)
print("LG Train roc-auc:{}".format(roc_auc_score(y_train, lg_ytrain_pred[:,1])))

lg_y_test_pred = lg_model.predict_proba(X_test)
print("LG Test roc-auc:{}".format(roc_auc_score(y_test, lg_y_test_pred[:,1])))

Output:

RF Train roc-auc:1.0
RF Test roc-acc:0.915948275862069
LG Train roc-auc:0.9226736566186108
LG Test roc-auc:0.9450431034482759

Selection Of Best Threshold Value For Accuracy

pred = []

for model in [rf_model, lg_model]:
pred.append(pd.Series(model.predict_proba(X_test)[:,1]))

final_pred = pd.concat(pred, axis=1).mean(axis=1)
print("Ensemble test roc-auc:{}".format(roc_auc_score(y_test,final_pred)))

Output:

Ensemble test roc-auc:0.9407327586206897

# Calculate the roc-curve

False_pos_rate, True_pos_rate, threshold = roc_curve(y_test, final_pred)

threshold

Output:

array([1.97525308, 0.97525308, 0.82803343, 0.82740393, 0.61206134, 0.5829135 , 0.48185878, 0.45554149, 0.45229139, 0.38156861, 0.21880427, 0.07000419, 0.06909258, 0.0069847 ])

These are some of the best candidates from which we can select our threshold value.

from sklearn.metrics import accuracy_score

acc = []

for thres in threshold:
    y_pred = np.where(final_pred>thres,1,0)

    #what ever prediction i am getting and if it is greater than threshold i'll be converting as 1 or i'll keep it as 0.

    acc.append(accuracy_score(y_test,y_pred,normalize=True))

    #Then i'll be computing my accuracy score with my y_test and then
    #append the accuracy inside acc list.

acc = pd.concat([pd.Series(threshold), pd.Series(acc)], axis=1)
acc.columns = ['threshold','accuracy']
acc.sort_values(by="accuracy", ascending=False, inplace = True)
acc.head()

These are top 5 threshold values.

import matplotlib.pyplot as plt
def plot_roc_curve(False_pos_rate,True_pos_rate):
     plt.plot(False_pos_rate, True_pos_rate, label="ROC")
     plt.plot([0,1],[0,1],color="Red",linestyle="--")
     plt.xlabel("False_Positive_rate")
     plt.ylabel("True_Positive_rate")
     plt.title("ROC Curve")
     plt.legend
     plt.show()

plot_roc_curve(False_pos_rate,True_pos_rate)

These are some of the best candidates from which we can select our threshold value.

Create A Model with 0.61 Threshold

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score, roc_auc_score, precision_score, classification_report

clf = LogisticRegression(class_weight="balanced")
clf.fit(X_train, y_train)

THRESHOLD = 0.61
preds = np.where(clf.predict_proba(X_test)[:,1] > THRESHOLD, 1, 0)

pd.DataFrame(data=[accuracy_score(y_test, preds), recall_score(y_test, preds), precision_score(y_test, preds), roc_auc_score(y_test, preds)], index=["accuracy", "recall", "precision", "roc_auc_score"])

A model with Changed threshold value

print(classification_report(y_test,preds))

print(confusion_matrix(y_test,preds))

[[28 1]
[ 9 23]]

So we hope that you find the solution. If you found this helpful then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Easy Number Plate Scanner Project Using OpenCV

2020-07-25T11:12:00.001+05:30

Easy Number Plate Scanner Project Using OpenCV

Before diving into the project let us discuss the term Artificial Intelligence, then we will see what is computer vision, and finally we will start with our project.

CAR

What is Artificial Intelligence?

The theory and the development of a computer system able to perform tasks normally requiring human intelligence, such as visual perceptron, speech recognition, etc.

Now let’s divide this term into 2 parts i.e Artificial and Intelligence.

So what do we mean by Artificial, Anything that is made by humans things which are not natural and what do we mean by Intelligence, It is the ability to understand think and learn.

When we combine both these terms Artificial + Intelligence we get a field where it seems like machines have human intelligence. The goal of AI is to mimic the human brain and create systems that can function intelligently and independently.

ARTIFICIAL INTELLIGENCE

What is Computer Vision?

Human vision is far advance and complex than any camera on this planet. As the AI field is developing rapidly computer vision plays an important role in it.

Computer Vision is the way of teaching intelligence to the machines and making them see things just like humans. So what happens when a human sees an image, that human will be able to recognize that image through patterns, colors, etc. So simply we can say that computer vision is what allows the computer to see and process visual data like humans. Computer vision involves analyzing images to produce useful information. For example, your face recognition system can easily detect your face and unlock your phone quite easily.

Computer vision is a form of Artificial Intelligence that let’s computer identify things using various algorithms trained to collect predefined features helping them pick objects out of the crowd potentially millions of objects with faster and faster recognition.

COMPUTER VISION

What is OpenCV?

OpenCV stands for Open Source Computer Vision it is a library of programming functions mainly aimed at real-time computer vision, it is originally developed by Intel and it was later supported by Willow Garage and now it’s supported and maintained by itseez.

OPENCV

Now OpenCV is a cross-platform library which means that we can use it on Mac, Windows, and Linux.

OPERATING SYSTEMS

You can work on OpenCV with languages such as C, C++, Python, etc. Your project will be done using python. OpenCV is free and Open source, easy to use, and install.

To install OpenCV just go to the command prompt and type:

pip install opencv-python

To know more about the image processing by the computer you can read this article, which will help you a lot.

https://setosa.io/ev/image-kernels/

Now let’s began with the project, so here is the final output of our project.

FINAL RESULT

This project will not only identify the number plate of the vehicle, but it will take a snap-shot of it and save it in a folder by clicking "S".

Don’t forget to download this and save it in the same folder or the folder you want.

harrcascade_russian_plate_number

This project will be created in five easy steps:

Step one, you will be importing some essential libraries.

Step two, you will be creating haarcascade variable.

Step three, you will be initializing the variables.

Step four, the most important step in this you will be creating a frame and creating an area where the number plate would be cropped in real-time.

Step five, you will be creating a snap-shot feature where if you click "S" the snap-shot will be taken of that cropped number plate and saved it in a folder.

# Import Import Libraries

import cv2                    #For computer vision
import numpy as np     # provides a high-performance multidimensional array

# ----------------------------------------------------------------------------------------------------------

# Creating Haarcascade Variable

nplateCascade =
cv2.CascadeClassifier("E:\OpenCV_PROJECT\Resources\haarcascade_russian_plate_number.xml")

# ---------------------------------------------------------------------------------------------------------

# Initializing Variable

minarea = 500
count = 0
framewidth = 6400
frameheight = 480

# ---------------------------------------------------------------------------------------------------------

# Number Plate Capturing

web_cap = cv2.VideoCapture(0)
web_cap.set(3,framewidth)
web_cap.set(4,frameheight)
web_cap.set(10,1000) # Brightness id = 10 and 100 intensity level

while True :
    success, img = web_cap.read()

    imgGray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    numberplates = nplateCascade.detectMultiScale(imgGray, 1.1, 4) # 4 : minimum neighbour

    # Create bounding box

    for (x, y, w, h) in numberplates:
        area = w*h
        if area > minarea:
            cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0),2) # (x,y) : Initial points & (x+w,y+h) : Diagonal points
            cv2.putText(img,"Number plate",(x,y-5),cv2.FONT_HERSHEY_COMPLEX,1,(255,0,0),2)
            imgRoi = img[y:y+h,x:x+w] # Region of number plate
            cv2.imshow("ROI IMAGE", imgRoi)

# ---------------------------------------------------------------------------------------------------------

# Saving Snap-Shot

    cv2.imshow("Video",img)
    if cv2.waitKey(1) & 0xFF == ord("s"):
        cv2.imwrite("Resources/scan/NoPlate "+str(count)+".jpg",imgRoi)
        cv2.rectangle(img,(0,200),(640,300),(0,255,0),cv2.FILLED)
        cv2.putText(img,"scan saved",(150,265),cv2.FONT_HERSHEY_COMPLEX,2,(0,255,0),2)
        cv2.imshow("Result",img)
        cv2.waitKey(5000)
        count += 1

So we hope that you enjoyed this project. If you did then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Create ChatBot Easily Using Python

2020-07-18T14:41:00.001+05:30

Create ChatBot Project Using Python

A chatbot is a computer program simulates a conversation between a user and computer through the auditory or textual method, it works as a real conversational partner and eases your work.

CHATBOT

Now let’s take the real-world example of this, I would be sure that you all know about Siri, Alexa, OkGoogle, Cortana, or even Natasha from the hike. They all are complicated chatbots which are made with the power of Deep Learning (Neural Network). Which actually makes them smart, quick to respond, and accurate.

You would be surprised that the chatbot is not new to us the first of it’s kind was developed in 1966 called ELIZA.

ELIZA

This chatbot was developed by Sir Joseph Weizenbaum.

Sir Joseph Weizenbaum

If you want to want to know more about the history of this, you can check out this link History Of Chatbots.

Now talking about the evolution of chatbots it took a very drastic turn during this decades and the future of the chatbots looks very promising as well, chatbots are expected to complete around 80% of all the works in the coming decade.

The task that is completed by the chatbots includes simply giving out the information or booking tickets etc.

Whenever we are making chatbot we have to consider a few things such as we should know our target audience and we must understand the natural language of communication as well.

What we are going to create?

FINAL CHATBOT APPLICATION

The chatbot which we are going to create can be used by any organization or community, but for example, we are going to create a chatbot specifically for the Computer Science Departement of college where parents will be able to know all the details about the CS department with the help of a chatbot.

We are going to create a chatbot application using:

- Python: Programming Language (Version: 3.6).

- Chatterbot:

* Library in python which generates the responses to the user input.

* It uses a number of machine learning algorithms to produce a variety of responses.

* It is easy to use and implement.

* With the help of chatterbot library we can train our chatbot in multiple languages it supports languages such a Hindi, English, Spanish, etc.

* It easily get trained.

If you want to know more about ChatterBot

Click Here: ChatterBot

- Tkinter: Python GUI ( Graphical User Interface ) Library.

- Notepad: To save our questions and user queries.

- Pycharm: IDE ( Integrated Development Environment ).

Let’s start to create a chatbot from scratch steps:

[ THE WHOLE PROJECT WILL BE CREATED IN WINDOWS & ALL THE STEPS WILL BE RESPECT TO WINDOWS USERS ]

* Install PyCharm in your system:

PyCharm

* Go to start Enter “cmd” a Type “ pip install chatterbot ” after installation à Type “ pip install chatterbot_corpus ”

[ WHILE INSTALLING “chatterbot & chatterbot_corpus” PLEASE SELECT THE SAME PYTHON VERSION WHICH YOU ARE GOING TO USE ]

*Now open your PyCharm :

-Create a new python file

-Now click on file a setting a Project: your_proj_name a Project Interpreter a click on “ + ” at the right side of the window a Now search and install the libraries:

*chatterbot

*chatterbot-corpus

*PyAudio

*NumPy

*pypiwin32

*pyttsx3

*pywin32

*PyYAML

All the required file will be available here :

chatbotdata

After all the steps you had done if you still get errors like these here are it’s solutions:

Error_1:

from chatterbot_corpus.corpus import DATA_DIRECTORY

modulenotfounderror: no module named 'chatterbot_corpus'

Solution:

(i) Go to: https://github.com/gunthercox/chatterbot-corpus

(ii) Click on download or clone a file

(iii) Download Zip

(iv) Save it at a certain location > unzip it

(v) Copy chatterbot_corpus folder from the unzipped file

(vi) Go to My_project(your_project_location) > venv > Lib > site-packages ; Now paste that chatterbot_corpus

problem solved.

Error_2:

import pythoncom

ModuleNotFoundError : No module name 'pythoncom'

Solution:

(i) Open Pycharm

(ii) Go to file > settings > Project:(Project_name) > Project
Interpreter > Click on the (+) which is near scrollbar.

(iii)search (pypiwin32) > install it

(iv) close all tab & import this into your project > import win32com.client

problem solved.

Error_3:

Error in set_trainer in chatterbot

I think the set_trainer was removed. You have to use the training method

from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

chatbot = ChatBot('Ron Obvious')

# Create a new trainer for the chatbot
trainer = ChatterBotCorpusTrainer(chatbot)

# Train the chatbot based on the English corpus
trainer.train("chatterbot.corpus.english")

# Get a response to an input statement

chatbot.get_response("Hello, how are you today?")

problem solved.

Now let’s start to create our chatbot:

Create a file where you would save all the files and go to your PyCharm and paste this code:

from chatterbot import ChatBot
from chatterbot.trainers import ListTrainer
from chatterbot.logic import logic_adapter
from tkinter import *
from tkinter import font
from tkinter import messagebox
import speech_recognition as s
import threading
import pyaudio
import pyttsx3 as pp
import win32com.client
import os
main=Tk()
frame = Frame(main)
ment=StringVar()

main.geometry("500x700")
main.title("CS Department")
img=PhotoImage(file='unnamed.png')
photo=Label(main,image=img)
photo.place(x=1,y=440)

eng=pp.init()

voices = eng.getProperty('voices')
print(voices)

i=IntVar()

def speak(word):
if i.get()==1:
eng.setProperty('voice', voices[0].id)
eng.say(word)
eng.runAndWait()

elif i.get()==2:
eng.setProperty('voice', voices[1].id)
eng.say(word)
eng.runAndWait()

else:
eng.runAndWait()

lbl=Label(main,text="Select Gender of voice :",font=40).place(x=100,y=50)

r1=Radiobutton(main,text="Male Voice",value=1,variable=i,command=speak,font=35).place(x=100,y=100)

r2=Radiobutton(main,text="Female Voice",value=2,variable=i,command=speak,font=35).place(x=100,y=150)

r3=Radiobutton(main,text="Sound ON/OF",value=3,variable=i,command=speak,font=35).place(x=100,y=200)

#import conversation
chatbot=ChatBot('Bot')

for _file in os.listdir('files'):
chats=open('files/' + _file,'r').readlines()
trainer = ListTrainer(chatbot)
trainer.train(chats)

sy=Scrollbar(main)

fnt = font.Font(size=10)

lbl2=Label(main,text="List of Questions you may ask :",font=35).place(x=1150,y=15)

txt=Listbox(main,width=55,height=44,font=fnt,yscrollcommand=sy.set, bd=5)

txt.insert(1," ADMISSION INFORMATION: "+"\n")

txt.insert(2,"\n")
txt.insert(3," what is the registration process"+"\n")
txt.insert(4," what document will be required"+"\n")
txt.insert(5," what are the timing for admission"+"\n")
txt.insert(6," Form where do we have to collect forms"+"\n")
txt.insert(7," How to fill form"+"\n")
txt.insert(8," can we cancel admission"+"\n")
txt.insert(9," Refund"+"\n")
txt.insert(10,"\n")
txt.insert(11,"\n")

txt.insert(12," FEES INFORMATION: "+"\n")

txt.insert(13,"\n")
txt.insert(14," What is the Fees for computer science"+"\n")
txt.insert(15," what document will be required"+"\n")
txt.insert(16," Fees for open categories"+"\n")
txt.insert(17," What is Fees for NT cast"+"\n")
txt.insert(18," Whats is Fees for Other Backward Class"+"\n")
txt.insert(19," What is the Fees for extracurricular activities"+"\n")
txt.insert(20,"\n")
txt.insert(21,"\n")

txt.insert(22," EXTRACURRICULAR INFORMATION: "+"\n")

txt.insert(23,"\n")
txt.insert(24," extracurricular activities are there in you college"+"\n")
txt.insert(25," what document will be required"+"\n")
txt.insert(26," what link of extracurricular activities will be provided"+"\n")
txt.insert(27," how will extracurricular activities will help to my child"+"\n")
txt.insert(28,"\n")
txt.insert(29,"\n")
txt.insert(30," CONCESSION INFORMATION: "+"\n")

txt.insert(31,"\n")
txt.insert(32," what kind of concession will be provided"+"\n")
txt.insert(32," Any specific concession for girls"+"\n")
txt.insert(33,"\n")
txt.insert(34,"\n")

txt.insert(35," PAYMENT INFORMATION: "+"\n")

txt.insert(36,"\n")
txt.insert(37," What are the Payment procedure"+"\n")
txt.insert(38," Can payment be done in installment"+"\n")
txt.insert(39," Do you support UPI"+"\n")
txt.insert(40," Any cash back is available for specific bank"+"\n")
txt.insert(41,"\n")
txt.insert(42,"\n")

txt.insert(43," STAFF INFORMATION: "+"\n")

txt.insert(44,"\n")
txt.insert(45," how is your teaching staff"+"\n")
txt.insert(46," how teachers will inform us about our childs progress"+"\n")
txt.insert(47," how technically advance are you laboratory"+"\n")
txt.insert(48," what placement opportunity will be provided to over child"+"\n")
txt.insert(49,"\n")
txt.insert(50,"\n")

txt.insert(51," CONTACT INFORMATION: "+"\n")
txt.insert(52,"\n")
txt.insert(53," I need a help"+"\n")
txt.insert(54," what is your contact number"+"\n")
txt.insert(55," Help"+"\n")
txt.insert(56," How can I contact you"+"\n")
txt.insert(57," how can I get further information"+"\n")
txt.insert(58," I do not get you"+"\n")
txt.insert(59," calling which number might help me"+"\n")

txt.config()
sy.pack(side=RIGHT,fill=Y)
txt.place(x=1125,y=45)

#I do not get you

lbl3=Label(main,text="Help us to Advance, Enter your unanswered Query :",font=35).place(x=525,y=585)
# txt1=Entry(main,bd=5,width=20)
# txt1.place(x=515,y=580)

txt1 = Entry(main,font=10, bd=5)
txt1.place(width=500,height=50,x=515,y=650)

def Unsolved():
q = txt1.get()

if len(q) < 5 :
messagebox.showinfo("Error","Please Enter Your Unsolved Query")
else:
f=open('Unsolved_Query.txt','a')
f.write(q+"\n")
f.close()
messagebox.showinfo("Congratulations", "Your Query is Accepted Successfully")
txt1.delete(0, END)

btn1=Button(main, text="Enter Query", font=("Times", 16),command=Unsolved).place(x=710,y=710)

sc=Scrollbar(frame)
scx=Scrollbar(frame,orient=HORIZONTAL)

msg=Listbox(frame,width=100,height=25,yscrollcommand=sc.set,xscrollcommand=scx.set, bd=5)
scx.pack(side=BOTTOM,fill=X)

scx.config(command=msg.xview)
msg.config()

msg.pack(side=LEFT,fill=BOTH,pady=10)
sc.pack(side=RIGHT,fill=Y)
frame.pack()

def take_query():
sr=s.Recognizer()
sr.pause_threshold=1
print("Your Bot is listening try to speak")
with s.Microphone() as m:
try:
audio = sr.listen(m)
question = sr.recognize_google(audio, language='eng-in')
print(question)
text.delete(0, END)
text.insert(0, question)
Ask_from_Bot()
except Exception as e:
print(e)
print("Not recognized")
def Ask_from_Bot():
question = text.get()

if question == 'Bye' or question == 'bye':
print("ChatBot:Bye")
main.destroy()
else:
ans = chatbot.get_response(question)
msg.insert(END, "You : " + question + '\n')
print(type(ans))
msg.insert(END, "Bot : " + str(ans))
speak(ans)
text.delete(0, END)
msg.xview()
msg.yview(END)

# Creating a text field
text = Entry(main, bd=5,font=("Times", 15))
text.place(x=650,y=450,width=220,height=35)

btn = Button(main, text="Ask From Bot", font=("Times", 16), command=Ask_from_Bot)
btn.place(x=695,y=510)

# Press Enter & get Output
def Enter_fun(event):
btn.invoke()

# going to bind the main window with entering key

main.bind('', Enter_fun)
def repeatL():
while True:
take_query()
t=threading.Thread(target=repeatL)
t.start()
main.mainloop()

Before executing this code first go to the location where all the files are saved, create a new Notepad file name it Unsolved_Query.txt, now at the same location creates a folder named files in that folder create new txt files names:

- admission.txt
- consession.txt
- contact.txt
- extracurricularactivities.txt
- fees.txt
- greeting.txt
- payment.txt
- teachingstaff.txt

[ ALL THE FILES WHICH ARE MENTIONED, ARE ALREADY IN THE LINK ABOVE IN THE "chatbotdata" LINK. ]

Now let's discuss all the steps :

[ CODE EXPLANATION ]

In the initial step, we imported all the important libraries let's discuss all the libraries one by one.

→ from chatterbot import ChatBot

ChatterBot is a python library that makes it easy to generate automated responses to a user's input. Before we do anything else, ChatterBot needs to be imported. The import for ChatterBot should look like as mentioned above.

→ from chatterbot.trainers import ListTrainer

If you want to create a list of conversation in your code directly like this;

[
    'How are you?',
    'I am good.',
    'That is good to hear.',
    'Thank you',
    'You are welcome.',

you can use this but in our project, we had created a Folder named as a file in which we had we had created multiple .txt files that hold your conversation.

→ from chatterbot.logic import logic_adapter

Logic adapters determine the logic for how ChatterBot selects a response to a given input statement. But in our project, we didn't implement this because it's a simply enquire chatbot not a chatbot for reasoning. So this import is optional in our project.

→ from tkinter import *

Tkinter to create a graphical user interface for the end-user.

→ from tkinter import font

This import is for font various fonts.

→ from tkinter import messagebox

Tkinter to create a graphical user interface for the end-user. This import is used to create a messagebox where our conversation will be placed.

→ import speech_recognition as s

This import is used for speech recognition before using this import you have to install it.

pip install SpeechRecognition

→ import threading

Python threading allows you to have different parts of the program run concurrently and can simplify your design.

→ import pyaudio

With PyAudio, you can easily use Python to play and record audio on a variety of platforms such as Linux, Mac, and Windows. Before using this import you have to install it.

pip install PyAudio

→ import pyttsx3 as pp

pyttsx3 is a text-to-speech conversation library in python. Unlike alternative libraries, it works offline and is compatible with both Python 2 and 3. Before using this import you have to install it.

pip install pyttsx3

→ import win32com.client

It is an essential library which is used to avoid a certain error in our program.

→ import os

The os module is used to read and write file directly.



After importing all the libraries  we will create a object of chatbot  and we named that object as chatbot and named your bot as Bot. Now we will create a list trainer object which we named as trainer but as we are going to have our conversation in .txt format we would read our folder named as files which contains all our .txt files, which will help you chatbot to learn.

We would also a function named as speak in which will help us to get the audio output both in male voice or female voice, for that we had created a radio button by clicking which the output will be get as male voice or female voice.

For end users we would also create a Listbox where we will show them what kind of conversations that they can do with our bot.

If any user didn't get appropriate out put or have any query they can post them in the textbox, where all the queries which are enteried by the end user will be saved in the Unsolved_Query.txt than when the developer want to update the chatbot they can add the unsolved or unanswered queries which are constantly asked by the end user.

With the help of the microphone user can even use speech in english to have a nice conversation with our chatbot this speech feature is availabe in the take_query() function.

And if end user type bye at the end of the conversation the chatbot will automatically get closed.
So we hope that you enjoyed this project. If you did then please share it with your friends and shpread this knowledge.
Follow us at :

Instagram : 

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Google Stock Price Prediction Using RNN - LSTM Python

2020-07-15T16:59:00.001+05:30

Google Stock Price Prediction Using RNN - LSTM Python

In this article, we are going to look at an outstanding end-to-end real-life Recurrent Neural Network (RNN) - LSTM project where we will predict the price of google stock. We will be using Python, Keras, Jupyter Notebook, and Tensorflow for this project.

Before directly diving into the code let’s have an overview of the topics such as RNN and LSTM network.

STOCKS IMAGE

Q.)What is RNN?

RNN also is known as Recurrent Neural Network has unique properties that allow them to be more effective for sequence data, sequence data can be a variety of data sources it can be anything from timestamps sales data or sequence of text in a sentence or biological data like heartbeat data overtime etc.

Basically, RNN helps us to predict future sequential data more accurately which learns from history.

To do this properly, we need to somehow let the neuron “Know ” about its previous history of outputs. One of the easy ways to do this is to simply feed it’s output back into itself as an input.

Let’s look at the normal neuron in feed-forward network

NEURON

Now how we will take advantage of being able to relay back the history with a neuron?

NEURON INVERSE

Here we can use a Recurrent Neuron. In Recurrent Neuron the main difference here not only sends the output out to the next layer, it takes that output and feeds it back to itself. So over time, we can unroll this.

Now go one timestamp into the past at Time(t-1) so as you can see we have some input coming in at t-1 this can be batch of sequence data which are aggregated and passed to the activation function and we get the output of this Recurrent a neuron at t-1.

FUTURE NEURON

As time goes on we end up doing for that next input batch of another sequence or input at time t, here we are not only going to feed in the input at time t but we also feed in the information of the output of Recurrent neuron times (t-1) which then give us the output at time t. Then we take that output at time t and feed it along with the input at a time (t+1) that way we are retaining the historical information. This is the Recurrent neuron essentially unrolled throughout time.

RECURRENT NEURON

Cells that are a function of inputs from previous time steps are known as memory cells.

RNN is also flexible in their inputs and outputs, for both sequence and single vector values.

We can actually, create entire layers of Recurrent Neurons.

Here is an simple diagrammatic representation of it.

(Artificial Neural Network) ANN :

NORMAL ANN

(Recurrent Neural Network) RNN :

NORMAL RNN

Now we can just unroll this layer in the same fashion. We pass input as time t = 0 into the layer and then the output of the layer is time t+1, than t+2, and so on.

RNN UNROLLED LAYER

As I have mentioned earlier that the RNN are very flexible in their inputs and outputs. There are different types of architectures we can use here. Let’s see some of them:

1.) Sequence to Sequence (Many to Many):

SEQUENCE TO SEQUENCE

In this, you pass in a sequence and you expect a sequence out.

For Example : We have an input of 5 words now you have to predict the output of 5 words.

2.) Sequence to Vector (Many to One):

SEQUENCE TO VECTOR

In this, you pass in a sequence and you expect a vector as an out.

For Example : We have an input of 5 words now you have to predict the output of the next 1 words.

3.) Vector to Sequence (One to Many):

VECTOR TO SEQUENCE

In this you pass in a vector i.e a single value and you expect a sequence as an out.

For Example : We have the input of 1 word now you have to predict the output of the next 5 words.

A basic RNN has a major disadvantage, we only really “remember” the previous output. If we think back that unrolled diagram we were only feeding in the of 1 timestamp into the past and what happens is if we have really long histories we begin to start to forget the older historical samples since we are only really looking at the out of the last previous t-1 and it will be really great if we could keep the track of long history and not just that short term memory. Another issue that arises during training is the “vanishing gradient”.

Before directly diving into the LSTM (Long Short Term Memory Units). We will have an overview of Vanishing Gradient.

Let’s discuss the 2 main issues i.e Exploding & Vanishing Gradients.

As our networks grow deeper and more complex, we have 2 issues arises:

-Exploding Gradients

-Vanishing Gradients

Recall that the gradient is used in our calculation to adjust weights and biases in our network, if you don’t have any idea about Gradient you may check our GradientDescent article.

This errors might arise during backpropagation :

NEURAL NETWORK

Now for complex data we such as complex image data or complex sequence data we end up needing deeper networks i.e we need more hidden layers in order to actually learn the patterns that are in our data.

Now what happens is there’s this vanishing or exploding gradient issue that arises during the backpropagation step. So recall we are going to calculate some sort of loss metrics on the output layer and then backpropagation error all the way back to the input layer and if we have a lot’s of hidden layers then we’re having the update to the weights and biases be a function of many other derivatives that we’re calculating along the way back.

- So backpropagation goes backward from the output to the input layer, propagating the error gradient.

- For deeper networks issues can arrive from backpropagation, vanishing and exploding gradients.

BACK PROPAGATION

- As you go back to the “lower” layers closer to the input layer, gradients often get smaller, to a vanishing gradient is usually the more common problem although they can technically explode on the way back. But as you are going back and back closer to those input layers there are gradients are getting smaller and eventually what happens is they’re so small by the time it gets to the input layer that the weights never really change that much at those lower input levels.

- That’s actually, a big problem because we want to be able to detect larger basic patterns in our data right out close to the input layer and have the deeper layers focus on the smaller details or in smaller details and patterns.

- And yes the opposite can also occur that the gradients explode on the way back, causing issues.

Now let’s discuss why this is actually occurring and how we can fix it and let’s also discuss how these issues specifically affect the RNN and how we can use the LSTM units and get a recurrent unit to also fix this to understand what actually happening let’s take a look at a really common activation function such as sigmoid. Now as we know that the sigmoid activation function squeezes the input to fit between 0 & 1.

SIGMOID

However let’s take a closer look at what happens when your inputs start to get further away from zero.

SIGMOID

The further away your input is from zero. The rate of change in the sigmoid function is actually decreasing rapidly and that rate of change that is the derivatives of the sigmoid functions and we already know that the backpropagation and the gradient calculation is essentially just calculating that derivative in multiple dimensions as you go back through into the hidden layers.

* When N hidden layers use an activation like the sigmoid function, N small derivatives are multiplied together.

* The gradient could decrease exponentially as we propagate down to the initial layers.

* We can use other activation function such as ReLU which doesn't actually saturate those larger positive values.

* The main benefit of using ReLU here is that doesn't matter how large your input value is going to be beyond 0. You are not going to exponentially decrease the rate of change.

The other a possible solution is the Batch Normalization where your model will normalize each batch using that particular batches mean and standard deviation, and that has also been founded to alleviate the issue of vanishing gradient descent.

So apart from things such as Batch Normalization, researchers have also used “Gradient Clipping”, where gradients are cut off before reaching a predetermined limit (eg: Cutt off gradients to be between -1 and 1).

RNN for Time Series presents their own Gradient Challenges, let’s explore special LSTM (Long Short Term Memory) neuron units that help fix these issues!

*LSTM ( Long Short Term Memory ):

Many of the solutions previously presented for the vanishing gradients can also apply to

RNN: different activation functions (ReLU), batch normalizations, etc…

However because of the length of time series input, these could slow down the training.

A possible solution would be to just shorten the time steps used to prediction, but this makes the model worse at predicting longer trends. So maybe looking back 20 time steps for the 21^st prediction you just look back 5-time steps to get the next prediction. However, this makes the model worse at predicting longer trends. So we still want to able to use a long time sequence in order to predict the next item in the sequence.

Another issue RNN faces are that after awhile the network will begin to “forget” the first inputs, as information is lost at each step going through the RNN.

We need some sort of “long-term memory” for our networks. So this is where LSTM (Long Short Term Memory) cell was created to help address these RNN issues.

Let’s see the working of an LSTM cell:

RNN CELL

Take a look at what a single Recurrent neuron would actually be doing is essentially it’s taking in both the previous output and then the current input and then producing the next output.

RNN CELL LABELED

instead of saying it output or input, we will refer to these as hidden state and then our current feature X going in. So basically we have Ht-1 going along with X of T and that produces Ht. So in a standard RNN essentially what we do is we just have a single hyperbolic tangent function and then what we are doing, we are combining Ht-1 with Ht multiply that with some weight matrix then adding a bias to it and then passing it through the hyperbolic tangent function and that gives us back our Ht.

RNN CELL LABELED FORMULA

And we just repeat that through the next recurrent neuron or an extra current layer.

So LSTM also have this chain-like structure. But a repeating module have slite difference to it and instead of just having a single NN layer. There would be actually going to be 4 layers working and interacting in a special way and the way we end up getting 4 is the fact that not only will we keep track of just a single historical memory with Ht-1 we’re keeping track of both long term memory input and the short term memory input and that creating a new long term memory output and then use short term memory output along with the current input at time t and then we produce the output with time t.

STEPS OF LSTM :

Before diving into the steps let’s see some of the notation we’re going to be using so essentially.

So essentially we are going to have 4 main components inside the LSTM.

LSTM 4 MAIN COMPONENTS

We have: Forget Gate, Output Gate, Update Gate, and an Input Gate.

*Forget Gate: As the name suggests, Forget Gate will decide what to forget from the previous memory units.

*Input Gate: As the name suggests, Input Gate will decide what actually accept into the neuron.

*Update Gate: As the name suggests, Update Gate will update the memories.

*Output Gate: As the the the name suggests, Output Gate will actually output the new long term memory.

SIGMOID CELL

A Gate optionally lets some information through and essentially we can just think of this as mathematically it’s a sigmoid function. It’s either going to end up a sequence between a 0 or 1 and if it’s 0 we don’t let that information go and if it’s a 1 we let it go.

GENERAL STRUCTURE OF LSTM

So here is the general structure of LSTM and we are accepting both Long Term and Short Term memory, we can think of this as going to be passed in through conveyor belts inside of this neuron and what we end up happening are it just ends up kind of running down straight the enitre chain and has some kind of linear interactions with a few functions inside of the cell.

Now for the purpose of a mathematical notation we’re gonna relabel some of these and we’re going to label them as such.

GENERAL STRUCTURE WITH NOTATION

So what are the actual linear interactions and functions going on inside of LSTM? Well, here we can see the entire LSTM cell.

Now let’s go through the process step by step :

1.] The 1^st step of LSTM is to decide what information is going to throw away from the cell state essentially what we are going to forget? So we end up creating is a forget gate layer or ft.

INSIDE OF LSTM

So we end up creating is a forget gate layer of ft. Remember those gates are essentially just passing things through a sigmoid function where the closer it is to 0 that means fewer weights we are giving it. The closer it is to 1 the more weights we are giving it. So in the context of the forget gate layer the closer it is to 0 means forget about it and get rid of it and if it closer to 1 than remember this it's important. So this is what ft is doing. Notice that it's essentially a linear combination of ht-1 that previously hidden state combines with the input Xt. Then we have our own sets of weights for this forget gate layer plus bias and then we pass it through an activation function and then we get ft then the next step after this is to decide what new information are we going to store into the cell state?

2.] In the 2^nd step we have to decide what new information we are going to store into the cell state. This has 2 parts to it :

INSIDE OF LSTM STEP 2

(i) We have a sigmoid layer that we’re going to label the input gate layer which is essentially going to decide what values are we going to update.

(ii) We have a hyperbolic tangent layer that creates a vector of new candidate values which will say Ct but we will label this with a tilde on top of it. These are the new candidate values that will eventually helpful in some sort of weighing be updating the cell state. So we have that input gate layer deciding which value we’re going to update in the hyperbolic tangent layer which is creating a vector of those new candidate values.

3.] Now it’s time to update the old cell state which is Ct-1. In order to calculate the new cell state Ct that we’re going to end up outputting.

INSIDE OF LSTM STEP 3

So what we are doing is that we multiply the old state by ft forgetting the things that we decided weren’t that important due to the forget gate layer. Then we add it times the new candidate values. So essentially these are the new candidate values for the cell state scaled by how much we decided to update each state value.

4.] Finally we have to decide what we are going to output? So this output will be based on our cell state. That actually a filtered version. First, we will end up doing is we run a sigmoid so that top equation which decides what parts of the cell state we are going to output. Then we put that cell state through the hyperbolic tangent function and what the hyperbolic tangent does is it pushes all the values to be between -1 and 1 and then we’re gonna multiply it by the output of that initial sigmoid gate of Ot so that we only output the parts of that we decided to.

INSIDE OF LSTM STEP 4

Before directly diving into the coding first, get the data :

Google_Data

This data set contains:-

1.) Data: Stock information date.

2.) Open: Opening on a particular date.

3.) High: Highest price on that data.

4.) Low: Lowest price on that date.

5.) Close: Stock pricing closed that date.

6.) Adj Close: Adjusted close price.

7.) Volume: Volume of share.

We will be reading this data in chunks of 30 days. So we will be training our neural network on 30 days data we will be predicting the 31st day and then similarity we will again skipping the last data and then again we will be taking from 1 to 31 days and then we will be predicting at 32nd day.

Steps To Build Stock Prediction Model:

i.) Importing & Preprocessing Data

ii.) Building The RNN model

iii.) Predicting Values

iv.) Checking The Accuracy

i.) Importing & Preprocessing Data:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = pd.read_csv(r"E:\New folder\GOOG.csv", date_parser=True)
data.head()

Output:

DATA

# Divide Data into Training and Test set

# Training Data

data_training = data[data['Date']<'2020-05-01'].copy()
data_training

Output:

TRAINING DATA

# Divide Data into Training and Test set

# Test Data

data_test = data[data['Date']>='2020-05-01'].copy()
data_test

Output:

TEST DATA

training_data = data_training.drop(['Date','Adj Close'],axis=1)
training_data.head()

Output:

TRAINING DATA HEAD

Here we have to predict the Open i.e opening price of that particular stock.

# Data Preprocessing

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

training_data = scaler.fit_transform(training_data)
training_data

Output:

MIN-MAX SCALER DATA

X_train = []
y_train = []

for i in range(30, training_data.shape[0]):
X_train.append(training_data[i-30:i])
y_train.append(training_data[i,0])

X_train, y_train = np.array(X_train), np.array(y_train)

X_train.shape

Output:

(173, 30, 5)

ii.) Build The RNN Model:

# Building LSTM

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, LSTM, Dropout

regression = Sequential()
regression.add(LSTM(units=50, activation="relu", return_sequences=True, input_shape=(X_train.shape[1], 5)))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50, activation="relu", return_sequences=True, input_shape=(X_train.shape[1], 5)))
regression.add(Dropout(0.3))

regression.add(LSTM(units=50, activation="relu", return_sequences=True, input_shape=(X_train.shape[1], 5)))
regression.add(Dropout(0.4))

regression.add(LSTM(units=50, activation="relu"))
regression.add(Dropout(0.5))

regression.add(Dense(units=1))

regression.summary()

Output:

MODEL SUMMARY

# Compile and Fit the model

regression.compile(optimizer='adam', loss="mean_squared_error")
regression.fit(X_train,y_train,epochs=550,batch_size=32)

Output:

COMPILE AND FIT DATA

iii.) Predict Values:

# Prepare test data

data_test.head()

Output:

TEST HEAD

past_30_days = data_training.tail(30)

df = past_30_days.append(data_test, ignore_index=True)
df = df.drop(['Date', 'Adj Close'],axis=1)
df.head()

Output:

FILTERED DATA

inputs = scaler.transform(df)
inputs

Output:

INPUT TRANSFORM DATA

X_test = []
y_test = []

for i in range(30, inputs.shape[0]):
X_test.append(inputs[i-30:i])
y_test.append(inputs[i,0])

X_test, y_test = np.array(X_test),np.array(y_test)
X_test.shape, y_test.shape

Output:

((49, 30, 5), (49,))

y_pred = regression.predict(X_test)
y_pred

Output:

PREDICTED DATA

scaler.scale_

Output:

array([2.13419869e-03, 2.17020477e-03, 1.96903103e-03, 2.12734298e-03,2.24300742e-07])

scale = 1/2.13419869e-03
scale

Output:

468.55993525138933

y_pred = y_pred * scale
y_test = y_test * scale

vi.) Visualize The Data and Check Accuracy:

# Visualize the data

plt.figure(figsize=(14,5))
plt.plot(y_test, color="red", label="Real Google Stock Price")
plt.plot(y_pred, color="blue", label="Predict Google Stock Price")
plt.title("Google Stock Price Prediction")
plt.xlabel('Time')
plt.ylabel('Google Stock Price')
plt.legend()
plt.show()

Output:

DATA VISUALIZATION

Classify Malaria Using CNN Python

2020-06-27T16:07:00.009+05:30

Classify Infected Malaria Cell Using CNN Python

CELL

In this article, we will look at end-to-end real-life Convolutional Neural Network (CNN) project where we will detect the cell is infected by malaria or not. We will be using Python, Keras, and TensorFlow for this project.

The data set which is used in this project is taken from the National Library Of Medicine.

National Library Of Medicine

(https://lhncbc.nlm.nih.gov/publication/pub9932)

This dataset contains multiple image’s of infected cells called as (‘parasitized’) :

INFECTED CELL

And multiple images of non-infected cells called as (‘uninfected’) :

Non-Infected Cells

So what we end up doing is attempting to build a model that just based on the image of a cell can predict whether or not it’s infected or not infected with malaria.

In real life, this may save doctors a lot of time by just running images into our model instead of having to manually look at these image’s themselves and determinant.

So now you can download this data from the official site or just click this link :

Cell Data: ftp://lhcftp.nlm.nih.gov/Open-Access-Datasets/Malaria/cell_images.zip

Now as you all have downloaded and extracted the data. Now it's time to execute some code.

Open your Jupyter-Notebook and let's begin.

The very first thing we will do is to take a look at the content which is present inside the folder.

import os
data = "D:\corona\CNN_data\cell_images"
os.listdir(data)

Output:

['test', 'train']

Now we will be importing some of the important Python libraries which will help us to visualize and manage the images.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

Now to read the images which are present in our file we will import imread function which is present in the matplotlib library.

from matplotlib.image import imread

As we saw earlier that we have a test set and training set inside of this. Let's go and set 2 variables the first variable will be called test_path and what we are going to do here is that we are going to concatenate our data path with test and for the second one will be called as train_path and we will do the same thing with that too.

test_path = data + "\\test\\"
train_path = data + "\\train\\"

Now to read the images which are present in our file we will import imread function which is present in the matplotlib library.

The purpose of doing this is to see what images are present inside that path. To check the path again just type test_path and it will show you the path

test_path

Output:

'D:\\corona\\CNN_data\\cell_images\\test\\'

So now let's go ahead and list the files what kind of files are inside.

os.listdir(test_path)

Output:

['parasitized', 'uninfected']

If you noticed that this data set contains 2 folders which are parasitized and uninfected which we saw above. These folders contains about 27000 images and the state asset is taken from a government repository on malaria data sets.

So as I mentioned earlier we are going to end up doing is attempting to build a model that just based off the image of a cell can predict whether or not that cell is infected or not infected with malaria.

So now we are going to look at the single image.

os.listdir(train_path+'parasitized')[0]

Output:

'C100P61ThinF_IMG_20150918_144104_cell_162.png'

Now let's visualize the single-cell image.

inf_cell = train_path+'parasitized\\'+'C100P61ThinF_IMG_20150918_144104_cell_162.png'

plt.imshow(imread(inf_cell))

Output:

Infected Cell Image

As we saw the infected cell above, now let's have a look at the non-infected cell.

os.listdir(train_path+'uninfected')[0]

Output:

'C100P61ThinF_IMG_20150918_144104_cell_128.png'

uninf_cell = train_path+'uninfected\\'+'C100P61ThinF_IMG_20150918_144104_cell_128.png'

plt.imshow(imread(uninf_cell))

Output:

Non-Infected Cell Image

Now let's go ahead and finally figure out the average shape of, one of these images. The important thing about this dataset is that the images which are present are real images and it's unlikely that they all are going to be the exact same shape. If we compare this dataset with other datasets such as MNIST or CIFAR dataset you have noticed that every image in this dataset has the same dimensions. But in the real dataset, it's another way around they going to have different dimensions.

There are lots of different ways we can do this but one way is to set up 2 list dim1 and dim2 than we will be applying a for loop where we will be iterating through every file and by chance the test path uninfected. Then we are going to check the shape of each of them and then append their first dimension (dim1) and second dimension (dim2). Now let's code this theory.

dim1 = []
dim2 = []

for img_filename in os.listdir(test_path+'uninfected'):
   img = imread(test_path+'uninfected\\'+img_filename)
   d1,d2,color = img.shape
   dim1.append(d1)
   dim2.append(d2)

Now let's visualize the dimensions of dim1 and dim2.

sns.jointplot(dim1,dim2)

Output:

Graphical Representation

So why is this important?

Well, the Convolutional Neural Network isn't going to be able to train on images of various sizes. So what we need to do is to make sure that, we are going to resize all the images to be the same size. So we have to choose what is the actual dimensions should we resize everything to. The answer is pretty simple we should choose is essentially the average of both dimensions and this shows you the actual distribution of the images and they all kind of center around basically 130 by 130 and you can confirm this by checking out the mean values in your dimensions.

np.mean(dim1)

Output:

130.925

np.mean(dim2)

Output:

130.750

So now we can say that our final image shape which we will be feeding in our Convolutional Neural Network is 130 by 130 by 3

img_shape=(130,130,3)

Go ahead and run that. And then later on we are actually preparing the data for the model we will be resize everything to these dimensions. So if it's a smaller photo we'll basically add padding so that it reaches these dimensions and if it's a larger photo we can either crop, shrink, or compress that image.

Now we are going to focus on this special object from TensorFlow Keras which is called the image data generator. What we are going to do of this class is that we are going to feed it in the directory over our actual image files are and you'll be able to perform a bunch of manipulations on our images and then feed those new images to your model.

Before diving into the concept keep in mind that there is really too much data for us to read in all this data at once. This file is much larger than the files such as MNIST and CIFAR. If you know the dimension of MNIST is 28 by 28 and CIFAR has the dimension of 32 by 32, and even that the smaller expansion from 28 by 28 to color images of 32 by 32 was a huge expansion in the amount of data if we look at the math i.e 28*28 = 784 data points and when we see CIFAR that was 32*32*3 = 3072 and our files are going to be even larger that is 130*130*3 = 50700. Now we are dealing with 50700 data points because we are not going to able to just feed in everything at once. Instead, we have to select batches of our images. The other idea that we want to be able to overcome is the fact that it should be robust enough to deal with images that are pretty different from images that it's seen before and one way we can do that is by manipulating and performing transformations on our images thing like rotation, resizing and scaling.

Okay enough with the talk now let's go ahead and explore this idea of manipulating images as well as flowing from a directory these new batches of files.

from tensorflow.keras.preprocessing.image import ImageDataGenerator

Go ahead and run that command. If you had ever worked with MNIST dataset we had 60000 images and that's a lot of images for a very simple file type essentially a very simple image that had a dimension of 28 by 28. Right now we have half of that size over our entire dataset. Our entire dataset is less than 30000 images. So want to do is to expand the number of images without having gathered more data. We can't just keep grabbing blood cells from people. So we can do things like take our current images and randomly rotate them, reshape them, rescale them, etc.

from tensorflow.keras.preprocessing.image import ImageDataGenerator

img_gen = ImageDataGenerator(rotation_range=20,
              width_shift_range=0.1,
              height_shift_range=0.1,
              shear_range=0.1,            #cutting part of image
              zoom_range=0.1,
              horizontal_flip=True,
              fill_mode='nearest')       #fill data in image voids

plt.imshow(imread(uninf_cell))

Output:

Non-Infected Cell

para_img = imread(inf_cell)
plt.imshow((para_img))

Output:

Infected Cell

Now let's have a look at our manipulated image.

plt.imshow(img_gen.random_transform(para_img))

Output:

Para-Infected Cell

So here is our randomized version of the image. Notice that we got to stretch, kind of like columns sticking out of the cell and that's because through this random transformation it looks like it got stretched out and fill in those values with the nearest pixel values and then notes it was also rotated makes a lot of sense to randomly rotate image here because their cell's that can be in any sort of rotational axis that they want. They can be floating around in their sample so depending again on the actual type of images you're looking at you're going to be playing around at these actual range values.

Okay so now with the fact that we can randomly transform these images we can essentially augment our dataset. We no longer restricted to just a single image from the cell we can randomly transform this many times over. So if we keep running this we'll see more and more random transformations. And this is the way artificially expanding our image dataset. Recall we have less than 30000 images but now we could do the random transformation on all those images and immediately double the size of our image dataset. Maybe we can do 5 random transformations and we went from something like 20000 images to 100000 images.

So this is the really powerful tool you have to keep in mind when you are dealing with kinda smaller datasets and when it comes to the Convolution Neural Networks it takes thousands and thousands of images to get something that performs really well.

plt.imshow(img_gen.random_transform(para_img))

Output:

Para-Infected Resized Cell

Okay now, how do we actually set up our directories to flow batches from the directory. The way we do that by using the flow from the directory function.

img_gen.flow_from_directory(train_path)

Output:

Found 24958 images belonging to 2 classes.

<keras_preprocessing.image.directory_iterator.DirectoryIterator at 0x1f9aabb8808>

When you run that, you saw some text. Now, what that text means? It means that our machine had found around 24958 number of images belonging to 2 classes. There are 2 folders in the directory which indicates each class, that's how our machine is able to distinguish between classes. This would be the same for the test_path but at test number of images would be 2600.

IMP: In order to use ".flow_from_directory", you must organize the image is sub-directories. This is an absolute requirement, otherwise, the method won't work. The directories should only contain images of one class, so one folder per class of images.

Okay now as we have learned about our dataset. Now it's time to create our model.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Dropout, Flatten

model = Sequential()

model.add(Conv2D(filters=32,
               kernel_size=(3,3),
               input_shape=img_shape,
               activation='relu'))

model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(filters=64,
               kernel_size=(3,3),
               input_shape=img_shape,
               activation='relu'))

model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(filters=64,
               kernel_size=(3,3),
               input_shape=img_shape,
               activation='relu'))

model.add(MaxPool2D(pool_size=(2,2)))

model.add(Flatten())

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(1,activation='sigmoid'))

model.compile(loss='binary_crossentropy',
               optimizer='adam',
               metrics=['accuracy'])

model.summary()

Output:

Model Summary

IMPORTANT POINTS:

* MaxPool2D and MaxPooling2D are aliases that mean there are multiple names for the same function.

* Because this is going to be a larger network we added Dropout to it. Which will help us to prevent overfitting.

* input_shape should be equal to the img_shape.

* Keep in mind if you choose an image shape that's too large especially if you are dealing with extremely large files your machine will run out of memory. That completely depends on your hardware.

* We added multiple Convolutional layers because the larger the image size the more complex of a task you are dealing with more Convolutional layer you should have.

* As we go deeper the network we increase the number of filters.

* Dropout value is 0.5 because we want to turn off half of the neurons to prevent overfitting.

Notice that we have 1605760 number of parameters by the time we get to the dense layer. So this model will take a long time to train.

To make sure we can pick the right amount of epochs to train for we can use callbacks early stopping.

from tensorflow.keras.callbacks import EarlyStopping

early_stop = EarlyStopping(monitor='val_loss',patience=2)

batch_size = 16

train_img_gen = img_gen.flow_from_directory(train_path,
               target_size=img_shape[:2],
               color_mode='rgb',
               batch_size=batch_size,
               class_mode='binary')

test_img_gen = img_gen.flow_from_directory(test_path,
               target_size=img_shape[:2],
               color_mode='rgb',
               batch_size=batch_size,
               class_mode='binary',shuffle=False)

Output:

Found 24958 images belonging to 2 classes.

Found 2600 images belonging to 2 classes.

train_img_gen.class_indices

Output:

{'parasitized': 0, 'uninfected': 1}

Now we are going to train our model. Remember that it's going to take a lot of time to train.

results = model.fit_generator(train_img_gen,epochs=20,
validation_data=test_img_gen,
callbacks=[early_stop])

Or else we have an alternative for this which is very easy to implement.

Pre-Trained File:

https://github.com/Vegadhardik7/ML_ALL_PROJECTS/blob/master/malaria_detector.h5

Just download this file and let's continue.

from tensorflow.keras.models import load_model

model = load_model('D:\corona\malaria_detector.h5')

model.summary()

Output:

Pre-Trained Model Summary

The minor drawback with this is, you won't get the model history. But if you want you can evaluate the model.

model.evaluate_generator(test_img_gen)

Output:

[1.9182854999828034, 0.8738462]

That's amazing we got an accuracy of 87% which is pretty good considering that base accuracy is 50%.

Now let's see how accurate our model is using some prediction values.

pred = model.predict_generator(test_img_gen)

pred

Output:

array([[0.],
           [0.],
           [0.],
           ...,
           [0.],
           [1.],
           [0.]], dtype=float32)

So if we take a look at the results here of pred we get the values between 0 and 1. So it actually doesn't return back the straight class calls, instead, it returns back the probabilities so you know that if the value of a point is 0.97 or 0.98 that means that our model is 97% or 98% sure that it belongs to class 1. So now what we will do is that if any value is greater than 0.5 i.e 50% it belongs to class 1 and if it's less than 0.5 it will belong to class 2.

predictions = pred > 0.5 # you can take any threshold value you want

predictions

Output:

array([[False],
           [False],
           [False],
           ...,
           [False],
           [ True],
           [False]])

So True and False are treated as 0 and 1 in NumPy. So we can directly pass it through this to our confusion matrix and classification report.

If the value is 1 the cell is not infected and if the value is 0 it is infected. Look the class values {'parasitized': 0, 'uninfected': 1}.

from sklearn.metrics import classification_report,confusion_matrix

print(confusion_matrix(test_img_gen.classes, predictions))

Output:

[[1274 26]
[ 299 1001]]

from tensorflow.keras.preprocessing import image

image.load_img(inf_cell)

Output:

Malaria Infected Cell

my_img = image.load_img(inf_cell,target_size=img_shape)

my_img

Output:

Reshaped Malaria Infected Cell

So when we think about this as a real-world situation, what actually happens is that the doctor will give us the image and we going to load the image and pass it to the model.

my_img_array = image.img_to_array(my_img)
my_img_array.shape

Output:

(130, 130, 3)

So what we are going to do here is simply resize this along the 0 dimensions and we can do that in several ways, but one of the ways is we can use np.expand_size function.

my_img_array = np.expand_dims(my_img_array,axis=0)

my_img_array.shape # We reshaped the image to 1 image of dimension 130,130,3

Output:

(1, 130, 130, 3)

model.predict(my_img_array)

Output:

array([[0.]], dtype=float32)

train_img_gen.class_indices

Output:

{'parasitized': 0, 'uninfected': 1}

This means that our model prediction is correct it is an infected cell.

So we hope that you enjoyed this project. If you did then please share it with your friends and spread this knowledge.

Follow us at :

Instagram :

https://www.instagram.com/infinitycode_x/

Facebook :

https://www.facebook.com/InfinitycodeX/

Twitter :

https://twitter.com/InfinityCodeX1

Top 10 Strategies Which Will Make You King Of RANDOM FOREST [2022]

2020-06-04T10:51:00.006+05:30

Top 10 Strategies Which Will Make You King Of RANDOM FOREST [2022]

FOREST

Random Forest is one of the Supervised Learning Techniques which is used to solve the Classification and Regression problem.

Evolution only comes when we found the limitations of something.

In this era of complete evolution, humans had found the limitations themselves. To overcome those limitations we try to create multiple solutions and go plus ultra.

To understand Random Forest completely, Let’s first understand what is Regression and Classification.

Regression: Regression is a method or an algorithm in Machine Learning that models a target value based on independent predictors. It is essentially a statistical tool used in finding out the relationship between a dependent variable an independent variable.

- Here we predict the output value as a specific number.

- In Regression, a Regression Tree is used when the target variable is numerical or continuous in nature.

- We fit a regression model to target variables using each of the independent variables.

- Each split is made based on the sum of the square error.

REGRESSION

Classification: Classification is a process of categorizing a structured or a nonstructured set of data in different classes based on certain features/categories.

CLASSIFICATION

The limitations of Decision Tree leads us to create a Random Forest. Let us see a quick overview of limitations of Decision Tree.

* The Primary disadvantages of the decision tree are overfitting. Overfitting occurs when the algorithm capture noise in the data.

* In the Decision Tree there is also a problem of high variance. The model can get unstable due to small variance in the data.

* In a decision tree is there is Low Biased tree. A highly complicated decision tree tends to have a low bias which makes it difficult for the model to work with new data.

* Calculation can get very complex particularly if many values are uncertain and if many outcomes are linked.

To overcome all the limitations which we had in the Decision Tree. We will be using Random Forest. After all, this algorithm creates the forest with n number of Decision Trees.

DECISION TREE AND RANDOM FOREST

In general more trees in the forest more robust the prediction and thus higher accuracy.

To understand this the concept clearly, here are some of the links which will help you to become not only King but KingKong of Random Forest.

KING KONG

→ Decision Tree

→ Supervised Learning

→ Unsupervised Learning

So before diving into Random Forest directly, here is the map that will help you become King.

RANDOM FOREST MAP

(1) Ensemble Learning Overview

Ensemble Learning is used in multiple learning algorithms at the same time to obtain predictions with an aim to have better predictions than the individual model.

ENSEMBLE LEARNING

Q.) Why use Ensemble Learning?

* Gives us better accuracy which means the error will be minimum.

* It avoids overfitting so the consistency is very high.

* The bias and variance errors are also reduced to a minimum.

Q.) When and Where to use Ensemble Models?

*When a single model like Decision Tree over fits we can use Random Forest or we can use Ensemble of multiple similar models.

*It can be used for both Classification and Regression problems.

(2) Why Random Forest?

It’s always important to understand, why we use any of the algorithms, for this article we will be talking about why we use Random Forest over the other algorithms.

Random Forest is an algorithm that helps us to get optimal output. It avoids overfitting, if you use multiple trees it automatically avoid the risk of overfitting.

For those who don’t know overfitting :

When a model performs well at training data but performs badly at test data.

In Random Forest the training time of the data is less.

The accuracy of the Random Forest is very high.

Random Forest also runs well especially in the larger database, for large data it produces highly accurate prediction in today's world of Big Data it is really very important. This is why Random Forest really comes in to help us.

It estimates the missing data, Data on today's world is very messy so when you have Random Forest for us which can maintain the accuracy when a large proportion of the data is missing.

(3) What Is Random Forest?

Random Forest is a type of ensemble learning method. It is a versatile algorithm capable of performing both Regression and Classification.

Random Forest or Random Decision Tree Forest is the method that operates by constructing multiple Decision Trees during the training phase.

RANDOM FOREST OF FRUITS

The decision of the majority of the trees is chosen by the Random Forest at the Final decision.

Random Forest is basically used for predictive modeling and machine learning technique.

(4) Decision Tree VS Random Forest

DECISION TREE VS RANDOM FOREST

(5) Application Of Random Forest

There are multiple application of Random Fores but some of them are :

(i) Banking

(ii) Remote Sensing

(iii) Object Detection

(iv) Medicine

APPLICATIONS OF RANDOM FOREST

(6) Features Of Random Forest

Here are the features of Random Forest

FEATURES OF RANDOM FOREST

(7) Disadvantages Of Random Forest

It’s not like that Random Forest is the perfect algorithm, everything has its disadvantages one way or other.

* More accurate ensemble require more trees, which means building and testing the model is a slower process.

* Random Forest is very good at Classification in comparison to Regression.

* In Regression Random Fores does not predict beyond the range of training data.

* Random Forest has been observed to overfit for some of the datasets with very noisy classification / regression tasks.

* For data including categorical variables with different number of levels, Random Forests are biased in favor of those attributes with more levels.

(8) How Does Random Forest Algorithm Works?

In Random Forest we grow multiple trees as opposed to a single tree in the CART model. To classify new objects based on attribute each tree gives a classification and we say the tree votes for that class.

The forest choose the classification having the most votes over all the other trees in the forests and in the case of regression, it takes the average of the outputs by different trees.

Steps related to working of the Random Forest Algorithm :

RANDOM FOREST ALGORITHM

T: Number of Trees to be constructed.

N: Number of Features.

O: The class with the highest vote.

(i) Select Random Features :

The first step is to actually, fill certain “m” features from N, where m less than N. As we know N is the total number of features, out of those features we will be selecting some random features out of those. The reason we are selecting certain features only is because if we select all the predictive variable that each of the Decision Tree will be the same and because of that our model is not learning something new.

(ii) Calculate Best Splitting Point :

For node d, calculate the best split point among them features. Here we pick up the most significant variable and then we split that particular node into further child nodes.

(iii) Split into n daughter nodes :

Split the node into n number of daughter nodes using the best splits.

(iv) Repeat the initial steps :

Repeat the first 3 steps until n number of nodes has been reached. Which means we have to repeat it until we rich the leaf nodes of the tree.

(v) Built your Forest :

Built your forest by repeating steps (i) till (iv) for T number of times.

After this (iv) steps we will have our 1 Decision Tree but Random Forest is about multiple Decision Trees.

This is not over yet, your final task will be to compile the results of all the Decision Trees and you will make a majority voting for the final result.

(9) Random Forest Algorithm Example In Python

Using Random Forest we will predict the person will get a loan or not is not, in this first, we will use a Decision Tree and then a Random Forest, in the end, we will compare both of them in term of accuracy.

DATA SET: Loan_dataset

#Importing Important Libraries

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

#Importing Dataset

data = pd.read_csv(r"D:\Dig\lending_club_data01.csv")
data.head()

Output :

#Getting Information of the Dataset

data.info()

Output :

#Overall Description of the Dataset

data.describe()

Output :

#Referring bad_loans column we create good_loans column where we say yes if 0 and no if 1

data['good_loans']=data['bad_loans'].apply(lambda y: 'yes' if y==0 else 'no')
data.head()

Output :

#For training: all columns except bad_loans and good_loans
#For testing: good loans

X=data.drop(['bad_loans','good_loans'],1)
y=data['good_loans']
print(X.shape,y.shape)

Output :

(1468, 7) (1468,)

# Split data into training and testing

from sklearn.model_selection import train_test_split
X_train , X_test , y_train , y_test = train_test_split(X,y,test_size=0.2)
print(X_train.shape,X_test.shape,y_train.shape,y_test.shape)

Output :

(1174, 7) (294, 7) (1174,) (294,)

# Decision Tree :

from sklearn.tree import DecisionTreeClassifier
model=DecisionTreeClassifier()
model.fit(X_train,y_train)

Output :

#Evaluate our model

predict = model.predict(X_test)
from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test,predict))

Output :
[[ 16 46]

 [ 38 194]]

#Result

print(classification_report(y_test,predict))

Output :

Decision Tree Accuracy is 71%

# Random Forest :

from sklearn.ensemble import RandomForestClassifier
rf_model=RandomForestClassifier(n_estimators=250)
rf_model.fit(X_train,y_train)

Output :

print(X_train.shape,X_test.shape,y_train.shape,y_test.shape)

Output :

(1174, 7) (294, 7) (1174,) (294,)

#Evaluate our model

rf_predict = rf_model.predict(X_test)
print(confusion_matrix(y_test,rf_predict))

Output :

[[ 10  52]
 [  8 224]]

#Result

print(classification_report(y_test,rf_predict))

Output :

Random Forest Accuracy is 80%

Hence it is proved that Random Forest is more accurate than the Decision Tree.

* Random Forest Conclusion:

CONCLUSION OF RANDOM FOREST

Did I Miss Anything?

Now I'd like to hear from you:

- Do you think Random Forest is awesome?

- What else would you like us to cover on this topic?

- What are the 5 most informative concepts you found in this blog?

Do comment your answers, and don't forget to share it with your friend so you can discuss more about this topic.

14 MOST ESSENTIAL Concepts About Decision Tree You Need To Know Right Now [BE A PRO]

2020-05-23T09:47:00.001+05:30

14 MOST ESSENTIAL Concepts About Decision Tree You Need To Know Right Now [BE A PRO]

TREE

Decision Tree is a classification algorithm which comes under the Supervised Learning Technique.

In this revolutionizing world of Machine Learning we had covered various topics such as:

and many more interesting topics in a very simple way with minimum use of complex mathematics. We had also included multiple examples of codes for a better understanding of there implementation.

So our today’s topics is Decision Tree. A topic that every Machine Learning enthusiastic is awarded about.

Before directly diving into Decision Tree. Let us see, what all things that we are going to accomplish in this blog.

(1) What is a Tree?

(2) CART(Classification And Regression Tree)

(3) What is Classification?

(4) Why we need to Classify?

(5) What is Decision Tree?

(6) Important Terminologies of a Decision Tree

(7) What is Entropy?

(8) How does Decision Tree work?

(9) What is the Gini Index?

(10) What is Information Gain?

(11) Application of Decision Tree

(12) Advantages of Decision Tree

(13) Disadvantages of Decision Tree

(14) Decision Tree Example

I hope The agenda of today's session is absolutely clear to you all.

Now Let’s dive in.

(1) What is a Tree?

- In linear the data structure, data is organized in sequential order, and in the non-linear data structure, data is organized in random order.

- The tree is a very popular data structure used in a wide range of applications. A Tree data structure can be defined as follows.

- A tree is a non-linear data structure that organizes data in a hierarchical structure.

- The tree represents the nodes connected by edges.

GLOSSARY TREE

Root:

-In a Tree data structure, the first node is called the Root Node. Every Tree must have a root node.

-Root node is the origin of tree data structure. In any tree, there must be only one root node. We never have multiple root nodes in a Tree.

Parent:

- In a Tree type data structure, the node which is a predecessor of any node is called as Parent Node.

- Parent Node can also be defined as “The Node Which Has Child/Children's”.

-The Root Node is the only node that does not have a Parent Node.

Child:

- The immediate successor of a node is called as Child Node.

- In a Tree, any parent node can have any number of Child Nods.

Siblings:

- Children of the same Parent is called as Siblings. In simple words, the nodes with the same parent are called sibling nodes.

Subtree:

- A Subtree is a set of nodes and edges comprised of a parent and all the descendants of that parent.

Leaf Node:

- The nodes which do not have any child is called Leaf Node.

- Leaf Nodes are called terminal nodes.

(2) CART( Classification And Regression Tree)

So let us see the 2 different categories where Decision Trees can be used.

CART

(i) Classification

(ii) Regression

(i) Classification:

- Classification where all the output is categorical i.e either they are True or False, Male or Female etc or the output belongs to certain category i.e did you scored A, B, C or D grade in your final examination etc.

- In Classification, a Classification Tree will determine a set of logical if-then conditions to classify problems.

Example : Identifying the fruit is an Apple or Orange.

CLASSIFICATION TREE

(ii) Regression:

- Here we predict the output value as a specific number.

- In Regression, a Regression Tree is used when the target variable is numerical or continuous in nature.

- We fit a regression model to target variables using each of the independent variables.

- Each split is made based on the sum of the square error.

REGRESSION TREE

For today’s topic we are going to only focus on the Classification Trees.

(3) What is Classification?

- Classification is a process of categorizing a given set of data into classes, It can be the performance on both structured or unstructured data.

- The process started with predicting the class of given data points.

- The classes are often referred to as target, label, or categories.

E-MAIL IS SPAM OR HAM

Classification is a technique of categorizing the observation into different category.

Basically we are doing is, we are taking data analyzing it, and on the basis of some of the conditions we are dividing it into various categories.

(4) Why do we classify?

- We classify it to perform predictive analysis on it.

For Example :

When we receive an E-mail, the machine predicts it either it could be a spam or not spam. On the bases of that prediction we add that particular mail to the respected folder.

In general our Classification algorithm handles questions like:

a.) Is this data belong to A, B, or C.

b.) It is a Spam or a Ham.

c.) Is that person is a Male or a Female?

(5) What is Decision Tree?

A Decision Tree is a graphical representation of all the possible solutions to a decision based on certain conditions.

We can also say that the decision tree is a flowchart or a tree shape diagram which always starts with a Root Node. Each branch of the tree represents a possible decision, occurrence, or reaction.

Decisions made can be easily explained.

Let’s see an example where you want to purchase a house.

DECISION TREE FOR PURCHASING HOUSE

In this example of the Decision Tree, we want to buy a but at certain conditions. Such as the price of the house should be ₹.5000000 if it's in Mumbai than buy it or else don't buy it and if it's in Pune check if it has 2 BHK or not if it's 2 BHK then purchase the house or else don't purchase the house.

(6) Important Terminologies of a Decision Tree

Basically there are 6 main terminologies regarding Decision Tree.

(a) Entropy

(b) Pruning

(c) Gini Index

(d) Information Gain

(e) Reduction in Variance

(f) Chi-Square

(a) Entropy:

Entropy is the measure of Randomness or Unpredictability in the Dataset.

FRUIT BASKET

If I tell you to pick you the fruit from the basket with your eyes close then the entropy will be max. But in real life entropy should be minimum.

(b) Pruning:

Basically we can say that this is the opposite of Splitting, in this we remove unwanted branches from the tree.

For Example :

PRUNING

We removed the unwanted branch from the tree because we don’t want it.

(c) Gini Index :

The measure of impurity (or purity) used in building Decision Tree in CART is Gini Index.

(d) Information Gain :

The information Gain is the decrease in entropy after a dataset is split on the basis of an attribute. Constructing a decision tree is all about a fighting attribute that returns the highest information gain.

Information Gain = Entropy(s) – [ (Weighted Average) * Entropy(each feature)]

(e) Reduction in Variance :

Reduction invariance is an algorithm used for continuous target variable (regression problems). This split with lower variance is selected as the criteria to split the population.

(f) Chi-square :

It is an algorithm to find out the statistical significance between the difference between sub-nodes and parent node.

Now the main question is how will you decide the best attribute? For now just understand that you need to calculate something called Information Gain.

The attribute with the highest information gain is considered as best. Before understanding information gain let’s first understand entropy.

(7) What is Entropy?

Entropy is the metric, which measures the impurity of something or in other words you can say that it is a first step to do before you solve the problem for Decision Tree.

Let’s understand a term known as Impurity before understanding Entropy.

LOW / HIGH ENTROPY

For Example :

Let’s say you have a dataset and you are trying to distinguish between various fruits and instead of taking multiple factors under consideration, you only took color as the only feature for differencing the fruits.

So what will happen is, for example there are green apple and red apple and strawberry in the dataset. You will keep the red apples and the strawberry in one basket. So here we can say that in this impurity is not zero which means that the impurity in this exists.

ENTROPY

Now come back to the word Entropy.

* Defines randomness in the data.

* Entropy is just a metric that measures the impurity.

* It is the first step to solve the problem of a decision tree.

ENTROPY GRAPH

From the above graph you can see that if the probability is 0 or 1 then it means that they are highly impure and if the value is 0.5 then it means that the value is very pure.

Now you will be thinking why the value of the entropy is pure at 0.5? Let me derive it mathematically.

As you can see that the formula of entropy is :

Entropy(s) = -P(yes) log2 P(yes) – P(no) log2 P(no)

Where;

* s is the total sample space.

* P(yes) is the probability of Yes.

* P(no) is the probability of No.

If number of yes = number of no i.e P(s) = 0.5

→ Entropy(s) = 1

If it contains all yes or all no i.e P(s) = 1 or 0

→ Entropy(s) = 0

Let’s start with conditions :

1.) Condition 1 Probability = 0.5

E(s) = -P(yes) log2 P(yes)

When P(yes) = P(no) = 0.5 i.e YES + NO = Total Sample(s)

E(s) = 0.5

Now substitute that value 0.5 into the formula.

E(s) = 0.5 log2 0.5 – 0.5 log2 0.5

E(s) = 0.5(log2 0.5 – log2 0.5)

E(s) = 1

2.) Condition 2 Either we have total YES or total NO

(i) Total YES :

E(s) = -P(yes) log2 P(yes)

When P(yes) = 1 i.e YES = Total Sample(s)

E(s) = 1 log2 1 … log 1 = 0

E(s) = 0

(ii)Total NO :

E(s) = -P(no) log2 P(no)

When P(no) = 1 i.e YES = Total Sample(s)

E(s) = 1 log2 1 … log 1 = 0

E(s) = 0

(8) How does Decision Tree work?

I am sure most of you have a general idea about how a decision tree works, but for better understanding and for clear knowledge let’s take an example dataset and understand it diagrammatically.

DATASET

In this dataset each row is an example and the initial 2 columns provide features or attributes that describe the data and the last column gives the label or the class we want to predict.

Now if you look at this dataset this is very straight forward except from 1 thing. If you look at our 4^th label column and compare that with 1^st and 2^nd labels you will notice that the features are the same but the label is different.

Let’s see diagrammatically how the decision tree handles this case.

In order to build this decision tree we will be using CART which we have discussed above.

Step1:

We will start with the root node as a tree and all the nodes receive a list of rows as input and root will receive the entire training dataset.

Step2:

Now each node will ask True or False questions about one feature and in response to that question we will split or partition the dataset into 2 different subsets these subsets then become an input to 2 child nodes. The goal of the question is to finally unmix the label as we proceed down or in other words to produce the purest possible distribution of the labels at each node.

SPLITTING

Step3:

The input of this node contains only one single type of label so we could say that it’s perfectly unmixed there is no uncertainty about the type of label as it consists of the only strawberry. On the other hand labels in this node are still mixed up. So we will ask another question to further drill it down. But before that, we need to understand which question to ask and when to do that we need to quantify how much question helps to unmix the label and then we can quantify the amount of uncertainty at a single node using a metric called Gini impurity and we can quantify how much a question reduces that uncertainty using a concept called Information Gain. We will use this to ask the best questions at each point and then we will iterate the steps and we will recursively build a tree on each of the new nodes.

FURTHER SPLITTING

Step4:

We will continue to divide the data until there are no further questions to ask and then we will finally reach our leaf.

FOUND LEAF NODE

(9) What is the Gini Index?

Gini index means that, if we select 2 items from a given dataset at random then they must be of same probability and class for this population should be pure.

It works with the categorical target variable “True” or “False”.

1. It performs only Binary (0/1) splits

2. Higher the value of Gini index is higher the homogeneity.

3. CART (Classification and Regression Tree) uses the Gini method to create binary splits.

Steps to Calculate Gini for a split :

1. Calculate Gini for sub-nodes, using formula sum of the square of probability for True and False (p²+q²).

2. Calculate the Gini index for split using the weighted Gini score of each node of that split.

(10) What is Information Gain?

In Information Gain, Less impure node requires less information to describe it and a more impure node requires more information to describe it. Information theory is a measure to define this degree of disorganization in a system known as Entropy. If the sample is completely homogeneous, then the entropy is 0 and if the sample is equally divided into half at each side i.e 50% - 50% then it has an entropy of 1.

Entropy can be calculated using the formula:

Entropy = -p log₂ p — q log₂q

Here p is the probability of success and q is the probability of failure in that node. Entropy is also used with a categorical target variable. It chooses the split which has the lowest entropy compared to the parent node and other splits. The lesser the entropy, the better it is.

Steps to calculate the entropy for a split:

Step 1: Calculate entropy of parent node.

Step 2: Calculate entropy of each individual node of split and calculate the weighted average of all sub-nodes available in the split.

We can derive information gain from entropy as 1- Entropy.

(11) Application of Decision Tree

We use Decision Tree more often then you think. Here are some of the real world application of Decision Tree which will blow you mind.

(i) Telecommunication Industry :

TELECOMMUNICATION

Telecommunication Industry uses this very often as if you want to chat with customer care, you might have noticed that they guide you, by instructing you to provide you the best services they can. For example Press 1 for Hindi, Press 2 for English so on.

(ii) Online Shopping:

ONLINE SHOPPING

Online The marketing industry uses the Decision Tree for several purposes, such as based on the last purchase or based on the last search of the product they show you more similar products. Amazon uses this very often, we search for sports shoes it will show us all kinds of sports shoes.

(iii) Handle our Craving :

PIZZA

I know this is funny, but give it a thought. Let’s say you have 20$ and we are very hungry. So what will you do? eat 1 pizza or eat 2 burgers. I leave this completely up to use guys😂.

(12) Advantages of Decision Tree :

Let us see the Advantages of Decision Trees :

* It is simple to understand.

* It is not difficult to interpret and visualize it.

* Very Less efforts are required for data preparation.

* We can handle both numerical and categorical data with the help of the Decision Tree.

* Performance of the Decision Tree is not Effected by the non-linear parameters.

(13) Disadvantages of Decision Tree :

Let us see the Disadvantages of Decision Trees :

* The Primary disadvantages of the decision tree are overfitting. Overfitting occurs when the algorithm capture noise in the data.

* In decision tree there is also a problem of high variance. The model can get unstable due to small variance in the data.

* In a decision tree is there is Low Biased tree. A highly complicated decision tree tends to have a low bias which makes it difficult for the model to work with new data.

* Calculation can get very complex particularly if many values are uncertain and if many outcomes are linked.

(14) Decision Tree Example :

Will that movie will win Oscar or Not?

CSV File : Classification_dataset

# Import all libraries

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# Import Dataset

df = pd.read_csv("E:\college_pics\Rainbow6\classification.csv",header=0)

df.head().T

Output :

# Display information of the dataset

df.info()

Output :

# Handel Missing Values

df['Time_taken'].fillna(value = df['Time_taken'].mean(),inplace=True)
df.head().T

Output :

df.info()

Output :

# Creating Dummy Variables

df = pd.get_dummies(df,columns = ['3D_available','Genre'], drop_first=True)
df.head().T

Output :

# X and y split

X = df.drop('Start_Tech_Oscar',axis=1)
type(X)

Output :

pandas.core.frame.DataFrame

X.shape

Output :

(506, 20)

y = df['Start_Tech_Oscar']
type(y)

Output :

pandas.core.series.Series

y.shape

Output :

(506,)

# Train-Test Split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=0)

X_train.shape

Output :

(404, 20)

X_test.shape

Output :

(102, 20)

# Training Classification Tree

from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=3)
model.fit(X_train,y_train)

Output :

DecisionTreeClassifier
(class_weight=None, criterion='gini', max_depth=3,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, presort=False,
random_state=None, splitter='best')

# Predicting Values using the trained model

y_train_pred = model.predict(X_train)
y_test_pred = model.predict(X_test)
y_train_pred.shape

Output :

(404,)

y_test_pred

Output :

array([0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1,
0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0], dtype=int64)

# Model Performance

from sklearn.metrics import accuracy_score, confusion_matrix

confusion_matrix(y_train, y_train_pred)

Output :

array([[172, 14],
[126, 92]], dtype=int64)

confusion_matrix(y_test, y_test_pred)

Output :

array([[39, 5],
[41, 17]], dtype=int64)

accuracy_score(y_test, y_test_pred)

Output :

0.5490196078431373

dot_data = tree.export_graphviz(model, out_file=None, feature_names=X_train.columns, filled=True)

from IPython.display import Image
import pydotplus
graph = pydotplus.graph_from_dot_data(dot_data)
Image(graph.create_png())

Output :

model2 = DecisionTreeClassifier(min_samples_leaf=20, max_depth=4)
model2.fit(X_train,y_train)

dot_data_2 = tree.export_graphviz(model2, out_file=None, feature_names=X_train.columns, filled=True)
from IPython.display import Image
import pydotplus
graph2 = pydotplus.graph_from_dot_data(dot_data_2)
Image(graph2.create_png())

Output :

accuracy_score(y_test, model2.predict(X_test))

Output :

0.5588235294117647

* Decision Tree Summary:

Let us summarize everything:

- We saw what is Tree and some important terms related to it.

- CART: Classification and Regression Tree.

- Then we saw what classification is and a SPAM and HAM image.

- After that, we saw why do we classify?

- We understood the Decision Tree with a House Purchasing example.

- Some Important Terminologies.

- Then we saw what is Entropy. In which we saw a diagram and a graph with values 0, 0.5, and 1.

- Then we saw how the Decision Tree works with an example.

- We also saw the Gini index and steps to calculate it.

- Which lead us to Information Gain and how to calculate it.

- Then we saw the various real-life application of Decision Tree.

- After that, we had a glimpse of the Advantages of the Decision Tree.

- Everything has its own disadvantages, So we also saw Disadvantages of the Decision Tree.

- Then we saw a coding step by step example of the Decision Tree.

Did I Miss Anything?

Now I'd like to hear from you:

- Which Concept did you like the most?

- What else would you like us to cover on this topic?

- What are the 5 most informative concepts you found in this blog?

Do comment, your answers and don't forget to share it with your friend so, you can discuss more this topic.