Web Snippets

THOUGHT AND CHARACTER

2020-06-23T05:08:00.002-07:00

A man is literally what he thinks, his character being the complete sum of all his thoughts.
Every act of man springs from the hidden seeds of thought, and could not have appeared without them.
Act is a blossom of thought and joy and suffering are its fruits.
Godlike character is not a thing of favor or chance, but is the natural result of continued effort of like thinking.
By patience and practice shall a man enter the door of the temple of knowledge
Courtesy: As a man thinketh. By James Allen

Human-AI Interaction Design Guidelines

2020-06-15T11:55:00.001-07:00

Human-AI Interaction Design Guidelines

INITIALLY
01 Make clear what the system can do.

Help the user understand what the AI system is capable of doing.

02 Make clear how well the system can do what it can do.

Help the user understand how often the AI system may make mistakes.

DURING INTERACTION
03 Time services based on context.

Time when to act or interrupt based on the user’s current task and environment.

04 Show contextually relevant information.

Display information relevant to the user’s current task and environment.

05 Match relevant social norms.

Ensure the experience is delivered in a way that users would expect, given their social and cultural context.

06 Mitigate social biases.

Ensure the AI system’s language and behaviors do not reinforce undesirable and unfair stereotypes and biases.

WHEN WRONG
07 Support efficient invocation.

Make it easy to invoke or request the AI system’s services when needed.

08 Support efficient dismissal.

Make it easy to dismiss or ignore undesired AI system services.

09 Support efficient correction.

Make it easy to edit, refine, or recover when the AI system is wrong.

10 Scope services when in doubt.

Engage in disambiguation or gracefully degrade the AI system’s services when uncertain about a user’s goals.

11 Make clear why the system did what it did.

Enable the user to access an explanation of why the AI system behaved as it did.

OVER TIME
12 Remember recent interactions.

Maintain short-term memory and allow the user to make efficient references to that memory.

13 Learn from user behavior.

Personalize the user’s experience by learning from their actions over time.

14 Update and adapt cautiously.

Limit disruptive changes when updating and adapting the AI system’s behaviors.

15 Encourage granular feedback.

Enable the user to provide feedback indicating their preferences during regular interaction with the AI system.

16 Convey the consequences of user actions.

Immediately update or convey how user actions will impact future behaviors of the AI system.

17 Provide global controls.

Allow the user to globally customize what the AI system monitors and how it behaves.

18 Notify users about changes.

Inform the user when the AI system adds or updates its capabilities.

MORE INFO:
https://www.microsoft.com/en-us/research/blog/guidelines-for-human-ai-interaction-design/

Machine Learning workflow

2020-02-05T16:45:00.002-08:00

Machine Learning Workflow consists of 3 components

Explore and process data
Modeling
Deployment

EXPLORE AND PROCESS DATA

This component consists of exploring and processing the data.

Retrieve

The first step is to retrieve the data, which includes test and train dataset. Lets take an example of housing dataset which contains csv files. We need to download the data from the source.

Clean and Explore

The data might have to be cleaned to identtify and remove anomalous values such as outliers and mistakes

Prepare and transform

The final step in this block is to prepare and transform the data
This steps involves normalization and converting the format of the data
This steps involves splitting data into training, validation and test sets

MODELING

This component focuses on developing the model that is deployed to production

Develop and train model

The first step where the model is developed and trained using the training dataset.

Evaluate and model

Final step is to evaluate and validate the model. Here we tune the model using the validation data set.

DEPLOYMENT

This component focuses on deployment, monitoring and updating the model in the production environment

Deploy to production

Deployment means making the model available for use by web or software applications. We can also deploy the model to a smartphone application

Monitor and update model and data

If there is any changes in the features and data, we can update the model and data

Error functions

2020-01-29T11:57:00.001-08:00

In most learning networks, error is calculated as the difference between the actual output and the predicted output.
The error function is which tells us how far are we from the solution.
The function that is used to compute this error is known as loss function.
Different loss functions will give different errors for the same prediction and thus would have a considerable effort on the performance of the model.

EXAMPLE:
Imagine, we are standing on top of a mountain(mount Everest) and we want to descend.It is not that easy and it is cloudy and it is big and we cant see the big picture.We would look at all the possible directions where we can walk.

Note:
If we constantly take step to decrease the error by decreasing the height then we would reach all the way down the mountain.

KEY MATRIX

In this case the key matrix that we use to solve the problem is the height.
We can call the height the error.This would say that how badly we are doing at the mountain and how far we are then an ideal solution.

GRADIENT DESCENT

We might also be stuck in a valley.This would often happen in solving real world problems. We have to resolve this issue.Many time the local min would give a pretty good solution to the problem.This method is called gradient descend

GOAL TO SPLIT DATA

How do we tell the computer how far are they from a perfect solution?

We can count the no of mistakes. (example that is our height)

Now let us try to decrease the no of errors

SOLUTION

1) After moving once step we had decreased the errors

2) After moving another steps we have decreased all the errors

ISSUE

The problem with this approach is that,algorithm would be taking very small steps.The reason for this is calculus. Tiny steps would be calculated by derivatives.
The problem with small steps is:

We start with 2 errors
We move a small amount
We are still at 2 errors
Even after moving a tiny amount we are at 2 error

This is equivalent to using gradient descend from an aesthetic pyramid with flat steps.

DISCRETE

If we are standing above and looking for errors, then we would always get 2 errors and we would get confused what to do.

CONTINUOUS

In this case we can figure out which direction we can decrease the most.
In math terms, in order for us , it means to do the gradient descent, our error function cannot be discrete. It should be continuos

LOG LOSS ERROR FUNCTION

As shown above in the figure, we have 6 points, out of which 4 are correctly classified and other 2 are incorrectly classified.

Assuming error function would give penalty to incorrectly classified points an small penalty to the 4 correctly specified points
Here we are representing the size of the points as penalty

Note:

Penalty is the distance from the boundary when the points are missclassified and 0 when they are correctly classified
Lets add all the errors from the corresponding points
The idea now is to move the line around to decrease these errors. In the fig below we have decreased the error.

We can now use gradient descent to solve our problem

SOLUTION

GRADIENT DESCENT
ERROR : In this example error is the sum of blue and red areas

We explore around to see what direction brings us the down most or equivalent.
We explore what direction we can move to reduce the error the most
We take a step in that direction

In the mountain we go one step down and in the graph we reduce the error a bit by classifying one of the points
Now we look again and follow the steps described above

On the left we have reduced the height and have successfully descended from the mountain and on the right we have reduced the error to the min possible value and successfully classified our points

Canny edge detector

2020-01-29T11:49:00.000-08:00

The Canny edge detector is an edge detection operator that uses a multi-stage algorithm to detect a wide range of edges in images. It was developed by John F. Canny in 1986. Canny also produced a computational theory of edge detection explaining why the technique works.

Canny edge detection is a technique to extract useful structural information from different vision objects and dramatically reduce the amount of data to be processed. It has been widely applied in various computer vision systems. Canny has found that the requirements for the application of edge detection on diverse vision systems are relatively similar. Thus, an edge detection solution to address these requirements can be implemented in a wide range of situations. The general criteria for edge detection include:

Detection of edge with low error rate, which means that the detection should accurately catch as many edges shown in the image as possible
The edge point detected from the operator should accurately localize on the center of the edge.
A given edge in the image should only be marked once, and where possible, image noise should not create false edges.

To satisfy these requirements Canny used the calculus of variations – a technique which finds the function which optimizes a given functional. The optimal function in Canny's detector is described by the sum of four exponential terms, but it can be approximated by the first derivative of a Gaussian.

Among the edge detection methods developed so far, Canny edge detection algorithm is one of the most strictly defined methods that provides good and reliable detection. Owing to its optimality to meet with the three criteria for edge detection and the simplicity of process for implementation, it became one of the most popular algorithms for edge detection.

Grayscale to RGB conversion

2020-01-29T11:38:00.001-08:00

we will convert an color image into a grayscale image. There are two methods to convert it. Both has their own merits and demerits. The methods are:

Average method
Weighted method or luminosity method
using skimage

Average method

Average method is the most simple one. You just have to take the average of three colors. Since its an RGB image, so it means that you have add r with g with b and then divide it by 3 to get your desired grayscale image.

Its done in this way.

Grayscale = (R + G + B / 3)

For example:

If you have an color image like the image shown above and you want to convert it into grayscale using average method. The following result would appear.

Explanation

There is one thing to be sure, that something happens to the original works. It means that our average method works. But the results were not as expected. We wanted to convert the image into a grayscale, but this turned out to be a rather black image.

Problem

This problem arise due to the fact, that we take average of the three colors. Since the three different colors have three different wavelength and have their own contribution in the formation of image, so we have to take average according to their contribution, not done it averagely using average method. Right now what we are doing is this,

33% of Red, 33% of Green, 33% of Blue

We are taking 33% of each, that means, each of the portion has same contribution in the image. But in reality thats not the case. The solution to this has been given by luminosity method.

Weighted method or luminosity method

You have seen the problem that occur in the average method. Weighted method has a solution to that problem. Since red color has more wavelength of all the three colors, and green is the color that has not only less wavelength then red color but also green is the color that gives more soothing effect to the eyes.

It means that we have to decrease the contribution of red color, and increase the contribution of the green color, and put blue color contribution in between these two.

So the new equation that form is:

New grayscale image = ( (0.3 * R) + (0.59 * G) + (0.11 * B) ).

According to this equation, Red has contribute 30%, Green has contributed 59% which is greater in all three colors and Blue has contributed 11%.

Applying this equation to the image, we get this

Original Image:

Grayscale Image:

Explanation

As you can see here, that the image has now been properly converted to grayscale using weighted method. As compare to the result of average method, this image is more brighter.

using skimage

import numpy as np
import matplotlib.pyplot as plt

from skimage import data
from skimage.color import rgb2gray

original = data.astronaut()
grayscale = rgb2gray(original)

fig, axes = plt.subplots(1, 2, figsize=(8, 4))
ax = axes.ravel()

ax[0].imshow(original)
ax[0].set_title("Original")
ax[1].imshow(grayscale, cmap=plt.cm.gray)
ax[1].set_title("Grayscale")

fig.tight_layout()
plt.show()

pyplot

2020-01-29T11:33:00.005-08:00

matplotlib.pyplot

Provides a MATLAB-like plotting framework.

pylab combines pyplot with numpy into a single namespace. This is convenient for interactive work, but for programming it is recommended that the namespaces be kept separate, e.g.:

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0, 5, 0.1);
y = np.sin(x)
plt.plot(x, y)

matplotlib.pyplot.acorr(x, hold=None, data=None, **kwargs)

Plot the autocorrelation of x.

Parameters:

Parameters:	x : sequence of scalar hold : boolean, optional, deprecated, default: True detrend : callable, optional, default: `mlab.detrend_none` x is detrended by the `detrend` callable. Default is no normalization. normed : boolean, optional, default: True if True, input vectors are normalised to unit length. usevlines : boolean, optional, default: True if True, Axes.vlines is used to plot the vertical lines from the origin to the acorr. Otherwise, Axes.plot is used. maxlags : integer, optional, default: 10 number of lags to show. If None, will return all 2 * len(x) - 1 lags.
Returns:	(lags, c, line, b) : where: `lags` are a length 2`maxlags+1 lag vector. `c` is the 2`maxlags+1 auto correlation vectorI `line` is a `Line2D` instance returned by `plot`. `b` is the x-axis.
Other Parameters:
	linestyle : `Line2D` prop, optional, default: None Only used if usevlines is False. marker : string, optional, default: ‘o’

x : sequence of scalar

hold : boolean, optional, deprecated, default: True

detrend : callable, optional, default: mlab.detrend_none

x is detrended by the detrend callable. Default is no normalization.

normed : boolean, optional, default: True

if True, input vectors are normalised to unit length.

usevlines : boolean, optional, default: True

if True, Axes.vlines is used to plot the vertical lines from the origin to the acorr. Otherwise, Axes.plot is used.

maxlags : integer, optional, default: 10

number of lags to show. If None, will return all 2 * len(x) - 1 lags.

Returns:

(lags, c, line, b) : where:

lags are a length 2`maxlags+1 lag vector.

c is the 2`maxlags+1 auto correlation vectorI

line is a Line2D instance returned by plot.

b is the x-axis.

Other Parameters:

linestyle : Line2D prop, optional, default: None

Only used if usevlines is False.

marker : string, optional, default: ‘o’

Notes

The cross correlation is performed with numpy.correlate() with mode = 2.

Examples

xcorr is top graph, and acorr is bottom graph.

(Source code, png, pdf)

Transfer learning

2020-01-23T15:52:00.000-08:00

TRANSFER LEARNING Using a pre trained network on images not in training set is known as transfer learning

DIFFERENT ARCHITECTURES WE CAN USE

THESE ARE THE TOP 1 AND TOP 2 ERROR

The numbers such as VCG-11 where 11 are the no of layers.
When we are using this we need to do the trade-off between accuracy and speed.
They are massively deep.They have 100's of hidden layer

MODELS

USING DENSENET

It has 121 different layers

LOADING AND ARCHITECTURE

After loading we can see the architecture.We have features and classifiers
The classifier has 1024 input features and 1000 output features
Imagenet dataset has 1000 different classes.The no of output should be 1000 for these classes.
The features would not be used and need to be freezed.

FREEZING THE FEATURES

We need feature part static, to this we would freeze our feature parameter as shown below
This would increase processing speed as it would not keep track of features

BUILD CLASSIFIER
We would build our classifier as shown below

ATTACH TO OUR MODEL

MOVE TO GPU

We can move all the computing to a GPU by specifying model.cuda . This would move all the parameters and model to the GPU

To move the tensors to the GPU, we need to use image.cuda
To specify cpu we need to change it to image.cpu, model.cpu

COMPARISON BETWEEN CPU AND GPU

We would get a speed of over a 100 times

MNIST GAN

2020-01-23T15:43:00.000-08:00

Here we have 2 neural networks GENERATOR and DISCRIMINATOR

DISCRIMINATOR:

Is a simple classifier that tries to classify the images as real from the training set or fake generated images.

GENERATOR:

Acting as an adversary to the discriminator.
It aims to trick the discrminator, giving it generated images that look as if they've come from the training set.
If the generator produces an image that the discriminator thinks is fake, then it would change it behavior and try again.
Generator would trick till the discrimnator starts thinking that its generated data comes from the real training set.( Goal is to force the discriminator to have a high error rate as possible.Note that at the same time discriminator is also training. it looks at examples of real and fake images and getting better at recognizing differences between them.its goal is to have a low error rate.)

Task is to have both discriminator and generator networks with opposing goals.We define opposing generator and discriminator loss functions.

At the end of the training discriminator should not be able to tell the difference between real and fake images

Glob module in python

2020-01-23T14:26:00.000-08:00

Usually, the programmers require to traverse through a list of files at some location, mostly having a specific pattern. Python’s glob module has several functions that can help in listing files under a specified folder. We may filter them based on extensions, or with a particular string as a portion of the filename.

All the methods of Glob module follow the Unix-style pattern matching mechanism and rules. However, it doesn’t allow expanding the tilde (~) and environment variables.

Python Glob() Methods

There are three primary functions of Glob module.

Contents [hide]

Glob(File_pattern, Recursive = False)

It retrieves the list of files matching the specified pattern in the file_pattern parameter.

The file_pattern can be an absolute or relative path. It may also contain wild cards such as “*” or “?” symbols.

The recursive parameter is turn off (False) by default. When True, it recursively searches files under all subdirectories of the current directory.

Let’s now check out some examples:

Check The Current Directory For Python Script Files

The below code checks for .py files in the current dir only.

>>> import glob
>>> for py in glob.glob("*.py"):
...    print(py)
...
copy_file1.py
copy_file2.py
decimal_sample.py

Another sample code – It checks for .py files in current dir and subdirectories.

>>> import glob
>>> for py in glob.glob("*.py"):
...    print(py)
...
copy_file1.py
copy_file2.py
decimal_sample.py
test_num.py
test_python_in_with_if.py
test_scope.py

List Files With A Pattern

We can provide a pathname pattern by including some wild cards like ? or numeric range [0-9]. The below code lists all files whose name starts with “test” followed by a number.

>>> for py in glob.glob("test[0-9].py"):
...    print(py)
...
test1.py
test2.py

Let’s check one more example using the question mark in the pattern.

>>> for py in glob.glob("?????.py"):
...    print(py)
...
quiz1.py
test1.py
test2.py

The above for loop statement printed all .py files having five letters.

The following statement would print the names of folders recursively in the current working directory.

>>> glob.glob('selenium/**/', recursive=True)
['selenium\', 'selenium\webdriver\', 'selenium\webdriver\firefox\', 'selen
ium\webdriver\firefox\amd64\', 'selenium\webdriver\firefox\x86\']

Iglob() Method | Python Glob

This method creates a Python generator object which can be used to list files under a given directory. You can call the next() function to print names of files.

Check the sample code below:

>>> gen = glob.iglob("*.py")
>>> type(gen)
<class 'generator'>
>>> for py in gen:
...    print(py)
...
copy_file1.py
copy_file2.py
decimal_sample.py
find_seed.py
frozen_set.py

Escape() Method

It allows for escaping the given character sequence. You can find it handy for locating files with certain characters in their file names.

Flattening in NLP

2020-01-22T14:02:00.001-08:00

Instead of representing this 4*4 matrix, we can construct a vector with 16 entries.
First 4 entries correspond to the first row.similarly 2nd 3rd and 4th

Note:After converting our image to an vector they can be fed to an input layer on NLP.

MoviePy

2020-01-22T13:57:00.002-08:00

MoviePy is a Python module for video editing, which can be used for basic operations (like cuts, concatenations, title insertions), video compositing (a.k.a. non-linear editing), video processing, or to create advanced effects. It can read and write the most common video formats, including GIF.

Here it is in action (run in an IPython Notebook):

More information is available at https://zulko.github.io/moviepy/

Do I need MoviePy ?

Here are a few reasons why you may want to edit videos in Python:

You have many videos to process or to compose in a complicated way.
You want to automatize the creation of videos or GIFs on a web server (Django, Flask, etc.)
You want to automatize tedious tasks, like title insertions tracking objects, cuting scenes, making end credits, subtitles, etc...
You want to code your own video effects to do something no existing video editor can.
You want to create animations from images generated by another python library (Matplotlib, Mayavi, Gizeh, scikit-images...)

And here are a few uses for which MoviePy is NOT the best solution:

You only need to do frame-by-frame video analysis (with face detection or other fancy stuff). This could be done with MoviePy in association with other libraries, but really, just use imageio, OpenCV or SimpleCV, these are libraries that specialize in these tasks.
You only want to convert a video file, or turn a series of image files into a movie. In this case it is better to directly call ffmpeg (or avconv or mencoder...) it will be faster more memory-efficient than going through MoviePy.

Advantages and limitations

MoviePy has been developed with the following goals in mind:

Simple an intuitive. Basic operations can be done in one line. The code is easy to learn and easy to understand for newcomers.
Flexible. You have total control over the frames of the video and audio, and creating your own effects is easy as Py.
Portable. The code uses very common software (Numpy and FFMPEG) and can run on (almost) any machine with (almost) any version of Python.

For the limitations: MoviePy cannot (yet) stream videos (read from a webcam, or render a video live on a distant machine), and is not really designed for video processing involving many successive frames of a movie (like video stabilization, you’ll need another software for that). You can also have memory problems if you use many video, audio, and image sources at the same time (>100), but this will be fixed in future versions.

Example code

In a typical MoviePy script, you load video or audio files, modify them, put them together, and write the final result to a new video file. As an example, let us load a video of my last holidays, lower the volume, add a title in the center of the video for the first ten seconds, and write the result in a file:

# Import everything needed to edit video clips
from moviepy.editor import *

# Load myHolidays.mp4 and select the subclip 00:00:50 - 00:00:60
clip = VideoFileClip("myHolidays.mp4").subclip(50,60)

# Reduce the audio volume (volume x 0.8)
clip = clip.volumex(0.8)

# Generate a text clip. You can customize the font, color, etc.
txt_clip = TextClip("My Holidays 2013",fontsize=70,color='white')

# Say that you want it to appear 10s at the center of the screen
txt_clip = txt_clip.set_pos('center').set_duration(10)

# Overlay the text clip on the first video clip
video = CompositeVideoClip([clip, txt_clip])

# Write the result to a file (many options available !)
video.write_videofile("myHolidays_edited.webm")

How MoviePy works

MoviePy uses the software ffmpeg to read and to export video and audio files. It also (optionally) uses ImageMagick to generate texts and write GIF files. The processing of the different media is ensured by Python’s fast numerical library Numpy. Advanced effects and enhancements use some of Python’s numerous image processing libraries (PIL, Scikit-image, scipy, etc.).

Basic concepts

The central objects of MoviePy are clips, which can be AudioClips or VideoClips. They can be modified (cut, slowed down, darkened...) or put mixed with clips to form new clips, they can be previewed (using either PyGame or the IPython Notebook) and rendered to a file (as a MP4, a GIF, a MP3, etc.). VideoClips for instance can be created from a video file, an image, a text, or a custom animation. They can have an audio track (which is an AudioClip) and a mask (a special VideoClip indicating which parts of the clip to hide when the clip is mixed with other clips). See Creating and exporting video clips and Mixing clips for more details.

A clip can be modified using one of moviepy’s numerous effects (like in clip.resize(width="360"), clip.subclip(t1,t2), or clip.fx(vfx.black_white)) or using a user-implemented effect. MoviePy implements many functions (like clip.fl, clip.fx, etc.) which make it very easy to code your own effect in a few lines. See Clips transformations and effects for more.

You will also find a few advanced goodies in moviepy.video.tools to track objects in a video, draw simple shapes and color gradients (very useful for masks), generate subtitles and end credits, etc. See Advanced tools for a description of these.

Finally, although MoviePy has no graphical user interface, there are many ways to preview a clip which allow you to fine-tune your scripts and be sure that everything is perfect when you render you video in high quality. See How to be efficient with MoviePy.

Gaussian Blur operation

2020-01-22T13:47:00.003-08:00

In Gaussian Blur operation, the image is convolved with a Gaussian filter instead of the box filter. The Gaussian filter is a low-pass filter that removes the high-frequency components are reduced.

You can perform this operation on an image using the Gaussianblur()method of the imgproc class. Following is the syntax of this method −

GaussianBlur(src, dst, ksize, sigmaX)

This method accepts the following parameters −

src − A Mat object representing the source (input image) for this operation.
dst − A Mat object representing the destination (output image) for this operation.
ksize − A Size object representing the size of the kernel.
sigmaX − A variable of the type double representing the Gaussian kernel standard deviation in X direction.

Example

The following program demonstrates how to perform the Gaussian blur operation on an image.

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.Size;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

public class GaussianTest {
   public static void main(String args[]) {
      // Loading the OpenCV core library
      System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

      // Reading the Image from the file and storing it in to a Matrix object
      String file ="C:/EXAMPLES/OpenCV/sample.jpg";
      Mat src = Imgcodecs.imread(file);

      // Creating an empty matrix to store the result
      Mat dst = new Mat();
    
      // Applying GaussianBlur on the Image
      Imgproc.GaussianBlur(src, dst, new Size(45, 45), 0);

      // Writing the image
      Imgcodecs.imwrite("E:/OpenCV/chap9/Gaussian.jpg", dst);
      System.out.println("Image Processed");
   }
}

Assume that following is the input image sample.jpg specified in the above program.

Output

On executing the program, you will get the following output −

Image Processed

If you open the specified path, you can observe the output image as follows −

Geometric Image Transformation

2020-01-22T13:41:00.001-08:00

Geometric Image Transformations

The functions in this section perform various geometrical transformations of 2D images. They do not change the image content but deform the pixel grid and map this deformed grid to the destination image. In fact, to avoid sampling artifacts, the mapping is done in the reverse order, from destination to the source. That is, for each pixel  of the destination image, the functions compute coordinates of the corresponding “donor” pixel in the source image and copy the pixel value:

In case when you specify the forward mapping  , the OpenCV functions first compute the corresponding inverse mapping  and then use the above formula.

The actual implementations of the geometrical transformations, from the most generic `remap()` and to the simplest and the fastest `resize()` , need to solve two main problems with the above formula:

Extrapolation of non-existing pixels. Similarly to the filtering functions described in the previous section, for some  , either one of  , or  , or both of them may fall outside of the image. In this case, an extrapolation method needs to be used. OpenCV provides the same selection of extrapolation methods as in the filtering functions. In addition, it provides the method `BORDER_TRANSPARENT` . This means that the corresponding pixels in the destination image will not be modified at all.

Interpolation of pixel values. Usually  and  are floating-point numbers. This means that  can be either an affine or perspective transformation, or radial lens distortion correction, and so on. So, a pixel value at fractional coordinates needs to be retrieved. In the simplest case, the coordinates can be just rounded to the nearest integer coordinates and the corresponding pixel can be used. This is called a nearest-neighbor interpolation. However, a better result can be achieved by using more sophisticated interpolation methods , where a polynomial function is fit into some neighborhood of the computed pixel  , and then the value of the polynomial at  is taken as the interpolated pixel value. In OpenCV, you can choose between several interpolation methods. See `resize()` for details.

convertMaps

Converts image transformation maps from one representation to another.

C++: void `convertMaps`(InputArray map1, InputArray map2, OutputArray dstmap1, OutputArray dstmap2, int dstmap1type, bool nninterpolation=false )

Python: `cv2.convertMaps`(map1, map2, dstmap1type[, dstmap1[, dstmap2[, nninterpolation]]]) → dstmap1, dstmap2

Parameters:

map1 – The first input map of type `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` .

map2 – The second input map of type `CV_16UC1` , `CV_32FC1` , or none (empty matrix), respectively.

dstmap1 – The first output map that has the type `dstmap1type` and the same size as `src` .

dstmap2 – The second output map.

dstmap1type – Type of the first output map that should be `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` .

nninterpolation – Flag indicating whether the fixed-point maps are used for the nearest-neighbor or for a more complex interpolation.

The function converts a pair of maps for `remap()` from one representation to another. The following options ( `(map1.type(), map2.type())` `(dstmap1.type(), dstmap2.type())` ) are supported:

. This is the most frequently used conversion operation, in which the original floating-point maps (see `remap()` ) are converted to a more compact and much faster fixed-point representation. The first output array contains the rounded coordinates and the second array (created only when `nninterpolation=false` ) contains indices in the interpolation tables.

. The same as above but the original maps are stored in one 2-channel matrix.

Reverse conversion. Obviously, the reconstructed floating-point maps will not be exactly the same as the originals.

See also

`remap()`, `undistort()`, `initUndistortRectifyMap()`

Parameters:	map1 – The first input map of type `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` . map2 – The second input map of type `CV_16UC1` , `CV_32FC1` , or none (empty matrix), respectively. dstmap1 – The first output map that has the type `dstmap1type` and the same size as `src` . dstmap2 – The second output map. dstmap1type – Type of the first output map that should be `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` . nninterpolation – Flag indicating whether the fixed-point maps are used for the nearest-neighbor or for a more complex interpolation.

getAffineTransform

Calculates an affine transform from three pairs of the corresponding points.

C++: Mat `getAffineTransform`(InputArray src, InputArray dst)

C++: Mat `getAffineTransform`(const Point2f src[], const Point2f dst[])

Python: `cv2.getAffineTransform`(src, dst) → retval

C: CvMat* `cvGetAffineTransform`(const CvPoint2D32f* src, const CvPoint2D32f* dst, CvMat* map_matrix)

Python: `cv.GetAffineTransform`(src, dst, mapMatrix) → None

Parameters:

src – Coordinates of triangle vertices in the source image.

dst – Coordinates of the corresponding triangle vertices in the destination image.

The function calculates the matrix of an affine transform so that:

where

See also

`warpAffine()`, `transform()`

Parameters:	src – Coordinates of triangle vertices in the source image. dst – Coordinates of the corresponding triangle vertices in the destination image.

getPerspectiveTransform

Calculates a perspective transform from four pairs of the corresponding points.

C++: Mat `getPerspectiveTransform`(InputArray src, InputArray dst)

C++: Mat `getPerspectiveTransform`(const Point2f src[], const Point2f dst[])

Python: `cv2.getPerspectiveTransform`(src, dst) → retval

C: CvMat* `cvGetPerspectiveTransform`(const CvPoint2D32f* src, const CvPoint2D32f* dst, CvMat* map_matrix)

Python: `cv.GetPerspectiveTransform`(src, dst, mapMatrix) → None

Parameters:

src – Coordinates of quadrangle vertices in the source image.

dst – Coordinates of the corresponding quadrangle vertices in the destination image.

The function calculates the matrix of a perspective transform so that:

where

See also

`findHomography()`, `warpPerspective()`, `perspectiveTransform()`

Parameters:	src – Coordinates of quadrangle vertices in the source image. dst – Coordinates of the corresponding quadrangle vertices in the destination image.

getRectSubPix

Retrieves a pixel rectangle from an image with sub-pixel accuracy.

C++: void `getRectSubPix`(InputArray image, Size patchSize, Point2f center, OutputArray patch, int patchType=-1 )

Python: `cv2.getRectSubPix`(image, patchSize, center[, patch[, patchType]]) → patch

C: void `cvGetRectSubPix`(const CvArr* src, CvArr* dst, CvPoint2D32f center)

Python: `cv.GetRectSubPix`(src, dst, center) → None

Parameters:

src – Source image.

patchSize – Size of the extracted patch.

center – Floating point coordinates of the center of the extracted rectangle within the source image. The center must be inside the image.

dst – Extracted patch that has the size `patchSize` and the same number of channels as `src` .

patchType – Depth of the extracted pixels. By default, they have the same depth as `src` .

The function `getRectSubPix` extracts pixels from `src` :

where the values of the pixels at non-integer coordinates are retrieved using bilinear interpolation. Every channel of multi-channel images is processed independently. While the center of the rectangle must be inside the image, parts of the rectangle may be outside. In this case, the replication border mode (see `borderInterpolate()` ) is used to extrapolate the pixel values outside of the image.

See also

`warpAffine()`, `warpPerspective()`

Parameters:	src – Source image. patchSize – Size of the extracted patch. center – Floating point coordinates of the center of the extracted rectangle within the source image. The center must be inside the image. dst – Extracted patch that has the size `patchSize` and the same number of channels as `src` . patchType – Depth of the extracted pixels. By default, they have the same depth as `src` .

getRotationMatrix2D

Calculates an affine matrix of 2D rotation.

C++: Mat `getRotationMatrix2D`(Point2f center, double angle, double scale)

Python: `cv2.getRotationMatrix2D`(center, angle, scale) → retval

C: CvMat* `cv2DRotationMatrix`(CvPoint2D32f center, double angle, double scale, CvMat* map_matrix)

Python: `cv.GetRotationMatrix2D`(center, angle, scale, mapMatrix) → None

Parameters:

center – Center of the rotation in the source image.

angle – Rotation angle in degrees. Positive values mean counter-clockwise rotation (the coordinate origin is assumed to be the top-left corner).

scale – Isotropic scale factor.

map_matrix – The output affine transformation, 2x3 floating-point matrix.

The function calculates the following matrix:

where

The transformation maps the rotation center to itself. If this is not the target, adjust the shift.

See also

`getAffineTransform()`, `warpAffine()`, `transform()`

Parameters:	center – Center of the rotation in the source image. angle – Rotation angle in degrees. Positive values mean counter-clockwise rotation (the coordinate origin is assumed to be the top-left corner). scale – Isotropic scale factor. map_matrix – The output affine transformation, 2x3 floating-point matrix.

invertAffineTransform

Inverts an affine transformation.

C++: void `invertAffineTransform`(InputArray M, OutputArray iM)

Python: `cv2.invertAffineTransform`(M[, iM]) → iM

Parameters:

M – Original affine transformation.

iM – Output reverse affine transformation.

The function computes an inverse affine transformation represented by matrix `M` :

The result is also a matrix of the same type as `M` .

Parameters:	M – Original affine transformation. iM – Output reverse affine transformation.

LinearPolar

Remaps an image to polar space.

C: void `cvLinearPolar`(const CvArr* src, CvArr* dst, CvPoint2D32f center, double maxRadius, int flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS )

Parameters:

src – Source image

dst – Destination image. It will have same size and type as src.

center – The transformation center;

maxRadius – The radius of the bounding circle to transform. It determines the inverse magnitude scale parameter too. See below

flags –
A combination of interpolation methods and the following optional flags:

CV_WARP_FILL_OUTLIERS fills all of the destination image pixels. If some of them correspond to outliers in the source image, they are set to zero

CV_WARP_INVERSE_MAP See below

The function `cvLinearPolar` transforms the source image using the following transformation:

Forward transformation (`CV_WARP_INVERSE_MAP` is not set):

Inverse transformation (`CV_WARP_INVERSE_MAP` is set):

where

and

Note

The function can not operate in-place.

To calculate magnitude and angle in degrees `cvCartToPolar()` is used internally thus angles are measured from 0 to 360 with accuracy about 0.3 degrees.

An example using the LinearPolar operation can be found at opencv_source_code/samples/c/polar_transforms.c

Parameters:	src – Source image dst – Destination image. It will have same size and type as src. center – The transformation center; maxRadius – The radius of the bounding circle to transform. It determines the inverse magnitude scale parameter too. See below flags – A combination of interpolation methods and the following optional flags: CV_WARP_FILL_OUTLIERS fills all of the destination image pixels. If some of them correspond to outliers in the source image, they are set to zero CV_WARP_INVERSE_MAP See below

LogPolar

Remaps an image to log-polar space.

C: void `cvLogPolar`(const CvArr* src, CvArr* dst, CvPoint2D32f center, double M, int flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS )

Python: `cv.LogPolar`(src, dst, center, M, flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS) → None

Parameters:

src – Source image

dst – Destination image. It will have same size and type as src.

center – The transformation center; where the output precision is maximal

M – Magnitude scale parameter. It determines the radius of the bounding circle to transform too. See below

flags –
A combination of interpolation methods and the following optional flags:

CV_WARP_FILL_OUTLIERS fills all of the destination image pixels. If some of them correspond to outliers in the source image, they are set to zero

CV_WARP_INVERSE_MAP See below

The function `cvLogPolar` transforms the source image using the following transformation:

Forward transformation (`CV_WARP_INVERSE_MAP` is not set):

Inverse transformation (`CV_WARP_INVERSE_MAP` is set):

where

and

The function emulates the human “foveal” vision and can be used for fast scale and rotation-invariant template matching, for object tracking and so forth.

Note

The function can not operate in-place.

To calculate magnitude and angle in degrees `cvCartToPolar()` is used internally thus angles are measured from 0 to 360 with accuracy about 0.3 degrees.

An example using the geometric logpolar operation in 4 applications can be found at opencv_source_code/samples/cpp/logpolar_bsm.cpp

Parameters:	src – Source image dst – Destination image. It will have same size and type as src. center – The transformation center; where the output precision is maximal M – Magnitude scale parameter. It determines the radius of the bounding circle to transform too. See below flags – A combination of interpolation methods and the following optional flags: CV_WARP_FILL_OUTLIERS fills all of the destination image pixels. If some of them correspond to outliers in the source image, they are set to zero CV_WARP_INVERSE_MAP See below

remap

Applies a generic geometrical transformation to an image.

C++: void `remap`(InputArray src, OutputArray dst, InputArray map1, InputArray map2, int interpolation, int borderMode=BORDER_CONSTANT, const Scalar& borderValue=Scalar())

Python: `cv2.remap`(src, map1, map2, interpolation[, dst[, borderMode[, borderValue]]]) → dst

C: void `cvRemap`(const CvArr* src, CvArr* dst, const CvArr* mapx, const CvArr* mapy, int flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, CvScalar fillval=cvScalarAll(0) )

Python: `cv.Remap`(src, dst, mapx, mapy, flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, fillval=(0, 0, 0, 0)) → None

Parameters:

src – Source image.

dst – Destination image. It has the same size as `map1` and the same type as `src` .

map1 – The first map of either `(x,y)` points or just `x` values having the type `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` . See `convertMaps()` for details on converting a floating point representation to fixed-point for speed.

map2 – The second map of `y` values having the type `CV_16UC1` , `CV_32FC1` , or none (empty map if `map1` is `(x,y)` points), respectively.

interpolation – Interpolation method (see `resize()` ). The method `INTER_AREA` is not supported by this function.

borderMode – Pixel extrapolation method (see `borderInterpolate()` ). When `borderMode=BORDER_TRANSPARENT` , it means that the pixels in the destination image that corresponds to the “outliers” in the source image are not modified by the function.

borderValue – Value used in case of a constant border. By default, it is 0.

The function `remap` transforms the source image using the specified map:

where values of pixels with non-integer coordinates are computed using one of available interpolation methods. and can be encoded as separate floating-point maps in and respectively, or interleaved floating-point maps of in , or fixed-point maps created by using `convertMaps()` . The reason you might want to convert from floating to fixed-point representations of a map is that they can yield much faster (~2x) remapping operations. In the converted case, contains pairs `(cvFloor(x), cvFloor(y))` and contains indices in a table of interpolation coefficients.

This function cannot operate in-place.

Parameters:	src – Source image. dst – Destination image. It has the same size as `map1` and the same type as `src` . map1 – The first map of either `(x,y)` points or just `x` values having the type `CV_16SC2` , `CV_32FC1` , or `CV_32FC2` . See `convertMaps()` for details on converting a floating point representation to fixed-point for speed. map2 – The second map of `y` values having the type `CV_16UC1` , `CV_32FC1` , or none (empty map if `map1` is `(x,y)` points), respectively. interpolation – Interpolation method (see `resize()` ). The method `INTER_AREA` is not supported by this function. borderMode – Pixel extrapolation method (see `borderInterpolate()` ). When `borderMode=BORDER_TRANSPARENT` , it means that the pixels in the destination image that corresponds to the “outliers” in the source image are not modified by the function. borderValue – Value used in case of a constant border. By default, it is 0.

resize

Resizes an image.

C++: void resize(InputArray src, OutputArray dst, Size dsize, double fx=0, double fy=0, int interpolation=INTER_LINEAR )

Python: cv2.resize(src, dsize[, dst[, fx[, fy[, interpolation]]]]) → dst

C: void cvResize(const CvArr* src, CvArr* dst, int interpolation=CV_INTER_LINEAR )

Python: cv.Resize(src, dst, interpolation=CV_INTER_LINEAR) → None

Parameters:

Parameters:	src – input image. dst – output image; it has the size `dsize` (when it is non-zero) or the size computed from `src.size()`, `fx`, and `fy`; the type of `dst` is the same as of `src`. dsize – output image size; if it equals zero, it is computed as: Either `dsize` or both `fx` and `fy` must be non-zero. fx – scale factor along the horizontal axis; when it equals 0, it is computed as fy – scale factor along the vertical axis; when it equals 0, it is computed as interpolation – interpolation method: INTER_NEAREST - a nearest-neighbor interpolation INTER_LINEAR - a bilinear interpolation (used by default) INTER_AREA - resampling using pixel area relation. It may be a preferred method for image decimation, as it gives moire’-free results. But when the image is zoomed, it is similar to the `INTER_NEAREST`method. INTER_CUBIC - a bicubic interpolation over 4x4 pixel neighborhood INTER_LANCZOS4 - a Lanczos interpolation over 8x8 pixel neighborhood

src – input image.
dst – output image; it has the size dsize (when it is non-zero) or the size computed from src.size(), fx, and fy; the type of dst is the same as of src.
dsize –
output image size; if it equals zero, it is computed as:

Either dsize or both fx and fy must be non-zero.
fx –
scale factor along the horizontal axis; when it equals 0, it is computed as
fy –
scale factor along the vertical axis; when it equals 0, it is computed as
interpolation –
interpolation method:
- INTER_NEAREST - a nearest-neighbor interpolation
- INTER_LINEAR - a bilinear interpolation (used by default)
- INTER_AREA - resampling using pixel area relation. It may be a preferred method for image decimation, as it gives moire’-free results. But when the image is zoomed, it is similar to the INTER_NEARESTmethod.
- INTER_CUBIC - a bicubic interpolation over 4x4 pixel neighborhood
- INTER_LANCZOS4 - a Lanczos interpolation over 8x8 pixel neighborhood

The function resize resizes the image src down to or up to the specified size. Note that the initial dst type or size are not taken into account. Instead, the size and type are derived from the src,``dsize``,``fx`` , and fy . If you want to resize src so that it fits the pre-created dst , you may call the function as follows:

// explicitly specify dsize=dst.size(); fx and fy will be computed from that.
resize(src, dst, dst.size(), 0, 0, interpolation);

If you want to decimate the image by factor of 2 in each direction, you can call the function this way:

// specify fx and fy and let the function compute the destination image size.
resize(src, dst, Size(), 0.5, 0.5, interpolation);

To shrink an image, it will generally look best with CV_INTER_AREA interpolation, whereas to enlarge an image, it will generally look best with CV_INTER_CUBIC (slow) or CV_INTER_LINEAR (faster but still looks OK).

warpAffine

Applies an affine transformation to an image.

C++: void `warpAffine`(InputArray src, OutputArray dst, InputArray M, Size dsize, int flags=INTER_LINEAR, int borderMode=BORDER_CONSTANT, const Scalar& borderValue=Scalar())

Python: `cv2.warpAffine`(src, M, dsize[, dst[, flags[, borderMode[, borderValue]]]]) → dst

C: void `cvWarpAffine`(const CvArr* src, CvArr* dst, const CvMat* map_matrix, int flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, CvScalar fillval=cvScalarAll(0) )

Python: `cv.WarpAffine`(src, dst, mapMatrix, flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, fillval=(0, 0, 0, 0)) → None

C: void `cvGetQuadrangleSubPix`(const CvArr* src, CvArr* dst, const CvMat* map_matrix)

Python: `cv.GetQuadrangleSubPix`(src, dst, mapMatrix) → None

Parameters:

src – input image.

dst – output image that has the size `dsize` and the same type as `src` .

M – transformation matrix.

dsize – size of the output image.

flags – combination of interpolation methods (see `resize()` ) and the optional flag `WARP_INVERSE_MAP` that means that `M` is the inverse transformation ( ).

borderMode – pixel extrapolation method (see `borderInterpolate()`); when `borderMode=BORDER_TRANSPARENT` , it means that the pixels in the destination image corresponding to the “outliers” in the source image are not modified by the function.

borderValue – value used in case of a constant border; by default, it is 0.

The function `warpAffine` transforms the source image using the specified matrix:

when the flag `WARP_INVERSE_MAP` is set. Otherwise, the transformation is first inverted with `invertAffineTransform()` and then put in the formula above instead of `M` . The function cannot operate in-place.

See also

`warpPerspective()`, `resize()`, `remap()`, `getRectSubPix()`, `transform()`

Note

`cvGetQuadrangleSubPix` is similar to `cvWarpAffine`, but the outliers are extrapolated using replication border mode.

Parameters:	src – input image. dst – output image that has the size `dsize` and the same type as `src` . M – transformation matrix. dsize – size of the output image. flags – combination of interpolation methods (see `resize()` ) and the optional flag `WARP_INVERSE_MAP` that means that `M` is the inverse transformation ( ). borderMode – pixel extrapolation method (see `borderInterpolate()`); when `borderMode=BORDER_TRANSPARENT` , it means that the pixels in the destination image corresponding to the “outliers” in the source image are not modified by the function. borderValue – value used in case of a constant border; by default, it is 0.

warpPerspective

Applies a perspective transformation to an image.

C++: void `warpPerspective`(InputArray src, OutputArray dst, InputArray M, Size dsize, int flags=INTER_LINEAR, int borderMode=BORDER_CONSTANT, const Scalar& borderValue=Scalar())

Python: `cv2.warpPerspective`(src, M, dsize[, dst[, flags[, borderMode[, borderValue]]]]) → dst

C: void `cvWarpPerspective`(const CvArr* src, CvArr* dst, const CvMat* map_matrix, int flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, CvScalar fillval=cvScalarAll(0) )

Python: `cv.WarpPerspective`(src, dst, mapMatrix, flags=CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS, fillval=(0, 0, 0, 0)) → None

Parameters:

src – input image.

dst – output image that has the size `dsize` and the same type as `src` .

M – transformation matrix.

dsize – size of the output image.

flags – combination of interpolation methods (`INTER_LINEAR` or `INTER_NEAREST`) and the optional flag `WARP_INVERSE_MAP`, that sets `M` as the inverse transformation ( ).

borderMode – pixel extrapolation method (`BORDER_CONSTANT` or `BORDER_REPLICATE`).

borderValue – value used in case of a constant border; by default, it equals 0.

The function `warpPerspective` transforms the source image using the specified matrix:

when the flag `WARP_INVERSE_MAP` is set. Otherwise, the transformation is first inverted with `invert()` and then put in the formula above instead of `M` . The function cannot operate in-place.

See also

`warpAffine()`, `resize()`, `remap()`, `getRectSubPix()`, `perspectiveTransform()`

Parameters:	src – input image. dst – output image that has the size `dsize` and the same type as `src` . M – transformation matrix. dsize – size of the output image. flags – combination of interpolation methods (`INTER_LINEAR` or `INTER_NEAREST`) and the optional flag `WARP_INVERSE_MAP`, that sets `M` as the inverse transformation ( ). borderMode – pixel extrapolation method (`BORDER_CONSTANT` or `BORDER_REPLICATE`). borderValue – value used in case of a constant border; by default, it equals 0.

initUndistortRectifyMap

Computes the undistortion and rectification transformation map.

C++: void `initUndistortRectifyMap`(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2)

Python: `cv2.initUndistortRectifyMap`(cameraMatrix, distCoeffs, R, newCameraMatrix, size, m1type[, map1[, map2]]) → map1, map2

C: void `cvInitUndistortRectifyMap`(const CvMat* camera_matrix, const CvMat* dist_coeffs, const CvMat* R, const CvMat* new_camera_matrix, CvArr* mapx, CvArr* mapy)

C: void `cvInitUndistortMap`(const CvMat* camera_matrix, const CvMat* distortion_coeffs, CvArr* mapx, CvArr* mapy)

Python: `cv.InitUndistortRectifyMap`(cameraMatrix, distCoeffs, R, newCameraMatrix, map1, map2) → None

Python: `cv.InitUndistortMap`(cameraMatrix, distCoeffs, map1, map2) → None

Parameters:

cameraMatrix – Input camera matrix  .

distCoeffs – Input vector of distortion coefficients  of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.

R – Optional rectification transformation in the object space (3x3 matrix). `R1` or `R2` , computed by `stereoRectify()` can be passed here. If the matrix is empty, the identity transformation is assumed. In `cvInitUndistortMap` R assumed to be an identity matrix.

newCameraMatrix – New camera matrix  .

size – Undistorted image size.

m1type – Type of the first output map that can be `CV_32FC1` or `CV_16SC2` . See `convertMaps()` for details.

map1 – The first output map.

map2 – The second output map.

The function computes the joint undistortion and rectification transformation and represents the result in the form of maps for `remap()` . The undistorted image looks like original, as if it is captured with a camera using the camera matrix `=newCameraMatrix` and zero distortion. In case of a monocular camera, `newCameraMatrix` is usually equal to `cameraMatrix` , or it can be computed by `getOptimalNewCameraMatrix()` for a better control over scaling. In case of a stereo camera, `newCameraMatrix` is normally set to `P1` or `P2` computed by `stereoRectify()` .

Also, this new camera is oriented differently in the coordinate space, according to `R` . That, for example, helps to align two heads of a stereo camera so that the epipolar lines on both images become horizontal and have the same y- coordinate (in case of a horizontally aligned stereo camera).

The function actually builds the maps for the inverse mapping algorithm that is used by `remap()` . That is, for each pixel  in the destination (corrected and rectified) image, the function computes the corresponding coordinates in the source image (that is, in the original image from camera). The following process is applied:

where  are the distortion coefficients.

In case of a stereo camera, this function is called twice: once for each camera head, after `stereoRectify()` , which in its turn is called after `stereoCalibrate()` . But if the stereo camera was not calibrated, it is still possible to compute the rectification transformations directly from the fundamental matrix using `stereoRectifyUncalibrated()` . For each camera, the function computes homography `H` as the rectification transformation in a pixel domain, not a rotation matrix `R` in 3D space. `R` can be computed from `H` as

where `cameraMatrix` can be chosen arbitrarily.

Parameters:	cameraMatrix – Input camera matrix . distCoeffs – Input vector of distortion coefficients of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed. R – Optional rectification transformation in the object space (3x3 matrix). `R1` or `R2` , computed by `stereoRectify()` can be passed here. If the matrix is empty, the identity transformation is assumed. In `cvInitUndistortMap` R assumed to be an identity matrix. newCameraMatrix – New camera matrix . size – Undistorted image size. m1type – Type of the first output map that can be `CV_32FC1` or `CV_16SC2` . See `convertMaps()` for details. map1 – The first output map. map2 – The second output map.

getDefaultNewCameraMatrix

Returns the default new camera matrix.

C++: Mat `getDefaultNewCameraMatrix`(InputArray cameraMatrix, Size imgsize=Size(), bool centerPrincipalPoint=false )

Python: `cv2.getDefaultNewCameraMatrix`(cameraMatrix[, imgsize[, centerPrincipalPoint]]) → retval

Parameters:

cameraMatrix – Input camera matrix.

imgsize – Camera view image size in pixels.

centerPrincipalPoint – Location of the principal point in the new camera matrix. The parameter indicates whether this location should be at the image center or not.

The function returns the camera matrix that is either an exact copy of the input `cameraMatrix` (when `centerPrinicipalPoint=false` ), or the modified one (when `centerPrincipalPoint=true`).

In the latter case, the new camera matrix will be:

where and are and elements of `cameraMatrix` , respectively.

By default, the undistortion functions in OpenCV (see `initUndistortRectifyMap()`, `undistort()`) do not move the principal point. However, when you work with stereo, it is important to move the principal points in both views to the same y-coordinate (which is required by most of stereo correspondence algorithms), and may be to the same x-coordinate too. So, you can form the new camera matrix for each view where the principal points are located at the center.

Parameters:	cameraMatrix – Input camera matrix. imgsize – Camera view image size in pixels. centerPrincipalPoint – Location of the principal point in the new camera matrix. The parameter indicates whether this location should be at the image center or not.

undistort

Transforms an image to compensate for lens distortion.

C++: void `undistort`(InputArray src, OutputArray dst, InputArray cameraMatrix, InputArray distCoeffs, InputArray newCameraMatrix=noArray() )

Python: `cv2.undistort`(src, cameraMatrix, distCoeffs[, dst[, newCameraMatrix]]) → dst

C: void `cvUndistort2`(const CvArr* src, CvArr* dst, const CvMat* camera_matrix, const CvMat* distortion_coeffs, const CvMat* new_camera_matrix=0 )

Python: `cv.Undistort2`(src, dst, cameraMatrix, distCoeffs) → None

Parameters:

src – Input (distorted) image.

dst – Output (corrected) image that has the same size and type as `src` .

cameraMatrix – Input camera matrix  .

distCoeffs – Input vector of distortion coefficients  of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.

newCameraMatrix – Camera matrix of the distorted image. By default, it is the same as `cameraMatrix` but you may additionally scale and shift the result by using a different matrix.

The function transforms an image to compensate radial and tangential lens distortion.

The function is simply a combination of `initUndistortRectifyMap()` (with unity `R` ) and `remap()` (with bilinear interpolation). See the former function for details of the transformation being performed.

Those pixels in the destination image, for which there is no correspondent pixels in the source image, are filled with zeros (black color).

A particular subset of the source image that will be visible in the corrected image can be regulated by `newCameraMatrix` . You can use `getOptimalNewCameraMatrix()` to compute the appropriate `newCameraMatrix` depending on your requirements.

The camera matrix and the distortion parameters can be determined using `calibrateCamera()` . If the resolution of images is different from the resolution used at the calibration stage,  and  need to be scaled accordingly, while the distortion coefficients remain the same.

Parameters:	src – Input (distorted) image. dst – Output (corrected) image that has the same size and type as `src` . cameraMatrix – Input camera matrix . distCoeffs – Input vector of distortion coefficients of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed. newCameraMatrix – Camera matrix of the distorted image. By default, it is the same as `cameraMatrix` but you may additionally scale and shift the result by using a different matrix.

undistortPoints

Computes the ideal point coordinates from the observed point coordinates.

C++: void undistortPoints(InputArray src, OutputArray dst, InputArray cameraMatrix, InputArray distCoeffs, InputArray R=noArray(), InputArray P=noArray())

Python: cv2.undistortPoints(src, cameraMatrix, distCoeffs[, dst[, R[, P]]]) → dst

C: void cvUndistortPoints(const CvMat* src, CvMat* dst, const CvMat* camera_matrix, const CvMat* dist_coeffs, const CvMat* R=0, const CvMat* P=0 )

Python: cv.UndistortPoints(src, dst, cameraMatrix, distCoeffs, R=None, P=None) → None

Parameters:

Parameters:	src – Observed point coordinates, 1xN or Nx1 2-channel (CV_32FC2 or CV_64FC2). dst – Output ideal point coordinates after undistortion and reverse perspective transformation. If matrix `P` is identity or omitted, `dst` will contain normalized point coordinates. cameraMatrix – Camera matrix . distCoeffs – Input vector of distortion coefficients of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed. R – Rectification transformation in the object space (3x3 matrix). `R1` or `R2` computed by `stereoRectify()` can be passed here. If the matrix is empty, the identity transformation is used. P – New camera matrix (3x3) or new projection matrix (3x4). `P1` or `P2` computed by `stereoRectify()` can be passed here. If the matrix is empty, the identity new camera matrix is used.

src – Observed point coordinates, 1xN or Nx1 2-channel (CV_32FC2 or CV_64FC2).
dst – Output ideal point coordinates after undistortion and reverse perspective transformation. If matrix P is identity or omitted, dst will contain normalized point coordinates.
cameraMatrix – Camera matrix .
distCoeffs – Input vector of distortion coefficients of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.
R – Rectification transformation in the object space (3x3 matrix). R1 or R2 computed by stereoRectify() can be passed here. If the matrix is empty, the identity transformation is used.
P – New camera matrix (3x3) or new projection matrix (3x4). P1 or P2 computed by stereoRectify() can be passed here. If the matrix is empty, the identity new camera matrix is used.

The function is similar to undistort() and initUndistortRectifyMap() but it operates on a sparse set of points instead of a raster image. Also the function performs a reverse transformation to projectPoints() . In case of a 3D object, it does not reconstruct its 3D coordinates, but for a planar object, it does, up to a translation vector, if the proper R is specified.

// (u,v) is the input point, (u', v') is the output point
// camera_matrix=[fx 0 cx; 0 fy cy; 0 0 1]
// P=[fx' 0 cx' tx; 0 fy' cy' ty; 0 0 1 tz]
x" = (u - cx)/fx
y" = (v - cy)/fy
(x',y') = undistort(x",y",dist_coeffs)
[X,Y,W]T = R*[x' y' 1]T
x = X/W, y = Y/W
// only performed if P=[fx' 0 cx' [tx]; 0 fy' cy' [ty]; 0 0 1 [tz]] is specified
u' = x*fx' + cx'
v' = y*fy' + cy',

where undistort() is an approximate iterative algorithm that estimates the normalized original point coordinates out of the normalized distorted point coordinates (“normalized” means that the coordinates do not depend on the camera matrix).

The function can be used for both a stereo camera head or a monocular camera (when R is empty).

Softmax function

2020-01-21T12:10:00.001-08:00

A wonderful activation function that turns numbers aka logits into probabilities that sum to one.
Outputs a vector that represents the probability distributions of a list of potential outcomes
Core element that is used in deep learning task

Lets predict a model to see if we receive a gift or not?

MODEL PREDICTIONS
The probability that we get a gift is 0.8
The probability that we would not receive a gift is 0.2

JOB OF MODEL
Based on the existing features (Birthdate,year) it calculates the linear model which would be the score
Then the probability that we get a gift or not is the sigmoid function applied to that score

MODEL WHICH WOULD TELL US WHICH ANIMAL WHAT WE SAW

Notice that the probability of the model needs to add to 1.

Assuming we have a linear model based on some inputs, We calculate linear function based on inputs like if it has a feather, teeth etc
After calculating linear functions based on these inputs , assuming that we get some scores

Note:

We need to change the scores to 1. (Requirement for probability)
The probability of duck is higher then beaver and the beaver is higher then walrus

We can take the sum of scores and divide it by zero

The probability would be 2/3 for the duck 1/3 for the beaver and 0 for the walrus

NOTE:
We need to store these scores into positive scores. This can be achieved using exponential function

It is called as softmax function

matplotlib and pyplot

2020-01-21T12:04:00.002-08:00

Startup commands

First, let’s start IPython. It is a most excellent enhancement to the standard Python prompt, and it ties in especially well with Matplotlib. Start IPython either at a shell, or the IPython Notebook now.

With IPython started, we now need to connect to a GUI event loop. This tells IPython where (and how) to display plots. To connect to a GUI loop, execute the %matplotlib magic at your IPython prompt. There’s more detail on exactly what this does at IPython’s documentation on GUI event loops.

If you’re using IPython Notebook, the same commands are available, but people commonly use a specific argument to the %matplotlib magic:

In [1]: %matplotlib inline

This turns on inline plotting, where plot graphics will appear in your notebook. This has important implications for interactivity. For inline plotting, commands in cells below the cell that outputs a plot will not affect the plot. For example, changing the color map is not possible from cells below the cell that creates a plot. However, for other backends, such as qt4, that open a separate window, cells below those that create the plot will change the plot - it is a live object in memory.

This tutorial will use matplotlib’s imperative-style plotting interface, pyplot. This interface maintains global state, and is very useful for quickly and easily experimenting with various plot settings. The alternative is the object-oriented interface, which is also very powerful, and generally more suitable for large application development. If you’d like to learn about the object-oriented interface, a great place to start is our FAQ on usage. For now, let’s get on with the imperative-style approach:

In [2]: import matplotlib.pyplot as plt
In [3]: import matplotlib.image as mpimg
In [4]: import numpy as np

Importing image data into Numpy arrays

Loading image data is supported by the Pillow library. Natively, matplotlib only supports PNG images. The commands shown below fall back on Pillow if the native read fails.

The image used in this example is a PNG file, but keep that Pillow requirement in mind for your own data.

Here’s the image we’re going to play with:

It’s a 24-bit RGB PNG image (8 bits for each of R, G, B). Depending on where you get your data, the other kinds of image that you’ll most likely encounter are RGBA images, which allow for transparency, or single-channel grayscale (luminosity) images. You can right click on it and choose “Save image as” to download it to your computer for the rest of this tutorial.

And here we go...

In [5]: img=mpimg.imread('stinkbug.png')
Out[5]:
array([[[ 0.40784314,  0.40784314,  0.40784314],
        [ 0.40784314,  0.40784314,  0.40784314],
        [ 0.40784314,  0.40784314,  0.40784314],
        ...,
        [ 0.42745098,  0.42745098,  0.42745098],
        [ 0.42745098,  0.42745098,  0.42745098],
        [ 0.42745098,  0.42745098,  0.42745098]],

       ...,
       [[ 0.44313726,  0.44313726,  0.44313726],
        [ 0.4509804 ,  0.4509804 ,  0.4509804 ],
        [ 0.4509804 ,  0.4509804 ,  0.4509804 ],
        ...,
        [ 0.44705883,  0.44705883,  0.44705883],
        [ 0.44705883,  0.44705883,  0.44705883],
        [ 0.44313726,  0.44313726,  0.44313726]]], dtype=float32)

Note the dtype there - float32. Matplotlib has rescaled the 8 bit data from each channel to floating point data between 0.0 and 1.0. As a side note, the only datatype that Pillow can work with is uint8. Matplotlib plotting can handle float32 and uint8, but image reading/writing for any format other than PNG is limited to uint8 data. Why 8 bits? Most displays can only render 8 bits per channel worth of color gradation. Why can they only render 8 bits/channel? Because that’s about all the human eye can see. More here (from a photography standpoint): Luminous Landscape bit depth tutorial.

Each inner list represents a pixel. Here, with an RGB image, there are 3 values. Since it’s a black and white image, R, G, and B are all similar. An RGBA (where A is alpha, or transparency), has 4 values per inner list, and a simple luminance image just has one value (and is thus only a 2-D array, not a 3-D array). For RGB and RGBA images, matplotlib supports float32 and uint8 data types. For grayscale, matplotlib supports only float32. If your array data does not meet one of these descriptions, you need to rescale it.

Plotting numpy arrays as images

So, you have your data in a numpy array (either by importing it, or by generating it). Let’s render it. In Matplotlib, this is performed using the imshow() function. Here we’ll grab the plot object. This object gives you an easy way to manipulate the plot from the prompt.

In [6]: imgplot = plt.imshow(img)

(Source code, png, pdf)

You can also plot any numpy array.

Applying pseudocolor schemes to image plots

Pseudocolor can be a useful tool for enhancing contrast and visualizing your data more easily. This is especially useful when making presentations of your data using projectors - their contrast is typically quite poor.

Pseudocolor is only relevant to single-channel, grayscale, luminosity images. We currently have an RGB image. Since R, G, and B are all similar (see for yourself above or in your data), we can just pick one channel of our data:

In [7]: lum_img = img[:,:,0]

This is array slicing. You can read more in the Numpy tutorial.

In [8]: plt.imshow(lum_img)

(Source code, png, pdf)

Now, with a luminosity (2D, no color) image, the default colormap (aka lookup table, LUT), is applied. The default is called viridis. There are plenty of others to choose from.

In [9]: plt.imshow(lum_img, cmap="hot")

(Source code, png, pdf)

Note that you can also change colormaps on existing plot objects using the set_cmap() method:

In [10]: imgplot = plt.imshow(lum_img)
In [11]: imgplot.set_cmap('nipy_spectral')

(Source code, png, pdf)

Note

However, remember that in the IPython notebook with the inline backend, you can’t make changes to plots that have already been rendered. If you create imgplot here in one cell, you cannot call set_cmap() on it in a later cell and expect the earlier plot to change. Make sure that you enter these commands together in one cell. plt commands will not change plots from earlier cells.

There are many other colormap schemes available. See the list and images of the colormaps.

Color scale reference

It’s helpful to have an idea of what value a color represents. We can do that by adding color bars.

In [12]: imgplot = plt.imshow(lum_img)
In [13]: plt.colorbar()

(Source code, png, pdf)

This adds a colorbar to your existing figure. This won’t automatically change if you change you switch to a different colormap - you have to re-create your plot, and add in the colorbar again.

Examining a specific data range

Sometimes you want to enhance the contrast in your image, or expand the contrast in a particular region while sacrificing the detail in colors that don’t vary much, or don’t matter. A good tool to find interesting regions is the histogram. To create a histogram of our image data, we use the hist()function.

In [14]: plt.hist(lum_img.ravel(), bins=256, range=(0.0, 1.0), fc='k', ec='k')

(Source code, png, pdf)

Most often, the “interesting” part of the image is around the peak, and you can get extra contrast by clipping the regions above and/or below the peak. In our histogram, it looks like there’s not much useful information in the high end (not many white things in the image). Let’s adjust the upper limit, so that we effectively “zoom in on” part of the histogram. We do this by passing the clim argument to imshow. You could also do this by calling the set_clim() method of the image plot object, but make sure that you do so in the same cell as your plot command when working with the IPython Notebook - it will not change plots from earlier cells.

In [15]: imgplot = plt.imshow(lum_img, clim=(0.0, 0.7))

(Source code, png, pdf)

Array Interpolation schemes

Interpolation calculates what the color or value of a pixel “should” be, according to different mathematical schemes. One common place that this happens is when you resize an image. The number of pixels change, but you want the same information. Since pixels are discrete, there’s missing space. Interpolation is how you fill that space. This is why your images sometimes come out looking pixelated when you blow them up. The effect is more pronounced when the difference between the original image and the expanded image is greater. Let’s take our image and shrink it. We’re effectively discarding pixels, only keeping a select few. Now when we plot it, that data gets blown up to the size on your screen. The old pixels aren’t there anymore, and the computer has to draw in pixels to fill that space.

We’ll use the Pillow library that we used to load the image also to resize the image.

In [16]: from PIL import Image
In [17]: img = Image.open('../_static/stinkbug.png')
In [18]: img.thumbnail((64, 64), Image.ANTIALIAS) # resizes image in-place
In [19]: imgplot = plt.imshow(img)

(Source code, png, pdf)

Here we have the default interpolation, bilinear, since we did not give imshow() any interpolation argument.

Let’s try some others:

In [20]: imgplot = plt.imshow(img, interpolation="nearest")

(Source code, png, pdf)

In [21]: imgplot = plt.imshow(img, interpolation="bicubic")

(Source code, png, pdf)

Bicubic interpolation is often used when blowing up photos - people tend to prefer blurry over pixelated.

Rolling average

2020-01-21T11:58:00.003-08:00

GRAPH SHOWING MOVING AVERAGE FOR THE 7 DAY PRECEDING AVERAGE.

With a 7-day preceding average line like this:

The Big Idea

Our first graph above is pretty noisy and hard to get useful information from. We can smooth it out by plotting a 7-day average on top of the underlying data. This can be done with window functions, self-joins, or correlated subqueries - we’ll cover the first two.

We’ll start with a preceding average, which means that the average point on the 7th of the month is the average of the first seven days.

Visually this shifts the spikes in the graph to the right, as a big spike is averaged over the following seven days.

First, Create an Intermediate Count Table

We want to compute an average over the total signups for each day. Assuming we have a typical users table with a row per new user and a timestamp created_at, we can create our aggregate our signups table like so:

select

created_at::date as date,

count(1) as value

from new_customers

group by 1

In Postgres and SQL Server you can use this as a CTE. In MySQL you can save it as a temporary table.

Postgres Rolling Average

Fortunately Postgres has window functions which are the simplest way to compute a running average.

select

date,

value,

avg(value)

over (order by date asc

rows between 6 preceding and current row) as avg,

from signups

order by 1 desc

This query assumes that the dates do not have gaps. The query is averaging over the past seven rows, not the past seven dates. If your data has gaps, fill them in with generate_series or joining against a table with dense date rows.

MySQL Rolling Average

MySQL lacks window functions, but we can do a similar computation using self-joins. For each row in our count table, we join every row that was within the past seven days and take the average.

select signups.date, signups.count, avg(signups_past.count)

from signups

join signups as signups_past

on signups_past.date between signups.date - 6 and signups.date

group by 1, 2

This query automatically handles date gaps, as we are looking at rows within a date range rather than the preceding N rows.

SQL Server Rolling Average

SQL Server has window functions, so computing the rolling average can be done in either the Postgres style or MySQL style. For simplicity, we’re using the MySQL version with a self join.

This is conceptually the same as in MySQL. The only translations are the dateadd function and explicitly named group by columns.

select signups.date, signups.count, avg(signups_past.count)

from signups

join signups as signups_past

on signups_past.date

between dateadd(day, -6, signups.date) and signups.date

group by signups.date, signups.count

Other Averages

We focused on the 7-day trailing average in this post. If we wanted to look at the 7-day leading average, it’s as simple as sorting the dates in the other direction. If we wanted to look at a centered average, we’d use:

Postgres: rows between 3 preceding and 3 following
MySql: between signups.date - 3 and signups.date + 3 in MySQL
SQL Server: between dateadd(day, -3, signups.date) and dateadd(day, 3, signups.date)

Difference between AWS glue and Hive warehouse

2020-01-21T11:50:00.003-08:00

Apache Hive vs AWS Glue: What are the differences?

Apache Hive: Data Warehouse Software for Reading, Writing, and Managing Large Datasets. Hive facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage; AWS Glue:Fully managed extract, transform, and load (ETL) service. A fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics.

Apache Hive and AWS Glue can be primarily classified as "Big Data" tools.

Some of the features offered by Apache Hive are:

Built on top of Apache Hadoop
Tools to enable easy access to data via SQL
Support for extract/transform/load (ETL), reporting, and data analysis

On the other hand, AWS Glue provides the following key features:

Easy - AWS Glue automates much of the effort in building, maintaining, and running ETL jobs. AWS Glue crawls your data sources, identifies data formats, and suggests schemas and transformations. AWS Glue automatically generates the code to execute your data transformations and loading processes.
Integrated - AWS Glue is integrated across a wide range of AWS services.
Serverless - AWS Glue is serverless. There is no infrastructure to provision or manage. AWS Glue handles provisioning, configuration, and scaling of the resources required to run your ETL jobs on a fully managed, scale-out Apache Spark environment. You pay only for the resources used while your jobs are running.

Apache Hive is an open source tool with 2.62K GitHub stars and 2.58K GitHub forks. Here's a link to Apache Hive's open source repository on GitHub.

According to the StackShare community, Apache Hive has a broader approval, being mentioned in 27 company stacks & 12developers stacks; compared to AWS Glue, which is listed in 13 company stacks and 7 developer stacks.

Lag and lead in sql

2020-01-21T11:48:00.000-08:00

LEAD() and LAG() Function

The LEAD() and LAG() function in MySQL are used to get preceding and succeeding value of any row within its partition. These functions are termed as nonaggregate Window functions.

The Window functions are those functions which perform operations for each row of the partition or window. These functions produce the result for each query row unlikely to the aggregate functions that group them and results in a single row.

The row on which operation occur is termed as current row.
The set of rows which are realted to current row or using which function opaerates on current row is termed as Window.

The LAG() function is used to get value from row that precedes the current row.

The LEAD() function is used to get value from row that succedes the current row.

Syntax:
For LEAD() function-

LEAD(expr, N, default) 
          OVER (Window_specification | Window_name)

For LAG() function-

LAG(expr, N, default) 
          OVER (Window_specification | Window_name)

The N and default argument in the function is optional.

Parameters used:

expr: It can be a column or any bulit-in function.
N: It is a positive value which determine number of rows preceding/succeeding the current row. If it is omitted in query then its default value is 1.
default: It is the default value return by function in-case no row precedes/succeedes the current row by N rows. If it is missing then it is by default NULL.
OVER(): It defines how rows are partitioned into groups. If OVER() is empty then function compute result using all rows.
Window_specification: It consist of query partition clause which determines how the query rows are partitioned and ordered.
Window_name: If window is specified elsewhere in the query then it is referenced using this Window_name.

AZURE BLOB STORAGE

2020-01-21T11:44:00.002-08:00

Service for storing large amount of unstructured object data, such as text or binary data
Expose data publicly to the world, or store application data privately
Common uses of blob storage

Serving images or documents directly to a browser
Storing files for distributed access
Storing data for backup and restore, disaster, recovery and archiving

STORAGE ACCOUNT

All access to storage is done through a storage account.
This account can be a General purpose or a blob storage.

CONTAINER

A container provides a grouping of a set of blobs
All blobs must be in a container
An account can contain an unlimited number of containers
A container can store an unlimited no of blobs

BLOB

A file of any type and size
Azure storage offers 3 types of blobs

CREATION OF BLOBS

Create a storage account
Create a container
Create blobs

TYPES OF BLOB
BLOCK BLOBS
Each blob can be of different size, up to max of 100 mb
used for stoing text and binary

Spark Executors and Drivers

2020-01-21T11:35:00.001-08:00

SPARK CONSISTS OF DRIVER PROGRAM

Our code that defines which datasources to consume from and which data sources to consume to which is driving all our execution
Responsible for getting the spark context
It would understand how to communicate with the cluster manager.(Yarn mesos or stand-alone)

CLUSTER MANAGER

Requests resources which are the worker node. This worker node contains executor
Driver is acting as a scheduler

EXECUTOR

Its a jvm, which can spawn different tasks
These tasks performs specific tasks what the driver tells them

EXECUTOR MEMORY (SPARK CACHE)

Each executor carries a memory that it keeps as cache. This is known as spark cache
There is a portion of jvm memory inside the executor

Note: Spark data off heap, so outside the jvm

DRIVER

Requests resources from a scheduler (YARN)
Once the program gets the resources it needs in the form of executors. Its the job of the driver to schedule the execution of tasks that it sends to the executors

SCHEDULING

There are 2 types of scheduler

CLUSTER MANAGER SCHEDULER (YARN)

Responsible for responding to resource requests for executors.

SPARK SCHEDULER(Driver)

Responsible translating transformation and actions into a DAG

CLUSTER MODE

There are 2 types of cluster modes

1. CLIENT MODE

When submitted as client mode, the machine or the client host that you are submitting your application from actually hosts the driver code
Their requests are made to yarn to submit the application and request the resources and then YARN creates an application master and containers that would communicate back to the application master, which the becomes aware of.
Host is responsible for the livelihood of the driver. Driver goes down then the whole application goes down.
Availability depends on client
Interactive (Spark Shell/REPL)

2. CLUSTER MODE

Driver itself (spark scheduler) is introduced as a container within your cluster manager
Driver actually leaves within the boundary of application master
It communicates with executors with other containers just as it does in the client mode
Cluster manger manages drivers livelihood
Better availability guarantees
Non interactive

CONTAINER

A container is an allocation or share of memory and cpu.
One job may need multiple containers
Containers will be allocated across the cluster depending upon the availability
The tasks will be executed inside the container

PARTITIONS

Each of these containers is spark executor. This carries different tasks and cache
Spark splits out its body of work that it creates partitions that it has to work on.
Every partitions consumes one of this task slots (similar to input splits)

APPLICATION MASTER

Stand alone application that YARN runs in a YARN container to manage spark application running in a YARN cluster
Framework specific library tasked with negotiating cluster resources from the YARN ResourceManager and working with YARN nodemanger to execute and monitor the task.

MASTER IN SPARKSESSION

The --master option specifies the master URL for a distributed cluster, or local to run locally with one thread, or local[N] to run locally with N threads. You should start by using local for testing.

And from here:

import org.apache.spark.sql.SparkSession
object Ingest {
  // Main Method
  def main(args: Array[String])
  {

     val spark = SparkSession
      .builder()
       .master("local[2]")
      .appName("Spark SQL basic example").getOrCreate()

   }
}

local : Run Spark locally with one worker thread (i.e. no parallelism at all).

local[K] : Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).

local[K,F] : Run Spark locally with K worker threads and F maxFailures (see spark.task.maxFailures for an explanation of this variable)

local[*] : Run Spark locally with as many worker threads as logical cores on your machine.

local[*,F] : Run Spark locally with as many worker threads as logical cores on your machine and F maxFailures.

spark://HOST:PORT : Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default.

spark://HOST1:PORT1,HOST2:PORT2 : Connect to the given Spark standalone cluster with standby masters with Zookeeper. The list must have all the master hosts in the high availability cluster set up with Zookeeper. The port must be whichever each master is configured to use, which is 7077 by default.

mesos://HOST:PORT : Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk://.... To submit with --deploy-mode cluster, the HOST:PORT should be configured to connect to the MesosClusterDispatcher.

yarn : Connect to a YARN cluster in client or cluster mode depending on the value of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable

YARN NODE MANAGER

YARN RESOURCE MANAGER

Loading facts in ETL

2020-01-21T11:30:00.000-08:00

LOADING FACTS
The most important thing about loading fact tables is that first you need to load dimension tables and then according to the specification the fact tables.

The fact table is often located in the center of a star schema, surrounded by dimension tables. It has two types of columns: those containing facts and other containing foreign keys to dimension tables.

FACT TABLE CONSISTS OF:
Measurements: additive-measures that can be added across all dimensions , non-additive cannot be added and semi-additive can be added across few dimensions.

METRICS
Facts - The fact table has frequently more than one measurement field and then each field is called a fact. Usually one fact table has at least three dimension tables.
Fact table granulatity
First point is creating the grain of a table.

GRAIN

The grain is simply a definition of a measurement event and allows dimension tables to fallow appropriate fact table. The connection between both appears through the foreign keys in the fact table and provide the context of the measurements. Every fact table should have only one grain(two or more are not recommended).

TYPES OF FACT TABLE
Transaction grain. Use only if the measurement events take place. It is for instantaneous actions. Those tables are the largest of all three types.
Periodic snapshot. Use for long tracking processes like bank account. It is a measurement at a specific point of time , regularly repeated and it refers to the period of time in the dimension table. The most common is monthly snapshot.
Accumulating snapshot. Use for processes that have defined beginning and end. It shows a status in every moment we need.
Preserving integrity
To preserve the integrity all foreign keys references should be correct and known. However there are two situations when this integrity may be broken. It is when we load fact tables with incorrect foreign key or when we delete a dimension with a related primary key. Then it is impossible to make a connection between those tables and we may loose some very important relationships.

Any query that references this relationship will fail. To avoid this problem you should analyze the loading phase in every step from preparation , through loading and after loading by scanning the fact table and looking for bad foreign keys.

Next point is checking the surrogate keys. When we load fact table we assume that all records are current. This operation refers to a surrogate key pipeline where the source data for the fact table contains natural keys that must be replaced by surrogate keys before the records are loaded. They are found through the line of little lookup surrogate tables which are updated every time a new dimension entity is created or changed.

Only with appropriate current surrogate key we may connect to a fact table with good results. However the lookup surrogate tables are not effective if we are handling with late arriving records. Then the dimension is a source of the surrogate key.

In the beginning of this article I have mentioned about the primary and surrogate key of the dimension table. Now we will look closer at those two elements considering two cases: the current database type and the temporary database type.

A current database stores only current data and the relationship between the surrogate and primary key is one to one so the surrogate key may be used as a primary key.

In the temporary database this relationship is many to one and then we cannot make this replacement.

The database should reflect the data of its source to provide appropriate business activity. In other words all data should be current and correct.

HANDLING INCORRECT DATA

Negate the fact (do nothing)
Update the fact (overwrite)
Delete and remove the fact - the most common way. We may delete facts physically and logically. When deleted physically we just delete permanently unwanted record while in logically deleting we create a delete column for unwanted records. The first one is easier but the second is much safer and most widely used.
Factless fact tables
Now we will say shortly about some other points of loading fact tables.

The fact less fact table is a table that contains only dimensional keys without any measured values.

TYPES OF FACT TABLE:

First type is used for capturing the event for information not calculation purposes. It is just an information about the even that happened.
Second used for support negative analysis report and guarantee coverage.
Multiple measures units
Sometimes the number of measurements is so big that presenting them may be quite inconvenient. To resolve this problem we use special units of measure in a fact table.

The currencies
In many databases which influence business decisions there is a need to express facts in many currencies. Then we create a foreign key in the fact table(like a sale fact table) which connect to the currency dimension table.

Late arriving facts
We locate those late arriving facts in historical position but we have to be very careful in choosing old contemporary dimension records. To make this operation appropriately you should follow these steps: First - find the corresponding dimension record, then replace the natural keys of late arriving facts with the surrogate key founded in dimension records and locate this late arriving fact into a correct place in the database where other fact tables from that time are stored.

Installing spark on windows

2020-01-21T11:23:00.000-08:00

https://www.youtube.com/watch?v=2CvtwKTjI4Q&vl=en

1) Download specific version of spark
http://spark.apache.org/downloads.html
2) Unzip and create a directory for spark

Wasb

2020-01-21T11:21:00.002-08:00

What is HDFS?

The Hadoop Distributed File System (HDFS) is one of the core Hadoop components, it is how Hadoop manages data and storage. At a high level, when you load a file into Hadoop the "name node" uses HDFS to chunk the file into blocks and it spreads those blocks of data across the worker nodes within the cluster. Each chunk of data is stored on multiple nodes (assuming the replication factor is set to > 1) for higher availability. The name node knows where each chunk of data is stored and that information is used by the job manager to allocate tasks and resources appropriately across nodes.

What is WASB?

Windows Azure Storage Blob (WASB) is an extension built on top of the HDFS APIs. The WASBS variation uses SSL certificates for improved security. It in many ways "is" HDFS. However, WASB creates a layer of abstraction that enables separation of storage. This separation is what enables your data to persist even when no clusters currently exist and enables multiple clusters plus other applications to access a single piece of data all at the same time. This increases functionality and flexibility while reducing costs and reducing the time from question to insight.

What is an Azure blob store, an Azure storage account, and an Azure container? For that matter, what is Azure again?

Azure is Microsoft's cloud solution. A cloud is essentially a collection of host data centers that you don't have to directly manage. You can request services from that cloud. For example, you can request virtual machines and storage, data services such as SQL Azure Database or HDInsight, or services such as Websites or Service Bus. In Azure you store blobs on containers within Azure storage accounts. You grant access to a storage account, you create collections at the container level, and you place blobs (files of any format) inside the containers. This illustration from Microsoft's documentation helps to show the structure:

How do I manage and configure block/chunk size and the replication factor with WASB?

You don't. It's not generally necessary. The data is stored in the Azure storage accounts, remaining accessible to many applications at once. Each blob (file) is replicated 3x within the data center. If you choose to use geo-replication on your account you also get 3 copies of the data in another data center within the same region. The data is chunked and distributed to nodes when a job is run. If you need to change the chunk size for memory related performance at run time that is still an option. You can pass in any Hadoop configuration parameter setting when you create the cluster or you can use the SET command for a given job.

Isn't one of the selling points of Hadoop that the data sits with the compute? How does that work with WASB?

Just like with any Hadoop system the data is loaded into memory on the individual nodes at compute time (when the job runs). The difference with WASB is that the data is loaded from the storage accounts instead of from local disks. Given the way Azure data center backbones are built the performance is generally the same or better than if you used disks locally attached to the VMs.

DStream vs RDD

2020-01-21T11:18:00.000-08:00

DSTREAM TRANSFORM

DSTREAM foreachRDD