Viking's Tech Blog

RDD in Spark | Features of RDD | Operations on RDD

2020-10-17T10:47:00.000-07:00

In the world of Big Data, Spark is a big name. Spark's in-memory computation overcomes the shortcomings of Hadoop's Map Reduce Architecture where data sharing is slow due to replication, serialization, and disk IO operations. Most of the Hadoop applications spend more than 90 percent of the time doing HDFS read and write operation. But Spark's in-memory computation comes at a cost: it requires high CPU utilization due to excessive memory usage.

What are RDDs in Spark ?

RDD or Resilient Distributed Datasets is a fundamental data structure of Spark. It is a read-only, partitioned collection of records. It can contain any type of python, scala, or java objects, including user-defined classes. This makes Spark more flexible in terms of accessing RDDs in different languages where scala is the official language of Spark. RDDs are capable of processing structured, semi-structured, and unstructured data.

How to create an RDD ?

There are two ways to create an RDD

Parallelizing an existing collection in your driver program
> var rdd1 = spark.sparkContext.parallelize(1 to 10).collect()
Referencing a dataset in an external storage system, such as a shared file system, HDFS, HBase, or any data source offering a Hadoop input/output format.
> var rdd2 = spark.sparkContext.textFile("file_path/data.txt")

Features of RDD

In-memory Computation - The data is kept in a Random Access Memory (RAM) instead of disk drives like Hadoop. This reduces the cost of memory to process, compute, and analyze the data but requires a significantly large amount of RAM for storing huge data. This increases infrastructural cost since RAMs are much costlier than disk drives.

Lazy Evaluation - In lazy evaluation, execution does not start until we trigger it. RDD transformations are lazy in nature. They keep a record of which operation is being called on the RDD. These operations can be executed anytime by calling an action on the RDD.

Distributed Environment - RDDs are divided into logical partitions. These partitions may be computed on different nodes of the cluster which enables parallel processing. Parallel processing helps to perform different tasks simultaneously, therefore increasing the speed of operation, provide optimization by reducing the number of queries.

Fault Tolerance - RDDs are highly resilient since the same data chunks are replicated across multiple executor nodes. Thus, even if one executor node fails, another will still process the data making the RDDs recover from any issue.

Persistence - This feature increases the reusability of RDDs by saving the intermediate result of the operation. It also provides the flexibility to select a storage strategy from memory_only, disk_only, memory_and_disk, memory_only_ser, memory_and_disk_ser.

Coarse-Grained Operations - We can apply from a group of operations like maps, filter by, or group by. These operations can be applied to all the elements.

Other features of RDD include immutability (RDDs cannot be divided further, therefore it is said to be a fundamental unit of Spark), partitioning, automatic data rollback, etc.

Operations on RDD

RDD supports two main operations namely,

Transformations
Actions

As mentioned earlier, RDDs are immutable and read-only in nature, then how does it apply transformations ?

Well, when we run any transformation, it runs those transformations on all the RDD and creates a new RDD instead of transforming the original one. This is mainly done to achieve fault tolerance and optimization reasons. Actions are applied to these new RDDs to produce the results.

Here, we come to an end of this post which covers the basics about RDD, its features, two ways to create RDD, and operation that can be performed on them.

Accuracy | Precision | Recall | F-Measure - Analyzing your Confusion Matrix

2020-09-25T10:08:00.008-07:00

When we are going through the Confusion Matrix, there are a lot many things that we can unearth like the type 1 error, type 2 error, accuracy, precision, recall, f-measure, etc. which helps us to understand the model performance. Please visit my earlier post where I have covered about the confusion matrix, type 1 and type 2 errors. In this post, let us understand the precision, recall, accuracy and f-measure from a confusion matrix and their significance on a model.

Accuracy

It is the ratio of the sum of correctly identified classes to the sum of all the classes.

The correctly identified classes are the ones which are -

Predicted positive provided they are actually positive i.e. True Positive.
Predicted negative provided they are actually negative i.e. True Negative.

Precision

It is the ratio of the correctly identified positive classes to the sum of the predicted positive classes.

The predicted positive classes are the ones which are -

Predicted positive provided they are actually positive i.e. True Positive.
Predicted positive provided they are actually negative i.e. False Positive.

This ratio provides information that out of all the positive classes we have predicted correctly, how many are actually positive.

Recall

It is the ratio of the correctly identified positive classes to the sum of the actual positive classes.

The actual positive classes are the ones which are -

Predicted positive provided they are actually positive i.e. True Positive.
Predicted negative provided they are actually positive i.e. False Negative.

This ratio provides the information that out of all the positive classes, how much we predicted correctly.

F-Measure

The F-Measure is also known as the F1-Score or the F-Score. It is a measure if a test's accuracy. We make use of this F-score in order to compare two models. F-measure considers recall and precision at the same time. It makes use of the harmonic mean instead of arithmetic mean to take care of the extreme values.

Above image provides a summary of what we have discussed in this post. Hope this post gives you the clarity pertaining to the different measures that are derived from a confusion matrix.

BeautifulSoup and Python: Writing your first web scraping code

2020-09-13T13:39:00.003-07:00

In our last post, we covered the introduction to web scraping, why do we need it, and the process of web scraping. In this post, we will look at the python library which is used for scraping and also write our first scraping code.

We are going to perform web scraping for Premier League Table 2019-20 and store the data in a pandas dataframe.

Let us get started ..

We will first import the required libraries -

The requests library will be used to perform get requests to a URL from where we have to scrape the data. It will help to get the HTML behind the webpage.

Making use of the requests library to get the HTML code

Enter the URL from where you want to get the data. The variable page is an HTML response object. We can get the status of our request by printing page.status_code (200 for successful responses). The page.text object will consist of the actual HTML code that we are interested in.

Create a BeautifulSoup() object named soup. The prettify attribute of this object will return the HTML code in a structured format that has better readability.

The next task is to identify the tags and their class names in which our required data is present.

You can make use of the browser devtools to identify these tags and the class names of the relevant data that you require. Once you have this ready with you, you can use the find_all() method to find all the relevant HTML code for your requirement. In our case, the tag is a table tag and the class is standing-table__table.

The league_table variable contains only the HTML content of the table.

We can use the indexing logic of python where index n will give you (n+1)th tag value. The .text attribute will give the text value that the tag contains. We have used the .strip() method of string to eliminate the additional spaces before and after the text content.

The rest of the code is self-explanatory shown below.

We have created a dataframe of the league table -

This was an introductory post about how we can write our first web scraping code. The BeautifulSoup library has very good documentation where you can visit and find out additional methods and attributes which can prove to be handy during your scraping process.