StatsCosmos

http://bit.ly/2deGavK

http://bit.ly/2eak6mO

http://bit.ly/2dMZ2EN

http://bit.ly/2ecgS3c

http://bit.ly/2e3W25D

http://bit.ly/29H9oHK

http://bit.ly/2ealpCg

http://bit.ly/2dM7BRf

http://bit.ly/1M7MAYL

http://bit.ly/2ecgEJF

http://bit.ly/2e85tph

http://bit.ly/2dN0s1K

http://bit.ly/2dV7LF8

http://bit.ly/2dP0VUW

http://bit.ly/2deEO41

http://bit.ly/2deG8Ui

http://bit.ly/1RythtQ

http://bit.ly/2deHMFM

http://bit.ly/2deFHJK

http://bit.ly/2d9ujUv

http://bit.ly/2dP4ACh

http://bit.ly/2e3YxFf

http://bit.ly/2dP667i

http://bit.ly/2ek7a2E

http://bit.ly/2ecl9DH

http://bit.ly/2dmMnc0

http://bit.ly/2deIv9H

http://bit.ly/2dyEJsL

http://bit.ly/2e40IZm

http://bit.ly/2dN0Ehz

http://bit.ly/2dMbPs9

http://bit.ly/2dVilMH
http://bit.ly/1tuxwkL

http://bit.ly/24HpXok

http://bit.ly/2ek9qqJ

http://bit.ly/2dZhmKx

http://bit.ly/2e43Iox

http://bit.ly/2dmPgtH

http://bit.ly/2e41lSC

http://bit.ly/2dMepOJ

http://bit.ly/2dmPqkq

http://bit.ly/29SNdwn

http://bit.ly/2earBdh

http://bit.ly/2dP9EGR

http://bit.ly/2e443aU

http://bit.ly/2d9zJPo

http://bit.ly/2e85tph

http://bit.ly/2ekbnmQ

http://bit.ly/2dN0s1K

http://bit.ly/2dyHiei

http://bit.ly/2dyEJsL

http://bit.ly/2dWSWlN

http://bit.ly/2e1scS2

http://bit.ly/2dzV9kx
http://bit.ly/2ebNLfh
http://bit.ly/2e5yaBn
http://bit.ly/29Hbbwn
http://bit.ly/2e27GzK
http://bit.ly/1Uo1MH8

http://bit.ly/268y3qY

http://bit.ly/2efZi2p

http://bit.ly/2eluk3B

http://bit.ly/29Crcy0

http://bit.ly/2e9DOBu

http://bit.ly/29I25d6

http://bit.ly/2dV7LF8

http://bit.ly/2em9Ehr

http://bit.ly/2emMXF7

http://bit.ly/2dW7Jeb

How to summarize Last.fm Social Tagging System Profiles using Go, Hadoop, MongoDB and Spark

2016-07-15T18:40:00.000-07:00

This post is designed for a joint installation of Apache Hadoop 2.6.0, MongoDB 2.4.9, Apache Spark 1.5.1 (pre-built for Hadoop) and Ubuntu 14.04.3 LTS. The purpose of the illustration is to show how one can construct and summarize a database of the Last.fm social system profiles for the GroupLens HetRec 2011 Last.fm dataset using the Go Programming Language. The methodology is that outlined in Cantador, Bellogin and Vallet (2010).

The approach involves implementing the MapReduce programming model using a mapper-reducer set extracted from a Go word count application. The specific HetRec 2011 Last.fm dataset is the assignments dataset. The approach involves counting four assignment categories. These are as follows.

Assignments made by each user (using all tags)
Assignments made to each artist (using all tags)
Assignments made by each user with a specific tag
Assignments made to each artist with a specific tag

The results of the counting exercise can then be used to construct the six core social tagging system measures outlined in the paper. The measures can, in turn, be used to construct the social tagging system profiles outlined in the paper.

1. Model

In social tagging systems, one has a folksonomy defined as a tuple ℱ = {T,U,I,A}, where T = {t₁,…,t_L} is the set of tags, U = {u₁,…u_L} the set of users, I = {i₁,…,i_L} the set of items annotated with the tags of T and A = {(u_m,t_l,i_n)} ∈ U ⨯ T ⨯ I is the set of assignments (annotations) of each tag t_l to an item i_n by a user u_m.

In these systems, the users create or upload content (items), annotate it with the tags and share it with other users. The whole set of tags then constitutes a collaborative classification scheme. The classification scheme can then be used to search for and discover items of interest.

In the system the users and items can be assigned profiles defined as weighted lists of social tags. Generally, a user will annotate items that are relevant for them, so the tags she or he provides can be assumed to describe her/his interests, tastes and needs. It can additionally be assumed that the more a tag is used by a user the more important the tag is for her or him.

Similarly, the tags that are assigned to an item usually describe its contents. Consequently, the more users annotate the item with a particular tag the better the tag describes the item’s contents. It is important, however, to keep in mind a key feature of the assumptions. If a particular tag is used very often by users to annotate many items, then it may not be useful when one wants to discern informative user preferences and item features.

The above constructs and assumptions allow one to formulate a social tagging system recommendation problem that can be used as the basis for the analysis. Adomavicius and Tuzhilin (2005) formulate the recommendation problem as follows.

Let U = {u₁,…,u_M} be a set of users, and let I = {i₁,…,i_N} be a set of items. Let g: U ⨯ I → ℛ, where ℛ is a totally ordered set, be a utility function such that g(u_m,i_n) measures the gain of usefulness of item i_n to user u_m.

Then, what is desired is to choose for each user u ∈ U a set of items i ^max,u ∈ I, unknown to the user, which maximize the utility function g:

∀ u ∈ U, i ^max,u = arg max _{i ∈ I} g(u,i).

In content-based recommendation approaches g() is then formulated as follows.

g(u_m,i_n) = sim(ContentBasedUserProfile(u_m), Content(i_n)) ∈ ℜ^k, where

ContentBasedUserProfile(u_m) = u_m = (u_m,1,…, u_m,K) ∈ ℜ^k is the content-based user preferences of user u_m(namely, the content features that describe the interests, tastes and needs of the user).
Content(i_n) = i_n= (i_n,1,…,i_n,K) ∈ ℜ^k is the set of content features characterizing item i_n.

The ContentBasedUserProfile() and Content() descriptions are usually represented as vectors of real numbers (weights). In the vector representation each component in the tuple measures the “importance” of the corresponding feature in the user and item representations. The function sim() quantifies the similarity between a user profile and an item profile in the content feature space. This set up then allows one to develop approaches to determine the basket of items i ^max,u in the problem.

The approach taken in the paper (Cantador, Bellogin and Vallet, 2010) and this illustration is to consider the social tags in such systems as the content features that describe both the user profile and the item profile. This allows one to study the different weighting schemes that can be used to measure the “importance” of a given tag for each user and item. These weighting schemes result in the social tagging content-based Profile models (from the paper) that we will consider in this illustration.

Given the above modelling structure and formulation of the folksonomy, the simplest way of defining the profile of user u_m, is as a vector u_m = (u_m,1,…,u_m,L), where u_m,l= |{(u_m,t_l,i) ∈ A | i ∈ I}| is the number of times the user has annotated items with tag t_l. In an analogous manner, the item profile of i_n can be defined as a vector i_n= (i_n,1,…,i_n,L), where i_n,l= |{(u,t_l,i_n) ∈ A | u ∈ U}| is the number of times the item has been annotated with tag t_l.

The paper (Cantador, Bellogin and Vallet, 2010) extends these two definitions of user and item profiles by using different expressions for the vector component weights.

The formulation and modelling structure result in the following table of core elements in the profile and recommendation models proposed in the paper (Cantador, Bellogin and Vallet, 2010).

The elements can be used to define three profile models that we will consider in this illustration. These are the TF Profile Model, the TF-IDF Profile Model and the Okapi BM25 Profile Model.

TF Profile Model

The TF Profile Model results from the simple approach to define user and item profiles.

Essentially, one can count the number of times a tag has been used by a user or the number of times a tag has been used by the community to annotate an item. This information is available for the Last.fm system if one defines the artist songs as the items and the user_taggedartists dataset (assignments dataset) with the second and third columns interchanged as the A set in the analysis.

The profile model for user u_m = u(m) then consists of the vector u_m= (u_m,1,…,u_m,L), where u_m,l= tf_u(m)(t_l).

The profile for item i_n = i(n) is then similarly defined as the vector i_n= (i_n,1,…,i_n,L), where i_n,l= tf_i(n)(t_l)

TF-IDF Profile Model

The second profile model that is proposed in the paper is the TF-IDF profile model.

The proposed model is formulated as follows.

u_m,l = tfiuf_u(m)(t_l) = tf_u(m)(t_l)iuf(t_l)

i_n,l= tfiif_i(n)(t_l) = tf_i(n)(t_l)iif(t_l).

Okapi BM25 Profile Model

The third profile model proposed in the paper is the Okapi BM25 model which follows a probabilistic approach.

The model is formulated as follows.

where b and k₁ are set to the standard values of 0.75 and 2, respectively.

One of the aims of this illustration is to show how one can quantify the six social tagging system measures in the Table for the Last.fm social tagging system using the GroupLens HetRec 2011 Last.fm dataset.

2. Prepare the data

The approach that will be followed will involve creating key datasets that will serve as inputs to the MapReduce implementation. This will involve creating four datasets from which to compile the measures.

The first dataset will use the first column of the assignment dataset as the keys. The second will use the second column. The third will use an index created with the first and third columns. The fourth will use an index created with the second and third columns. The values to the keys are created in the mapping phase when the data is processed in Hadoop, MongoDB and Spark (using Go for Hadoop and Spark). The reduce phase in the processing will generate the core measures.

The MapReduces will thus produce datasets with which to quantify the first, second, fifth and sixth measures directly. The number of observations in the fifth and sixth output datasets will provide the numerator values for the third and fourth measures, respectively. The first and second datasets will provide the inputs for the calculation of the denominators for the third and fourth measures, respectively.

The resulting quantifications will then be used to formulate the profile models for each user and each artist.

3. Prepare the Mapper-Reducer set (experimental/under development)

The approach to the MapReduce will be to use a word count application prepared in Go. The source Go word count application was prepared by Damian Gryski and posted on Github on November 2, 2012. In this illustration minor modifications were made to the code in order to allow the resulting application to read the (generated) user, artist, artist-tag and user-tag indexes from the datasets.

The wordcount application was prepared using the dmrgo library. The dmrgo library is a Go library for use with the Hadoop Streaming protocol and is thus ideal for MapReduce using Hadoop Streaming and the Spark Pipe facility. The guide to the library provides the information required to create and run the MapReduces in this illustration. The instructions include how to include the mapper-reducer code in one’s scripts and most importantly, how to build the word count application.

The code for the application for this illustration was prepared using this Go programming tutorial, this Go tour and is as follows.

The Go application can be saved in a local system folder , < Local System Go Application Folder>. The basic approach for including the scripts in the applications (and the program) is to create the following two bash files and save them in local system folders. In the remainder of the illustration the bash files are treated as the mapper and the reducer.

4. Process the data in Hadoop, MongoDB and Spark

Hadoop Streaming

The first MapReduce can be implemented in Hadoop using the Hadoop Streaming facility. The commands for customizing the Hadoop run can be found in this tutorial.
In order to conduct the first MapReduce the following arrangements need to be made.

Input data file: InputData.txt

Hadoop Distributed File System (HDFS) input data folder: < HDFS Input Data Folder>

Local system Hadoop Streaming jar file folder: <Local System Hadoop Streaming jar File Folder>

Hadoop Streaming jar file: hadoop-streaming-2.6.0.jar

Local system mapper folder: <Local System mapper Folder>

Local system reducer folder: <Local System reducer Folder>

Local system mapper bash file: Mapper.sh

Local system reducer bash file: Reducer.sh

HDFS Output data folder: <HDFS Output Data Folder>

The next step is to run the following command on Ubuntu 14.04.3.

These are the contents of the resulting output file.

MongoDB

The four MapReduces can be implemented in MongoDB. The first step is to import the four datasets into MongoDB using mongoimport. The dataset can be called LastFM. The collections can be named as follows.

MapReduce one: User Tags
MapReduce two: Artist Tags
MapReduce three: User Profile
MapReduce four:Artist Profile

The collections can be viewed using the db.<LastFM collection>.find().pretty() command. For example, to view the input collection for the third MapReduce one can switch to the LastFM dataset. One can then use the db.UserProfile.find().pretty() to view the UserProfile collection.

The other collections can be viewed analogously.

The next step is to run the following program in order to run the MapReduce for the user tag counts.

Then one can run the db.UserTags_Counts.find().pretty() method to view the results.

It is also possible to use the method to run individual queries. For example to find the results for user 2 and tag 13 one can run db.UserTags_Counts.find({"_id": “2;13;”}).pretty() method.

The results can be run analogously for the other collections. The find().pretty() method will yield the following results for the MapReduces for the remainder of the collections.

The next step is to run the second MapReduce using PySpark.

PySpark (application)

The second MapPreduce can be implemented in the Spark Pipe facility using a PySpark application. The application can also include SQL queries from Spark SQL and NoSQL queries from MongoDB (PyMongo and rmongodb). The PyMongo query can be included in the application. The rmongodb script can be saved in a separate file.

In order to conduct the second MapReduce and generate the queries the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: <Local System Input Data Folder>

Local system mapper folder: <Local System mapper Folder>

Local system reducer folder: <Local System reducer Folder>

Local system output data folder: <Local System Output Data Folder>

Local system mapper bash file: Mapper.sh

Local system reducer bash file: Reducer.sh

MongoDB instance: Have a MongoDB instance with the arrangements outlined in the MongoDB illustration

Local system rmongodb query Script Folder: <Local system PySparkQueryScript Folder>

PySpark query script: PySparkQueryScript.R

The PySpark Pipe application and the query script can be prepared using the tutorials in this post, the Spark Quick Start website, this Spark Guide, this post, this post, this post and this post. The application and query script are as follows.

The application and the script can be saved in appropriate local system folders. The application can be run using the bin/spark-submit script.

The application will generate the following output file contents, statistical summary measures, Spark SQL query and NoSQL MongoDB query.

What just happened? The PySpark application conducted the second MapReduce and wrote the results to file as shown in the first image. The application then took the in-memory results, created an SQL data frame and showed us the dataframe as shown in the second image. The application then took the second column in the dataframe, calculated statistical summary measures and showed us the results as shown in the third image.

The application then made a connection with MongoDB using PyMongo in the local machine, conducted a MapReduce on the ArtistTags collection from the MongoDB illustration and generated a results MongoDB collection on the LastFM database called PyMongoresults1. The application then read 10 records from the collection and printed the ten records on screen as shown in the first ten JSON entries in the last image. The application then generated two individual JSON queries from the PyMongoresults1 collection for artist-tag combinations 10002;127; and 10005;5770;. These two were also printed on screen. These are the next two JSON entries in the last image.

The application then called an R script which also made a connection with MongoDB using rmongodb. The script then made two queries. The first query was for User 2 from the UserProfile collection in the LastFM database. The second was for Artist 52 in the ArtistProfile collection in the LastFM database. The script then closed the connection (NULL entry) and returned the results to the application. The application took the results (two JSON queries and the NULL entry) from the R script and printed them on screen. These are the last two JSON entries and the NULL entry in the last image.

The next step is to run the third MapReduce using SparkR.

SparkR (application)

The third MapReduce can be implemented in the Spark Pipe facility using a SparkR application. The application can also include SQL queries from Spark SQL and NoSQL queries from MongoDB (PyMongo and rmongodb). The rmongodb query can be included in the application. The PyMongo script can be saved in a separate file.

In order to conduct the second MapReduce and generate the queries the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: < Local System Input Data Folder>

Local system mapper folder: <Local System mapper Folder>

Local system reducer folder: <Local System reducer Folder>

Local system output data folder: <Local System Output Data Folder>

Local system mapper bash file: Mapper.sh

Local system reducer bash file: Reducer.sh

MongoDB instance: Have a MongoDB instance with the arrangements outlined in the MongoDB illustration

Local system SparkR query Script Folder: <Local System SparkRQueryScript Folder>

SparkR query script: SparkRQueryScript.py

Local system qplot folder: <Local System Qplot Folder>

Local system qplot Histogram .png file: Qplot.jpeg

The SparkR Pipe application and the query script can be prepared using the tutorials in this post, the Spark Quick Start website, this Spark Guide, this post, this post , this post, this post and this post. The application and the supporting script are as follows.

The application and the script can be saved in appropriate local system folders. The application can be run using the bin/spark-submit script:

The application will generate the following output file contents, qplot Histogram .jpeg file, Spark SQL query, statistical summary measures and NoSQL MongoDB query.

What just happened? The SparkR application ran the third MapReduce and wrote the results to file as shown in the first image. The application then took the second column from the RDD, generated a Histogram using the qplot function and also saved it to a file. This is the graph in the second image. The application then took the in-memory data and used it to create an SQL data frame. The application then showed the dataframe as shown in the third image.

The application then took the second column in the dataframe, calculated statistical summary measures and showed us the results as shown in the fourth image. The application then established a connection with MongoDB using rmongodb in the local machine (indicated by the TRUE) and generated two JSON queries. The first was a query on the UserTags collection (in the MongoDB illustration) for user-tag combination 2;13;. The second was query on the ArtistProfile_id_Counts collection for artist 52. The application then printed the results of the queries on screen. These are the first two JSON entries in the last image. The application then closed the connection which is indicated by the NULL entry in the last image.

The application then called a SparkRQuery python script. The script made a connection with the MongoDB LastFM UserProfile collection on the local machine. The Python script conducted a MapReduce on the UserProfile collection from the MongoDB illustration (using PyMongo) and generated a results collection called PyMongoresults2 in the LastFM database.

The script then read 10 records from the collection and made two queries. The first query was basic and the second was specific. The basic query used the PyMongo find_one() method. The specific query also used the find_one() method but used the specific document functionality. The specific query was for user 52.

The script then made a connection with the PyMongoresults1 collection from the PySpark PyMongo MapReduce and made two queries. The two queries were similar to the preceding two queries in that the first was basic (using find_one()) and the second was specific (also using find_one()). The second query was for the artist-tag combination 10002;12051;. The script then returned the 14 JSON queries to the application. The application printed the results on screen as shown in the last image.

The next step is to conduct the fourth MapReduce in the Scala Spark-shell.

Scala Spark-shell program

The fourth MapReduce can be run in the Spark Pipe facility in a Scala Spark-shell program. The shell program can also include SQL queries from Spark SQL and NoSQL queries from MongoDB (PyMongo and rmongodb). The rmongodb and PyMongo scripts can be saved in separate files.

In order to run the fourth MapReduce and generate the queries the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: <Local System Input Data Folder>

Local system mapper folder: <Local System mapper Folder>

Local system reducer folder: <Local System reducer Folder>

Local system output data folder: <Local System Output Data Folder>

Local system mapper bash file: Mapper.sh

Local system reducer bash file: Reducer.sh

MongoDB instance: Have a MongoDB instance with the arrangements outlined in the MongoDB illustration

Local system rmongodb query Script Folder: <Local system rmongodb query Script Folder>

Local system PyMongo query Script Folder: <Local system PyMongo query Script Folder>

rmongodb query script: ScalaSparkShellrmongodbScript.R
PyMongo query script: ScalaSparkShellPyMongoScript.py

The Scala Spark-shell Pipe program and the query script can be prepared using the tutorials in this post, the Spark Quick Start website, this Guide, this post, this post and this post. The program and scripts are as follows.

The program will generate the following output file contents, Spark SQL query, statistical summary measures and NoSQL MongoDB query.

What just happened? In the Scala Spark-shell program the fourth MapReduce was run and the results were written to a file. The contents of the file are shown in the first image. The results were then used to create an SQL data frame and show it. This part of the program will result in the output shown in the second image.

The next part of the program then takes the second column in the dataframe, calculates statistical summary measures and asks Spark to show the results. This part of the program will result in the output shown in the third image.

The next part of the program then calls an R script which makes a connection with MongoDB using rmongodb on the local machine and makes two JSON queries from the LastFM database. The first is on the UserTags_Counts collection for user-tag combination 2;13;. The second is on the ArtistProfiles_id_Counts collection for artist 52. The script then closes the connection, returns the results to the program and puts them to a Scala string variable called rmongoquery. The contents are shown on the fourth image (print statement).

The next part of the program then calls a Python script. The Python script makes a connection with the MongoDB ArtistProfile collection in the LastFM database.
The script then runs a MapReduce on the collection and generates a LastFM database collection called PyMongoresults3. The script then reads 10 records from the collection, conducts a basic find_one() query and a specific query for artist 52.

The script returns the results to Spark and they are put into a string variable called pymongoquery. The program then prints the contents of the variable in screen. This is shown in the last image.

This sequence can run run analogously using a Spark Java application.

Java Spark (application)

The fourth MapReduce can also be run in the Spark Pipe facility in a Spark Java application. The application can also include SQL queries from Spark SQL and NoSQL queries from MongoDB (PyMongo and rmongodb). The rmongodb and PyMongo scripts can be saved in separate files.

In order to conduct the fourth MapReduce and generate the queries the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: <Local System Input Data Folder>

Local system mapper folder: <Local System mapper Folder>

Local system reducer folder: <Local System reducer Folder>

Local system output data folder: <Local System Output Data Folder>

Local system mapper bash file: Mapper.sh

Local system reducer bash file: Reducer.sh

MongoDB instance: Have a MongoDB instance with the arrangements outlined in the MongoDB illustration

Local system rmongodb query Script Folder: <Local System SparkJavaQueryRscript Folder>

Local system PyMongo query Script Folder: <Local System SparkJavaQueryPythonscript Folder>
rmongodb query script file: SparkJavaQueryRScript.R

PyMongoDB query script file: SparkJavaPythonScript.py

JavaScriptSubmit bash file: SparkJavaQueryScriptsSubmit.sh

The Java Spark Pipe application and query scripts can be prepared using the tutorials in this book, this book, the Spark Quick Start website, this Spark Guide, this post, this post, and this post. The query script can be called from the Java application using a bash file which one can name JavaScriptSubmit.sh. The application, bash file and query scripts are as follows.

The application java file can be exported to a jar file. The jar file, bash file and scripts can be saved in appropriate local system folders. The application can be run using the bin/spark-submit script.

The application will generate the following output file contents, Spark SQL query and NoSQL MongoDB query.

What just happened? The application ran the fourth MapReduce and wrote the results to file whose contents are shown in the first image. The application then created a dataframe containing the in-memory results. The application then ran an SQL query on the dataframe for artist 52 and printed the results on screen. These are the first two entries in the last image.

The application then called a bash query submit script to conduct two MongoDB queries. The bash query submit script calls two scripts, a rmongodb query script and a PyMongo query script. The rmongodb query is run first. The rmongodb query script makes a connection with MongoDB on the local machine. The script then runs two queries in the LastFM database. The first query is on the UserTags_Counts collection for user-tag combination 2;13;. The second is on the ArtistProfile_id_Counts collection for artist 52. The script then closes the connection and takes the results to bash. Bash in turn prints the results returned to screen.

The bash query script then calls the PyMongo query script. The PyMongo query script also makes a connection with MongoDB on the local machine. The script then makes a connection with the ArtistProfile collection in the LastFM database and conducts a MapReduce that computes the artist profile sizes.

The results of the MapReduce are written to a LastFM collection called PyMongoresults3 in MongoDB. The script prints ten records from the database. It then conducts a basic query using the find_one() method on the PyMongoresults3 collection and prints the results. The script then conducts a specific query for artist 52 and prints the results. The printed results are taken to bash. Bash prints the results returned to screen.

The application prints all the output printed to screen by bash to screen in Spark as shown in the remaining entries in the last image.

5. Query the data

The data housed in MongoDB can additionally be queried using a ShinyMongo application from GitHub. The GitHub Gist also includes installation instructions and additional tutorials on Shiny Apps can be obtained from the Shiny tutorials.

The arrangement in this illustration requires Shiny, rmongodb, rjava, an instance of R and an instance of MongoDB (as outlined in the MongoDB illustration). The ShinyMongo application is available in downloadable form and in script form. The downloadable form can be run using the following commands in the R shell.

The command will launch the Shiny App. The graphical user interface has the following features.

The queries can be entered in the experimental JSON field and the size of the query is limited to 100 entries.

The application can be run locally using the ui.R and server.R scripts in the Gist (and following the Shiny tutorials). The Shiny App can be launched using the following commands in the R console.

In the case of the MongoDB (including rmongodb and PyMongo) part of the illustration the resulting locally based interface has the following features.

6. Analyze/Summarize the data using Histograms

The data can be further analyzed/summarized using the SGPlot Procedure in the SAS software. One can prepare a program using the tutorial in this post to generate the top 20 counts from MapReduce output data in turn. Suppose we have a Microsoft Excel dataset with the following structure.

One can then import the data into a SAS software work dataset, say, called import. If one runs the following code for <var> being UserID and chooses the first title option on line 8.

One will obtain the following plot.

The other plots can be generated similarly. In the case for <var>=ArtistID and appropriate dataset.

In the case for <var>=UserIDTagID, second title option on line 8 and appropriate dataset.

In the case for <var>=ArtistIDTagID, appropriate title option and appropriate dataset.

The next step is to obtain the statistical summary measures using the stat.desc() function in the pastecs package in R.

In the case of the user tags counts the statistical summary measures are as follows.

In the case of the artist tags counts.

In the case of the user profile counts (user profile size).

In the case of the artist profile counts (artist profile size).

The data can then be analyzed using Relative Frequency Histograms in R using the HistogramTools package. The histograms can be written to file using R grDevices.
The program has the following general structure for a numeric vector x.

The Relative Frequency Histograms for each of the output files from the MapReduces, the TF-IDF Profiles (user and artist) and the Okapi BM25 Profiles (user and artist) are as follows.

The user tags Relative Frequency Histogram.

The artist tags Relative Frequency Histogram.

The user profile Relative Frequency Histogram.

The artist profile Relative Frequency Histogram.

The user TF-IDF profile Relative Frequency Histogram.

The artist TF-IDF profile Relative Frequency Histogram.

The user Okapi BM25 profile Relative Frequency Histogram.

The artist Okapi BM25 profile Relative Frequency Histogram.

7. Conclusions

The illustration showed how one can construct social tagging system profile models for the Last.fm dataset using the profile models proposed in the paper Cantador, Bellogin and Vallet (2010). The results from the dataset are exciting as well as interesting. The resulting findings and constructions from the illustration can be used as inputs to an analysis of the dataset using the similarity measures proposed in the paper.

Interested in other posts on Big Data and Cloud Computing from the Stats Cosmos blog?

Check out my other posts

Subscribe via RSS to keep updated

Check out my course at Udemy College

Check out our Big Data and statistical services

Sources

http://bit.ly/1M7MAYL

http://bit.ly/1tusyEE

http://bit.ly/1R89nos

http://bit.ly/1rDzezL

http://bit.ly/29LIeOq

http://bit.ly/1RythtQ

http://bit.ly/1TAOH9I

http://bit.ly/29UbmS7

http://bit.ly/1YxRLtN

http://bit.ly/1V5cK57

http://bit.ly/1WSfBB5

http://bit.ly/29CfRhn

http://bit.ly/29CfFyD

http://bit.ly/1QttFIJ

http://bit.ly/29Gqv6B

http://bit.ly/1oXvms2

http://bit.ly/1omcBNO

http://bit.ly/29H9oHK

http://bit.ly/29H9saD

http://bit.ly/293dfgO

http://bit.ly/29X5F5s

http://bit.ly/1Y08OWK

http://bit.ly/1VR19sn

http://bit.ly/29N9Phk

http://bit.ly/29H9GOM

http://bit.ly/29H9LlK

http://bit.ly/29SNdwn

http://bit.ly/1T76xr7

http://bit.ly/1QcuOe5

http://bit.ly/262BQcE

http://bit.ly/29AGgNj

http://bit.ly/1OuZA19

http://bit.ly/1YiCkUc

http://bit.ly/21TGWAx

http://bit.ly/1M0puc7

http://bit.ly/1YiCkUc

http://bit.ly/1QcuOe5

http://bit.ly/28UOcrU

http://bit.ly/1Ueuwjj

http://bit.ly/268CGRV

http://bit.ly/29zjkC1

http://bit.ly/1omcG4d

http://bit.ly/2aenpZo

http://bit.ly/29HaW4L

http://bit.ly/29X7jnD

http://bit.ly/29NcgQO

http://bit.ly/2aenBrr

http://bit.ly/29UGW4W

http://bit.ly/1M0qSLy

http://bit.ly/29Hbbwn

http://bit.ly/29SPmIl

http://bit.ly/29GslnM

http://bit.ly/29Crcy0

http://bit.ly/1PFIW8p

http://bit.ly/29Z6sCZ

http://bit.ly/2agxtko

http://bit.ly/1YiCkUc

http://bit.ly/1QttFIJ

http://bit.ly/29JhOhT

http://bit.ly/1OkRxhP

http://bit.ly/24HpXok

Or check out our statistics and e-learning services

http://bit.ly/29KS0z1

http://bit.ly/29JKenQ

http://bit.ly/29I25d6

http://bit.ly/29ProK5

How to apply MapReduce to the Delicious dataset using Hadoop, MongoDB and Spark (Spark-shell, PySpark, Spark Applications, SparkR and SparkSQL) – Part Two

2016-06-21T06:27:00.001-07:00

This post is designed for a joint installation of Hadoop 2.6.0 (single cluster), MongoDB 2.4.9, Spark 1.5.1 (pre-built for Hadoop) and Ubuntu 14.04.3. The illustration builds on the steps covered in part one of the post on the application of the MapReduce programming model to the GroupLens HetRec 2011 Delicious dataset. The procedure involves applying seventeen MapReduces to the dataset. The first six were outlined in part one. The underlying mathematical model to the approach is outlined in the paper Cantador, Bellogin and Vallet (2010).

1. Model

The model starts with a social tagging system with a set of users U, items I, annotations A and tags T, constituting a folksonomy, F.

The users can be assigned profiles based on their tag assignments and the items can be assigned profiles based on the tags used on them. The user profile provides a reflection of the user's tastes, interests and needs. The item’s profile provides a reflection of its contents.

The illustration rests on two key assumptions. The first is that users will annotate items that are relevant for them, hence, the tags they provide can be assumed to describe their interests, tastes and needs. The second is that the tags assigned to an item usually describe its contents. The first follow on assumption is that the more a user uses a particular tag, the more important the tag is for them. The second follow on assumption is that the more an item is annotated with a tag, the better the tag describes its contents. The limitation to the assumptions are that tags that are used by users to annotate many items may not be useful to discern user preference and item features.

The recommendation problem, as formulated in Adomavicius and Tuzhilin (2005), is then, for a given set of users, U = {u₁,....,u_M} and items, I = {i₁,....,i_N} to define g: U * I → ℜ, where ℜ is a totally ordered set, a utility function such that g(u_m,i_n) measures the gain of usefulness of item i_n to user u_m. Then, for each user u ∈ U, the aim is to choose a set of items i^max,u ∈ I , unknown to the user, which maximize the function g:∀ u ∈ U, i^max,u= arg max _i∈Ig(u,i). In content-based recommendation analyses g() can be formulated as:

g(u_m,i_n) = sim(ContentbasedUserProfile(u_m),Content(i_n)) ∈ ℜ, where
ContentbasedUserProfile(u_m) = u_m = (u_m,1,....,u_m,K) ∈ ℝ^k is the content-based preferences of user u_m and
Content(i_n) = i_n= (i_n,1,....,i_n,K) ∈ ℝ^kcis the set of content based features of item i_n.

ContentbasedUserProfile(u_m) and Content(i_n) can usually be represented as vectors of real numbers where each vector component measures the "importance" of the corresponding feature in the user and item representations.

The sim() function measures the similarity between the user profile and the item profile in the content feature space.

The key to the MapReduce constructs is the assignment set, A = {(u_m,t_l,i_n)}∈ U * T * I , of each tag t_l to item i_n by each user u_m. This is available as the assignment dataset if the bookmarked URL's are defined to be the items in the model.

Essentially, a folksonomy can then be defined as a tuple F ={T, U, I, A}, where T={t₁,....,t_L} is the set of tags, U ={u₁,....,u_M} is the set of users that annotate, I ={i₁,....,i_N)} is the set of items that are annotated and A = {(u_m,t_l,i_n)} is the set of annotations. This notation allows one to define a simple profile for user u_m as a vector u_m = (u_m,1,....,u_m,L), where u_m,l = |{(u_m,t_l,i) ∈ A| i ∈ I }| is the number of times user u_mhas annotated items with tag t_l. The profile for item i_n can be defined as the vector i_n= (i_n,1,....,i_n,L), where i_m,l = |{(u,t_l,i_n) ∈ A | u ∈ U }| is the number of times item i_n has been annotated with tag t_l.

In the part one illustration, each of the social tagging system components and the TF-based similarity measures were explored. The relevant constructs provided an illustration of how a particular solution to the social tagging system problem could be formulated using the MapReduce programming model.

The aim of this part two post is to build on the solution (with its constructs) in order to generate other solutions and constructs using the MapReduce programming model.

The core constructs of the MapReduce are the six quantities described in this table from the paper Cantador, Bellogin and Vallet (2010).

The constructs are then the inputs with which to formulate the profile models. The Profile models are the following.

TF Profile model

TF-IDF Profile model

Okapi BM25 Profile model

where b and k₁ are the standard values 0.75 and 2, respectively.

The models allow for the formulation of the similarity measures . The Similarity measures are the following.

TF-based Similarity measures

TF Cosine-based Similarity measure

TF-IDF Cosine-based Similarity measure

Okapi BM25-based Similarity measures

Okapi BM25 Cosine-based Similarity measure

The key approach to keep in mind in the construction of the core components and the profile models is how one defines the key, value pairs for the MapReduce processing. For an example, for the TF Profile Model model, u_{_m,l} defines the tag frequency for tag t_l by user u_mand i_{_n,l}the tag frequency for tag t_l on item i_{_n}.

The formula g(u_{_m},i_{_n}) in the TF-based Similarity measure essentially means that one can take the tf_u(m)(t_l) = u_{_m,l} and tf_i(n)(t_l) = i_{_n,l}terms, attach, the {u_m,i_{_n}} key to the appropriate frequencies in the model (i.e. tf_u(m)(t_l) for non-zero frequency tags in i_{_n}'s profile in the first measure andtf_i(n)(t_l) for non-zero frequency tags in u_{_m}'s profile in the second measure) and conduct MapReduce in order to generate the required numerator sums. In part one the illustration showed how the relevant key indices could be constructed from the A set of the data (i.e. the Assignment dataset).

The data compilation component of this illustration involves constructing the appropriate MapReduce keys for the constructs and profile model terms (namely, unweighted and weighted frequency terms). The keys and values can be compiled into datasets that can be processed. The keys and the profile model terms can then be processed using MapReduce to quantify the Similarity measures using the Similarity measure formulae.

The best way to illustrate this process is to begin, as was shown in part one, from the A set data and core measures table from Cantador, Bellogin and Vallet (2010). In terms of the core measures in the table, the first measure can be constructed using an index of the first column and third columns of the A set. The second measure can be constructed using an index created from the second and third column of the A set. The fifth measure can be constructed using an index created from the first column of the A set. The sixth measure can be constructed using an index created from the second column of the A set. The number of observations in the output file of the fifth MapReduce and the outputs of the first MapReduce can be used to construct the third measure. The number of observations in output file of the sixth MapReduce and the outputs of the second MapReduce can be used to construct the fourth measure.

The profile model formulae can then be used to construct the profile model frequencies (namely, weighted frequencies). The Similarity measure formulae can be used to construct the model Similarity measures (as was shown in part one for the TF-based Similarity).

The remaining eleven MapReduces can be implemented in Hadoop, MongoDB and Spark. The Hadoop and Spark MapReduces can be implemented using mapper-reducer sets prepared in Perl, Python, R and Ruby. The MongoDB MapReduces can be implemented in the MongoDB 2.4.9 shell using mapper and reducer functions.

The MapReduces can be arranged into two categories, namely, one set and three set.
The three set MapReduces can be implemented in Hadoop and Spark using mapper-reducer sets prepared in R and Python. The one set MapReduces can be implemented in Hadoop, MongoDB and Spark. The non-MongoDB one set MapReduces can be implemented using mapper-reducer sets prepared in Perl, Python and Ruby.

In the case of the three set MapReduces, the Python mapper-reducer set is compiled for the MapReduces in Hadoop Streaming, the Spark Pipe facility in a PySpark application (and shell) and Spark Pipe facility in a SparkR application (and shell). The R mapper-reducer set is designed for the Spark Pipe facility in a Java application.

The one set MapReduces are designed for use in Hadoop Streaming, Spark Pipe in the Scala Spark-shell, Spark Pipe in a PySpark application, and Spark Pipe in a SparkR application. In the scheme, the one set MapReduce configuration is designed for the calculation of the Okapi BM25 Similarity measures (MapReduce nine and ten) from the core measures and profile model terms.

2. Prepare the data

The Similarity measure MapReduces can be referenced as one to thirteen, the first two being the TF-based Similarity measures from part one.

Similarity measure MapReduce three, four and five (TF Cosine-based Similarity)

In compiling the data, one can begin with the output files from the core measures and profile model calculations. The next step will be to take from each user and item frequency ( i.e. u_m,land i_n,l in the output files of the core measures) the first part of the index (i.e. the user u_mand i_n, respectively) and create two key, value combinations (i.e. the user u_m,tf_u(m)(t_l) and i_n,tf_i(n)(t_l), respectively) for the Similarity measure MapReduce.

In the mapping phase, the numerator values can be the cross-products (tf_u(m)(t_l) * tf_i(n)(t_l)), and the denominator values can be the squares of the individual values (i.e. (tf_u(m)(t_l))²and (tf_i(n)(t_l))²). In the reduce phase, the values of the numerator cross-products can be summed and the totals output by key. In the case of the denominator entries, the square roots of the sums can be output for each key. This will generate the outputs required by the Similarity measure formulae. This is essentially how the input files for the three set MapReduces can be constructed.

Similarity measure MapReduce six, seven and eight (TF-IDF Cosine-based Similarity)

The approach for the Similarity measure MapReduce six, seven and eight input file creation is similar to that of Similarity measure MapReduce three, four and five, except that one will first create tf_u(m)(t_l)* iuf(t_l) and (tf_i(n)(t_l)* iif(t_l), for the user u_mand item i_n, respectively, according to the TF-IDF Profile model. The rest of the steps follow analogously to that of Similarity measure MapReduce three, four and five.

Similarity measure MapReduce nine and ten (Okapi BM25-based Similarity)

The approach for the Similarity measure MapReduce nine and ten input file creation is similar to that of MapReduce three, four and five.

In the case of Similarity measure MapReduce nine, the approach will be to take the first part of the indices (i.e. the user u_m and item i_n) from the Okapi BM25 Profile model calculations and create a new key (u_m,i_n) for the Similarity measure MapReduce. The next step, is to identify from the items's profile the tags that have non-zero (item) weighted frequencies (i.e. i_n,l= bm25_i(n)(t_l) in the Okapi BM25 Profile model) and allocate the user's weighted tag frequencies (i.e. u_m,l = bm25_u(m)(t_l) from the Okapi BM25 Profile model) for each tag t_l(respectively). This will generate the key, value input file for Similarity measure MapReduce nine.

In the case of Similarity measure MapReduce ten the process is similar but instead one makes use of the user's profile and the item's frequencies. Hence, after the keys have been created in the same manner as in the case of MapReduce nine, one identifies from the user's profile the tags that have non-zero (user) frequencies. The next step is to allocate the item's weighted tag frequencies (i.e. i_n,l = bm25_i(n)(t_l) from the Okapi BM25 Profile model) for each tag t_l(respectively). This will generate the key, value input file for Similarity measure MapReduce ten.

Similarity measure MapReduce eleven, twelve and thirteen (Okapi BM25 Cosine-based Similarity)

The approach for the compilation of the input file for the Similarity measure MapReduce eleven, twelve and thirteen is similar to that of MapReduce three, four and five. The first part will, however, involve creating the key value pairs using the weighted frequencies bm25_u(m)(t_l) and bm25_i(n)(t_l), for the user u_mand item i_n, respectively, according to the Okapi BM25 Profile model. The rest of the steps follow analogously to that of Similarity measure MapReduce three, four and five.

3. Prepare the mapper-reducer sets

Three set mapper-reducer sets

Python mapper-reducer set

The Python mapper-reducer set can be prepared according to the tutorials in this post, this post and this post.

Mapper

Reducer

R mapper-reducer set

The R mapper-reducer set can be prepared according to the tutorials in this gist, this post and this post.

Mapper

Reducer

One set mapper-reducer sets

The one set mapper-reducer sets can be prepared in Perl, Python and Ruby.

Perl mapper-reducer set

The Perl mapper and reducer sets can be prepared according to the tutorials in this post and this post.

Mapper

Reducer

Python mapper-reducer set

The Python mapper and reducer sets can be prepared according to the tutorials in this post and this post.

Mapper

Reducer

Ruby mapper-reducer set

The Ruby mapper-reducer set can be prepared according to the tutorials in this post, this post and this post.

Mapper

Reducer

4. Process the data in Hadoop, MongoDB and Spark

Three set MapReduce

Hadoop Streaming

In order to implement the first three set MapReduce using the Hadoop Streaming facility the following arrangements need to be made.

Input data file: InputData.txt (tab-separated)
Local system Hadoop Streaming jar file folder: <Local System hadoop streaming jar file folder>
Local system mapper file folder: <Local System mapper File Folder>
Local system reducer file folder:<Local System reducer File Folder>
Hadoop Distributed File System (HDFS) input data folder: <HDFS Input Data Folder>
HDFS output data folder: <HDFS Output Data Folder>

The similarity measure MapReduce three, four and five can be conducted in the Hadoop Streaming facility using the following command on Ubuntu 14.04.3:

These are the contents of the resulting output file.

PySpark application

The similarity measure MapReduce six, seven and eight can be conducted in the PySpark Pipe facility. In order to implement the MapReduce using the PySpark Pipe facility the following arrangements need to be made.

Input data file: InputData.txt (tab-separated)
Local system input data folder: <Local System Input Data Folder>
Local system mapper file folder: <Local System mapper File Folder>
Local system reducer file folder:<Local System reducer File Folder>
Local system output data folder: <Local System Output Data Folder>

The next step is to prepare the following application prepared using the tutorials in this book, this post, this post, this post, this guide, this guide, this post, and this post.

The next step is to save the application in a Python file (i.e. PySparkThreesetPipeApp.py) in a local system folder and use spark-submit to run the application.

This will result in the following SQL query (of the contents of the output file).

Java Spark application

The similarity measure MapReduce nine, ten and eleven can be conducted in the Java Spark application Pipe facility. In order to implement the MapReduce using the Java Pipe facility the following arrangements need to be made.

Input data file: InputData.txt (tab-separated)
Local system input data folder: <Local System Input Data Folder>
Local system mapper file folder: <Local System mapper File Folder>
Local system reducer file folder:<Local System reducer File Folder>
Local system output data folder: <Local System Output Data Folder>

The MapReduce can implemented using the following application prepared using the tutorials in this website, this guide, this book, this book, this programming guide, this post, the SparkSQL website, this website and this post.

The next step is to save the application in a java file (i.e. JavaSparkThreesetPipeApp.java) in a local system folder, export the java file to a jar file JavaSparkThreesetPipeApp.jar (in a local system folder) and use bin/spark-submit to run the application.

This will result in the following SQL query.

One set MapReduce

The next step is to implement similarity measure MapReduce nine and ten. These can be implemented in MongoDB, the Spark Pipe facility using a SparkR application and the Spark Pipe facility using a Scala Spark-shell program.

MongoDB-shell

The one-set MapReduce for the Okapi BM25-based similarity for the User and Item measure can be prepared using programs in the MongoDB 2.4.9 shell.

The first step is to read the data into MongoDB database <MongoDB database> (in this illustration DeliciousMR) and collection<MongoDB collection>.

In this illustration the collection for the BM25 User collection is BM25UserSimilarity and the collection for the BM25 Item collection is BM25ItemSimilarity. The MapReduce collections are map_reduce_BM25UserSimilarity for the BM25-based User Similarity measure and map_reduce_BM25ItemSimilarity for the BM25-based Item Similarity measure.

The use db command can be used to switch to the DeliciousMR database.

The db.BM25UserSimilarity.find().pretty() command can be used to view the BM25-based User Similarity measure collection.

The next step is to run the following program for the MapReduce prepared using the tutorials in this post and this post.

This will generate the following output.

The db.collection.find().pretty() command in the Mongo shell will generate the following output for the BM25-based User similarity.

The db.BM25ItemSimilarity.find().pretty() command can be used to view the BM25-based Item Similarity measure collection.

The MapReduce procedure can then be implemented for the BM25-based Item similarity.

The find().pretty() command will generate the following output.

The results can then be queried in the Scala Spark-shell and in Spark applications.

SparkR Application

In order to implement the first one set MapReduce using a Spark Pipe SparkR application and query the results in MongoDB (RMongoDB and PyMongo) the following arrangements should be made.

Input data file: InputData.txt (tab-separated)
Local system input data folder: < Local System Input Data Folder>

Local system mapper file folder: < Local System mapper File Folder >

Local system reducer file folder: < Local System reducer File Folder >

Local system output data file folder: < Local System Output Data Folder>
MongoDB instance: Have an instance of MongoDB with the arrangements in the MongoDB illustration

The MapReduce (and query) can be implemented using an application and a script file. The script (for the application and the supporting script) can be prepared using the tutorials in this post, this post, this post, this document, this post, this post, this post, this post, this post, this post, this post, this post and this post. The next step is to save the following SparkR application and SparkRApplicationScript files in appropriate local system folders.

The application can be run using the bin/spark-submit script.

These are the contents of the resulting output file/SQL query/NoSQL query.

Scala Spark-shell

The similarity measure MapReduce nine can be conducted using the Spark Pipe facility in a Scala Spark-shell program. The program can be complemented with MongoDB queries using RMongoDB and PyMongoDB. The first step is to make the following arrangements.

Input data file: InputData.txt (tab-separated)

Local system input data folder: < Local System Input Data Folder>

Local system mapper file folder: < Local System mapper File Folder >

Local system reducer file folder: < Local System reducer File Folder >

Local system output data file folder: < Local System Output Data Folder>
MongoDB instance: Have an instance of MongoDB with the arrangements in the MongoDB illustration

The next step is to save the following scripts in appropriate local system folders.

The next step is to run the following program prepared using the tutorials in this post, this post, this post, this post, this post, this guide and this post.

These are the contents of the resulting output file/SQL query/NoSQL query.

5. Query/Analyze the results

Once all the output data has been generated, one can conduct queries using MongoDB, a PySpark application, a SparkR application and a Java application. The BM25-based measures were calculated for user 8 and item 1. The queries using MongoDB, the Spark Pipe facility in a SparkR application (including RMongoDB) and the Spark Pipe facility in a Scala Spark-shell program were shown in the last section.

The three set TF-IDF Cosine-based Similarity measure query for user 8 and item 1 can be generated using a SparkR application prepared using the tutorials in this post, this post, this post, this document, this post, this post, this guide, this guide, this post, this post, this post and this post. The query can also be complemented with the one set MapReduce queries in MongoDB and PyMongo.

The first step is to make the following arrangements.

Input data file: InputData.txt (tab-separated)

Local system input data folder: < Local System Input Data Folder>

Local system mapper file folder: < Local System mapper File Folder >

Local system reducer file folder: < Local System reducer File Folder >

Local system output data file folder: < Local System Output Data Folder>

MongoDB instance: Have an instance of MongoDB with the arrangements in the MongoDB illustration

In order to generate a query the following application file and application script file can be saved in local system folders.

The application can be run using the bin/spark-submit script.

This will generate the following output.

The three set TF Cosine-based Similarity measure query for user 8 and item 1 can be generated using a Java application. This can be done by firstly appending the Java application code from the last section in line 110 and line 111 as follows.

The next step is to run the application (using the bin/spark-submit) with the input file for Similarity measure MapReduce three, four and five. This will generate the following output.

The three set TF-IDF Cosine-based Similarity measure query for user 8 and item 1 can be generated using a Java application with the appended code and using the input file for Similarity measure MapReduce six, seven and eight. The bin/spark-submit run will generate the following output.

The three set Okapi BM25 Cosine-based Similarity measure query for user 8 and item 1 can be generated using a Java application with the appended code and using the input file for Similarity measure MapReduce eleven, twelve and thirteen. The bin/spark-submit run will generate the following output.

The one set TF-IDF Cosine-based Similarity measure for user 8 and item 1 can be generated using a PySpark application. In order to implement the first one set MapReduce using a Spark Pipe PySpark application and query results in MongoDB (using RMongoDB and PyMongo) the following arrangements should be made.

Input data file: InputData.txt (tab-separated)

Local system input data folder: < Local System Input Data Folder>

Local system mapper file folder: < Local System mapper File Folder >

Local system reducer file folder: < Local System reducer File Folder >

Local system output data file folder: < Local System Output Data Folder>

MongoDB instance: Have an instance of MongoDB with the arrangements in the MongoDB illustration

The PySpark application script (and supporting file script) can be prepared using the tutorials in this post, this post, this post, this post, this post, this guide, this guide, this post and this post. The following PySpark application file (PySparkPipeOnesetApp.py) and PySpark Application script file (PySparkOneSetAppScript.R) can be saved in local system folders.

The next step is to use spark-submit to run the application..

This will generate the following output.

6. Conclusions

The post provided an illustration of how to implement the MapReduce programming model to the GroupLens HetRec 2011 dataset using the methodology outlined in Cantador, Bellogin and Vallet (2010). The approach can be further fine tuned to conduct the other analyses outlined in the paper.

Interested in other Big data analyses and Cloud computing resources from the Stats Cosmos blog?

Check out our other blog posts

Subscribe to our RSS feeds for blog material updates

Or get a 28% discount to our exciting training opportunity bundle

http://bit.ly/1M7MAYL
http://bit.ly/1tusyEE
http://bit.ly/1Y07BPl
http://bit.ly/1rDzezL
http://bit.ly/1V5cK57
http://bit.ly/1RythtQ
http://bit.ly/1TAOH9I
http://bit.ly/1YxRLtN
http://bit.ly/1WSfBB5
http://bit.ly/1rtVawg
http://bit.ly/231JrU3
http://bit.ly/1Q5vX1t
http://bit.ly/1omcG4d
http://bit.ly/1TiHqjD
http://bit.ly/1Y08OWK
http://bit.ly/1M0oCUO
http://bit.ly/1PFIW8p
http://bit.ly/21TGWAx
http://bit.ly/1Uo1MH8
http://bit.ly/262B8Mv
http://bit.ly/1UTyEr2
http://bit.ly/1RKV5dQ
http://bit.ly/1W4xED9
http://bit.ly/21rwvVv
http://bit.ly/1OuZA19
http://bit.ly/1UFeOgy
http://bit.ly/262BQcE
http://bit.ly/1T76xr7
http://bit.ly/1UeusA6
http://bit.ly/268y3qY
http://bit.ly/1Ueuwjj
http://bit.ly/1sKO0oT
http://bit.ly/1Y07GTd
http://bit.ly/1QcuOe5
http://bit.ly/1UFf3IF
http://bit.ly/1Qc7Gc8
http://bit.ly/1WlCMnp
http://bit.ly/1NoLZIf
http://bit.ly/1tutIQq
http://bit.ly/1sKOIlR
http://bit.ly/1rtWuzr
http://ibm.co/1T3h0ml
http://bit.ly/1SZPyVw
http://bit.ly/1UgJ47t
http://bit.ly/21rwXDt
http://bit.ly/1Y09F9Z
http://bit.ly/1SN27EA
http://bit.ly/1WlCMnp
http://bit.ly/1Y08WFT
http://bit.ly/24WQCvF
http://bit.ly/1UBhVdw
http://bit.ly/1tuuFIF
http://bit.ly/1UezwnY
http://bit.ly/268CGRV
http://bit.ly/1UFi8bv
http://bit.ly/1tuxwkL
http://bit.ly/1Qcz518
http://bit.ly/1Ueuwjj
http://bit.ly/1QcuOe5
http://bit.ly/1sKRrMc
http://bit.ly/1UBka0r

Sources

http://bit.ly/1Upk0YR

http://bit.ly/28UOcrU

Or check out the Stats Cosmos Big Data, statistics and e-learning services

http://bit.ly/293dfgO

How to apply MapReduce to the Delicious dataset using Hadoop, MongoDB and Spark (Spark-shell, PySpark, Spark Applications, SparkR and SparkSQL) – Part One

2016-05-10T18:44:00.000-07:00

This post is designed for a joint installation of Apache Hadoop 2.6.0, Apache Spark 1.5.1 (pre-built for Hadoop), MongoDB 2.4.9 and Ubuntu Server 14.04.3 LTS. This illustration shows how one can use MapReduce to construct metrics in content-based recommendation models for social tagging systems. The specific system is the GroupLens HetRec 2011 Delicious Bookmarks dataset system. The illustration is composed of two posts.

The theoretical framework is that outlined in the paper: Content-based Recommendation in Social Tagging Systems by Cantador, Bellogin and Vallet, published in 2010. The MapReduce implementation approaches are Hadoop Streaming, Spark WordCount (Scala program and Python Application), Spark Pipe (Scala program and Python Applications), Spark SQL (SparkR and Scala Spark-shell) and MongoDB. The core dataset is the Bookmark Assignments dataset which translates to a version of the A set in the paper.

The procedure translates to 17 MapReduce jobs which can be categorized into two phases. The first phase involves constructing the element definitions (and weights) in the paper and the second phase the utility metrics (similarity measures). The first phase is composed of four MapReduce jobs and will be outlined in this post.

The second phase is composed of the remaining jobs. The first two MapReduce jobs in the second phase involve constructing the first similarity measure (two metrics) and will also be outlined in this post. The remaining eleven MapReduce jobs will be outlined in the second post.

1. The Model

In a social tagging system (Cantador, Bellogin and Vallet, 2010), users create or upload content, annotate it with their own words and share it with other system users. The content is referred to as items and the annotations as tags. The tagging system is then an unstructured collaborative content classification system called a folksonomy. The classification system can then be used by system users to search for and discover items of interest.

The modeling of the system rests on two key assumptions. The first is that users will generally annotate items that are relevant for them and thus their tags can be seen to be a reflection of their interests, tastes and needs. It can additionally be assumed that the more a tag is used by a certain user, the more the important the tag is for them.

The second assumption is that tags assigned to items describe their contents. Similarly, the more a certain item is annotated with a particular tag, the better the tag describes the item’s contents.

It is important to keep in mind, however, that if a tag is used very often by users to annotate many items, it may not be useful to discern the information assumed.

A folksonomy ℱ can be defined mathematically as a tuple ℱ = {T, U, I, A}, where T = {t₁,......, t_L} is the set of tags, U = {u₁,......, u_M} is the set of users and I={i₁,......, i_N} is the set of items. A set A = {(u_m, t_l, i_n)} ∈ U * T * I is then the set of annotations, tag t_l to item i_n by user u_m.

The paper then outlines the following formulation of the recommendation problem according to Adomavicius and Tuzhilin (2005):

For a totally ordered set ℜ, one can define a utility function g, g: U * I ➙ ℜ, such that g(u_m,i_n) measures the gain of usefulness of item i_n to user u_m. The aim of the analysis is then, for each user u ∈ U, to find items i ^{max, u}∈ I, unknown to the user, that maximize g:

∀ u ∈ U, i ^{max, u}= arg max_i_∈Ig(u,i).

In the modeling framework of the paper, content-based recommendation approaches formulate g as the metric:

g(u_m,i_n) = Sim(ContentBasedUserProfile(u_m), Content(i_n)) ∈ ℜ, where,

ContentBasedUserProfile(u_m) = u_m = (u_m,l,........,u_m,K) ∈ ℝ^k, k ∈ ℕ, is the content-based preferences of user u_m (i.e. described in assumption one).
Content(i_n) = i_n = (i_n,l,........,i_n,K) ∈ ℝ^k, k ∈ ℕ, is the set of content features characterizing item i_n (i.e. described in assumption two).

The vectors can then be represented as vectors of real numbers, called weights, in which each array element provides a measure of the importance of the corresponding feature in the user and item model representations. The similarity function Sim() then computes the similarity between a user profile and the item profile in the content feature space.

The next critical component is then to consider the tags as the content features that describe both the user and item profiles (as per assumptions one and two). In our MapReduce formulation these descriptions are housed in the elements computed in the first phase.

These elements allow for the study of the weighting schemes outlined in the paper. The schemes are the TF Profile Model, TF-IDF Profile Model and the Okapi BM25 Profile Model.

The tag weighting schemes can then be used to construct the similarity (utility) functions. In our MapReduce formulation, this is the second phase.

The basic profile model defines the profile of user u_m as a vector
u_m = (u_m,1,........,u_m,L), where u_m,l = |{(u_m,t_l,i) ∈ A | i ∈ I}| is the number of times a user u_mhas annotated items with tag t_l.

The item profile of i_nis the vector i_n = (i_n,l,........,i_n,L), where i_n,l = |{(u,t_l,i_n) ∈ A | u ∈ U}| is the number of times item i_nhas been annotated with tag t_l.

The important point to note in this definition is that both the user and item profiles are defined from components of the A set, which can be constructed from the Bookmarkassignments dataset. In this illustration we use the simple formulation of defining the items to be the bookmarked URL's (in the Bookmarks file) in order to use the Bookmarkassignment dataset as our A set.

In the next step we use the methodology of the paper to define the other profile models from the basic profile model. These involve the following elements, which constitute phase one of our MapReduce approach.

The TF User and TF Item Profile

The TF User and TF Item Profiles can be constructed as follows.

TF User profile (user u_m and tag t_l)

TF Item profile (item i_n and tag t_l)

The TF-IDF User and TF-IDF Item Profile

The TF-IDF User and TF-IDF Item Profiles can be constructed as follows.

TF-IDF User profile (user u_m and tag t_l)

TF-IDF Item profile (item i_n and tag t_l)

Okapi BM25 User and Okapi BM25 Item Profile

The Okapi BM25 User and Okapi BM25 Item Profiles can be constructed as follows.

Okapi BM25 User Profile (user u_m and tag t_l)

Okapi BM25 Item Profile (item i_n and tag t_l)

The values b and k₁ are the standardized values, 0.75 and 2, respectively.

TF-based Similarity

In terms of the similarity measures, the TF-based similarity metrics that can be constructed from the basic model are as follows:

The similarity measures constitute the first two MapReduces in phase two. The maxima for scaling the measures can be obtained from the resulting files using Spark.

2. Prepare the data

The dataset used is the user_taggedbookmarks dataset and has the following columns:

userID
bookmarkID
tagID
day
month
year
hour
minute
second

The first three columns can be used to construct the A set entries. The dataset is composed of 437593 observation tuples (rows). The MapReduces generally involve constructing key indices using the columns of the A set (phase one) and key value pairs from the processed data (phase two). For example, for the first MapReduce, to determine the tag assignments by user u_m for each tag t_l (i.e. u_m,l ) we construct the index u_m;t_l; (from the first and third columns) and conduct Hadoop/Spark word count. This provides the u_m,l element in the paper elements.

An analogous approach can be followed for the i_n,l using the second and third columns to define the index i_n;t_l; and conduct Hadoop/Spark word count.

The fifth and sixth elements, namely, the user and item profile sizes (i.e. |u_m| and |i_n|, respectively) can be determined by defining the first and second columns, respectively, as the MapReduce keys (index) and conducting the Hadoop/Spark word counts. The output files from the word counts yield M, the number of users (in the A set) and N the number of items. One can then additionally, use the output files to construct avg(|u_m|) and avg(|i_n|) required for the Okapi BM25 Profile model weights.

The third and fourth elements, can be constructed using the M and N from the output files of the third and fourth MapReduce. The six components can then be used to construct the remaining Profile model weights. The components are then used as input components to the two sum MapReduce jobs that are used to construct the first two similarity measures.

3. Prepare the Mapper-Reducer sets

The Mapper-Reducer sets in the illustration are prepared in Ruby, Python, R and Perl.

Ruby mapper-reducer set

The Ruby mapper-reducer set was prepared using the tutorials in this post and this post.

Ruby mapper

Ruby reducer

Perl mapper-reducer set

The Perl mapper-reducer set was prepared using the tutorials in this post and this post.

Perl mapper

Perl reducer

R mapper-reducer set

The R mapper-reducer set was prepared using the tutorials in this gist and this post.

R mapper

R reducer

Python mapper-reducer set

The Python mapper-reducer set was prepared using the tutorials in this post and this post.

Python mapper

Python reducer

4. Process the data in Hadoop and Spark

PySpark WordCount Application

The first MapReduce (Simple PySpark Word Count Application) can conducted on a column of data composed of an index created using the first and third columns of the A set. In order to conduct the MapReduce the following arrangements should be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Choose the data delimiter to be a space.

The following program, prepared using the tutorial in the Spark Examples website, can be saved in a Python file (SimplePysparkWCApp.py).

The MapReduce can be run using the following command in Ubuntu 14.04.3 obtained from the Spark Quick Start website.

These are the contents of the output file.

Scala Spark-shell WordCount

The second MapReduce (Scala Spark-shell Word Count program) can be conducted on a column of data composed of an index created using the second and third columns of the A set. In order to conduct the MapReduce the following arrangements should be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Choose the delimiter to be a space.

The following Simple program, prepared using the tutorial in the Spark Examples website can be run in the Scala Spark-shell (local mode on four cores).

These are the contents of the output file.

Hadoop Streaming (Ruby mapper-reducer set)

The third MapReduce (Ruby mapper-reducer set Hadoop Streaming) can be conducted on a column of data composed of an index created using the first column of the A set. In order to conduct the MapReduce it is necessary to make the following arrangements.

Input data file: InputData.txt

HDFS input data folder: HDFS Input Data Folder

HDFS output folder: HDFS Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Hadoop Streaming jar: hadoop-streaming-2.6.0.jar

Local system Hadoop Streaming jar folder: Local system Hadoop streaming jar Folder

Mapper file: Profile_mapper.rb

Reducer file: Profile_reducer.rb

Choose the delimiter to be a space.

The MapReduce can be run using the following command in Ubuntu 14.04.3.

These are the contents of the output file.

PySpark Pipe WordCount Application (Ruby mapper-reducer set)

The fourth MapReduce (Simple PySpark Pipe Word Count Application) can be conducted on a column of data composed of an index created using the second column of the A set. In order to conduct the MapReduce the following arrangements should be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Mapper file: Profile_mapper.rb

Reducer file: Profile_reducer.rb

Choose the delimiter to be a space.

The following program, prepared using the tutorial in this post, the Databricks Guide/Tutorials on the book and the Spark Quick Start website, can be saved in a Python file (SimplePysparkPipeWCApp.py) in Ubuntu 14.04.3.

The MapReduce can be run using the following command in Ubuntu 14.04.3 obtained from the Spark Quick Start website.

These are the contents of the output file.

Hadoop Streaming (Python mapper-reducer set)

The fifth MapReduce (Hadoop Streaming) can be conducted on the outputs of the first MapReduce (i.e. u_m,lfrequencies) for user 8 summed over the non-zero i_n,lfor each item i_n. This can be achieved by attaching user 8’s u_m,l = f_{u_m}(t_l) (tag frequencies) to the non-zero i_n,lfor each tag t_lin the i_nprofiles. This is the method of Noll and Meinel (2007). The sums can then be scaled, in line with the methodology proposal in the paper (i.e. Cantador, Bellogin and Vallet, 2010), to values between 0 and 1 by dividing with the maximum numerator sum value.

In order to run the MapReduce the following arrangements need to be made.

Input data file: InputData.txt

HDFS input data folder: HDFS Input Data Folder

HDFS output folder: HDFS Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Hadoop Streaming jar: hadoop-streaming-2.6.0.jar

Hadoop Streaming jar folder: Local system Hadoop streaming jar Folder

Mapper file: Summapper.py

Reducer file: Sumreducer.py

Choose the delimiter to be a tab.

The MapReduce can be run using the following command in Ubuntu 14.04.3.

These are the contents of the output file.

PySpark Pipe Sum Application (Python mapper-reducer set)

The sixth MapReduce (Simple PySpark Pipe Sum Application) can be conducted on the outputs of the second MapReduce (i.e. i_n,lfrequencies) for item 1 summed over the non-zero u_m,lfor each user u_m. This can be achieved by attaching item 1’s i_n,l= f_{i_n}(t_l)(tag frequencies) to the non-zero u_m,lfor each tag t_lin the u_mprofiles. The sums can then be scaled, in line with the methodology proposal in the paper (i.e. Cantador, Bellogin and Vallet, 2010), to values between 0 and 1 by dividing with the maximum numerator sum value.

In order to run the MapReduce the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Mapper file: Summapper.py

Reducer file: Sumreducer.py

Choose the delimiter to be a tab.

The MapReduce can be run using the following command in Ubuntu 14.04.3 obtained from the Spark Quick Start website.

These are the contents of the output file.

5. Check the results (Hadoop, MongoDB and Spark)

MongoDB

The first MapReduce can be checked by reading an index created using the first and third columns of the A set into MongoDB and conducting the MapReduce.

Once the data is in MongoDB one can run the following commands prepared using the tutorial in this post, to view the collection UserID_TagID in database Delicious.

UserID tag specific counts

The MapReduce for the User 8 tag 1 counts can be conducted by running the following program prepared using the tutorial in this post.

This will generate the following output.

The MapReduce programs and output for the remaining MapReduces have the same structure.

ItemID tag specific counts

The Item 1 tag1 counts.

UserID total tag counts (User Profile size)

The UserID 8 tags counts.

ItemID total tag counts (Item Profile size)

The Item 1 tag counts.

Scala Spark-shell Pipe Sum program (R mapper-reducer set)

The results of the fifth MapReduce of the TF-based User similarity metric can be checked by running a Scala Spark Pipe Sum MapReduce program. In order to run the MapReduce the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Mapper file: Summapper.R

Reducer file: Sumreducer.R

Choose the delimiter to be a tab.

The next step is to run the following Scala Spark Pipe Sum MapReduce program prepared using the tutorial in this post and the Spark Quick Start website (Spark local mode on four cores).

These are the contents of the output file.

PySpark Pipe Sum Application (Perl mapper-reducer set)

The PySpark Pipe Sum Application/program with the Perl mapper-reducer set can be used to check results of the sixth MapReduce of the TF-based Item similarity metric. The procedure is thus to change the mapper file and reducer file reference in the program to the Summapper.pl and Sumreducer.pl files, respectively (instead of Summapper.py and Sumreducer.py). In order to run the MapReduce the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

Local system output folder: Local system Output Folder

Local system mapper folder: Local system mapper Folder

Local system reducer folder: Local system reducer Folder

Mapper file: Summapper.pl

Reducer file: Sumreducer.pl

Choose the delimiter to be a tab.

These are the contents of the output file.

6. Query/Analyze the results using MongoDB, SparkR and Spark SQL (Scala Spark-shell)

The outputs from the MapReduce can then be used to generate datasets containing the key-value columns (i.e. separate column for keys and for values) of the Profile Size, TF Profile, TF-IDF Profile, Okapi BM25 Profile (User and Item). The results can also be used to generate the TF-based similarities.

In this illustration we will take a look at the utility function for User 8 and Item 1.

SparkR (User Profile Size, Item Profile Size and Maxima)

The first step is to extract the Profile sizes and maxima using SparkR. This can be done from the output files from the third and fourth MapReduce. An alternative is to extract them from the A set. In order to conduct the SparkR query the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder
Choose the delimiter to be a space.

The User Profile size for user 8, maximum Profile size, and average Profile size can be obtained using this simple program prepared using the tutorials in this post, the SparkR Programming Guide website and this post (i.e. preparing the metrics from the A set user index/key from the MapReduce three source file).

This will generate the following output.

The Item Profile Size for item 1, maximum Profile size, and average Profile size can be obtained by replacing the input file (with the A set database item index/key from the MapReduce four source file) and the last three lines of the above code replaced with these three lines of code (prepared from the same sources).

This will generate the following output.

MongoDB (TF, TF-IDF and Okapi BM25 Profile weights)

The components from the four MapReduces are part of the MongoDB checks above. The Profile size metrics were generated using SparkR. The M and N were obtained from the output files from the third and fourth MapReduces. For example, the User 8 Profile size is 153, the Item 1 Profile size is 5, the User 8 tag 1 frequency is 28 and Item 1 tag 1 frequency is 2.

The information available can be used to generate the (TF, TF-IDF and Okapi BM25) Profile weights for User 8 and Item 1 for tag1 using the formulae from the paper (Cantador, Bellogin and Vallet, 2010) as follows.

The TF User and TF Item Profile

TF User profile (user u₈ and tag t_l)

TF Item profile (item i₁ and tag t_l)

The TF-IDF User and TF-IDF Item Profile

TF-IDF User Profile

TF-IDF Item Profile

Okapi BM25 User and Okapi BM25 Item Profile

Okapi BM25 User Profile

Okapi BM25 Item Profile

The values for b and k₁ are the standardized values of 0.75 and 2, respectively.

Spark SQL in Scala Spark-shell (TF-based similarity using User and Item Profile frequencies)

The dataset for the TF-based similarity measure prepared for User 8 (from the fifth MapReduce output file) can be queried using the Spark sqlContext in the Scala Spark-shell. In order to conduct the query, from the output file of the fifth MapReduce, the following arrangements need to be made.

Input data file: InputData.txt

Local system input data folder: Local system Input Data Folder

A query for the User 8 similarity measure frequencies (i.e. the numerator) can be achieved by running the following simple program in the Scala Spark-shell, prepared using the Spark SQL dataframes guide and this post.

This will generate the following output.

The summary statistics from describe function.

The dataset for the TF-based similarity measure for Item 1 prepared in the sixth MapReduce can be queried using the Spark sqlContext in the Scala Spark-shell.

A query for the Item 1 similarity measure frequencies (i.e. the numerator) can be achieved (analogously to the user frequency program) by running the following simple program, prepared using the Spark SQL dataframes guide and this post.

This will generate the following output.

The summary statistics from the describe function.

TF-based Similarity

In terms of the similarity measures, the TF-based similarity, based on the user tag frequencies (for user 8) and item tag frequencies (for item 1), can be constructed from the compiled metrics in the illustration. This is the similarity measure for user 8 and item 18, calculated using user 8 tag frequencies for tags appearing on item 18 profile.

In the case of user 101624 and item 1, the similarity measure calculated using item 1 tag frequencies for tags appearing in user 101624's profile.

7. Conclusions

The post illustrated how one can apply MapReduce to the Delicious dataset and conduct simple query analyses for content-based recommendation. The approach can be further refined according to specific information requirements by developing more specific datasets and program sets. Part two of the post will explore the remaining measures in the paper by Cantador, Bellogin and Vallet (2010).

Interested in other Big Data analyses from Stats Cosmos blog?

Check out my other blog posts

Or, subscribe via RSS to keep updated

Or check out my Course at Udemy College

Or check out the Stats Cosmos resources page

Sources

http://ntrda.me/1q2eCR1
http://bit.ly/1Yzm2qe
http://bit.ly/1YiCkUc
http://bit.ly/1T76xr7
http://bit.ly/1T76ziF
http://bit.ly/1T76zzf
http://bit.ly/1M0puc7
http://bit.ly/1rN3pnV
http://bit.ly/1M0oCUO
http://bit.ly/1NoLrCe
http://bit.ly/1oXvms2
http://bit.ly/1TAOH9I
http://bit.ly/1M0oCUO
http://bit.ly/24HpXok
http://bit.ly/1Q5vX1t
http://bit.ly/1TiHqjD
http://bit.ly/231JrU3
http://bit.ly/1omcG4d
http://bit.ly/1rDzezL
http://bit.ly/1MBuJif
http://bit.ly/1R89nos
http://bit.ly/1OkRxhP
http://stanford.io/1OcLqRZ
http://bit.ly/1T76NpO
http://bit.ly/1UO6dN7
http://bit.ly/1VR19sn
http://ibm.co/1T3h0ml
http://bit.ly/1SZPyVw
http://bit.ly/1s8IU5C
http://bit.ly/219mwIJ
http://bit.ly/1Qc7Gc8
http://bit.ly/1NoLZIf
http://bit.ly/1WlCMnp
http://bit.ly/1nh2Osx

How to apply MapReduce to the MovieLens 1M datasets using Hadoop Streaming, Spark Pipe, Spark Simple Applications and SparkR

2016-03-30T15:30:00.001-07:00

This post is designed for a joint Apache Hadoop 2.6.0 single cluster, Apache Spark 1.5.1 and Ubuntu Server 14.04.3 LTS installation. This is a follow on post to my previous post: How to set up Hadoop Streaming to analyze MovieLens data. In the present post the GroupLens dataset that will be analyzed is once again the MovieLens 1M dataset, except this time the processing techniques will be applied to the Ratings file, Users file and Movies file.

The MapReduce approach has four components. The first is to integrate the GroupLens MovieLens Ratings, Users and Movies datasets. The second is to design the MapReduce processing model. The third is to design a system for checking the results from the processing for consistency and accuracy. The last is to summarize/analyze the results.

1. Prepare the data

The Ratings file has the following variables:

The Movies file has the following variables:

The Users file has the following variables:

The three datasets can thus be linked using the UserID variable and the MovieID variable. Essentially, from a processing point of view, each of the columns can be processed together or individually.

For example, the Ratings and MovieID columns can be processed together to generate the average rating for each MovieID. An example of individual processing is using the gender column to generate the number of ratings by female users and male users.

In this approach, the individual columns selected were Gender, Occupation, Age, ZIP codes and Genres. The MapReduce will be conducted with files containing the individual columns (for example a file containing the Gender column for the Gender MapReduce).

The joint column consideration is the MovieID and Ratings columns. The MapReduce will conducted using a file with the UserID, MovieID and Ratings columns.

The MovieID and Ratings column can also be fused to generate a MovieIDRatings (fused text and number) column that can be used for checking the results. The MapReduce (for this purpose) will be conducted using a file with the fused MovieID and Ratings column.

2. Prepare the mapper and reducer sets

The MapReduce design has two components, a processing component and a checking component.

The MapReduce processing component combines mapper-reducer sets, Apache Hadoop Streaming and Apache Spark Pipe. The MapReduce processing model is housed in the mapper-reducer sets. The data is processed using the mapper-reducer sets within the Apache Hadoop Streaming and Apache Spark Pipe facilities (of Hadoop and Spark, respectively).

A mapper-reducer set prepared using Perl can be used with the Hadoop Streaming facility. Mapper and reducer sets prepared in R, Ruby and Python can be used with the Spark Pipe facility.

The MapReduce checks component firstly runs a joint column consideration of MovieID and Ratings columns to calculate the Average Ratings per MovieID (i.e. replicates the Spark Pipe run) in Hadoop Streaming. The next step is to run Spark Simple Applications prepared in Java and Python on all the column datasets (including the fused MovieID and Ratings column). The procedure is repeated using SparkR. The last check is to run the Hadoop Grep worked example on the fused MovieID and Ratings column data.

Perl mapper-reducer set

The Perl mapper-reducer set was prepared using the tutorial in this post.

The mapper.

The reducer

Ruby mapper-reducer set

The Ruby mapper-reducer set was prepared using the tutorial in this post.

The mapper.

The reducer

R mapper-reducer set

The R mapper-reducer set was prepared using the tutorial in this post.

The mapper.

The reducer

The Python mapper-reducer set

The Python mapper-reducer for the MovieID ratings average set was prepared using the tutorial in this post.

The mapper.

The Python reducer

Once the mapper-reducer sets have been prepared the data can be processed in Hadoop and Spark.

3. Process the data in Hadoop and Spark

In this processing design, the Gender variable is processed in Hadoop using the Streaming facility. The remainder of the variables are processed in Spark using the Pipe facility.

Hadoop Streaming

The Gender variable MapReduce uses the mapper-reducer set prepared in Perl.

Gender variable

In this step one can make the following arrangements/selections.

Input data file: InputData.txt

Input data folder in Hadoop Distributed File System (HDFS): <HDFS Input folder>

Local system mapper folder: <Local system mapper folder>

Local system reducer folder: <Local system reducer folder>

Hadoop Streaming jar file: hadoop-streaming-2.6.0.jar

Local system Hadoop Streaming jar file folder: <hadoop-streaming-2.6.0.jar file local folder>

Output folder in HDFS: <HDFS Output folder>

Mapper file: mapper.pl

Reducer file: reducer.pl

Then one can run the following command on Ubuntu 14.04.3. LTS

This will generate the following results.

Spark Pipe

The data can be processed using the R, Ruby and Python mapper-reducer sets in Spark using the Spark Pipe facility. A simple program can be prepared using the tutorial in this post.

The datasets for the Age, Occupation, Genre and Zip code variables are assumed to have a semi-colon at the end of the values. This is used for the line split for creating the first Resilient Distributed Dataset (RDD) in line 2 of the program.

An excerpt of the first three lines from the Occupations data file (Occupations.txt) is as follows.

The “;” at the end of each value is placed to match the lines.split(“;”) method in line 2 of the simple program. The files for the other user variables can be prepared analogously for the program.

Occupations

The next step is to make the following arrangements/selections.

Input data file: InputData.txt

Input data folder on local system: <Local system Input data folder>

Output data folder in local system: <Local system Output data folder>

Local system mapper folder: <Local system mapper folder>

Local system reducer folder: <Local system reducer folder>

Mapper file: mapper.R

Reducer file: reducer.R

One can then run the following simple program prepared using the tutorial in this post.

This will result in the following output for the Occupations variable.

Age

For the Age variable, the Occupation variable Spark Pipe arrangement can be run using the Age variable file as the input data file and the R mapper-reducer set replaced with the Ruby mapper-reducer set.

This will generate the following results.

Genre

For the Genre variable, the Occupation variable Spark Pipe code arrangement can be run using the Genre variable file as the input data file.

This will generate the following results.

Zip codes

For the ZIP codes variable the Occupation variable Spark Pipe code arrangement can be run using the ZIP code variable file as the input data file and the R mapper-reducer set replaced with the Ruby mapper-reducer set.

This will generate the following results.

Movie Average Rating

In the case of the Python Average mapper-reducer set one needs a different Spark Pipe program and a minor modification to the Input dataset. The colon needs to be used for the line split instead of the semi-colon at the end of the line (i.e. the data needs to have a colon after the third column).

The second consideration is the .coalesce(1) transformation in the pipe in line 5 instead of the .repartition(1) transformation. The coalesce transformation can also be used for the program for the other Spark Pipe program above (in order to get one data partition instead of two).

An excerpt from the MovieID Average Ratings (input) data file is as follows.

The input data adjustment is to match the lines.split(“:”) method in line 2 of the simple program. One can then run the following simple program.

This will generate the following results.

4. Check the results of the processing

It is a good idea to check the results of the MapReduce before proceeding with the summary. The Spark submit facility, Hadoop Streaming facility, Hadoop examples, Spark examples and SparkR have some extremely useful functionalities that can be used for checks.

At the most basic level one can run the Hadoop or Spark wordcount code. In this illustration the approach used makes use of Hadoop Streaming, Hadoop Grep example, Java Simple Application, Python Simple Application and SparkR. The results can be used to check the results from the Hadoop Streaming and Spark Pipe processing above.

Simple Applications

The code for the Java Simple Application can be obtained from the Spark 1.6.1 examples website and is as follows for the gender variable.

For the Simple Applications the following arrangements can be made:

Input data file: InputData.txt

Input data folder local system: <Local system Input data folder>

Java Simple Application folder local system: <Java Simple Application folder>

Python Simple Application folder local system: <Python Simple Application folder>

Java Simple Application jar file: SimpleApplication.jar

Python Simple Application file: SimpleApplication.py

Then one can run the following code at $YOUR_SPARK_HOME/bin for the Java Simple Application check (using the ./bin/spark-submit to execute).

The equivalent code for the Python Simple Application can be obtained from the Spark 1.6.1 examples website and is as follows.

One can run the following code at $YOUR_SPARK_HOME/bin for the Python Simple Application check (using the ./bin/spark-submit to execute).

The code for the Simple Applications can be adapted to include more variables as required. This is the output that will be obtained for the Gender variable (i.e. same input file as the Perl Hadoop Streaming) from the Spark submit of the Java Simple Application.

This is the output for the Age variable (Java Simple Application).

This is the output for the Occupation variable (Java Simple Application).

This is the output for the Zip codes variable (Java Simple Application).

This is the output for the Genre variable (Java Simple Application).

This is the output for the fused MovieID and Ratings column (Java Simple Application).

The average for the MovieID can be calculated from the output using this formula.

where i is the rating and the number of ratings are obtained from the MapReduce counts (of the fused MovieID-Rating variable).

This is the output obtained using the fused MovieID and Ratings column for the Python Simple Application.

Apache Hadoop Grep example

In order to run the Hadoop examples Grep code the following arrangements can be made.

Input data file: InputData.txt

Input data folder in HDFS: <HDFS Input folder>

Hadoop Mapreduce examples jar file: hadoop-mapreduce-examples-2.6.0.jar

Local system Hadoop MapReduce examples folder: <Local system Hadoop examples folder>

Output folder in HDFS: <HDFS Output folder>

The Apache Hadoop examples Grep code can be run on the fused column data using the following command outlined in this post.

This will generate the following output.

Ubuntu 14.04.3 LTS and HDFS

The next step is to use Ubuntu 14.04.3. LTS to scroll to the 3645 MovieID average in the Spark Pipe MovieID average output file for the check. This can be done using the cat | less command outlined in this post.

The next step is to remove the colon at the end of the line in the UserID, MovieID and Ratings input dataset (i.e. on the right of the Ratings variable). The input data can then be processed using the Python mapper-reducer set in the Hadoop Streaming arrangement (i.e. Perl arrangement) to obtain the MovieID average ratings output in HDFS.

The HDFS version of the cat | less command can be run on the output file processed using the Python average mapper-reducer set through the Hadoop Streaming facility. The average rating for the 3645 MovieID can be identified by scrolling through the HDFS output using this command outlined in this post.

The checks above are useful for checking a few values, however, when one is interested in running structured (and larger) checks one can use SparkR.

SparkR

Essentially, the best way of running the checks in the present scenario is by reading the output datasets from the MapReduce and running the checks.

An alternative method is to run a fresh (and independent) Spark wordcount from the source data. This can be done by reading the Gender column, Occupation column, Age column, Zip code column, Genre column and the fused MovieID-Ratings column into SparkR (individually) from a local system folder. A SparkR dataframe can then be constructed and used to run SparkR SQLContext queries.

One essentially needs to make the following arrangements.

Input data file: InputData.txt

Input Data folder in local system: <Local system Input data folder>

The check can be run using the following simple program obtained from the tutorial in this post, this post, the SparkR programming guide, the SparkR API documentation and the Spark SQL guide.

This is the output for the Gender column (processed individually).

This is the output for the Occupation column.

This is the output for the Age column.

This is the output for the Zip code column.

This is the output for the Genre column.

In the case of the fused MovieID-Ratings column data the above code and additional code needs to be run. The additional code needs to be run in order to get a subset of the results for the counts belonging to the 2831 and 3645 MovieIDs. The additional code is as follows.

This will generate the following output that can be used for the checks.

The tools and their results are extremely useful for checking the results of the MapReduce. The next step is to generate the results for the summaries/analyses.

5. Summarize/Analyze the data

The MapReduce results can be summarized using bar charts and basic summary statistics. The bar charts can be generated using PROC SGPLOT in the SAS software using the method outlined in this post. The summary statistics from PROC UNIVARIATE in the SAS software can be used to generate some additional checks on the MapReduce results.

The bar chart for the gender variable shows that there are roughly three more times more ratings made by male users than female users.

The additional checks for the Gender MapReduce from the summary statistics are N=2 and Sum Observations 1000209.

The Age code with the most number of ratings is the 25.

The additional checks for the Age variable MapReduce are N=7 and Sum Observations 1000209.

The Occupation code with the most number of ratings is other or unspecified.

The additional checks for the Occupational variable MapReduce are N=21 and Sum Observations 1000209.

The Genre category with the highest number of ratings is the Comedy genre.

The additional checks for the Genre variable MapReduce are N=301 and Sum Observations 1000209.

The ZIP code with the most ratings is 94110 (San Francisco, California).

The additional checks for the Zip codes MapReduce are N=3439 and Sum Observations 1000209.

The highest MovieID average rating is 5.

The additional check for the MovieID Averages MapReduce is N=3706.

The post gives a brief illustration of the kind of MapReduce processing designs that can be applied to the GroupLens MovieLens 1M datasets. I hope this post proves useful in your big data analyses, especially in the building of Spark Simple Applications.

Interested in other big data and cloud computing resources from the Stats Cosmos blog?

Check out my previous posts?

Or subscribe to the Stats Cosmos RSS feeds to keep updated

Or check out Stats Cosmos Services

Or check out my course at Udemy College

Or check out the Stats Cosmos blog resources page

Sources

http://bit.ly/1q2ctoH

http://bit.ly/1pKFYup

http://bit.ly/21TFhLr

http://bit.ly/1PFHzGM

http://bit.ly/1RythtQ

http://bit.ly/1QEh5pW

http://bit.ly/1V5cK57

http://bit.ly/1oXvms2

http://bit.ly/1omcBNO

http://bit.ly/1TiHqjD

http://bit.ly/1omcG4d

http://bit.ly/231JrU3

http://bit.ly/1M0orIV

http://bit.ly/1TiHK1B

http://bit.ly/1M0oCUO

http://bit.ly/1MBuJif

http://ntrda.me/1q2eCR1

http://ibm.co/1T3h0ml

http://bit.ly/219mwIJ

http://bit.ly/1Qc7Gc8

http://bit.ly/1PFIW8p

http://bit.ly/1M0puc7

http://bit.ly/1ROM1e9

http://bit.ly/1ome6vv

http://bit.ly/1SmNqD0

http://bit.ly/1SxgxGm

http://bit.ly/1UUEZop

http://bit.ly/1WXVkGR

http://bit.ly/21TGWAx

http://bit.ly/1RKV5dQ

http://bit.ly/1qj52Jq

http://bit.ly/1Y0Q59A

http://bit.ly/1ZKXOKg

http://bit.ly/1M0qSLy

http://bit.ly/1WXW5zC

How to summarize the Book-Crossing dataset using Hadoop 2.6.0 and Spark 1.5.1

2016-03-09T19:23:00.000-08:00

This post is designed for a joint installation of Ubuntu Server 14.04.3 LTS, Apache Hadoop 2.6.0 single cluster and Apache Spark 1.5.1 (pre-built for Hadoop 2.6 and later). The steps outlined involve setting up a job to summarize the GroupLens Book-Crossing dataset using Hadoop Streaming and Spark word count (in local mode). The Hadoop Streaming part of the job is implemented using mapper-reducer sets prepared in Python and Ruby. The Spark word count part of the job is implemented in the SparkContext within the Scala Spark shell (in local mode).

The Book-Crossing dataset is a great information source for book ratings. The information is particularly useful when analyzed in relation to the GroupLens MovieLens datasets and other GroupLens datasets.

1. Prepare the data

The Book-Crossing data was collected by Cai-Nicolas Ziegler in a 4-week crawl (during the August/September 2004 period) from the Book-Crossing community with the kind permission of Ron Hornbaker (CTO of Humankind Systems). The reference to the dataset also has excellent additional resources for methods on Book rating related analyses.

The specific dataset considered in this illustration is the BX-Book-Ratings dataset. The dataset version selected for the illustration is the CSV Dump version of the BX-Book-Ratings dataset. The other version that is available is the SQL Dump.

The BX-Book-Ratings dataset contains three variables:

UserID
ISBN
Rating (for each ISBN)

The summary of the notes on the BX-Book-Ratings variable is as follows:

Ratings are either explicit, expressed on a scale from one to ten (with higher values denoting higher appreciation), or implicit, expressed by 0.

In this illustration I will show how to determine the number of UserIDs that rated, the number of ISBNs that were rated, the average rating for each ISBN and the number of ratings in each ISBN rating score category.

In a simple MapReduce approach to the job, this amounts to four jobs. These are:

Counting the number of UserIDs in a UserID text file created from the first column of the dataset
Counting the number of ISBNs in a UserID text file created from the second column of the dataset
Calculating the average rating for each ISBN in a text file created from the second and third columns (or all the columns) of the dataset
Calculating the number of times each score rating value occurs in a text file created from the third column of the dataset

The approach will be to show how to complete the jobs in the following manner:

Complete the first job in Spark running on local mode using Resilient Distributed Dataset (RDD) processing methods
Complete the second and third jobs using Hadoop Streaming with mapper-reducer sets written in Python
Complete the fourth job using Hadoop Streaming with a mapper-reducer set written in Ruby

The resulting output files yield the required information for analysis (or further processing).

In this illustration the ISBN text file must be created in such a manner that the ISBNs are more than one character long (i.e. all field lengths must be greater than one).

The average rating for each ISBN is obtained by using the ISBN column and the Ratings column of the dataset. In this illustration the whole dataset is used for the third job (i.e the UserID column is also included with the ISBN and Ratings columns). The rationale for this approach is to take advantage of the linked nature of the dataset.

For example, the same dataset can be used (with some minor adjustments to the Python code or by using another mapper-reducer set) to calculate the average of the ratings made by each user.

The average ISBN rating mapper-reducer set Python code can, however, be easily modified to exclude the UserID column if a more compact dataset is desired for the job.

The four resulting datasets prepared in this manner can be processed using Hadoop and Spark. The next step is to prepare the mapper-reducer sets for the Hadoop Streaming part of the job. The Spark part of the job can be run using a simple program (within a SparkContext) in the Scala Spark shell run in local mode.

2. Prepare the mapper and reducer sets

Python Mapper and Reducer set for the ISBN counts

The Python mapper-reducer set for the ISBN counts was prepared using the tutorial in this post.

The mapper.

The reducer.

Python Mapper and Reducer set for the ISBN rating averages

The Python mapper-reducer set for the ISBN average ratings was prepared using the tutorial in this post.

The mapper.

The reducer.

Ruby Mapper and Reducer set for the rating category counts

The Ruby mapper-reducer set for the rating category counts was prepared using the tutorial in this post.

The mapper.

The reducer.

The mapper-reducer sets complete the set up for the Hadoop Streaming part of the job. The next step is to process the datasets in Hadoop and Spark.

3. Process the data in Hadoop and Spark

The Hadoop Streaming configuration that will used in the illustration is the standard set up without any modifications to enhance performance. The Spark shell set up in Scala is the standard local mode set up with four cores without any jar files and/or Apache Maven dependencies (using Maven coordinates).

UserID counts using the Scala Spark shell

The Scala Spark shell running in local mode creates a special interpreter-aware SparkContext in the variable called sc. The RDD for the job is created using the SparkContext’s textFile method on the UserID column text file from the local file system. The results are saved on a text file in the local file system. The Spark word count example can be found in Apache Spark Examples website.

The assumption in the job is that the UserID column text file is called InputData.txt, has been saved in the local folder- <InputData folder> and the local system output folder has been selected to be - <Output folder>.

The word count is implemented by running the following simple program within the Scala Spark shell.

This is an excerpt from my results.

ISBN counts using Hadoop Streaming (Python Mapper-Reducer set)

In this section the assumption is that the ISBN column text file is called InputData.txt.

In terms of the Hadoop Streaming part of the code the assumption is that the InputData.txt file has been successfully loaded into the Hadoop Distributed File System (HDFS) input folder - <HDFS Input folder>, the Hadoop Streaming jar file is called hadoop-streaming-2.6.0.jar, the Hadoop Streaming jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper (BCmapper.py) is located in the local system folder - <Python mapper>, the Python reducer (BCreducer.py) is located in the local system folder - <Python reducer>, and the output folder in HDFS has been selected to be - <HDFS Output folder>. The next step is to run the following command on Ubuntu Server.

This is an excerpt from my results.

ISBN average rating using Hadoop streaming (Python Mapper-Reducer set)

In this section the assumption is that the UserID, ISBN column and Rating column text file is called InputData.txt.

In terms of the Hadoop Streaming part of the code the assumption is that the InputData.txt file has been successfully loaded into the HDFS input folder - <HDFS Input folder>, the Hadoop Streaming jar file is called hadoop-streaming-2.6.0.jar, the Hadoop Streaming jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper (Bavgmapper.py) is located in the local system folder - <Python mapper>, the Python reducer (Bavgreducer.py) is located in the local system folder - <Python reducer>, and the output folder in HDFS has been selected to be - <HDFS Output folder> . The next step is to run the following command on Ubuntu Server.

This is an excerpt from my results.

Rating counts using Hadoop streaming (Ruby Mapper-Reducer set)

In this section the assumption is that the Rating column text file is called InputData.txt.

In terms of the Hadoop Streaming part of the code the assumption is that the InputData.txt file has been successfully loaded into the HDFS input folder - <HDFS Input folder>, the Hadoop Streaming jar file is called hadoop-streaming-2.6.0.jar, the Hadoop Streaming jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder>, the Ruby mapper (Bmapper.rb) is located in the local system folder - <Ruby mapper>, the Ruby reducer (Breducer.rb) is located in the local system folder - <Ruby reducer>, and the output folder in HDFS has been selected to be - <HDFS Output folder> . The next step is to run the following command on Ubuntu Server.

This is an excerpt from my results.

4. Results summary and highlights

The results yielded some very interesting and wonderful book ratings information. The results can be nicely summarized using bar charts, summary statistics and histograms. The bar charts and histograms can be generated in R using the lessR package. The specific functions are BarChart for the bar chart and Histogram for the histogram. The summary statistics can be generated in R using the stat.desc function in the pastecs package.

UserID Ratings

The UserID with the most ratings was UserID 11676 with 13602 ratings.

The basic summary statistics provide analysis information for the UserID rating counts and diagnostic checks for the Spark job. The key metric is the 1149780 rating counts total. The number of UserIDs (nbr.val of 105283) reflect the anonymized UserIDs.

ISBN Ratings

The ISBN with the most number of ratings was ISBN 971880107 with 2504 ratings.

The basic summary statistics provide analysis information for the ISBN rating counts and diagnostic checks for the Python Hadoop Streaming job. The key metrics are the 1149780 rating counts total and the number of ISBNs (nbr.val) of 339528. The number of ISBNs (nbr.val) balance with number of ISBNs from the Python Hadoop Streaming job for the rating averages.

ISBN Rating Averages

The histogram of rating averages shows that the implicit average rating had the most counts.

The basic summary statistics provide analysis information for the ISBN rating averages and diagnostic checks for the Python Hadoop Streaming job. The key metric is the nbr.val of 339528 (number of ISBNs) which balance with the Python Hadoop Streaming job for the ISBN counts.

ISBN Rating counts

The ISBN rating value counts shows that the implicit rating had the most counts.

The basic summary statistics provide analysis information for the ISBN rating average counts and diagnostic checks for the Ruby Hadoop Streaming job. The key metric is the sum of 1149780 (number of ratings) which balance with the Spark job UserID rating counts total and the Python Hadoop Streaming job ISBN rating counts total. The availability of metrics for diagnostic checks across jobs illustrate the advantages of the linked nature of the dataset.

Summary

The illustration shows how to set up a scheme to extract some basic results from the Book-Crossing Book-Ratings dataset. The dataset is very rich in information about book user ratings. The scheme was designed to yield results that can be used with results from the (GroupLens) MovieLens and other GroupLens datasets. For example, some movies in the MovieLens datasets have book counterparts in the Book-Crossings dataset and vice versa.

The results combine will provide for a richer analysis of user ratings across the products in the GroupLens datasets. I hope this post proves useful in your big data analyses.

Interested in other posts about Hadoop Streaming and cloud computing?

Check out my other posts.

Or subscribe to the Stats Cosmos RSS feeds to keep updated

Alternatively, check out the Stats Cosmos services page.

Stats Cosmos services

Or our blog resources page and training course.

Stats Cosmos blog resources

Stats Cosmos training

Sources:

http://bit.ly/1LSkoOI
http://bit.ly/1Yzm2qe
http://bit.ly/1M7MAYL
http://bit.ly/1R89nos
http://ibm.co/1T3h0ml
http://bit.ly/1Qc7Gc8
http://bit.ly/1oXvms2
http://bit.ly/1SN27EA
http://bit.ly/1p9hgn7
http://bit.ly/1Watnek
http://bit.ly/1QNk68v
http://bit.ly/1nh2Osx

How to set up Hadoop Streaming to analyze MovieLens data

2016-02-28T15:47:00.000-08:00

This post is designed for an Apache Hadoop 2.6.0 single cluster installation. The job uses a Hadoop Streaming design with C++, Ruby and Python. The MapReduce configuration is a standard or simple configuration without any tweaking on the streaming, mappers and reducers for enhanced performance. The dataset used is the GroupLens MovieLens 1M dataset. The file used is the ratings file (and the README.txt file).

1. Prepare the data

The data for the analysis is relatively easy to prepare. It can be imported (after extraction from the zipped format) into Microsoft Excel. The file can then be arranged into four columns (whose metadata can be obtained from the README.txt file packaged with the data). These are:

UserIDs range between 1 and 6040

MovieIDs range between 1 and 3952

Ratings are made on a 5-star scale (whole-star ratings only)

Timestamp is represented in seconds since the epoch as returned by time(2).

There are a number of MapReduce jobs that can be set up on the dataset. The one selected for this illustration is to determine the following metrics:

Number of ratings made by each UserID

Number of ratings for each MovieID

Average rating for each MovieID

Number of ratings in each score category in the list {1,2,3,4,5}.

The data can be arranged into four files for the job. The UserID column can be extracted into a text file for analysis using wordcount MapReduce. The rationale for the approach is that number times a UserID occurs in the file column is equal to the number ratings made by the UserID. The scheme for generating the number of ratings for each movie follows analogously for a MovieID column text file.

The average rating calculation outlined uses all four columns of the data in csv format, however, the Python (mapper/reducer) code can be easily adapted to include only the MovieID column and the Ratings column. The number of ratings for each score category can be determined (analogously to the UserID and MovieID approach) by using the ratings as words (instead of numbers).The Timestamp variable is not used in the illustration.

Once the four files have been prepared they can be loaded into the Hadoop Distributed File System (HDFS) for the job. The next step is to prepare the mapper-reducer sets for the job.

2. Prepare the Mappers and Reducers

In this illustration the four mapper-reducer sets were prepared in three scripting languages (C++, Ruby and Python).

C++ mapper-reducer set

The mapper-reducer set for the UserID wordcount MapReduce job was prepared in C++ according to the tutorial in this blog post.

C++ Mapper

C++ Reducer

The mapper.cpp file and reducer.cpp file have to be compiled into *.out files using the following commands on Ubuntu 14.04.3 LTS.

These will generate a mapper.out file and a reducer.out file which can be used in the Hadoop streaming.

Ruby mapper-reducer set

The mapper-reducer set for the MovieID and Rating category wordcount MapReduce jobs was prepared in Ruby according to the tutorial in this blog post.

Ruby Mapper

Ruby Reducer

Python mapper-reducer set

The mapper-reducer set for the Average Ratings MapReduce job was prepared in Python according to the tutorial in this blog post.

Python Mapper

Python Reducer

It is important to run the chmod procedure for each file created in each of the mapper-reducer set creation processes. The mapper-reducer sets can then be saved in appropriate folders in the Hadoop local environment. The next step is to process the data in Hadoop using streaming.

3. Process the data in Hadoop

UserID file

The assumption is that the UserID input dataset (in HDFS) is called InputData.txt. In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder> , the C++ mapper is located in local system mapper folder - < mapper folder>, the C++ reducer is located in local system reducer folder - < reducer folder>, and the HDFS output name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

This is an excerpt of my UserID rating counts output (6040 in total).

MovieID file

The assumption is that the MovieID input dataset (in HDFS) is called InputData.txt. In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder> , the Ruby mapper is located in local system mapper folder - < mapper folder>, the Ruby reducer is located in local system reducer folder - < reducer folder>, and the HDFS output name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

This is an excerpt of my MovieID rating counts output (3706 in total).

MovieID Ratings file

The assumption is that the MovieID Ratings input dataset (in HDFS) is called InputData.csv. In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder> , the Python mapper (KeyAvgmapper.py) is located in local system mapper folder - < mapper folder>, the Python reducer (KeyAvgreducer.py) is located in local system reducer folder - < reducer folder>, and the HDFS output name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

This is an excerpt of my MovieID average ratings output (3706 in total).

Score Rating (category) counts file

The assumption is that the Score Rating (category) counts input dataset (in HDFS) is called InputData.txt. In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in the local system folder - <hadoop-streaming-2.6.0.jar local folder> , the Ruby mapper is located in local system mapper folder - < mapper folder>, the Ruby reducer is located in local system reducer folder - < reducer folder>, and the HDFS output name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

This is an excerpt of my category counts output.

4. Highlights from the results

The results from the MapReduce job were very interesting and exciting. I give a very brief summary from the SAS software PROC SGPLOT procedure using techniques from the tutorial in this post. I also supplemented the illustration with results from the SAS software PROC UNIVARIATE. The top 20 User_ID rating volumes (counts).

The basic summary measures provide useful statistics for analyzing the User_ID rating volumes and also brief checks for the MapReduce job. These are the number of User_IDs that rated, 6040, and the sum of the observations which equals 1000209.

The top 20 Movie_ID rating volumes.

The brief checks for the MapReduce job and summary statistics of the Movie_ID rating volumes.

The top 20 Movie_ID average ratings. The highest average rating was 5. This was the average rating for the top ten movies.

The brief check for the MapReduce job (only the 3706 in this case) and summary statistics of the Movie_ID rating averages.

The category counts are commensurate with the summary statistics of the movie averages.

Summary

The MapReduce programming model provides a very powerful method to process data. The Apache Hadoop Streaming facility provides a way to customize the processing according to programming style and cluster resources.

The MovieLens dataset has a lot of information about user ratings. The MapReduce jobs considered in this post provide a simple way to begin to analyze the dataset.

I hope this post proves useful to you in applying MapReduce and analyzing the MovieLens dataset(s).

Interested in finding out more information about simple approaches to Hadoop Streaming and cloud computing?

Check out my previous blog posts.

Or check out our statistics and e-learning services

Or check out my course at Udemy College

Sources:

http://bit.ly/1QEh5pW

http://fla.st/1Qc7nhC

http://ibm.co/1T3h0ml

http://bit.ly/1Qc7Gc8

How to incorporate Python and R into a Hadoop 2.6.0 MapReduce job using Hadoop Streaming

2016-02-22T11:51:00.001-08:00

This setup guide is designed for an Apache Hadoop 2.6.0 installation. Hadoop streaming is a utility/facility that allows one to create and run MapReduce jobs with any executable or script as the mapper and/or reducer. The functionality is part of the Hadoop distribution. A detailed explanation of Hadoop streaming and Hadoop 2.6.0 can be found in the Apache Hadoop project website. In this post I will explain how to execute the Hadoop 2.6.0 MapReduce examples word count, word mean and word standard deviation. The examples word count, word mean and word standard deviation are also part of the Hadoop distribution. In this scheme, Hadoop Streaming is used for the word count MapReduce instead of the Hadoop distribution word count (implemented in my previous blogpost).

The first part of the post gives the setup and execution of word count using Hadoop Streaming MapReduce. The Hadoop Streaming MapReduce setup has mapper/reducer set in Python script and a set in R script. The second part of the post gives the setup and execution of the word mean and word standard deviation using the standard Hadoop MapReduce.

The MapReduce job is designed to analyze four sets of aggregates. These are the 2014 global population, 2014 global internet user population, 2012 Facebook population and the spatial time series variance-covariance matrix (annual steps) for the global internet user population between the years 2008 to 2014. The analysis of the first three sets of aggregates in Hadoop was for testing purposes and the last set was the main analysis.

In the scheme the word count MapReduce job was implemented using Python for all the sets. The word count MapReduce job using R script was implemented only for the global internet user population spatial time series variance-covariance matrix aggregates (the fourth set). The word mean and word standard deviation standard MapReduce jobs were also only implemented for the fourth set of aggregates.

1. Prepare the data

A detailed account of the aggregates can be found in my previous blog post: 5 matrix decompositions for visualizing the global internet user population spatial time series variance-covariance matrix. The data preparation essentially involved categorizing the aggregates into decile categories (classes). The decile classes are then given word values whose length gives an indication of the size of the figure. For example, the first decile class, namely, decile_one, had a word length that is shorter than that of the second decile class, namely, decile_two_. The naming convention is designed to facilitate the word mean and word standard deviation of the analysis.

Decile classes for the 2014 Global population aggregates

The decile classes for the 2014 global population aggregates are shown in the following table.

Decile classes for the 2014 Global internet user population aggregates

The decile classes for the 2014 global internet user population aggregates are shown in the following table.

Decile classes for the 2012 Facebook user population aggregates

The decile classes for the 2012 Facebook user population aggregates are shown in the following table.

Decile classes for the Global internet user population (2008 to 2014) spatial time series variance-covariance matrix aggregates

In processing the matrix, the first step is to obtain the absolute value for all the entries (in order to handle the cases of negative variance-covariances). It is also worth noting that variance-covariance matrices are symmetric so in this analysis one half of the off-diagonal elements can be omitted from the processing. There are advantages (lower number of values to process) and disadvantages (transforming the final Hadoop results before presenting them, possibilities of errors from further processing of the matrix because of its size and more complex processing procedures) of following this procedure. In the present analyses, however, they were retained in the processing because of the resulting disadvantages.

The decile classes for the 2008 to 2014 annual global internet user population aggregates are shown the following table.

The classified aggregates were then read into the Hadoop Distributed File System (HDFS) in preparation for the MapReduce job. The procedure for loading data into the HDFS can be found in the Apache Hadoop project website.

2. Prepare the Mappers and Reducers for Hadoop Streaming

The next step is to prepare the mappers and reducer scripts that will be used in the Streaming job.

Python mapper and reducer

The Python mapper and reducer for the word count jobs were obtained and prepared according to the tutorial in this post. The improved Python mapper and reducer combination was selected. The Python mapper is as follows:

The Python reducer is as follows:

R script mapper and reducer

The R Script mapper and reducer for the word count jobs were obtained and prepared according to this post. The R script mapper is as follows:

The R script reducer is as follows:

3. Analyze the data in Hadoop

The next step is to execute the jobs in Hadoop.

2014 Global population word count job (Python)

In this section the first assumption is that the 2014 global population data is called InputData.txt (i.e. name in HDFS). In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS in the input folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in a local system folder called - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper is located in a local system mapper folder called - <Python mapper folder>, the Python reducer is located in a local system reducer folder called - <Python reducer folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

In my case I obtained the following category counts.

2014 Global internet user population word count job (Python)

In this section the first assumption is that the 2014 global internet user population data is called InputData.txt (i.e. name in HDFS). In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS in the input folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in a local system folder called - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper is located in a local system mapper folder called - <Python mapper folder>, the Python reducer is located in a local system reducer folder called - <Python reducer folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

In my case I obtained the following category counts.

2012 Facebook user population word count job (Python)

In this section the first assumption is that the 2012 Facebook population data is called InputData.txt (i.e. name in HDFS). In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS in the input folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in a local system folder called - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper is located in a local system mapper folder called - <Python mapper folder>, the Python reducer is located in a local system reducer folder called - <Python reducer folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

In my case I obtained the following category counts.

2008 to 2014 Global internet user population spatial time series variance-covariance matrix word count job (Python)

In this section the first assumption is that the 2008 to 2014 global internet user population spatial time series variance-covariance matrix data is called InputData.txt (i.e. name in HDFS). In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS in the input folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in a local system folder called - <hadoop-streaming-2.6.0.jar local folder>, the Python mapper is located in a local system mapper folder called - <Python mapper folder>, the Python reducer is located in a local system reducer folder called - <Python reducer folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

In my case I obtained the following category counts.

2008 to 2014 Global internet user population spatial time series variance-covariance matrix word count job (R script)

In this section the first assumption is that the 2008 to 2014 global internet user population spatial time series variance-covariance matrix data is called InputData.txt (i.e. name in HDFS). In terms of the MapReduce Streaming code the assumption is that the data has been successfully loaded into HDFS in the input folder - <HDFS input folder>, the hadoop-streaming-2.6.0.jar file is located in a local system folder called - <hadoop-streaming-2.6.0.jar local folder>, the R script mapper is located in a local system mapper folder called - <R script mapper folder>, the R script reducer is located in a local system reducer folder called - <R script reducer folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The next step is to run the following command in Hadoop.

In my case I obtained the following category counts.

2008 to 2014 Global internet user population spatial time series variance-covariance matrix word mean and word standard deviation jobs

In this section the first assumption is that the 2008 to 2014 global internet user population spatial time series variance-covariance matrix data is called InputData.txt (i.e. name in HDFS). In terms of the standard Hadoop 2.6.0 MapReduce code the assumption is that the data has been successfully loaded in the input folder - <HDFS input folder>, the hadoop-mapreduce-examples-2.6.0.jar file is located in the local system folder - < hadoop-mapreduce-examples-2.6.0.jar folder> and the HDFS output folder name has been selected to be - <HDFS output folder>. The word mean is obtained by running the following command in Hadoop.

In my case I obtained the following decile class based word mean value.

The word mean value (i.e. mean of the decile class data) is a function of the mean of the original data (quantitative values).

The word standard deviation is obtained by running the following command in Hadoop.

In my case I obtained the following decile class based word standard deviation value.

The word standard deviation (i.e standard deviation of the decile class data) is a function of the standard deviation of the original data (quantitative values). The scheme can be refined to use percentile divisions that have a finer granulity. An example of the approach is to use 5% percentile interval cut-offs, 2.5% and so on.

Summary

In the post I outlined how to setup a MapReduce job that can be used to generate summaries of a big annual spatial time series variance-covariance matrix of the global internet user population between the years 2008 to 2014. The summaries can be used to generate more specific/elegant/specialized analyses of the spatial time series variance-covariance matrix.

The procedure is simple to execute in the sense that there are essentially three sets of MapReduce jobs/procedures that were run. For example, the R script procedure replicates the Python MapReduce job. Secondly, a Python word count procedure wthat has the same structure was run on four different data sets. This essentially reduces the jobs to a Hadoop Streaming Python MapReduce word count job, a standard Hadoop MapReduce word mean job and a standard Hadoop MapReduce word standard deviation job.

I hope this post proves useful for your own analyses. Check out my other related blog posts for a better context on how you can use the procedure in your own analyses.

Or check out our statistics and e-learning services

Also check out my Introduction to Website Survey Sampling Course course at Udemy College

Sources:

http://bit.ly/219mwIJ

http://bit.ly/1SN27EA

http://bit.ly/1PW2Ktg

http://bit.ly/1oXvms2

http://bit.ly/1SQAQhu

http://bit.ly/1oDhovm

http://bit.ly/1oXwBXW

http://bit.ly/1QttFIJ

How to set up the Hadoop 2.6.0 MapReduce wordcount example to summarize digital, social and mobile data

2016-02-14T23:46:00.002-08:00

This post outlines a simple framework for analyzing digital, social and mobile (technology) data using a simple framework that can be set up using Microsoft Excel (gnumeric in Ubuntu), Apache Hadoop version 2.6.0, R and SAS software. The setup requires elementary programming knowledge, although MicrosoftVisual Basic, R, RHadoop and Java programming can greatly simplify (and enhance) the process. The approach ultimately involves setting up a data lake which grows according to a Kronecker product multiplication. The mathematical foundation of the framework was introduced and discussed in my previous blog posts (whose links are included at the end of the post).

The prerequisites for implementation are an Apache Hadoop 2.6.0 (Hadoop) installation (single node cluster and examples), Microsoft Excel (or any other Microsoft Excel-like spreadsheet), R (or SAS software), elementary statistics and Internet World Stats data sets (2014 Internet user population, 2012 Facebook population and the 2008 to 2014 population spatial time series).

1. Obtain the data sets

The procedure for setting up the data sets is outlined in my previous blog post: The sixty-five Regional Digital, Social and Mobile in 2015 statistics list that every blogger, writer or internet (content)specialist should take a look at to get a good mathematical basis from which to formulate social media and digital statistical content.

The first step is to load the population data into Microsoft Excel. The figures must be converted into ordinal categories that have been set up to have a name length that gives an indication of the (size of the) values.

For example sm_cat one (small category one) has a name length that is shorter than sm_cat_two_ (small category two). This is how one can tell Apache Hadoop MapReduce that the word sm_cat_one is attached to a number that is smaller than that of sm_cat_two.

This will set up the classification for the 2014 (global) internet user population, 2014 (global) population and 2012 (global) Facebook user population data sets. The cut-offs can be set up according to the choice of the programmer/analyst. The next step is to set up the spatial time series variance-covariance data.

The details for calculating the spatial time series variance-covariance matrix for the global population using the Kronecker product is outlined in my previous blog post: 5 matrix decompositions for visualizing the global internet user population spatial time series variance-covariance matrix. The spatial time series variance-covariance matrix I selected for the analysis was the 2008 to 2014 global population spatial time series.

The next step was to calculate the absolute value of the matrix values in order to handle the cases of negative variance-covariances. A procedure that can be followed is to leave the matrix values as is and adjust the value categories to represent the numbers (i.e. the sign and the size of the values). The 2008 to 2014 global internet user population spatial time series variance-covariance matrix can be set up similarly with different (or the same) cut-offs. The categorization of the variables needs to be adjusted because separable spatial time series variances-variances are on a different scale to the original data.

The approach I will illustrate for the analysis involves using a double squared scale of the original data. The rationale is for a squared scale for the spatial covariances and a squared scale for the time covariances. In my case the decision also involved an analysis of the magnitude of variances-covariance values.

The data sets can then be saved into a text file. It is important to make sure that there are leading and trailing spaces for each word so that Hadoop can separate the words in the wordcount mapreduce code.

2. Process the data in Hadoop

The four data sets can then be loaded into your input folder on the Apache Hadoop Distributed File System (HDFS). The next step will be to analyze the data sets using the procedure outlined in the Apache Hadoop project.

For example, from an Ubuntu 14.04.3 terminal in Hadoop, the following line of code can be run where the MapReduce examples programs are housed:

$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount HDFS_input_folder/<analysis file>.txt HDFS_output_folder

2014 Global population

In my case the resulting category wordcounts for the 2014 global population were as follows.

The data can be extracted and plotted in R, SAS software or Microsoft Excel (or gnumeric). The procedure can be further fine-tuned according to the requirements of the analyst.

2014 Internet user population

In the case of the 2014 internet population I obtained the following counts for the categories.

2012 Facebook users

In the case of the 2012 Facebook population I obtained the following counts for the categories.

2008 to 2014 Global population spatial time series variance-covariance matrix

In the case of the 2008 to 2014 spatial time series variance-variance matrix, I obtained the following counts for the categories.

The next step is to head to the R-console and find out which category contains the mean for the variance-covariance data (absolute value of the original variance-covariance data). In my case I obtained the following output.

The values can be plotted in R using the gvisBarChart function GoogleVis package.

The mean is in large_cat_two which is a little counter-intuitive. It is important, however, to keep in mind that larger values will pull the mean toward them (i.e. have a larger “implicit weight” in the equal weight mean value calculated in R). The second consideration is that the numbers in each category depend on the cut-offs. In my case, the next logical step will be to repeat the procedure with the newly available information from Hadoop.

The exciting feature about the Hadoop output is that this scheme represents a home-made/simple/elementary decomposition of the spatial time series variance-covariance matrix. I have found that the decomposition, because of its simplicity, is useful for constructing (enhancing) the more sophisticated spatial time series variance-covariance decompositions like the singular value decomposition/spectral decomposition, QR decomposition, polar decomposition, and Fourier/spectral density decomposition.

Conclusions

The Hadoop version 2 framework is a great resource for generating efficient solutions to data lake related problems. In this post I was able to achieve this by combining statistical programming with the built-in Hadoop version 2 tools. A similar approach can be taken with other Hadoop version 2 capabilities to solve other data lake problems. In the next Hadoop examples post I will explore this approach with the wordmean, wordstandarddeviation and other Hadoop examples.

Before then, would you like to get more information which you can use to customize Hadoop to your own setting? Then check out our other resources.

Check out our other blog posts and screencast series

Subscribe to our RSS feeds for blog material updates

Blog post RSS feeds

Screencast RSS Feeds

Or get a 50% discount to our exciting training opportunity bundle

Or check out our statistics and e-learning services

Sources:
http://bit.ly/1Pgq57i

Or check out our statistics and e-learning services

Setting up an Ubuntu OpenStack cloud using the Oracle VM VirtualBox, Ubuntu Linux Server 14.04.3 LTS and Ubuntu OpenStack Autopilot

2016-02-08T01:33:00.000-08:00

This blog post is part of a series on how to set up and grow a Hadoop version 2 ecosystem for your Big data analyses. The methods outlined can be implemented on a single computer or a cluster of computers. Similarly, the methods can be implemented for business applications as well as non-business applications. The main non-business applications include academic and personal research.

The main technology fields include computing (software and hardware), programming (statistical and non-statistical), cloud computing, applications development, (Big) data science, mathematical statistics and (pure) mathematics. The tools and applications that will be considered can be applied in e-commerce, environmental science, library science, linguistics and many other fields.

The series has two aims. The first is to provide a simple overview of the field of DevOps. The second is to give an account of what can be achieved with a very minimal set of DevOps tools. This distinction will naturally be reflected in the hardware and software settings. A professional DevOps specialist would be deploying a much (much) more advanced configuration. This is especially so for a data science DevOps specialist from a company/organisation like Apache, Google, Canonical, DataStax, Yahoo!, Amazon, Cloudera, MapR, Hortonworks, IBM, Microsoft Azure, SAS, Microsoft Revolution Analytics and others.

The idea is to think of each communication between two machines over the internet as a private cloud. The private cloud can then be generalized to more complex scenarios based on the purpose of establishing the cloud. The cloud can be private or public according to your preference and purpose.

The post describes the steps involved in setting up an OpenStack cloud from the context of a single Windows host machine. The OpenStack cloud will have the characteristics of a Hadoop version 2 ecosystem (i.e. without the Hadoop version 2).

1. Oracle VM VirtualBox

The Oracle VM VirtualBox (OVB) User Manual gives the following key reasons for virtualization:

Run multiple operating systems simultaneously
Easy software instructions
Testing and disaster recovery
Infrastructure consolidation

Below are two screenshots of the OVB Virtual Manager and Virtual Machines.

2. Oracle VM VirtualBox download

The OVB can be downloaded from the OVB website downloads section. The site also includes downloadable user manuals, online user manuals, tutorials and other materials. The site also has other update information as well as the OVB Extension pack. The extension pack should be downloaded with the OVB in order to fully access its features, like for example shared folders. It is also a very good idea to download a copy of the user manual and to fully setup the OVB's features.

3. Ubuntu Linux Server and OpenStack Autopilot download

The next step after the downloading the OVB components is to download the operating system and cloud software for the Virtual Machine (VM). The download for Ubuntu Server 14.04.3 LTS can be found at Canonical’s Ubuntu website at the Server downloads section.

The download for Ubuntu OpenStack Autopilot can be found at Canonical’s Ubuntu website at the Cloud downloads section.

The software bundles are suited for a DevOps project because of the various cloud communications. The most important being the Apache Hadoop project and other Hadoop Ecosystem projects. The notable ones being Apache Ambari, Apache Avro, Apache Chukwa!, Apache Zookeeper, Apache Hive, Apache Maven, Apache HBase, Apache Pig, Apache Cassandra (and DataStax Apache Cassandra), Apache Mahout, Apache Spark, Apache Storm and Apache Tez. It is is also very important for our purposes to keep in mind the excellent projects under the Google Cloud platform like Hadoop on Google Cloud Platform, Google Cloud BigQuery project and other related projects.

Canonical Ubuntu also has two other exciting options that can be used at this stage. The first is to test drive installing OpenStack Autopilot on vSphere. The second is to manually install Ubuntu OpenStack using MAAS and Juju.

The test drive of OpenStack Autopilot on vSphere can be found in the Cloud downloads section. The installation of OpenStack using MAAS and Juju can be found in the Cloud downloads section.

4. Virtual Machine Setup

The installation files can be saved in a windows folder like c:/Downloads/ or ~/Downloads in Ubuntu Linux. The manual guide gives a more detailed account of the steps in setting up the OVB and VM.

Install the OVB by double clicking on the downloaded file. This will start the wizard as shown below. The various options will be according to preference(s).

After the installation completes successfully, the Virtual Manager Window will appear. Click on New to create your VM. Give your machine a name. Choose your type to be Linux. Choose your version to be Ubuntu (64-bit).

Select Memory size to be 4G and click next.

In the next step, select Create a virtual hard disk now.

Select VHD (Virtual Hard Disk).

Select Dynamically allocated.

Specify the file location and size of the disk. My recommendation is 25 G. However, one can start off with 8G but a smaller size will mean more management later like adding new hard drives, drive partitioning, and so on. This will just add complications in managing your Hadoop cluster ecosystem.

The important issue to consider is that for DevOps you need enough local space for: your Hadoop cluster; management of the cluster; managing interactions with the cluster; developing cluster applications; data sets for your applications,;system log files; Ubuntu 14.04 server updates and features; Ubuntu OpenStack cloud management; Juju OpenStack cloud management, managin; your cloud server accesses; Oracle Virtual Box features; and so on.

The Virtual Manager is where you manage the VMs and their settings and preferences. This is also where you can create new hard drives, manage network connections, preferences and settings. It is most important to read the OVB manual to set up all the features of your VMs and in our case our Ubuntu Server (or private cloud).

5. Installing Ubuntu 14.04.3 LTS Server and Ubuntu OpenStack Autopilot on the Virtual Machine

The installation instructions for Ubuntu Server 14.04.3 LTS can be found at the Canonical’s Ubuntu website under the Download>Server Installation guides section.

The installation instructions for OpenStack can be found at the Canonical’s Ubuntu website under the Download>Cloud Installation guides section. The instructions for the required installations in the VM involve combining the instructions for installing a server and software for an OpenStack cloud.

6. Setting up Ubuntu OpenStack cloud

Setting up the OpenStack cloud involves going to the Canonical Ubuntu website Cloud> OpenStack section for the getting started guides on Ubuntu OpenStack.

The next step is to go to the Canonical Ubuntu Cloud>Juju section for the getting started guides on Juju.

The next step is to go to the Canonical Ubuntu Cloud> MAAS section for the getting started guides on MAAS.

The next step is to go to the Apache Hadoop website for the downloads and installations for the Hadoop version 2 Ecosystem.

It will also be very important to go and visit the Google Cloud Big Data projects, especially the Hadoop on Google Cloud Platform, Google BigQuery project and its related projects.

A good place for online courses on Apache Cassandra is the DataStax Academy.

I hope this post was helpful to you.The next post in the series will outline the procedures in setting up OpenStack, Juju, Hadoop version 2.6.0 and the Hadoop version 2 Ecosystem.

In the meantime, interested in seeing other digital and social media materials from Stats Cosmos blog?

Check out our other blog posts and screencast series

Subscribe to our RSS feeds for blog material updates

Blog post RSS feeds

Screencast RSS Feeds

Or get a 50% discount to our exciting training opportunity bundle

Do you have statistical products to sell?

Why not try selling them on Amazon?