http://statscosmos.blogspot.com/Academic blog. The blog posts on key statistical product related Information and Communication Technology (ICT) developments. The blog has a resources page with educational content; and statistical technologies that can be used by bloggers for big data analysis, cloud computing, multi-media content creation and content marketing. The blog also provides writing, statistics, and e-learning services for bloggers.ennoreply@blogger.com (Hariz Naam)Tue, 21 Jun 2016 05:22:51 PDTBlogger http://www.blogger.com1315Statistical,products,statistics,blogs,big,data,and,HadoopnoStatistical,products,statistics,blogs,big,data,and,HadoopStatistics bloggingStatistics blogging unlocks new technologieshttp://creativecommons.org/licenses/by/2.0/https://feedburner.google.com/fb/images/pub/fb_pwrd.gifblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comblogspot/flanPhttps://feedburner.google.comTechnology/Tech Newsblogspot/flanPhttps://feedburner.google.comHarizNaam@gmail.comhttp://feedproxy.google.com/~r/blogspot/flanP/~3/66HmUCtDy8M/how-to-apply-mapreduce-to-delicious.htmlSat, 14 May 2016 13:13:54 PDTtag:blogger.com,1999:blog-3427862540497685710.post-52896169047911992510no This post is designed for a joint installation of Apache Hadoop 2.6.0, Apache Spark 1.5.1 (pre-built for Hadoop), MongoDB 2.4.9 and Ubuntu Server 14.04.3 LTS. This illustration shows how one can use MapReduce to construct metrics in content-based recommeHarizNaam@gmail.com This post is designed for a joint installation of Apache Hadoop 2.6.0, Apache Spark 1.5.1 (pre-built for Hadoop), MongoDB 2.4.9 and Ubuntu Server 14.04.3 LTS. This illustration shows how one can use MapReduce to construct metrics in content-based recommendation models for social tagging systems. The specific system is the GroupLens HetRec 2011 Delicious Bookmarks dataset system. The illustration is composed of two posts. The theoretical framework is that outlined in the paper: Content-based Recommendation in Social Tagging Systems by Cantador, Bellogin and Vallet, published in 2010. The MapReduce implementation approaches are Hadoop Streaming, Spark WordCount (Scala program and Python Application), Spark Pipe (Scala program and Python Applications), Spark SQL (SparkR and Scala Spark-shell) and MongoDB. The core dataset is the Bookmark Assignments dataset which translates to a version of the A set in the paper. The procedure translates to 17 MapReduce jobs which can be categorized into two phases. The first phase involves constructing the element definitions (and weights) in the paper and the second phase the utility metrics (similarity measures). The first phase is composed of four MapReduce jobs and will be outlined in this post. The second phase is composed of the remaining jobs. The first two MapReduce jobs in the second phase involve constructing the first similarity measure (two metrics) and will also be outlined in this post. The remaining eleven MapReduce jobs will be outlined in the second post. 1. The Model In a social tagging system (Cantador, Bellogin and Vallet, 2010), users create or upload content, annotate it with their own words and share it with other system users. The content is referred to as items and the annotations as tags. The tagging system is then an unstructured collaborative content classification system called a folksonomy. The classification system can then be used by system users to search for and discover items of interest. The modeling of the system rests on two key assumptions. The first is that users will generally annotate items that are relevant for them and thus their tags can be seen to be a reflection of their interests, tastes and needs. It can additionally be assumed that the more a tag is used by a certain user, the more the important the tag is for them. The second assumption is that tags assigned to items describe their contents. Similarly, the more a certain item is annotated with a particular tag, the better the tag describes the item’s contents. It is important to keep in mind, however, that if a tag is used very often by users to annotate many items, it may not be useful to discern the information assumed. A folksonomy ℱ can be defined mathematically as a tuple ℱ = {T, U, I, A}, where T = {t1,......, tL} is the set of tags, U =&nbsp;{u1,......, uM} is the set of users and I={i1,......, iN} is the set of items. A set A = &nbsp;{(um, tl, in)}&nbsp;∈&nbsp;U * T * I is then the set of annotations, tag tl to item in by user um.&nbsp; The paper then outlines the following formulation of the recommendation problem according to Adomavicius and Tuzhilin (2005): For a totally ordered set&nbsp;ℜ, one can define a utility function g, g: U * I&nbsp;➙&nbsp;ℜ, such that&nbsp;g(um,in)&nbsp;measures the gain of usefulness of item in to user um. The aim of the analysis is then, for each user u&nbsp;∈&nbsp;U, to find items i max, u&nbsp;∈&nbsp;I, unknown to the user, that maximize g: ∀ u ∈ &nbsp;U,&nbsp;i&nbsp;max, u &nbsp;= arg maxi∈I&nbsp;g(u,i). In the modeling framework of the paper, content-based recommendation approaches formulate g as the metric: &nbsp; g(um,in) = Sim(ContentBasedUserProfile(um), Content(in)) &nbsp;∈&nbsp;ℜ, where, ContentBasedUserProfile(um) =&nbsp;um&nbsp;= &nbsp;(um,l,........,um,K)&nbsp;∈&nbsp;ℝk, k&nbsp;∈&nbsp;ℕ,&nbsp;is the content-based preferences of user um (i.e. described in assumption one).Content(in) =&nbsp;in&nbsp;= &nbsp;(in,l,........,in,K)&nbsp;∈&nbsp;ℝk, k&Statistical,products,statistics,blogs,big,data,and,Hadoophttp://statscosmos.blogspot.com/2016/05/how-to-apply-mapreduce-to-delicious.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/xtudj1VLS04/how-to-apply-mapreduce-to-delicious.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/ulWngjQQe70/how-to-apply-mapreduce-to-delicious.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/R6-GfI5njk8/how-to-apply-mapreduce-to-delicious.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/DEi_cpGryaE/how-to-apply-mapreduce-to-delicious.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/kNuJtyoKMog/how-to-apply-mapreduce-to-delicious.htmlThis post is designed for a joint installation of Apache Hadoop 2.6.0, Apache Spark 1.5.1 (pre-built for Hadoop), MongoDB 2.4.9 and Ubuntu Server 14.04.3 LTS. This... Click on the... Click on...<br/> <br/> Click on the blog post title above to visit my blog and read the rest of the post.HarizNaam@gmail.comhttp://feedproxy.google.com/~r/blogspot/flanP/~3/9AZna_I6Z7Y/how-to-apply-mapreduce-to-movielens-1m.htmlThu, 07 Apr 2016 05:21:31 PDTtag:blogger.com,1999:blog-3427862540497685710.post-21189436110122918040no This post is designed for a joint Apache Hadoop 2.6.0 single cluster, Apache Spark 1.5.1 and Ubuntu Server 14.04.3 LTS installation. This is a follow on post to my previous post: How to set up Hadoop Streaming to analyze MovieLens data. In the present poHarizNaam@gmail.com This post is designed for a joint Apache Hadoop 2.6.0 single cluster, Apache Spark 1.5.1 and Ubuntu Server 14.04.3 LTS installation. This is a follow on post to my previous post: How to set up Hadoop Streaming to analyze MovieLens data. In the present post the GroupLens dataset that will be analyzed is once again the MovieLens 1M dataset,&nbsp;except this time the processing techniques will be applied to the Ratings file, Users file and Movies file. The MapReduce approach has four components. The first is to integrate the GroupLens MovieLens Ratings, Users and Movies datasets. The second is to design the MapReduce processing model. The third is to design a system for checking the results from the processing for consistency and accuracy. The last is to summarize/analyze the results. 1. Prepare the data The Ratings file has the following variables: The Movies file has the following variables: The Users file has the following variables: The three datasets can thus be linked using the UserID variable and the MovieID variable. Essentially, from a processing point of view, each of the columns can be processed together or individually. For example, the Ratings and MovieID columns can be processed together to generate the average rating for each MovieID. An example of individual processing is using the&nbsp;gender column to generate the number of ratings by female users and male users. In this approach, the individual columns selected were Gender, Occupation, Age, ZIP codes and Genres. The MapReduce will be conducted with files containing the individual columns (for example a file containing the Gender column for the Gender MapReduce).&nbsp; The joint column consideration is the MovieID and Ratings columns. The MapReduce will conducted using a file with the UserID, MovieID and Ratings columns.&nbsp; The MovieID and Ratings column can also be fused to generate a MovieIDRatings (fused text and number) column that can be used for checking the results. The MapReduce (for this purpose) will be conducted using a file with the fused MovieID and Ratings column. 2. Prepare the mapper and reducer sets The MapReduce design has two components, a processing component and a checking component. The MapReduce processing component combines mapper-reducer sets, Apache Hadoop Streaming and Apache Spark Pipe. The MapReduce processing model is housed in the mapper-reducer sets. The data is processed using the mapper-reducer sets within the Apache Hadoop Streaming and Apache Spark Pipe facilities (of Hadoop and Spark, respectively). A mapper-reducer set prepared using Perl&nbsp;can be used with the Hadoop Streaming facility. Mapper and reducer sets prepared in R, Rubyand Python can be used with the Spark Pipe facility. The MapReduce checks component firstly runs a joint column consideration of MovieID and Ratings columns to calculate the Average Ratings per MovieID (i.e. replicates the Spark Pipe run) in Hadoop Streaming.&nbsp; The next step is to run Spark Simple Applications prepared in Java and Python on all the column datasets (including the fused MovieID and Ratings column). The procedure is repeated using SparkR. The last check is to run the Hadoop Grep worked example on the fused MovieID and Ratings column data. Perl mapper-reducer set The Perl mapper-reducer set was prepared using the tutorial in this post. The mapper. The reducer Ruby mapper-reducer set The Ruby mapper-reducer set was prepared using the tutorial in this post. The mapper. The reducer R mapper-reducer set The R mapper-reducer set was prepared using the tutorial in this post. The mapper. The reducer The Python mapper-reducer set The Python mapper-reducer for the MovieID ratings average set was prepared using the tutorial in this post. The mapper. The Python reducer Once the mapper-reducer sets have been prepared the data can be processed in Hadoop and Spark. 3. Process the data in Hadoop and Spark In this processing design, the Gender variable is processed in Hadoop using the StreamStatistical,products,statistics,blogs,big,data,and,Hadoophttp://statscosmos.blogspot.com/2016/03/how-to-apply-mapreduce-to-movielens-1m.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/tpW_Cg-fO54/how-to-apply-mapreduce-to-movielens-1m.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/FM0hv9GiHqU/how-to-apply-mapreduce-to-movielens-1m.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/npGf5nw2UCg/how-to-apply-mapreduce-to-movielens-1m.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/2E6bIoljjjM/how-to-apply-mapreduce-to-movielens-1m.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/UJNYm1_HgZE/how-to-apply-mapreduce-to-movielens-1m.htmlThis post is designed for a joint Apache Hadoop 2.6.0 single cluster, Apache Spark 1.5.1 and Ubuntu Server 14.04.3 LTS installation. This is a follow on post to my previous... Click... Click on...<br/> <br/> Click on the blog post title above to visit my blog and read the rest of the post.HarizNaam@gmail.comhttp://feedproxy.google.com/~r/blogspot/flanP/~3/DgP7mc0vgcw/how-to-summarize-book-crossing-dataset.htmlFri, 11 Mar 2016 19:32:10 PSTtag:blogger.com,1999:blog-3427862540497685710.post-26967458872234448130http://statscosmos.blogspot.com/2016/03/how-to-summarize-book-crossing-dataset.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/bjdup24LfLk/how-to-summarize-book-crossing-dataset.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/jYjqMfVRFXo/how-to-summarize-book-crossing-dataset.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/ESWP7clFsoc/how-to-summarize-book-crossing-dataset.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/ZvmFp1amaQA/how-to-summarize-book-crossing-dataset.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/70VYBBG1PAg/how-to-summarize-book-crossing-dataset.htmlThis post is designed for a joint installation of Ubuntu Server 14.04.3 LTS, Apache Hadoop 2.6.0 single cluster and Apache Spark 1.5.1&nbsp;(pre-built for Hadoop 2.6 and... Click... Click on...<br/> <br/> Click on the blog post title above to visit my blog and read the rest of the post.HarizNaam@gmail.comhttp://feedproxy.google.com/~r/blogspot/flanP/~3/BL0Qks6CESY/how-to-set-up-hadoop-streaming-to.htmlMon, 21 Mar 2016 14:14:17 PDTtag:blogger.com,1999:blog-3427862540497685710.post-36017622072119658320http://statscosmos.blogspot.com/2016/02/how-to-set-up-hadoop-streaming-to.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/GPKZ8dk00-Q/how-to-set-up-hadoop-streaming-to.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/q0sPnvbC5Ng/how-to-set-up-hadoop-streaming-to.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/cgdpaC3myog/how-to-set-up-hadoop-streaming-to.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/U9mwMGWBNwk/how-to-set-up-hadoop-streaming-to.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/aGwiiiRT-ac/how-to-set-up-hadoop-streaming-to.htmlThis post is designed for an Apache Hadoop 2.6.0 single cluster installation. The job uses a Hadoop Streaming design with C++, Ruby and Python. The MapReduce configuration... Click on... Click...<br/> <br/> Click on the blog post title above to visit my blog and read the rest of the post.HarizNaam@gmail.comhttp://feedproxy.google.com/~r/blogspot/flanP/~3/CfiU_gd4Lgo/how-to-incorporate-python-and-r-into.htmlFri, 04 Mar 2016 05:24:03 PSTtag:blogger.com,1999:blog-3427862540497685710.post-88645367298814442980http://statscosmos.blogspot.com/2016/02/how-to-incorporate-python-and-r-into.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/j2VMxoIUhfQ/how-to-incorporate-python-and-r-into.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/4pxe2bRrU4o/how-to-incorporate-python-and-r-into.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/VPHj7iQ-eWA/how-to-incorporate-python-and-r-into.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/mmnqS2NnVm4/how-to-incorporate-python-and-r-into.htmlhttp://feedproxy.google.com/~r/blogspot/flanP/~3/8n7y9bf6P_A/how-to-incorporate-python-and-r-into.htmlThis setup guide is designed for an Apache Hadoop 2.6.0 installation. Hadoop streaming is a utility/facility that allows one to create and run MapReduce jobs with any... Click on... Click on...<br/> <br/> Click on the blog post title above to visit my blog and read the rest of the post.HarizNaam@gmail.comnonadultStatistics bloggingnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadultnonadult