Vancouver Data Blog by Neil McGuigan

Less Painful AJAX / Javascript Web Scraping

2012-02-11T20:36:00.000-08:00

If you read my previous post, you'll see that scraping ajax pages can be a pain. So I wrote a little Java program to make it easier. It takes a list of URLs to scrape, and will render them in a browser, and save the (normal and ajax) rendered HTML and screenshots to a folder. Here's the how-to video: You need Firefox 3+ installed, as well as Java 1.6. This is a beta project, and no warranty

Web Scraping AJAX Pages

2012-02-09T16:01:00.000-08:00

This is part four of a series of video tutorials on web scraping and web crawling. Part 1: Web scraping with Google Spreadsheets and XPathPart 2: Web Crawling with RapidMinerPart 3: Web Scraping with RapidMiner and Xpath This post explains how to capture HTML from Ajax / Javascript generated pages. Here is the accompanying video. The first thing you should know is that it is a major, major

On Making Videos

2012-01-29T17:59:00.000-08:00

Here is what i use to make my videos: 1. CamStudio. This is a nice free and open-source desktop video capture program. Make sure to use their Lossless Codec, and go with these settings: Set Keyframes Every 30 frames Capture Frames Every = 50 milliseconds Playback Rate = 20 frames per second Video codec: CamStudio Lossless Codec Quality: 70% 2. Handbrake Video Transcoder. This will help you

Happy New Year

2011-12-31T19:38:00.003-08:00

75,000 pageviews this year! Thanks to everyone for visiting. I will post some new material in the new year. Have a safe and fun 2012 Neil

My new blog about learning ExtJS

2011-11-04T14:39:00.001-07:00

I have a new blog. It's about learning to use ExtJS, a great rich internet application library in javascript. Here it is: http://extjs-tutorials.blogspot.com/ Check it out. Thanks

How Obama's data-crunching prowess may get him re-elected

2011-10-09T22:26:00.000-07:00

An article on CNN about how the Obama 2012 campaign has hired many data miners and statisticians to help boost fundraising and support. http://www.cnn.com/2011/10/09/tech/innovation/obama-data-crunching-election/index.html?hpt=hp_c1

Text Analytics with RapidMiner Part 6 of 6 - Applying the Model to New Documents

2011-10-08T15:31:00.001-07:00

After my last series, I got a lot of questions about how to apply a model to new data, so here is the real final installment in the series. I show how to save a wordlist and model to the repository. I use them later to read the wordlist and model and apply them to new documents that RapidMiner hasn't seen before. It correctly labels 11 of the 12 documents.

September sunset

2011-09-02T21:06:00.000-07:00

RapidMiner ETL - Transforming Attributes with Functions

2011-08-27T20:01:00.002-07:00

In this video I show how to transform features in RapidMiner using operators such as log, sqrt, absolute value, and multiplying columns.

RapidMiner ETL - Normalizing, Discretizing, Recoding

2011-08-27T20:01:00.000-07:00

In this video I show how to normalize an attribute, including z-normalization, how to discretize a column, and how to recode values

RapidMiner ETL - Sampling, Selecting Rows, Attributes

2011-08-25T18:18:00.000-07:00

In this video I show how to sample rows, including balancing class labels, bootstrap sampling. I also show how to filter rows by value, and select a subset of attributes. You can get the dataset here

RapidMiner ETL - Combining Datasets

2011-08-25T17:58:00.000-07:00

In this video, I show how to combine multiple datasets into one, and join columns and append rows.

And We're Back. A video series on ETL with RapidMiner

2011-08-25T17:33:00.000-07:00

Back with some more videos! Sorry for the long wait, and thanks for your patience. This series is on ETL: Extract, Transform, Load with Rapidminer. The first video shows how to combine multiple datasets into one, by joining columns and appending rows. The second videos is on sampling and selecting rows and attributes. More videos coming soon.

A rainy sunday in downtown Vancouver

2011-04-10T18:13:00.001-07:00

My blog should look better on mobile devices now.

Web Scraping with RapidMiner and XPath

2011-04-04T14:53:00.001-07:00

In this video I show how to load 500 html files from a previous web crawl, loop through each of them, and use XPath to grab values from each page, and put them in a data table for later analysis. Part 1: Web scraping with Google Spreadsheets and XPathPart 2: Web Crawling with RapidMinerPart 3: Web Scraping with RapidMiner and Xpath Part 4: Web Scraping AJAX Pages

Web Crawling with RapidMiner

2011-04-04T14:52:00.000-07:00

Here is part 2 of my series of videos on web crawling with RapidMiner. In this video I show how to crawl about 500 pages from a site, and discuss user agents, crawling rules, and robot exclusion files. Part 1: Web scraping with Google Spreadsheets and XPathPart 2: Web Crawling with RapidMinerPart 3: Web Scraping with RapidMiner and Xpath Part 4: Web Scraping AJAX Pages

More X-Path Goodness

2011-04-03T15:13:00.000-07:00

Got a RapidMiner crawling/scraping video coming up, but for now, here are some more X-Path ideas to play with: //* return all nodes //*[contains(., 'Search Text')] return all nodes that contain Search Text in their content. Case sensitive search. //div[@id='div1']/following-sibling::* return the next sibling of a specific node (not sure if this works in RapidMiner) //div[@id='div1']/../

Web scraping with Google Spreadsheets and XPath

2011-02-27T17:36:00.000-08:00

This is part one of a series of video tutorials on web scraping and web crawling. In this first video, I show how to grab parts of a web page (scraping) using Google Docs Spreadsheets and XPath. Google Spreadsheets has a nice function called importXML which will read in a web page. You can then apply an XPath to that page, to grab various parts of it, such as one particular value, or all of the

A Data Explosion Remakes Retailing

2011-01-07T10:31:00.000-08:00

From the New York Times: "Retailing is emerging as a real-world incubator for testing how computer firepower and smart software can be applied to social science — in this case, how variables like household economics and human behavior affect shopping." http://www.nytimes.com/2010/01/03/business/03unboxed.html

Computers That Trade on the News

2011-01-03T19:26:00.000-08:00

I missed this one around Christmas time...must have been the drive up to Prince George. In the NY Times: The number-crunchers on Wall Street are starting to crunch something else: the news. Math-loving traders are using powerful computers to speed-read news reports, editorials, company Web sites, blog posts and even Twitter messages — and then letting the machines decide what it all means for

10,000 views on my Youtube videos, 5,000 on my blog. Thanks everyone!

2010-12-27T17:22:00.000-08:00

Blown away with the number of visitors! While I'm here, here's a good article from Wired magazine about AI: The A.I. Revolution Is On

Next video series: Web crawling and scraping

2010-12-16T11:09:00.000-08:00

I'll be working on a video series on web crawling and scraping over Christmas, for release at the end of December or the first week of January at the latest. Web crawling and social network analysis were neck and neck on the poll, with social slightly ahead, but I am going to do web crawling first as I'm working on a web crawling project, so it will be fresher in my mind.

How to Filter By Value in RapidMiner

2010-12-14T16:09:00.000-08:00

Use the Filter Examples operator on an exampleset (Data Transformation > Filtering) set condition class to attribute_value_filter set parameter string like attribute=value example: Category=healthcare or Category=customer service|healthcare Both attribute and value are case-sensitive Spaces are allowed Use the | character for the "or" operator.

Custom stemming dictionary

2010-12-14T15:25:00.000-08:00

You can create your own stemming dictionary in RapidMiner. Add the Text Processing -> Stemming -> Stem (Dictionary) operator, and choose your dictionary file (plain text). Your format should be like this: stem:inflection stem:inflection example: fish:fished will turn fished into fish. You can also use wildcards: fish:fish.* will turn fished, fishes, fishing or anything beginning with

A regular expression to find "word A near word B" in RapidMiner

2010-12-14T15:09:00.000-08:00

You can use the Text Processing->Extract Information operator to match regular expressions. If you put the Extract Information operator inside a Process Documents operator, it will add a column to your dataset with the results of the match. Turn on "add meta information" option on the Process Documents operator. Here's a simple regular expression to find a word near another word: (word1\W+(?