Vancouver Data Blog by Neil McGuigan

AWS Redshift: How Amazon Changed The Game

2013-05-16T12:41:00.001-07:00

A good blog post on Amazon RedShift - their Postgres-based massive data warehouse. Some good analysis on performance and costs: http://blog.aggregateknowledge.com/2013/05/16/aws-redshift-how-amazon-changed-the-game/

Vancouver Training: Introduction to Data Mining and Predictive Analytics with RapidMiner - Save $500

2013-04-18T22:53:00.001-07:00

I'll be teaching a RapidMiner course here in Vancouver next week:Tuesday, April 23, 2013 at 8:30 AM - Wednesday, April 24, 2013 at 5:00 PM (PDT)Details here:http://rapid-i_us_20130423-eorg.eventbrite.com/Save $500 with the coupon VAN_BLOG !

Google's Data Mining Research Papers

2013-02-12T17:56:00.000-08:00

In case you missed it, here are Google's 104 data mining research papers: http://research.google.com/pubs/DataMining.html

The Google F1 slides

2012-12-20T20:15:00.000-08:00

Google F1 is a relational database query engine that works on top of Google Spanner, which is a distributed storage system that sits on top of Google File System. Got it? :) Basically, it's a really big, distributed relational database, and Google is using F1 to replace MySQL for Adwords. http://www.stanford.edu/class/cs347/slides/f1.pdf

Chomsky on Where AI Went Wrong

2012-11-01T22:56:00.001-07:00

If one were to rank a list of civilization's greatest and most elusive intellectual challenges, the problem of "decoding" ourselves -- understanding the inner workings of our minds and our brains, and how the architecture of these elements is encoded in our genome -- would surely be at the top. Yet the diverse fields that took on this challenge, from philosophy and psychology to computer science

The father of fractals

2012-11-01T22:54:00.002-07:00

A nice little piece on Mandlebrot in the Economist: http://www.economist.com/node/2246127

As I predicted, Self-driving cars a reality for 'ordinary people' within 5 years, says Google's Sergey Brin

2012-09-26T13:53:00.002-07:00

Link here: http://www.computerworld.com/s/article/9231707/Self_driving_cars_a_reality_for_39_ordinary_people_39_within_5_years_says_Google_39_s_Sergey_Brin

The Google Spanner Paper

2012-09-26T13:13:00.001-07:00

Google spanner is a massively distributed database. It needs atomic clocks on each machine to work though... http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/spanner-osdi2012.pdf

The Google Dremel Paper

2012-09-07T20:30:00.002-07:00

Here is the paper describing Google Dremel, which may replace Hive one day. There does not seem to be anyone working on an open-source version though Link (PDF) Update: Apache Drill is the open source version of Dremel (hat tip to Zoltan). Also, Cloudera's Impala looks simlar.

Self-driving cars: The next revolution

2012-09-07T20:22:00.001-07:00

Here is a recent report from KPMG about self-driving cars: http://www.kpmg.com/US/en/IssuesAndInsights/ArticlesPublications/Documents/self-driving-cars-next-revolution.pdf

Google’s Self-Driving Cars Are Going to Change Everything

2012-08-07T17:36:00.000-07:00

Recent News: Google’s Self-Driving Cars Complete 300K Miles Without Accident, Deemed Ready For Commuting http://techcrunch.com/2012/08/07/google-cars-300000-miles-without-accident/ Here's what is going to happen in the next 5-10 years. It won't all happen right away. The car insurance industry will cease to exist. These cars aren't going to crash. Even if there are hold-outs that drive

Less Painful AJAX / Javascript Web Scraping

2012-02-11T20:36:00.000-08:00

If you read my previous post, you'll see that scraping ajax pages can be a pain. So I wrote a little Java program to make it easier. It takes a list of URLs to scrape, and will render them in a browser, and save the (normal and ajax) rendered HTML and screenshots to a folder. Here's the how-to video: You need Firefox 3+ installed, as well as Java 1.6. This is a beta project, and no warranty

Web Scraping AJAX Pages

2012-02-09T16:01:00.000-08:00

This is part four of a series of video tutorials on web scraping and web crawling. You can probably skip this one, and go to the easy version. Part 1: Web scraping with Google Spreadsheets and XPath Part 2: Web Crawling with RapidMiner Part 3: Web Scraping with RapidMiner and Xpath This post explains how to capture HTML from Ajax / Javascript generated pages. Here is the accompanying video

On Making Videos

2012-01-29T17:59:00.000-08:00

Here is what i use to make my videos: 1. CamStudio. This is a nice free and open-source desktop video capture program. Make sure to use their Lossless Codec, and go with these settings: Set Keyframes Every 30 frames Capture Frames Every = 50 milliseconds Playback Rate = 20 frames per second Video codec: CamStudio Lossless Codec Quality: 70% 2. Handbrake Video Transcoder. This will help you

Happy New Year

2011-12-31T19:38:00.003-08:00

75,000 pageviews this year! Thanks to everyone for visiting. I will post some new material in the new year. Have a safe and fun 2012 Neil

My new blog about learning ExtJS

2011-11-04T14:39:00.001-07:00

I have a new blog. It's about learning to use ExtJS, a great rich internet application library in javascript. Here it is: http://extjs-tutorials.blogspot.com/ Check it out. Thanks Don't worry, I'll keep posting here too

How Obama's data-crunching prowess may get him re-elected

2011-10-09T22:26:00.000-07:00

An article on CNN about how the Obama 2012 campaign has hired many data miners and statisticians to help boost fundraising and support. http://www.cnn.com/2011/10/09/tech/innovation/obama-data-crunching-election/index.html?hpt=hp_c1

Text Analytics with RapidMiner Part 6 of 6 - Applying the Model to New Documents

2011-10-08T15:31:00.001-07:00

After my last series, I got a lot of questions about how to apply a model to new data, so here is the real final installment in the series. I show how to save a wordlist and model to the repository. I use them later to read the wordlist and model and apply them to new documents that RapidMiner hasn't seen before. It correctly labels 11 of the 12 documents. Files from the video.

September sunset

2011-09-02T21:06:00.000-07:00

RapidMiner ETL - Transforming Attributes with Functions

2011-08-27T20:01:00.002-07:00

In this video I show how to transform features in RapidMiner using operators such as log, sqrt, absolute value, and multiplying columns.

RapidMiner ETL - Normalizing, Discretizing, Recoding

2011-08-27T20:01:00.000-07:00

In this video I show how to normalize an attribute, including z-normalization, how to discretize a column, and how to recode values

RapidMiner ETL - Sampling, Selecting Rows, Attributes

2011-08-25T18:18:00.000-07:00

In this video I show how to sample rows, including balancing class labels, bootstrap sampling. I also show how to filter rows by value, and select a subset of attributes. You can get the dataset here

RapidMiner ETL - Combining Datasets

2011-08-25T17:58:00.000-07:00

In this video, I show how to combine multiple datasets into one, and join columns and append rows.

And We're Back. A video series on ETL with RapidMiner

2011-08-25T17:33:00.000-07:00

Back with some more videos! Sorry for the long wait, and thanks for your patience. This series is on ETL: Extract, Transform, Load with Rapidminer. The first video shows how to combine multiple datasets into one, by joining columns and appending rows. The second videos is on sampling and selecting rows and attributes. More videos coming soon.

A rainy sunday in downtown Vancouver

2011-04-10T18:13:00.001-07:00

My blog should look better on mobile devices now.