Never Read Passively

Mobile Game Engines

2012-03-30T08:59:00.001+11:00

I'm launching http://MobileGameEngines.com today. It's a simple site that lets anyone compare between all known iPhone, Android and HTML5 game engines.

I had the problem of trying to decide which game engine to choose for an iPhone game project and wrote about my efforts in a blog post in 2009: Review of 3D Engines for the iPhone I noted that many others had been through the same pain and had written a similar blog post and figured someone would come along and address this problem.

It has been a few years and the number of posts on quora and stackoverflow asking "what is the best iphone game engine" and "what is the best android game engine" continue to be posted. I decided to make a site to really nail the answer to this question, the result was MobileGameEngines.com

I've been in contact with the majority of the game engine developers listed on the site to ensure all the information is correct and useful. We are sitting on 65 game engines listed and still growing as more are uncovered.

Take a look and let me know what you think.
Pass the link on to anyone you think might be experiencing this pain.

DailyAIFeed: Your daily dose of Artificial Intelligence news

2012-03-06T17:50:00.000+11:00

I've been running a little email list for about a year called DailyAIFeed. I started it because I got sick of checking my RSS and email each morning, preferring to just check email.

I've recently cleaned it up a little and opened it up to general consumption at www.DailyAIFeed.com

The list provides a daily email of the top interesting news in the fields of Artificial Intelligence, Machine Learning, Natural Language Processing, Computational Intelligence, Infographics and Data Science.

You can see examples of the types of messages on the blog at blog.DailyAIFeed.com

Go and check it out, and if you have any suggestions on how to further improve it, please get in touch.

Hosted Data Analysis Project Workflow

2012-02-16T09:02:00.001+11:00

I have been thinking a lot about the previously mentioned "Data Analysis Project Management: SaaS".

Such a system could be focused on the project management side, say a Basecamp for Data Analysis projects. But I think it could something different, something more. I think there are pain points in the data analysis workflow that could be 1) systematized and 2) automated. I also suspect that large data platforms from IBM and Oracle may offer solutions, but I question whether the any such solutions exist for smaller scale/cost projects. I think there is an opportunity for a Hosted Data Analysis Project Workflow system.

I've been trying to think hard about possible pain points, and here's what I got:

Revision control - keeping track of changes to project files.
Research Journal - keeping track of what questions have been asked and what findings have been made.
Reproducibility - ensuring there is a recipe to recreate a past result.
Collaboration - working with others on a particular on all aspects of the analysis workflow.
Interpreting Data - Looking at tables and graphs and generating hypotheses to go and test.
Executing Models - configuring, running, tuning models.
Verifying Models - assessing models on test and verification datasets.
Blending Models - comparing and combining model outputs.

I have also been thinking about solutions in the market that address specific pain points. This is much harder - the Internet is a big place. Here is what I could put together quickly:

Many Eyes - Collaborate on interpretation from visualization.
Google Predict - Hosted models and data
Github (and similar) - Hosted project files, forking, collaboration
Kaggle, TunedIT - Data competitions, competitive community around problems, Data Spec Work / R&D Outsourcing.
Cross Validated - Q/A community around statistics and machine learning
Google data explorer - Visualization and interpretation public datasets

Do we need a system that combines all or some of these together? Why can't this problem be solved with existing systems, why isn't it such?

Data Analysis Project Management: SaaS

2012-02-15T07:00:00.053+11:00

The idea for developing a Data Analysis Workflow SaaS as been percolating in my brain for a few days.

I read a post on the Kaggle HHP forum entitled "Project Management software for Data Analysis" by Dan Becker and it got me thinking. With almost every data or machine learning competition I enter, I end up solving the wrong problem. Rather than solving the problem presented by the competition, I solve the problem of automating and optimizing my work flow by writing scripts and software.

It makes sense, it's what I do for a living. I write and maintain software. Why not expose the problem for what it is and solve it. The real question: Is there a market for such a solution? and secondarily, What features would the software have?

My ad hoc competition-wise solutions focus on result reproducibility (good science), summary statistics, visualization, research journal, generating webpages of experiments performed and above all, automation. I figure that the more I can automate, the faster I can test ideas and the higher my velocity will be on the problem of the competition. Except, most of my effort goes into the automation rather than ideas for the subject data.

Whether I build the software or not, it is a good research topic to consider.

The aforementioned forum post lists some interesting resources, specifically:

dryWorkflow project Don't Repeat Yourself Workflow for efficient data analysis. Focused on R.
ProjectTemplate: Automates the creation of new statistical analysis projects. Project Webpage, Github project, CRAN project.
CRAN Task View: Reproducible Research with some interesting ideas.
Book: The Workflow of Data Analysis Using Stata, J. Scott Long, 2009

Some google'ing resulted in some additional useful links: How to efficiently manage a statistical analysis project? on CrossValidated which provides some excellent answers and great links to go off and read. Other CV links that were pretty useful included Best Way to Aggregate and Analyze Data and What is a practically good data analysis process?

I also came across Organizing Your Approach to a Data Analysis by Scott Emerson which provides some excellent motivating questions.

Know any other good resources?

I'm keen for feedback, should such a software (web-based) platform exist in the world?

Are their data analysis people that would use this, pay for this, beyond competition patrons?

Preview of Clever Algorithms: Statistical Machine Learning Recipes

2012-02-14T13:55:00.000+11:00

I am feverishness working to complete a first draft of my next book in the Clever Algorithm series:

Clever Algorithms: Statistical Machine Learning Recipes

You can take a sneak peek at some early chapters online. I am developing the project in plain view on github, take a look at my source R files or LaTeX files if you're into that sort of thing.

The book will provide a treatment of the field of Machine Learning much like the first book provided a treatment of the field of Computational Intelligence and Biologically Inspired Computation.

Here is a preliminary blurb:

Implementing an Machine Learning algorithms is difficult. Algorithm descriptions may be incomplete, inconsistent, and distributed across a number of papers, chapters and even websites. This can result in varied interpretations of algorithms, undue attrition of algorithms, and ultimately bad science.
This book is an effort to address these issues by providing a handbook of algorithmic recipes drawn from the field of Machine Learning, described in a complete, consistent, and centralized manner. These standardized descriptions were carefully designed to be accessible, usable, and understandable.
An encyclopedic algorithm reference, this book is intended for research scientists, engineers, students, and interested amateurs. Each algorithm description provides a working code example in R.

I have started to distribute chapters to copy editors and technical editors. I'm also trying hard to nail down the table of contents and specifically which algorithms will appear in the text.

If you know anyone who might be interested in technical editing or any editing, get into contact: jasonb@CleverAlgorithms.com

House Price Regression: Vermont South, Melbourne

2011-05-17T19:03:00.009+10:00

While looking for a house I maintained statistics on the main suburbs we were visiting, and more specifically, on the houses we looked at.

Each house we visited was about 4 bed rooms and generally had the same kinds of attributes - attributes we thought we wanted in a house. For each house we visited, I recorded the address, size of land in square meters, date of sale, sale type (auction, private), asking price, sale price, and other assorted details. An additional contrived measure was the driving distance to main shops reported by Google Maps, in kilometers.

I also supplemented the dataset with additional matching houses in the area when data was available. I found prices were sometimes available from the auction results, although in other cases I had to call to find out, and.or scourer the web.

Rather than let this information go to waste, I thought I would share some of the collected data. This post provides data I collected for the Melbourne suburb of Vermont South.

The following graph simply shows sale price by date, quite boring.

The following graph shows the sale price by land size in square meters.

The following graph shows the sale price by distance to a specific set of shops in kilometers.

I found the data generally useful for plugging in new places and using simple linear regression to help answer questions about expected price at auction or private sale.

Some of this data may be available for purchase from various retail data providers, but I found collecting and entering the data myself made it a lot more personal and gave me some additional focus when inspecting properties and talking to agents about trends.

So, we bought a house

2011-05-16T18:00:00.030+10:00

So we finally bought a house. We've been looking on and off for about 12 months although things got serious about 3 months ago.

We first looked at the place we bought last Saturday, and walking in the door I knew it was a strong contender. We looked at three other places that day, and they all paled in comparison.

The place had been passed in at auction nearly three months before, and we were told that initially the vendors expectations were too high. We saw this as a good opportunity to negotiate and try to broker a good deal. The market had been slumped for a few months and the early figures for the quarter had indicated a ~2.5% drop in median house price for the city.

We did another inspection on the following Tuesday and enlisted all of the troops (extended family) to give the place a good once over. We then sat down and signed a formal offer. It was rejected. We upped the offer $5K and it was accepted, although at the insistence of my wife we made the offer contingent on the outcome of a builders inspection.

A found a company in the yellow pages and had the inspection done on the last day of the 3 day cooling off period. The building and pest report was incredibly detailed, providing photos and a room-by-room summary, inside and out. We learned a lot about the types of preventative maintenance the place will need over the next 5-10 years, and more importantly, we learned that the upstairs balcony had some major structural problems.

The report said that the wood used was popular in the decade that the balcony was built and was known to rot unless property treated. Rather than expecting the vendor to return the balcony to new condition, we made an offer to split the difference, deducting half of the cost of the repair from our offer.

All of the negotiating occurred on the last day of cooling off period, a Friday. I had my wife on one had, adamant that she didn't want to pay a thing to have the balcony fixed, and the agent on the other hand threatening to open the property for inspection on the next day. I really liked the place and I was feeling totally strung out (to say the least).

We managed to broker a deal in the end and initial the final amendment on the Saturday, one week from our first inspection. With previous auctions and negotiations, I tried to remain emotionless, time was on our side and we could wait for a deal. I really liked this place and it was beginning to dawn on me that our remaining time to find a place (before the baby came) had shrunk to a matter of a few months. We're both happy we finally got there and have high hopes for turning the property into our home.

We learned a lot throughout the process. My analysis of median house prices, suburb selection, crime rates, and even travel time studies months ago were interesting, although in the end did not directly affect the outcome. Even the detailed suburb house price regressions I was building up were not used, as we ended up buying in a completing different suburb, inspecting the house on a whim.

I was told early on that buying a home is different from buying an investment, and it bit me in the end, because its emotional. If/when there is a next time, at least our expectations - that it is a long hard emotional roller coaster - will mean we'll be better prepared. Hopefully.

Rather than letting them go to waste, I'll post some regression analysis for a selected suburb soon.

Quake AI Programming Book

2011-05-03T07:00:00.028+10:00

I intend to write a follow-up book to the Nature-Inspired Clever Algorithms book on Machine Learning. I have a lot going on this year, so I was thinking of postponing it until 2012. If I do decided to go down this road, I was thinking of taking on a different project in 2011 that would be smaller in scope, less taxing, although still interesting and rewarding.

I have been thinking of writing a book about the AI in the Quake series of computer games. I was thinking of either writing a book that analysed the Artificial Intelligence architecture in each game in the series, or analyze the AI in the bot modifications. Perhaps both. The book would walk through monster or bot case studies and describe how they fit together, think, and behave. Perhaps with small experiments and demonstrations along the way. The kind of book that would have captured me as a game programming hacker 15 years ago.

In pondering this idea, I thought it prudent to explore other books written on or related to this idea. The following is a list of books that I found:

Quake Series Programming Books

Focus On Mod Programming in Quake III Arena, 2002
Quake 4 Mods For Dummies, 2006

Related Programming Books

These are by no means the cream of the crop of game AI programming, and there are in fact many level design books in there as well.

All of these books are focused on teaching some form of programming or game development using an existing game as a medium. The advantage of the Quake series is that the source code is released under the GPL. The Unreal series and the Half-Life (Source Engine) series are not released as open source, although do provide access to some aspects of the source under restricted licence for the modding community.

It is clear that there is interest/demand for books on game development based on the Unreal series, which makes a lot of sense given their general success in licencing the technology.

Some concerns about tackling such a project include:

Interest: The games in the Quake series are old (10-15 years). The methods may be outdated, they may not be relevant to modern computer games, and it is more than likely that no one will care
Low Barrier: It is more than likely that no one has undertaken such a project because the barrier is so low. One can simply read the code and understand what is happening, no analysis is necessary.
Copyright: Although the source code is released under the GPL, the game assets are not. One may have to acquire a licensed copy of the game to do any meaningful development. Additionally, my use of game screenshots may be restricted (fair use!?).

There is some effort required to produce such a work. Getting each project setup may be involved, especially across the three main platforms (Windows, Mac, Linux). The work would be primarily analysis: reading source code, experimenting and communicating what is happening with diagrams and descriptions. This tinker-write cycle is slightly more relaxed than the deep research needed for each algorithm in a machine learning text.

Is there interest in the market? Would you read or skim such a book?
Let me know what you think in a comment or email.

The Little Taxonomist

2011-05-02T07:00:00.000+10:00

We are expecting our first child in a bunch of months and I have been thinking about all kinds of science experiments to perform on/with the bub. I had an idea last week for what I think is a cool little web app that allows a a parent and their child to catalogue the native species around their home and learn more about their local environment. I am referring to this idea as "Little Taxonomist".

I'll present the idea in the context of some stories:

Story #1
A father and son are curious about the plants and animals in and around their house and neighbourhood. In an effort to learn more, they decide to begin to catalogue the things they see in their backyard. They select a subject (say a flower, tree, or insect), photograph it, and note down a few descriptive phrases. They enter this information into a web application. The web application accepts the image and structured description and makes informed guesses (based on location and time of year and subjects collected by others) as to the exact species of the subject. The web application also suggests interesting subjects that are known or expected to exist in their area (probabilistically based on entries of other entries in the area), and some information about where they might be found, creating a context sensitive scavenger hunt. Slowly, a subject every few days, more on weekends, over weeks, they build up a catalogue of plants and animals in and around their home. Together they have learned more about the specifics of the suburban flora and fauna.

Story #2
A primary school class have a week or month long assignment that is a scavenger hunt of genera and/or species in the school ground. The class is split into pairs or small groups and allocated a flip camera (or equivalent single button point-and-shoot). They have a list of hints or requirements as well as blank pages, structured to capture basic taxonomic descriptions. Students return to the classroom and use the computer to copy the photos from the camera, drag them into the web application and add their descriptions. Students are awarded badges and points for the breadth and depth of species described, teams are ranked, and some indication of what other groups are finding is provided.

Vision
A web application for children to be completed in groups or with a guardian. The objective is to describe subjects in the local area (home or neighbourhood) and in so doing learn more about the local flora and fauna. The system encourages the collection of species by intelligently guessing based on brief descriptions and photos as to the actual known species. A gamaification layer is provided that includes badges, points, leaderboards, and similar extrinsic motivators. Additionally, the system uses the localized information in aggregate to suggest subjects to look for (to "collect"), and a probabilistic expectation that they can be spotted (you have a 90% chance of seeing a fruit bat between 5pm and 8pm by looking up). This probabilistic understating of what can be seen in the local area would be coupled with the gamification system, highlighting rare finds. The system would provide all data in aggregate (anonymize) allowing kids to explore what others are finding in their neighbourhood and the types of descriptions being used.

The following are some mock screens I hacked together in Google Docs:

Mock: Add Subject

Mock: List of Subjects

Forget kids, I want to use this. There might be a general case for adults with smart phones.

I am not sure whether I will build it yet, I figure it needs a 5-10 year old child to make it fun. I figure it could make money by selling some cheap cameras with a website subscription or maybe targeted advertising (kids+science).

A friend pointed me to Project Noah, which is a similar idea, but not the same.

I'm eager to hear what people think, seen anything like this? Would you use it yourself or with a child?

May Challenge: Touch-type Faster!

2011-05-01T16:30:00.000+10:00

We learned touch-typing in high-school. I think we did it for two or three years. Nevertheless, I suck at touch-typing, and as a programmer it may be considered embarrassing.

I can type without looking at the keyboard, so technically, I touch-type all day long. What I don't do is type as I was taught was "correct". I think that as a consequence I am starting to get some wrist pain.

Anyway, in the constant battlefield of self-improvement, I thought I would take the month of May as an opportunity to improve my touch typing and hopeful start typing "correctly" from the end of May onward in day-to-day computer interaction.

The May Challenge is to perform one lesson in touch-typing each day and to measure a standard typing word count each day. Hopefully this word count and/or accuracy will improve by the end of the month and my confidence in correctly touch typing will also improve.

At this stage, I believe I will use the free online service www.typingweb.com because it provides lessons and ad word speed tests, and of course it's free. I just took a test using the "correct" home-row based method and scored: 18 WPM average, 23 WPM gross, and 96% accuracy. It would be interesting to see what my score would be with my unorthodox method - a bad idea, I suspect it may reinforce the bad method.

Again, as with April, I will adopt a penalty-based approach to the challenge. For each day that I miss, I will have to donate $20 to an open source project of choice (not a .NET project as with last month). This is less relaxed (I support opensource, generally), because I am concerned that commitment required may mean that I miss a day or two here and here. We'll see. $20 a day is still a decent disincentive.

I'll be sure to summarize progress at the end of the month.

Image copyright Wouter Verhelst.

April Challenge Over: Watch One Tech Video Every Day

2011-04-30T07:25:00.001+10:00

I set my self a challenge for April to watch one technical video every day throughout the month. My rationale was that if I could apply the same discipline that I use to discriminate what I eat to what media I consume, that it would have a beneficial effect. Sure, fluffy, but the challenge was measurable and the penalty for missing one day was a donation of $20 to a .NET opensource project.

The month has come to and end and I did manage to watch one technical video each day, so no donations needed.

I intended to spend most of the month watching university lectures and Google Tech Talks, which I mostly did. It became harder and harder towards the end of the month to find an hour+ to watch a tech talk. I ended up catching a quick (15 minute) TED talk instead. TED talks are good (some can be great), but they are so brief that my retention is poor, and so high-level that I finish thinking that I have not learned very much at all.

The following image provides a breakdown of the sources for videos I watched. The "other" category includes random tech videos on youtube that do not fit into one of the other broader and popular categories.

I enjoyed almost all videos. I ranked each with a real-valued scoring between 0 and 5 and provided my own description in a spreadsheet. Five real highlight videos (in no particular order) were as follows:

Properties of Exoplanets: Planet detection, Kepler, Nature of planets found so far (bad sound, but whatever!)
Brains, Meaning and Corpus Statistics: Text statistics in, map of neural activity out!
Getting in Shape for the Sport of Data Science: Tools for data mining competitions, intro. to random forests
Sugar: The Bitter Truth: Sugar is bad for you, it is a poison
"All Questions Answered" by Donald Knuth: Q&A

I found that I read a lot less RSS feeds in my Google Reader. I also found that I consumed a lot less 'acquired' media in the form of US TV shows (selling the family media player helped here, no doubt).

I had a good time with this challenge and will attempt to stick to it and record my progress in a spreadsheet.

AIFeeds Part 4: Automating an AIFeed aggregator

2011-04-29T07:00:00.113+10:00

In Part1, we prepared a large list of RSS feeds and filtered them down to something workable. In Part2, we processed all of the articles in the feeds and presented posts from the last five days as a static RSS reader. In Part3 we used a number of JSON APIs for social networking website to gauge the popularity of articles and highlight those popular posts at the top of the page on our static reader.

In this 4th and final post in the series we will explore different ways to disseminate the results of our filtering.

Step1: Send via Email
Last time in Part3 we ended up with a result that was quite passable. The scripts generated an HTML page that promoted popular AI, Data Mining, Machine Learning, etc. articles from the last 7 days, followed by a listing of all those other articles that, though were deemed to be less popular, may be of interest - organized by day. A simple approach to disseminate the results of this script is to send an email. The objective here is to receive the equivalent of an AI-themed version of the most excellent Hacker Newsletter.

I have a Google Gmail account and I assume most programmers do. The first step is to prepare a script that can generate an HTML email message and use the Gmail SMTP server to send the email. We are not focused on mass distribution here, just emailing the results to ourselves, at the moment, on demand.

The built-in SMTP handling in the Ruby standard library more than meets our needs here. The Gmail SMTP details are also easily obtained. The result is a script with two simple functions: the first for building a standard SMTP message with support for text/html content (mimetype), and the second for connecting to the Gmail SMTP server and posting the email. The script provides a spot test that will ask for your Gmail credentials and use them to send you a hello world email. Easy as pie.

See sendemail.rb

See below for a screenshot of my Gmail inbox with the resulting test email.

Step 2: Cron Send Email
The next step is to prepare a script that can be executed each day, generate a summary of the AIFeed output and email it to you. The easiest way to do this on any Linux or Mac machine is with cron.

A variation of the listpopulardayarticles.rb script from Part3 is used as the basis for the email. A new function is defined that generates the html content of the email and sends it. The script accepts two parameters on the command line: a gmail email address and a gmail password. These credentials are then used to send the email using the script prepared above.

See dailyfeed.rb

To execute the script we can create a shell script that contains the call to the script and the Gmail credentials used to send the email. For example, the shell script may be called run_dailyfeed.sh and look as follows:

#!/bin/sh
cd /path/AIFeeds/part4/
ruby dailyfeed.rb [email] [password] >> /path/AIFeeds/part4/dailyfeed.log

The script is three lines: a shebang, change directory to the script location, and a call to the ruby script with parameters. Replace /path/ with the path to your AIFeed directory, and replace the [email] and [password] with your gmail login details. The output of the script (and any errors) are output to a new log file dailyfeed.log.

The crontab for the current user can be opened as follows:

crontab -e

Add an entry that looks something like the following:

00 5 * * * /path/AIFeeds/part4/run_dailyfeed.sh

This is all one line with tabs in between the fields. Again replace /path/ with the path to your AIFeed directory. Cron will execute the shell script once each day at 5am local time.

The following is an example of a resulting email sent to my email account.

Improvements and Extensions
This section summarizes possible improvements and extensions to this part in the series.

Cron is an easy way to schedule a task on your machine. A better approach would be to set this up on a server (such as Heroku, AppEngine, or AWS) and send an email to an email list (via something like MailChimp).
An interesting extension to this project would be to turn the output into a webpage that is re-generated every hour or so. This might provide a useful diversion to reddit and hacker news, with a targeted corpus of links to scan over.

In this fourth and final part in the series we have hacked together simple script to send email via Google Gmail and scheduled the script to email ourselves a list of popular AI articles each morning at 5am. Not bad for a few ~~days~~ hours work. Sure, there are some rough edges, but the result is entirely functional and I think useful.

If you would like to see this as a service or perhaps a website, drop me a comment or an email. I'd be happy to clean it up further and automate it for a broader crowd if I knew that others as passionate as me about AI and Machine Learning were interested!

Don't forget all code and data for this series is available on the AIFeeds github project.

AIFeeds Part 3: Intelligently filter AI posts by social impact

2011-04-28T07:00:00.122+10:00

In Part1 we learned how to prepare a master list of Artificial Intelligence and Machine Learning RSS feeds. In Part2 we learned how to process articles from the list of feeds and present them as either a fire-hose or in a more compact day-based manner. In this part we explore how we may filter articles by their perceived social impact.

As with Part2, the curated_feeds.txt file is used as the basis for locating articles to consider listing.

Step 1: Computing Social Impact
Computing social impact for a give URL involves determining how many times it has been shared, up-voted, stumbled, etc on popular social networking websites. All of the large sites offer APIs and almost all offer JSON APIs.

I started out thinking I would need a ruby gem for each social networking site, one for Twitter, one for Facebook, etc. I started looking around and found that there were some gems here and there and most were quite outdated. I figured that because most APIs were in JSON format, that I could just interact with each in turn manually. I stared reading API docs for various sites and quickly got bogged down. For example, searching for a URL on twitter is not enough because people use URL shortening services. In fact, these services are quite pervasive.

The requirements here are quite specific: For a given article URL, how popular is it in a given social networking website? After a little more searching around, I came across a very useful website on Shared Count that provides a listing of the JSON API calls needed to answer this question across a host of social networks. I combined this with some of the API code I had already written and built a little URL scoring sub-system.

Given that all of the APIs we are dealing with are JSON based, we need some robust functions for downloading and parsing JSON files. I installed the json gem and wrote a robust function for making arbitrary JSON API calls (HTTP GET requests) that used similar socket timeout handling as to that which was used in Part1 for RSS downloading.

See parsejson.rb

Next, the specific query formatting and data processing needed for each social networking website was prepared and tested based on the JSON downloading and parsing code. The scoring of a URL against a a given API was relegated one per file allowing any special handing (like MD5 digests in the case of delicious) and spot checks.

API handling was written for 8 services: (each links to the specific ruby code) Delicious, Digg, Facebook, Google Web Search, Google Buzz, Reddit, Stumbleupon, and Twitter.

Step 2: Article Scoring
The next step is to combine the scoring for a given URL. There are many ways to do this and the more thought and experiment put into the scoring, the more meaningful the scores will become.

The chosen method here is a simple sum of the scores. This first approximation will provide a number that can be compared, but the relation between the scores in the sum from different websites is unknown. For example, a large score on Facebook or Twitter likely has more meaning than a large score on the Google web search. This is an obvious area for further research.

A URL scoring function was prepared that evaluated a given address against each website in a different thread. Digg was dropped from the cannon of scoring functions because the results did not appear to be meaningful and the API was always slow to respond.

See scoreurl.rb

Step 3: List Popular Articles
Now that we can score articles by their impact on the world, we can combine the scoring information with the list of articles posted in the last week to create a list of popular articles.

Starting with the viewarticles.rb and listarticles.rb scripts prepared in Part2, an updated list of popular articles can be created. In this case, the score for each article can be computed and all articles can be listed by their scoring. This promotes those articles that are expected to be interesting to the top of the page with the intent that they may be read first.

See listpopulararticles.rb

The following provides a screenshot from the last time that the script was executed showing an example of the resulting populararticles.html file.

Step 4: Combine List by Day and Popular Articles

The resulting listing of posts is long, it contains 7 days worth of articles. Additionally, taking a closer look at the posts in the list shows that most of them have a score of 0. It is expected that the older a post is, the larger its potential to have a meaningful score. This is because it gets indexed and shared more over that time. It is also expected that most articles posted within the last day or two will not have a score. In fact, it is expected that most AI articles in general will not have a score - it is a niche field of interest after all.

In this step we separate "popular" articles (those with a score) from the "unpopular" articles (those without a score). Popular articles are listed at the top of the page and the remaining articles that have not achieved a score yet are listed below the popular articles, broken down by day.

See listpopulardayarticles.rb

The following screenshot shows an example of the resulting populardayarticles.html generated by the script.

Improvements and Extensions
In this section a number of improvements and extensions to this part are summarized.

An important improvement to this part is the assessment of the URLs. Specifically, permalinks reported by some ATOM and feedburner feeds involve a proxy that redirects to the page. The effect is that the popularity of these proxy URLs are assessed against the social networking websites rather than the permalinks themselves, resulting in incorrect scorings. This may be addressed by following the redirect in the proxy URLs or using a different feed parsing library that has knowledge of the different feed types.
A natural first extension to this part is to add scoring from more social networks and sources. For example: Hacker News, Google Blog Search, Friendfeed, etc.
The scoring used is very simplistic. A good extension would be to make better use of the information available for each URL and combine them in more intelligent ways. Such as a weighted sum of their relative importance, more weighting to comments and up-votes, etc.
Additional meta information about a given URL on social networking sites could be used in conjunction with an intelligent parsing of the articles themselves to determine broader categories that articles could be put into. A new list could be constructed with articles by category and popularity.
The generated HTML pages will only ever contain AI articles that are actually in the source RSS feeds. A useful extension of this project would be to monitor social networking sites and other news sites for articles on AI and Machine Learning and include them in the mix.

In this part we looked at the social interest of each article from the last 7 days and used that information to promote "popular" articles to be read above "unpopular" articles. The dynamic nature of the social scoring function means that popularity can be re-calculated each time the script is executed, responding to changes in the spread and sharing of a given story.

In the next and final article in the series we will look at automating the generation of the AIFeed and in disseminating it via email and the web.

Don't forget that all code and data for this series is available for free on the AIFeeds github project.

AIFeeds Part 2: Recent AI Articles

2011-04-27T07:00:00.104+10:00

In Part1, we went through the process of aggregating and filtering a bunch of RSS feeds for Machine Learning, Data Mining, Artificial Intelligence and related topics. In this continuation we will explore the properties of articles listed within a feed and provide a way of generating a static list of recent articles.

Step 1: List articles from the past week
A problem with this effort is that there are a lot of inconsistencies in the real world data. Some feeds are missing fields and some articles have fields that do not match expectations. The first step is to explore the data that we have by listing articles from the past week. This includes all fields that we think might be usefully related to an article, a feed, and its source website.

To start with, the final list of feeds prepared in Part1 was taken and curated based on eyeballing the output in feedlist.html (also generated in Part1). The result is the file curated_feeds.txt with some of the more obviously wrong and non-interesting feeds removed.

Next, a script was prepared to generate listing of all articles from the last seven days drawn from the feeds in the curated list. Almost all information for each article is provided such that we can discern what might be appropriate or interesting in further experiments.

It was interesting that some articles had neither a date_published or last_updated field. Some articles would put the content in the description and others in the content field. These may be artifacts of the chosen RSS library, or may be artifacts of the feeds themselves, more investigation is needed. There are also Unicode issues with strange characters all over the place where apostrophes and quotes should be. Again, it is not clear whether this is because of the library or because some custom parsing is needed, nevertheless, as a static RSS the result is very basic.

See viewarticles.rb

The following provides a sample screenshot at the time I last executed the script to produce the articlelview.html file.

Step 2: Summarize articles by day
In recent months I have become a big fan of the Hacker Newsletter. It is basically an email sent once per-week that contains a summary of the top stories of the week on the website Hacker News with links to comments as well as other relevant and interesting sections. I like the layout of the email and I think it provides a good pattern to head towards in this project.

The previous step explored how to filter the feeds and how to extract the relevant information from each article, but it also demonstrated how ugly rendering the raw data can be. One could invest effort into replicating an RSS reader, or in doing something simpler and move toward a better summary view on the same data. This step explores such a simplified view where each article is presented on one line.

A script was prepared to produce an article summary for the last seven days. The same filtering was performed as in Step1, although rather than writing out a long list of articles, the script was updated to group articles by date and only list the article title and source webpage. The result is a compact listing of recent AI and Machine Learning blog posts. The resulting page also provided clearer insight into the nature of the blogs included in the master list. As a result, the curated list of feeds was further culled.

See listarticles.rb

The following figure provides a screenshot from the last time I executed the script to produce the articielist.html file.

Improvements and Extensions
This section lists some potential improvements and extensions to this part in the series.

Curating the master list of source feeds by hand is a bad idea. It seems feasible that all articles from each feed could be scanned and their appropriateness to the project determined using heuristics on the structure or content (keywords).
Unicode characters in the feeds need to be rendered correctly. This still effects the titles of the articles and the blogs themselves.

In this part we have explored the content for articles in RSS feeds and two methods for presenting the data: a fire-hose and a daily summarization, the latter currently preferred.

Next in Part3 we explore additional social-web based methods for filtering the list of articles and promoting those that are more likely to be interesting to the reader.

Don't forget, all the code and data for this series can be downloaded from the AIFeeds github project.

AIFeeds Part 1: Artificial Intelligence and Machine Learning blogs

2011-04-26T07:00:00.195+10:00

Recently, I have been neglecting my RSS feeds. I'm a Google Reader user and my unread count is far from zero. I've been distracted and too busy. The problem though, is that there are many excellent Artificial Intelligence and Machine Learning blog posts that are being written every day and I get a lot of enjoyment in reading them and keeping abreast with current research and interesting projects.

An obvious solution would be to simply rely on social media to filter such articles and let the best float to the top on Hacker News, Reddit, and Twitter. Maybe.

I felt like some hacking over my Easter break (in between the numerous family responsibilities), so I decided to have a crack at this problem. This post represents the first of four posts of my exploration of aggregating and filtering RSS feeds from popular blogs in the areas of computer science, data mining, data visualization, machine learning and artificial intelligence. I've called the project AIFeeds and have hosted the code and data for all parts on github if you want to skip ahead.

The objective of this first part is to simply build a list of a large number of blogs that match my loose criteria. Specifically, the objective of this part is to prepare a list of working RSS feeds for blogs on (AI and related fields) without any duplicates. This will then be used as the basis for the subsequent parts in the series.

Step 1: Extract relevant blogs from my Google Reader Account
For starters, I have a modest collection of relevant feeds in my google reader. Going to google reader, I clicked settings, Import/Export, then Export your subscriptions as an OPML file. I opened the OPML file in a text editor and delete all those feeds that I knew were not on topic.

Step 2: Search for lists of relevant blogs
The next step is to Google search for lists of machine learning blogs. I remember seeing such lists on Quora and MetaOptimize before so I kept an eye out for these sites in the search results. Here are some good lists I found:

Good Machine Learning Blogs (MetaOptimize)
Machine Learning Resources (Inductio ex Machina)
Updated Machine Learning/Statistics blog list (Machine Learning, etc)
Data Mining Blogs (Data Mining Research)
What are the best blogs about data? (Quora)
What are the best machine learning blogs? (Quora)

The result is a bunch of links in webpages and OPML files. Additionally, many of the linked blogs themselves provide a blog roll to related blogs.

Step 3: Aggregate
The first easy thing to do is to write a script to extract all feeds from the collected OPML files. Using the built-in REXML library in Ruby I bashed out a script to read in all .opml files in an opml/ subdirectory, parsing each entry for the rss attribute and writing a opml_feeds.txt file.

See parseopml.rb

I then did something stupid, slow, but effective. I clicked a lot of links, read a lot of blog rolls, viewed source on a ton of blogs, and copy-pasted links to blog RSS feeds into a byhand_feeds.txt. I could have written scripts for this, but there is a lot of pain to deal with and I suspected I could do this once of task by hand faster than I could write code to parse a bunch of pages, extract (the right) links, and then download the pages and extract links to rss feeds. I took an hour or so to do by hand - not a big deal.

Step 4: Filter list of feeds
So now we have two lists of RSS feeds. Some may work, some may not. There are likely many duplicates and even many that are not strictly on topic. An easy quick win is to write a script that opens a connection to each url, parses the contents and builds a list of unique feeds.

The first thing that is needed is a library that can process RSS feeds in ruby, at least RSS1, RSS2 and Atom. Google throws up a few options, such as RubyRSS, Ruby RSS in the stdlib, Ruby-Feedparser, Simple RSS, feed-normalizer, and others. I tried a few and was somewhat frustrated. I ended up going with feed-normalizer (rdoc) because it was really easy to use and provided a consistent data structure for RSS and Atom feeds.

I wrote a small script using feed-normalizer to parse a feed and return the data structure. Importantly, it provided safety for the URL failing to open, the XML being bad, and for socket timeouts. An error is reported to stdout on error and a nil is returned. I used this script to test some rss, atom and even some bad URLs. This file provides a home for helper functions for loading and parsing RSS feeds.

See parserss.rb

Using this parser function, I wrote a script to read in lists of RSS feeds in text files (opml_feeds.txt and byhand_feeds.txt), parse each and build a unique list based on the website name. There are many hundreds of feeds to check in those files and I'm lazy. I wrote a small function to hammer the feeds in the file using lots of threads and then check all of the data structures in memory. Lazy but effective. The result is a list of 236 rss feeds in filtered_feeds.txt.

See filterfeeds.rb

Step 5: Summarize list of feeds
A final step is to generate a summary of all the available feeds. An easy way to do that is to generate an HTML page that summarizes each feed in turn, ordered by the name of the website. A script was prepared to parse each feed and list the name of the site, link to the site, and finally link to the actual rss feed. The list was ordered by site name.

See listfeeds.rb

The following provides a screenshot of the generated html page providing the final list of webpages and their rss feeds in feedlist.html.

Improvements and Extensions
There are a number of improvements and extensions that could be made to the first part of this effort, such as:

The final list contains some oddball and even some off-topic feeds. A high-level human-based curation of the list would be a good start. For example, removing Q&A feeds and delicious feeds would be useful.
Some of the links to the sites in the generated feedlist.html are suspect, meaning that some feeds do not accurately report the site URL or site title. It may be possible to carry over this information from the source documents, either the website themselves and/or the OPML files. Ideally, the final corpus of feeds could/should be stored in an OPML file for portability and reuse.
Build a larger corpus of feeds. Naturally, this is the easy extension to the whole project and there are may ways to go about collecting more feeds starting with some simple google searches. Getting more OPML files and including them in the opmlparser.rb script would be a great first step.
Write a script to extract links of blogs from some of the above listed collections. Once such links to blogs are collected, write a script to download the HTML, parse it and extract the link to the RSS or Atom feed. This is most commonly listed in the head tag of the HTML file as an alternative link (at least it is for blogger and wordpress blogs).

Done
So we now have a list of RSS feeds for blogs, most of which are likely to provide posts on or related to our topics of interest. Each feed has been tested, and duplicates have been removed.

Next, in AIFeeds Part 2 we will use this list of sources to prepare a summary of posts as a simple static RSS reader.

Remember, all data and code is provided on the AIFeeds github project.

Automatically Generate Regular Expressions (regex)

2011-04-04T07:17:00.000+10:00

I had a thought about two weeks ago that I really wanted a web app that would generate regular expressions for me. I was lazy and had spent some time hacking expressions together for a script I was working on.

I wrote some notes and posted them to a gaggle of programmers and engineers I email regularly. Something like:

A programmer normally google’s for examples of their problem or similar problems online, copies the code and adapts it for their needs. There are websites dedicated to discussing programming problems, and websites dedicated to listing solutions to common problems. The objective of this system is to write a new program for the user from scratch that exactly addresses their needs.

A user defines some sample input and some sample output. The system examines the data and generates a program for the user to perform the transform. This program can be downloaded, executed in place, or used as a service. The more information provided by the user, the better the robot programmer can do its job. The robot programmer may not get it right the first time, providing an opportunity for the user to work with the robot to better define the problem and tinker with candidate programs that may be used as the basis for the robots second and subsequent tries.

I abstracted the problem and proposed a generic "robot programmer" or "robot coder" that would compute arbitrary transformations based on sample input and output. I pitched both the general case and the specific regex case and got a variety of responses and some good technical discussions.

Using Heroku, I hacked together a prototype of the interface and sent it around for comment. The following is a snap shot of the regex mock:

The following is a snap shot of the general case mock, here the example requires a program to print the input as well as all the input summed together.

Ideally, the service would be free for the simple case, and cost a dollar or two for a more advanced case. The free case would allow a large corpus of examples to be collected and the system refined, and the complex case would provide useful pay-only features such as: save, private, download in preferred language, and pair-program with the system. This last case would be the truly useful incarnation of the system where one could modify the expressions or programs and I/O and iterate the solver until a desired solution was achieved.

I still think it is a good idea, but I just couldn't think of a good way to monetize it. It is an interesting technical challenge and the service would make an interesting novelty, but it is not an earner and I'd rather hack on other things for now.

I have some designs and small technical prototypes for the ensemble of inductive automatic programming solvers and heuristic solvers that would be needed, but I'm keeping the details to myself for now, just in case the idea gets some traction or I can think of a better way to make a few bucks from it (Excel plug-in?).

Sure the general problem is intractable, but the specific case is results-driven. It only has to be good enough for the users needs - a monkey patch for whatever data conversion or scripting they happen to be doing at the time. The subset of actual problems that people have will be much smaller than all of possibility space, and the web is a fertile ground for seed and test instances.

Anyway, if anyone has a burning desire for either or both of these services to exist or has some further ideas, drop me a comment or an email.

April Challenge: Watch 1 Tech Video Every Day

2011-04-01T07:01:00.000+11:00

I was recently thinking about my junkfood media consumption. My thesis was that one could apply the same discipline that one uses for discerning the food they eat to media consumption, ideally with the intention of self improvement.

If one must consume to relax (as opposed to drinking gin), then why not impose discipline to avoid the fast food of consumption (reading online news, twitter, American TV shows, etc.) and consume with intent.

The April challenge is self imposed (unlike the March Challenge) and is as follows:

Watch one free online technical video for each day of April. For each day missed, you (I) must donate $20 to .NET open source projects.

I'm experimenting with a "loss" as a motivating facto, rather than the "gain" of the March challenge. I'm really starting to get into this public challenge approach to self improvement :) Also, I recently sold our family media solution, so there is far less opportunity for general lazy consumption.

Seriously, why would one write open source using the .NET framework? It has oxymoron written all over it. Donating to open source is good, and $20 is not too much pain if I miss a day. It quickly starts adding up though, 5 days is a $100 and I'd much rather give that $100 to some noble device driver hacker working on Linux than some .NET hacker playing around in Visual Studio, or whatever it is they do.

This challenge does not preclude my normal media consumption, although it imposes constraint on at least one hour of that dedicated time, typically in the evenings. If I have a busy night, I guess I'll need to make time in the morning or during my lunch hour. I intend to maintain a spreadsheet of what I watch, it's duration, and its URL. I'm sure I'll think of ways to slice and dice this collected data.

Some sources of media may include:

I am passionate about technical topics, specifically Artificial Intelligence and Machine Learning, so this directed form of media consumption does align with my interests. I have to focus and concentrate for tech talk, take notes, think, quite different from what I need to do when I consume acquired TV shows. It will be interesting if I can/should switch to more passive consumption for interesting subject matter (what if I forget something amazing!?!). Alternatively, the penalty for remaining active in "relax time" may be worse, irritability, increased tiredness, etc. It will be an interesting experiment.

I'm looking forward to this challenge!

Image copyright by schmilblick.

March Challenge: All Over

2011-03-31T07:10:00.008+11:00

Today is the last day of March, and the last day of my March Challenge to lose 10 kilograms. I did my final weigh-in this morning. The results were as I had expected earlier: I did not lose enough weight to get an iPad. No big deal, I didn't know what I would do with one anyway :)

I lost a total of 8 kilograms (17.63 pounds), falling 2 kilograms short of my goal. The following graph shows my weight loss over the month in kilograms and days. I think that big initial drop was some water weight or something. I remember my first few runs took a bit out of me, sweated a lot and weighed myself right after the runs. In recent weeks I've been eating light lunches and don't even feel hungry (shrunk stomach?)

My regime was strict. I weighed myself each morning, typically before breakfast. I stuck to a slow carb diet and avoided all white carbohydrate foods (rice, flower, sugar, potato, etc). I did not have one day off a week as suggested in that diet. I did crave foods, but not badly enough to require a day off the diet. I ate a lot of beans, meat, and vegetable, with one piece of fruit most days (technically fruit is bad, natural sugars and all). I kept making the same dish (chili con carne with tons of kidney beans) and ate it one or two times a day. I also had a week and a half of canned tuna lunches. I mostly stuck to a small serve of baked beans for breakfast. If we ate out, I carefully chose my dishes, focusing on lean meat and vegetable. I did not stop drinking coffee (black, no sugar), and stuck to a glass of red wine each night (a man needs to relax). I did have one sneaky beer over the whole month, it was worth it!

I walked for 1 hour every work day (I do anyway), and I ran 5km most days either before or after work. I qualitatively noticed more wait loss when I stopped running for a few days!? Once I noticed, I stopped running all together (last week and a half of the month). This might just be the inertia of my system or an interplay of cardio and the diet...

I also measured the circumference of my belly at the belly button each day. I have sort of a pair shape, so this seemed an effective measure of mid-section fat. I lost a total of 6.55 centimeters (2.57 inches) from my belly! The following shows my belly circumference decrease in centimeters and days. That data shows a lot more variability than weight.

I feel that the month was a success regardless of meeting the initial (and aggressive) 10kg target. I feel fitter and I also feel like my brain is working better. I have been more productive both at work and in my after hours pursuits. More recently, I've been abuzz with ideas, and am furiously writing blog posts, talking peoples ears off and writing a lot.

My wife has proposed a 15 kilogram challenge for March through to the end of April. That means I would need to lose a further 7 Kilograms over the next 30 days (easy enough). I am sticking with the daily measuring schedule, exercise, and diet. If I make the April 30 weigh-in then that is a bonus. Ideally, I would like to hit close to my running weight, but I'm still a fair way off.

If I had to do the challenge again, I would change the design. A loss is a bigger motivator than a gain. Rather than gaining an iPad for winning the challenge, I should have had to buy an iPad for someone else, putting my out of pocket for losing. To make it more interesting, I should have to buy an iPad for someone I don't like very much, or for someone who doesn't need or want it. That being said, I don't want to do that for the so-expanded March-April challenge.

I will re-visit these graphs at the end of April.

Buying a House at Auction: Some Lessons

2011-03-22T20:30:00.032+11:00

Last weekend we attended an auction with the intention to buy. We did our homework and expected to come home with a deal or nothing at all. We ended up not getting the house, which was fine, it fit into our expectations. Before we move on to more harrowing house auction experiences, I wanted to capture what happened, the things we did right, and some lessons we learned for next time.

We got to the house early and did the pre-auction walk through. We opened cupboards, draws and generally tried to get a final feeling for the quality of the house. We did the walk through with my parents who had no inhibitions about commenting on all the good qualities on the property (shh). This was good because it confirmed some of my own thoughts, but also further tipped our hand that we were interested in the place to anyone listening in. The auction kicked off with a solid but expected figure. I stood and tried to look as nonchalant as I could. There were no bids. The auctioneer put in a vendor bid of an extra $10K. One, Two, Three.

The auctioneer walked back inside and conferred with the vendor, and eventually came back out. "Time to get serious" he said, looking right at my wife and I. The auctioneer's minions moved about the crowd, and one started needling my father, and then my wife and I. "This is your last chance to get an exclusive opportunity on this property". Whatever. I stood my ground. I my head I was thinking "No one has bid, the market is not interested in this property, let is pass in then negotiate". Last call, One, Two... (my wife turned to me "Should we bid $5K?" she asked. "Nope" I said.) ... and Three.

The auctioneers made noises about a Dutch auction. I wasn't too concerned. I saw one or two other couples walk into the house. We walked into the house knowing we could walk out at any time we wanted, it felt powerful. We learned the unrealistically high reserve and made a low offer, which involved reading and signing the contract. It was rejected. We inched up the offer. It was rejected. We made our final offer at our pre-agreed fair price and draw a line in the sand. Eventually, it too was rejected. The vendors showed movement in their expectations, but not enough. We learned that there were three people in the room all trying to work the deal - a brother, sister and the sisters husband. I could imagine bickering and maybe even fighting, either of which might have worked in our favor if one of the group was powerfully motivated to sell.

The agents worked us. In retrospect we didn't have to budge a dollar from our first offer. There appeared to be no one else in the house putting in offers, and the lack of bids suggested the markets sentiment towards the property at the asking price. The agents worked a few angles, trying to befriend us and convince us that they were on our side and wanted a deal, and eventually pushing us that we needed to get serious and make a reasonable offer.

All -in-all it was a great learning experience, and most importantly, it didn't cost a cent. We noticed that the property is now back on the market for private sale at a price closer to our final offer. Maybe they'll call back in a week or two. In the mean time we have lined up a number of follow-on candidates and I'm looking forward to the games.

Advice we were given: (I'm paraphrasing)

Go to lots of auctions and get a feeling for the process. (awesome advice)
Bid as little and as late as possible. (awesome advice)
Be civil, don't be agressive (awesome advice, it's a big game)
Agents lie, it's about their bottom line. (awesome advice)
The reserve price is usually 5%, 8%, 10%, even 20% above the asking price (I've seen it many times in practice!)
Stories of aggressive bidding, making the first bid and intimidating the other bidders from the beginning. (did not gel with me at all)
Read the Section 32 many times, a solicitor can read it just as well as you can, why pay. (great advice)
Location is what matters. (maybe for resale, but so does a comfortable place to live)
Size of land is what matters. (sure, if you want to demolish and build units, not applicable)
Pay extra to get a place you love. (na, humans are very adaptable, and houses can be renovated)
You must get a building inspection. (maybe, but only if I'm suspicious of the vendor/agent/property)
Schools is what matters. (agreed it is a consideration)

What we did right:

We were not emotionally invested in the property. It was one of many and we talked plainly about the good features and the bad, and focused on the things that needed to be fixed/renovated and their cost relative to a fair purchase price.
We worked hard to estimate a fair price for the property based on an investigation of the area and recent sales. Regression is hard simply because the data is not available, but attending lots of auctions and looking at lots of houses helped to build a qualitative model.
We clearly defined and agreed upon what constituted a deal, a fair price, our soft limit and our hard brick wall limit in terms of what we were willing to spend on the property.
We read, re-read, and re-re-read the section 32. We researched all things that looked interesting and generally learned more about the documents it contained.
We spoke with the agent a number of times. We called the agent and asked a bunch of dumb questions. This clearly tipped them off that we were interested in the place, but also allowed us to learn more about the property.
We asked lots of questions about the vendors. We asked about their status (brother and sister), were they selling to buy (no), was it an investment (yes), etc.
We talked to neighbours and asked about the street and the other neighbours. This is all to my wife's credit. She seems to have no fear in approaching random people and speaking plainly. We got some useful tidbits.

What we did wrong:

We did not get a copy of the contract before the auction. Normally the contract is provided with the Section 32, but in this case it wasn't.
We should have addressed a builder's inspection before the auction. We tried to organize this late in the week before the auction and even tried to have the contract amended on the spot to be subject to a builders inspection. Both of these measures were too late (although, I thought we should be able to amend and initial a contract - it's just a contract after all).
We should not have increased our offer. The market was not interested in the property at the auction price. We should have made our offer at the auction price and left it there. This is debatable, because one could argue that the auction price was not necessarily a fair price (the vendor needs to be placated as well).
The only stream of information is from the auctioneer and his minions. How do we really know what the vendor was saying or thinking? All we have to go off is what we have been told by the auctioneer. Next time I will listen to them less and try to just think through the situation.

I get the feeling that there are a lot of anecdotes and lot of folk advice, but few people actually study this process (I do know of some exceptions). Sometimes I get quite conflicting advice from seasoned house buyers. With that in mind I'm sure my "lessons learned" would be a different story if I had messed it up. I just hope I can keep my head, try to think rationally and see the big picture next time.

If you've got any additional advice, I'd love to hear it.

Image copyright by Ian Muttoo.

Sony NEX-5

2011-03-15T20:57:00.000+11:00

So I bit the bullet and bought the Sony NEX-5, and I'm happy. The camera was built for me.

It has the body of a compact, the sensor of a DSLR and allows me to switch lenses as needed. It provides point and shoot in an "intelligent automatic" configuration, and then provides all the DSLR features in software for when I can be bothered.

I don't want to list all the facts and features. There are plenty of sites that do that and I'll link to them. I'll just go over the few things that I'm really pleased with.

The camera can do Full HD at 1080i resolution and stereo sound. I bought the version that included two kit lenses, specifically a 16mm pancake (wide angle) and an 18-55mm zoom. There is a 200mm lens around for it that I'm secretly thinking about. The camera has hardly any buttons and all the core functionality is handled in software. Buttons are programable, so I can configure my favourite features to be one step rather than menu-driven. The camera features slide-panorama, adjustable background defocus (effect- rather than setting-driven behaviour), auto HDR and the list goes on. Mucking around with it, my happy-snaps all of a sudden look a whole hell of a lot better!

I managed to bargain in a 16GB memory card and carry bag when I purchased it, so I was ready to go immediately. The only thing I think I might need at this stage is a second battery.

For some more information on the camera see:

Sony NEX-5 on Wikipedia
Official Sony product page
dpreview review of NEX-3/5 (must read if you're keen on this beast!)

Check out the following Sony advertisements. Specifically, the David McLain shorts are good clean fun.

The day after I bought it I set myself a small assignment to try out as many features as I could throughout my normal day. I managed quite nicely and thought I'd put up some of the nicer shots (click to enlarge). I'm clearly no photographer.

The following shot was taken with the background defocus feature and the 55mm lens.

The following is a shot taken with the macro feature and the 55mm lens.

Finally, this shot was taken with the automatic panoramic sweep option that stitches the image together in the camera.

As I said, I'm no photographer, but this camera is a joy to use.

I'm trying to organize a "go to the zoo and play with camera's" day with some mates to really burn in this new toy. I need to hang around some guys who can remind me what ISO and aperture are all about again so I can make better use of the features. Looking forward to it!

Junk food media consumption

2011-03-14T08:48:00.150+11:00

Make things. Work on stuff that matters. Produce. I try to optimize my time such that I'm producing more than consuming, but we all have limits. You cannot work all the time, and more than axiomatic advice, I have learned this lesson the hard, more than once. Thinking turns to mud long after the sun goes down and no doubt productivity with it. Over an extended period, eventually you burn out.

I make time in the early mornings before work to hack and I do the same thing with pockets of time after work. But I also make time to not-hack. I'm not just talking about making time for loved ones or whatever, I talking about making time for active consumption. I now take it as assumed that one needs to turn off, that that turning off more than likely means consuming media.

So, is sitting in front of the TV for a few hours a total write off? Is it better or worse than reading news articles on the web? How about reading science fiction?

We're not massive TV consumers in our house, we have a small CRT and if when we do use it, it's to watch an episode of a carefully procured show via our Media player rig. To find out what is happening in the world or what the weather will be, we check a local or internal website, typically once or twice a day (generally we don't consume broadcast TV).

Nevertheless, even with a "carefully procured show" or movie, I feel guilty after sitting impassively for an hour or two. I could be watching a documentary and taking notes, I could be reading a text book and taking notes. Should I be?

The natural breakdown (in my head) of permissive media consumption is as follows (descending):

Socially (the other humans)
Reading (books, the web!?)
Listening (radio, music!?)
Watching (video, TV, movies!?)

Here, I use permissive to refer to general social acceptance. For example: I watched LOST last night < I listened to radio national last night < I read an awesome book last night < I went to a dinner part last night. Here the ordinal relationship between these activities is something like utility or benefit. One may be interested in optimizing such things.

Maybe this is generally accepted, at least I believe other people around me might have similar opinions. The breakdown might be as such because there may be built-in assumptions around the level of attention required or the level of knowledge acquisition. I don't like this argument all by itself. How useful is having partial memories of random tech factoids in your head from the 20 technical articles one reads in a sitting? Is this better or worse than one deep but inconsequential conversation, or compared to reading a Hugo award winning novel? Naturally, the argument is not so simple. What I have pointed to is the obviously orthogonal concern of content.

A lot of my non-work time is concerned with consumption of some sort or another. I want to be productive and make good use of my time. Can a balance be found? Can or should one optimize or actively consume media to "turn off"?

I have been on my health kick for nearly two weeks now and I've been thinking. It is easy to snuggle up and watch an episode of some TV show, but is that like eating junk food? To be generally making better use of my time on this planet, should I in fact be watching something more education? A documentary? Something more cultured? An opera? Should I be reading the books from my shelf more than slurping down popular tech articles?

One has to eat as one has to "turn off", but I now think the discipline one applies to the consumption of food applies just as equally to ones consumption of media.

Do I simply have the taste for science fiction television shows and technical new articles on the web? Are they fatty foods I could be doing better without or do they broaden me, my imagination, my dreams? ultimately my personality? Is this even testable?

There's lots of rhetoric here, I don't the answers, but I am also beginning to think that this generalized discipline of consumption may also apply to production and making things.

Image from Wikipedia.

Publishing a book on Lulu

2011-03-13T09:01:00.098+11:00

I published my most recent book "Clever Algorithms" using Lulu. The process meet my expectations and I cannot be happier with how the whole process unfolded.

I thought it best to capture some of my notes and findings before I forget them completely. I do intend to write and publish more books in the future, and I will more than likely use Lulu for the process.

To get started, I bought two books from Lulu with the same general market as my book. I wanted to get a feeling for the quality of the printing, the feel of the paper, the clarity of the cover images, and generally a feel for the publisher themselves. I purchased:

A Field Guide to Genetic Programming, by Poli, Langdon, McPhee, 2008 (sales rank 368)
Essentials of Metaheuristics, by Luke, 2010 (sales rank 25,727)

The first book on Genetic Programming was very close fit for what I was trying to achieve. They provided a free PDF version of their book on Lulu, and priced their paperback low. Also the form-factor matched what I wanted for Clever Algorithms (6x9 inches).

The second book also had a similar model, providing a free PDF on a separate webpage and a modestly priced paperback. Interesting, Luke chose a USLetter form-factor for the book, likely keeping with the "course notes" intention of the text. Also, the paper quality was better, where Luke seemed to have chosen a whiter white rather than the default beige.

I thought both paperbacks were well made, the binding was solid, the books held together nicely. Most importantly the text was clearly presented.

For the cover, I chose 99Designs. I chose to go with US Trade 6x9 inch sized cover and provided templates for the front and back covers as well as the one piece. I bashed out a brief with some help from a friend and spent a lot of time over the course of the ~7 day competition discussion my expectations designers and providing detailed feedback on candidate designs. Eventually, I selected a top 8-10 designs and posted a message to the MachineLearning reddit (as well as friends and colleagues) in order to tender votes (you can still see all the finalists up there). Ultimately I went with my gut and picked what I thought was the most complete and suitable design.

Once I had a cover, I printed a proof copy of the book even though the copy was not finalized. I ended up printing 2 or 3 proof copies to get the cover size, spine size and barcode perfect. I found some articles on the Lulu support website useful during this process, answering the general questions I had such as:

Lulu provided the ISBN and I used another program to generate the barcode image. I then used GIMP or Photoshop to add the barcode to the one-piece cover image and convert it to PDF.

I prepared all my book content in LaTex (available under Creative Commons on github). The PDF created by my LaTex Makefile did not pass the Lulu checks. I think I had some embedded fonts inside some GNUPlot images. I simply used the command line too ps2pdf13 to embed all fonts into the PDF.

Finally, I purchased a distribution deal to release the book on Amazon and to make it available to traditional book stores. This and availability on Google Books are still pending, but should come through soon enough.

One area of weakness has been eBook format. I didn't consider it, and in fact, I still don't think reading my book on a Kindle or an iPad is appropriate - it's a reference text. I need to think about this area, maybe even get a device and start paying around before I make up my mind.

All in all, I've had a very positive experience. If I could change one thing, it would be the Lulu sites ability to report on the number of PDF downloads. I know there must have been many thousands of downloads, but I can only speculate - I'd love some hard numbers.

Pondering a new Camera

2011-03-12T15:31:00.001+11:00

The last digital camera I bought was in 2006 for a trip to the States. It is an Olympus mu Digital 600 (pictured left). It is smal, fast, 6 megapixel but generally takes crappy photos. Specifically, the photos are really really grainy!

I need a new camera. Bad.

I did some thinking and figured I have 3 options: 1) Get another point and shoot digital and likely be let down again, 2) Get a full on digital SLR and re-learn how to drive one of those crazy things, 3) get an compact with DSLR-like features and try out the middle ground.

What the hell do I know about digital cameras? I at least know that the crappy camera on the back of my mobile phone is not good for anything other than ad hocery. I sent the word out to my camera-knowledgeable friends, who came back with some generalities:

Point and Shoot: Small and high-resolution. Slow to focus. Poor zoom. Poor in low light. Look for fast shutter speed and selective focus.
Compact: Cheaper and smaller than a full DSLR. Better lens than point and shoot. Close to the same functionality as a basic DSLR.
DSLR: Full control over everything. Expensive and physically large. Likely lack video support.

I also got some specific product and brand suggestions:

Point and Shoot: Canon IXUS 220 HS
Compact: Canon PowerShot G9, Canon PowerShot G11, Sony NEX-3, Sony NEX-5
DSLR: Sony SLT-A33, Sony SLT-A55VL

Recommendation of the Canon G Series came from two different groups of people. I note that the most recent model is the Canon PowerShot G12. Reading up a little on the Sony NEX series suggests some seriously cool tech on a seriously tiny frame!

I think a compact is the way to go and I think the Sony and the Canon are good contenders. I intend to get myself to a camera shop or two and try out some of these models before making a decision.

Thanks to Rod, Martin, Rick, and Hasan for the advice.

10 Great Ideas for using Clever Algorithms

2011-03-08T20:56:00.000+11:00

The Clever Algorithms book provides working, although minimal, implementations of 45 algorithms from the fields of Computational Intelligence, Biologically Inspired Computational, and Metaheuristics.

The intent of providing minimal implementations is to allow people to play with the systems, to make them come alive, and most importantly, provide the basis for extension.

I'm really keen to see what people can do with the algorithms. To see the creative side in how these methods can be applied, not just in science and engineering.

In this post I'll suggest 10 projects uses the algorithm implementations, and maybe provoke something interesting. The majority of these ideas are proposed as exercises for learning more about one or more algorithms described in the book or about the genetic research or application of such methods.

The following 5 ideas relate to a single given algorithm.

Change the demonstration problem. The problems presented in the book were idealized and simplistic. They were selected for the simplicity and ability to demonstrate a given method. A great place to start would be to simply change the problem being solved by a give algorithm. This may be as simple as changing the number of dimensions on a continuous function optimization problem, to changing a method to work with a new representation all together. Remember, most methods are generic, therefore careful attention must be paid to the both the representation and the chosen cost function.
Extend an algorithm implementation. Many of the implementations provide a minimal working version of the algorithm. This leaves a lot of room to add in all the fancy bells and whistles (extensions discovered to improve performance over intervening studies). Most of the time I mention these extensions. In some cases, such as many Evolutionary Algorithms, one can extend an algorithm implementation by switching out a high-level operator. These algorithms are described in such a way that the principle operators and sub-processes are modular and can be exchanged. Perhaps for starters, consider switching out the selection, crossover, and/or mutation genetic operators in the Genetic Algorithm or related evolutionary algorithm.
Study an algorithm. Something I really get a kick out of is going deep on a given method. Vary one parameter at a time and control all others, including the problem domain. Run each variation multiple times (control for the random starting position) and evaluate the algorithm configuration on each configuration. You will quickly build up matrices of configuration-performance behaviours that you can slice, dice, and graph. Sample parameters randomly or test them in a tightly stratified manner (such as each 0.01 increment between 0 and 1 for the crossover rate in the genetic algorithm). By performing such studies across multiple test problems or across multiple algorithms you can start to define equations that describe the behaviour of a method under specific conditions. You might even learn something new about a method (and publish)! These are complex systems, there is much to learn.
Implement another algorithm. I only managed to describe 45 algorithms in the book. There are many many other algorithms and variations I didn't have time to describe. Pick one and implement it. If you're super keen, try describing it using the same algorithm description template.
Apply to something fun. One can run off and attempt to solve difficult science and technology problems where there is limited domain knowledge, complex constraints, and where such a method would be appropriate. There are also those problems that are slightly less serious, but whose outputs can be very provocative. For example, start framing problems that require innovation or creativity as optimization or function approximation problems and apply a technique. For example music composition, art, and types of design problems. For inspiration, see these 5 fun genetic algorithm examples.

The next following 5 ideas relate to a collection of algorithms, maybe all the algorithms in the book.

Race algorithms. The natural thought when one has a bunch of competing methods at ones disposal is to race them. Although I discourage this behaviour in the book (from a scientific merit perspective), it can be enticing. Construct a bake-off between a number of closely related methods, fairly tune each in turn on one or a small suite of test problems and test-away. You can produce some thought provoking graphs and perhaps begin speculating about the relative merit of this method or that. Remember to think deeply on the actual transferability of your findings to other algorithms or problems.
Port algorithms to a favorite programming language. Ruby is pretty awesome in my humble opinion. It is concise, productive, and fun to program in, which is why I chosen it for the implementations in the book. That being said, it is not everyone's favorite programming language. Consider porting one or more algorithms to YOUR favorite programming language. For example Python is a natural fit given the close similarities to Ruby, but I'd be keen to see ports to Javascript, ANSI C, Go, Java, even LISP languages like Clojure. You would be sharing these interesting AI methods with people from your language tribe while learning the innards of each algorithm as you ported them. Win-Win.
Implement a plug-in for existing software. There are many great software packages out there that might benefit by having some of the optimization and function approximation methods described in the book provided as plug-in's. For example, one might write plug-in's for WEKA, Matlab, Mathematica, or Octave.
Implement a software library. Rather than simply port the algorithms to a new language or provide implementations as plugin-in's for existing software workbenches, one could build a new library of optimization algorithms. One could start building a library of genetic Artificial Intelligence methods, data representations and visualization, or could focus on a specific problem domain and build up a suite of methods specifically for solving it (such as continuous function multiple objective problems)
Translate the book. Not really an algorithm thing, but if you can speak (write in) another language and want to really absorb this content, I can't think of a better way than translating algorithm descriptions to another language. Chinese? Russian? Spanish? I'm sure you would have a gaggle of devoted fans for doing so.

All of the content (including the algorithm implementations themselves) have been released under a permissive Creative Commons license. Go nuts! If you do end up exploring one or more of these ideas, shoot me an comment or email. I'd be very keen to see what you come up with!

If there is any general interest, I'd really like to add a "projects" section to the Clever Algorithms website. A place where one can suggest a project idea, read over tutorials on applying the algorithms in the book, and download interesting code examples. Let me know what you think?

Computational Intelligence on Wikipedia

2011-03-06T08:14:00.000+11:00

During the hoopla of the Clever Algorithms book launch, an interesting question came up:

Why didn't I just write a whole heap of Wikipedia articles (to give back) rather than write a standalone book?

It's a good question. I started out writing Wikipedia articles. I wrote an entry for Extremal Optimization way back when. I expanded the Artificial Immune Systems entry and wrote an entry for the Clonal Selection Algorithms. I also wrote a bunch of other articles. I enjoyed the interface and the fact that many thousands of people would benefit from the information every day.

The more I contributed, the more I disliked the lack of control. Although I'm a world expert on this or that algorithm or method, I cannot self cite. Sometimes I can mold an article to my will, other times some random person or bot will come and wipe out carefully crafted content. Ultimately, Wikipedia did not align closely enough with my desire (need!) to present a large corpus of algorithms in a consistent way.

Anyway, I recently trawled through Wikipedia and collected a bunch of algorithms that are in or could have been in my Clever Algorithms book. I then organized them by the rough taxonomy I used in the book. Rather than keep this hidden away, I though I may as well share it more broadly than my Wikipedia user page.

The Field

Stochastic Algorithms: Stochastic Optimization

Evolutionary Algorithms: Evolutionary Algorithm, Evolutionary Computation

Physical Algorithms

Probabilistic Algorithms

Swarm Algorithms: Swarm Intelligence, Collective Intelligence

Immune Algorithms: Artificial Immune Systems, Immunocomputing

Clonal Selection Algorithm

Neural Algorithms: Artificial Neural Networks,

A strange mixture of seminal methods and techniques I've never heard of before! If I missed anything interesting, please post a comment. I note that there category lists, such as the Optimization Algorithms category, but the mixture of algorithms in there is diverse to say the least.

After scanning through a couple of entries, the differences in organization and quality of the content is obvious. What sucks is that each article gets such a high ranking in a Google search result, although the quality is generally so poor. Nevertheless, I find it interesting to see them all together and compare entries. I might even prepare a similar set for Scholarpedia or Mathworld, two sites that also have entries for some of the algorithms described in the book.

Image taken from Wikipedia.