Never Read Passively

What does it take to create for the creative?

2012-07-29T16:34:00.004+10:00

With my work on Mobile Game Engines, I'm in awe of what mobile game engine developers are able to achieve. And by how many developers or projects are able to achieve it.

How do engine developers maintain a high-level of software quality while keeping abreast of the continuous changes to the devices and their capabilities?

Building high-quality software is hard enough without building upon technologies like mobile that are moving so quickly.

In attempting to build a mobile game engine, I suspect that I would spend all my time reading the device APIs and other peoples code to see how to handle all the odd edge cases cases. This approach would be at the determent of the quality of the code in the game I was trying to build because it would become a hodgepodge of code snippets and device API wrappers. Mobile game engine developers need a vision that ties the project together.

How do engine developers design their APIs for use by a large and diverse base of developers?

This is a great question, because APIs are hard, fully stop. I am forever refactoring APIs I write for myself, let alone APIs I write for other developers. I can imagine huge resistance if a game API changed frequently and dramatically. How do developers manage consistency of interface with innovation. How does a consistent engine API emerge from the contributions of multiple different developers?

How do engine developers structure their projects and manage their teams of developers and contributors?

What is like to have hundreds, thousands or tens of thousands of deployed games built on technology that they envisioned and put into the world?

These questions and more like them motivated me to write my next book: Interviews with Mobile Game Engine Developers. The book will feature interviews with the developers behind the popular and modern game engines for iOS, Android and HTML5. Each developer is probed with a barrage of questions to get to the heart of just how they are able to do what they do - how they are able to create platforms that allow so many tens or hundreds of thousands of developers be creative making mobile games.

If this sounds like the type of book you'd like to read, leave your email address and I'll ping you when the book is finished. Like my last book, it will be released as a paperback and eBook in all the standard formats.

If you are a mobile game engine developer (or you know one), reach out, I'd love to talk about interviewing you for the book.

Some sample questions I will be asking game engine developers include:

Why did you start working on this game engine?
What are some challenges in supporting multiple mobile platforms?
What is your approach for staying abreast of changes to the underlying technology and device APIs?
What are some game ideas you have had but do not have the time to work on?
How much revenue have you generated from your game engine?
What are some trends you are seeing in mobile gaming?

Got a good idea for a question I should ask game engine developers? Reach out and let me know.

Family Histories: Interview relatives and preserve their memories in a family book

2012-07-27T09:20:00.002+10:00

Photo by robotpolisher

I've been chewing on a idea for a few weeks. It's an idea to put together a collection of personal interviews with all the people in my extended family.

Obviously the target audience are the people being interviewed, this is a non-profit venture.

The motivation for this is that although I "know" my family, each relationship is specific and limited. I have a lot of mundane questions for my relatives that I would like to know the answer to to have more complete picture of them as whole people. I figure others in my extended family may have the same desire or may be convinced that that such an enlightened perspective would be interesting and valuable.

I also figure different generations have altered perspectives. They know that nephew Jonny plays with computers, but they have no idea what he does, what his dreams are and how he met his wife. I may know that a particular grandfather was a fitter and turner, but I have no idea whether he liked the job or which companies he worked for.

I think it's an easy sell for participation and a feasible value proposition for the end product.

I've floated the idea with a few family members and their has been some genuine interest with suggested questions to ask. Specifically in the phrasing of questions for family members that have two or three families through remarriages. I've been told that this is a normal reaction to having a child, that others have expressed interest in interviewing older family members with a tape recorder, or have explored more formal genealogy.

This last point got me. It's the rationale for the blog post, because if their is a trend here, I'd really like to know. This might be a broader opportunity.

Going forward, I'm going to build up the question set and have it reviewed by a few of the matriarchs in the family. From there, I will set up a google form for those that are tech-savvy and send out a printed set with a return self addressed envelope and cover letter for those that are not.

This should be an interesting experiment.

From Developer to Entrepreneur: Must Read Books

2012-07-23T06:00:00.000+10:00

Photo by Keithius

I'm not an entrepreneur (yet). Other people have to name you such.

Anyway, I've been reading a lot and I've read a number of books on and around entrepreneurship that I think are must read.

I suspect there's nothing new in this list, I read them because other people recommended them, but I want to pay that forward.

This list is for the upstart programmer who is thinking of entrepreneurship and is looking for the kick to get started and the tools (processes) to actually put some ideas into the world and measure the effects.

I'm assuming your a developer, a programmer and that you want to make software-based services or products. There's no books in here about learning to program.

You may also want to brainwash yourself with some podcasts. I use Mixergy, Startups for the Rest of us, Lifestyle Business Podcast, and the Stanford Entrepreneurial Thought Leaders series.

Where to start?

You start with Start Small, Stay Small: A Developer's Guide to Launching a Startup by Rob Walling. It's robs no nonsense process for market first web-based software entrepreneurship. It will either kick you into immediate action or you're realise the game is not for you.

What's Next?

Theory. I like process and theory and understanding some whys after some initial hows.

Geoffrey A. Moore's Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers will turn on the marketing light. You'll get some notions on how to think about the the interplay of technology products and markets. Much of the book is at the wrong level for being an upstart, but it's an important foundation and I'm big on primary sources.

Steven Blank's The Four Steps to the Epiphany: Successful Strategies for Products that Win. Just read it. Become one with it. All the process questions I had while reading Start Small, Stay Small and Crossing the Chasm were addressed in this book. This is the the play book and I know everyone says to read it, how many actually do? Buy it now and read it cover-to-cover. Then reread it every few months. Seriously.

More Actionable Process

I love listening to talks by Eric Ries, he so honed to inflame action in me. His book The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses is good and you should read it. But I say so less strongly than I do with the two books above. Frankly if you become one with The Four Steps to the Epiphany and have read some agile software development books by Kent Beck then I think The Lean Startup is the cliff notes and the natural merging of the ideas. Nevertheless, it's a solid reinforcing read.

I think Ash Maurya's Running Lean: Iterate from Plan A to a Plan That Works and Chris Guillebeau's The $100 Startup: Reinvent the Way You Make a Living, Do What You Love, and Create a New Future are also great books to read at this point. But could be skipped if you feel like you have enough process and rah rah in your head.

Down to Grunt Work

After I filled my head with high-level processes, I felt I needed more grounding in the grunt work of throwing together reasonable webpages and writing lots and lots of sales copy. This phase of my reading was strongly influenced by Walling's post Why Startup Founders Should Stop Reading Business Books.

Maria Veloso's Web Copy That Sells: The Revolutionary Formula for Creating Killer Copy That Grabs Their Attention and Compels Them to Buy is a must read. Read it twice. After programming, the one thing I want to be killer at is writing solid web copy and this book lit that fire in me. It has check lists to analyse your own sales copy! I wish more books promoted how-to's for analysing your own work.

Steve Krug's Don't Make Me Think: A Common Sense Approach to Web Usability is the go-to volume for web design best practices. Pair with with Web Copy that Sells above and you can get a long wait to throwing together very meaningful minimum viable product's for your ideas. I also read Lance Loveday's Web Design for ROI: Turning Browsers into Buyers & Prospects into Leads which I also think is a good complement, it covers a lot of the same ground but also critiques websites to help with that whole "how evaluate your own work" problem.

Bryan Eisenberg's Always Be Testing: The Complete Guide to Google Website Optimizer gives a lot of great actionable ideas of things to test for different page types and Avinash Kaushik's Web Analytics: An Hour a Day has more information on web analytics than I ever want to know (actually, the book is a real slog to get through).

From here...

Books are great, but you have to play the home game. You must be building and launching things while reading a long. Your project will frame the advice you'll be reading and you will not forget it.

Any must read books for the developer to entrepreneur transition that I have missed? Hit the comments.

Teardown of 4 Technical eBook Websites

2012-07-21T06:00:00.000+10:00

I listen to the "startups for the rest of us podcast" and the on the episodes summarising this and least years MicroConf the hosts Rob and Mike described doing public website teardowns at the conference. It's an awesome opportunity for learning, and maybe a great idea for a lightweight consulting business.

Anyway, I'm enamoured with the idea and thought I would apply it to learn how I might improve my ebook product landing pages.

After reading a number of books on copy writing I've drilled a few golden rules into my head that I try to follow or think about. They may or may not be best practices (I think they are) and I may or may not follow them for a given landing page I build (I usually start with them). As follows:

Use a sales page structure like AIDA or the problem methodology.
Have your call to action after you have spent some building interest in the product in the reader.
Describe the product a few different ways appealing to different ways people consume information: emotionally, visually, analytically, etc.
Use phrases like "you find out" rather than "you will learn".
Describe in terms of benefits rather than features.
Focus on the reader and what they want or get rather than on yourself and what you have done.
Do not provide distractions (links) on the path to your call to action. The site should guide the reader to clicking the one thing you want them to click.
Try to read the webpage from the other side of the room, the heading and call to action should be obvious.

I decided to pick a few modestly popular ebooks. I have no idea that that popular, modest or otherwise other than I have seen them pop-up a few times on social news websites.

I processed each site for about 5 minutes listing observations of things I saw and didn't see. It's an analytical exercise, nothing personal to these great authors. I'm not a UI/UX guy, so you may have to bear with me.

#1 Working with Unix Processes

Price: $27 USD
URL: http://workingwithunixprocesses.com

The book is sold directly on the site using a Shopify store (the author works for Shopify).
The domain name is the book title.
The page heading is not the book title, but is instead a summary of the value proposition and audience.
The first content after the heading is a testimonial.
There are three links on the page before the call to action, and specifically a tweet widget right before the call to action.
There is a list of technical features (rather than benefits) next to a 3D version of the book.
There are many testimonials on the site.
The call to action appears twice on the page and really stands out as a bold orange button on a pale background. Both have the text "Buy It now!
The details of the product (formats and no PayPal login required) are duplicated at each call to action.
There are a single copy and team license options.
There is a link to the kindle store and pragmatic bookshelf versions after the second call to action.
There are small links to sample chapters (blog posts) after the second call to action.
Author information at the bottom with a contact email address.
Money back guarantee at the bottom of the page.
The page is long.

#2 Riak Handbook

Price: $29
URL: http://riakhandbook.com

The domain name is the book title.
The book is sold on the site using a paypal button (FastSpring form).
The heading is not the book title, it is a value proposition for the book and the technology the book is about.
A cover image is provided directly after the heading and shown in an ipad, suggesting ebook format.
The call to action is next to image and less bold than the cover.
A link to a free sample PDF is provided directly after the main call to action.
The call to action appears twice more on the page between sections on the page, and a third time as a group license option.
The page is quite long.
A screenshot of the free sample is provided with a link in between a number of testimonials. The screenshot of the page encourages you to click.
Testimonials are provided along side the details about the book.
The summary is in terms of features rather than benefits. Uses "you will learn" rather than "you will find out".
Promotes the technology using two paragraphs, appealing to the audience interested in that technology.
Provides a bullet list of features mixed with benefits "you will learn", the list is further down the page.
Heavy use of highlighting of phrases to draw attention.
Provides a PDF of the table of contents.
Provides the hard sell paragraph towards the bottom of the page right after a ebook call to action and before a group license call to action.
About the author details provided at the bottom of the page with contact information. No author photograph.

#3 Exception Ruby

Price: $15 USD
URL: http://exceptionalruby.com

The book title is the domain name and the page title.
The tag line is a second header. It's a great tag line that reads well and highlights the value proposition.
There is a 3D book image for eBook.
The book is sold directly on the site using getdpd
The call to action at the top under book image and really stands out.
There is list of features (rather than benefits) with highlighting very prominent on the page (main body)
There is a 50 second video from a talk that inspired the creation of the book (I am not normally a video consumer online).
There is a small tweet button below the list of features.
Author information at the bottom of the page with large photograph of the author. The photos is a little cheesy.
There are three short testimonials at the bottom of the page.
The page is short and to the point, not much scrolling.
I like the clean design, site works on a few levels (design, copy, value proposition), it makes me want to buy right now. Nice work.

#4 Start Small, Stay Small

Price: $19 for ebook, $25 for paperback
URL: http://www.startupbook.net

The book name not in the domain name.
You can purchase the ebook and paperback directly from the site using a PayPal button.
Title is an image including the title of the book, value proposition and 3D book image.
There are testimonials at the top of the page directly under the heading. There are more testimonials below the calls to action and at the bottom of the page. It is hard to notice the second and third blocks as they look like normal text.
The page is simple page, short with not much scrolling (which I like).
There is a block of calls to action for the three different versions: ebook, audiobook and paperback. It's overwhelming.
The "Free Sample" link is a button and is just as large as the other calls to action. It looks like a product.
Three paragraphs: audience, value proposition, about the author. Good content, but hard to differentiate the paragraphs. I don't know which one to read, so I read none.
There are subtle links in the about the author paragraph.
No social networking buttons, but some kind of spread the word link at the bottom of the page - very hard to notice. Email the author link also at the bottom of the page.
This is the only book of the three that I have actually read and I strongly recommend reading it to any software developers interesting in taking the next step.

Actionable Outcomes

Having a domain name (URL structure?) that matches the title of the book might be a good idea (I suspect for wining the SEO for the book title).
Use a page title the communicates the value proposition rather than the book title.
Technical ebooks can have a high list price. The four examples I covered were all $15 or above, with two that were close to $30 USD.
Sell the book on the site directly and let other markets (Amazon, Smashwords, etc.) worry about getting traffic to their landing pages.
Have a 3D version of the book cover, even if it is just an ebook. Use a dead-tree book metaphor or an eReader metaphor (cover on an iPad, etc.).
Sell both single license and team/group licenses for the ebook.
Take a screenshot of the first page and use that as the download link for a free sample.
Keep the page short ant to the point. (long form sales copy can work too, test)

Imagine being able to pay a reasonable sum to have proven experienced sales or marketing people do this to your sites. I wonder if a "I'll tear your site down if you tear mine down" reciprocal thing could be set-up? Any takers?

Of course, all of this post goes against the modern "test" mantra. One should be A/B testing (multivariate testing) changes to a page to see which converts the best. I agree in principle, but I also believe it pays to do your homework and have a good base to work from. To build some sales and design intuition.

Depending on how I feel (and the backlash), I may apply this process again to books with some commercial success. Looking a few quickly (the lean startup, rework, etc) I note they make quite different design decisions.

How to create a product description for a landing page

2012-07-20T06:00:00.000+10:00

Photo by Matt Jiggins

As a programmer, I struggle putting together a product description. Marketing is an area I'm desperately learning about, but I have a long way to go yet (know any must-read books?).

For each new product page I put together, I re-imagine my process with an eye for improving. The proof is always in the pudding: the conversion rate. You can iterate towards a better conversion rate but this product description process is about better understanding a new product and being able to communicate what it is all about.

See my Interviews with Mobile Game Engine Developers homepage for my most recent effort.

My method is about generating the content that provides the basis for a landing page, for blurbs and product descriptions around the web.

I thought I would summarise my current method and try to flush out some new ideas.

I start with a google doc and work through a series of product description methods to try and explore all the different ways of thinking about my new product. It is an ongoing iterative process that never ends. I block out the document at the start of the project and update it all the way up to and beyond the product launch.

The following is a list of headings in the google doc that I fill out and keep updated for a given product:

Product Name

I start with the name of the product. If it is a book, I generate a list of 10 or 20 titles and keep adding to it as my understanding of the product improves.

Rationale

A bullet point summary of the honest personal reasons reasons I want the product to exist in the world. Typically the reasons float around cash flow, ego, and self-education.

Technical Details

I capture technical details about the product in bullet point form. Again, if it is a book, I'll list facts like the number of pages, the ISBN, the number of words. I will invent numbers as targets or use numbers available from the work.

Tag Lines

Tag lines are critical. For me they become the titles of blog posts, the titles of landing pages, one line descriptions I throw around in tweets and forums, and even sub-titles of books. When I get in the mood I will generate tag lines in massive batches. I like to fill a page or two with them. They are critical because they provide a one line pitch for the product, so they have to be descriptive and punchy.

Audience

I think really hard about who might be interested in the product and for who the product is a must-have item. Again, I list each group or type of person on each line and under it I try to brainstorm the reasons why, starting each line with a "because". Really, each audience is a market or market segment and will tie into the benefits later and into the keyword research I typically do before deciding to take on a new project.

Features and Benefits

I think in terms of features. It's easier because they are the things I'm building or writing or making sure exist in my product. I make a bullet point list of all the features I can think up in the product. Then, for each feature, I list all of the benefits that the feature provides. This is hard, but I find it very useful (you can see that I currently put the resulting list directly on the landing page). If I find that I start repeating myself, I consolidate features. If I find I'm listing too many benefits (more than 3-5) I split the feature into two. Benefits sell. Benefits also tie right into items in the list of possible audiences. The big interesting benefits become titles or tag lines.

Human Description / Blurb

At this point I typically feel confident enough to bash out a few human readable paragraphs describing the product. For me this is typically the book blurb or project vision and may open with a tag line, list key audiences, and highlight key benefits. I have tried to structure this, but the structure never holds as I always iterate towards readability.

Geoffrey Moore Product Template

In Geoffrey Moore's book Crossing the Chasm he lists a simple fill-in-the-blanks elevator pitch value proposition template for describing a software product or service. I have used this for years to help understand my small open source projects. I find it is less useful for my book and website projects, but I still like it as it provokes thoughts of competitors. The template verbatim is:

For (target customers)
Who are dissatisfied with (the current market alternative)
Our product is a (new product category)
That provides (key problem-solving capability).
Unlike (the product alternative),
Our product (describe the key product features).

There is an elaborated version with examples here.

Problem Methodology

I recently read Web Copy That Sells to improve my copy writing. It is full of great advice, but most importantly had large actionable check lists of things to do and think about. One of these check lists had to do with the problem that product solves. To paraphrase the method, it asks 5 questions that are to be answered from the perspective of the prospect, your product, and the problem it solves, summarised as follows:

What's the problem?
Why hasn't the problem been solved?
What's possible now?
What's different now?
What should you do now?

AIDA

This is the classic AIDA method for copy writing. It has always felt abstract to me and as such I don't really like it very much and typically omit filling it out for a product. It is more a template for a sales page or sales letter than a product description, but it can help in framing the product for the prospect.

Attention
Interest
Desire
Action

Feedback

I capture all feedback I read. I capture them mainly from emails, blog posts and forum posts. I list them verbatim with the authors name, details, and source information (link if available). In addition to boosting my ego when I feel low, these provide the basis of testimonials. I also have this fear that someone will call BS on one of my testimonials, so having the raw feedback available (and source info) mitigates this fear.

Testimonials

Testimonials are great for landing pages and product description pages. I even put them on about pages for webpages. I take them from feedback I have received (asking for permission where needed). I'm careful to include the salient product affirmation, excluding unnecessary elements and fixing typos.

Solicited Benefits

Early in a project I will solicit a lot of feedback on the product. I will email and post on forums providing a description and link to the current landing page asking for "what interests you in the product" and "what problem would it solve for you". This works great, but you have to choose the market carefully. For example, with "Interviews with Mobile Game Developers" I pinged others developers using game engines features in the book, who were very responsive.

I use all of this information to craft a first cut product landing page. I typically put the landing page up early and iterate like crazy. I collect email addresses and use some A/B testing, using the email collection as the call to action and the thank-you page as the goal.

I'll print the landing page out and then and read through it looking for typos and inconsistencies. I don't do this enough and feel embarrassed when someone points out bad copy.

I think I am improving and I am forever guilty for not going back and applying my improved methodology to past products (I will get around to it).

Got any suggestions to improve this process? What process do you follow?

A programmers approach to self-publishing a book

2012-07-18T06:15:00.000+10:00

I launched my latest book last week after approximately 3 months in development. It's great to complete another project and put something physical in the world (as opposed to just software).

The book is called "Mobile Game Engines: Interviews with Mobile Game Developers" and features 38 interviews with developers of mobile games about how the choose and work with modern mobile game engines. See the home page for more information.

I prepared the book in LaTeX. I didn't think too hard about this decision, and in retrospect, I may as well have prepared it all in plain text. I used LaTeX for my previous book and I knew that the resulting PDF provided a high-quality basis for a paperback book. I maintained the source for the project in revision control. This included the LaTeX files, Makefile, as well as assets like the book cover and screenshots from the games.

I asked a designer friend to create the cover for me. He did a great job, and created a suite of covers that I can use for the series of books I plan to complete on Mobile Game Engines. Each has the same design, although the main color varies, allowing me to refer to each project based on the colour (this project was the Green Book, the next will be the Red Book of interviews with engine developers themselves).

I wanted a paperback edition of the book. It's not rational given the huge uptake of eBooks. I wanted it for myself. I wanted to be able to slide book into my bookcase and feel that sense of satisfaction (totally worth it). I used Lulu for the paperback, the same as my previous book. There was no cost. They allocated an ISBN for my book, and I provided the PDF and the book cover. Unlike last time I used their cover designer and placed the A4 Version of my book cover on the front, and added the blurb to the back and title to the spine. The result came out just fine.

Lulu offer 30% of the list price on their site. They also coordinate the distribution through partners like Amazon and Barnes and Nobel where they take a larger percentage. At the time of writing I priced the paperback at $17.50 USD. I theoretically get $5.31 USD per book when sold on Lulu and $0.71 USD from other retailers. This is reduced by a further 30% because I do not have the required US tax documentation.

See Interviews with Mobile Game Developers paperback on Lulu

I used Smashwords for the eBook edition and it was my first time using the service. I am very happy thus far. At the time of writing Smashwords only accept MS Word .doc files and rigorous documentation on how to format your book document ready for processing. Their system creates many ebook formats (epub, sony reader LRF, Kindle mobi, Pall PDB, PDF, RTF and plain text) and distribute to many stores (Apple in 32 countries, Barnes & Noble, Sony, Kobo, Baker & Taylor, Diesel, and more to come). I was allocated a free ISBN, submitted my file and it was immediately available on their store. I had to wait a few weeks to be accepted for distribution - the formatting rules are very strict and minor infractions block the winder distribution of your book. The modify-resubmit process is very straightforward.

I priced my book at $2.99 USD which leaves me with $2.21 USD (74%) per book purchased from Smashwords. Most stores offer 70% royalties, and smashwords take another 10% from that.

The conversion of my book content from LaTeX to MS Word .doc took some work. I already had some Ruby scripts to generate HTML from LaTeX for my last book. I used those as a base and generated a single HTML file for the book. I did not want to deal with images, so I excluded them from the eBook version. I then opened the file with my version of MS Word on my Mac (2008) and saved it as a .doc file. I followed the Smashwords Style Guide to the letter. I read it a few times, printed it and highlighted all of the rules that I had to worry about. I then wrote a repeatable procedure in a .txt file for each sequence of commands I needed to perform in Word to get the file ready for submission to Smashwords. This last step was critical because I had to create the doc file more than 10 times before I had it right.

Two things that got me were bullets and first line indent both in the mobi format. Mobi was nothing but pain, but totally worth it. Bullets rendered on a friends kindle on a new line, this was a problem as it was a roadblock from my book having wider distribution. I ended up removing all numbering and bullets and using "*" at the start of the line for a simulated bullet or list.

The book is a collection of interviews with a question/answer format (block paragraph). The mobi format kept performing a first line indent which really messed up the Q/A format. The solution was in the style gude, but some how I glossed over it. It required a first line indent set to 0.01cm which is basically imperceptible to the eye and looks just like a block paragraph format.

See Interviews with Mobile Game Developers eBook on Smashwords

Smashwords does not support the Amazon Kindle Store (KDP) at the time of writing. As such I had to sign up to Kindle Direct Publishing to get distribution on Amazon for the Kindle version. I used the doc file I prepared for smashwords and uploaded it to KDP and it worked first time, perfectly.

The Kindle store was really easy to use and my book was online in about 24 hours. I priced it the same as on Smashwords and KDP offers a 70% USD royalty. I did not get an ISBN for the Kindle version. All credit to the Smashwords Style Guide, I am quite sore it was the reason that the KDP process was so smooth.

See Interviews with Mobile Game Developers for Kindle on Amazon

I got there in the end and I am generally happy with the result. The book reads fine on my iPhone, on my friends Kindle and the paperback has a suitable heft and looks slick. I will use this same process for the next book, allowing me to focus more on the content and marketing.

Collecting and editing the interviews for the book was an interesting process and I learned a lot. I expect to post something about this in the future.

If you have any questions about the process, please reach out, I'm more than happy to provide further details on what I did.

Tapping my network for the Must-Read books on Developing Software

2012-07-15T15:46:00.000+10:00

Image by chillihead.

I've been reading a lot of books recently, 1-3 a week for about 6 months, mostly entrepreneurial and business books.

I started having hard questions around user experience, web page design, online marketing strategy, etc. so I pinged a few friends and asked them for book recommendations.

The results were mixed (the basis of a future rant), but it gave me an important new perspective: The people I am connected to know many things and I have much to learn from them, the difficulty is thinking up creative ways to mine that knowledge.

Anyway, I generally earn my living designing and building software systems, and I'm always interested in improving my craft. I decided to ping many of the engineers in my network to find out the must read books for software engineers. Note, there are lots of links to books in this post, all of them to Amazon with my referrer code.

I am at that stage in my career where I'm learning toward project theory, management theory, etc. and less interested in hard core technical subjects. I still enjoy learning new programming languages, but I prefer to read books on how to manage time and resources.

Before I sent my email out, I made my own list of must read books, books that I had read and that had influenced the way I approach my work. They included:

I then googled around for similar lists - I know this question has come up many times on reddit and hacker news. There were a lot of lists and many books (I got bored quickly - how could I trust these links?), but the few I added to my to-read list based on their synopsis and reviews were:

Finally, I collated the list sent through from friends and colleagues of yesteryear. I excluded those I had already listed (above). I also excluded books that were too technical that I didn't want to read (MySQL performance tuning, etc) and those recommended by people that clearly hadn't read them (like "just read Knuth").

It was a very interesting exercise that really helped my update and re-prioritize my software book to-read list (I have a massive google doc with lists-of-lists of books to read).

As a bonus round, I pinged other friends of specific topics close to their specialty, the results are below:

Software Testing (consolidated, I received a list of many 10's of books)

Web Design / Design

Python

Functional Programming (in order they were suggested to be read)

Erlang

Programming Erlang: Software for a Concurrent World

I have done this for other fields, such as investing and entrepreneurship and I may post my results for those too if there is interest.

Thanks again for everyone who contributed and put up with my harassing questioning emails.

Have I missed an important book? Reach out and let me know.

Interviews with Mobile Game Developers

2012-06-10T14:33:00.000+10:00

I am proud to announce my upcoming book: "Mobile Game Engines: Interviews with Mobile Game Developers", Jason Brownlee, 2012. It is a book of 38 interviews developers, each tracing the history of a mobile game from the selection of an off-the-shelf game engine to getting the game in the app store.

I'm going through final copy editing at the moment, but you can read more about the book and its contents on the home page:
Mobile Game Engines: Interviews with Mobile Game Developers

Mobile Game Engines

2012-03-30T08:59:00.001+11:00

I'm launching http://MobileGameEngines.com today. It's a simple site that lets anyone compare between all known iPhone, Android and HTML5 game engines.

I had the problem of trying to decide which game engine to choose for an iPhone game project and wrote about my efforts in a blog post in 2009: Review of 3D Engines for the iPhone I noted that many others had been through the same pain and had written a similar blog post and figured someone would come along and address this problem.

It has been a few years and the number of posts on quora and stackoverflow asking "what is the best iphone game engine" and "what is the best android game engine" continue to be posted. I decided to make a site to really nail the answer to this question, the result was MobileGameEngines.com

I've been in contact with the majority of the game engine developers listed on the site to ensure all the information is correct and useful. We are sitting on 65 game engines listed and still growing as more are uncovered.

Take a look and let me know what you think.
Pass the link on to anyone you think might be experiencing this pain.

DailyAIFeed: Your daily dose of Artificial Intelligence news

2012-03-06T17:50:00.000+11:00

I've been running a little email list for about a year called DailyAIFeed. I started it because I got sick of checking my RSS and email each morning, preferring to just check email.

I've recently cleaned it up a little and opened it up to general consumption at www.DailyAIFeed.com

The list provides a daily email of the top interesting news in the fields of Artificial Intelligence, Machine Learning, Natural Language Processing, Computational Intelligence, Infographics and Data Science.

You can see examples of the types of messages on the blog at blog.DailyAIFeed.com

Go and check it out, and if you have any suggestions on how to further improve it, please get in touch.

Hosted Data Analysis Project Workflow

2012-02-16T09:02:00.001+11:00

I have been thinking a lot about the previously mentioned "Data Analysis Project Management: SaaS".

Such a system could be focused on the project management side, say a Basecamp for Data Analysis projects. But I think it could something different, something more. I think there are pain points in the data analysis workflow that could be 1) systematized and 2) automated. I also suspect that large data platforms from IBM and Oracle may offer solutions, but I question whether the any such solutions exist for smaller scale/cost projects. I think there is an opportunity for a Hosted Data Analysis Project Workflow system.

I've been trying to think hard about possible pain points, and here's what I got:

Revision control - keeping track of changes to project files.
Research Journal - keeping track of what questions have been asked and what findings have been made.
Reproducibility - ensuring there is a recipe to recreate a past result.
Collaboration - working with others on a particular on all aspects of the analysis workflow.
Interpreting Data - Looking at tables and graphs and generating hypotheses to go and test.
Executing Models - configuring, running, tuning models.
Verifying Models - assessing models on test and verification datasets.
Blending Models - comparing and combining model outputs.

I have also been thinking about solutions in the market that address specific pain points. This is much harder - the Internet is a big place. Here is what I could put together quickly:

Many Eyes - Collaborate on interpretation from visualization.
Google Predict - Hosted models and data
Github (and similar) - Hosted project files, forking, collaboration
Kaggle, TunedIT - Data competitions, competitive community around problems, Data Spec Work / R&D Outsourcing.
Cross Validated - Q/A community around statistics and machine learning
Google data explorer - Visualization and interpretation public datasets

Do we need a system that combines all or some of these together? Why can't this problem be solved with existing systems, why isn't it such?

Data Analysis Project Management: SaaS

2012-02-15T07:00:00.053+11:00

The idea for developing a Data Analysis Workflow SaaS as been percolating in my brain for a few days.

I read a post on the Kaggle HHP forum entitled "Project Management software for Data Analysis" by Dan Becker and it got me thinking. With almost every data or machine learning competition I enter, I end up solving the wrong problem. Rather than solving the problem presented by the competition, I solve the problem of automating and optimizing my work flow by writing scripts and software.

It makes sense, it's what I do for a living. I write and maintain software. Why not expose the problem for what it is and solve it. The real question: Is there a market for such a solution? and secondarily, What features would the software have?

My ad hoc competition-wise solutions focus on result reproducibility (good science), summary statistics, visualization, research journal, generating webpages of experiments performed and above all, automation. I figure that the more I can automate, the faster I can test ideas and the higher my velocity will be on the problem of the competition. Except, most of my effort goes into the automation rather than ideas for the subject data.

Whether I build the software or not, it is a good research topic to consider.

The aforementioned forum post lists some interesting resources, specifically:

dryWorkflow project Don't Repeat Yourself Workflow for efficient data analysis. Focused on R.
ProjectTemplate: Automates the creation of new statistical analysis projects. Project Webpage, Github project, CRAN project.
CRAN Task View: Reproducible Research with some interesting ideas.
Book: The Workflow of Data Analysis Using Stata, J. Scott Long, 2009

Some google'ing resulted in some additional useful links: How to efficiently manage a statistical analysis project? on CrossValidated which provides some excellent answers and great links to go off and read. Other CV links that were pretty useful included Best Way to Aggregate and Analyze Data and What is a practically good data analysis process?

I also came across Organizing Your Approach to a Data Analysis by Scott Emerson which provides some excellent motivating questions.

Know any other good resources?

I'm keen for feedback, should such a software (web-based) platform exist in the world?

Are their data analysis people that would use this, pay for this, beyond competition patrons?

Preview of Clever Algorithms: Statistical Machine Learning Recipes

2012-02-14T13:55:00.000+11:00

I am feverishness working to complete a first draft of my next book in the Clever Algorithm series:

Clever Algorithms: Statistical Machine Learning Recipes

You can take a sneak peek at some early chapters online. I am developing the project in plain view on github, take a look at my source R files or LaTeX files if you're into that sort of thing.

The book will provide a treatment of the field of Machine Learning much like the first book provided a treatment of the field of Computational Intelligence and Biologically Inspired Computation.

Here is a preliminary blurb:

Implementing an Machine Learning algorithms is difficult. Algorithm descriptions may be incomplete, inconsistent, and distributed across a number of papers, chapters and even websites. This can result in varied interpretations of algorithms, undue attrition of algorithms, and ultimately bad science.
This book is an effort to address these issues by providing a handbook of algorithmic recipes drawn from the field of Machine Learning, described in a complete, consistent, and centralized manner. These standardized descriptions were carefully designed to be accessible, usable, and understandable.
An encyclopedic algorithm reference, this book is intended for research scientists, engineers, students, and interested amateurs. Each algorithm description provides a working code example in R.

I have started to distribute chapters to copy editors and technical editors. I'm also trying hard to nail down the table of contents and specifically which algorithms will appear in the text.

If you know anyone who might be interested in technical editing or any editing, get into contact: jasonb@CleverAlgorithms.com

House Price Regression: Vermont South, Melbourne

2011-05-17T19:03:00.009+10:00

While looking for a house I maintained statistics on the main suburbs we were visiting, and more specifically, on the houses we looked at.

Each house we visited was about 4 bed rooms and generally had the same kinds of attributes - attributes we thought we wanted in a house. For each house we visited, I recorded the address, size of land in square meters, date of sale, sale type (auction, private), asking price, sale price, and other assorted details. An additional contrived measure was the driving distance to main shops reported by Google Maps, in kilometers.

I also supplemented the dataset with additional matching houses in the area when data was available. I found prices were sometimes available from the auction results, although in other cases I had to call to find out, and.or scourer the web.

Rather than let this information go to waste, I thought I would share some of the collected data. This post provides data I collected for the Melbourne suburb of Vermont South.

The following graph simply shows sale price by date, quite boring.

The following graph shows the sale price by land size in square meters.

The following graph shows the sale price by distance to a specific set of shops in kilometers.

I found the data generally useful for plugging in new places and using simple linear regression to help answer questions about expected price at auction or private sale.

Some of this data may be available for purchase from various retail data providers, but I found collecting and entering the data myself made it a lot more personal and gave me some additional focus when inspecting properties and talking to agents about trends.

So, we bought a house

2011-05-16T18:00:00.030+10:00

So we finally bought a house. We've been looking on and off for about 12 months although things got serious about 3 months ago.

We first looked at the place we bought last Saturday, and walking in the door I knew it was a strong contender. We looked at three other places that day, and they all paled in comparison.

The place had been passed in at auction nearly three months before, and we were told that initially the vendors expectations were too high. We saw this as a good opportunity to negotiate and try to broker a good deal. The market had been slumped for a few months and the early figures for the quarter had indicated a ~2.5% drop in median house price for the city.

We did another inspection on the following Tuesday and enlisted all of the troops (extended family) to give the place a good once over. We then sat down and signed a formal offer. It was rejected. We upped the offer $5K and it was accepted, although at the insistence of my wife we made the offer contingent on the outcome of a builders inspection.

A found a company in the yellow pages and had the inspection done on the last day of the 3 day cooling off period. The building and pest report was incredibly detailed, providing photos and a room-by-room summary, inside and out. We learned a lot about the types of preventative maintenance the place will need over the next 5-10 years, and more importantly, we learned that the upstairs balcony had some major structural problems.

The report said that the wood used was popular in the decade that the balcony was built and was known to rot unless property treated. Rather than expecting the vendor to return the balcony to new condition, we made an offer to split the difference, deducting half of the cost of the repair from our offer.

All of the negotiating occurred on the last day of cooling off period, a Friday. I had my wife on one had, adamant that she didn't want to pay a thing to have the balcony fixed, and the agent on the other hand threatening to open the property for inspection on the next day. I really liked the place and I was feeling totally strung out (to say the least).

We managed to broker a deal in the end and initial the final amendment on the Saturday, one week from our first inspection. With previous auctions and negotiations, I tried to remain emotionless, time was on our side and we could wait for a deal. I really liked this place and it was beginning to dawn on me that our remaining time to find a place (before the baby came) had shrunk to a matter of a few months. We're both happy we finally got there and have high hopes for turning the property into our home.

We learned a lot throughout the process. My analysis of median house prices, suburb selection, crime rates, and even travel time studies months ago were interesting, although in the end did not directly affect the outcome. Even the detailed suburb house price regressions I was building up were not used, as we ended up buying in a completing different suburb, inspecting the house on a whim.

I was told early on that buying a home is different from buying an investment, and it bit me in the end, because its emotional. If/when there is a next time, at least our expectations - that it is a long hard emotional roller coaster - will mean we'll be better prepared. Hopefully.

Rather than letting them go to waste, I'll post some regression analysis for a selected suburb soon.

Quake AI Programming Book

2011-05-03T07:00:00.028+10:00

I intend to write a follow-up book to the Nature-Inspired Clever Algorithms book on Machine Learning. I have a lot going on this year, so I was thinking of postponing it until 2012. If I do decided to go down this road, I was thinking of taking on a different project in 2011 that would be smaller in scope, less taxing, although still interesting and rewarding.

I have been thinking of writing a book about the AI in the Quake series of computer games. I was thinking of either writing a book that analysed the Artificial Intelligence architecture in each game in the series, or analyze the AI in the bot modifications. Perhaps both. The book would walk through monster or bot case studies and describe how they fit together, think, and behave. Perhaps with small experiments and demonstrations along the way. The kind of book that would have captured me as a game programming hacker 15 years ago.

In pondering this idea, I thought it prudent to explore other books written on or related to this idea. The following is a list of books that I found:

Quake Series Programming Books

Focus On Mod Programming in Quake III Arena, 2002
Quake 4 Mods For Dummies, 2006

Related Programming Books

These are by no means the cream of the crop of game AI programming, and there are in fact many level design books in there as well.

All of these books are focused on teaching some form of programming or game development using an existing game as a medium. The advantage of the Quake series is that the source code is released under the GPL. The Unreal series and the Half-Life (Source Engine) series are not released as open source, although do provide access to some aspects of the source under restricted licence for the modding community.

It is clear that there is interest/demand for books on game development based on the Unreal series, which makes a lot of sense given their general success in licencing the technology.

Some concerns about tackling such a project include:

Interest: The games in the Quake series are old (10-15 years). The methods may be outdated, they may not be relevant to modern computer games, and it is more than likely that no one will care
Low Barrier: It is more than likely that no one has undertaken such a project because the barrier is so low. One can simply read the code and understand what is happening, no analysis is necessary.
Copyright: Although the source code is released under the GPL, the game assets are not. One may have to acquire a licensed copy of the game to do any meaningful development. Additionally, my use of game screenshots may be restricted (fair use!?).

There is some effort required to produce such a work. Getting each project setup may be involved, especially across the three main platforms (Windows, Mac, Linux). The work would be primarily analysis: reading source code, experimenting and communicating what is happening with diagrams and descriptions. This tinker-write cycle is slightly more relaxed than the deep research needed for each algorithm in a machine learning text.

Is there interest in the market? Would you read or skim such a book?
Let me know what you think in a comment or email.

The Little Taxonomist

2011-05-02T07:00:00.000+10:00

We are expecting our first child in a bunch of months and I have been thinking about all kinds of science experiments to perform on/with the bub. I had an idea last week for what I think is a cool little web app that allows a a parent and their child to catalogue the native species around their home and learn more about their local environment. I am referring to this idea as "Little Taxonomist".

I'll present the idea in the context of some stories:

Story #1
A father and son are curious about the plants and animals in and around their house and neighbourhood. In an effort to learn more, they decide to begin to catalogue the things they see in their backyard. They select a subject (say a flower, tree, or insect), photograph it, and note down a few descriptive phrases. They enter this information into a web application. The web application accepts the image and structured description and makes informed guesses (based on location and time of year and subjects collected by others) as to the exact species of the subject. The web application also suggests interesting subjects that are known or expected to exist in their area (probabilistically based on entries of other entries in the area), and some information about where they might be found, creating a context sensitive scavenger hunt. Slowly, a subject every few days, more on weekends, over weeks, they build up a catalogue of plants and animals in and around their home. Together they have learned more about the specifics of the suburban flora and fauna.

Story #2
A primary school class have a week or month long assignment that is a scavenger hunt of genera and/or species in the school ground. The class is split into pairs or small groups and allocated a flip camera (or equivalent single button point-and-shoot). They have a list of hints or requirements as well as blank pages, structured to capture basic taxonomic descriptions. Students return to the classroom and use the computer to copy the photos from the camera, drag them into the web application and add their descriptions. Students are awarded badges and points for the breadth and depth of species described, teams are ranked, and some indication of what other groups are finding is provided.

Vision
A web application for children to be completed in groups or with a guardian. The objective is to describe subjects in the local area (home or neighbourhood) and in so doing learn more about the local flora and fauna. The system encourages the collection of species by intelligently guessing based on brief descriptions and photos as to the actual known species. A gamaification layer is provided that includes badges, points, leaderboards, and similar extrinsic motivators. Additionally, the system uses the localized information in aggregate to suggest subjects to look for (to "collect"), and a probabilistic expectation that they can be spotted (you have a 90% chance of seeing a fruit bat between 5pm and 8pm by looking up). This probabilistic understating of what can be seen in the local area would be coupled with the gamification system, highlighting rare finds. The system would provide all data in aggregate (anonymize) allowing kids to explore what others are finding in their neighbourhood and the types of descriptions being used.

The following are some mock screens I hacked together in Google Docs:

Mock: Add Subject

Mock: List of Subjects

Forget kids, I want to use this. There might be a general case for adults with smart phones.

I am not sure whether I will build it yet, I figure it needs a 5-10 year old child to make it fun. I figure it could make money by selling some cheap cameras with a website subscription or maybe targeted advertising (kids+science).

A friend pointed me to Project Noah, which is a similar idea, but not the same.

I'm eager to hear what people think, seen anything like this? Would you use it yourself or with a child?

May Challenge: Touch-type Faster!

2011-05-01T16:30:00.000+10:00

We learned touch-typing in high-school. I think we did it for two or three years. Nevertheless, I suck at touch-typing, and as a programmer it may be considered embarrassing.

I can type without looking at the keyboard, so technically, I touch-type all day long. What I don't do is type as I was taught was "correct". I think that as a consequence I am starting to get some wrist pain.

Anyway, in the constant battlefield of self-improvement, I thought I would take the month of May as an opportunity to improve my touch typing and hopeful start typing "correctly" from the end of May onward in day-to-day computer interaction.

The May Challenge is to perform one lesson in touch-typing each day and to measure a standard typing word count each day. Hopefully this word count and/or accuracy will improve by the end of the month and my confidence in correctly touch typing will also improve.

At this stage, I believe I will use the free online service www.typingweb.com because it provides lessons and ad word speed tests, and of course it's free. I just took a test using the "correct" home-row based method and scored: 18 WPM average, 23 WPM gross, and 96% accuracy. It would be interesting to see what my score would be with my unorthodox method - a bad idea, I suspect it may reinforce the bad method.

Again, as with April, I will adopt a penalty-based approach to the challenge. For each day that I miss, I will have to donate $20 to an open source project of choice (not a .NET project as with last month). This is less relaxed (I support opensource, generally), because I am concerned that commitment required may mean that I miss a day or two here and here. We'll see. $20 a day is still a decent disincentive.

I'll be sure to summarize progress at the end of the month.

Image copyright Wouter Verhelst.

April Challenge Over: Watch One Tech Video Every Day

2011-04-30T07:25:00.001+10:00

I set my self a challenge for April to watch one technical video every day throughout the month. My rationale was that if I could apply the same discipline that I use to discriminate what I eat to what media I consume, that it would have a beneficial effect. Sure, fluffy, but the challenge was measurable and the penalty for missing one day was a donation of $20 to a .NET opensource project.

The month has come to and end and I did manage to watch one technical video each day, so no donations needed.

I intended to spend most of the month watching university lectures and Google Tech Talks, which I mostly did. It became harder and harder towards the end of the month to find an hour+ to watch a tech talk. I ended up catching a quick (15 minute) TED talk instead. TED talks are good (some can be great), but they are so brief that my retention is poor, and so high-level that I finish thinking that I have not learned very much at all.

The following image provides a breakdown of the sources for videos I watched. The "other" category includes random tech videos on youtube that do not fit into one of the other broader and popular categories.

I enjoyed almost all videos. I ranked each with a real-valued scoring between 0 and 5 and provided my own description in a spreadsheet. Five real highlight videos (in no particular order) were as follows:

Properties of Exoplanets: Planet detection, Kepler, Nature of planets found so far (bad sound, but whatever!)
Brains, Meaning and Corpus Statistics: Text statistics in, map of neural activity out!
Getting in Shape for the Sport of Data Science: Tools for data mining competitions, intro. to random forests
Sugar: The Bitter Truth: Sugar is bad for you, it is a poison
"All Questions Answered" by Donald Knuth: Q&A

I found that I read a lot less RSS feeds in my Google Reader. I also found that I consumed a lot less 'acquired' media in the form of US TV shows (selling the family media player helped here, no doubt).

I had a good time with this challenge and will attempt to stick to it and record my progress in a spreadsheet.

AIFeeds Part 4: Automating an AIFeed aggregator

2011-04-29T07:00:00.113+10:00

In Part1, we prepared a large list of RSS feeds and filtered them down to something workable. In Part2, we processed all of the articles in the feeds and presented posts from the last five days as a static RSS reader. In Part3 we used a number of JSON APIs for social networking website to gauge the popularity of articles and highlight those popular posts at the top of the page on our static reader.

In this 4th and final post in the series we will explore different ways to disseminate the results of our filtering.

Step1: Send via Email
Last time in Part3 we ended up with a result that was quite passable. The scripts generated an HTML page that promoted popular AI, Data Mining, Machine Learning, etc. articles from the last 7 days, followed by a listing of all those other articles that, though were deemed to be less popular, may be of interest - organized by day. A simple approach to disseminate the results of this script is to send an email. The objective here is to receive the equivalent of an AI-themed version of the most excellent Hacker Newsletter.

I have a Google Gmail account and I assume most programmers do. The first step is to prepare a script that can generate an HTML email message and use the Gmail SMTP server to send the email. We are not focused on mass distribution here, just emailing the results to ourselves, at the moment, on demand.

The built-in SMTP handling in the Ruby standard library more than meets our needs here. The Gmail SMTP details are also easily obtained. The result is a script with two simple functions: the first for building a standard SMTP message with support for text/html content (mimetype), and the second for connecting to the Gmail SMTP server and posting the email. The script provides a spot test that will ask for your Gmail credentials and use them to send you a hello world email. Easy as pie.

See sendemail.rb

See below for a screenshot of my Gmail inbox with the resulting test email.

Step 2: Cron Send Email
The next step is to prepare a script that can be executed each day, generate a summary of the AIFeed output and email it to you. The easiest way to do this on any Linux or Mac machine is with cron.

A variation of the listpopulardayarticles.rb script from Part3 is used as the basis for the email. A new function is defined that generates the html content of the email and sends it. The script accepts two parameters on the command line: a gmail email address and a gmail password. These credentials are then used to send the email using the script prepared above.

See dailyfeed.rb

To execute the script we can create a shell script that contains the call to the script and the Gmail credentials used to send the email. For example, the shell script may be called run_dailyfeed.sh and look as follows:

#!/bin/sh
cd /path/AIFeeds/part4/
ruby dailyfeed.rb [email] [password] >> /path/AIFeeds/part4/dailyfeed.log

The script is three lines: a shebang, change directory to the script location, and a call to the ruby script with parameters. Replace /path/ with the path to your AIFeed directory, and replace the [email] and [password] with your gmail login details. The output of the script (and any errors) are output to a new log file dailyfeed.log.

The crontab for the current user can be opened as follows:

crontab -e

Add an entry that looks something like the following:

00 5 * * * /path/AIFeeds/part4/run_dailyfeed.sh

This is all one line with tabs in between the fields. Again replace /path/ with the path to your AIFeed directory. Cron will execute the shell script once each day at 5am local time.

The following is an example of a resulting email sent to my email account.

Improvements and Extensions
This section summarizes possible improvements and extensions to this part in the series.

Cron is an easy way to schedule a task on your machine. A better approach would be to set this up on a server (such as Heroku, AppEngine, or AWS) and send an email to an email list (via something like MailChimp).
An interesting extension to this project would be to turn the output into a webpage that is re-generated every hour or so. This might provide a useful diversion to reddit and hacker news, with a targeted corpus of links to scan over.

In this fourth and final part in the series we have hacked together simple script to send email via Google Gmail and scheduled the script to email ourselves a list of popular AI articles each morning at 5am. Not bad for a few ~~days~~ hours work. Sure, there are some rough edges, but the result is entirely functional and I think useful.

If you would like to see this as a service or perhaps a website, drop me a comment or an email. I'd be happy to clean it up further and automate it for a broader crowd if I knew that others as passionate as me about AI and Machine Learning were interested!

Don't forget all code and data for this series is available on the AIFeeds github project.

AIFeeds Part 3: Intelligently filter AI posts by social impact

2011-04-28T07:00:00.122+10:00

In Part1 we learned how to prepare a master list of Artificial Intelligence and Machine Learning RSS feeds. In Part2 we learned how to process articles from the list of feeds and present them as either a fire-hose or in a more compact day-based manner. In this part we explore how we may filter articles by their perceived social impact.

As with Part2, the curated_feeds.txt file is used as the basis for locating articles to consider listing.

Step 1: Computing Social Impact
Computing social impact for a give URL involves determining how many times it has been shared, up-voted, stumbled, etc on popular social networking websites. All of the large sites offer APIs and almost all offer JSON APIs.

I started out thinking I would need a ruby gem for each social networking site, one for Twitter, one for Facebook, etc. I started looking around and found that there were some gems here and there and most were quite outdated. I figured that because most APIs were in JSON format, that I could just interact with each in turn manually. I stared reading API docs for various sites and quickly got bogged down. For example, searching for a URL on twitter is not enough because people use URL shortening services. In fact, these services are quite pervasive.

The requirements here are quite specific: For a given article URL, how popular is it in a given social networking website? After a little more searching around, I came across a very useful website on Shared Count that provides a listing of the JSON API calls needed to answer this question across a host of social networks. I combined this with some of the API code I had already written and built a little URL scoring sub-system.

Given that all of the APIs we are dealing with are JSON based, we need some robust functions for downloading and parsing JSON files. I installed the json gem and wrote a robust function for making arbitrary JSON API calls (HTTP GET requests) that used similar socket timeout handling as to that which was used in Part1 for RSS downloading.

See parsejson.rb

Next, the specific query formatting and data processing needed for each social networking website was prepared and tested based on the JSON downloading and parsing code. The scoring of a URL against a a given API was relegated one per file allowing any special handing (like MD5 digests in the case of delicious) and spot checks.

API handling was written for 8 services: (each links to the specific ruby code) Delicious, Digg, Facebook, Google Web Search, Google Buzz, Reddit, Stumbleupon, and Twitter.

Step 2: Article Scoring
The next step is to combine the scoring for a given URL. There are many ways to do this and the more thought and experiment put into the scoring, the more meaningful the scores will become.

The chosen method here is a simple sum of the scores. This first approximation will provide a number that can be compared, but the relation between the scores in the sum from different websites is unknown. For example, a large score on Facebook or Twitter likely has more meaning than a large score on the Google web search. This is an obvious area for further research.

A URL scoring function was prepared that evaluated a given address against each website in a different thread. Digg was dropped from the cannon of scoring functions because the results did not appear to be meaningful and the API was always slow to respond.

See scoreurl.rb

Step 3: List Popular Articles
Now that we can score articles by their impact on the world, we can combine the scoring information with the list of articles posted in the last week to create a list of popular articles.

Starting with the viewarticles.rb and listarticles.rb scripts prepared in Part2, an updated list of popular articles can be created. In this case, the score for each article can be computed and all articles can be listed by their scoring. This promotes those articles that are expected to be interesting to the top of the page with the intent that they may be read first.

See listpopulararticles.rb

The following provides a screenshot from the last time that the script was executed showing an example of the resulting populararticles.html file.

Step 4: Combine List by Day and Popular Articles

The resulting listing of posts is long, it contains 7 days worth of articles. Additionally, taking a closer look at the posts in the list shows that most of them have a score of 0. It is expected that the older a post is, the larger its potential to have a meaningful score. This is because it gets indexed and shared more over that time. It is also expected that most articles posted within the last day or two will not have a score. In fact, it is expected that most AI articles in general will not have a score - it is a niche field of interest after all.

In this step we separate "popular" articles (those with a score) from the "unpopular" articles (those without a score). Popular articles are listed at the top of the page and the remaining articles that have not achieved a score yet are listed below the popular articles, broken down by day.

See listpopulardayarticles.rb

The following screenshot shows an example of the resulting populardayarticles.html generated by the script.

Improvements and Extensions
In this section a number of improvements and extensions to this part are summarized.

An important improvement to this part is the assessment of the URLs. Specifically, permalinks reported by some ATOM and feedburner feeds involve a proxy that redirects to the page. The effect is that the popularity of these proxy URLs are assessed against the social networking websites rather than the permalinks themselves, resulting in incorrect scorings. This may be addressed by following the redirect in the proxy URLs or using a different feed parsing library that has knowledge of the different feed types.
A natural first extension to this part is to add scoring from more social networks and sources. For example: Hacker News, Google Blog Search, Friendfeed, etc.
The scoring used is very simplistic. A good extension would be to make better use of the information available for each URL and combine them in more intelligent ways. Such as a weighted sum of their relative importance, more weighting to comments and up-votes, etc.
Additional meta information about a given URL on social networking sites could be used in conjunction with an intelligent parsing of the articles themselves to determine broader categories that articles could be put into. A new list could be constructed with articles by category and popularity.
The generated HTML pages will only ever contain AI articles that are actually in the source RSS feeds. A useful extension of this project would be to monitor social networking sites and other news sites for articles on AI and Machine Learning and include them in the mix.

In this part we looked at the social interest of each article from the last 7 days and used that information to promote "popular" articles to be read above "unpopular" articles. The dynamic nature of the social scoring function means that popularity can be re-calculated each time the script is executed, responding to changes in the spread and sharing of a given story.

In the next and final article in the series we will look at automating the generation of the AIFeed and in disseminating it via email and the web.

Don't forget that all code and data for this series is available for free on the AIFeeds github project.

AIFeeds Part 2: Recent AI Articles

2011-04-27T07:00:00.104+10:00

In Part1, we went through the process of aggregating and filtering a bunch of RSS feeds for Machine Learning, Data Mining, Artificial Intelligence and related topics. In this continuation we will explore the properties of articles listed within a feed and provide a way of generating a static list of recent articles.

Step 1: List articles from the past week
A problem with this effort is that there are a lot of inconsistencies in the real world data. Some feeds are missing fields and some articles have fields that do not match expectations. The first step is to explore the data that we have by listing articles from the past week. This includes all fields that we think might be usefully related to an article, a feed, and its source website.

To start with, the final list of feeds prepared in Part1 was taken and curated based on eyeballing the output in feedlist.html (also generated in Part1). The result is the file curated_feeds.txt with some of the more obviously wrong and non-interesting feeds removed.

Next, a script was prepared to generate listing of all articles from the last seven days drawn from the feeds in the curated list. Almost all information for each article is provided such that we can discern what might be appropriate or interesting in further experiments.

It was interesting that some articles had neither a date_published or last_updated field. Some articles would put the content in the description and others in the content field. These may be artifacts of the chosen RSS library, or may be artifacts of the feeds themselves, more investigation is needed. There are also Unicode issues with strange characters all over the place where apostrophes and quotes should be. Again, it is not clear whether this is because of the library or because some custom parsing is needed, nevertheless, as a static RSS the result is very basic.

See viewarticles.rb

The following provides a sample screenshot at the time I last executed the script to produce the articlelview.html file.

Step 2: Summarize articles by day
In recent months I have become a big fan of the Hacker Newsletter. It is basically an email sent once per-week that contains a summary of the top stories of the week on the website Hacker News with links to comments as well as other relevant and interesting sections. I like the layout of the email and I think it provides a good pattern to head towards in this project.

The previous step explored how to filter the feeds and how to extract the relevant information from each article, but it also demonstrated how ugly rendering the raw data can be. One could invest effort into replicating an RSS reader, or in doing something simpler and move toward a better summary view on the same data. This step explores such a simplified view where each article is presented on one line.

A script was prepared to produce an article summary for the last seven days. The same filtering was performed as in Step1, although rather than writing out a long list of articles, the script was updated to group articles by date and only list the article title and source webpage. The result is a compact listing of recent AI and Machine Learning blog posts. The resulting page also provided clearer insight into the nature of the blogs included in the master list. As a result, the curated list of feeds was further culled.

See listarticles.rb

The following figure provides a screenshot from the last time I executed the script to produce the articielist.html file.

Improvements and Extensions
This section lists some potential improvements and extensions to this part in the series.

Curating the master list of source feeds by hand is a bad idea. It seems feasible that all articles from each feed could be scanned and their appropriateness to the project determined using heuristics on the structure or content (keywords).
Unicode characters in the feeds need to be rendered correctly. This still effects the titles of the articles and the blogs themselves.

In this part we have explored the content for articles in RSS feeds and two methods for presenting the data: a fire-hose and a daily summarization, the latter currently preferred.

Next in Part3 we explore additional social-web based methods for filtering the list of articles and promoting those that are more likely to be interesting to the reader.

Don't forget, all the code and data for this series can be downloaded from the AIFeeds github project.

AIFeeds Part 1: Artificial Intelligence and Machine Learning blogs

2011-04-26T07:00:00.195+10:00

Recently, I have been neglecting my RSS feeds. I'm a Google Reader user and my unread count is far from zero. I've been distracted and too busy. The problem though, is that there are many excellent Artificial Intelligence and Machine Learning blog posts that are being written every day and I get a lot of enjoyment in reading them and keeping abreast with current research and interesting projects.

An obvious solution would be to simply rely on social media to filter such articles and let the best float to the top on Hacker News, Reddit, and Twitter. Maybe.

I felt like some hacking over my Easter break (in between the numerous family responsibilities), so I decided to have a crack at this problem. This post represents the first of four posts of my exploration of aggregating and filtering RSS feeds from popular blogs in the areas of computer science, data mining, data visualization, machine learning and artificial intelligence. I've called the project AIFeeds and have hosted the code and data for all parts on github if you want to skip ahead.

The objective of this first part is to simply build a list of a large number of blogs that match my loose criteria. Specifically, the objective of this part is to prepare a list of working RSS feeds for blogs on (AI and related fields) without any duplicates. This will then be used as the basis for the subsequent parts in the series.

Step 1: Extract relevant blogs from my Google Reader Account
For starters, I have a modest collection of relevant feeds in my google reader. Going to google reader, I clicked settings, Import/Export, then Export your subscriptions as an OPML file. I opened the OPML file in a text editor and delete all those feeds that I knew were not on topic.

Step 2: Search for lists of relevant blogs
The next step is to Google search for lists of machine learning blogs. I remember seeing such lists on Quora and MetaOptimize before so I kept an eye out for these sites in the search results. Here are some good lists I found:

Good Machine Learning Blogs (MetaOptimize)
Machine Learning Resources (Inductio ex Machina)
Updated Machine Learning/Statistics blog list (Machine Learning, etc)
Data Mining Blogs (Data Mining Research)
What are the best blogs about data? (Quora)
What are the best machine learning blogs? (Quora)

The result is a bunch of links in webpages and OPML files. Additionally, many of the linked blogs themselves provide a blog roll to related blogs.

Step 3: Aggregate
The first easy thing to do is to write a script to extract all feeds from the collected OPML files. Using the built-in REXML library in Ruby I bashed out a script to read in all .opml files in an opml/ subdirectory, parsing each entry for the rss attribute and writing a opml_feeds.txt file.

See parseopml.rb

I then did something stupid, slow, but effective. I clicked a lot of links, read a lot of blog rolls, viewed source on a ton of blogs, and copy-pasted links to blog RSS feeds into a byhand_feeds.txt. I could have written scripts for this, but there is a lot of pain to deal with and I suspected I could do this once of task by hand faster than I could write code to parse a bunch of pages, extract (the right) links, and then download the pages and extract links to rss feeds. I took an hour or so to do by hand - not a big deal.

Step 4: Filter list of feeds
So now we have two lists of RSS feeds. Some may work, some may not. There are likely many duplicates and even many that are not strictly on topic. An easy quick win is to write a script that opens a connection to each url, parses the contents and builds a list of unique feeds.

The first thing that is needed is a library that can process RSS feeds in ruby, at least RSS1, RSS2 and Atom. Google throws up a few options, such as RubyRSS, Ruby RSS in the stdlib, Ruby-Feedparser, Simple RSS, feed-normalizer, and others. I tried a few and was somewhat frustrated. I ended up going with feed-normalizer (rdoc) because it was really easy to use and provided a consistent data structure for RSS and Atom feeds.

I wrote a small script using feed-normalizer to parse a feed and return the data structure. Importantly, it provided safety for the URL failing to open, the XML being bad, and for socket timeouts. An error is reported to stdout on error and a nil is returned. I used this script to test some rss, atom and even some bad URLs. This file provides a home for helper functions for loading and parsing RSS feeds.

See parserss.rb

Using this parser function, I wrote a script to read in lists of RSS feeds in text files (opml_feeds.txt and byhand_feeds.txt), parse each and build a unique list based on the website name. There are many hundreds of feeds to check in those files and I'm lazy. I wrote a small function to hammer the feeds in the file using lots of threads and then check all of the data structures in memory. Lazy but effective. The result is a list of 236 rss feeds in filtered_feeds.txt.

See filterfeeds.rb

Step 5: Summarize list of feeds
A final step is to generate a summary of all the available feeds. An easy way to do that is to generate an HTML page that summarizes each feed in turn, ordered by the name of the website. A script was prepared to parse each feed and list the name of the site, link to the site, and finally link to the actual rss feed. The list was ordered by site name.

See listfeeds.rb

The following provides a screenshot of the generated html page providing the final list of webpages and their rss feeds in feedlist.html.

Improvements and Extensions
There are a number of improvements and extensions that could be made to the first part of this effort, such as:

The final list contains some oddball and even some off-topic feeds. A high-level human-based curation of the list would be a good start. For example, removing Q&A feeds and delicious feeds would be useful.
Some of the links to the sites in the generated feedlist.html are suspect, meaning that some feeds do not accurately report the site URL or site title. It may be possible to carry over this information from the source documents, either the website themselves and/or the OPML files. Ideally, the final corpus of feeds could/should be stored in an OPML file for portability and reuse.
Build a larger corpus of feeds. Naturally, this is the easy extension to the whole project and there are may ways to go about collecting more feeds starting with some simple google searches. Getting more OPML files and including them in the opmlparser.rb script would be a great first step.
Write a script to extract links of blogs from some of the above listed collections. Once such links to blogs are collected, write a script to download the HTML, parse it and extract the link to the RSS or Atom feed. This is most commonly listed in the head tag of the HTML file as an alternative link (at least it is for blogger and wordpress blogs).

Done
So we now have a list of RSS feeds for blogs, most of which are likely to provide posts on or related to our topics of interest. Each feed has been tested, and duplicates have been removed.

Next, in AIFeeds Part 2 we will use this list of sources to prepare a summary of posts as a simple static RSS reader.

Remember, all data and code is provided on the AIFeeds github project.

Automatically Generate Regular Expressions (regex)

2011-04-04T07:17:00.000+10:00

I had a thought about two weeks ago that I really wanted a web app that would generate regular expressions for me. I was lazy and had spent some time hacking expressions together for a script I was working on.

I wrote some notes and posted them to a gaggle of programmers and engineers I email regularly. Something like:

A programmer normally google’s for examples of their problem or similar problems online, copies the code and adapts it for their needs. There are websites dedicated to discussing programming problems, and websites dedicated to listing solutions to common problems. The objective of this system is to write a new program for the user from scratch that exactly addresses their needs.

A user defines some sample input and some sample output. The system examines the data and generates a program for the user to perform the transform. This program can be downloaded, executed in place, or used as a service. The more information provided by the user, the better the robot programmer can do its job. The robot programmer may not get it right the first time, providing an opportunity for the user to work with the robot to better define the problem and tinker with candidate programs that may be used as the basis for the robots second and subsequent tries.

I abstracted the problem and proposed a generic "robot programmer" or "robot coder" that would compute arbitrary transformations based on sample input and output. I pitched both the general case and the specific regex case and got a variety of responses and some good technical discussions.

Using Heroku, I hacked together a prototype of the interface and sent it around for comment. The following is a snap shot of the regex mock:

The following is a snap shot of the general case mock, here the example requires a program to print the input as well as all the input summed together.

Ideally, the service would be free for the simple case, and cost a dollar or two for a more advanced case. The free case would allow a large corpus of examples to be collected and the system refined, and the complex case would provide useful pay-only features such as: save, private, download in preferred language, and pair-program with the system. This last case would be the truly useful incarnation of the system where one could modify the expressions or programs and I/O and iterate the solver until a desired solution was achieved.

I still think it is a good idea, but I just couldn't think of a good way to monetize it. It is an interesting technical challenge and the service would make an interesting novelty, but it is not an earner and I'd rather hack on other things for now.

I have some designs and small technical prototypes for the ensemble of inductive automatic programming solvers and heuristic solvers that would be needed, but I'm keeping the details to myself for now, just in case the idea gets some traction or I can think of a better way to make a few bucks from it (Excel plug-in?).

Sure the general problem is intractable, but the specific case is results-driven. It only has to be good enough for the users needs - a monkey patch for whatever data conversion or scripting they happen to be doing at the time. The subset of actual problems that people have will be much smaller than all of possibility space, and the web is a fertile ground for seed and test instances.

Anyway, if anyone has a burning desire for either or both of these services to exist or has some further ideas, drop me a comment or an email.

April Challenge: Watch 1 Tech Video Every Day

2011-04-01T07:01:00.000+11:00

I was recently thinking about my junkfood media consumption. My thesis was that one could apply the same discipline that one uses for discerning the food they eat to media consumption, ideally with the intention of self improvement.

If one must consume to relax (as opposed to drinking gin), then why not impose discipline to avoid the fast food of consumption (reading online news, twitter, American TV shows, etc.) and consume with intent.

The April challenge is self imposed (unlike the March Challenge) and is as follows:

Watch one free online technical video for each day of April. For each day missed, you (I) must donate $20 to .NET open source projects.

I'm experimenting with a "loss" as a motivating facto, rather than the "gain" of the March challenge. I'm really starting to get into this public challenge approach to self improvement :) Also, I recently sold our family media solution, so there is far less opportunity for general lazy consumption.

Seriously, why would one write open source using the .NET framework? It has oxymoron written all over it. Donating to open source is good, and $20 is not too much pain if I miss a day. It quickly starts adding up though, 5 days is a $100 and I'd much rather give that $100 to some noble device driver hacker working on Linux than some .NET hacker playing around in Visual Studio, or whatever it is they do.

This challenge does not preclude my normal media consumption, although it imposes constraint on at least one hour of that dedicated time, typically in the evenings. If I have a busy night, I guess I'll need to make time in the morning or during my lunch hour. I intend to maintain a spreadsheet of what I watch, it's duration, and its URL. I'm sure I'll think of ways to slice and dice this collected data.

Some sources of media may include:

I am passionate about technical topics, specifically Artificial Intelligence and Machine Learning, so this directed form of media consumption does align with my interests. I have to focus and concentrate for tech talk, take notes, think, quite different from what I need to do when I consume acquired TV shows. It will be interesting if I can/should switch to more passive consumption for interesting subject matter (what if I forget something amazing!?!). Alternatively, the penalty for remaining active in "relax time" may be worse, irritability, increased tiredness, etc. It will be an interesting experiment.

I'm looking forward to this challenge!