YT!

Land and Waste New Zealand - #mixandmashnz

noreply@blogger.com (YT Kuah) — Sun, 25 Sep 2011 00:01:00 +0000

Land And Waste New Zealand

Here's a behind the scenes write-up for my Mix and Mash NZ mashup.

Check it out here :

http://www.landandwaste.co.nz

This mashup attempts to show land and waste data as it changes, and hopefully illuminates any improvements of our environmental impact as well as regressions.

The journey

It started out as an idea to gather all kerb recycling data from all 75 odd local councils in New Zealand. That meant scraping the data off the websites of various local and unitary councils. I quickly backed away once I found that lots of councils forbade any re-use, and the data that's available is too inconsistent for me to compare and visualize.

Therefore much time and effort was spent looking for suitable data. I was interested in mainly environmental type data, which was to found mainly on the Ministry for the Environment website. I was also interested in regional level data, instead of a single national figure, which wasn't that interesting .

In the end, I found suitable data for land use, rates, farm animal counts, population, and waste. Given more time, I would have hunted for transport data, which would then completely encompass all the major responsibilities of a regional council.

Putting it together
This mashup was made with Ruby on Rails, TileMill (http://mapbox.com/tilemill/), d3 (http://mbostock.github.com/d3/) and polymaps (http://polymaps.org/)

The excellent Koordinates.com put up a CC licensed shapefile of various land types in New Zealand. Instead of reaching for the de-facto choice of Google maps, this time I chose to try my hand at creating custom map tiles from that data. This had the advantage of being able to design the map to be far friendlier for statistical visualizations like cartograms, etc.

TileMill is simply a brilliant piece of software, which allows once to pull data from various sources and create beautiful maps, using any style you want. In addition, there are options for mouseover and click events.

There is an export function which will allow you to host the map tiles online ala Google Maps. I chose to utilise the hosted service at TileStream (http://mapbox.com/#/tilestream). It is completely possible to self-host as well, as the hosting code has been made available athttps://github.com/mapbox/tilestream.

The final map I produced using TileMill can be seen at http://www.landandwaste.co.nz/land.

Importing and processing the tabular data seen on the mashup was just a matter of using a combination of Google Docs, Google Refine, and csv files. Choosing a suitable graphical representation for each set of data was slightly tougher. I used a combination of sparklines, quantile maps, and just plain text. This was done using d3 and polymaps.

Land use area visualization (example here for Bay of Plenty)

Waste visualization for each region in New Zealand

Where to from here
It was a huge learning experience. I learnt that creating something useful is very possible, and takes less time than I expected. Creating something polished requires far more time. I will definitely find a designer or improve my own design skills next time.

Fix home/end keybindings in Terminal.app on OS X

noreply@blogger.com (YT Kuah) — Sun, 12 Sep 2010 23:07:00 +0000

http://www.yaroze.org/?p=79

Go all the way to the end of the less buffer! Drove me crazy the way end behaves exactly like page down instead. This tip fixed that for me.

See this Amp at http://amplify.com/u/abkv

Why Users Fill Out Forms Faster With Top Aligned Labels | UXMovement.com

noreply@blogger.com (YT Kuah) — Wed, 01 Sep 2010 22:56:00 +0000

URL: http://uxmovement.com/design-articles/faster-with-top-a...

See this Amp at http://amplify.com/u/9kjp

Untitled

noreply@blogger.com (YT Kuah) — Mon, 23 Aug 2010 02:35:00 +0000

Amplify’d from martinfowler.com

Identity Map

Ensures that each object gets loaded only once by keeping
every loaded object in a map. Looks up objects using the map when
referring to them.

An old proverb says that a man with two watches never knows
what time it is. If two watches are confusing, you can get in an even
bigger mess with loading objects from a database. If you aren't
careful you can load the data from the same database record into two
different objects. Then, when you update them both you'll have an
interesting time writing the changes out to the database
correctly.

An Identity Map keeps a record of all objects that have been
read from the database in a single business transaction. Whenever you
want an object, you check the Identity Map first to see if you already
have it.
Read more at martinfowler.com

See this Amp at http://amplify.com/u/92bh

Facebook - greedy defaults and jarring configs

noreply@blogger.com (YT Kuah) — Mon, 23 Aug 2010 00:29:00 +0000

Commited Facebook Suicide for the second time. Where do I start? ...

The first thing that jarred me was that my full birthday is published. This in combination with search engine visibility means that it's spammer time.

Next thing that happened is that the privacy settings had "Everyone", "Friends of Friends", and "Friends". Nice and simple, I thought. But no, I have to exclude my full birthday again from everyone. (I know about how your friend's applications can just mine everyone's data through friends).

Ok, I'll just configure that again.

What do I see next. Social Ads. Ads. Default On. In my name. I hate Ads that purport to be in my name. That's it. Goodbye.

--

Here are some tools below that can help you, but to be honest, I don't think anyone can keep up with the myriad of configurations.

http://www3.untangle.com/saveface

http://www.reclaimprivacy.org/

http://github.com/mjpizz/reclaimprivacy

http://itunes.apple.com/us/app/s-n-a-p/id384773305

See this Amp at http://amplify.com/u/9252

Bash hackery to get around old rspec

noreply@blogger.com (YT Kuah) — Fri, 18 Jun 2010 03:07:00 +0000

I have a problem where a project is using an older version of RSpec (currently the lastest version is RSpec 1.3.0). Therefore running

spec spec/model/some_model_spec.rb

gives me an error

/spec/models/../spec_helper.rb:23: undefined method `use_transactional_fixtures='

The solution would be to use the older, vendored spec. Now, the spec binary that is installed as part of the gem installation should do this (some gems do checked for vendor'ed binaries) and my rspec is installed under vendor/plugins, not vendor/gems. Therefore, I use some bash trickery to dynamically redefine the "spec" command. Put the code below in .bashrc :


vendored_or_path_spec() {
  if [ -x vendor/plugins/rspec/bin/spec ];
  then 
    echo 'vendor/plugins/rspec/bin/spec';
  else
    echo '*spec';
  fi
}
alias spec='$(vendored_or_path_spec) $@'

Some explanations. The hardest thing to figure out was how to dynamically alias something. After such scrumbling and asking on vark.com, I managed to find command substitutions, which is the $(command) part. Combining that with $@ for all arguments meant that I can run any command in the alias.

Because I'm aliasing spec, I use *spec to refer to the original un-aliased spec.

That's it!

How to upgrade to Firefox 3.6 for Karmic (Ubuntu 9.10) - firefox-stable

noreply@blogger.com (YT Kuah) — Fri, 23 Apr 2010 10:09:00 +0000

There are a plethora of install instructions out there for Firefox 3.6 Karmic (which uses Firefox 3.5 by default), however it mostly suggests the mozilla-daily build.

By chance, I found the firefox-stable ppa (https://launchpad.net/~mozillateam/+archive/firefox-stable/)

Here's how (enter each in turn):

sudo add-apt-repository ppa:mozillateam/firefox-stable

sudo apt-get update

sudo apt-get upgrade

sudo apt-get dist-upgrade

Here's what it looks like:

~ $ sudo apt-get upgrade

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages have been kept back:

firefox firefox-3.5 firefox-3.5-branding firefox-gnome-support

0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

~ $ sudo apt-get dist-upgrade

Reading package lists... Done

Building dependency tree

Reading state information... Done

Calculating upgrade... Done

The following NEW packages will be installed:

firefox-branding

The following packages will be upgraded:

firefox firefox-3.5 firefox-3.5-branding firefox-gnome-support

4 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 12.5MB of archives.

After this operation, 32.4MB of additional disk space will be used.

Do you want to continue [Y/n]? y

You will now get /usr/binfirefox as Firefox 3.6

Rails 3 presentation

noreply@blogger.com (YT Kuah) — Sun, 07 Feb 2010 03:02:00 +0000

At the January WellRailed (Wellington Rails User group), I gave a presentation on the upcoming Rails 3.0, with special emphasis on cool new things.

You can view the slides after the jump. There are a huge number of changes, but it should be all good fun :)

Rails 3 : Cool New Things by Thong Kuah

View more presentations from Y. Thong Kuah.

Note: Rails 3.0pre was released after the presentation

Nuggets of inspiration

noreply@blogger.com (YT Kuah) — Tue, 27 Oct 2009 09:33:00 +0000

Some good Software Engineering pearls of wisdom :

Coding Horror: The Xanadu Dream

The bottom line is that a lot of the time it's OK to create a solution that solves 80% of the problem. Always remember that shipping is a feature.

Agile development, startups and government policy

My recommendation to just about anyone with an idea is to just build the thing, iterate until you have some user traction, then pitch angel investors based on that traction. This is very much in line with the old IETF motto of "rough consensus, running code."

And from the startup side of things:

Finding Your Co-Founders

It’s most comfortable to hang out with people like ourselves, but those are exactly the folks you probably don’t want to co-found a startup with. ...The best founding team for a startup is a group of two or three people who have synergistic – not overlapping – skills. Note that it’s also important your goals and passions be similar.

This matches well to what the Black Swan book says - maximise your serendipity, you increase your chances of success this way

Wellington Hazard Map

noreply@blogger.com (YT Kuah) — Mon, 19 Oct 2009 08:42:00 +0000

Reading this article about Wellington Hazards just makes me want to plot it on a map. It took a while, but here is a map of Wellington Hazards / Contamination. Below is a screenshots of the hazards on Google Earth and Google Maps (Google Earth shot nicely prepared by Cameron). Let me know what do you think?

Read on after the jump for hyper-technical explanation of how I made the map.

Here's the lowdown on the technical wizardry. The data is based on http://static.stuff.co.nz/files/WCC-slur.pdf. I had to extract the data out from the pdf, obtaining a spreadsheet that looks like this:

No & STREET & City           HAIL & CLASSIFICATION& CONTAMINANTS
50 BUCKLE ST EASTERN         Service Stations Verified History of Hazardous Activity or Industry
75 DARLINGTON RD EASTERN     Landfill Verified History of Hazardous Activity or Industry
501 EVANS BAY PDE EASTERN    Service Stations Verified History of Hazardous Activity or Industry
9 -11 KENT TCE EASTERN       Storage Verified History of Hazardous Activity or Industry

The address leaves a lot to be desired. Having data like 'Eastern' for the city column makes no sense to anyone or a program. Cleaning that up and adding "Wellington NZ" would make it easier for geocoding. We now have this:

HAIL & CLASSIFICATION& CONTAMINANTS                           Full Cleaned Address
Service Stations Verified History of Hazardous Activity or Industry 50 BUCKLE ST Wellington New Zealand
Landfill Verified History of Hazardous Activity or Industry           75 DARLINGTON RD Wellington New Zealand
Service Stations Verified History of Hazardous Activity or Industry 501 EVANS BAY PDE Wellington New Zealand
Storage Verified History of Hazardous Activity or Industry            9 -11 KENT TCE Wellington New Zealand

We can now geocode, using the helpful spreadsheet geocoder (instructions here). The only limitation was that only 100 addresses can be geo-coded at one time. But that's fine as I only have 592 addresses.

Now here is the tricky bit, there is no good way to get items from a spreadsheet onto a map. The best way I could was use KML. I used the Spreadsheet Mapper v2.0, but it was far too painful with the browser locking with just 600 rows. Here is the resultant spreadsheet: http://spreadsheets.google.com/pub?key=tXlYA3d_eh42qOboocs8zug&output=html. I will most likely use PostGis and the asKML function next time.

Data spreadsheet: http://spreadsheets.google.com/pub?key=t_4WKBgw34UvW_zQMChuVKw&gid=0

Google Maps (my maps): http://maps.google.co.nz/maps/ms?ie=UTF8&hl=en&msa=0&msid=105368590499327803003.000475dc20f9a8d9d5142&t=h&z=11

KML file (.kml): http://maps.google.co.nz/maps/ms?ie=UTF8&hl=en&t=h&msa=0&output=nl&msid=105368590499327803003.000475dc20f9a8d9d5142

Rands In Repose: Your People

noreply@blogger.com (YT Kuah) — Tue, 29 Sep 2009 09:41:00 +0000

Came upon this really good post about people you should cherish:

Rands In Repose: Your People

Really good insight - "As we edit our days into these stories, there is always a risk of fiction. This is why you need to identify and nurture Your People."

What are Your People?

"You tell these stories to Your People without reservation. Your People love your stories — fiction and all. They love how you tell them, they laugh about the lies you tell yourself, and then they stop and they tell you the truth."

You need someone to keep you honest. Read the post for a clear definition of Your People.

Strengths, weaknesses and games

noreply@blogger.com (YT Kuah) — Tue, 22 Sep 2009 08:52:00 +0000

StrengthsFinder is an amazing book, and an assessment tool that helps you discover your strengths. Its central concept is that you should focus on your strengths, not your weaknesses. I was introduced to it at the Summer of Code (NZ) program several years ago. The concept of focusing on strengths was just brilliant.

This really reminded me of my game playing style, especially in RPGs, where you have to increase the levels and skills of your character(s) in order to be able to fight enemies that pop up with increasing difficulty and toughness.

What is the usual strategy here? Increase your strong attributes up to insane levels. Weaker attributes would be increased only for the sake of 'balance'. To translate into gamer-speak:

Level up all magic skills for the mage, and have some combat skills to keep up with the group. Increase the tank's combat and defense skills to insane, and increase healing and magic if there's a chance.

Balance is important as weaknesses cannot be so abysmally weak that you cannot function if a team member which has that strength was unavailable. In gamer-speak again:

Tank has to be able to self-heal and revive mage if mage goes down, and hang on until mage revives and heals group.

Imagine instead we concentrate on improving weakness more than improving strengths. What would result is a group that has similar strengths and weakness, i.e. a homogeneous group, but at levels lower than peak strengths. This team of 'all-rounders' would then be able to do each other's jobs, but none would available to handle particularly difficult problems. In game-land, this is referred to as:

Boss kills mediocre party in one stroke.

Whereas a team with people who has different and fully-developed skills will have a better chance of tackling the problem. In game-land, this scenario is better:

Boss unleashes stroke of hell, tank survives and manages to revive others. Or Mage casts protective spell powerful enough to protect party.

Which brings us to complementary strengths. The team must have a variety of different strengths to deal with different problems that crop up. This, ideally should be addressed when you are starting to hire for the team. Of course it's hard to figure out strengths in an interview. Take time, figure out what strengths are needed for teams, as you never know what ~~high-level boss~~ crazy, hairy problem your team will face...

Ruby and Rails links, Best of

noreply@blogger.com (YT Kuah) — Tue, 15 Sep 2009 10:13:00 +0000

Monthly collection of Ruby links that I'll regret forgetting about:

find_each bites

noreply@blogger.com (YT Kuah) — Tue, 08 Sep 2009 11:11:00 +0000

ActiveRecord::Base#find_each is an awesome idiom for batch processing large sets of data. It combines the neat DSL of ActiveRecord and allows you to operate at the level of the individual item.

So, imagine my surprise when I debugged my code and discovered that find_each introduces a scope into ActiveRecord calls within the block. Demonstrating by example:

Slam.find_each(:conditions => {:value => nil}) do |slam|

#....

  Slam.find_by_key(slam.key)   #Find  related key

  #However, in effect becomes Slam.find_by_key(:key => slam.key, :value => nil)

end

Looking at the SQL in the logs, I expected to see:

select * from slams where key = 'abcde';

Instead I saw:

select * from slams where key = 'abcde' and value is null;

It took me fifteen minutes to check and verify there is no code that should be generating that SQL, so the next candidate was find_each. Lo and behold, it was. Turns out find_each is using with_scope.

Looking around, there are already some bug reports for this behaviour:

Question: Is this intended or this a bug? I think, given the "side-effect" feel of it, it's a bug

Update: I have reviewed one promising patch to fix this. Needs one more person to review and approve this patch!

Fear the singularity

noreply@blogger.com (YT Kuah) — Sun, 06 Sep 2009 09:17:00 +0000

I found several interesting things this month.

Super fast SSD hard disks
21GB pirate bay index
Singularity in thirty years

Ok, the last one is really out there, but it points the way. Imagine the whole sum of human creative endeavor (at least the popular ones) indexed in just 21GB. All we need is just some sort of instant hash expansion for that 21GB (and 20 more years of PC power advancement). And so on and so forth. Value of data or content would be worth next to nothing.

Will that happen? Probably, after flying cars.

Trying to do the impossible

noreply@blogger.com (YT Kuah) — Sun, 06 Sep 2009 09:01:00 +0000

Someone always asks this question - how do you check types in Ruby? (*Hint: Ruby use dynamic typing) This typical newbie question would ask this and be answered with various levels of helpfulness - typically none - sadly.

However this thread is just beautiful in its helpfulness. http://groups.google.com/group/comp.lang.ruby/msg/eb4b49a511e44be6?hl=en

What happended snarkiness, or Slashdot / Youtube style comments? All this helpfulness must come from somewhere. I blame Twitter.

Hackfest at the OpenGovtDataNZ

noreply@blogger.com (YT Kuah) — Mon, 31 Aug 2009 09:47:00 +0000

Hacking for the greater good : the good being helping to free up important government information. A Barcamp and Hackfest was held just this weekend, in the last days of August. I participated in the Hackfest, which was held at the amazing Southern Cross Bar. Great food and nice private room made for a good environment for coding.

The Hackfest was a bit more structured than previous events I have been to, but I'll say this had a positive effect of helping people choose what and who they wanted to work with. There were some suggested ideas from the Barcamp, but people were free to work on their ideas too. Here are some projects that was worked on at this event :

And many more, which is listed at:

http://wiki.open.org.nz/Hackfest

It was great collaborating and working with great people and programmers. Hope to do this again very soon!

Solr, and Sunspot

noreply@blogger.com (YT Kuah) — Mon, 24 Aug 2009 09:35:00 +0000

Update: I have found a nicer way of doing autocomplete. Basically, use the string type with the "starting_with" method. See this ticket comment.

Solr is really nice search engine which has the added advantage of being able to return items that you put into the search engine, unlike say, sphinx (which is really fast but only returns the document id).

However, Solr is very complex.

Thankfully, there is a very neat ruby abstraction for Solr, called Sunspot. Given I needed to do autocomplete on it, i.e. "query*" type stuff, I had to fork Sunspot. Here is the relevant change I made to enable wildcard searches with Sunspot:

http://pastie.org/592897

For those interested in how this works, it uses the q.alt parameter as Sunspot uses the rather "user-friendly" DisMaxRequestHandler, which ignores "*" characters.

Throw away code

noreply@blogger.com (YT Kuah) — Sat, 22 Aug 2009 05:39:00 +0000

Let's start simple. Here is a situation I have encountered more than once. We have this big long class or method, and embedded in it are several blocks of code, all commented out.

def some_method(a)
do_something(a)
calc_another

# wow
# do_do
# calc_calc
# zcalc(z)
end

What should I do? Given the code is version controlled, I would happily throw it away.

The code is commented out and will have no effect whatsoever on any program or user. Another programmer may be affected by it, though. So, make a note in the commit message :

"removed commented-out code in such-and-such method. haven't been used in a while"

Caveats apply for very large blocks of commented-out code, or it has warnings not to remove. But these can be easily solved with some programmer-to-programmer communication.

There are other things that can be theoretically thrown away, such as:

Once off experiments / prototype
Quick and dirty functions
Generally ugly code in the project

However, each of this has it's own set of code, teamwork and business-related problems. Yes, there is the urge to reduce mental strain - working what does piece of code do - which leads to "let's rewrite" moments. I would love to know what a programmer or a team can do in each of this instances. I have some ideas, but these will be topics for later discussions.

* I have always wanted to write down some of my experiences as I embark on a programming life. This is my starting contribution.

Riding Rails: Community Highlights: IronRuby

noreply@blogger.com (YT Kuah) — Wed, 12 Aug 2009 00:23:00 +0000

Consider this question from Riding Rails: Community Highlights: IronRuby (emphasis mine).

Matt: Are there any limitations that our readers should be aware of before starting to develop on IronRuby?

Jimmy: The main limitation is that IronRuby does not support any of the C-based Ruby libraries, and only after 1.0 will we consider building an interop layer between the Ruby C API and IronRuby. In the meantime, people have been porting their favorite C-based Ruby libraries over to C# so it can be used from IronRuby, like Hpricot. While this seems like a large limitation, most of the C-based libraries Ruby code depends on have an equivalent API in the .NET framework, which IronRuby has direct integration with, making either using directly or porting really easy. For example, the Rails app I showed at RailsConf did image resizing directly with the System.Windows.Drawing APIs rather than ImageMagick. If your code does not depend on anything outside of the Ruby standard library that is C-based, you should have no problems. Take a look at the documentation for running Rails on IronRuby to make sure things go smoothly: http://ironruby.net/Documentation/Rails.

Sure reminds me of Embrace, extend, extinguish. I'm sure the interop layer is being considered, but the absence of it will tie IronRuby to .NET and away from Ruby. I hope the interop layer will be done soon.

Git Workshop

noreply@blogger.com (YT Kuah) — Wed, 05 Aug 2009 23:21:00 +0000

I did a Git workshop for Summer of Code NZ students. Here it is:

Code Management

View more documents from kuahyeow.

Full details at the Summer of Code blog:

http://blog.summerofcode.co.nz/2009/07/31/code-management-git-workshop/

Rails 3 updates

noreply@blogger.com (YT Kuah) — Wed, 05 Aug 2009 23:16:00 +0000

(Cross-posted from http://blog.projectxtech.com/2009/08/06/rails-3-updates/)

Rails 3 is a rewrite / merger with Merb, and includes Yehuda Katz, Merb lead developer working on it full-time. Here are some posts which were published recently:

http://yehudakatz.com/2008/12/23/rails-and-merb-merge/

http://weblog.rubyonrails.org/2009/7/30/community-highlights-yehuda-katz

http://yehudakatz.com/2009/03/06/alias_method_chain-in-models/

http://www.engineyard.com/blog/2009/6-steps-to-refactoring-rails-for-mere-mortals/

There are lots of lessons and techniques in ruby, rails and general programming to be learnt from these posts. Enjoy!

Jaunty, Twitter and 'Growl' type notifications (notify-osd)

noreply@blogger.com (YT Kuah) — Sun, 17 May 2009 21:51:00 +0000

It's been a long, two-stepped journey to get to Jaunty from Hardy, all so that I can experience notify-osd with Twitter updates.

The upgrade has been long and slow (NZ internet!) but I found the end result strangely unsatisfying. Yes, notifications work, and there's a new "Computer Janitor", but that's it. A bit let down.

Then I found out about Gwibber, the new twitter client on Ubuntu. It is by far superior to other clients on Linux. Previously, pidgin-twitter had to suffice, but no longer. However, Gwibber does not seem to do notifications on my Jaunty laptop. In my Intrepid desktop, notification bubbles do pop up, though in non-NotifyOSD style at the bottom.

In other news, I've hit a bug / issue with my Intrepid desktop where I can't open any folder in the File Manager. :( Gnome or Nautilus experts that can help would be much appreciated

'View as single page'

noreply@blogger.com (YT Kuah) — Tue, 09 Sep 2008 10:42:00 +0000

Just wanted to rant about how news websites have the annoying feature of splitting up articles into 'pages' - of just two or so paragraphs. There'll be three or four of these 'pages'. But once you click on the 'view as single page', you realise how ridiculously short the article is, to even split to any page. I wish there's a magic unpage! bookmarklet out there.

Google Talk for your domains - Pidgin

noreply@blogger.com (YT Kuah) — Wed, 27 Aug 2008 04:19:00 +0000

Just found out that Pidgin now has Google Talk. That means my GAFYD GTalk now works. Awesome!