Eadz :: Blog

Trademe's New API

2010-12-22T00:00:00+13:00

If you haven't already heard, Trademe has recently released a public API. I believe this is big news, as Trademe is arguably the biggest thing on the .NZ Web.

Over at Webforce, I have been busy integrating with the Trademe ( and Xero ) API for various customers.

Trademe uses this API internally for their iPhone App, which is quite nifty and worth checking out! The app also gives you an idea of what is possible through the API. However the possibilities are quite large.

I see Trademe API integrations breaking down mainly into Buyer apps and Seller apps.

Three examples of Trademe API Buyer Apps

One of the first Buying apps was based on this XKCD Comic:

The bot will automatically buy small value items from Trademe. Not the most useful application of the Trademe API, but entertaining none the less! You can read more about it here.

Other 'buyer' type apps may involve reformatting Trademe listing data to be easier to use. For example Find my Flat uses Trademe's real estate data and displays it on a Google Map. This provides the buyer different a more visual representation of where available flats are.

Another good example is MagicCards.co.nz ( Made by myself @ Webforce ). It allows the buyer to browse trademe listings by sorting and searching by methods not available on Trademe. For example, Magic Card buyers would otherwise have to manually look through the entire card listings on trademe to find all of the cards from a particular set ( regular release of cards ).

Seller Apps

Apps for helping sellers will come thick and fast. At Webforce I have made 3 so far, and these will be released in the new year. I do custom work for power sellers on Trademe to enable them to automate various parts of their business, for example listing products based on their product database and processing orders automatically and quickly as they come in saving many man hours of manual work.

The API itself

The API itself will be familiar to many as a 'RESTful' API. It can output in various formats such as XML and JSON, and the authentication is done via oAuth.

I use Ruby on Rails and the Ruby oAuth gem to connect with Trademe, however the API can be accessed from any languages you might want to use as it is an open web based API.

For a young API, I am impressed. While it doesn't cover everything, it does expose a lot of Trademe to API developers. Xero also has an API I use, and by comparison even basic stuff is still missing ( such as customer balances ) - and Xero has had an API for much longer than Trademe
- so I just say 'Good Work' Trademe.

How Vodafone broke my hosts file and redefined "tech" in the process

2010-08-24T00:00:00+12:00

This post is going to be very technical, so to my non-technical readers ( and Vodafone tech support, see below ), I apologize.

Among other things, I am a web developer. Part of a web developer's job is to get the website up and running on a web server. Sometimes we need to change websites from one server to another. Often when doing this we rely on our "hosts" file to override the IP address our browser tries to connect to. Before I go any further, if I've lost you already, there no point carrying on.

So you type in www.eadz.co.nz into your browser. Your browser asks your OS for the IP address of www.eadz.co.nz through the DNS system. Once we have the IP address, say 1.2.3.4 our browser connects to port 80 of 1.2.3.4 and starts a HTTP dialogue to request the web page www.eadz.co.nz. Right? Well with vodafone, wrong!

The end result of whatever proxy/routing/cisco magic Vodafone is using is that Vodafone completely ignores the IP address when I visit a site, causing the hosts file not to work.

A quick demonstration; lets telnet to Google's server and ask for Yahoo's website. In a ISP setup this wouldn't work.

telnet google.com 80
Trying 66.102.11.104...
Connected to google.com.
Escape character is '^]'.
GET / HTTP/1.1                  
Host: www.yahoo.com

HTTP/1.1 302 Found
Date: Mon, 16 Aug 2010 18:38:30 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="...snip..."
Cache-Control: private
Location: http://nz.yahoo.com/?p=us
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Age: 0
Server: YTS/1.18.5
......

As you can see I get a response redirecting me to nz.yahoo.com, this response did not come from the ip address 66.102.11.104 which I tried to telnet to, but from what Vodafone believes yahoo's servers to be.

It is easy to see how this breaks the hosts file. You put an entry of 123.123.123.123 www.mydomain.com in your hosts file and you try to visit the site. While your browser will try to connect to the IP 123.123.123.123 vodafone will intercept this request and instead ask whatever IP Vodafone believes to belong to www.mydomain.com instead for the website.

For another couple of examples of this sillyness, lets connect to an unassigned IPv4 address. With IPv4 addresses running out very soon this demonstration will no longer be possible. So lets to go IANA and look for the blocks of unassigned ip addresses. So 105/8 is unallocated.. so there should be no website there right? Especially IANA's..

telnet 105.1.1.1 80
Trying 105.1.1.1...
Connected to 105.1.1.1.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.iana.org

HTTP/1.1 200 OK
Server: Apache/2.0.63 (CentOS)
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Date: Tue, 31 Aug 2010 08:51:02 GMT
Content-Length: 5200
Connection: Keep-Alive
Age: 203

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML
<html xmlns="http://www.w3.org/1999/xhtml" 
<head>
	<title>IANA — Internet Assigned Numbers Authority</title>

So once I had decided my hosts problem was due to Vodafone, I called them up. I tried both the consumer internet support and the business line, and neither contact centers had anyone who could help me, due no no-one knowing much about how HTTP works. I'm under no illusion that for the majority of people this crazy proxy stuff doesn't affect them, and that Vodafone's tech support will be perfectly adequate for most users. However it is a real shame that there was no one who could even discuss this matter with me. In frustration I asked if there was anyone there with a networking qualifications or computer science degree but I was told there was not, so my problem remains unsolved.

So is this Tech support? Vodafone sells access to the Internet, and when it comes down to it the Internet is a collection of protocols. So should their "Tech" support be able to support the protocols that make up the "Internet"? Or just Layer 7 services like email and web browsing? What does "Tech Support" from an ISP mean to you?

With all that said this is not a dig at Vodafone, I'm pretty happy with their services and I don't expect much different from other large ISPs. In the interests of full disclosure I called at the weekend, but vodafone's website advertises "24/7 tech support" so this shouldn't make a difference.

2 Ruby on Rails security tips

2009-09-11T00:00:00+12:00

Working on open source rails applications, I sometimes come across security issues. Rails makes writing secure code reasonably easy - it's hard to make a SQL injection hole in a rails app without trying - but there are a couple of common issues I see repeatedly.

Regular expressions are oven used for validation. However ruby's regexes can behave differently to what you'd expect. Say you're validating an IP address and you have a regular expression that matches an IP address, do not use /^[regex]$/, use /\A[regex]\z/ instead

In ruby, the ^ and $ characters match the beginning and end of line. An attacker can bypass the validation by inserting a newline after a valid IP address, then whatever content he wants. \A and \z should be used instead of ^ and $ - they match the start and end of the string instead of the line. If you want to test out your ruby regular expressions, check out Rubular.

This second tip is well known, but I'll mention it here because it's important.
I'm not going to repeat the API docs, so I'll just say, make sure you understand attr_accessible: and why you would want to use it. Hint: if you have an admin variable on a user model for example, and it isn't protected from mass assignment, I can become an admin by modifying a user edit form.

Best VPS for Ruby on Rails ( or anything )

2009-08-18T00:00:00+12:00

I have been using Clustered.net for about 2 years. In that time the service has been excellent and there has been practically no downtime ( their record is 99.994% over the past 2 years ).

What I like best is their redundancy – redundant everything – cisco switches, firewalls, disks, network cards etc.
I recommend if you are looking for a quality vps provider to check them out.

RailsCamp AU 2009

2009-08-16T00:00:00+12:00

I’m heading over to Melborne for RailsCamp #6 on the 20th to 23rd of November. Any other Kiwi’s going to join me for some RoR hacking?! If you haven’t seen it already, check out ToolmanTim’s awesome project from a previous rails camp, "bananajour ":http://toolmantim.com/articles/bananajour

I’m looking forward to meeting some other local and not-so-local Ruby on Rails developers. Should be great!

How to get Safari to open in new Tab instead of new Window

2009-08-04T00:00:00+12:00

When clicking a link in Safari, sometimes it will open a new window. If you want it to open in a Tab instead, use the following command in the Terminal, then restart safari.

defaults write com.apple.Safari TargetedClicksCreateTabs -bool true

Easy as that!

Free Web Design Quote Service

2009-06-18T00:00:00+12:00

Recently launched, Quoter is a free service for people looking for web design. If you’re after a web site, pop your details into quoter.co.nz and up to 5 New Zealand web design companies will respond with quotes!

Go get your free web design quotes!

NZ Speed Camera Map

2008-11-13T00:00:00+13:00

My first mashup :) Take a gps file of new zealand speed camera locations, add in some google maps and you have a map of nz fixed speed camera locations

Google Aotearoa

2008-09-16T00:00:00+12:00

It seems google.co.nz is now available in M�?ori !

Free car rentals in New Zealand ( Relocation Deals )

2008-09-01T00:00:00+12:00

Maybe it’s a well kept secret, maybe not, but it is possible to get free one way car rentals in New Zealand. Car rental companies sometimes have too many cars in one location and require them to be relocated. They give you free rental, and even pay for your ferry crossing if traveling between the North and South Islands. I used these free rentals a couple of months ago to travel from Dunedin to Auckland and back again. I got a free car from Dunedin to Christchurch, and from Christchurch to Auckland I got a 2007 4 berth camper van/motor home/RV. The motor home wasn’t free, I paid a token $5/day, but usually these are $120-$200/day so I’m not complaining.

Until now, if you wanted one of these deals you had to ring up all the car rental companies and ask if they had needed any cars relocated. But today a new site has launched which lists all these car rental relocation deals across New Zealand, http://www.transfercar.co.nz/. Enjoy!

Commerce Commission's report on broadband, and why Woosh isn't in it.

2008-06-18T00:00:00+12:00

The NZ Commerce Commission released a report this week comparing the performance of New Zealand broadband providers. I was interested in the report to see what it has to say about the best ISPs. It turns out the Orcon, which happens to be govt owned ranked highly.

So how did Woosh do?

Well is woosh even broadband? According to Woosh’s webpage, yes, it is
bq. “Woosh means personal, portable high-speed broadband and calling.”

What did the commerce commission say?

… because of limitations in the technical capability of its fixed wireless technology,
and therefore no longer attempts to directly compete with DSL providers. …

I think they are right, to a point. Maybe one day Woosh will compete with broadband, but for the moment it’s marketing material reads “Woosh is much faster than dial-up.”. I’ll keep you posted on Woosh’s progress at competing with Dial-up.

See also NZ herald coverage and the report in PDF format.

New Zealand SEO Seminars & Training

2008-01-30T00:00:00+13:00

Searchmasters are holding SEO (Search Engine Optimisation) seminars this year in Auckland on May the 19th to 20th.
More Details & Register

SimpleDB and ActiveRecord

2007-12-16T00:00:00+13:00

Amazon has announced SimpleDB – a database-like web service. While it has limitations, if an ActiveRecord adapter for Ruby on Rails can be made, these limitations could possibly be made transparent to the Ruby/Rails developer.

As the service is limited beta, and I know of no one at this stage with access, it’s a wait and see game.

Knol Forum

2007-12-16T00:00:00+13:00

Google has announced Knol, which has been billed by some as a competitor to wikipedia.

Ironically, the best article about Knol is at Wikipedia with over 18 references.

As for the title of the post, Knol Forum, I plan to start a site dedicated to Knol, including at forum at www.knolcentral.com coming soon ( not live yet ).

Will woosh get their act together in 2008? Or will they still suck?

2007-12-10T00:00:00+13:00

Woosh has been around for a number of years now, but the issues for customers continue. paradoxsm’s account was cut with no warning. Paradoxsm’s account on woosh was “woosh” and he claims to have received many emails from unhappy customers claiming woosh does indeed suck.

The best part is that my username was “woosh”. I should reply to all those customers that emailed me showing their disgust for the company and say yes, woosh does suck..

Paradoxsm is not alone with his view about woosh, as many users at geekzone are reporting slower-than-dialup speeds

Even worse than just bad internet service, when canceling their account, some woosh customers are reporting that
Woosh wants to charge them a $99 disconnection fee even though they claim they were not on a contract.

For more information and opinions about Woosh, check out Woosh Sucks #1 and Woosh Sucks #2

Until next time..

Xen on Ubuntu Gutsy

2007-12-06T00:00:00+13:00

Xen is awesome, and so is Ubuntu, but there are some bugs in the ubuntu xen packages. However, these have easy fixes and you can be up and running on xen in a matter of minutes.

This is a quick walkthrough of getting it working with gutsy as Dom0 ( host ) and gutsy as the DomUs ( virtual machines ), however it’s not a complete how-to. If you are unsure about an aspect ( e.g. LVM or loopback) google it.

Load up a computer with ubuntu gutsy server. If you have hyper-threading on your CPU, make sure it is disabled in the BIOS.

apt-get install ubuntu-xen-server libc6-xen

This will install all packages needed to run xen. Then reboot to make sure the xen kernel works on your hardware.

Edit /etc/xen/xend-config.sxp and comment out
(network-script network-dummy)
then uncomment
(vif-script vif-bridge)

That will get your network bridge going after you restart xend with /etc/init.d/xend restart

Now for xen-tools. Xen tools are a set of scripts that can create virtual images by debootstraping debian/ubuntu.
Edit /etc/xen-tools/xen-tools.conf and fill in your settings – select gutsy for your distro. The main one is LVM or loopback images for the virtual disks. I use LVM and it’s great, but there is a learning curve, if you’re just testing loopback images will be easier.

ubuntu-xen Bug #1 : The console doesn’t work.
Fix: add
extra = ' TERM=xterm xencons=tty console=tty1'
to the end of /etc/xen-tools/xm.tmpl

ubuntu-xen Bug #2 : The xen instance hangs on boot, caused by the virtual machine trying to set the hardware clock.
Fix: edit /usr/lib/xen-tools/gutsy.d/15-disable-hwclock
underneath the line
chroot ${prefix} /usr/sbin/update-rc.d -f hwclock.sh remove
add
chroot ${prefix} /usr/sbin/update-rc.d -f hwclockfirst.sh remove
rm -f ${prefix}/etc/init.d/hwclock.sh ${prefix}/etc/init.d/hwclockfirst.sh ${prefix}/etc/udev/rules.d/85-hwclock.rules

ubuntu-xen Bug #3 : DomU networking fails to work after a reboot. ( This bug doesn’t always happen, it may only happen when you have multiple ethernet cards )
Fix : Always give a mac address when creating a virtual machine. Xen has it’s own mac address prefix assigned, so make mac addresses start with 00:16:3E e.g. 00:16:3E:11:12:22

Now you can create your virtual machines :
xen-create-image --ip=10.1.1.5 --hostname=myvm --mac=00:16:3E:11:12:22

and once it’s made, you can start it with xm create /etc/xen/myvm.cfg and access it’s console with xm console myvm

Hope this helps.

The music revolution

2007-10-09T00:00:00+13:00

The shit has really hit the fan for record companies. World class bands such as Nine in Nails and Radiohead announcing, with joy, that they have left their record companies. Radiohead are releasing their latest album online, and the price? Whatever you want to pay for it. Nine in nails have gone a step further and released the source files for tracks of their latest album, so you can remix them if you want.

NZ Ruby on Rails forum and portal launched

2007-09-25T00:00:00+12:00

New Zealand hasn’t had a Ruby on Rails forum or any kind of community site, until now!

May I present www.RoR.co.nz, a rails site, with a forum powered by beast – great open source ROR forum software.

The site is small and simple now, but will grow with the NZ rails community. Check it out here

GPL to be legally tested in New York

2007-09-21T00:00:00+12:00

Yet another case to watch

For the first time in the U.S., a company and software vendor, Monsoon Multimedia, is being taken to court for a GPL violation.
….cont…..
The developers of BusyBox came to the SFLC after trying to talk Monsoon into honoring the conditions of the GPLv2. Unsuccessful with this, the SFLC has filed suit on the developers’ behalf against Monsoon.

It seems like a classic case of GPL violation. The great thing about the GPL licence is trying to get it declared invalid doesn’t help someone breaking it, as the GPL is the only thing that gave them any rights to distribute the software in the first place.

update The case is to settle out of court

Ruby on Rails IDE? Yes!

2007-09-20T00:00:00+12:00

The question is often asked, does ruby on rails have an IDE?

Until now, the answer was ‘kind of’, with radrails.

Now comes 3rdRail from Borland , yet another large company jumping on the rails train. It’s not a free product, but has a 30 day free trial. Best of all, the IDE is cross-platform for Linux, OSX and Windows.

There is a video tour of the 3rdRail RoR IDE here