codesimple

That Big Shop

2009-09-22T17:51:00.003+01:00

I've always found shopping at Amazon a bit of a cumbersome experience. The pages seem a little cluttered and when choosing between products there's lots of back-and-forth as you switch between search results and product detail.

So I thought I'd make use of the Amazon Product Advertising API and write my own interface. The result is That Big Shop. It's not old browser or search engine friendly but it does, I think, have a more responsive feel to it than the standard Amazon interface.

To create the site, I choose to use the Cappuccino framework and I'm glad I did. The painful part of writing web applications is getting them to look and work the same across browsers. With Cappuccino I didn't need to worry about that; as long as I coded to the framework API, Cappuccino would take care of rendering what I wanted consistently across browsers.

QR codes for cogteeth.com

2008-08-12T12:36:00.001+01:00

With a little help from the Google Chart API, I've added the ability to create QR code t-shirt, mugs and cards over at cogteeth.com.

QR codes are quite nifty little two-dimensional barcodes which can store a surpising amount of data in a small space. There are numerous readers available to decode them including mobile phone apps which enable you to decode them using your phone camera.

Any data can be stored in these codes but there are a number of standards or conventions recognised by the readers. For example, many will detect an encoded URL and automatically pull up the referenced web page. The documentation for the zxing reader gives a good summary of these conventions.

Google App Engine

2008-04-15T17:12:00.003+01:00

It took less than two hours to convert cogteeth.com to run with the Google App Engine. Admittedly this is a pretty simple example since cogteeth doesn't use a database and Django is supplied with the App Engine environment.

All that was necessary was:

Create a main.py: Since the app is Django based this was just a modified copy of the version on this page.
Create an app.yaml to pass all URLs onto main.py and serve the static files directly.
Modify some URL fetching code that previously used urllib2 to use urlfetch.

When I say it's running with the App Engine that's not strictly true. It runs locally with the SDK, now I just have to hope they work through the wait list quickly and send me an invite...

Blogger custom domains

2007-10-11T17:34:00.000+01:00

This blog is currently hosted at http://www.codesimple.net. This is a pity since the more succinct http://codesimple.net would be preferable. Unfortunately if I try and set the custom domain in Blogger to just "codesimple.net" it complains with a somewhat dubious "Another blog is already hosted at this address" message.

I think perhaps this is related to also having Google Apps active on this domain. Both services require you to point your DNS at ghs.google.com.

Also slightly odd is that Google do an HTTP redirect for any requests to http://codesimple.net to http://www.codesimple.net. I can't find any setting in Google Apps to control this "feature".

WordPress to Blogger

2007-10-11T16:25:00.000+01:00

I recently decided to switch my blog from Wordpress to Blogger. Not because I think Blogger is better but because I wanted to avoid maintaining my own instance of Wordpress and Blogger would host my blog on my domain for free.

To copy the posts across I wrote a small Python script which takes a file of exported Wordpress posts and uses the Blogger Data API to upload them to Blogger.

If you want to do this yourself, you will need to:

Download the script wptoblogger.py.
Ensure you have Python 2.5, BeautifulSoup and Google's GData Python Client.
Create your target Blogger blog and note its ID.
Export your Wordpress blog to an XML file (I think this is only available in version 2 and above of Wordpress).
Run wptoblogger:
python wptoblogger.py -u your_blogger_username -b your_blog_id -a author_for_posts wordpress_xml_file

There are a few limitations though...

It doesn't support multiple blog authors - all posts will be marked with the author_for_posts.
A limitation with the Blogger API means that comment authors can't be set (they always appear as the blog author). As a workaround, I prefix each comment with a "Comment from..." line (see the existing comments on this blog).
Blogger impose a limit on the number of posts you can create via the API in a certain time period. So if you have a lot of posts to transfer you may need to modify the script to post them in batches over a number of days.

Join and split

2007-06-21T11:39:00.004+01:00

It always amazes me when I try out a new programming language and find that the standard library doesn't include join and split functions. join is used to combine a list of strings into a single string with each item separated by a delimiter. split takes a delimited string and divides it into sub-strings. Python, thankfully, does provide these functions:

>>> ':'.join(['a','b','c'])
'a:b:c'
>>> 'a:b:c'.split(':')
['a', 'b', 'c']

Erlang provides regexp:split but it looks like I have to add my own join function. Not sure this is the most efficient implementation but it will do the job:

intersperse(_, []) -> [];
intersperse(Element, List) ->
tl(lists:reverse(lists:foldl(fun(X, A) -> [X,Element|A] end, [], List))).

join(List, ListOfLists) -> lists:append(intersperse(List, ListOfLists)).

> intersperse(":", ["a","b","c"]).
["a",":","b",":","c"]
> join(":", ["a","b","c"]).
"a:b:c"

And since strings in Erlang are just lists of integers this can be applied to other lists as well.

Flickr bookmarklet

2006-11-16T18:54:00.004Z

I've just added Flickr support to the Cogteeth Bookmarklet. Obviously it worked with Flickr before (since it works with most web pages) but now it will automatically detect Flickr images and provide the following additional features:

It will retrieve the largest available version of the selected image from Flickr to ensure the best quality product is produced.

Make the image clickable so that the user can easily go to the Flickr photo page.

Provide links to any other available sizes of the image.

Show the license that the image poster has attached to it.

Create products from worldinpictures.org

2006-10-17T13:15:00.005+01:00

Integration of Cogteeth.com and worldinpictures.org now means that you can create Zazzle products from any images you see on worldinpictures.org. Just click the "Make product..." links.

Cogteeth Bookmarklet

2006-10-17T09:54:00.005+01:00

Just added a new feature at Cogteeth: the Cogteeth Bookmarklet. Once installed, this allows you to create a Zazzle product from any image on the web.

Probably half the development time on this was spent getting it to work with Internet Explorer. In the end I had to produce a slightly different version specifically for IE which may not look so pretty when used on some sites. But of course there's an easy solution to this - switch to a decent browser.

Cogteeth decoder added

2006-10-10T17:37:00.005+01:00

I've added a message decoder to Cogteeth.com so if you don't want to buy a t-shirt you can use it as a general UTF-8 coder/decoder!

Cogteeth.com - personalized T-Shirts

2006-10-06T11:09:00.005+01:00

Partly as an experiment with the new Zazzle API, I've created a new site Cogteeth.com.

At Cogteeth you can create customized T-Shirts by entering messages for the back and front of the shirt. The text of the messages is converted to UTF-8 and the resulting codes displayed in hex. The result is a T-Shirt with a coded mesage that only those with too much time on their hands can read.

Here's what I've learnt from this exercise:

Passing an empty field to http://www.zazzle.com/api/create is the same as not including it in the request (i.e. the text in the template will be used). So If you want to leave a text field blank, you must pass in a SPACE (e.g. http://www.zazzle.com/api/create?frontText=%20&blah=...).

The Zazzle API does not yet support non-ASCII characters. Passing a UTF-8 encoded character sequence (e.g. http://www.zazzle.com/api/create?frontText=%C2%B7) will NOT result in the UTF-8 C2:B7 character. Zazzle say this may be supported in future.

Javascript uses UTF-16 strings, so if you want to use Javascript to convert form field values to UTF-8 (and ensure Unicode characters above 0x10000 are handled correctly) you must first decode from UTF-16. Javascript appears to have no built in support for this kind of thing so I ended up writing my own functions to decode from UTF-16 and encode to UTF-8.

Although the HTML 4 spec suggests that only ASCII values can be used in HTTP GET form submissions, most browsers will sensibly encode non-ASCII characters in UTF-8 and then URL encode them before submission. (This assumes the page containing the form was sent as UTF-8 encoded).

Why isn't there an "onFieldValueChange" event for form fields? I currently use the "keyup" event but I'm not interested whether the change was due to a keystroke, I just want to know if a change occured. The onChange event is no use here - it only fires when the field loses focus.

54:68:61:6E:6B:73
20:66:6F:72:20:72
65:61:64:69:6E:67
2C:20:6E:6F:77:20
67:6F:20:62:75:79
20:61:20:54:2D:53
68:69:72:74:21:00

UK place name support for worldinpictures.org

2006-09-26T09:32:00.005+01:00

Thanks to the excellent Geo-Names webservices you can now enter UK place names at worldinpictures.org to see photos from that location.

Previously I'd been relying on the Google geocoder which doesn't (yet) support the UK or a number of other countries, so photos from unsuppoted countries could only be viewed by entering a latitude/longitude (or post code for the UK). Now Geo-Names fills in the gaps so many more locations are supported.

Geocoded YouTube videos

2006-09-08T15:54:00.005+01:00

I had a look at adding videos to worldinpictures.org this morning. YouTube looked like a good source for these and offers a limited API to get hold of them.

Unfortunately the grand total of geotagged videos at present on YouTube is... 38. Admitedly it wouldn't make a lot of sense to geocode many of the videos on there, but I thought there might be a few more.

Retrieving officially geocoded flickr photos

2006-09-07T18:48:00.005+01:00

I've been updating worldinpictures.org to use the new flickr geocoding API. Unfortunately it doesn't provide a "find all photos near to a point ordered by distance" function, so I still need to keep a database of photo locations.

To keep my database up-to-date, I now to need to find the latest photos that have been "officially" geocoded, as well as those using the old unofficial geo:lon/geo:lat tags.

Turns out you can retrieve just the flickr photos that have been "officially" geocoded by setting the bbox parameter to cover the whole globe:

flickr.photos.search(bbox=-180,-90,180,90)

This is different to leaving out the bbox parameter (which would return all photos). Instead it essentially applies a filter which returns only those photos with official geocode information.

Firefox trounces IE

2006-09-06T10:27:00.005+01:00

worldinpictures.org made it on to the front page of digg yesterday and as a result got rather popular. My poor server was stuggling in the morning so I invested in an extra 128Mb. The nice guys at memset got this upgraded promptly and it survived the rest of the day admirably.

Looking at the stats, it's interesting to see which browsers visitors were using:

Firefox: 66%

Internet Explorer: 21%

Safari: 9%

Opera: 2%

This is probably not representative of general users since digg visitors are probably more technically savvy than most but encouraging nevertheless.

Google maps tabbed windows

2006-09-04T11:06:00.005+01:00

For the new map view at worldinpictures.org, I needed to support the situation where the search results contain several images taken at the same point. The obvious soltuion was to take advantage of the tabbed info windows provided by Google Maps ("info windows" are the pop-ups that display when you click on a map marker).

Unfortunately the tabs are quite wide so if you want a lot of tabs (and I could need 12) you need to have a very wide info window. So I had to implement my own "tabbing" system using simple numbered links. This seems to work well though and keeps the window to a reasonable size.

Now I need to change the title that appears when you hover over a marker to indicate the number of photos at that point. That can wait for another day though...

Maps added to worldinpictures.org

2006-08-30T17:01:00.005+01:00

I've just added a "map view" to worldinpictures.org. There's now a couple of links below the search results to switch between "gallery view" and "map view". Select "map view" to see the location of the photos on a Google Map. Hover over a marker to see the photo title, click the marker to see the image.

There's a couple of things I need to improve but I thought it was useful enough to release as is. It needs to cope better when several photos have the same location - at present only one of the photos will be available on the map view. I might have a look at using tabbed info windows to cope with this.

Python unicode function weirdness

2006-08-30T11:02:00.005+01:00

Python has a built-in function called unicode which is intended to convert strings to unicode.

When called with only one argument (the string to convert) it will assume the string is encoded in the default encoding. This is normally ASCII but can be overridden in site.py.

I would normally want to write code that would work regardless of the default encoding. Thankfully unicode can take an additional argument to allow you to specify an encoding rather than using the default. Unfortunately, and for no reason I can think of, supplying this argument causes the function to behave differently when given a unicode string as input:

>>> unicode(u"abc")
u'abc'

>>> unicode(u"abc", "ascii")
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: decoding Unicode is not supported

This strikes me as rather bizarre behavior. Surely unicode(s) ought to bahave exactly the same as unicode(s, sys.getdefaultencoding())?

Unexpected ElementTree behavior

2006-08-30T10:45:00.006+01:00

I've been using the Python ElementTree library for parsing web service responses for my worldinpictures.org site and generally found it reliable and easy to use.

Character encoding issues have caused me a number of problems recently and I've come across another one with ElementTree:

>>> from elementtree import ElementTree as ET
>>> ET.XML('<?xml version="1.0" encoding="utf-8" ?><title>Good morning Mazatl\xc3\xa1n!</title>').text
u'Good morning Mazatl\xe1n!'

>>> ET.XML('<?xml version="1.0" encoding="utf-8" ?><title>Good morning Mazatln!</title>').text
'Good morning Mazatln!'

It seems that if the element contains any non-ASCII characters then the result will be a unicode string otherwise it will be a plain string.

It would be preferable to have a consistent return type (e.g. always unicode or always in the input encoding).

So, in my case, I pass the result through unicode() to ensure I always get a unicode result.

(There's an issue here with the unicode function and its reliance on the default encoding but that belongs in another post...)

worldinpictures.org - see more

2006-08-25T10:24:00.006+01:00

I've added nearer and further links to worldinpictures.org, so now you can see more than just the 12 nearest images to your chosen place.

Google maps UTF-8 problem

2006-08-24T12:58:00.006+01:00

A while ago I came across a problem with the google geocoder apparently returning Latin1 encoded characters rather than UTF-8. I posted an enquiry to the Google Maps API group but didn't get any responses.

Now I've had time to look at this in more detail and found how to fix it. From my investigations I found that:

wget, curl and requests made with Python urllib2 all returned responses encoded in Latin1. Requests made with Firefox returned responses encoded in UTF-8.

Regardless of the actual encoding returned, the XML always stated encoding="UTF-8".

The Content-Type header in the HTTP response correctly gave the returned encoding (either UTF-8 or ISO-8859-1).

So it looked like this had something to do with the headers sent in the HTTP request. I used curl to play around with these and see if I could get a UTF-8 response. The obvious ones (e.g. Accept-Charset: utf-8) didn't work. But what did work was changing the User-agent header. So, if you want to ensure you get a UTF-8 response, pretend to be Firefox:

curl -H'User-Agent: Mozilla/5.0' 'http://maps.google.com/maps/geo?key=&q=cologne&output=xml'

All this means that you can now search for cologne on worldinpictures.org and it will display Köln rather than K�ln.

Ubuntu xserver problem

2006-08-23T14:05:00.006+01:00

If you find X suddenly stopped working on Ubuntu in the last few days you may have the same problem I had. Yesterday a routine update installed xserver-xorg-core 1:1.0.2-0unbuntu10.3. Turns out this "breaks PCI setup for many users", so the solution is to do another update and get xserver-xorg-core 1:1.0.2-0unbuntu10.4 which reverts the change.

Unfortunately I spent several hours trying to fix the problem before doing the sensible thing and checking if an new update was available.

Note to self: always check for an update first.

worldinpictures gets UK postcodes

2006-08-22T10:09:00.012+01:00

With the help of the geocoder web service available at worldkit, worldinpictures.org now understands UK postcodes.

The geocoder only gives accuracy down to the first part of the postcode but I allow searches on full ("SW1A 0AA") or partial codes ("SW1A").

Unfortunately the geocoder doesn't give results for all postcodes - "N1" for example doesn't get resolved. Not sure if this is due to the worldkit web service or the source data it uses (which comes from jibble.org).

Still this makes searching within the UK that bit easier - at least until the google geocoder supports it...

Detabifying

2006-08-17T09:56:00.006+01:00

If that was a word I'm not sure that would be the way to spell it. I'm referring to the process of replacing tabs by spaces.

The unix command expand does this job - taking a file and replacing tabs with a specified number of spaces.

What's the best way to run this command over a number files? find is likely to be useful here to return the files to process. In my case these are all .php and .css files in and below the current directory:

find . -name '*.php' -o -name '*.css'

will get me a list of these.

I thought perhaps I could use find's -exec option to run expand on the resulting files. Unfortunately expand will only send its output to stdout and as far as I can see there's no way of specifying that you want the output of an -exec'd command redirected.

However, I can iterate over the results with for and this does the trick (copying the results to /tmp/x in this case):

cp -r . /tmp/x
for f in `find . -name '*.php' -o -name '*.css'`; do expand -i -t4 $f > /tmp/x/$f; done

The initial cp is just a crude way of creating the directory structure in the destintion location to stop expand failing when sub-directories don't exist.

worldinpictures.org to do list

2006-08-15T16:37:00.006+01:00

There's plenty of improvements I'd like to make to worldinpictures.org in the (hopefully) near future. These include, in no particular order:

Improve geocoding: The google service I currently use doesn't support all countries. Specifically it doesn't support the UK, so if you want to view images from the UK you have to find out the latitude and longitude yourself. Not very convenient. I think worldkit.org and/or http://www.jibble.org/ukpostcodes/ may help here.

Add a "more" button to view more than the initial 12 images currently displayed.

Add a feed feature so that users can subscribe to a feed of photos for a particular location.