Codelog

I'm addicted to Wholesale Meat

2011-11-27T22:58:00.000+10:30

If there's one thing I like; it's meat. If there's another I like: bulk. Gawler River Cattle Co seems to provide both, happily.

That is all

I need a new lawnmower

2011-11-20T22:54:00.000+10:30

Right now, I have an out of control back yard and a dire need for a lawnmower. What kind of thing do people suggest?

Why is Gwibber so slow?

2011-11-13T22:42:00.001+10:30

Gwibber seems horribly slow and CPU intensive. From what I've read, there is much finger pointing and little actual solution - just vague hand waving about desktopcouch being a terrible backend.

Development seems dead in the water. How have we ended up at this point? Why don't I have ways to integrate TweetDeck or any other client as well? Why does gwibber-service need to run all of the time?

Managing multiple job configurations for Jenkins

2011-11-13T15:10:00.001+10:30

If you are in the same boat as I am, you find you have too many packages to look after with Jenkins.

The beauty of Jenkins is the simplicity at setting up a job with the web frontend - but once you get over a certain level of complexity this is actually one of the bigger drawbacks.

Sure, we've got some templates, but how far can you really stretch it?

In my situation, I need to:

Trawl SVN/other version control for all packages available - several hundred
Only if the package has tests, add an entry to the CI suite
Adapt to packages which require E_ALL & ~E_STRICT to run happily under that
Packages which require dependencies, but can't be installed, still need a mechanism to install said dependencies
And some which need to be invoked with the legacy AllTests.php
Detect when a package has migrated to github
... and update an existing build/job with a new tool when required

I had tackled part 1 with pear's "packages-all" SVN link, which pointed to the trunk branches of all relevant code, and written some scripts for cruisecontrol to find all directories with a /tests/, but I find myself in need of something more.

So, my code is on github for now, and you can see the current CI system where those scripts have installed new jobs.

I'm quite sure that pyrus and a local installation will deal with the dependencies; as they are all described with PEAR's package.xml format. Also; detecting when a package has shifted to github should be fairly easy to tackle, as there is much work underway to deal with migration.

The one area I need to explore is manipulating jenkins jobs via xpath, to understand what parts of a job are already present and what need updating - basically number seven in the above list.

I'm curious who's done this sort of thing before, regardless of language, and if there are any libraries which make it easier to do this sort of thing.

Ausgrid data sets

2011-10-31T13:35:00.001+10:30

AusGrid are publishing data on how much people use; plus much more about their network. It's a pity it's NSW only; but if you were wondering how green your LGA was; this is your answer.

Obviously; a heatmap would make a neat visualisation - but what else could you do with this data?

Goodreads, Freebase

2011-04-30T17:15:00.003+09:30

Goodreads provides an API; and has lots of data about books, editions, reviews and authors, Freebase has quite a lot of data about the same individuals.

It would be good to reconcile all of the Goodreads authors with freebase/dbpedia entries; or to populate freebase with links to goodreads reviews by ISBN.

On being busy

2011-04-05T21:20:00.002+09:30

Work is busy right now; and I think for the next few years it will be too.

XML_GRDDL, BestBuy & Good Relations

2011-03-19T13:48:00.004+10:30

Digg used to publish rdfa, but it appears to have given it the boot.

So who is out there publishing useful rdfa? Best Buy of course.

While they appear to have sold out of their example rdfa product you can still get a heck of a lot of data out about the store itself.

The code:


$url = 'http://stores.bestbuy.com/577/fairless-hills-pa/products/open-box/frigidaire-30-freestanding-range/0012505540066/?uid=118';

$options = XML_GRDDL::getDefaultOptions();
$options['log'] = Log::singleton('console');
$grddl = XML_GRDDL::factory('xsl', $options);

$data = $grddl->fetch($url);

$data = $grddl->appendProfiles($data, array('http://ns.inria.fr/grddl/rdfa/'));

$stylesheets = $grddl->inspect($data, $url);

$rdfXml = array();
foreach ($stylesheets as $stylesheet) {
   $rdfXml[] = $grddl->transform($stylesheet, $data);
}

$result = array_reduce($rdfXml, array($grddl, 'merge'));

print $result;

The result? 80 or so triples come out describing everything from the facebook account of the store; the geolocation; the address; the telephone; their email; their opening hours and more.

Give it a go yourself:


$ pear install -f XML_GRDDL
$ cd /usr/share/php/doc/XML_GRDDL/docs
$ php bestbuy-rdfa.php | less

Rabbit VCS

2011-03-18T18:44:00.001+10:30

James points me to Rabbit VCS.

"total clone of tortoise... I'm so happy right now."

XBMC vs Boxee vs file browsers

2011-01-09T22:14:00.003+10:30

I ditched Boxee the other day. I was sick of it eating far too many resources, never quite keeping up with my mouse, fighting me with strange user interface metaphors; ignoring some files it would never recognize and providing me with not much enjoyment.

I installed XMBC and got rid of boxee. That was before I realized how good I had it: Boxee's scraper doesn't work off regexp and demand you do crazy things to your filesystem.

What I am struggling to understand is why I can't simply have a damned metadata layer and scraper for my desktop which plugs easily into the file browser.

All that both Boxee and XBMC are really doing is scrape, fetch, and organise. XBMC does it better with a few widgets like "recently discovered espisodes"; but even then; how hard is it?

I advocate the semantic web stack just because this is the very problem those tools should be able to solve in a trivial fashion; but I don't understand why this has not been achieved by anyone else in the last decade.

It's not like web scraping is mind bogglingly hard, nor matching little bits of string against certain sources. Musicbrainz solved the problem for mp3s some time ago: if the same can't be done for video content, I would be surprised. Additionally, it's not like relational databases are a new thing: they have been around for a bit, and that's all you really need: schema, inserts, and the web scraper organising it all.

So it really, really pains me when the Boxee/XBMC/Gnome world can't get it right. Why does Gnome/Gnautilus still treat media basically like this:

Whilst your original design might have been GUI filesystem explorer, what I actually want from you is GUI-like-XBMC-Metadata-displayer.

Dog Day Afternoon

2010-12-08T21:27:00.003+10:30

What do you get if you cross a small horse, a sound a cow makes, and showing off to your neighbours?

I like to think of it as the all animal variety hour. It all began at the behest of my vegetarian girlfriend, who insisted upon a pasta bake which didn't harm an animal. Agreeing, I ~~helped~~ watched her make tea, then decided quite unexpectedly to have an idea.

"Dear," I said. "Why don't we walk Gus, the giant horse dog?". I paused. She paused. I belieive the air was sniffed, in search of the scent of treachery. However, seeing no major obstacles nor any way this could ruin tea, my partner agreed.
We fetched the leash, the giant dog, one or two beers for the journey, and set off to explore Rosewater.

Shortly after walking a mere 25m or so; I decided that frightening my neighbours with a small horse was not sufficient. How would I ever make friends with them? Would anyone at work believe my tall tails (cough)?

With this thought in mind, I set off to visit my neighbour and workmate, Brook. By set off, I don't mean to imply that I really had much to do with the direction we took: one Mr. Gus, Bullmastiff had more to do with that.

Luckily Gus may be as unfit as I am, and quickly tired around the location of Brook's house. I knocked on the door, surprised her partner; and proceeded to hold an impromtu dog and pony show. With only one animal. Playing both the part of dog, and pony.
After a short period, I realised that there was to be a mutual display of newfound pet ownership: Dex the Devon Rex Kitten was present.
I allowed him to claw me for a while, remarked on his corpselike nature, and scurried off into the uneventful night, Christina in tow.

Uneventful, did I say? On what appeared to be a quiet rosewater street, we decided to let Gus off the leash for a bit of Run-Between-One-Person-Pretending-To-Have-Food-And-The-Other. About three rounds in, dog still not tired, it all went wrong.
At this point, you see, a second dog joined the party. No collar, no owners, a black labradour who was in LOVE.

If I were a building...

2010-11-19T12:59:00.002+10:30

.... I think I would quite enjoy it.

Every time people got into my lifts, I would quitely think NOM NOM NOM. Whenever I got hot, I would simply turn on my air conditioner. I would stand and glint as the sun sets over me; quietly resplendent in steel and glass.

If I were a building, I could stand free and tall.

If.

Transport

2010-11-12T13:52:00.001+10:30

Boston's real-time transit data: "Better than winning the World Series" vs Adelaide.

Depressing.

Rapportive

2010-11-08T09:59:00.002+10:30

Rapportive is a plugin for gmail, which trawls the social graph.

I would class it as one of the few applications which makes use of foaf (perhaps more the ideals than the direct execution); but it lets you view someone's connectivity to you on the social web.

LIXI forum 2010

2010-11-05T07:10:00.004+10:30

Over the past two days, I have been lucky enough to attend the 2010 LIXI forum.

LIXI is a set of standards relating to how financial institutions can deal with loan applications, valuations, and a few other items.
Typically it's serialized into an XML version of a paper form; and gets sent around by SOAP to various workflow engines.

Previously, my involvement with LIXI has been in the valuations space, with ValEx being a big driver of LIXI usage; but no one really talking about the standards (or asking for improvements).

So, it was much to my surprise when I got to meet much of the industry over the last two days.

Probably one of the most interesting ideas put forward was by Christopher Joye. He proposed a national clearing house for data relating to credit:

Moving from the general to the specific, I would propose that the Commonwealth establish a central electronic ‘clearinghouse’ of all residential, personal and business credit originated in Australia. For simplicity’s sake, let’s call it the National Electronic Credit Register (NECR).

The main purpose of this information would be to provide much more accurate, real time, granular data to understand how individuals are actually fairing, rather than trying to guess at an aggregate level.
This is primarily to allow entities like the RBA to act quickly and accurately to help manage our economy.

I personally think it is a quite valid idea, as much of my frustration with how statistics and modeling takes place stems from the use of assumptions, or aggregate data.

There are obviously obstacles, not least of which is the potential for scope creep and how that relates to individual's privacy; but as a concept, it quite impressed me.

Other presenters touched on everything from the cost of software failure to a comparison of Australia's railway gauge mess to the current fragmentation of the LIXI standards.

The big move at the moment seems to be to standardize on a mechanism to transmit business rules.
Three vendors all have the same problem of trying to communicate changing credit policy for specific products from a lender to many brokers; and there are a number of solutions out there on the cusp of realization.

We saw practical demonstration of real time transmission of changed requirements (application A hits the save button, changing the age limit on a product, application B becomes aware of it within seconds).

Whilst people did not get into the technical details behind that particular demo, a technical training session on schematron was run.
I had initially dismissed the idea of schematron, but hearing Rick Jelliffe (one of the creators of XSD) talk about many of the concepts that drove the design; I'm more sold on it.

If you haven't used schematron, in short it differs from XSD by being much more relaxed. You write specific xpath rules to target segments of a document as being valid.
XSD on the other hand ends up being much more restrictive. For instance, it forces you to make choices about sequence of nodes and tends to fail validation if you have extra content.

The extra content item is a killer for the LIXI ecosystem, as everyone seems to have practice of grabbing a copy of the XSD at a given point in time; and if need be; doing minor modifications for their patch of it.
It's not on the web, because no one wants to expose a core banking system to the internet.
As an example, I introduced a simple additional Time element to ValEx's schema and the LIXI standards. It has taken months of testing with multiple organisations just to take a tiny step forward, and in the end a partner application still ended up ceasing to communicate with us.
The most painful part of it all? The only bits which "broke" in every single system were the XSD validation components. Every other bit of code simply ignored the additional XML.

With such a minor change requiring so much effort, you can understand how we've ended up in a fragmented state, and why XSD has been harmful.

I would be very keen to see the entire basis of validation (both at a syntax/expected data level and a business rules level) shift to schematron; or something else - much of the RDF world does not have the XSD problem, because no one uses XSD to validate!

Overall, I made quite a few contacts in all sorts of places, touching on topics such as the use of unique identifiers (ala Linked Data, even if we're not an RDF based standard); to improving physical documentation handling with our existing partners.

I look forward to the coming year, and the opportunities it may create.

On automation and ownership

2010-10-26T11:18:00.005+10:30

I'm writing this here rather than getting into an email trail or pissing people off by telling them that I disagree via IRC.

What's the quickest way to annoy an open source community?

Impolite automation.

I was pretty happy when I annoyed enough people that I was granted QA karma. It's not just the technical karma, but the social karma to be a bit bossy and push out releases, fix bugs, scold maintainers.

I have slowly built up a hodge podge of continuous integration tools, scripts and other bits and pieces for the QA work I do in pear.

To date, I've been getting pretty positive feedback - the cruisecontrol instance, pumping out fixes to unit tests, pumping out releases for old packages, etc.

Sick of doing things manually, I scripted up a quick way to scrape a changelog, prepare a release, and hey presto: QA releases just got easier.

What's the problem with that? It turns out if you do screen scraping and sort the results in any particular way; you end up straying from "that package last released way back in 2005" into the newer stuff - some of it maintained.

PEAR also has a problem with its code ownership model - strong code ownership is the order of the day.
This is not just my opinion, but evidenced by the fact we have many packages which are just abandoned (I would guesstimate over 60%). We have maintainers apply on some, but if we don't get around to doing anything; those packages sit there and rot.

SVN karma generally only allows a maintainer to touch one package they specifically apply for.
The process we have (PEPr) is great for newcomers and sorting out the wheat from chaff, but not so great for encouraging members of the community to adopt more code.

So; mix all of that together, throw in a range of developers who work in a range of styles; and what do you get?

Maintainers who feel angry when features & bugfixes they have authored are released to users.

So; first thing's first: Sorry to those who feel angry. I can fix this by communicating better, and tweaking the scraping tools to be more precise; but in all honesty I think I will probably end up causing more upset at a later time by doing something similar.
Why would I do this again, am I insane? I would put it down to my code ownership views vs the status quo.

Secondly: How do we change the culture of PEAR from strong to something between weak and collective ownership of packages?

My answer to this in a work environment has been brutal automation of the trivial. There are no exceptions to rules: only the code is wrong or the automated measure is wrong. The result is the same - it gets fixed.
Make a commit? It works, keep it. It causes regression? Revert or fix.
It's not just the person who authored the change who has that responsibility; it's everyone.

(Ssh, work colleagues: just because I am the most fragrant violator of these rules!)

The net result turns out to be much more maintainable code, and you get to be lazier (let the machines do machine things, I'm here to make elegant code).

My current answer to this for PEAR is to give every darned active maintainer we trust QA karma; then make it so easy to do QA work there is very little excuse not to do it via tools.

That way at least it's not only me pissing everyone off!

Third: Is my zealous view of automation and collective code ownership completely the wrong model for PEAR? Does the community want less trampling over polite expectation by rogue zealots with too much power?

This one I don't have an answer for; and I think I ask the community for a vote (ie: get the annoyed with QA karma, revoke mine if so decided, and get everyone deciding a new direction for QA. Let's not forget the most important step of doing it either!)

If you're looking into ways of improving the quality & completing rates of your company's employees try finding solutions with performance appraisal tools from Halogen

Queries

2010-10-01T13:31:00.002+09:30

I ran across a thread related to queries, CBA and ValEx.

As someone who is building this particular functionality, it's interesting to see what effect it has on the average person.

PEAR 1.9.0 won't upgrade on windows

2010-09-30T08:00:00.000+09:30

(9:45:34 PM) lera: Hello, guys! Can anyone help me with pear installer 1.9.1 installation?
(9:47:02 PM) cweiske: just ask your question
(9:47:31 PM) lera: I currently have pear installer of 1.9.0 version, but I need 1.9.1 to upgrade my phpunit to 3.5, so I'm trying 'pear upgrade --force pear' command, but pear keeps saying that its version remains 1.9.0 (to figure out version I use 'pear version' command)
(9:48:11 PM) lera: Though, installation seems to pass successfully
(9:48:44 PM) lera: I use php 5.3.3 zip package and 1.9.0 was installed using go-pear
(9:50:33 PM) lera: Any ideas?
(9:54:30 PM) cweiske: please pastebin the output of "pear upgrade pear"
(9:55:24 PM) lera: Here is it: http://pastebin.com/f6Mhiq1f
(9:56:37 PM) cweiske: pear info pear
(9:56:53 PM) cweiske: what do you get?
(9:59:17 PM) lera: yes, here: http://pastebin.com/PeFdeFxQ
(10:00:35 PM) cweiske: ABOUT PEAR.PHP.NET/PEAR-1.9.1
(10:00:45 PM) cweiske: so we have a difference here
(10:01:13 PM) lera: so, what can I do about it?
(10:01:20 PM) cweiske: which path has the pear executable that you run on command line?
(10:01:59 PM) cweiske: I suppose it is different to the one you get in "pear list-files pear", the "script" one
(10:02:15 PM) lera: C:\Program Files\PHP\pear.bat
(10:03:12 PM) lera: yes, that's true, they are different
(10:03:24 PM) cweiske: then there is your problem
(10:03:30 PM) cweiske: remote the one in PHP
(10:03:33 PM) cweiske: and use the other one
(10:05:33 PM) lera: but with that pear, whose path is equal to that mentioned in pear list-files pear, there is just the same problem - I can't update it either
(10:05:50 PM) cweiske: didn't you say it's different?
(10:05:55 PM) cweiske: what is the 2nd path?
(10:06:00 PM) cweiske: the one from pear list-files pear?
(10:06:28 PM) lera: yes, exactly
(10:07:15 PM) lera: I have PEAR here: C:\Program Files\PHP\pear.bat C:\Program Files\PHP\pear.bat
(10:07:18 PM) lera: And here: D:\PHP\php-5.3.3\pear.bat
(10:07:33 PM) lera: And pear list-files pear points to D:\PHP\php-5.3.3
(10:07:47 PM) cweiske: what do you get if you execute the D one?
(10:07:52 PM) cweiske: (as version)
(10:07:54 PM) lera: just the same
(10:08:07 PM) cweiske: how do you execute it?
(10:08:11 PM) cweiske: the full command line
(10:08:38 PM) lera: I've tried pear upgrade --force pear is went fine
(10:08:50 PM) lera: then i use pear version and it gives 1.9.0
(10:09:06 PM) cweiske: please post the FULL command line how you execute the pear.bat in D
(10:11:16 PM) lera: D:\PHP\php-5.3.3>pear upgrade --force pear
(10:11:31 PM) lera: this is the full command line
(10:12:23 PM) lera: Here is it as well... http://pastebin.com/DqTFcPR2
(10:12:23 PM) PEARgirl has changed the topic to: http://pear.php.net/ | just ask your question | please use a pastebin for long lines of text, eg pastebin.com | 604 open bugs (1.08 bugs/package)
(10:14:34 PM) lera: Ok, the problem is that two pears got mixed up, right?
(10:14:41 PM) cweiske: I suppose so
(10:15:31 PM) cweiske: please pastebin the pear list-files pear from D
(10:16:20 PM) lera: Yes, please: http://pastebin.com/Nx0fTPYC
(10:17:21 PM) cweiske: please pastebin the D pear.bat
(10:18:20 PM) lera: here you are: http://pastebin.com/c3NBda5w
(10:21:15 PM) cweiske: it might be that you have some environment variable set, PHP_PEAR_INSTALL_DIR
(10:21:29 PM) cweiske: which points to the old version
(10:21:45 PM) cweiske: the most easy way to "fix it" is to run php scripts/pearcms.php info
(10:21:52 PM) cweiske: instead of using pear.bat
(10:21:58 PM) cweiske: try that and see if you get 1.9.1
(10:23:30 PM) lera: PHP_PEAR_INSTALL_DIR is set to D:\PHP\php-5.3.3\pear
(10:24:02 PM) lera: Where can I find pearcms.php? There is no such file under php root
(10:24:08 PM) cweiske: pearcmd.php
(10:24:10 PM) cweiske: scripts
(10:24:21 PM) cweiske: it should be in the pear list-files pear output
(10:25:12 PM) lera: oh, right
(10:29:26 PM) lera: that's great! It works! Thank you so much!!!
(10:29:40 PM) lera: Sorry, for taking so much of your time!
(10:30:00 PM) cweiske: good to see

Tomboy notes

2010-07-17T13:50:00.002+09:30

I wish that I had a tomboy notes plugin which was like Zemanta; or could otherwise plug in freebase data.

DotNetNuke, RDFa and GoodRelations via NB_Store

2010-07-01T16:51:00.003+09:30

Martin Hepp says:

The latest release of the NB_Store module for e-commerce sites based on DotNetNuke CMS seems to support GoodRelations in RDFa.

See here for details:

http://nbstore.codeplex.com/releases/view/45017

Local Government Areas and IDs

2010-06-30T15:11:00.001+09:30

The ABS have made their data creative commons; but it's still a pain to get to the data you want.

I'm trying to reconcile LGAs against our work application, freebase, and the census data - but it's a pain.

To make things a bit easier for others treading this path, here's a spreadsheet of LGAs and ABS ids for them, 2006

Most Ridiculous Design Ever

2010-06-28T08:21:00.002+09:30

At work, we've got roaming profiles. I've just spent 20 minutes logging on, because I have a 7-8gb shared mailbox in my outlook.ost; and that makes things unhappy.

What beggars belief is how you move your outlook.ost.

Call me stupid; but I rather thought we'd mastered the basics of a "move" file within our operating systems.
Apparently, the outlook 2007 team has some reason to make that impossible to do - the user interface fights you every step of the way.

Unless you disable a tonne of options which should have no bearing on your ability to move, copy, or change the location of your offline mailboxes; Outlook will not let you touch this setting.

Ridiculous.

Home automation and eating

2010-06-26T18:46:00.003+09:30

I want a Unitech HT580 Wireless Barcode Scanner; but cheaper.

I want to holster the thing on the side of my cupboard or fridge; so when I'm loading up the fridge I can quickly tag barcoded products.

Most importantly; I want to be able to sync that with my computer to make an instant shopping list.

Failing all of this; I'll just have an internet fridge please

I'm a research subject

2010-06-11T11:17:00.001+09:30

Check out The Anatomy of a Large-Scale Human Computation Engine, figure 4.

This application alone received more than 1 million contributions over 14 months with the top contributor casting more than 120k judgments and the top 10 contributors casting half of the total judgments.

Beats the heck out of medical research!

Playdeb, and ubuntu wishlist items

2010-05-16T17:47:00.003+09:30

I'm a huge fan of playdeb. The fact there is an apt URI handler available in ubuntu makes everything trivial - and I'd encourage lots of existing games to simply link to playdeb for a stable version.

What does annoy me though - because software installation in ubuntu is nearly flawless as it is - is that I can't have two package managers running at the same time.

Why not? Why can't I queue up several things to install? I can do that in windows; why do I have to wait around and install things one at a time?

If another package manager is working; why not prompt me with a question; rather than an error? "Another package manager is working, would you like to queue this installation task until after it has finished?"

There's some inklings about this, but most miss the point - Bug #74134 wants to make the failure more user friendly; but why should it fail at all?

I've opened a blueprint for it; which is probably a dupe; but I couldn't find anything specifically.

I imagine this to roughly look like the below for apt (if apt were magically written in PHP):


$stdin = fopen("php://stdin");
$path = ...; // /path/to/apt/lock/file/majigger
if (!is_writable($path)) {
   print "Another package manager appears to be working; would you like to queue this installation? [Y|n]";
   $ans = fgets($stdin);
   if ($ans === "n") {
       exit(0);
   }

   while (!is_writable($path)) {
       sleep(1);
   }
}
install($package);

Perhaps you'd want to cover a few more edge cases, or make it unit testable, but you get the idea. It doesn't seem to hard either for others, like update-manager or synaptic - probably beyond papercut size though.