Abstract Generator Factory

How I got here, part 16 - Job No. 8

2010-08-08T15:06:00.000+01:00

It's time that the tale were told, of Job No. 8, the place I worked at when my ideas of freelancing got tied up in an inevitable knot. My original ideas of a short-term contract didn't go very far, this mostly seemed to be the fault of the financial crisis creating a large group of unemployed contractors, breaking in to that as a newbie didn't offer much promise.

This left me with the choice of a permanent job or nothing at all, and to continue on with Plan A (in whichever iteration it had reached by that stage).

A permanent job wasn't too hard to come by. The first recruitment agency I talked to mentioned a company that "sounds like a good fit" and he soon set me up with an interview later the same week. When I got there I initially met with the development manager, who wouldn't look me in the eye, and the project manager, a quite fierce looking middle aged woman. I really didn't know what to make of them. This was followed by two of the techies coming to talk to me, one of whom described the company was "the most laid back place you'll ever work".

I was called back the very next day for even more interviews. I didn't know if this was a good sign or not, but since I've been there they are quite quick to throw out hopeless candidates so it must have been a good thing. I managed to find out some more useful things, like the financial standing of the company, etc.

It didn't take long for an offer to arrive. Laid back company with enough money to see out the financial crisis and take the world by storm on the other side? Done and done. I accepted and began the following week.

For the first week I still really wasn't sure about it. There was a pretty standard probationary period, so I had my chance to run away. The reasons I wasn't sure was many and varied: The fact that the code base was very, very old; the strangely sarcastic, but very settled, developers; and many, many other reasons.

The first assignment was a deliberate dead-end designed to let new hires get to know the code base. The code base was ancient (well, ancient by Java standards, early 2000's; it wasn't proper ancient), and highly idiosyncratic. After I overcame the shock of some of it, and saw that it was regularly worked on and improved (i.e. not just a static blob, like Job No. 3) I began to see it as a good thing. I'd never worked on a proper legacy code-base before; I'd certainly inherited old applications, but they had to be put out of their misery. Most of the code-bases I'd worked on in the past had had a very short life indeed, if this one had lasted so long, and was still going, then there must be some value in it.

The original developers were long gone; the second generation developers had all left; we were now on the third generation at least. This had positive side effects, it meant there was a healthy distance between the developers and the code; it wasn't "my code" or "your code", it was "the code". This also led to a "no blame" culture for defects; anything found was logged and dealt with completely dispassionately. There was none of the histrionics from previous jobs. Not that there were many defects... there's a certain irony here, companies which are most paranoid about bugs are more likely to have them.

It wasn't all legacy crap though, or I would have gone mad, my first real assignment was on a brand new application to replace part of the legacy application suite. During this time I began to realise I'd made the right choice in joining the company. The fierce project manager I wasn't sure of during the interview turned most of her anger outwards, she kept "the business" in check, if they wanted anything doing they were going to have to go by her rules. This kept all releases consistent, with a managed scope, etc. Luxury!

Shortly afterwards it was my turn to take a project through a testing cycle for the first time. At previous jobs this had always caused pain. The difference at Job No. 8, however, was that they had professional testers (although they would begin to cause problems, but more on that later); so they understood what a release was, the need to log defects correctly (well some of them didn't get that point, but lets not get bogged down in that).

All-in-all they were a rarely competent bunch, of the type I didn't think actually existed. Everyone did what they did, did it quite well (usually), without much in the way of arguing at all.

The first significant project that came my way was a compete ground-up rebuild of the companies main source of revenue. This gave me a ringside seat to see the tensions that were bubbling under the surface. The project itself went well, the result was faster, less buggy, and increased revenue - win/win/win. But "the business" were not happy. Specifically they were annoyed that they didn't know at the start of the project how long it was going to last, that was because the spec they produced consisted of mainly "TBC" in all the sections. The scope was only agreed about half way through development, but despite that they wanted us to have guaranteed a delivery date, then they could squeeze in as much stuff as possible. Needless to say, the technical management were too wise to agree to this and gave it short shrift. "We'll give you a guaranteed delivery date as soon as you give us a guaranteed final spec."

What they actually had, while unpredictable, worked quite well. They had the whole hippie "working side-by-side" nonsense that so many tech journalists long for. And it worked quite well, but no-one appreciated it.

The other crack along the retaining wall of sanity came towards the end of that project. There was an extended period between the end of development and the beginning of testing (I did warn they would get involved in this story), this lead to a lot of hand-wringing amongst management that we needed more testers. This was absurd, as we already had as many testers as we had developers; but it would get more absurd as rumours abounded they were going to cut the development team in half and use the money to hire more testers. The company was small, and while doing quite well it didn't really have any "unique selling points"; it's competitors were larger and better funded, if we didn't innovate faster we'd be dead. But yet serious consideration was given to abandoning new development just so incremental low-risk changes could be tested thoroughly.

They didn't axe the development team, in the end. Although they didn't replace natural wastage either, so the end result was largely the same. Instead management came up with, what they thought was a genius idea: an all-encompassing methodology which would be followed to the letter. Which well-known methodology did this resemble most closely? Why, waterfall of course, the complete opposite of what had worked well for them in the past.

The main tenets of this was that projects would have stringent entry criteria. This would enable development to accurately estimate and deliver projects, but also would provide the testers with enough information that they would be able to test stuff faster, without having to do their own research all the time.

It sounded good in theory, but no-one except management expected it to work in practice. There were early warning signs, like the fact that no-one in the development team was allowed any say into the process; it was monopolised by the two senior testers and their manager. This included an incredibly detailed document detailing how development works, which was complete bollocks; any feedback we gave was drowned out in "well, yes, but, you see, for what it's worth, we have to take into account everything, that, notwithstanding the details, for what it's worth.... zzzz"

Did it work? No.

The first consequence was that no projects were approved. Nothing "the business" came up with met the entry criteria. For three whole months the team did nothing. Literally nothing. Eventually senior management exerted enough pressure to move some projects on, but "the business" still wasn't doing their part. What this meant was that projects moved into development without a complete spec, exactly the same as what used to happen before the system was changed; but, unlike the "good old days", it was now the developers responsibility to give a complete, accurate and detailed "risk assessement" and "use case/process flow" documentation to the testing team. All-in-all the workload on the developers increased greatly, despite their being fewer of them, but the testers retained veto on a project if they didn't want to test it for whatever reason (and some of the reasons were genuinely piss-takes).

Because of this all significant projects were still vetoed. Instead only very small and/or self-contained projects were approved, each would only require minimal development work (sometimes less than one day's worth), but the developers would still have to spend up to a month working on all the information that the testers needed. The developers, who were originally sceptical about the methodology were now openly hostile; there was no where for management to deny it anymore, it was simply a takeover of the software development process by the testing team at the expense of development. It was a coup, plain and simple.

More developers left, the companies financials got worse and worse. Competitors now had obviously better offerings, and getting better by the month. But still, even the most business critical changes were vetoed "because it'll take too long to test". Things were getting quite desperate, people were saying "fuck testing, if we can't do this we won't last the year, if it doesn't work we haven't lost anything!" but no; the testing team said no, therefore it will never, ever happen.

"The business" by this stage had been tapped up by the developers, just to make it clear that the developers were on their side. We wanted to make those changes, it wasn't us that was the problem. I don't think they quite believed us, they were quite suspicious about development, like the fact we got paid more despite never seeming to do any work.

About the only reason this ridiculous methodology lasted so long was because the management of the company was in-flux with an interim CEO. Things changed, and changed quickly when a new permanent CEO was introduced. He asked precisely three questions: "why are there so many testers?", "why has nothing being delivered for nearly a year?", and "what are you going to do about it?"

The technology director was gone by the very same afternoon. The two senior testers were persuaded to leave that same week. The new CEO was more worried that the developers would react badly to the end of the methodology, he was quite surprised when we said we never supported it anyway and would be only too happy to see it gone.

The testers, on the other hand, were well miffed. They'd never had it so good, they had got used to everything being delivered to them on a plate and never having to do some original work of their own. They also spent quite a lot of time in denial about the methodology's demise; one tester gave me a headache for an entire afternoon demanding I gave him the kind of use cases they used to get. I refused, my work was self-contained, I gave incredible detail about my changes but wasn't going to get bogged down in a full audit of the entire application stack, like what used to happen. He wouldn't let it go, and kept shouting in an ever louder voice.

"It's not worth my time, I'm not going to do it," was my response. The shouting only stopped when he booked a meeting with everyone from me to the new CEO, to discuss "testing requirements". As soon as he sent the meeting request, the new CEO took him into a room, he came out after two minutes, cancelled the meeting, and never said a word about it ever again.

Reality bites. The testing team will get used to it, but they still didn't like it. They kept complaining about the "bad old days". Yeah, "bad" old days; those days we made changes that earned the company money and kept us ahead of the competition. Woe betide us if they were to happen again!

Do I want an iPhone 4?

2010-06-24T10:09:00.000+01:00

So then, it's been nearly two years since I bought my iPhone 3G, and a top-notch phone it has been too. Since then there's been the iPhone 3GS, of course, and now there's the iPhone 4. Given that my 18-month contract has long been completed, is it not time I upgraded?

Well, I don't know actually. There's a surprisingly long list of reasons not to get one:

The fact that O2 are only allowing upgrades via their retail store. WTF? In fact I can't even see what price I'd expect upgrading on their website. For fuck's sake people, this worked three years ago when the iPhone was new and rare, but pulling these fake scarcity tactics now is just tiresome. You seriously expect me to take several hours out of my day, clutching my "two forms of ID" down to the nearest O2 store, just so I can have a spotty git lamely trying to guilt me into buying insurance, just to get my hands on a (what is now) bog standard bit of electronics?

The undeniable fact that Apple has turned a corner and is now pro-actively evil. Their belief that it is their right to protect the "user experience" regardless of how it affects customers or third-party vendors. They can get away with that nonsense now, and they can get away with it for mobile phones which were always locked down anyway; but they fact they're trying to extend the same tactics via the iPad is a trend that must be stopped.

The news that if you opt-out of allowing Apple access to your real-time location data, you are never allowed to install anything from iTunes (which is, of course, the only way to install anything).

The miscellaneous unrecognisable problems I've had with my current iPhone. Once every two weeks iTunes will not recognise it and force me to reset it as a new phone. It's fucking USB, how hard is it?

So really. With all that. Do I want an iPhone 4?

The nonsense

2010-05-12T21:46:00.000+01:00

It's been a while since I've written one of these, but I have to make comment about this ridiculous fluff-piece in The Independent (of all papers) today.

"The housing market benefited from its traditional spring revival during April as activity levels continued to increase, research showed today."

Hmm, so the property market is accelerating from the bounce it had last year (which I, and I alone, predicted)? That's quite significant if true as: mortgage approvals have been going down recently, the Halifax's most recent report shows falling prices and the Land Registry (the most accurate measure) shows falling prices.

So what is this research? It was conducted by The National Association of Estate Agents, their survey showed that more property was being put up for sale; but only a slight increase in number of buyers. Woo, activity's good, boost, spring bounce, buy now! ZOMG!!!

Or that's what they want you to think. Meanwhile anyone with the slightest grounding of economics knows that increased supply and static demand results in a downward pressure on prices. Perhaps there was other "good news" that I somehow missed. What did the rest say?

Hmm, sales per estate agent are down on last year. So are lower than they were before the mini-bounce last spring. OK, that doesn't sound very "positive" to me.... what else?

Apparently there is a lack of new buyers entering the market, but that's OK because "they'll need more and more government support" - so a healthy ever growing market we have there then.

I've ranted against this nonsense enough times before, but this is by far the most bizarre. It's like it's not even trying. Three major indicators all pointing downwards, but the headline "even more evidence of infinite wealth in the housing market", the article says nothing of the sort.

Meanwhile, back in the real world. The only thing that has kept house prices above water this past year has been zero interest rates; and if zero interest rates produce a market where sales volumes are only one-fifth of the 2007 peak; where no-one can enter the market; etc.; then you can see just how fragile the property market actually is.

What's going to happen next depends largely on this new government's actions. It should be interesting to follow. The value of sterling has been rising already, which means currency traders believe that government borrowing is going down and/or interest rates are going up. Both is good news, we shall see what happens. Although the Bank of England reckons interest rates are going to stay as they are for two years, but the Bank of England has ballsed-up it's inflation predictions for the past five years (according to an ex-governor, deliberately and wilfully massaged the figures), so make of that what you will.

Trousers

2010-03-04T21:54:00.000Z

One of the more mundane areas of life, one I should have got used to by now, is clothing; in particular, trousers. Really, dressing myself at my age ought not to be a problem.

The garment I have most problems with: trousers, and even more specifically than that: work trousers. Trousers smart enough to go to work in.

This has been a problem going back years: I manage to introduce a subtle split; I wear them out; they suddenly become too right or too roomy... it doesn't make any sense, but no matter how optimal they may have been at the outset, they are unsightly rags within a couple of weeks. Every, single, time.

Now, casual jeans I might wear the rest of the time... no problem. They will last me for ages. Years. I've got a wardrobe full of them. Never wear out.

So what the hell is going on?

I thought I'd cracked the problem quite recently. Taking advantage of my current workplace's relatively flexible dress-code - you still have to be smart, but there's no rule on shirts/ties/etc.; plain t-shirts, etc. are fine - I'd buy a pair of black jeans. Smart enough to get away with, but durable like my pile of normal jeans at home.

Genius or what? What? Today, after a grand-total of seven days wear: the seam has split down both calves - left and right. How the hell? That's the least-tight area they're used to cover, plenty of space. The pair were exactly the same as some normal blue jeans I have (only black, obviously - same cut etc.) which have lasted months so far.

I just don't get it. I'm constantly buying work trousers, and now I'm going to have buy yet more! I still think black jeans is a winning idea, I'll try a different brand next.

Suspicious Wi-Fi

2010-02-15T21:26:00.000Z

So, there was I this lunchtime on my way to buy some lunch, as you do. When I suddenly remembered I needed to check my bank balance. "Balls" thought I, I should have done it upstairs in the office, but never fear I can use my trusty iPhone.

And I did, apart from it being slow as mollases obviously, everything went well. Except a highly suspicious moment...

Banking via 3G may not seem wise, it's a complex protocol but snoopable, but still it's an SSL connection so should be safe. (Well, there's theoretical vulnerabilities against SSL, but if it's broken we're all fucked anyway.) Everything was going as - very slow - clockwork. Until, that was, the iPhone spontaneously switched to Wi-Fi. It does this, it's supposed to do this, my phone contract allows me use of various public Wi-Fi networks.

Now, Wi-Fi is definitely snoopable, with very cheap equipment. But still, it's an SSL connection, that should defeat any snoopers.

But, and it's a big but, BUT!, the next page I attempted to access flashed up the warning "the servers certificate is invalid for this URL" - woah there, giant red flag. I manually switched off Wi-Fi, went back to 3G, and everything was good again.

Which Wi-Fi network was this? BT OpenZone.

Are BT OpenZone conducting man-in-the-middle attacks against unwary customers? What are they snooping on and what are they doing with the information?

The web browser warned me of this, but it's not the only SSL application of course. I have my iPhone setup to regularly poll Gmail for my email; and, of course, wary of session-hijackers on public Wi-Fi networks I've switched SSL on. But does this mail application warn of invalid certificates; or have BT OpenSnoop been listening every time I walked past a phone box?

Weather

2010-01-09T10:30:00.000Z

This weather is beginning to piss me off. It hasn't snowed since Wednesday, and most of the main roads/transport/town centres/etc. are clear, but it's still a pain to get anywhere; especially the dice-of-death that is the route from where I'm sitting right now to the main road and therefore the outside world.

This is bad with it being a weekend because it means I don't really have to be anywhere, and the pain of getting somewhere almost certainly means "nuts to this, I can't be arsed"; but, then again, not going outside for two whole days is very boring. If everything was snowed in, it would be fine, I could deal with it, but it's not; if I could easily get to the nearest main road everything would be continuing as normal. I'll feel quite guilty if I don't make an attempt to do something, it's a waste of time otherwise.

Not that I don't have a million very important things to be doing right here, of course, there's a huge backlog of tasks - everything from clearing my clothes washing backlog (rock 'n' roll!), to updating this blog with the last set of Gamblotron changes.

I do need to do some shopping actually. On that score I don't have any options, so I'm going to have to give that a go shortly. I'd normally combine such mundaneness with a good dose of fresh-air by taking the scenic route, but I expect that avenue is closed to me today; the bag-carrying weight distribution-on-ice problem on my way back is also going to be quite interesting.

I have no problem with snow, as long as it's all gone and clear for a few days before the next lot!

Do you see what I have to contend with?

Windows still sucks

2009-12-23T19:30:00.000Z

Which I pretty much expected, to be honest, but it still needs to be said what with all the idiots going on about Mac OS X needing to play catch-up to Windows 7. They wish.

I can freely acknowledge that Windows 7 is the best Windows, this is beyond doubt. But, really, come on. I've only had my machine a couple of weeks so far, but had: one old-school Blue Screen of Death; two refusals to wake up after going to sleep; countless driver failures and memory leaks (O.K., not strictly Microsoft's fault, but still problems that don't seem to happen on Linux - Mac OS X has the advantage of supporting limited hardware subsets).

Compare this with my old iMac - I literally never turned the bastard off, and only rebooted it to apply security updates - three years continuous service. (Then it started crashing randomly, almost certainly due to hardware failure - overheating graphics card I reckoned.)

And the old iMac, even when knackered, would still go from zero to Firefox in the time it takes for this new machine to get to the "Loading Windows..." screen.

Microsoft fans - give it up, you've got a textbook case of Stockholm Syndrome. By all means acknowledge the wider software availability, the wider hardware compatibility, etc., etc., but Windows still sucks. Always has; always will.

It appears Microsoft still can't quite get their heads around quality, the same way that other organisations can. For example, I wanted to see what the graphic capabilities of this machine was; I don't play many games so I thought I'd download the DirectX SDK samples as a kind of benchmarking exercise. This caused to download Visual C++ Express, not too much of a hardship, until: "There is an error, part of the application has not been installed."

That was the full extent of the error message. And this was a brand new Windows 7 installation, so zero legitimate causes of conflicts.

Now, as a long time user of Open Source technologies I'm well familiar with customising things as I need them, so the error didn't fill me with fear, I would just: find the cause, change my system as needed, install the bastard. Unfortunately, as a long time user of Open Source technologies, I'm also well familiar with a open vibrant community to tap into; this does not exist in Microsoft-land. They think it does, but it doesn't. Or, at least, they're damned hard to find and the vagueness of the error message made it impossible to track down any forums for blog posts where the same error had happened before.

The only advice that seemed to refer to the same error was repeated twice, by two seperate people, both "Microsoft MVP" whatever the hell that means, saying: "Yeah, that error is caused by installing Visual C++ Express; uninstall it, it'll be fine."

Well, yes, it is fine, but I don't have Visual C++ either! That's it.

Compare this to researching any kind of bizarre gcc bug on Linux, you'll find several billion articles on the subject, going in to great length the differences of Linux distributions and what autoconfig settings you need, etc.

In other words, all the forces that led me to first trying Linux back in 1996 are all still there. All of them. The only things have changed is ripping-off the OS X dock, and they haven't even done that right.

Gamblotron Lives (almost)

2009-12-20T17:03:00.000Z

A bit of an over-excited headline, but we're not far off. I've finally rebuild the development environment on my new laptop to work more-or-less how I want it too.

I've taken the existing code and regressed it several steps, to the extent where it's pretty much the wrapper around the Betfair API and nothing else.

A quick test, to list the currently open markets:

ID=100618627 NAME=Winner 2009/10 EVENT=\Soccer\English Soccer\Carling Cup DATE=07/07/2003 23:00:00
ID=100680945 NAME=PFA Player of Year EVENT=\Soccer\Specials DATE=19/12/2004 23:59:59
ID=100613837 NAME=Winner 2009/10 EVENT=\Soccer\Austrian Soccer\tipp3-Bundesliga DATE=07/03/2005 14:30:00
ID=100721929 NAME=Next Labour Leader EVENT=\Politics\UK DATE=05/05/2005 06:00:00
ID=2725554 NAME=Most Seats EVENT=\Politics\UK\Next General Election DATE=10/05/2005 15:13:48
ID=4772641 NAME=Overall Majority EVENT=\Politics\UK\Next General Election DATE=10/05/2005 15:13:48
ID=20485444 NAME=Party Leaders EVENT=\Politics\UK\Next General Election DATE=10/05/2005 15:13:48
ID=20706198 NAME=Conservative Seats EVENT=\Politics\UK\Next General Election\Seat Markets DATE=10/05/2005 15:13:48
ID=20706199 NAME=Labour Seats EVENT=\Politics\UK\Next General Election\Seat Markets DATE=10/05/2005 15:13:48
ID=20706200 NAME=Lib Dem Seats EVENT=\Politics\UK\Next General Election\Seat Markets DATE=10/05/2005 15:13:48
ID=100247228 NAME=Date and Most Seats EVENT=\Politics\UK\Next General Election DATE=10/05/2005 15:13:48
ID=100498549 NAME=Party Seats Line EVENT=\Politics\UK\Next General Election\Seat Markets DATE=10/05/2005 15:13:48
ID=100958612 NAME=Election Date - Month EVENT=\Politics\UK\Next General Election DATE=10/05/2005 15:13:48
ID=100590765 NAME=Winner 2009/10 EVENT=\Soccer\English Soccer\Blue Square Premier DATE=02/07/2005 09:47:10
ID=100721223 NAME=Promotion 2009/10 EVENT=\Soccer\English Soccer\Blue Square Premier DATE=02/07/2005 09:47:10

And so on...

A more specific test to list details of a market; odds (and therefore implied probability), amount matched, etc. The 2010 Formula One Driver Champion market:

ID=100934070 STATUS=ACTIVE REFRESH=20/12/2009 16:50:53
*--> ID=2311131 (Lewis Hamilton) AMOUNT_MATCHED=3676.32 LAST_PRICE=4.1 LAST_PROBABILITY=0.24390243902439
*--> ID=2311125 (Fernando Alonso) AMOUNT_MATCHED=2699.12 LAST_PRICE=4.8 LAST_PROBABILITY=0.208333333333333
*--> ID=2445649 (Sebastien Vettel) AMOUNT_MATCHED=1546.04 LAST_PRICE=7.4 LAST_PROBABILITY=0.135135135135135
*--> ID=2710636 (Michael Schumacher) AMOUNT_MATCHED=1821.48 LAST_PRICE=9.2 LAST_PROBABILITY=0.108695652173913
*--> ID=2311128 (Jenson Button) AMOUNT_MATCHED=1565.58 LAST_PRICE=11.5 LAST_PROBABILITY=0.0869565217391304
*--> ID=2311126 (Felipe Massa) AMOUNT_MATCHED=931.82 LAST_PRICE=14.5 LAST_PROBABILITY=0.0689655172413793
*--> ID=2311143 (Nico Rosberg) AMOUNT_MATCHED=813.62 LAST_PRICE=15.5 LAST_PROBABILITY=0.0645161290322581
*--> ID=2311135 (Robert Kubica) AMOUNT_MATCHED=247.22 LAST_PRICE=23 LAST_PROBABILITY=0.0434782608695652
*--> ID=2311142 (Mark Webber) AMOUNT_MATCHED=461 LAST_PRICE=24 LAST_PROBABILITY=0.0416666666666667
*--> ID=2311133 (Rubens Barrichello) AMOUNT_MATCHED=208.02 LAST_PRICE=44 LAST_PROBABILITY=0.0227272727272727
*--> ID=2311139 (Nick Heidfeld) AMOUNT_MATCHED=441.86 LAST_PRICE=50 LAST_PROBABILITY=0.02
*--> ID=2311124 (Kimi Raikkonen) AMOUNT_MATCHED=461.68 LAST_PRICE=95 LAST_PROBABILITY=0.0105263157894737
*--> ID=4304180 (Nico Hulkenberg) AMOUNT_MATCHED=57 LAST_PRICE=160 LAST_PROBABILITY=0.00625
*--> ID=2311134 (Jarno Trulli) AMOUNT_MATCHED=8.64 LAST_PRICE=200 LAST_PROBABILITY=0.005
*--> ID=2893725 (Timo Glock) AMOUNT_MATCHED=59.46 LAST_PRICE=200 LAST_PROBABILITY=0.005
*--> ID=2311149 (Adrian Sutil) AMOUNT_MATCHED=43.14 LAST_PRICE=220 LAST_PROBABILITY=0.00454545454545455
*--> ID=2311129 (Heikki Kovalainen) AMOUNT_MATCHED=31.92 LAST_PRICE=230 LAST_PROBABILITY=0.00434782608695652

All the stuff that's needed for a system like mine.

But, to put it another way, this brings me to the same state the application was in within seven hours of me originally having the idea back in 2008!

Next steps: design the database schema; point these results at the database; begin work on my strategy testing spreadsheet.

Mark my words!

The rebirth of Gamblotron

2009-12-08T23:21:00.000Z

Right then, I have the idea in my head, all I need to do is do it.

My idea, like all great ideas, is so simple an idiot could have devised it. It comes in three parts: part one, to build a new development environment; part two, to migrate the currently working parts to the new platform; part three, to begin again at the stage where I got stuck last time, namely optimising the different signalling and staking plans.

Part one

Part one is currently under way. My existing computing hardware was getting on a bit, and randomly crashing, which didn't exactly help. And, although it would have done, it sounded like a good enough excuse to buy something shiny.

I'd previously written about how I needed a new laptop, and came to the conclusion that I needed a Windows laptop (even though all my instincts are UNIX based) because of the lessons learned trying to get WalkDB running on IE6, 7 and 8. So I recently bought a, quite massive, Windows laptop: quad-core, etc., should do me well for a while.

Although the first thing I've noticed is that WalkDB seems to run fine on IE8! It didn't on IE7, but neh!, it's not like I'm going to change it anyway.

This new machine will become my new development environment, all the current components of Gamblotron are cross-platform technologies anyway (i.e. Python), so the underlying OS isn't exactly a significant factor.

Part two

So the highly complicated plan for this part is to install a good old-fashioned relational database. The existing data-gathering scripts I have will then be modified to save to the database.

Why a database? Well, the performance problems I had with the initial version were, while not caused by entirely, exacerbated by the old evil of premature optimisation. The data gathered was purely linear, as it was intended to run in real-time. And that was a safe assumption as you can't place bets in the past; but what I hadn't accounted for was the amount of testing/analysis/etc. that would be needed. The sheer amount of IO required to "replay" the data accounted for a large part of the over-all running time of the tests. A good old-fashioned relational database will present the data in a neutral open format, allowing me to get at the bits I need to get at, even though I don't know what they are yet. It will also do the classic sequential "real-time replay" thing too.

The only bit of part two that is going to require any thought is the schema design; I should have everything else in place already, more or less.

Part three

This is where the real fun and games will begin. This is the stuff that I should have done the first time round. Instead of diving straight in to a platform designed to run interchangeable strategy scripts, I shall instead do some proper testing of the strategies themselves.

Again, the performance problems of the last time are a driver here. Running the strategy scripts took time, this was because they were optimised for real-time decisions so they pre-loaded all the data needed; they were still fast enough, taking about 50ms to process 1s worth of data. But still, that meant a ten minute run-time to "replay" a two-hour event.

But, thinking about it, ten minutes was a hell of a long time just purely for testing. It was madness. Even one data-point-per-second is only 60 a minute, 3,600 per hour, etc. How the hell was it taking five minutes to process that? (Well, the answer is in the previous paragraph, it was eagerly trimming data and doing all sorts of other housekeeping to keep response-time high when a match was found; but none of that is needed when just testing a strategy.)

Calculating the Relative Strength Index of a 7,200 row Excel spreadsheet would take mere seconds, perhaps not even one whole second. This would bring my hill-climbing and simulated annealing optimisation plans into feasibility, taking around an hour or two to get meaningful results rather than 14 days...

Now, I hate spreadsheets, and with good cause. I've seen millions of the little bastards, all badly conceived collections of wishful thinking and bad maths. But, just recently, I've been working at a place that did spreadsheets seriously; actually using them for proper calculations, not just a big database with a SUM at the end. It's almost, but not quite, almost, given me a respect for the things. Especially as when you add some correctly written (i.e. not automatically recorded) macros in there, and you suddenly have a fully self-hosting programming environment which can be used for some genuinely impressive things.

It has also left me with more knowledge of Excel bugs than I ever wished to have; everyone knows about the 65k row limit (yes, Excel 2003, etc.), but there's much more than that. And if you cross any of these lesser-known boundaries bad things happen; either: formulas fail, and random numbers are substituted; formulas are silently rewritten; you get a cached result that refuses to recalculate; the whole thing becomes uneditable, and point-blank refuses to save (the only solution is to open and re-save in OpenOffice.org!). Excel is a uniquely horrible piece of shit. No matter how sensible using it for rapid prototyping of Gamblotron, for my own sanity I cannot use it for a "fun" project.

So I've been thinking of trying Resolver One. They say it's free for non-commercial uses, which I think Gamblotron counts as, until it starts making money at any rate. But, of course, I don't know what kind of limits it has; it may have none at all, just limited by memory available... But it is very programmer friendly (Python - well IronPython, but I can ignore the Microsoft parts and use the standard Python bits) and might be interesting.

Conclusion

So that's all well and good, now I need to put it into action.

Where does all the time go

2009-11-21T21:31:00.000Z

I mean I haven't even had enough time to post anything ranty on here for a while, let alone get anything else done. I don't remember being busy either, but neither have do I remember having much time spare.

The only reason I have the five minutes to take the time to write this mini-moan is that I've finally had an idea on how to progress with Gamblotron, but it's nearly half-past nine on a Saturday night (so I really shouldn't have had time to think that, but there we go); I haven't got time to begin working on it now, and neither will I have tomorrow, or next week! Possibly next weekend, but I'd be quite surprised if that didn't change between then and now.

Finally, after being stuck thinking about some quite intractable problems, I've had an idea. It's not necessarily a solution, but it is an idea. The problem was that I'd originally written everything as a monolithic application - all except the strategy scripts which were interchangeable - the data-gathering/analysis/risk-management/bet-placing processes were all one and the same thing. On this I'd tacked on a basic "replay" function to allow testing and fine-tuning. The clumsiness of this approach, combined with some other implementation details (deciding to use Python), meant that performance was woeful, which in-turn derailed my plan to investigate some optimisation techniques - simulated annealing and all that stuff.

The idea is simple, it's dealing with each of those sub-systems as independent sub-systems. My initial plan, which needs a bit of thinking before I try and implement it, is to keep things simple and independent - multiple processes working on a single database (vendor to be determined). The various parts could then be run in different configurations, and lead to quicker investigation/turn-round-times/etc. The data-gathering bit only needs to run when data gathering; the analysis part could access the database directly, removing the need to "replay" the data, which did take a while doing not very much.

This should be quite an efficient process to re-architect too, the existing python data-gathering code wouldn't need too much work to write to persistent storage. I could then worry about any performance problems in smaller bits of code.

None of this is guaranteed to lead to a working system, but it's a better starting point than the single blob it currently is.

All I need now is the time to get on with it.

Things that annoy me, part 304

2009-11-07T19:33:00.000Z

As I've said in the past, I hate mobile phones; or, at least, that was until I got my iPhone.

The main thing that annoyed me about most of the early 21st century phones was they were ridiculous; they were expensive to use £1 per access of train times plus 20p/MB data traffic, and locked down in absurd ways. What impressed me with the iPhone was that it was essentially an always-on internet tablet that fitted in my pocket; that it also came with a phone was just a bonus to cut down pocket clutter.

(Yes, I know, Apple is the new Microsoft what with its locked down App Store and everything... but back in reality, the access policy is quite promiscuous and the web browser lets you access whatever you want without paying a premium; compare that with 2004...)

This leads me on to my current annoyance: the number of recent TV ads for other "smartphones". The least-bad ones are just trying to copy Apple, but they have no chance of success - the Ovi store? nice try. The worst ones though, what the hell? They're all for "social" phones and other such bastardisations of the English language.

There's the one that proclaims "all your calls, emails, tweets, on the same timeline". Why the hell do I want that? In my experience there's a definite pecking order: phone calls are quite important; SMS messages are somewhat important; emails are mostly circulars but worth checking in detail a couple of times per day; all the rest are just pure-dross, the best you can get out of them is very mild stimulation in the cases of extreme boredom. Why the hell would I want to drown out important communication from people I want to hear from with the latest mass-mock-outrage-campaign-bullshit-twittery? Answer me that!

But the worst is the new proprietary one from Vodaphone, which seems to advertise itself by recruiting ginger people to invade Russia.

Stop it, just stop it. I don't want a phone to have features, I want it to give me access to the things I want access to. I don't want lock-in to a fixed set of long-past-their-sell-by-date social networks, let me worry about all of that.

To summarise: Get off my lawn!

Wheee!

2009-09-20T09:26:00.000+01:00

Silly BBC have forgotten to write a story again. How this keeps happening I don't know... they are only too keen most of the time to commentate on the housing market, and editorialise on the subject, but on Friday they forgot to do that. Can't think why...

On Friday, the Council of Mortgage Lenders announced that August mortgage lending was down 13% on July. This was entirely expected, and fully consistent with what all rational analysis already knew. The positive mortgage/house price figures through the second quarter of this year were all due to the traditional spring house hunting season, the housing market is one of the most seasonal markets you can get; the numbers always were going to decline come the autumn, they always do.

But, and it's a big but, BUT! The same applies during the spring, the numbers will always rise, always! 2008 was the exception, when the collapse of the market outweighed the traditional seasonal movements, for every other year this is a true statement. So the market movements so far this year are not unexpected at all, they have merely highlighted the fact that 2009 isn't as bad as 2008. But this didn't stop the headlines: "Mortgage lending soares 7% in April".

So this is the story that the mainstream media has missed. If a 7% rise is "soar", then surely a 13% fall is "plummet". Yet, I cannot find a "Mortgage lending plummets in August" story anywhere... this doesn't make sense, surely the BBC cannot be biased? Surely not?

Oh, here it is, on the bottom of the "economy" section of the Business page. Apparently there's nothing to see here, it's just a "summer" thing. That's alright then...

Meanwhile, back in reality, that article lets the cat out of the bag. It says:

The CML said UK gross mortgage lending totalled about £12.6bn, down from July's revised total of £14.5bn.

Hmm, down from a "revised total". I wonder what the unrevised total was? Google is a wonderful thing... it was £16bn. Which is interesting. Last month's mortgage lending headlines were full of "mortgage lending rises 26%", which I talked about at the time. But that 26% "soar" was up to a number which has since been revised.

This leads me to think, what would that percentage rise have been if calculated against the new number? Well, fortunately the CML publish a spreadsheet, so I can calculate it myself. The answer, a much less impressive: 14.4% rise.

So in July it "soared" 14.4%, but in August it "summer adjusted" 13%. In other words the two pretty much cancelled each other out and we're almost back to the same place we were in June?

Well, this is where mathematics gets fun. This is an old trick used by people all the time. I let you believe that the August fall was smaller than the July rise because 13 is a smaller number than 14.4. People fall into this trick all the time, thinking that the 20% house price rises in 2004 were bigger than the 18% declines in 2008. The key here is that the falls happen from a bigger starting point than the rises, so the 13% fall is actually a bigger fall than the 14.4% rise.

The raw numbers (monthly mortgage lending, according to the CML): June - £12,666m; July - £14,493m; August - £12,600m.

Clear and conclusive. August lending is lower than June. So much for this resurgence in the housing market. Add this in with further declines as we move into winter and we can hopefully explode this myth once and for all, and accept the fact that we're witnessing a dead-cat bounce. I'm not holding my breath though, by the time the facts become undeniable we'll be at next March and next years "Spring Bounce" will begin.

The moral of this story is that you can't judge anything by purely month-on-month statistics. This is especially true for a market which has a clear, defined, and well understood annual cycle. The only numbers that are useful are year-on-year statistics, so lets have a look.

August 2009 vs. August 2008 shows a whopping 36.58% fall, or "adjustment", as I'm sure the BBC would call it, if only they quoted that number once - they didn't. The BBC, however, did have this to say:

Gross lending in the month was still more than a third down on the August 2008 figure of £19.9bn.

Which is correct, but only after they said saying:

Mortgage lending, sales and house prices have all picked up this year, although from a very low level.

Which is quite clearly untrue, from a mortgage lending perspective. Lending in December 2008 was £13.7bn. So both on year-on-year, and year-to-date measures, mortgage lending has fallen in 2009. The amount of spin we get from mainstream sources makes Shane Warne jealous.

So, given all these facts, what has been causing the rise in house prices we've seen recently?

Well, the rate of decline of mortgage lending is clearly slowing down. If the mainstream media wanted to report "good news" this is the only real non-made-up good news they could use. In the early part of this year it was running at -55%, now it's just -36%. But that's still a negative number, there's still less money going in to the system. I can't take claims of a genuine clamour of new buyers seriously.

The theory that cash rich buyers are gaining confidence is certainly the one that holds the most water. They have cash, and they have equity. For them the problems of the mortgage market - e.g. needing a 40% deposit - don't apply to them; they can get cheap mortgages with no bother whatsoever.

But, for that theory to be true, these "cash rich" buyers would need to believe the hype that the housing market is moving, or at least bottomed out. They want in before the hoi-poloi gets in on it.

This does seem to be my experience. The most equity-endowed person where I work has been spending 75% of the working day looking for a new house, "this will be the last chance we have to trade up". This all sounds very familiar to 2005, when house price growth reached zero, only to take off again as people suddenly panicked and thought this was their "last chance".

But it gets worse. He's not selling his old house, "no one has offered enough yet, but it'll be worth enough next year". Using a mortgage on two properties to buy one, at a time that there's no new people entering the market to produce the wholesale upsurge required to make his gamble work. It's like watching a train crash, it really is.

If we're two years in to the steepest house price decline on record, and there are people who, by good fortune, have got themselves a nice house with plenty of equity, and they still gamble the whole thing "to get in on the next boom"... all the while mortgage lending is collapsing again, first-time buyers never got going anyway...

Oh the humanity.

BBC Watch, part II

2009-08-28T15:33:00.000+01:00

Getting better. Now moved off the top slot to the second row:

But still a long way from useful. Measuring rate of change of a rate of change is a minefield at the best of times, especially on something that is so seasonal as house prices. It implies a kind of natural progression where none exists; it was only last month that the Land Registry monthlies was positive for the first time, for example.

Step in the right direction though, keep going.

BBC Watch

2009-08-28T12:30:00.000+01:00

It gets worse. This is the front-page now:

Looks the same as the last one? Look closely at the bottom, see how they've added a link to a nice friendly video telling you how to be a good citizen: buy a house, pay your taxes, vote Labour; the war in Iraq was just, the NHS is wonderful, Harriet Harman isn't a lunatic.

(I think this is driving me quite, quite mad. I'm going to go and lie down...)

Right, that's it!

2009-08-28T12:18:00.001+01:00

Just when I think the mainstream media have reached the nadir of their self-serving reality detached agenda. They go and disprove me and reach a whole new low.

Today, just after writing the linked post, the Land Registry posted their latest house price statistics. The headlines: a month-on-month rise of 1.7%; year-on-year is down 11.9%. So far, so expected, given the news from other statistics.

The Land Registry is still the most accurate, if that's what they say then it must be true. But, the BBC on the other hand:

"Biggest rise for years" - only if you count month-on-month statistics, and only if you ignore any other index (which they should do, but they don't when the Land Registry is showing smaller numbers than the other indexes, which it does 98.5% of the time). Jesus H. Christ. Also notice the prominent positioning, top story no less, the complete abdication of any of the secondary statistics. Guess where they used to put the "Sharpest house price fall on record" stories during the latter half of 2008? Oh, they forgot to write any! Silly them...

I've included the above as an image, rather than a link, as this sort of thing has happened before only for the BBC to tone-down the posting after a few hours.

Anyway, for any poor souls who both: a) read this blog (not very many); and b) still believe what the mainstream media say, I urge you read the statistics direct from the horses mouth. Download the PDF from the bottom right corner. Particular attention must be brought to the annual changes per county:

And the sales per month graph:

The most important piece of new information (we could guess the prices based on other information) is the amount of sales: 42,699. Look at the above graph, boom years were in excess of 120,000. This is the key statistic, prices may be moving, but the health of the market (i.e. probability to make a speedy sale at a good price) is still down; not at it's lowest point, but not exactly soaring either, it's been mostly flat for six months.

That number is significant anyway, but doubly so today. The final sentence of that BBC article says:

This has proved to be bad news for tenants, who have seen rent levels rise again as former "reluctant landlords" put their homes back on the market.

There is nothing in the Land Registry PDF that says any such thing, sales are still at a near-record low. Text like the above can only have been fed from the various property vested interests, back to their old tricks, trying to drum up business.

This is not a sustainable boom people! This is not even a long-term rally. Although, if the Bank of England keep printing money (ostensibly to fight "deflation" - ho ho, do not make me laugh - but really to stop government bond auctions failing) then it will be a long-term rally; it'll be an infinite rally, "One bed flats in Swindow to exceed £1,000,000 by 2011". Hmm, I think I've got a TV programme to pitch to Channel 4...

Wood? Can't see any wood. Not with all these trees 'round here...

2009-08-28T10:12:00.000+01:00

As you know, and as I've said numerous times before, we're all doomed. The collective concious of humanity has got so little grip on reality it's amazing. I'm sure this has always been a problem, but it must never have been this bad in the past? I put it down to a lack of recent wars, there's nothing like imminent destruction to focus the mind.

Yes, I am talking about our elected leaders, and their mainstream media puppets; and the bizarre decisions they make. The list is endless - one day the World Health Organisation plead to avoid the overuse of Tamiflu, lest Swine Flu become resistant; what does our government do? Send it out by the bucket load to everyone that wants it. Why? Because the Daily Mail would crucify them if they didn't. Reckless isn't the word...

And it doesn't end there, of course. This is the cause of much of the disjoint thinking that can be seen in economic reporting.

No, don't panic, this isn't going to turn into another "no one measures house prices correctly" rants. But, before I get to my main point, it is worth mentioning some recent developments:

All of the mortgage lending statistics are still showing 45,000 per month as being current lending rates. They've been around that level for quite a while. But still they spin "up 26% in July"; just look at the graph on that page, to see that 26% from a low base isn't much at all (it puts us back at the level of December 2008), and will require another couple of years to grow back to normal - and that's was never really normal, if you think Northern Rock/Bradford and Bingley lending (125% mortgages with no proof of income) was "normal" and want to see it return.

Some of the old-school cheerleaders who were temporarily stunned into silence during 2008 have started making noises again. David Smith (economics correspondent for The Sunday Times), who said in May 2007 "the world economy is strong and house prices rises will be slow but positive" (that was paraphrased for space, full analysis here), yesterday took time to point out the "trend" in the Nationwide House Price Index. (Incidentally, late 2007 was when Smith stopped allowing comments on his blog, I wonder why that was.)

The trend was that, annualised, the Nationwide House Price Index is showing a 15% growth rate. Hurrah, normality has returned, everyone knows house prices always rise at more than 10%, some bloke on Newsnight said it, and now it's true. (See, I could write articles for The Times, I'm already more insightful than Anatole Kaletsky.)

The truth is that a rise of such magnitude has taken mainstream commentators by surprise. They didn't predict the crash, they even tried to pretend it wasn't happening, but when it became undeniable they just reverted to form and ended up quoting each others predictions of a flat or slightly falling market. None of them predicted a mid-year boom. Some people (i.e. me), on the other hand, did predict a mid-year boom, describing such things as a "forthcoming dead-cat bounce". Those of us who did both: a) see the crash coming, and b) saw the bounce coming, generally take quite a different interpretation to these recent headlines than the mainstream pundits who are all falling back into their pre-crash set patterns.

For you see. This is a 15% rise (not that annualising part of a year is particularly helpful, it always produces wildly eccentric numbers, but I'll stick with it for the time being) on a very low number of sales. It is, therefore, difficult to make any extrapolations from this. This could be caused entirely by waiting cash-rich buyers being tempted to trade-up due to record low mortgage rates; if so, this will end quite soon. On the other hand, this could be the start of mortgages becoming more widely available again, but: a) we're still not realistically ever going to get back to early 2007 levels of lending, that only happened because of the willingness of now-departed banks (Lehman Brothers) to buy anything with the word "mortgage" on it; b) if such low volumes are causing a 15% hike, then larger volumes will cause an unsustainable spike; c) interest rates will need to be raised to avoid the inflation problem getting worse.

Meanwhile, back in the land of political expediency, the Bank of England is printing just enough money to buy all the government debt that the private sector doesn't want (half of it); and since this is unable to stop anytime soon, low interest rates will continue. It is this political shenanigans which is going to guarantee that everything ends badly, and is the main thing that has caused this dead-cat bounce in the first place. By monetising the debt, they are rekindling the same kind of asset-price bubble that caused all these problems in the first place. But those idiots who regarded 2006-2007 as "normal" and the "good times" will welcome such a return with open arms; little do they know that it practically guarantees another crisis that'll make 2007-2008 look like a minor spout of petulance. Can you imagine, if the "boom" lasts another six months, there will be a very large percentage of home owners on essentially zero rates (mainly variable because fixed mortgages have already gone up); when rates need to rise... kaboom.

All the weapons in the armoury have been used for this first crisis, even though only half of them were needed to stop the contagion; the rest of them have been used just for morale boosting purposes. When we need them again, they will not work. If the next crisis happens at 1.5% interest rates, cutting them will have no stimulative effects whatsoever. And if inflation is up at 10% by then, printing money will be a waste of time, it'll devalue faster than it can't be spent.

This is not going to end well. Not well at all.

We're still living in a world where the belief that continually rising asset prices based on ever expanding lending is both sustainable and desirable. The whole notion of having economic growth, wealth creation, etc. never gets a look in; or when it does get mentioned it is only in the context of needing to sacrifice parts of it "to increase lending" - yeah, let's destroy investment, then people will have to borrow. Makes sense...

The final evidence of this, and the whole point of the post (must stop getting side-tracked), is the way the "problem" of bankers bonuses is being handled. The current best ideas that are being bandied about is either a: high pay commission, which would have the power to essentially confiscate large earnings; or a bonus tax, punishing companies that pay bonuses.

Both are, quite clearly, madness. Complete, total, utter lunacy.

Are banking bonuses too high - almost certainly yes. Are banking bonuses deserved - almost certainly no. But, at the same time, banking is highly profitable (because the government pays billions to them every time they don't naturally make a profit - which is the real outrage), and considering the amount of money sloshing about why shouldn't the workers get the money.

You hear similar arguments about Premier League football, they're overpaid, etc... well, yes, but there's billions flooding around the game, this is caused by the competition for media rights more than anything else. And, given that fact, surely it's better that the people who create the entertainment get the money rather than the owner of the Dallas Cowboys or other such hangers-on?

The bonuses are a symptom, they are not the disease. The disease is that banking is such a privileged closed world. In any other industry such large profit margins would lead to new entrants, and provide incentive for game-changing alternatives to find investment. Not in banking, they keep sitting there, creaming the rest of the economy... This is the problem!

It's a problem that is made worse by the willingness of the government to keep it afloat at all costs. Providing unlimited insurance against losses, etc. No other industry gets such treatment. Of course, banking is special, if it all collapsed we really would have a Mad Max scenario; this is why the original bailout was valid, it's everything that's happened since that's the problem. Several times this year, for example, the government has increased its stake in various banks "to provide more capital for lending". Freaking reckless lunacy.

They know what they're doing. And what they've done has worked, there's no longer any risk of High Street banking failures; but they've gone too far. The economy is, at the best case, sluggish for years because of the inefficiencies that have been added by overcompensating the bailout.

You can see now that bonuses aren't the problem. When the whole bank has a guarantee, there's no incentive for them not to take risks, it doesn't matter how much or how little the bankers themselves are getting paid; there will still be pressure for results.

To summarise: Expect a sequel to the financial crisis, bigger and better than ever before, late 2010 or sometime during 2011. Mark my words. The only real question is what form it'll take; currently I'm expecting a hyperinflationary flame out.

Whatever I've said about The Guardian in the past...

2009-08-12T20:15:00.000+01:00

...was still all entirely accurate, but occasionally they make steps to redeeming themselves.

Take this article today, it's exactly what I've been saying for months. Months! It has to be the first time I've seen from a mainstream source exactly the problem we're now in, it covers all the bases:

Over-eagerness to have "rising" house prices, why that's a bad idea, and why the statistics which currently show the very thing are only the beginning of the story - 45,000 sales per month is not a healthy market.
Risk of inflation from quantitative easing.
The lack of lessons learned about the causes of the crisis in the first place. It wasn't the drying up of credit that caused the problem, but rather the massive expansion of credit over the previous five years.
The probability that everything that's been done to fix the problem is quite likely to make things worse, potentially to the extent of their being an even greater crisis two, three, five, ten years down the line.

It's madness that such comments are few and far between. Everyone else seems to believe that doubling M0 money supply every three months, just to give a few bankers a nice bonus and get house price statistics nominally positive, is actually a healthy thing to do. I know which attitude makes most sense to me.

Tracking the decline and fall of western civilisation

2009-07-18T22:08:00.000+01:00

On Friday, The Guardian has a live blog (a fucking live blog) on "the best week for the FTSE all year." I'll let that sink in for a second.

What?

The Guardian, the well known slightly socialist newspaper spent all Friday cheerleading a divorced statistic that doesn't really have anything to do with anything? Why? What?

Is it just me, or does this make anyone else both simultaneously bewildered and slightly nauseous? Quite why it has this effect I do not know; it's not that the stock market isn't interesting, it is, to a certain extent. Nor is it that people have a vested interest in it; that's a good thing, we'd all be better off if more people had direct interest in the stock market rather than putting all their eggs in the housing market basket. It was more the tone, the blinkered one-dimensional tone...

If the Financial Times had done such a thing it wouldn't have bothered me, that's what it's their for, and it would have done it in a more transparent way. They would have reported the simple facts of the matter, as they happened; and indeed they do, every day, that's practically their entire purpose.

The Guardian annoyed me because their tone was that this was some sort of significant event, yet more "proof", that "the worst is over". All such talk is nonsense... why? Because "worst" and "over" all betray the unspoken belief that the economic problems of the last two years were just the downward sweep of a U-shape, and that the inevitable rise must come on the other side. It is my conjecture, as I have described many times before, that these problems go far deeper than that. The crisis didn't start in September 2007; the crisis began with the dot-com crash of 2000 (and by extension, the initial trigger must have been the dot-com boom itself). The dot-com crash, and the small matter of September 11th 2001, created a sustained period of historically low interest rates which the world has become addicted to. This created an unsustainable asset-price explosion, which most people confused with genuine wealth.

The problem is that each cut in rates has a diminishing return; but each step up becomes twice as painful. Hence all the problems in 2007, when rates were "high" (still would have been considered low in the 1990's). We've now got 0.5% rates, and the biggest GDP drop for donkey's years. There's nowhere else to go.

Movements in the FTSE are irrelevant, except for monitoring investments or the general health of the stock exchange. It's easy to forget that the stock exchange isn't that important, it has little bearing on the wider economy; it's main importance comes from it's role as a secondary market for shares, and a secondary market is only important for its effect on the primary market for shares (IPOs).

A buoyant stock market is therefore important, because it provides a means for companies to raise capital, without borrowing (remember those days). This is the stuff that a healthy economy is made, a means of sustaining growth without being dependent on ever lower (and therefore unsustainable) interest rates.

But the FTSE isn't as buoyant as people claim, not by a long way. It may have had the best week of the year, but that was only because it started the week at the lowest point for three months! And it's still short of its most recent high, 4,500 in June. It's still lower than the 4,600 it was in January. And well, well short of the all-time high reached TEN FUCKING YEARS AGO! (Ahem.) The current level of the FTSE100 is the same was it was in April 1997.

This is a testament to how markets which all conventional wisdom says should be one-way, barring occasional short-term crashes, can actually permanently stagnate. Dig out your time machine, go back to the middle of 1999, and tell people that the FTSE will spend the entire of the next ten years lower than the current valuations. (Of course, when you take inflation into account, the "real terms" fall of the FTSE is much worse.) People would have laughed and/or resorted to ultraviolence.

Exactly the same reaction people have now to people that say the housing market is going to take ten years to recover to 2007 values.

So the real reason that that Guardian blog angered me is that it's another nail in the sanity coffin. When the only economic debate you can find is a national newspaper saying "go on FTSE, you can do it" like it was some sort of football player, we're all fucked.

The "booming" economy of 2000's was nothing of the sort. We've been in a cryptorecession since 2000; any activity has been nothing more than cheap-credit fuelled speculation. (Perhaps I shouldn't complain too much, it's been keeping me employed in a wide variety of boondoggle projects - or then again, perhaps that's why they all sucked, a bit of bottom-line reality would have done them all the world of good.)

(Damnit. Here was me thinking I'd just invented the word "cryptorecession", but Google finds a total of one other reference. You'll all be using it in a couple of years time, mark my words!)

And, of course, what's the cure? More cheap money, of course. Yep, that'll work.

I would like nothing more than to be wrong about any of this, but nothing I read convinces me otherwise. I just find patronising rubbish written by Anatole Kaletsky, saying that the only problem is that interest rates might ever be higher than 0.5% - oh no! We can never let that happen. NOO!

Apparently all us people predicting quantitative easing fueled inflation are wrong because "M4 money is being replaced with M0 money, but the total is remaining the same" (paraphrased) - yes, at the moment, but the new M0 money hasn't worked through the fractional reserve banking system yet; then we'll get ten times the M4 money we ever had before!

For the good of my sanity, someone tell me why I'm wrong. Please!

"For fuck's sake!" moment of the day

2009-07-09T07:32:00.000+01:00

A quick one, I'm running late.

Yesterday, it was announced that the Halifax house price index had turned negative again (and would have been worse were it not for the "seasonal adjustment" that added value rather than subtracted - standard practice for June).

Today, on the front page of BBC News, good news for the housing market as Nationwide introduce a 125% mortgage. Yay! Because they worked so well last time!

Only, of course, they haven't. They've introduced a standard 90% mortgage, with an expensive option for people to carry-over a limited amount of negative equity. It's not eligable for new buyers, and priced such that only the very desperate to move would even consider it.

So why the hell is this front-page news?

It doesn't make any sense. There is an orchestrated campaign of misinformation going on here. Who is behind it? Who!

Same old, same old - part two

2009-07-05T00:13:00.000+01:00

For the other side of the story, listen to this vested interest circle jerk. In traditional BBC style, the presenter of the programme, which is intended to offer financial guidance to it's listeners, is "wishing property prices goes up", and is talking to the bloke who wrote that piece about house prices rising at 17% annualised. (Alright, they did have a sceptic talking too.)

It's quite interesting how they mention much of the same things I did, but yet come to radically different conclusions. They interpret the rapidly rising costs of fixed-rate mortgages as "bad news for those left behind" (typical propaganda); while I interpret the same as proof that there is no fuel to motor any sustained growth.

Anyway, I won't start ranting again. Listen to that and read my previous post and draw your own conclusions as to who's talking the most bollocks.

Same old, same old

2009-07-04T23:57:00.000+01:00

I'm getting bored of talking about it, but I feel the need to externalise my sense of outrage. Every month is the same, every year, forever. I'll be long dead in my grave before I start accepting what I'm told in press-releases at face value.

A quick recap of what I'm talking about: May house price statistics. The end-of-May indexes from both Halifax and Nationwide contained some quite strong upward movement, enough for some over-excitable vested interests to proclaim an annual growth rate of 17%. Last week the Land Registry published their index for May; how much growth did that show? Fuck all, that's what. Prices fell 0.2% over the month, and fell 15% year-on-year. So much for "house prices rising again".

The problem is (well, it shouldn't be a problem, but people are sheep) that only three days after the Land Registry index is published, the Nationwide published their numbers for the next month. So "bad news" is only in the public concious for three days, and "good news" is all anyone uses for reference the rest of the month; this gives people false confidence in "good news". The publication dates shouldn't matter, but people tend to value "recent" data better than old data, regardless of accuracy.

The simple explanation for this constant gulf between the Land Registry and the Halifax/Nationwide is simple. The Land Registry is better!

"You would say that, you're constantly arguing that prices are low and/or surpressed!" I hear you shout. Which is true, but that doesn't mean I'm wrong. Anyway, I don't have any particular vested interest one way or the other; I have enough direct property investment to dwarf the Plan A Development Fund, so I should be cheering all the money-printing and other efforts to "kick start the housing market." The reason I'm not is quite simple: property is grossly overvalued, does not stand up to any reason (i.e. all the "supply and demand" you hear about is based on faulty definitions of "supply" and "demand"), and a correction is vital to a healthy economy. And, even more fundamentally, even after two years of falling prices, current valuations are still unsustainable. Valuations have been protected by economic policy, but at the cost of destroyed savings (which are vital to a healthy banking sector, and therefore the future property market) and a forthcoming inflation shock.

Anyway, back to the point. Why is the Land Registry better? I refer to my previous post on the subject, but can be summarised in two points: 1) a larger, and more representative sample size (i.e. includes cash purchases and specialist mortgages); and 2) superior statistical methods, geometric mean and regression based on difference in achieved prices for the same house rather than a reliance on mix adjusting and seasonal fiddle-factors.

So what's going to happen next? The Land Registry still showed a distinct trend that monthly falls are getting smaller, plus the past few weeks contained evidence that other indicators were becoming positive. But evidence of any real structural solution are very few and far between. This led me to finally declare the dead cat bounce I had long been predicting.

So what about those June house price statistics? Halifax hasn't reported yet, that'll be next week. Nationwide says 0.9% up in June. Which, of course, gets reported unquestioningly as "good news". This is an interesting number, as it represents a smaller growth than the last two months; June is also the final traditional house-buying month, the last month to get seasonally adjusted downwards. It will be interesting to see what happens next.

The only real predictor of house prices is, of course, mortgage approvals. So what do they say? "Mortgage approvals still rising", says the BBC, with helpful commentary of how this underlines the continuing recovery. So how much did they rise? Was it 8% like the last month? No, it was 200. That's right, two hundred mortgages more than the previous month. That's a soaring market sparked by "pent up demand" if I've ever seen it. Mortgage approvals have peaked at around half the rule-of-thumb neutral level of 90,000. Recovery? Do not make me laugh.

Not only that, but net lending was the lowest ever recorded. This was also painted as good news (which it is, from an economic stability point-of-view), it represents a quite severe shortage of money entering the property market, and therefore is going to be a brake on house prices.

This dead cat bounce may be shorter than I reckoned. I was expecting to last between six months and a year, but at this rate the August house price statistics will be undeniably negative once again; where they will firmly stay for the rest of the year, at least.

The rest of the world is beginning to latch on to my ideas, they always do, in the end. The Times published quite a good synopsis of the current state of events last week, it still put too much of an upbeat spin on it for my liking, but acknowledged that a housing recovery is unlikely to be underway just yet. And most of the rest of the mainstream media have been cooling on the whole "green shoots" thing, they're firmly hedging their bets.

In other, related news. My cash ISA which had been paying 0.60% (I would have transferred it were it not for two factors: a) I couldn't be bothered; and b) nowhere else offers significantly more) is now paying 2.70%. Now, this isn't the bank being generous, it's more a reflection of wholesale money-market rates; and these are the best predictors for future interest rate moves.

The inflation/interest rate shock is coming. Then where will we be? This is going to cause significant pain. There are quite a few people, anyone who's fixed rate has ended soon, who have been happy to fall back to a variable mortgage as "it's essentially free money, I'll fix again when rates go up" - oh no you won't, you can have 0.5% variable, or 3% + fixed. There are no cheap fixes (well, 3% is cheap by historical circumstances, but that's more due to reckless monetary policy than anything else - that's a separate rant). That gap is only going to get wider, the jumping point might have already gone. (And most of these fixes requires a LTV ratio that existing mortgagees can't meet anyway.)

So what should anyone do about any of it? Fucked if I know, but it's not going to be easy.

Cooking

2009-06-28T10:59:00.000+01:00

I made the mistake of thinking that 9a.m. should be plenty cool enough to go outside. I was wrong; I could only travel at half my normal speed, and the sweat has left me weak, I need a saline drip or something.

None of this bodes well for the next couple of months. July is usually hot, and the hottest day of the year usually happens in August.

There must be a way of learning heat tolerance. Either that or I need to find a building with air-conditioning that won't mind me sitting there for much of the day.

At least the forecast says that humidity levels will be lower beginning tomorrow. The temperature is expected to stay until Friday.

I think it's finally time...

2009-06-14T00:17:00.000+01:00

...to call this dead cat bounce.

For a long time I've been claiming that the next significant economic event will be a dead cat bounce. On the 11th of January this year, I wrote: "the deflationary forces in the economy will temporarially mask the inflationry effects of the scheme, and if enough money is printed then there will be a dead-cat bounce". I continued to refine and develop this theory in late January, April, and a myriad of occations before and since.

I know I get accused of being a doom-and-gloom merchant, but on that score my prediction was a hell of a lot more upbeat than the cheerleaders, and also a hell of a lot more accurate. But again, I am always right. (It is this arbitary categorisation of bulls and bears that also stop debate, it's physically impossible for any genuine independent balanced analysis to make any kind of widespread impact. This is doubly true in the mainstream media.)

There's now enough evidence that various economic indicators are actually moving, before hand people were just reading non-existant patterns from sampling errors.

So why do I think this is a dead cat bounce, rather than a recovery? For many reasons, most of which I have explained in previous posts - any recovery that happens at the same time as unlimited government guarentees and money printing isn't a recovery, it's a subsidy - that sort of thing. So for this post I want to concentrate on one particular theme:

The reason why I believe this to be a dead cat bounce is precisely because people think it isn't a dead cat bounce. I'm talking about good old-fashioned delusion. What does delusion have to do with it? Quite a lot, unfortunately; people's perceptions of the causes and effects of the great 2007-2009 financial crisis will control what happens next. Decision makers, whether: government, Bank of England, academia, individual businesses, etc., will all make huge judgments based on this delusion.

For example, today David Blanchflower was given a CBE. Apparently, he's a visionary because he was calling for rate cuts several years before they were needed. How is that visionary? Surely that's a textbook example of a stopped watch being correct twice a day? Low interest rates for too long were one of the major errors during this decade, that's what allowed the whole unsustainable debt bubble to get so ridiculously big. And even if it wasn't, the jury is still out on whether low rates is helping with a recovery or not; the problem was always one of liquidity, which could have been tackled equally well with higher rates.

You see, people are already being congratulated on their "vision" even before the consequences of their actions are known. This is what we have to deal with. The establishment has already decided that permanently low rates and inflation can suit itself is the one true way.

This is just lunacy.

Your average newspaper columnist is no better. I'm not going to mention names, but they know who they are. At the beginning of the year: "a tentative recovery will begin, fueled by quantative easing, but a sustained recovery may be a long time in coming." Now: "service sector output rose in May, that means we've fixed all the problems and everything is super!"

Essentially what's happening is fully consistent with the "tentative" hypothesis, but that's been dropped in favour of "good times are back again!" - a view which is unsupported by any real evidence.

It has to be some form of bad news fatigue, or something. People are leaping on any positive indicator, then spontaneously forgetting about any other issues and declaring everything as done, finished, sorted.

Why? All of the reasons for believing a recovery will be slow are still in place - government debt is high, requiring future spending cuts or higher taxes; oil prices are rising, which will add to inflation; etc. (The "elephant in the room" - god I hate that phrase, but it seems apt - is that the "boom years" were based on a lie, it was that dawning realisation that caused the credit crunch in the first place. That makes the "boom times are back" a physical impossibility, they never existed in the first place. There will only be "good times" when we have a functioning non-subsidised economy that's driven by something other than financial voodoo.)

It's something to do with the perception that the good news has come along all of a sudden, that has shocked people, they weren't ready. Well I was ready. It's inevitable. If you guarentee all banks so they can't lose money, then print an extra few hundred billion on top, well fuck me, some of it gets spent. That's why things are moving upwards people, it doesn't take a rocket scientist to work that one out.

This spending was the entire point of making those policies, of course. But yet all the descriptions of activity fail to mention this. If this were a spontaneous recovery then the 0.5% interest rates are dangerously low, we'll have 15% GDP growth by the end of the year and a huge economic burn-out.

And of course, it's this key small matter that's the real issue. The beatificaton of Blanchflower, are conspiring to keep rates low; and they will remain low for too long. The next crisis, a crisis that is going to dwarf what has happened so far, is a crisis of inflation. It's been a long time coming, and it won't start for a while yet, but it's coming.

Despite rates being low, real financial products have already started moving up - like mortgages.

Even if GDP growth is now positive, which was claimed by one group this week, that's not the end of the matter. Not by a long way.

The final example of delusion: Double-dip recessions are not uncommon. Given the evidence before us now: seemingly positive indicators, plus high probability of a future inflation spike; then a double-dip recession is certainly a distinct possibility for the current problem. The "recovery" may last for six months, it may last for a year, but the chance of it all slipping back is certainly in the realms of probability.

How many mainstream media citations of the possibility of a double-dip have you seen? That's right: none. None whatsoever.

More on Getting Things Done

2009-06-06T00:29:00.000+01:00

A week ago, as you may remember, I had a minor breakdown (it took all of twenty-five minutes to come and go). Although my analysis that things weren't as bad as I first thought is a bit wrong, it's much worse; much, much worse!

I'll spare you the horrible details and instead talk about something more positive. It's pointed out a huge glaring failure in my GTD system, which may or may not explain why I never get anything finished.

The original crisis came about when I set myself a goal and a deadline, and I suddenly realised it was significantly more difficult than I had originally envisaged. This sort of panic had been rare in the past, because the full horror has been masked by the reasuringly clogged todo list. But the crisis is right, it gets to the nub much, much quicker than the todo list.

You see, the problem is that the todo list understands the concept of dependencies. If a task is marked as a dependency of another task, then it inherits a slightly higher priority. As such, the items at the top of the list are those at the end of the longest dependency chains. They have lost all meaning, with respect the underlying task, like a game of Chinese whispers. And, the root item might not even be particularly important, it just happens to have the longest chain.

This doesn't make the todo list worthless, it's good to analyse cause and effect and find tasks which seem irrelevant but can be proven to have good effects down the chain. But it does mean it's possible to spend infinity on peripheral tasks.

My addition to the GTD system, therefore (even though I haven't got around to describing it yet), is to make a habit of deadline setting. Where the deadlines are driven by root items of the todo list, not taking dependencies into account. For a task to be eligable for a period of focus it must be expressable in the form of an unambiguous goal, with a sensible time attached to it.

Root and branches damnit.

Just where the hell...

2009-06-05T18:28:00.000+01:00

...is the chocolate I bought yesterday? Where?

It's on the receipt, and I'm quite sure I would have noticed it missing when I unpacked if I didn't have it. I must have picked it up with another item, and put it in a place I don't usually put chocolate. God knows where that is, it could take months to find.

If I didn't have enough problems already, now this.

I suppose there's an outside chance it was missing from the bag, but I'm sure everything was packed before leaving, double checking is a very annoying but highly-effective habit I've picked up over the years. In that case it either: a) fell out, somewhere along the way; or b) was snatched by the checkout minion... hmm, he did look a bit dodgy, he didn't know how to use the till properly.

Hmm...