The official said the state department contacted Twitter over the weekend to urge it to delay a planned upgrade that could have cut daytime service to Iranians.

So, let me get this straight. The US state department monitors Twitter traffic and is reviewing what Iranians are commenting on the topic of their government. The department wanted to prevent an upgrade to Twitter that might stop the flow of revolutionist propaganda getting back out on the street and starting a larger movement.

Don’t get me wrong, it’s a good call. It just caught me a little by surprise.

I guess I shouldn’t actually be surprised by this but it marks a very definite transition (to me at least) for Twitter. Folks at Twitter have probably been feeling it for years though. The point is though that it’s no longer a fun toy.

Twitter is now, amongst other things, a marketing place for the public, politicians, and companies. But for government departments it’s also a socio-political barometer.

Innocence lost.

Even so, I’ve sort of been following it from the sidelines for a while and I’m excited to say that Mono 2.4 seems to be a drop-in replacement for the bits of the .NET SDK I was actually using. Now with the release of MonoDevelop 2.0 it seems that I’m also able to get a functioning alternative to the Visual Studio GUI I’ve been using. It’s a colossal achievement and one that makes me even more ashamed. Ashamed that I didn’t think it was possible, and ashamed that I didn’t help out.

To get it working was a bit of a trial though. It turns out that because I already had the default MonoDevelop 1.0 installed that when I started trying to build a new version things got very confused. Not only that it turns out that the build instructions for MonoDevelop 2.0 aren’t exactly correct. After a lot of messing around I think I managed to damage the default installation and cross-link a few libraries here and there. I think if I was doing it again I should probably have followed these instructions Parallel Mono Environments before doing anything. Too late now though. After much building and swearing I ended up uninstalling all Mono environments and then rebuilding everything for Mono 2.4. I didn’t put the Hardy version back yet, I’ll do that if I need to.

Here’s what I did, in case it helps anyone.

First of all I decided that this is a temporary solution and that eventually I would be able to get MonoDevelop 2.0 running from APT, therefore I decided to install everything in /usr/local.

Ubuntu Development Packages

As ever with Ubuntu you will quite probably need to install a bunch of development libraries to make it work. I probably had a lot of the dependencies I needed installed before I began but here are the ones that I didn’t have that I needed. Your mileage might vary.

sudo apt-get install bison  gnunet-gtk-dev libpango1.0-dev libgtk2.0-dev
libglade2-dev libgnome-dev libart-dev libgnome2-dev libgnome-dev libgnomeui-dev
libgnomeprint2.2-dev libgnomeprintui2.2-dev libgnomecanvas2-dev
libpanel-applet2-dev

Mono

As per the parallel environments guide I set up this environment in a script and executed it.

#!/bin/bash
MONO_PREFIX=/usr/local
GNOME_PREFIX=/usr
export DYLD_LIBRARY_PATH=$MONO_PREFIX/lib:$DYLD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MONO_PREFIX/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=$MONO_PREFIX/include:$GNOME_PREFIX/include
export ACLOCAL_PATH=$MONO_PREFIX/share/aclocal
export PKG_CONFIG_PATH=$MONO_PREFIX/lib/pkgconfig:$GNOME_PREFIX/lib/pkgconfig
PATH=$MONO_PREFIX/bin:$PATH
MONO_GAC_PREFIX=/usr/local
PS1="[mono] w @ "

I then downloaded the following libraries from the Mono 2.4 sources page.

wget http://ftp.novell.com/pub/mono/sources/libgdiplus/libgdiplus-2.4.tar.bz2
http://ftp.novell.com/pub/mono/sources/mono/mono-2.4.tar.bz2
http://ftp.novell.com/pub/mono/sources/gtk-sharp212/gtk-sharp-2.12.8.tar.bz2
http://ftp.novell.com/pub/mono/sources/gnome-sharp220/gnome-sharp-2.20.1.tar.bz2
http://ftp.novell.com/pub/mono/sources/mono-addins/mono-addins-0.4.zip
http://ftp.novell.com/pub/mono/sources/mono-tools/mono-tools-2.4.tar.bz2
http://ftp.novell.com/pub/mono/sources/mono-debugger/mono-debugger-2.4.tar.bz2
http://ftp.novell.com/pub/mono/sources/monodevelop/monodevelop-2.0.tar.bz2
http://ftp.novell.com/pub/mono/sources/monodevelop-debugger-mdb/monodevelop-debugger-mdb-2.0.tar.bz2
http://ftp.novell.com/pub/mono/sources/monodevelop-database/monodevelop-database-2.0.tar.bz2

Extract all these and then run the below command in each. Because this is a dependency stack you will need to do them roughly in the order they’re listed above.

./configure --prefix=/usr/local
make && sudo make install

After that you should be left with a working MonoDevelop 2.0!

Troubles

I encountered a few problems on the way, mostly there were problems relating to not having the correct Ubuntu development libraries and headers installed when the parts of the stack were configured.

I also ran into some problems with the GAC having pre-installed versions of conflicting libraries installed. Eventually though this problem was resolved by uninstalling the default Hardy version of Mono and starting again. Hopefully you won’t have this problem because you will have setup a parallel environment in the first place …

We’re programmers, we love this esoteric shit. The more hardcore the better. IMHO, the force of numbers, and its pedigree, will probably make the big fat git wade through the opposition like the monster it is.

Anyway, for a few months now I’ve been seeing this github thing linked here and there, I figured it was about time I checked it out.

I went, I saw, and it was good. Fast, easy-to-use and beautifully in the spirit of free-software (you only have to pay to use it for private repositories).

So in the spirit of free lovecode I’ve added cl-mysql.

Code … data … code.

A very dim light went on in my head telling me that that sounded very much like a Lispy problem.

However, I particularly wanted to use Mono (and not Java, say) for the solution and so I started writing my own interpreter. After about 4 hours of toil I realised that maybe I should finish reading Lisp in Small Pieces before getting carried away. But the urge to create was large, so I went looking for a Scheme interpreter written in .NET and found S#.

10 minutes of fiddling in MonoDevelop later and I had an R4RS interpreter running in .NET on Linux. I heart open source …

stkni@ubuntu:~/source/SSharp-Mono-0.38a/SSharp.Console/bin/Debug$ ./SSharp.Console.exe 

 This is: SharpScheme v0.38 - R4RS compatible Scheme interpreter.
 A pure C# port of Peter Norvig's Scheme for the .NET Framework.
 License here: http://www.norvig.com/license.html 

 Running on: Unix  

>>> (+ 1 2)
3

Here are some of the highlights of v0.2

• Connection pooling - Thread safe allocation and release of connections from a central pool.
• Use result/Store result - Ability to use mysql_use_result as well as mysql_store_result. This means that CL-MYSQL should be able to handle the processing of very large datasets without running out of memory.
• Convenience functions/macros - with-rows / nth-row

The main difference between v0.1 and v0.2 is that version 0.1 didn’t really manage its connections. I decided that allowing the user to choose between pooled and non-pooled connections is a hassle. Much better then to allow the user to create as many connection pools as they want and allow them to specify the maximum and minimum number of connections that the pool can hold. After all, a single connection is simply a special case of a pool with only one connection.

However, in theory this could hurt performance when attempting to do large number of INSERT/UPDATE’s because every call would require the connection pool to be locked and a connection to be aquired. This could be overcome though by making use of the fact that CL-MYSQL will correctly pass multiple statements to the server so you could concatenate a large string of updates and execute them all at once.

The good news though is that the API has changed only very slightly in the optional arguments it accepts. However I have changed the way the result data comes back from query. Because CL-MSQL returns multiple result sets it’s necessary to place all of them into a sequence. Additionally, I did not like the way I was placing the column headers into the first item of the result data. It means you always have to allow for it. I considered doing it the way that CLSQL does it by returning the column data in a value struct but I find this awkward to manage. This is because every layer of the API (and client code) must multiple-value-bind the columns out and either repackage them as a sequence or create a new value structure to pass them up the call-chain.

Therefore I have changed the result sequence structure to be as follows:

query-result ::= (<result-set>*)
result-set ::= (<result-data> <column-data>)
result-data ::= (<row>*) | <rows-affected>
row ::= (<column>*)
column-data ::= ((<column-name> <column-type> <column-flags>)*)

I appreciate that this is a little complex, I did consider turning the result data into a struct but this complicates how the user processes the data. For this reason I have added: with-rows and nth-row to simplify the processing of this result data.

Finally, the whole thing is still only SBCL/x86 Linux compatible, that might change .

More information is available here. As always, any feedback is appreciated.

I did though, in the process, create an alternate library to using CLSQL which has support for stored procedures that return result sets, better (IMHO) type inference and is far simpler to deploy. So it wasn’t totally in vain. I intend to extend the library a little to work on the performance and provide a more faithful Common Lisp implementation in the near-future.

Full details of the cl-mysql library are available on this page.

Thanks to the power of Lisp’s macros I am able to make the low-level API hooking dependent upon a compile-time special variable. That way I can generate the additional code for the API hooking, if I want to, or leave it out whilst I’m working on other aspects of the library. Don’t you just love Lisp?! Hopefully we’ll revisit this topic some time later too.

Finally, I realise now that announcing anything on April Fool’s day is possibly a mistake. It seems that the signal-to-noise ratio in the world has gone down quite significantly in the last-few-hours . I guess that makes me the fool then.

Now and again, though, I find myself requiring features that are not exposed in a Super API. They’re not exposed because the feature is either too niche or because it is not standard across implementations. Creating a Super API is in some ways like plastering. Once you’ve created that nice even finish across an entire wall you might find that you’re missing a few light switches.

I was reminded of this the other day when one particular super SQL API: CLSQL uses another super API: UFFI, what they do and how they do it is largely irrelevant. The important point is that I’m only going to be concerned about the operation of one of the combination of technologies (MySQL with SBCL:SB-ALIEN). I’m concerned because there is a particular SBCL feature that I want CLSQL to make use of when it calls SBCL:LOAD-SHARED-OBJECT from via UFFI. Confused? Don’t worry, it’s not that important.

In the end I found a (most heinous) way to do what I want but it occurred to me that there has to be a better way. I appreciate that there is no way that the library writer could know what I want before I want it and neither could they potentially anticipate future underlying API changes to provide for them. However, what if the library writer gave me the option of registering a per-API first-chance callback function? This function could then perform custom processing for my platform combination and then yield control back to the library when it was completed. That way I could inject any code I wanted at potentially any supported depth.

I don’t doubt that it would be difficult to produce an API like this. Certain classes of API (non reentrant APIs being one that jumps to mind) could potentially break in new and unforseen ways and library maintainers would have to be creative about how they detect and cope with errors happening in and around user injected code.

But if the injected code was simply a one-line replacement for the one-line that would otherwise have be called then it might work. Perhaps it might even be fun to try it.

It seems that everyday things have something called ‘affordances’ which should give us mental clues about how they work. The design of everyday things attempts to teach us that poorly designed things lead us down wrong mental-paths and make mistakes when using those things. Mistakes that at best might get our fingers burnt or at worst cost lives.

So far so good. What do you do then when the mental-paths that people are sent down get so worn that they can’t think straight anymore? Of course, I’m talking about spreadsheets again. It seems spreadsheets are so ubiquitous and are so well understood by so many people that some people when faced with technology view it as one enormous spreadsheet. Which is ironic because we know by now that spreadsheets simply can’t be enormous because they don’t scale.

I really don’t think technologists should beat non-technologists up about their lack of technical expertise. That’s just counter-productive and plain wrong. But what do you do when you are faced with someone who has convinced themselves that it’s all one big spreadsheet? How do you explain to them that in my world there is no F9? Where’s that pencil …

The problem is that spreadsheets are a special type of code, and I’m not talking about the Excel ‘macros’ I’m talking about the formulas. As such they probably need to be treated the same way as other types of code, but their very nature makes this difficult. But I’m getting ahead of myself, let’s first look at some of what is good and bad about spreadsheets.

Pros

Spreadsheets are remarkable for their:

• Utility - we can bend them into almost any shape we want because they give one way to represent almost any business process;
• Portability - we can pick up our little gobbets of data and logic and relocate them to almost anywhere inside or outside the company, in file-systems, mail servers and web-sites;
• Simplicity - you don’t have to explain a spreadsheet to anyone. They might have to be a proto-genius to figure out how it works but the working knowledge they would need to get started is pre-loaded in their heads and ready-to-run.

Cons

So they sound pretty useful, and I like to think that I’m a pragmatic guy, so why do I hate them so much? Many have noted about the shortcomings of spreadsheets. The page on spreadsheets at Wikipedia spells it out clearly enough so I’ll paraphrase:

1. Productivity - Working with spreadsheets requires a lot of “sheet-shuffling” to reach the required goal. The bigger the sheet, the more time is spent copying, cutting and pasting cells around.
2. Reliability - Although what consitutes an error in a spreadsheet is subjective, the paper A Critical Review of the Literature on Spreadsheet Errors” (pdf) reveals a series of studies (some more recent than others) that have shown that approximately 5% of cells contain errors.
3. Collaboration - Sharing a spreadsheet is difficult. Having two independent people working on the same sheet and merging their results is as far as I know impossible.

The first two items don’t bother me overly. Yes, it’s a problem but then the alternatives aren’t that great either. Consider what you would do if you didn’t have a spreadsheet to fulfill the task. You’d either do it with a bit of paper and a calculator (i.e. simulate a spreadsheet) or get a programmer to do the task for you. Either way the amount of productivity loss/gain and the amount of errors aren’t going to be that significantly different from using a spreadsheet. Don’t get me wrong, I love my fellow programmer, but we make a LOT of mistakes too. The difference perhaps is that bespoke systems usually end up getting audited (and hence fixed) and spreadsheets often don’t. Although this point is probably moot.

Good + Bad = Too Bad

My real beef is with what happens when you have the ‘pro’ of high portability with the ‘con’ of low collaborative power. You have no way of knowing which version of the spreadsheet you have is the “true” one, and which version is duff. Every copy, whether it be inadvertently through forwarding a sheet by email to someone else or explicitly by taking a ‘backup’ is a 12 foot tall baby-eating, business-crushing monster waiting to rip you and everyone you love apart.

Hug the Monster, Then Run

The thing is we kind of have to embrace the baby-business-beating monster because it’s about all we’ve got. There are some tasks, as a programmer, that I’m really happy that you as the non-programmer don’t bother me with and solve yourself in sheets. Want to set-up an intra-company phone-book as a spreadsheet so you don’t have to bother will all that “Access” voodoo? Be my guest, but I’m watching you. Want to set-up a spreadsheet to run your fantasy football so you don’t have to add two numbers together? Go right ahead, I’ll even drive you to the game so you don’t miss the turn. Want to set up a spreadsheet to calculate payments and and do a mail-merge with the results … STOP. RIGHT. NOW.

The truth is though that you might not know that you’re creating the mother-of-all spreadsheets when you start. I might not know it either but there will probably come a time when a line is crossed and then I will want to know what you’ve been doing and who you’ve been doing it with. I’m just like that.

Unless you are small company (and hence don’t have a lot of choice) you have to be very afraid of trusting anything that might lose you money to a spreadsheet. You need to be very aware of the risks and the potential-costs you are letting yourself in for. Here in Europe there is even a special interest group dedicated to highlighting the risks of spreadsheets. Those guys must throw wild parties …

The Missing Links

In my opinion there is something missing, something that can fill the gap between spreadsheet and system.

I think we need something that can:

1. Track spreadsheet changes - Not knowing which spreadsheet is “true” and which lies (by being able to identify revisions of the sheet that have happened after yours was ‘branched’), and not being able to merge sheets is a problem. Perhaps someone solved it already, if they had that would be great.
2. Track spreadsheets themselves - Having some more information about what sort of corporate-data was being accessed, who was using it and how frequently they ran it might alert us to potential spreadsheet monsters being born.
3. Narrow the gap - Making spreadsheets more like traditional software systems, without significantly castrating the usefulness of the spreadsheet, would be great too. This is a little like asking for the moon on a stick though.

Perhaps I’ll make something like this one day. I have to admit it’s not a terribly exciting project but it has some potential I think. Perhaps I could spice it up by throwing a party and invite the guys from the “European Spreadsheet Risks Interest Group”. Now we’re talking. How will I budget for the 7-up, party hats and streamers? In a spreadsheet of course.

When you’re making software in-house there is a well-known danger of gilding the lilly. Indeed, in-house the law of diminishing returns comes into force if you spend too much time making your in-house project of the very highest quality. The law of diminishing returns can best be described as:

… in a production system with fixed and variable inputs (say factory size and labor), beyond some point, each additional unit of variable input yields less and less output. Conversely, producing one more unit of output costs more and more in variable inputs.

Since I don’t work for a software house, if I can release my code in-house when it is only partially complete, then I should be able to get a lot more stuff done for less cost.

So far so good. A problem arises, though, because I have observed that different programmers place different levels of importance on software quality. This is as probably as you’d expect, one thing we have in common is that we’re all individuals.

The sad truth is that, based on someone’s own personal standards, the bare minimum of what is required to get a job done is usually all that is done. This probably goes someway to explaining the appalling state of some of the in-house software I’ve seen, and written myself. It’s understandable because often, once the main problem has been solved the other issues like usability, maintainability, extensibility and support can be overlooked without any immediately dire consequences.

But here’s the sting, once you release something you usually have to support it too. That’s just the way in-house software works (sucks?), I guess. If you made a poor job of it then you’ll probably pay for it many many times over in support queries. In the end it seems to me that unless you quit (or are fired!) the diminishing deliverable cost turns into a potentially huge support cost. Especially if you end up layering new solutions on to an originally broken solution.

All, in all, you might as well have tried to make something a little more durable and complete at the outset. Sure you can re-factor your mistakes later, but even refactoring costs a lot more to do later than it does to get it ‘right’ at the beginning.

It seems to me that it is essentially a problem of planning. In-house software projects, big and small, aren’t usually planned properly. As a result secondary factors that would improve the overall quality are not included in any estimates. As a result of this bad planning those in-house projects are often late and buggy.

This problem is not getting any better. Years of software-development in an in-house setting have shown me that where in-house plans & design meetings exist issues of usability, maintainability and support are very minor concerns if they are concerns at all. Perhaps that should change. Just a little …