Parcell's Posts

Thoughts on If this, then that

Sat, 17 Sep 2011 14:27:00 -0700

My friend George pointed out a link to an Infoworld article about a new startup, If this then that

I've been looking at internet-wide event processing for a little while. My feeling is that messaging and event-processing have been big and useful in enterprise IT for a while, but that so far there hasn't been much on the wider web ecosystem. And we are increasingly generating and consuming events - facebook status updates, google+ updates, location data, share prices, real-time sports data. So my hope has been that this is an area that can grow up and offer us some great new capabilities in the next few years.

So if this then that looks interesting. Unfortunately, it appears their site is down right now, so I need to follow up further, but my initial impressions from the infoworld article are

This is cool, and not too far from the lines I was thinking - I hope it generates some interest.
It reminds me a little of Yahoo's pipes, which was cool, but never really went anywhere. Albeit an event-based version, so as SQL is to CEP, pipes is to this I guess, in my mind.
That it relies on ifttt people hooking into your site. I hope I am wrong about this - it would be a shame for me to have to build a userbase of tens of millions of users before it's worth their while hooking into my API.
That it's proprietary, and that's reasonable given the amount of IP in those integrations. But then it depends on their integration with various APIs, and your event-processing remains locked in their system. This is a shame, and that's why I hope that underlying protocols for event dissemination get standardized, so that systems like this can become commoditized.

In my mind, what is needed is a standard for events, as RSS standardized blog data etc. So that just as google reader doesn't have to have different processing for wordpress vs blogger vs slashdot vs xkcd vs ... then a given event processing program, library, add-in or extension wouldn't have to speak facebook, google+, twitter, yahoo finance, google finance, betfair, formula 1, etc.. Given that I think there is scope for event processing in the cloud, on the desktop as a stand-alone application, on the desktop integrated with the OS, on the desktop as an add-in/extension to various programs eg Excel add-in, a notification programme, and integrated with the OS on mobile, the sheer number of data sources and consumers, tends to suggest to me that events won't really reach their full potential until there is a common protocol in the middle.

Anyway, these are just my thoughts, and ifttt seems like an exciting step increasing general excitement around events, and hopefully a baby-step to a brave new world where consuming, processing and emitting events across sites seems as commonplace as looking at a blog in google reader.

Permalink | Leave a comment »

How I added my Hacker News saved stories to my Posterous sidebar

Wed, 15 Sep 2010 10:23:00 -0700

I acknowledge that adding something to the sidebar of my Posterous blog is not going to set the world on fire. The real meat here is that I have created a nascent library to pull data from the Hacker News website and I'm interested to see what all the creative people that hang out over there do with it. The code is available on github - help yourselves.

If you interested in software, or startups and don't already read Hacker News, you should. It's what reddit was before it got big, and what slashdot was before it got big before that. Fortunately, Paul Graham know what's what, so Hacker News has proved more robust.

I've spent quite a bit of time following articles on Hacker News, upvoting the ones that I like, and know I'll want to read again one day. It occurred to me that it would be awesome to make the stories I've upvoted available as a side-bar on this site. In this day of RSS that should be easy right? Wrong. Hacker News is deliberately basic, partly due to PG's time-constraints, and partly I suspect because keeping it basic means only people actively interested in the subject matter go there. As a result, it doesn't have an RSS feed for the list of your saved stories that it maintains. And Posterous doesn't seem to have any option to add an RSS feed to a side-bar anyway.

Architecting and building this little monstrosity

I have a couple of virtual private servers that I keep around to perform tasks like this. The plan here was to have a VPS periodically scrape the Hacker News site, and generate a static HTML file, which could then be included in the Posterous sidebar.

My choice of language for tasks like this is Python. There are a stunning number of libraries built into it, and it's easy to augment its power using packages from the PyPI repository. I'm sure that Ruby and Perl would be similarly capable, but this is definitely a task best suited for a scripting language.

I chose to use the Mechanize library to control interaction with the Hacker News site. Mechanize provides you with a 'browser' object that can be easily told how to fill in forms, or to click a link with particular text. This relieves the strain of having to keep track of state between calls to an HTTP library, and having to parse the returned HTML. This made it a lot easier to handle the stream of hidden forms and clicks that have to be done to log in to Hacker News using clickpass.

The library wraps around the Mechanize browser object, allowing the library user to easily log in to Hacker News (currently only available using a google login - patches/pull requests welcome if you'd like to contribute). Once logged in, the user uses the nav_* functions to navigate to the appropriate page, and the get_current_page_stories function to get a list of dictionaries, each item representing a story. An example interactive session might look like this:

>>> import hnscraper >>> h=hnscraper.HNScraper() >>> h.login_with_google('USERNAMEGOESHERE','PASSWORDGOESHERE') >>> h.nav_saved_stories() >>> stories=h.get_current_page_stories() >>> list(stories)[0] {'score': 254, 'link': u'http://adgrok.com/why-founding-a-three-person-startup-with-zero-revenue-is-better-than-working-for-goldman-sachs', 'user': u'cjg', 'title': u'Founding a startup with zero revenue is better than working for Goldman Sachs', 'ord': 1, 'comhead': u' (adgrok.com) '} >>> h.nav_more() >>> stories=h.get_current_page_stories() >>> list(stories)[0] {'score': 61, 'link': u'http://www.aaronstannard.com/post/2010/06/12/The-Myth-of-the-Single-Person-Startup.aspx', 'user': u'Aaronontheweb', 'title': u'The Myth of the Single-Person Startup', 'ord': 31, 'comhead': u' (aaronstannard.com) '}

The get_current_page_stories function uses the BeautifulSoup library to parse the currently open web page. BeautifulSoup provides a pragmatic interface for pulling data out of HTML pages, allowing access through the tree of tags using dictionaries and lists, which allows you to quickly drill down to the data you need. For example, to get all the <td> tags for a particular row:


>>> s.findAll('tr')[49].findAll('td')</div>
<div>
<div>[<td align="right" valign="top" class="title">16.</td>,</div>
<div> <td><center><img src="http://ycombinator.com/images/s.gif" height="1" width="14" /></center></td>,</div>
<div> <td class="title">Cold Calling Techniques<span class="comhead"> (businessballs.com) </span></td>]</div>
<div>

As there is no class assigned to indicate which table rows represent stories, the get_current_page function naively looks through all table rows in a page, trying to build story dictionaries for those that are in a plausible format. This works pretty reliably in practice.

Building a page for Posterous to include

Martin Ruiz has a great blog post on how he uses a similar technique to add items to the sidebar of his blog. He uses a dynamicly generated webpage, whereas I build a static webpage to serve - in the event that I hit slashdot, maybe this saves my server from melting.

The page itself is included in Posterous using an iframe added to my custom Posterous template:

<section id="hackernews"> <h1>Articles I've liked on Hacker News</h1> <div class="sidebar-block"> </div> </section>

The key thing to remember is that the included page does not inherit any of the CSS from the page importing it. To get the look and feel right, I opened up my posterous home page in Chrome, and fired up the Developer Tools. I can use this to see what CSS is getting applied to each element on a page, and replicate that in the page I will be generating.

I used the mako template library to build the final output - this is very similar to using a templating engine to dynamically generate content. I built a script, make-html.py, to drive the HNScraper, pass the scraped data to Mako, and save down the output ready to be served.

The End Result

You can hopefully now see the items I've found interesting in the sidebar on my blog. The fantastic libraries available for Python meant that I just had to write a bit of glue code to get this working - 170 lines in total. Working out how to get Mechanize to push through clickpass took a little time, as did getting the parsing right on anonymous-looking tables using BeautfulSoup, but all-told, I've probably spent longer on this blog post than writing code.

As I said, I've made the code available on github. I hope you find it useful, and I'd love to see what mash-ups people can use it for.

Permalink | Leave a comment »

Only a moron would use a SQL database in 2010

Mon, 06 Sep 2010 09:32:00 -0700

There you go: in time-honoured blogging fashion, I've used a highly inflammatory title to get your attention. Hopefully now I'll have got that out of my system, and the rest of this post can be reasoned and insightful.

A friend drew my attention to this animation at Alex Popescu's excellent myNoSQL blog. It features a pig and bear (?) arguing the merits of relational versus NoSQL databases. The bear is a NoSQL fan-boy, parroting lines such as "shards are the secret ingredient in the web-scale sauce - they just work". The pig condescendingly suggests /dev/null may be a better place to store data, performance-wise, and rants that he will move to farm rather than work in the same industry as the bear. The debate is not moved forward.

Now, the relative merits of relational and NoSQL databases is well trodden ground by now. Relational databases are reliable and proven versus the scalability and flexibility offered by schema-less NoSQL databases. Each has their place. There are some situations where reliability, and transactions, and all the other relational goodness is important - bank accounts are the standard example. Conversely, if you need to scale cheaply, and no-one cares all that much about a few missing tweets, likes or status updates, then NoSQL is the obvious solution. There are gray areas in the middle, where either or neither may be appropriate. But it is hard to argue that there is no place for one or the other, and you'd have to be a moron to make that choice (regardless of title of this blog post).

So why the animosity between the relational database crowd and the NoSQL crowd?

After all, there are plenty of areas where there is more than one solution. No one argues vociferously that I must use Google or Bing for all my searches - sometimes one is better, sometimes the other. Sometimes Excel is a more convenient tool for analysis, sometimes Matlab would be a better choice, and everyone seems happy with that.

Although the argument is often framed in terms of facts - "MongoDB is web scale", "You might as well write to /dev/null" - I think it is more telling to look at the interests of those making the arguments. Well, one difference between the examples above and database technologies is that there is a group of people with a vested interest in relational databases: DBAs.

I suspect that DBAs, en masse, think that NoSQL databases will ruin their livelihoods with their lack of schemas and tables to CREATE and UPDATE. On the other side of that, web developers think they would like to cut out the bureaucracy of endless CREATE and SELECT statements as they make web applications, and as extension of that, not having to defer to a DBA on matters of database style, and schema-less databases seem to promise that.

I understand the defensiveness, and I understand the desire to make dealing with data easy. But I think both of these arguments are wrong.

Dealing with large amounts of data has never been easy. DBAs are valuable not because they now they know every clause a SELECT statement can have, but because they understand data, and particularly the data under their care. They know what indexes are going to help performance. They know how to defend against the application layer inserting junk rows with inappropriate NULLs, or against application programmers who want to make a column called "COL_1" to contain comma-separated data because it would just be convenient. A good DBA can free developers from thinking about data-quality issues, and act as a multiplier on development productivity.

This is not going to go away just because some applications run against a MongoDB back-end, rather than a MySQL back-end. Sure, the commands will be different, but the skills will not, and much of the knowledge and experience transfers.

In short, I think data volumes are exploding, both for relational and non-relational databases, and the DBA skill set will be in higher and higher demand over the next few years. Hopefully, when this comes to pass, we can stop the defensiveness, stop the fan-boyishness, and just use the best tool for whatever job we need to do.

Or we can use CSVs and VBA.

Update: My friend George has written a thoughtful response to this post, arguing that NoSQL solutions will benefit from standardized query languages, and query planners, and that it may be as easy to adapt existing solutions to be distributed as it is to re-invent the wheel.

Permalink | Leave a comment »

A new low (number of bytes in your program)?

Fri, 13 Aug 2010 02:35:00 -0700

Occasionally it's fun to get away from "serious" programming and do something just for fun. Serious programming these days tends to require using several, possibly tens of external frameworks and libraries. That's fine, and you get a lot more bang per line of code that way. But it can be fun to go back to basics, and make something from scratch.

With that in mind, I've been looking at the JS1k competition. The idea is to do something interesting in under a kilobyte of JavaScript. The great thing about this is that you can get started with just notepad and a browser. The new canvas element in HTML5 makes it surprisingly easy to do interesting graphical effects in JavaScript. But it requires some ingenuity to get the size down, particularly any data you might want to use is probably better generated procedurally.

My effort is a spinning tetrahedron. I enjoyed Steven Wittens description of making his demo, so I thought I might describe one aspect of mine here.

My initial effort ran to over 2k, so I looked at a few techniques to get the size down. Some were pretty obvious: changing variable names to be single letters helped. Inlining functions further reduced the size to 1.3k - each function requires you use at least "function" and "return", but you also get quite a lot of redundancy assigning the results to a local variable for example.

One method that helped, and is a bit more interesting, was to generate the data, rather than store it. The data in the case are the vertices and the line list [lines from vertex 0 to vertices 1, 2, 3, from 1 to 2 and 3, and from 2 to 3]. Taking the vertex list first, one way to represent it would be as a list of lists, with each inner list having 3 items representing the x,y and z coordinates, using 65 bytes:

P=[[-0.5,-0.5,-0,5],[-0.5,0.5,0.5],[0.5,-0.5,0.5],[0.5,0.5,-0.5]]

Accessing a coordinates using something similar to "P[i][0]" uses 7 bytes, and this will be done once for each of the x, y and z coordinates for a total cost of 86 bytes - a not insubstantial portion of the 1024 allowed.

A better way is to not explicitly store the data, but to try and use bitwise operators to calculate it. Looking at the first coordinate, we can see that it is that it is -0.5 for the 0th and 1st vertices, and 0.5 for the 2nd and 3rd. I notice that 0, 1, 2 and 3 in binary are 00, 01, 10 and 11, and that the second digit, less a half, would be precisely what I want. So I can replace "P[i][0]" with "(i/2&1)-0.5". Similarly, the second coordinate is the first digit, minus a half, and so "P[i][1]" can be replaced with "(i%2)-0.5". The final digit is a little harder - it is 0.5 when i is 1 or 2. One way to think of this, is that it is 0.5 when precisely one of the binary digits is 1 - so this sounds like a job for xor. Sure enough, the final coordinate can be represented by "((i/2^i)&1)-0.5". Each of these expressions is longer than using the indexing operator on P, but it is now possible to throw away P. In fact, using this representation uses 34 bytes rather than the 86 used by explicitly creating the vertex data.

Similarly, the line list does need to be stored explicitly. In a pyramid, every vertex is connected directly to every other vertex, so it is sufficient to use a double-loop over the vertices "for(i=0;i<4;++i)for(j=i;j<4;++j)".

With a tetrahedron, these savings a relatively small. For a cube, with twice as many vertices, they are larger: the vertex list can be stored by using the jth binary digit to represent the jth coordinate, and the line list can be represented by double-looping over each pair of vertices again, but only drawing those where exactly one binary digit differs [ie vertex a and b are connected if a^b is a power of 2, which is true if (a^b)&((a^b)-1) - check bit-twiddling hacks for more bitwise goodness]. For an icosahedron, it is not immediately clear what a good representation

Permalink | Leave a comment »

Thoughts on the Barclays Cycle Hire scheme

Wed, 11 Aug 2010 06:58:00 -0700

I just had my first ride on a Barclays Cycle Hire bike. Here are some random thoughts on it:

I think they should be called Barcycles. I like portmanteaux.

It's less scary riding in London than I thought. I used to ride occasionally in Cambridge, but mostly on back-roads, so I always found the thought of cycling in London pretty terrifying. While I chickened out of joining 3 lanes of traffic on Euston Road on my first ride, cycling with traffic wasn't as scary as I expected - I expect that traffic is more wary than usual of people on.

The bikes themselves are pretty good. Perfectly pleasant to ride for half an hour or so. I've never driven a carbon-fibre featherweight bike, so I don't know what I'm missing, but I have walked a long way in London, and these bikes are definitely faster than that.

FipLab's London Cycle: Maps & Routes app is cool. I looked at a few contenders, and all seemed reasonable. FipLab's effort seemed best. I just sent an email to the company, and they responded in a minute, letting me know that they are looking at colour-coding the pins based on bike/space availability in the near future. Awesome.

There are a few organization teething problems. I couldn't find a docking space at the dock nearest my work. I understand that there is a large inward flow of bikes in the morning, and and a large outward flow in the evening. I hope that this will get dealt with in time, as demand stabilizes, and TfL can better plan how much redistribution is needed. In the meantime, it's a little frustrating to have to walk back from a distant cycle station.

Very positive for cycling in London. This makes it a lot easier to cycle in London. No need to find somewhere to keep a bike now. More importantly, I think it stands a good chance of creating a virtuous circle which fixes the lack of cycle routes in London. More people cycle, so more cycle routes are created, so more people cycle.

Barclays got themselves a good deal. Put in £25m over 5 years, less than half the estimated cost for the first year of the scheme, but get their logo and colouring plastered all over London. I've no objection to the corporate sponsorship - if anything it's positive - but Boris's team could have pushed for a better price.

Cycle shops could take better advantage. I went to a couple of shops to look at buying a helmet. I asked the very helpful guy at Cycle Surgery if there had been many people using the Barclays Cycle Hire scheme coming in for helmets - he didn't know. With 32,000 members in the first week, that has to be the biggest upsurge in cycling the capital has ever seen. Ok, so not many of these people are going to spend £1,000 on a bike frame, but I would expect that most of these people would value the contents of their skulls at more the £50 a reasonable helmet costs. The cycle stores could pushing this. Stick a board outside the store proclaiming that you sell helmets for people using the scheme. Pay people to hand out flyers by cycle stations offering a discount on helmets, or even just describing what you have. But it seems like they are blithely ignoring it.

Be interested to hear of your experiences, or any hints and tips to get the most out of the scheme.

Permalink | Leave a comment »

How to pick a product name

Mon, 02 Aug 2010 11:22:00 -0700

I just read Jason Cohen's advice on the Smart Bear blog about how to pick a company name. It's excellent advice, and I wish that article had been written before I was trying to come up with a name for Cobalt Quantware last year. It also reminded me that I wanted to write a blog post about the process of coming up with a product name, which I have just been through.

The core of Jason's advice is that the name is a lot less important than what the company does. I hope that is true, and I think it is. It is good advice, because it takes the pressure off you to think of the perfect name. Nonetheless, when you fire up your IDE or editor of choice to create the new project or solution that will be your product, it will ask you for a name. So you shouldn't put pressure on yourself to come up with a perfect name, but you do need a name.

Don't: Use a half-assed Product Name

My first instinct was to use the first name that came into my head, and hope that I thought of something better later. I spent about 3 minutes coming up with the name. This was a mistake for several reasons, and so I'd advise you to avoid doing that.

I was never really keen on the name. This reduced my engagement with the project. Not by a huge amount, just marginally. Nonetheless, trying to bootstrap a new business and create a product is hard. It is not wise to throw away motivation.

Customers and advisors were not keen on the name. One of the repeated comments I would get is that the product looked useful, but the name wasn't great. Customer and advisor feedback is invaluable. It is a waste of both your time for you to explain why you have a crappy name that you intend to change. Worse, it is a wasted opportunity if they are telling you something that you already know. Far better to fix it, and hear about the feature that they really wish was there that hadn't even occurred to you.

Do: Use a codename if you can't come up with a Product Name

Later, I adopted a codename for the project. A codename is just a cool-sounding word that is a temporary placeholder for the final name.

I decided to use names of British railway stations for codenames, using a station with a cool-sounding name beginning with "A" for the first codename, "B" for the second codename, and so on. There are a hundred themes you could use: cool-sounds birds, cool-sounding cities, cool-sounding metals etc.. This can be done pretty well without getting others involved, and without taking a large amount of time.

When I did this, I started getting better quality feedback. Customers knew that this wasn't the final name, so it wasn't up for discussion. We could move on to discussing important topics, like what they wished the product would do.

Do: Come up with a Product Name as early as possible

As well as a solution/project name for the codebase, you may also need a name for JIRA, or other bug tracking software, for a wiki, for documentation. Some of those places can be easily changed. Some less so: you can't easily change the project code used by JIRA for emails, for example. The sooner you come up with a final product name, the fewer places you will need to change, and the fewer headaches you will have making those changes.

Later, but before release, you will need to start creating marketing materials - a document describing your product to send to potential customers, for example. At this point, you pretty much need a final product name. Your product name will be in large type at the top of that document. You don't want an asterisk next to it.

Eventually you will release your product, and at that point you will be pretty irrevocably committed to the name you have used. Yes, products change their name, but they lose a lot of brand recognition when they do.

Do: Get a group together to brain-storm for Product Name candidates

I naturally tend to gravitate towards solving problems on my own, rather than in a group. I tried to do that with the problem of coming up with a product name. That worked badly. I partially filled a whiteboard with suggestions that I wasn't happy with. One of my advisors suggested than in a large company she had worked for, they would typically gather a group to brain-storm for product names. That was great advice - this is a type of problem that screams out for group brain-storming.

In brain-storming, you have two aims: to suggest ideas that could be good solutions, and to suggest things that could trigger ideas in others. It's important to create a positive environment where people feel free to suggest anything, no matter how off-the-wall, and it's important not to criticize or evaluate ideas at this point. There are plenty of good resources on brain-storming on the web. I'd recommend businessballs.com for this and other topics.

One problem for me was that I'm a sole-founder, so I didn't have anyone to brainstorm with. I solved this by gathering a group of friends in a meeting room I hired for a couple of hours over a weekend. I traded with them the promise that I would owe each of them a couple of hours (but that I'd appreciate it if they didn't choose tasks that would make me cry).

I decided on a group size of 6, as being large enough to bounce around ideas, but small enough not to be intimidating. I deliberately chose friends that didn't know each other. This helped us focus on the task, rather than pal around, and it maximized the variation in suggestions. We also had lunch together afterwards. I think everybody had a good time. Common friends, unsurprisingly, got on well, and people enjoyed the unusual exercise.

We generated several hundred suggestions, many of which I was very happy with, and all it has cost me so far is building a barbeque and helping a friend move.

Do: Filter down to a shortlist

I transferred the several hundred suggestions from flipcharts to a word document. From the several hundred suggestion, I filtered through several times, first down to about 100 names, and then to a shorter list of around 25, and finally to 10 or so names. I did this at different times, to ensure that the names that made it through weren't purely based on a particular whim at the time.

I also found that some suggestions were fragments - either a good descriptive name for the product, or a cool-sounding adjective that could be incorporated into a product name. I tried to save these as I filtered, and combined them to form a few extra names for the shortlist.

I checked the shortlist to ensure that I wasn't infringing on trademarks, and that the names were not already in use, ideally for any product, but certainly for similar products, or other products that our customers might already use.

In the final filter, from 25 to 10, I did not filter purely on quality - I was pretty happy with most of the 25 - but as well to ensure some variation in the final set. I would rather get feedback on more variable set of names, than a higher-quality set of very similar names.

Do: Get feedback on the shortlist

I gathered feedback from about 50 people over the course of a week or so, including customers and potential customers, advisors, friends, family, people I had just met, and so on. Everyone has an opinion on your shortlist, and most people are happy to give it.

The easiest way is just to ask if they'd like to see the shortlist of names for your product (you were talking about your product anyway, right?), present them with the list, shut up and make notes. Do shut up, because people will often volunteer things like liking one part of one name, and one part of another, which they won't if you just ask them which their favourite is. Do make notes, because you won't remember what someone said about 10 different names over dinner. In a pinch, I have recorded people on my phone, but people are generally a little more reticent to be recorded.

When you get back to base, systematically organize the feedback. Hopefully, it should quickly become apparent that there are a few names that people are responding positively to. Remember, these are being positively evaluated with respect to the names that you preferred out of a large set.

Final Thoughts

The process of coming up with a name was not actually that time-consuming. In hindsight, it is pretty obvious too. But coming up with a process that worked well for me was a little trickier. In particularly, the key realization was that as a sole-founder I would need, and could get, others to help on this task. I hope that if other people are stuck on this task, or on other tasks, they can use my "mini-crowdsourcing" experiment as inspiration.

For me, having a final product name that I was happy with, and my customers liked, was a big punch-the-air happy moment. There was a huge contrast: Not having a product name was an increasing source of anxiety as we moved towards launch. Having a product name really renewed my energy and engagement towards the project.

I'm really excited about Cobalt Quantware again, and about our first product. We currently in stealth mode, but I look forward to filling you in later, at an appropriate time.

Permalink | Leave a comment »

NRN: How about a convention for emails that don't require responses?

Thu, 01 Jul 2010 08:01:00 -0700

NRN = No Response Necessary

A few years back, I was thinking about learning to fly. Other things got in the way, and it didn't happen, but it's something I may go back to in the future. That didn't stop me massively geeking out around flying though. One of the things I liked to do was listen to air traffic control online. It's actually quite relaxing as background noise, in the same way that listening to the shipping forecast is. I highly recommend http://www.atcmonitor.com/, where you can see live radar around Atlanta as well as listen to the air traffic controllers.

One of the things I've noticed is that air traffic controllers and pilots don't waste a lot of time when talking on the radio. They need to say what needs to be said, and then stop talking. They are busy, and need to focus on flying and directing planes, and there is probably another person who needs to transmit a message.

It's quite common for them to finish a transmission with "Good day". The implication is that they have nothing further to say, and aside from the other party repeating what they have just said to confirm, they do not expect to speak again. They are not being rude, but busy, and considerate of the next guy who needs to speak.

In a lot of ways, it would be nice to have a similar convention in other avenues of life. Sometimes emails seem to bounce politely backwards and forwards, long after everyone has run out of things to say.

So I'd like to propose inserting "NRN:" in the subject line of an email to indicate that the other party is not obliged to respond. It's short for "NO RESPONSE NECESSARY". The idea is that I won't be offended if you don't respond.

So if I you send me an email asking a question, and I give you an answer, I could stick "NRN:" in the subject line. You are relieved from any duty you might have perceived to send me a thank you email. And you don't have to flag the email for further response. You can if you want, or have something else you need to say, but don't feel obliged. Don't worry - we are done here and everything is fine.

Permalink | Leave a comment »

Getting started with RabbitMQ in Windows using C#

Wed, 30 Jun 2010 11:43:00 -0700

In a previous blog, I wrote how excited I was by ZeroMQ, a very lightweight brokerless messaging system. I wanted to also try RabbitMQ using C# on my Windows development machine, but couldn't find any step-by-step introduction to get me to the point of running my first program. I'm there now, so I thought it would be good to remedy that.

I'd like to note that I'm not a RabbitMQ or messaging whiz. What you see here may or may not be best practice. If it's not, hopefully someone will chime in with some comments to put me back on the right path.

Here are the steps I took to install RabbitMQ and create a couple of simple C# clients to check that everything was working.

Download the Windows bundle zip file from the RabbitMQ download page. The current version at the time of writing is 1.8.0. It contains the Erlang installer, the RabbitMQ server, and bindings for Java and C#.
Unzip the bundle.
Install Erlang using the OTP installer by running otp_win32_R13B03.exe from the unzipped bundle.
Create a subdirectory "RabbitMQ" in "C:\Program Files". (Or "C:\Program Files (x86)" if you are running on 64-bit).
Unzip the rabbitmq-server-windows-1.8.0 zipfile in the bundle to a directory rabbitmq-server-windows-1.8.0.
Copy the rabbitmq-server-windows-1.8.0 directory you just unzipped to the RabbitMQ directory you just created. Copy the rabbitmq-server-windows-1.8.0 directory itself rather than its contents. ie You should have a directory "C:\Program Files\RabbitMQ\rabbitmq_server-1.8.0" when you have done this.
You need to add ERLANG_HOME to the environment variables for your system so RabbitMQ can find Erlang. I can't remember precisely how to do this in XP - I think it's in Control Panel...System... somewhere, and I don't have Windows 7 yet. Perhaps someone could comment for these versions of Windows. In Vista, this is done by:
1. Open the control panel.
2. Search "environment". As you type, the option to "Edit the system environment variables" should present itself. Click this - you will be asked for permission to take this action.
3. In the System Properties...Advanced tab which appears, click the "Environment Variables..." button.
4. Click the bottom "New..." button to add a system variable.
5. Enter "ERLANG_HOME" as the variable name.
6. Enter "C:\Program Files\erl5.7.4" as the variable value for 32-bit systems. Use "C:\Program Files (x86)\erl5.7.4" for 64-bit systems.
7. Click Ok, Ok, Ok and close the control panel.
Run the server by running "C:\Program Files\RabbitMQ\rabbitmq_server-1.8.0\sbin\rabbitmq-server.bat". A command-line window should appear. When it says "broker running" the server is running ok.
Install the .Net libraries by running the rabbitmq-dotnet-client-1.8.0.msi installer in the unzipped bundle.

You can also install RabbitMQ as a service. The RabbitMQ documentation has more details on this.

Ok, so now you've installed RabbitMQ, and you have a RabbitMQ broker running (that's the server that all your message producers and message consumers will connect to). Before going any further, it's worth taking a little time to come up to speed with the concepts behind RabbitMQ. Dmitriy Samovskiy has an excellent, and quick, set of introduction slides which should tell you what you need to know (when I clicked publish, I saw that Posterous automatically embedded this - kudos, Posterous crew):

In case you are too impatient to read through that, the gist is that a publisher application sends messages to an exchange, which sits on the server. Message queues sit on the server and filter messages from exchanges. Finally a consumer application consumes messages from a message queue.

A simple publisher

Let's start by writing an application that publishes a message each second which contains "Hello World":

Start a new project in Visual C#, called "RabbitMQPub".
Add a reference to "RabbitMQ Client Library for .Net", by right-clicking the "References" entry in the Solution Explorer, clicking "Add Reference...", selecting the ".NET" tab, and double-clicking the "RabbitMQ Client Library for .Net" entry.
In the program you will want to add using statements to bring in the appropriate RabbitMQ namespaces:
using RabbitMQ.Client;using RabbitMQ.Client.Framing.v0_8; </span>
The first step is to create a ConnectionFactory, and configure it:
ConnectionFactory factory = new ConnectionFactory(); factory.Protocol = Protocols.FromEnvironment(); factory.HostName = "localhost";
There are other configuration items, which are taking default values here. This will work on a virgin RabbitMQ instance, but in production you should look at setting up access control and VHosts.
Then from the ConnectionFactory, create a Connection object:
IConnection conn = factory.CreateConnection();
And from the connection, create a Channel object:
IModel ch = conn.CreateModel();
We need to create an exchange to publish our messages to:
ch.ExchangeDeclare("exch", ExchangeType.Direct);
And a queue to filter messages from the exchange. A queue is made to filter messages from a particular exchange by "binding" it to that exchange:
ch.QueueDeclare("queue"); ch.QueueBind("queue", "exch", "key", false, null);
Then we loop, using the Channel object's BasicPublish method to publish a new message every second, and printing a message to the display to indicate we have done so.
while (true) { Console.WriteLine("Sending message."); byte[] messageBody = Encoding.UTF8.GetBytes("Hello world"); ch.BasicPublish("exch", "key", null, messageBody); Thread.Sleep(1000); }
Note that we are expected to present byte arrays to send in a message. When sending strings, it's a good idea to use UTF-8 encoding as shown.
Finally, we close the channel and the connection. This will never execute in this case, and the channel and connection will be shutdown by program termination instead, but if you are writing a client that isn't guaranteed to loop forever, this is how you should terminate the connection:
ch.Close(Constants.ReplySuccess, "Closing the channel"); conn.Close(Constants.ReplySuccess, "Closing the connection");

A simple consumer

We now need to create a simple application to listen for these messages, and print them.

Create a new project Follow steps 2-6 and 10 above. This should connect to the RabbitMQ broker, but otherwise is inactive.
Add code to consume messages from a queue:
QueueingBasicConsumer consumer = new QueueingBasicConsumer(ch); ch.BasicConsume("queue", null, consumer);
In an infinite loop, listen for messages, decoding each one, and printing its body
while (true) { BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue(); IBasicProperties props = e.BasicProperties; byte[] body = e.Body; Console.WriteLine("Received message: " + Encoding.UTF8.GetString(body)); ch.BasicAck(e.DeliveryTag, false); }
Note that messages must be acknowledged using BasicAck, or the broker will resend them.

Source code

I have made the source code for these projects, as well as other sample projects, through GitHub: http://github.com/edparcell/Samples-CS

Where's the Fudge?

In a previous post, I mentioned that I would like to give an example of using Fudge to encode messages. That is still my intention. However, I thought this blog post had become long enough already, so I'd like to defer that to another post.

Comments

Some random comments on RabbitMQ so far:

The bundle seems quite large. I know the bulk of this is the OTP installer, but it makes it quite hard to consider embedding RabbitMQ in a lot of small-scale cases, such as deploying to a single end-user machine. Not a problem for an enterprise perhaps, but this is the case I have right now.
[Update: I'd like to clarify that 80 megs is not a huge download by modern standards, especially for such a useful and full-featured product. Just that it would look odd to my end-users if I replace the named pipes in a few programs with embedded RabbitMQ, and the installer balloons from 1 meg to 80 megs].
Would be nice to create an installer to automate these steps, so installation is simply clicking "Next" a few times.
The .Net user guide seems a little out-of-sync with the latest library in places. Not terribly, but it throws you a curveball sometimes, such as suggesting that ConnectionFactory.CreateConnection takes 3 parameters, when in the latest version you should set some member variables on the factory and call the method with no parameters.

Overall, RabbitMQ seems like a great application, albeit a little tough to get started in on .Net. Hopefully this short tutorial will help with that.

I'd gratefully welcome any corrections - as I said, I'm no messaging guru, so if I've met any errors, let me know and I'll correct them.

Permalink | Leave a comment »

Excited by ZeroMQ

Tue, 29 Jun 2010 01:42:00 -0700

ZeroMQ looks like an interesting project. It's a light-weight messaging stack, supporting point-to-point, publish-subscribe and other topologies, without a broker.

ZeroMQ aims to sit at a slightly higher level than raw sockets. It's easy to write a socket "Hello World" server and client. The server prints whatever it receives, and the client connects to the server and emits "Hello World". Scaling up to real world needs is hard though. Sockets tend to fall over. Messages certainly will be fragmented. Your socket may be handling several clients at the same time. Messaging layers such as ZeroMQ aim to provide an abstraction, so you can put a string or an object in one end, and have it come out the other with minimal fuss.

In the past, I've looked at ActiveMQ, and RabbitMQ. Both seemed interesting, although as I understand it, both require a broker i.e. a server. This is fine for enterprises, the main target of messaging systems to date, but a little trickier for an individual - sure you can run a broker on your machine, but it's just something to be down at inopportune moments.

As far as I can see ZeroMQ is to Active/RabbitMQ as SQLite is to MySQL or PostgreSQL. You can embed ZeroMQ in your applications, and get the benefit, but the user doesn't have to care, and critically, doesn't have to run a server on their desktop.

And I think that having personal message systems on the desktop is exciting. I already have one, sort of, kind of. Growl is an application that sits in the background listening for "messages". Various applications support growl to pop up status messages. So when a friend comes online, Growl pops up a bubble to notify me. Or when various things update. It's a nice and useful program, but it's not really a messaging system - the "messages" are one-way, and do one thing: pop up on the screen.

A real messaging system goes further. One program could listen for a certain type of message to pass on to Growl. My servers could send messages detailing their status. A third could monitor those messages to check whether limits were breached, and send messages for Growl if so. A library for Excel could monitor messages, and give the latest value for various things, even updating in real-time, allowing quick-and-easy status board. Messages could be sent for stock and other financial data.

People are already doing all of this, of course. But you need to be an enterprise, or prepared to act like one, to use an enterprise messaging system. ZeroMQ will bring the bar down, and in the past, when that's happened, there have always been exciting new products. I hope I've given some, but, as usual, I expect the most exciting new uses will be the ones that we can't even see at the outset.

I've been playing a little with ZeroMQ at weekends, and in a future post, I'll describe how to install ZeroMQ, and give some simple examples of how to use it.

While I'm talking about messaging, I'd like to give a quick mention to the Fudge messaging project who are building libraries to support a slick binary format for encoding messages.

Permalink | Leave a comment »

I've put some C# code samples on github

Sat, 26 Jun 2010 01:15:00 -0700

When I have to use a new library or functionality, I tend to build a minimal console application to check that I can get something working in isolation from the codebase it will eventually be integrated into. My first blog post grew out of a sample I was building to try out JQuery and CouchDb. Often, there's good sample code out there on the web, but sometimes not. In any case, I often need to do something different to the other guy, so it's as well to make my own sample.

In the past, I've tended to throw away these samples once they are working, for lack of anywhere reasonable to put them. Recently though, I've noticed that often, great working code samples are available on github, so I've decided to join the crowd there.

My first github repository - Samples-CS - is set of code samples for various things in C#. Currently there are simple console applications to encipher and decipher using AES, convert a PNG to an ICO, and a simple system tray application. They are released under an MIT licence. I'll be adding more as things come up, as well as repositories for other languages.

Hopefully they will be useful to someone who needs to do the same things as me. If not, I lose nothing, and I can refer back to my code samples later if I need to.

Permalink | Leave a comment »

Mockup of EasyInstaller - an "App Store for Excel"

Wed, 19 May 2010 16:42:00 -0700

In my previous blog post, I talked about an "App Store for Excel", allowing Excel users to download extensions from a central repository on the web. A picture being worth a thousand words (and more likely to get looked at), I thought I'd put together a mock-up of how this might actually look:

I also coined the name EasyInstaller. Let me know what you think of the idea and the name.

On a separate note, I used Balsamiq to create this mock-up. I recommend it, in that it's very easy to use. I'm currently weighing up whether I want to spend $79 on it. It's a great piece of software, and I love that I can do application interfaces as well as websites, but I rarely mock up interfaces. In the meantime, does anyone have any recommendations for other mock-up software I should look at?

Permalink | Leave a comment »

How about... an "App Store for Excel"

Fri, 07 May 2010 10:41:11 -0700

In my previous blog post, I looked at the complementary roles of open- and closed-source development in a software ecosystem. In particular, I focused on how the lack of a fully interoperable alternative to Excel is hurting innovation in the spreadsheet ecosystem. In this post, I'd like to look at a way of stimulating that ecosystem.

I'd like to propose an "App Store for Excel". That is, an equivalent of Apple's App Store for iPhones, but built to integrate with, and to provide extensions for Excel.

This is something that I'm personally excited about. It would be incredibly useful, and I would love this to exist. At the moment, most of my time is going into iterating for Cobalt Quantware's initial release of our product. Cobalt Quantware is currently in stealth mode, and I hope to talk about both the experience of creating a start-up and our product more later. In the meantime, I'm putting this idea out there, because I don't have the spare capacity to execute on it. If someone wants to build it, great, and I'll provide whatever support I can. If not, I may pick up on it later. In the meantime, consider this an A-to-B test of stealth mode vs loudly talking about an idea. I want to provoke debate, and I would love to hear feedback on this idea - if you love it, if you hate it, or if you think it needs to work differently. Please email me, or comment below.

I'll deal later with how I think this would improve the spreadsheet ecosystem, but first, I'll outline the mechanics of how the "App Store for Excel" might work.

How might an "App Store for Excel" work

I imagine that the "App Store for Excel" would be accessible through a menu item. It would show a dialog that would list extensions that are available from a central repository. One could search for a particular extension using a search bar. Clicking on an extension would give a detailed description, and double-clicking would download and install the extension.

An extension may offer additional functions to be used in formulae - for example, a simple extension may offer a version of doing linear interpolation, or a way to access data from web pages in a spreadsheet. It may offer additional menu items - for example, a better alternative to manage named ranges.

An extension could be packaged in one of several ways. Immediate possibilities that spring to mind are that an extension could be a VBA module, or an XLL file, or it could be a ZIP file that contains many file resources, including a file that describes how the extension could be installed. I'll discuss advantages and disadvantages later.

The central repository itself might simply be a single XML or JSON file served by a webserver over HTTP, listing the available extensions, together with download URLs, descriptions, release dates, version information and other metadata. In the case of ZIP file extensions, it could even be compiled from the information file in the ZIP itself.

The "App Store for Excel" may support using several repositories simultaneously - I may have access to a global central repository of good stuff, but also to a repository specific for my company, or my group.

This is a broad brush-strokes overview of how an "App Store for Excel" may work. You may agree with what I've sketched out, or you may feel it needs some changing. In any case, there are a lot of details that would need to be filled out and specified more precisely between this description, and even a working prototype.

What an "App Store for Excel" offers

The promise is of a global central repository, packed with functions that allow you to access data from the web, from databases, and to do high quality analysis of it, right in Excel. Instead of cutting and pasting several tables from web pages and emails to Excel, pull it in using a function from the central repository. Data changed, no problem, and no need to manually recopy the data - just recalculate.

The examples I gave before - linear interpolation, pulling data from the web, and managing named ranges - are functions that I have built recently. I'm pretty sure that many people have built functions to do these things. The duplication of effort is crazy! It would save years of effort, is if they were written once, and available centrally.

There are two other nice side-effects. First, Excel users would be better able to interoperate between companies, if they were using common functions for linear interpolation, and all the other things. Second, these common functions would see much more use, than the versions in existence at even the largest companies do currently. The more these functions are pounded, the more bugs are shaken out; and the functions in a central repository are likely to be very high quality.

Above all, what I hope you do feel, if you are an Excel user is that you listen to this, and think "I want this to exist". I know I do.

Why this improves the spreadsheet ecosystem

The most obvious way this improves the spreadsheet ecosystem is by putting more capabilities in the hands of users. Want to pull a stock quote from Yahoo Finance - there's an extension for that. Want to pull analytics data on who is visiting your blog into Excel - you can do that too.

For me, there is an even closer analogy than Apple's amazingly successful App Store, and that is the repositories that have sprung up around various programming languages in the last 10 years. Perl has CPAN, Python has PyPI, Ruby has RubyGems, Java has Maven, and so on. To my mind, these repositories have been one of the two huge production-boosts for programmers in the 2000s (the other is that lists and dictionaries have become first-class data structures in most languages). With a single command, I can extend Python with a world class numerical library, or an industrial-strength web framework, or a web scraping library, or ... All things that would require a huge amount of effort for me to create from scratch. In fact, due to the success of these libraries, the nature of programming itself has changed from creating from scratch, to glueing together libraries and systems.

But this post is meant to be about improving the spreadsheet ecosystem, rather than simply improving the Excel ecosystem, and there is a second, more subtle way that this central repository could help achieve that, and that harks back to the power of formal written standard that I discussed in my previous post. By having a specification, it allows other spreadsheets to interoperate. The same extension that accesses BetFair from Excel, could be made available in OpenOffice, and in Apple's Numbers, and so on. This would be tricky for extensions written in VBA, but should be achievable for other forms of extension. For example, extensions written in C#, and exposed to Excel through ExcelDna, could be compiled using Mono, and exposed automatically to other sheets.

This would provide a key interoperability step. As the central repository grows, it would become more and more attractive to build spreadsheets using only built in functions, and those available in the "App Store for Excel". There would be no dependence on custom-built VBA, and those spreadsheets would be portable. And once this happens, the stage is set for several commercial vendors to compete by adding value, with open-source competitors keeping them innovating.

As I said before, Excel has 400 million users, versus Java's 9 million. To me, the thought of creating a software ecosystem to hugely empower nearly 6% of the world's population is very exciting.

Monetization

As I said before, I do not currently have the capacity to pursue creating this "App Store for Excel". But I have thought a little about how it might be monetized. To me, the importance of this is that it provides sustainability, rather than dot-com riches. If it can be monetized, it can support a population of permanent developers, who can maintain, support and extend it. Although more power to you if you can make dot-com riches as well.

The obvious way to obtain revenue from an "App Store for Excel" is the same way that Apple do: by providing the facility for authors to charge for add-ins, and to take a percentage of this. In that case, this had a very strong stimulative effect on the community of developers, especially considering that development for Apple systems beforehand would be considered nichey, at best. I would hope that providing authors with a simple way to charge for Excel extensions would provoke a similar effect on development around the Excel ecosystem.

Another way to obtain revenue might be by providing a certification scheme, charging to certify that extensions meet certain criteria.

A third way might be to provide a product targeting the corporate market, allowing easy distribution, authentication, and access control within a company. I understand this is the route chosen by the makers of the repository software used in the Java ecosystem.

I haven't thought too deeply about this aspect, but these are just some initial thoughts that spring to mind on how to make an "App Store for Excel" pay for itself. I'd be interested in any comments or further thoughts people have.

Conclusion

In this post, and the previous one, I've looked at the dynamic of open- and closed-source software working together for the best pace of innovation. My particular focus has been the spreadsheet ecosystem, the largest software development ecosystem by number of users, which has been dominated for many years by a single closed-source commercial solution, Excel. I've put forward a proposal of a central repository for extensions to Excel, and other spreadsheets - an "App Store for Excel". I think that this would be immediately useful, and exciting in itself. More importantly, longer term, I hope this would hope to stimulate the existence of open- and closed-source solutions competing in the spreadsheet ecosystem, and to restore a high pace of innovation.

Permalink | Leave a comment »

The tortoise, the hare and the problems of the Excel ecosystem

Wed, 28 Apr 2010 23:29:45 -0700

Chris Dixon has a great post about the tradeoff between open and closed. One thing that I found particularly interesting was the quote from Professor Eisenmann that "markets may evolve in directions that favor[sic] the presence of one strong closed player and one strong open player".

The tortoise and the hare

I agree with the thinking. It's like Aesop's tortoise and hare exploring an unknown continent. The hare goes scampering off to the near distance, and starts napping, while the tortoise plods along, slowly and continuously, catching up to the hare, and then passing it. Eventually the hare wakes up, and again runs ahead of the tortoise. They are going to run a long way.

Without the tortoise, the hare is going to run off in the distance once, fall asleep, and never bother to explore the rest of the continent. You can see this after the first set of browser wars. Internet Explorer had become dominant, and Mozilla took a long sabbatical to be redeveloped from scratch as an open-source product. Microsoft didn't release a version of Internet Explorer for five years. So the job of the tortoise is to make sure that the hare doesn't nap.

The job of the hare is more subtle. The hare runs faster, and can explore the area near them both in less time. I would argue that companies with closed source products tend to innovate faster, and that open source products tend to take their lead from closed source products where possible. To pick 3 examples I can see on my desktop right now, Linux/BSD came from the commercial UNIXes, Firefox was built to compete with IE, Thunderbird with Outlook, and so on. Without the hare, the tortoise will get there, eventually, but it will take far longer to explore the possibilities. The hare helps give the tortoise direction.

Now I would hardly argue that this is hardly a complete analysis of the distinctive aspects of open- and closed-source cultures, and the various ways they compete and co-operate. But it does help cast some light on where this innovation process is failing. And for me, the prime example has to be Excel.

Excel

Excel is used by 400 million people worldwide, to build what are to all intents and purposes simple business applications. And yet from 1995 to 2007, we have seen the following innovations in Excel:

a clipboard that can hold multiple items
a talking paper-clip
the ability to have more the 65,536 rows
an arguably improved menu-system

I don't think I'm missing too much in that list. I haven't used 2010 yet, but I'm not holding my breath. And Microsoft is acting perfectly rationally in failing to innovate. Say they could spend $1 million developing a new feature, then for them that's $1 million of lost profit. They won't gain a single customer, because what else were you going to do than buy Excel?

Now, there are open-source spreadsheets - OpenOffice and Gnumeric spring to mind. And there are commercial alternatives, such as iWork. And if you prefer the cloud, there's Google Documents. But year after year, they fail to make a significant dent in Excel's dominance. Why is that?

What none of them have, properly, is interoperability. Sure, if I make a simple table in Excel, I can port that around. If I start using the more fancy functions in Excel, I start running into trouble. If I make my own functions in VBA, fairly quickly I end up unable to use my spreadsheet, even on Mac Excel.

Where do we go from here

The internet, and the web in particular, have always had interoperability. While Internet Explorer may be closed source, it is based on open standards. And in a lot of ways, IE has done a good job as the hare, exploring possibilities - ActiveX for example, was expedient, and gave use-cases which set targets for later web standards to do properly.

If I'm asked to imagine what the web would be like if the entire ecosystem was closed, I think of Lotus Notes. I first used Notes in 1997, eight years after its first release, and it still seemed innovative at that time. But in 2010, I would not consider it for a new deployment. It's clunky, and has been largely eclipsed by web technologies, improving slowly over time.

The spreadsheet ecosystem needs to take this lesson to heart. Standards allow interoperability. And interoperability allows real competition. And that makes better products for end-users.

So one possibility is to start defining standards based on what already exists in Excel and other spreadsheets. One standard may define allowable worksheet functions, another may define the core of a scripting language, another a standard file format for interchange and so on. There is a lot of work here, no doubt. In fact, I would say it is an impractical amount of work, amounting in effect to reverse engineering Excel.

This blog post is already long enough, so in a future blog post, I'd like to outline an alternative direction, based on the creation of a repository for VBA modules, XLLs and other extensions. Think CPAN/PyPI/CRAN/App Store for Excel. The aim is to create something of immediate utility, with a view to allowing proper interoperability.

In the mean time, especially if you're a keen Excel user, I'd be interested in any feedback you have on this post.

Permalink | Leave a comment »

Tutorial: Numerical Analysis in Excel using C# with ExcelDna and AlgLib

Thu, 22 Apr 2010 12:15:00 -0700

Quite often, I need to use numerical techniques more advanced than those that ship with Excel. Sometimes this involves writing my own algorithms from scratch, but when possible, I prefer to make use of existing libraries. Tens or hundreds of thousands of open source developers have worked to make great libraries - not only for numerical analysis, but for messaging, web scraping, and a jillion other tasks. Exposing this functionality to Excel makes it far more powerful. With that in mind, I'd like to share the workflow I use to access an open source numerical library, AlgLib using C#.

So why C# rather than VBA?

C# lets me use version control. Once you've used version control, you can't go back.
C# is more maintainable than VBA. Even with the best of intentions, VBA code tends to become a bit of a mess - functions grow to several pages, algorithms become enmeshed with code to manipulate the sheet. C# is a well-designed modern object-oriented language. It's the easiest language I've ever come across to just bash code out in, and be pretty sure that you will be able to understand it when you come back in 6 months.
A C# library is more manageable than VBA modules. When I start a new spreadsheet, I have one add-in to load, rather than 20 separate VBA modules.

You can download Visual C# 2010 Express from Microsoft at http://www.microsoft.com/express/Downloads/#2010-Visual-CS, which will be sufficient for this article. Standard or Professional editions are recommended for more serious work.

AlgLib

In this article, I'll be looking at accessing numerical routines from the AlgLib library. AlgLib is a neat project run by Sergey Bochkanov and Vladimir Bystritsky, aiming to provide robust implementations of numerical algorithms for several languages, including C#. This is great for several reasons. Having code in the language you are familiar with makes it very easy to integrate into your project or existing codebase. But more importantly, at a deeper level than that, it encourages you to look at the code, to play with it, and to adapt it to your own needs. Numerical algorithms are intrinsically hard - you need a lot of maths to come up with an efficient solver or integration routine - and I'm not anticipating that every one who looks at the code will be able to do that. Nonetheless, the most important property of numerical routines is that they be robust. There is nothing worse than running a solver for a day, just to find it has crashed, or given you, very precisely, the wrong solution. And the way numerical routines become robust is by people pounding on them, and fixing edge cases. And for this, AlgLib is a much better base for the 2010s, than JPL's excellent, but nearly dead, Fortran routines - code may not decay, but code ecosystems seem to, and there aren't so many Fortran programmers left.

I should mention the other open source numerical libraries. GSL is robust, mainly written in C, which can make it a challenge to integrate, memory-management-wise. Boost (C++) now has some mathematical libraries, although fewer, and less advanced, but its coverage of statistical distributions is quite complete. On the commercial side, IMSL and NAG are well recognized as robust numerical libraries, with hefty price tags. If there is interest, I can cover how I expose C++ libraries to Excel in another post.

You can download the latest version of AlgLib from their website at http://www.alglib.net/.

ExcelDna

I'll be using ExcelDna to expose C# functions to Excel. ExcelDna is a magical, but under-publicized library, which can be used to automatically expose all public static functions in a .NET assembly to Excel through an xll addin. Everyone I have recommended ExcelDna to has either immediately switched to using it, or lamented (hard) the fact that for one reason or another they were unable to immediately switch to it.

You can download the latest version of ExcelDna from http://exceldna.codeplex.com/.

How to put it all together

The first thing to do is to expose C# functions to Excel. To do that, follow these instructions:

Open Visual C# 2010 Express, and select start a new project from the start page. In the new project dialog box, select the Class Library template and enter "AwesomeNumericalLibrary" as the name at the bottom of the dialog.
Put the cursor in the name of the class, "Class1", and hit F2 to rename the class to "ExcelFunctions". You should also rename the files. This is personal choice, but my preference is to have the functions I expose to Excel to be in a single file, which only contains functions exposed to Excel, rather than intermingled with other code. As the library grows, I tend to split the file based on categories of function eg statistical distributions in one file, interpolation routines in another.
Add a simple function, so we can test that we can expose functions to Excel. Again, as a personal choice I tend to begin all Excel functions in a single library with a common prefix. This allows them to be easily found using Excel's Insert Function dialog or intellitype.
public static double AwesomeLifeUniverseEverything() { return 42; }
From the ExcelDna zip file, add ExcelDna-0.25\Distribution\ExcelDna.xll to your project. You should add it to the project root itself, rather than a sub-folder. This can be done by dragging the ExcelDna.xll file from a Windows Explorer window onto the text AwesomeNumericalLibrary in the Visual C# Solution Explorer.
In the solution explorer, rename ExcelDna.xll to AwesomeNumericalLibrary.xll.
Add a new text file to the project root, and rename it to AwesomeNumericalLibrary.dna. The xll file looks for a dna file with a similar name to tell it what .NET assemblies it should load and expose functions to Excel from. In our case, there will be a single dll, so add the following text to AwesomeNumericalLibrary.dna:
<DnaLibrary> <ExternalLibrary Path="AwesomeNumericalLibrary.dll" /> </DnaLibrary>

You must tell Excel to copy the .dna and .xll files to the output directory when it builds the project. To do this, select the two files in the solution explorer. Then right click and select properties from the context menu. In the properties dock, change the Copy To Output Directory setting from "Do not copy" to "Copy always".
At the time of writing, ExcelDna does not support .NET 4.0 assemblies. To fix this, right click on the project in the solution explorer and hit properties. In the properties window, change Target Framework to ".NET Framework 3.5". There will be a warning message about the CSharp reference. Expand the References section for the project in the solution explorer, and remove the reference to CSharp.
Now we are ready to test whether our function is accessible from Excel, so save the project (easiest way seems to be to close and re-open it!) and build the project (by hitting F6).
Fire up Excel, and open the Add-Ins dialog box. In Excel 2003 and older, it is Tools..Addins. If you are using Excel 2007 or newer, the menus are hidden - the magic key combo to get to the Add-Ins dialog box is "Alt-T, I" (it is also available through Excel options).
Click Browse, and then navigate to where you saved the project, and then within the "bin\Release" directory, select AwesomeNumericalRecipes.xll and hit Open.
In a cell in the sheet, type "=AwesomeLifeUniverseEverything(10)" and verify that you get the result 42.

Well done! You have just exposed your first C# function to Excel.

Now to incorporate AlgLib into our project, and expose a function to Excel:

Unfortunately AlgLib does not have a project file, so we have to create our own. Right click the solution in the solution explorer, and click Add..New Project. In the Add New Project dialog, select Class Library and enter the name "AlgLib".
Delete the class file, Class1.cs, that is created by default.
From the AlgLib zip file, add all the .cs files in the csharp\src directory to the AlgLib project. You can do this by selecting the files in Windows Explorer and dragging them to the AlgLib project in the Visual Studio Solution Explorer.
Again, you must tell Visual Studio you want a .NET Framework 3.5 library, using the steps above on the AlgLib project.
Add a reference to the AlgLib project in the AwesomeNumericalLibrary project. In the Solution Explorer, expand the AwesomeNumericalLibrary project. Right-click References, and click Add Reference. In the Add Reference dialog, select the Projects tab, select the AlgLib project and click OK.
Add a function to the ExcelFunctions class in the AwesomeNumericalLibrary project. I've chosen here to expose Dawson's integral:
public static double AwesomeDawson(double x) { return alglib.dawson.dawsonintegral(x); }
Build the solution (hit F6). The AlgLib project may generate many warnings. You may get an error that AwesomeNumericalLibrary.dll is open by another process and cannot be written. If this is the case, then open up the Add-Ins dialog in Excel, and remove the tick from by "AwesomeNumericalLibrary" to unload the dll from Excel. Occasionally you may also have to restart Excel.
Go back to Excel and reload the library. It should remain in the Addins dialog, so you can just re-select it. Then in a cell, type "=AwesomeDawson(2)". It should give you the result 0.30134.

Congratulations! You have just exposed a numerical routine from AlgLib to Excel.

AlgLib has a large number of functions, so I'll let you expose the rest as you need. I've chosen a pretty simple function to expose here, but ExcelDna also makes it possible to take ranges as inputs, and give ranges as outputs using array formulae. This can be useful for linear algebra, or generating large amounts of random data. Error handling is another issue - this function takes one number and always returns a result, but not so with all functions. If you're interested, let me know, and I'll cover these issues in a future blog post.

How to Distribute

To distribute an Excel add-in you have made using ExcelDna you must distribute the (renamed) ExcelDna.dll file, your .dna file, and any .dll files that your add-in is using directly or indirectly.

AlgLib is distributed under a GPL license, so if you distribute your add-in, you must also make the source code available under the GPL license. ExcelDna is distributed under a more permissive license. Please check for yourself that you are complying with the respective projects' license terms.

Further steps

There are several directions you can take from this quick tutorial. I'd recommend looking through the ExcelDna documentation a little. ExcelDna is able to take object arrays and other types as parameters, and to return them as results, which can be powerful. You can also use attributes on your functions to better control how they work, and how they appear in Excel.

You might want to look at exposing other libraries or functionality to Excel. Some things I've done that were quite fun and useful were a web scraper, which allows me to suck out values from a live web page using XPath right in the Excel formula, and more targetted add-ins for Yahoo finance and Betfair.

Permalink | Leave a comment »

Book review: Influence: The Psychology of Persuasion by Robert Cialdini

Wed, 14 Apr 2010 19:10:07 -0700

I initially came across Cialdini's "Influence: The Psychology of Persuasion" as a recommendation by Charlie Munger in his excellent speech "The Psychology of Human Misjudgement", and with my reading pile near empty, ordered it from Amazon on a whim. You can (and if you've got time, should) read a transcript of that speech at http://vinvesting.com/docs/munger/human_misjudgement.html. Here is my review of that book:

In Influence, Cialdini looks at some of the universal principles used by what he calls compliance professionals. This includes salesmen and estate agents, looking to make you compliant so that you might buy a fridge, a car or a house, but also includes more extreme examples - cult leaders and Chinese POW guards are two of the more colourful examples.

The book naturally targets two groups of people. First, those who want to put these principles into action. I would expect this would naturally include sales and marketing professionals, but I'm hard-pressed to imagine who wouldn't want the magic power to make others do their bidding. The audience also includes those who want to know more about these principles, how they are susceptible to them, and how to defend against them. I suspect this group also includes pretty much everybody. There is a natural tension between these audiences. It would be easy to alienate the general public with a barrage of ways they are naive, or to go the other way and turn off sales people looking for concrete techniques. The author successfully diffuses this tension, identifying himself in the first sentence of the book as a life-long patsy for all these techniques, but not shying away from giving details on how they can be used to best effect.

The author Robert Cialdini, is an expert researcher in this field, and this book builds upon his years of research. Nonetheless, while it is backed by high quality research and thinking, this book is no research paper. The book is deliberately approachable, outlining ideas with a minimum of fuss, and giving plenty of concrete examples from the author's own experience, and interpreting well-known events or situations.

The book is organized into seven chapters. The first is an introductory chapter which provides motivation for the reader by giving a taste of the power of the sort of techniques outlined in the book. Each of the subsequent chapters deals with a specific principle which can be used to motivate an intended action. The principles covered are Reciprocation, Commitment and Consistency, Social Proof, Liking, Authority and Scarcity. Within each chapter, the author uses a number of examples to drive the message, outlining the general principle at work, looking at when they are most effective, and how to say no when you are on the receiving end.

For example, in the chapter on Reciprocation, the author looks at the facet of human nature that causes us to feel indebted when someone does something for us, or gives us something. Examples given include Hare Krishnas handing out flowers at airports, and various political careers. He goes on to describe how this effect works even when the gift is unreciprocated, to the point where people will try to avoid receiving unsolicited gifts to avoid the burden of reciprocation. He uses the example of a boy scout initially offering circus tickets at 5 dollars a piece, but when rejected, backing off to offering cookies at 1 dollar a piece to illustrate how reciprocal concessions can be used to make a sale. Finally, he covers how to avoid being trapped into an action by the principle of reciprocation, without resorting to rudeness that may itself be counter-productive.

The most severe criticism I can level at the book is that occasionally it veers towards becoming a succession of anecdotes, with less attention paid to explicitly describing the principle under consideration. These colourful examples do all have a point, illustrating some aspect of a principle or a way it can be more or less effective, but it is possible to lose track of the higher level view, and just start reading the book as a series of interesting stories.

And that would be a great shame, because this is one of those books, that once you have read it, you cannot help but view the world through the prism of the ideas in this book. Apple delaying the iPad internationally yet again, and me wanting one even more - perfect example of the principle of scarcity. In hindsight, the credit bubble leading to the financial crisis of 2007 onwards seems pretty much a perfect storm of all the principles from all the parties.

One might think that in the 26 years since it was first published, it would have aged. Certainly, advertising and marketing techniques have evolved hugely in that time, to the point where modern consumers could be expected to be pretty immune to say, a television advert from the early 80s. Nonetheless, underlying the principles examined in the book are fundamental aspects of human behaviour, and these do not change so quickly.

In any case, it is useful to explicitly list, describe and examine how it is that people come to make decisions, both to be aware of where our own decision-making can be taken advantage of, and to help others come to decisions that we would prefer them to.

The author abstains from comment on the desirability of using the powers of influence it describes, although it deals with examples from the positive (increasing racial integration in schools, effectively promoting environmental causes) to the strongly negative (propaganda in North Korean POW camps). Like any tools, the author's principles can be used for good or evil, but are inherently neither.

In many cases, the principles are based on shortcuts that we could reasonably expect to work; a more expensive or more in demand item is likely to be a better item. Furthermore, with the vast choice of actions and purchases available to us in the modern world, we need these shortcuts. It would take a long time to complete a supermarket shopping trip if one was to calculate the cost per gram of every item in the shopping cart, and take into account the subjective quality differences - far easier to just go by the shiny packaging. The author recognizes this, and to his credit does not recommend attempting to totally avoid these signals. Instead, he adopts an approach of recognizing when the signals are being used in a deliberately manipulative way, giving mental tricks to flip the manipulation around, and negate its effect.

The author leaves it to you to decide whether you want to try to influence others, or simply to shield yourself from their undue influence. Nonetheless, he does on occasion advocate a course of trying to disrupt those who use these techniques in a dishonest way. For example, social proof is too useful a tool in deciding what to do to allow it to be subverted.

In the end, I found this an interesting book. The subject matter, how people try to influence other people, was always going to be good, and its handled at a good pace. Simple enough that it's approachable for anyone, but with deep enough ideas to make it worth reading. More than that, the book is likely to be useful reading whichever camp you primarily fall into, a manipulator or a manipulatee.

Recommendation: At £5.50 on Amazon, this book is a steal. BUY IT!

Permalink | Leave a comment »

Tutorial: Using JQuery and CouchDb to build a simple AJAX web application

Tue, 13 Apr 2010 17:21:00 -0700

I initially wrote this post a couple of months ago, while trying out writing a blog on wordpress. That blog didn't go anywhere, but I thought this tutorial on writing an AJAX web application with JQuery and CouchDb might still be of interest to developers playing with CouchDb. Certainly I couldn't find anything similar at the time

It also gives me a chance to see what the code formatting facilities on posterous are like. I enjoyed using CouchDb, but ultimately found that MongoDb better met my needs. That said, I was very impressed with CouchDb as a light, stand-alone solution, and look forward to seeing how it progresses. I hope that CouchDb hasn't moved so far forward in the intervening time that this tutorial is out of date.

I could not find too much on the web about using jQuery and CouchDb together, so I decided to put together a little "Hello World" couch app of my own. Having done that, I thought I'd put together a little step-by-step instruction in case it's of use to others.

I should warn that I'm not too familiar with jQuery, so I may be doing things in odd ways - please let me know if you see something that could be done better.

I assume that you have CouchDb 0.10 and CouchApp 0.5 installed. Later versions may work, but no guarantees.

First, go to http://127.0.0.1:5984/_utils and create a database called "addressbook". Because we'll be making the functionality to display data before the functionality to create or update it, it's a good idea to use the built-in CouchDb web interface to add a couple of documents. The initial version of our webapp will let us store a mobile number for each person, so add the following 2 documents:

{"type": "address", "name": "Fred", "mobile": "555-0001"} {"type": "address", "name": "Barney", "mobile": "555-0002"}

CouchDb uses design documents to expose the database to the world, but editing design documents directly gets a little painful. Couchapp eases this pain, by breaking design documents out into a set of directories and files on the local filesystem representing shows, views, and so on, which can be pushed to the CouchDb server at will. We need to create a couchapp design document, and also a view to retrieve the phone numbers, which we can do using couchapp at the command-line:

couchapp generate app addressbook
couchapp generate view addressbook phonenumbers

The first line creates a directory called addressbook with a skeleton design document set up, and the second creates a view in the design document.

We can push this to the server using:

couchapp push addressbook http://127.0.0.1:5984/addressbook

If you look at the database using the web interface now, you should see the design document "_design/addressbook" in addition to the documents representing the two phone numbers.

The view we will need simply gives each name and mobile number, with name as key, so the view is sorted by name. No reduce function is required, so delete addressbook/views/phonenumbers/reduce.js, and put the following in addressbook/views/phonenumbers/map.js

function(doc) { if (doc.type && doc.type == "address" && doc.name && doc.mobile) { emit(doc.name, doc.mobile); } }

Push the couchapp to the server using

couchapp push addressbook http://127.0.0.1:5984/addressbook

and check that you can see the view at http://127.0.0.1:5984/addressbook/_design/addressbook/_view/phonenumbers

Couchapp has set up an initial index.html for us, which we can edit at addressbook/_attachments/index.html, and view at http://127.0.0.1:5984/addressbook/_design/addressbook/index.html. Make sure that you can view the file ok. You should then alter index.html to contain:

<!DOCTYPE html> <html> <head> <title>Simple address book</title> <link rel="stylesheet" href="style/main.css" type="text/css"> <span style="white-space: pre;"> </span><script src="/_utils/script/json2.js"></script> <span style="white-space: pre;"> </span><script src="/_utils/script/jquery.js?1.3.1"></script> <span style="white-space: pre;"> </span><script src="/_utils/script/jquery.couch.js?0.9.0"></script> <span style="white-space: pre;"> </span><script type="text/javascript"> <span style="white-space: pre;"> </span></script> </head> <body> <h1>Simple address book</h1> <span style="white-space: pre;"> </span><div id="add"><button type="button" id="add">Add</button></div> <div id="addressbook"></div> </body> </html>

The head contains a title, a link to a stylesheet, imports jQuery and jQuery Couch. These are used by the CouchDb web interface, and included with CouchDb by default. It then has a space for our own javascript to go - more on this later. The body contains an add button within a div, and an empty div called addressbook. The app will display address book entries in the addressbook div.

We use javascript to access data in the CouchDb, and to process events from the browser, such as buttons and links being clicked. Conceptually, we can think of CouchDb as the Model, the DOM representing the HTML shown to the user as being the view, and our javascript as being the Controller, albeit with relatively little separation in this example.

Javascript typically uses a continuation-style to process events. For example, we call asynchronous functions, for example to request data from CouchDb, and pass them a function to call when they return, eg to update the view with the data we receive. A simple example:

$.couch.db("addressbook").saveDoc( {type: "address", name: "Wilma", mobile: "555-003"}, {success: function() { alert("Saved ok."); }} );

This example saves a new document to the Couch database, and if successful, pops up an alert.

Another example is jQuery, which calls functions when links or buttons are clicked. The following example will trigger an alert whenever an "a" element is clicked (ie a hyperlink):

$("a").click(function() { alert("Click"); })

jQuery uses selectors to control what elements of an HTML document will trigger functions. In the above example, the selector applied to all "a" elements. We can use "a#blah" to select all "a" elements with an id attribute of "blah", or "a.blah" to select all "a" elements with a class of blah. Selectors form a mini-language and the jQuery documentation gives more details.

The first javascript we add is called when the base document has finished loading. It retrieves the JSON object of the phonenumbers view from CouchDb. When the JSON object is retrieved a continuation iterates over each row in the view, adding a div for each row containing the name, phonenumber and links to edit and delete it:

$db = $.couch.db("addressbook"); function refreshAddressbook() { $("div#addressbook").empty(); $db.view("addressbook/phonenumbers", { success: function(data) { for (i in data.rows) { id = data.rows[i].id; name = data.rows[i].key; phonenumber = data.rows[i].value; html = '<div class="address">' + '<span class="name">' + name + '</span> ' + '<span class="phonenumber">' + phonenumber + '</span> ' + 'edit '+ 'delete '+ '</div>'; $("div#addressbook").append(html); } }}); } $(document).ready(function() { refreshAddressbook(); });

A few things to note:

We have created a variable $db to access the CouchDb database. If we change the name of the database, this means we only have to change one place.
The function called on success(ful retrieval of data from CouchDb) takes a parameter, data, which is the data returned fromt the view.
We call the function from a continuation passed to $(document).ready(). This ensures that we won't request data until the HTML document is fully loaded.
The links have an id, which is the id of the document representing the address book entry.

If we push to the server now, using

couchapp push addressbook http://127.0.0.1:5984/addressbook

and refresh in the browser, we should see the page load and then populate with phone numbers from the database. The add button and the edit and delete links will be inactive, but at least the R part of our CRUD app works.

The next thing to focus on is deleting entries. A nice way for this to work is for the delete link, when clicked, to pop up further links to confirm or cancel deletion. We can use jQuery to insert these links in the div for the address book entry when the delete links are created.

But this poses a problem - jQuery binds functions to the DOM elements that exist at the time. The links we are proposing adding do not exist when the $(document).ready continuation is called, so we cannot bind functions to them until they are created. Fortunately, there is a trick we can use. Events such as clicks propogate up the DOM tree if they are not processed - so a click will first be processed by the a element, if a handler exists, else it will be processed by the containing div, for example, if a handler exists, all the way up to the document itself. The continuation takes a parameter, which can be used to determine the exact element that was clicked.

In our example, we can bind a function to the click event of div#addressbook, and then check whether the click came from an edit link, a delete link, or the confirm or delete links we will create. Add the following to the $(document).ready continuation:

$("div#addressbook").click(function(event) { var $tgt = $(event.target); if ($tgt.is('a')) { id = $tgt.attr("id"); if ($tgt.hasClass("edit")) { // TODO: implement edit functionality } if ($tgt.hasClass("delete")) { html = '<span class="deleteconfirm">Sure? Yes No</span>'; $tgt.parent().append(html); } if ($tgt.hasClass("dodelete")) { $db.openDoc(id, { success: function(doc) { $db.removeDoc(doc, { success: function() { $tgt.parents("div.address").remove(); }}) }}); } if ($tgt.hasClass("canceldelete")) { $tgt.parents("span.deleteconfirm").remove(); } } });

This code runs when any element in the addressbook div is clicked. It gets the clicked element, the id of that target element (which we set above to be the relevant CouchDb document's id), and then runs different code depending on the class of the clicked element. If the link with class "delete" is clicked, we append two new links - "Yes" with class "dodelete", and "No" with class "canceldelete" - within a span with class deleteconfirm. Then, if a link with class "canceldelete" is clicked, we remove the relevent "confirmdelete" span from the DOM to delete the confirm message. On the other hand, if a link with class "dodelete" is clicked, we call $db.openDoc to retrieve the document to delete from CouchDb, and pass it, using a success continuation to $db.removeDoc. We need to retrieve the document prior to deletion in CouchDb, to avoid deleting from an older version than the most current in the database. If this is successful, we remove the parent address div from the DOM, to match the database. Take a deep breath, and read that again.

If you push this design document to CouchDb, you should be able to delete entries from the address book. However, you will have to use the CouchDb built-in interface to re-add them at this point.

We adopt a similar approach for editing or creating new entries - clicking the button or link exposes a form to enter new details. CouchDb allows us to save a document with an id to indicate an update, or without an id to indicate that one should be generated, and a new document created. Consequently, we can use the same form for creating and editing if we are careful.

Add the following function between the refreshAddressbook function and the $(document).ready

function addUpdateForm(target, existingDoc) { html = '<form name="update" id="update" action=""><table>' + '<tr><td>Name</td><td>Number</td></tr>' + '<tr>' + '<td><input type="text" name="name" id="name" value="' + <span style="white-space: pre;"> </span> (existingDoc ? existingDoc.name : "") + '"/></td>' + '<td><input type="text" name="mobile" id="mobile"/ value="' + <span style="white-space: pre;"> </span> (existingDoc ? existingDoc.mobile : "") + '"></td>' + '<td><input type="submit" name="submit" class="update" value="' + <span style="white-space: pre;"> </span> (existingDoc?"Update":"Add") + '"/></td>' + '<td><input type="submit" name="cancel" class="cancel" value="Cancel"/></td>' + '</tr>' + '</table></form>'; target.append(html); target.children("form#update").data("existingDoc", existingDoc); }

Apologies for the inelegant table. I would love to hear a better way for laying out forms. In the meantime, this does the job. And the job is that it appends a form to the target element. If existingDoc is provided, its name and mobile values are used to populate the form, else blanks are used. Further, the existing document is attached to the form, so it can be updated with new values provided by the user to save data later.

To display the form when an edit link is clicked, add the following in place of the edit TODO above:

if ($tgt.hasClass("edit")) { $("button#add").show(); $("form#update").remove(); $db.openDoc(id, { success: function(doc) { addUpdateForm($tgt.parent(), doc); }}); }

And to display the form when the add button is clicked, add a continuation to the bottom of the $(document).ready continuation:

$("button#add").click(function(event) { $("form#update").remove(); $("button#add").hide(); addUpdateForm($("div#add")); });

Note that we hide the add button when it is clicked, and show it otherwise. We also remove any existing forms, so only one form shows at any time.

Finally, we need to process the events when the update(/add) button is hit, or the cancel button. We could adopt the same approach as above, of binding to an element up the hierarchy, but I've chosen to demonstrate a different approach. The .live method can be used to bind to a selector, and to any future elements that selector applies to. That is, as new DOM elements are created, the continuation will apply to them also. Here is an example to bind the click events, of input elements of class "cancel", which hides the form and shows the add button when the cancel button is pressed:

$("input.cancel").live('click', function(event) { $("button#add").show(); $("form#update").remove(); return false; });

You should add this to the end of the $(document).ready continuation.

Finally, we need to process the update/add event. The key here is to pick up the existing CouchDb document, if it exists, from the form DOM element we attached it to. This ensures we are updating a CouchDb document with the appropriate id and revision. If there is no existing document, we can populate an empty JSON object, and CouchDb will save it as a new document with a uniquely generated id. Here is the continuation and binding to add to $(document).ready:

$("input.update").live('click', function(event) { var $tgt = $(event.target); var $form = $tgt.parents("form#update"); var $doc = $form.data('existingDoc') || {}; $doc.type = "address"; $doc.name = $form.find("input#name").val(); $doc.mobile = $form.find("input#mobile").val(); $db.saveDoc( $doc, {success: function() { $("button#add").show(); $("form#update").remove(); refreshAddressbook(); }}); return false; });

Note that we remove the form and show the button when the document has been saved.

Ok, that's it. Push this to CouchDb, and reload the page. You should find that you have a perfectly working, albeit minimal, CRUD webapp.

There are a few other things that you would want to do in production code, which I have left out here for clarity and brevity. Verification functions should be added to the design document, to avoid badly formed data being passed to CouchDb. Errors should be handled, as well as successes. Also, you might want slightly more from an address book webapp. Nonetheless, I hope this is a useful example of using CouchDb and jQuery to make an AJAX webapp.

Permalink | Leave a comment »