“Writing a proposal. Oh boy do I hate word processors. There must be a better way.”

It turns out there is, though I ended up forging my own path to suit my skills and working methods. A little background…

I need to produce decent looking proposals for web development projects. Normally I bite the bullet and suffer the awkwardness of Word or OpenOffice or LibreOffice or something, and then export to PDF. It’s all very clever: it took some great minds to produce this software. But it really goes against the grain. It feels so wrong.

I like plain old text, and as a web developer I’m used to defining the style of documents by hand, in CSS, rather than using another program to do it for me, as I write. I would go further and claim that this is not just a matter of personal taste: word processors are bad for the world. Sure, people are used to them, and they cannot imagine a better way, and – most importantly – there is probably no equally mature, friendly, well-supported and funded software in place to allow for a move away from them as things stand. But it does not follow from this that they are good.

What’s Wrong With Word Processing?

The name, to start with. But most of all WYSIWYG. One thing I’ve learned from web development is the importance of respecting the logical separation of style and content. WYSIWYG is not so respectful. Everything happens in the same place and at the same time. Effectively, documents are typeset as you go, character by character. This is silly.

Formatting your document’s style has nothing to do with its content. What you are trying to say is a different concern from how it should look. And I do not mean that the two human operations are more conveniently handled by separate software operations. I mean that they are different human, conceptual, natural concerns. Of course it has to look good (although a. it doesn’t always have to, and b. even plain text looks way better than a clipartified Word document). Yes, of course your document has to conform to a company standard, or your own standard, and how it looks might be partly what you intend to convey. But all this precisely means that these standards and styles ought to be defined separately, beforehand, leaving you to get on with the writing: to concentrate on what it is you have to say. It is the difference between talking to somebody at a party and deciding what you are going to wear to that party.

Using WYSIWYG software such as the horribly feature-rich Word is like going to a party with several changes of clothes and proceeding to change them every time you think that what you’re saying doesn’t go with stripes. Sure, do the best you can to look good, but prepare for that, by going shopping, trimming your beard, and putting on your best Y-fronts. After all that’s done you can get on with talking to people and getting drunk, safe in the knowledge that you’re looking great. Underdressed? Well, if you didn’t care enough to prepare, to research, and to tune your attire accordingly, then you shouldn’t care about being underdressed.

Now, moving on from that straining analogy, the biggest benefit of separating style and content is that content becomes independent of how it is presented. If standards of presentation change, or a piece of writing needs to be reproduced in different formats, or you’ve finally got tired of Comic Sans, then you shouldn’t be going anywhere near your content to achieve the desired formatting changes. But word processors force you to do this. You actually have to open and edit your document, just to re-style it. This wrongheadedness has been spectacular in its worldwide detrimental impact.

(Note that I’m quite ignorant of Word and the no-doubt very sophisticated setups that can be achieved in organizations where stanadardized presentation is important. For all I know maybe you don’t always have to go back and edit your document or re-assign a template of whatever. However, achieving this is just a workaround, and likely would not have a perfect success rate; it seems clear that the default expectation of the software is that you do mix up content and presentation.)

And faced with the option, the expectation, to fiddle about with fonts and bullets, it takes discipline, imposed from above or by yourself, to beat down that natural inclination. This is not a matter of freedom. In my case, I have to struggle with formatting issues almost every time I write – or rather, process – one of these documents, because I might be using different word processors on different platforms with different fonts or whatever. This should not even be an issue, because we already know, deep down, that style and content are logically separate categories (note that this does not mean that they do not sometimes overlap). And in a company environment, standard templates for Word and detailed strictures and guidelines from management on formatting, fonts and colours, are awkward and only partly successful workarounds for a problem that would not exist without WYSIWYG.

The writer of a document has expertise in the subject addressed by that document. They may not have expertise in typography and layout, and usually they do not. The latter, then, becomes nothing but a distraction and a huge waste of time. And … I didn’t ask you to send me a bloody Word attachment in the first place!

The natural paradigm for me, which I get from web development, is to write stuff in a text file, and to define how it looks in another text file, and only then to view how it will look to the document’s recipient. I believe this should go for print documents as well as HTML web pages.

It is already the way of things in printing, publishing and professional writing. I don’t believe Proust was worrying about typefaces when he painstakingly described the memory of a hawthorn bloom, and the prevalence of Word has not made this any less true of professional writers today. Why should any writer, or anyone who becomes a writer for brief periods – whether of poems or proposal documents or annual reports – have to worry about such things? Why has the division of labour, both human and machine, which is so thoroughly advanced in the modern world, been bypassed in this case?

Lightweight Markup

A few years ago I discovered Textile, which is a lightweight markup language. What a Godsend. I can get on with writing:

All in a text editor. You know, like Notepad only with nice colours and productivity tools. There’s nothing and nobody messing around processing what I write. Processing comes later, based on separate settings that I have set up for the scenario.

My Great Solution Which, Though Great, Could Most Probably Be Vastly Improved Upon

Up till now I’ve only really used Textile to generate HTML for display on web sites, usually blog posts, but now that I’ve (mostly) moved away from Windows and on to Linux, manipulating text becomes all the more easy. I’ve created a nice wee setup to generate PDF documents styled with CSS straight from a Textile file. I should stress that this involves the terminal and bash scripts, so it’s probably for geeks only.

The upshot is that when I want to write a proposal, I just run a couple of commands to set up a proposal project based on a template, and then get down to writing my document in Textile, and when I want to see what it’ll look like I just run ~/gopdf.sh, which generates a PDF.

The gopdf.sh bash script does three things in order:

A. Generates an HTML file from the textile file using pandoc
B. Generates a PDF from the HTML file using xhtml2pdf
C. Opens the PDF

(If I had actually got to the stage where I’ve built up a nice PDF stylesheet I would have shown you something more fancy, but there you go. And the sharp-eyed among you will notice that the line of text immediately following the list is squashed up to it. This might be a problem with pandoc’s understanding of textile, which is not perfect, favouring the alternative Markdown as it does. You can include HTML in among the Textile markup, so a line break here is easy to achieve, or else a CSS bottom margin would probably do it, and would probably be the better choice.)

What it Allows Me to Do

Before I describe how I make all this possible, here in more detail is what I now do every time I get a new client who wants a proposal.

One

So I meet with Bob of Bob’s Cake Company fame, and he tells me he wants a cake web app. I say sure, no problem, then I go home, start up Linux and create a folder called bobs_cake_company, and cd into it. Then…

The first command copies over my standard proposal template folder to bobs_cake_company as a sub-folder called “proposal”. The second one just opens the current folder, bobs_cake_company, in Sublime Text 2 (I seem to remember doing some things to get that “subl” command working, but I’m not sure what. Probably just a symlink.)

Two

I write the document in the file that is there waiting for me, proposal.textile. Remember, by writing I just mean writing. Not fiddling with margins or choosing bullets or struggling with alignment issues or fixing pasted text or making sure that the screenshot isn’t blurry or anything else like that.

Three

In the terminal…

This generates a stylesheet from the SASS files in which I define the CSS stylings. Full disclosure: currently my method is in its early stages so I don’t yet have a good standard CSS stylesheet for PDFs. I’ll be building this as I go along for a project or two before I can just leave it alone. So right now I’m actually running this Compass watcher when I begin writing, because I’m styling as I go, like a human word processor. Not for long.

Four

Then in the terminal (in a new tab):

This creates a PDF (or overwrites the one that already exists), and then opens it in the default PDF viewer. Thereafter I just run “~/gopdf.sh proposal” whenever I want to see my changes in the PDF.

Five

Send proposal to client and have a cigarette.

Making it All Possible

The process I’ve just described depends on a few pre-existing elements:

• pandoc
• xhtmlpdf
• startdoc.sh bash script
• gopdf.sh bash script
• a proposal_template folder with goodies in it
• Compass (optional)
• Sublime Text 2 (optional)
• A dangerous, expensive and anti-social nicotine addiction (optional)

You do the following stuff once only.

Get pandoc: Pandoc is a Haskell library that converts between lots of different kinds of documents. I would have used it alone had it been able to convert Textile and CSS to PDF directly. (I couldn’t make it work and concluded that pandoc couldn’t do it, but I could be wrong.)

Installation instructions are here

Get xhtml2pdf: xhtml2pdf is a python package that generates PDF files from HTML and CSS. It can be installed with easy_install or pip and also available to download here

If you are so inclined, get Compass: Compass is a CSS authoring framework built on Ruby and SASS. You write in SASS and Compass generates CSS files for you. The latter are what you actually run on the server. I’ve been using it for a year or so and I love it. One of the reasons I like it is just because I can avoid the curly braces and semi-colons of CSS itself. I’m addicted to significant whitespace. (Though it’s worth noting that they’ve been going after designers and have opted to make their SCSS syntax the default because SASS looks too much like programming).

More details on installing Compass here

If you need a great text editor, get Sublime Text 2:It’s all the rage, and for good reason. Get it here

Create startdoc.sh:

This copies the proposal template folder, renames it depending on your argument, and also renames the .textile file.

Create gopdf.sh:

This is where the action happens. It runs pandoc to convert from Textile to HTML – along with print.css – and then runs makepdf.py (which in turn runs xhtml2pdf, as we shall see) to take proposal.html and generate proposal.pdf.

Create the proposal_template folder

We need to create a standard proposal template that will thereafter be copied over to a new project any time you run ~/startdoc.sh. It consists of a folder structure which looks like this:

The folder sass contains the Compass SASS file; stylesheets contains the CSS that is produces from the SASS, namely print.css; config.rb is the Compass config file; and makepdf.py runs xhtml2pdf, but is itself executed by the gopdf.sh bash script. The folder fonts contains font-files that can be included in the PDF by using @font-face in the stylesheet.

Once all this is copied over by startdoc.sh, proposal.textile is where all the writing happens.

The Python script, makepdf.py, looks like this:

This is called in gopdf.sh, specifically this line:

So running ~/gopdf.sh proposal will get pisa to take proposal.html and generate proposal.pdf. Pisa, by the way, is the original name – or central core – of xhtml2pdf.

In the near future I hope to fill the standard print.css with lots of proposal-specific stylings, and proposal.textile will probably contain my standard section headings and table of contents.

By the way, when creating the standard template, the Compass stuff can be set up automatically by running the following command.

(If you like the SASS syntax, that is)

WordPress

I’m writing this in Textile right now, but it struck me that I only had a WordPress blog to post it to. I can use Textile for my philosophy blog, because I built the engine myself in Django, but I hadn’t got far in the past trying to get Textile working in WordPress. Thankfully, Veeti Paananen has provided a plugin, Simple-Textile which seems to work nicely.

It’s really time to move this blog, and a redesign is probably overdue.

Afterthoughts

If you’re not a coder, but you’re an author or just do a lot of writing, and you don’t have to worry too much about making your words look good enough for Bob of Bob’s Cake Company to print out — but you do want a nice, pleasing, peaceful, distraction-free writing environment, then there are some nice minimalist editors out there at the moment, such as OmmWriter and WriteRoom.

I get basically the same thing in Sublime Text when I put it in “distraction-free mode”, so my programming and web design editor also functions as a nice place to write about Wittgenstein, Heidegger, Karl Marx and the evils of traffic control.

But when it comes to writing something big or complex like a book or a thesis, folks sometimes opt for specialist software, such as editors that revolve around the LaTex typsetting markup system. These are usually built on the same paradigm that I’ve been describing, namely the separation of style and content, which is especially important with documents primarily intended for professional print publishing. Word processors are strictly for home and office use, and I imagine those in the world of printing never touch them with a bargepole. It must be galling that word processors are these days thought to be the last word in setting and styling type, because while they have cast that noble profession into obscurity – though certainly not into obsolescence – their own ability to do it well is tragically lacking. Next time you’re in a print shop ask them what they think of Microsoft Word, so long as you don’t find anti-Microsoft rants boring.

Thanks to all of the wonderful and generous programmers out there without whom I would likely still be shouting at Word like a madman.

Further Reading

I’m not the only one, you know.

Why Do I Hate Word Processors?
Word Processors: Stupid and Inefficient
What has WYSIWYG done to us?

After working primarily in Django for the last while I had to do some WordPress work yesterday, implementing SEO improvements to one of our e-commerce client’s web site. I added a function to functions.php and got everything working nicely, but then I went to /wp-admin only to find a blank screen, with no error messages. Neither could I find anything in the server error logs. I could get to /wp-login.php and seemingly log in succesfilly, but then the same thing would happen – I couldn’t get in to the back end, even though the public site was working just fine.

Most of the proposed and unsurprisingly successful fixes I found in blogs and on the WordPress support forums involved re-installing WordPress, but one suggestion stood out: delete any blank lines at the end of your functions.php file. I’m in the habit of adding extra lines at the bottom of a file to give me more space to work in my text editor, so this looked promising. And it worked.

I don’t know why.

I haven’t been blogging here much because I’ve been doing it over at my philosophy blog, critique-of-pure-reason.com. Go take a look and leave a comment, if you’re into that sort of thing. If you’re here for web development…I really should write something soon. I’ve just launched chop-chop.co.uk, and it’s worth a write-up, as it’s my first large project using Django – But I may wait until I’ve done the reservations system. Here’s a screenshot:

Ever since I learned about it I’ve made use of it in everyday life, increasingly without thinking about it; but it’s not something I would necessarily have picked up anyway. It’s not intuitive: I had to be told about it. It’s a different kind of thing from hitting a nail with a hammer, because it requires the application of a counter-intuitive principle; and yet just like using a hammer it becomes a part of my complement of skills and tools. Actually, perhaps it’s more that it’s a mental tool, as opposed to the hammer, which is physical. A hammer requires conscious use, but it requires no conscious application of scientific knowledge. We intimately know the physics of a hammer blow, but some phenomena, like granular convection, are remote from us: they are not built into our bodies.

In this video, the guy is demonstrating the effect with exactly the same muesli that I eat every morning. It’s ideal for the purposes of a demo because it’s full of brazil nuts. But he’s doing it wrong – all you have to do is shake it gently from side to side – so I’m sceptical about his explanation of how it works.

I’ve come to regard it as quite ordinary, but there is more to it than first appears. The particles actually divide and circulate in a flowing motion, very much like liquid convection. I don’t know how it works. But then, nobody really does.

Pony-Powered Enlightenment Bullshit

This sounds like bullshit, of course, and I confess I’m spinning this out more than is warranted, but there is an immense calm, neatness and harmony about developing with Django. Everything’s at just the right level; everything fits together in just the right way. Everything’s in the right place, everything’s in nice little pieces that fit together like a jigsaw.

You know when you read a textbook and you get the feeling that the author knows exactly what you’ll have trouble with? Or a teacher who always seems to be able to anticipate your difficulties. Well that’s what it’s like to work in Django. There have been countless occasions where I have exclaimed, after a bit of struggling, “oh, you mean I can just do this!” Just where you need it to be, it’s all done for you. You know the bits I mean: the tedious shit. The painful repeated Ctrl-C Ctrl-V crap that you go through every time you do a form.

Django is high-level, but in a low-level way. There’s no fucking about with shitty interfaces, arcane meta-languages and strange customs, just to put a website up that has editable content; there’s none of the horrible mess of PHP; there’s no need to download three thousand PHP files and work with a horribly complex interface just to set up a wee blog for God’s sake. With Django, you build things up yourself, in just the way you want. If you think this sounds like a lot of work, it’s not – it’s way less work, because Django knows roughly the kind of things you’ll be doing.

I confess that I never got around to trying a PHP MVC framework. No doubt all this great stuff I’m talking about is largely owing to the general MVC pattern, rather than to Django itself – but I’m not so sure. Django’s helper classes, such as forms, make the whole web development experience fun. Anyway, it’s so much nicer to write in Python.

Demon-Driven Enlightenment Bullshit

(This section doesn’t have much bullshit in it actually, but I couldn’t stand to leave the first two titles unbalanced.)

I’ve been blogging over at my philosophy blog, critique-of-pure-reason.com, which is why this is the first post here for a long – shit, it’s October already! That blog and the studies from whence its content sprouts have been taking up a lot of my time. Right now I’m studying scepticism, in particular Descartes, who suggested that for all I know I might be dreaming at this moment, or I might be the victim of a malicious deceiver demon who is creating all of my perceptions for me; so I cannot take my perceptions for what they seem to be, and so I have no ground for the belief that I am sitting at my computer writing; even though I bloody well know I’m not dreaming.

Now, you would think that “I bloody well know I’m not dreaming” would be absolutely inadmissable in philosophy, but most answers to Descartes seem to be saying something similar, especially those from Moore and Wittgenstein. Anyway, that’s enough of that. This is neither the time nor the place. This is the place if you’re interested. I did the blog engine from scratch in Django, by the way. Have I mentioned how much I like Django? Come to think of it though, the site’s sadly lacking in navigation, categories, tags and all that. Well, I did it over a single weekend so what do you expect?

So I’ve been staying up till all hours poring over philosophical papers from the journal Mind, just trying to prove that I’m not dreaming. It’s not as fruitless as it sounds though: mountains and valleys of thought are appearing from over the horizon. I’m moving ever nearer, exploring the foothills on my way to the dizzy precipices of [insert more bullshit here].

Not Enough Hours

And that’s not to mention all the music: saxophone, piano, theory and composition. And the maths. I’m working through a book of undergraduate mathematics, and Euclid’s Elements – oh and I’m half-way through a textbook on logic. Am I insane? Is this whole paragraph a cliché? I think it is you know. Fuck.

I was in the basement of Waterstone’s at the west end of Princes Street. Three young members of staff, one female and two male, were talking loudly. The males were teasing the woman in a deliberately petty, repetitive fashion. I was quite distracted by this incessant stupid chatter, and although I was in a public place and we can’t expect the silence of a library in a high street shop, I was irritated. I think I was justified:

1. Different standards and conventions – loose codes of conduct – apply on different floors of a bookshop. We do expect a quieter, more peaceful atmosphere on floors other than the ground floor.

2. There are chairs on these floors so that you can sit and read. In general, the shop itself creates the peaceful ambience or the expectation of it.

3. I’ve been in this particular Waterstone’s many times and the basement is normally very quiet and peaceful. This is probably one reason for my frequent visits.

4. On this occasion, there was no other sound in the room, so their loudness was unnecessary, uncivil and discourteous.

5. This loudness was just one aspect of their swaggering manner, in which they showed a disregard for the customers, acting as if we were not there and as if the whole room belonged to them. This was manifest in their movements and bearing, the way they would talk across the full width of the room even if there were people in between, and so on.

In any case, I was irritated. I was trying to choose between two books based on what I could gather from their prefaces about the standard of knowledge required to read them. The unpredictable monkey-like noises of the staff – interspersed with their occasional macaw-like screeches – made it difficult to concentrate.

Transformation

But then something unexpected happened. There was suddenly a new noise: whispering, between two customers, a middle-aged man and woman, who must have just come down the stairs. That someone should think it suitable to whisper was absurd enough, given that the staff continued to dribble out their inanities for everyone to hear; but, worse than that, the whispering was loud, considerably louder than most normal speech. I don’t have proof of it, but I’d guess that if they had talked at a normal, quiet volume, I would hardly have noticed them.

If you’ve never heard loud whispering, let me tell you: it’s bloody irritating. I think this is partly because it’s so stupid, more than because of the intrinsic qualities of the sound.

Anyway, there I was, trying to read a preface to a book on mathematics, my ears buffeted by irregular waves of jungle-like cacophony coming from the staff and scoured by the rasping voiceless speech of the whisperers; and I wasn’t irritated any more. I just smiled at the wonderful absurdity of my situation.

Logicomix, An Epic Search For Truth
By Apostolos Doxiadis, Christos Papadimitriou, Alecos Papadatos and Annie Di Donna

I have enjoyed comics since I was a child, when, fascinated and entranced by Hergé’s Adventures of Tintin, I argued heatedly with my friends, who all preferred Asterix.

I still read Tintin occasionally, and I enjoy more consciously grown-up graphic novels, but I never thought I’d be treated to a comic-book account of Bertrand Russell’s quest to establish the foundations of mathematics. The idea of such a book is incredible, yet here it is, a story told passionately and illustrated beautifully, which does justice to the men and their ideas. And they’re all in here: Frege, Hilbert, Cantor, Gödel and Wittgenstein….

Read the rest on my philsophy blog, critique-of-pure-reason.com »

Anyway, I was looking at Mark Dooley’s biography of Scruton on Amazon and noticed this 5-star review:

This certainly is not my kind of book. It is a Christmas present for which I was asked so clearly it IS the kind of book enjoyed by the person who asked for it.

But wait. There are CSS compilers, and then there are the libraries and frameworks and plugins that are built around them. Well, I guess that’s where the term “meta-framework” comes in.

One such meta-framework is Compass, which is built around the core language and compiler Sass. I mentioned Compass briefly in my other post. At the time, I had dismissed it as being more than I required, or more than I wanted to get into, but in the end I realized if I was going down this route at all I might as well do it right – and Compass does it right. It seems by far the most thorough, mature solution I’ve found, with a good heritage and a lot of success.

So that’s what I’ve settled on using now. There might be some kind of tighter integration achievable if you’re using Rails, but what I’m doing is using it to generate CSS as I locally develop a site in WordPress, and the same for a site built on the GetSimple CMS, and a Django project as well. You can use it with anything, and when you come to deploy it you can exclude all the Compass stuff, leaving no trace of it on the live site, except perhaps in the tell-tale auto-generated layout of the CSS code.

Your best bet is probably to go read the excellent documentation, but here’s some basic stuff from my own experiences…

Initial steps:

1. Install Ruby
2. Install Ruby Gems
(if you’re on a Mac you may have these already. I guess you could find out by carrying out the next step)

3. Install Compass from the command line:

With these installed, whenever I begin a project I set up Compass in the theme or template directory by running the following command inside that directory:

There are loads more options for the create command (See the documentation). Note also I’m using the SASS syntax, because I prefer it. There are two choices, and projects default to SCSS. See here for more information on the differences.

Incidentally, it’s interesting that SCSS is now the default syntax for Sass (confusingly, the name of the language/compiler is also the name of one of its two syntaxes). The SCSS syntax looks just like CSS itself, and the idea is to lessen the unfamiliarity for designers who are already very skilled in CSS. After all, there’s no reason in principle why they should have to learn a new syntax in order to take advantage of the power of Sass and Compass, which is independent of the syntax, and independently valuable.

However, I’m sticking with the SASS syntax, because I find it neater and easier to write, and I don’t mind admitting that it’s simply a welcome novelty.

That create command creates a configuration file, config.rb, and three directories, most important of which are src and stylesheets. src is the where you write the code, and stylesheets contains the generated CSS.

The last essential piece of the puzzle is how the CSS is generated when you’re developing. All you need to do is run the following command in the directory in which you set up Compass:

This runs in the background and watches for changes, then re-generates the CSS when it finds them.

Getting ready to code

I’m currently working with a config.rb that looks like this:

It’s good when starting out to use a _base.sass “partial” to set up your variables and imports. Mine looks like this:

See here for more on Compass best practices.

Mixins

Notice at the top of the base partial I’m importing another partial file, _my_mixins.sass. Mixins are wonderful: they let you collect together styles and mix them into your stylesheet wherever you like. Compass comes with many of them, but you can also define your own, which is what _my_mixins.sass is for:

These are used in the actual stylesheet source code (eg. in screen.sass) like this:

Grids

I’m also using the 960 grid system (Compass also comes with Blueprint). With Compass, using a grid system is very nice indeed.

One thing that’s good about this set-up is that you can code using a grid system without using any unsemantic class names like “prefix_2 grid_8.” Compass takes care of that when it generates the CSS. Effectively you’re using the grid as it really should be used, only in design and development. Compass calculates the relevant widths for your semantically-named elements, based on the 960 mixins, and there’s no underlying skeleton of unsemantic classes left behind in your HTML and CSS.

NOTE: Using +grid-container instead of +grid-system(12) means you don’t end up with all the 960 grid classes in your style sheet, which won’t be required because of your use of the mixins.

This is just the basics. There’s lots of stuff online, but you can’t do better than the Compass documentation, and it’s worth taking a look at the site of Chris Eppstein, the man we have to thank for it all.

I think I’ve unearthed a conspiracy. In the wonderful world of the world-wide web (that’s WWWWW for short), with so much information available, and so many companies selling online, surely there must be a collection of standard procedures for implementing an online shop, and reliable sources of information covering them? Apparently not. How else to explain this except an evil conspiracy? In particular, an evil conspiracy against web designers and developers.

This post is drawn from an e-mail that I sent to a tech savvy client, a small business owner. The email summarized their options for e-commerce, in the specific context of the UK, and it was the result of some hours of research. Reading it over later I realized it could be useful to web developers, given the aforementioned global conspiracy. It is therefore my duty as a good web citizen to reveal this sacred, jealously guarded information – despite the dangers.

NOTE: I’ve called it a starting point, but really this post is just a record of my own starting point, and there’s a bias here towards WordPress.

Introduction

There is one way of selling online that I won’t really go into here, which is selling through your regular PayPal account or with Google Checkout. These are pretty good options, because they’re so easy, but most businesses don’t want customers being directed away to PayPal or Google when they check out. A more integrated solution is preferable because it gives customers confidence in the professionalism of the business; the business can have direct access to its product, order and customer data within a single system; and you – the developer – get more control all round. That’s what this post is really about, but the shopping cart software mentioned here can work nicely with the simpler PayPal set-up as well.

An online shop that accepts credit cards usually needs these four things:

• An Internet Merchant Account (IMA)
• A Payment Gateway
• Some shopping cart software (including product listings, product pages, a cart and a checkout page)
• (in some cases optional) Shop CMS (admin for shipping rates, product management, inventory, order management, etc.)

The shopping cart and CMS can normally be treated as separate and independent. For the payment gateway and IMA, I would normally recommend PayPal’s Website Payments Pro, which bundles these together. But if a substantial portion of your market is in the UK, there’s a problem:

Maestro and 3D Secure

This is where it gets complicated. In fact, it looks to me like the whole thing’s a bit of a mess.

Mastercard enforces the use of 3D secure to process Maestro payments (and this is becoming very common for payments with the other cards too). All Maestro transactions will be declined if this isn’t implemented. 3D Secure is an extra step in the checkout process, which allows real-time authentication with the customer’s bank. It commonly comes in the guise of “Verified by Visa” and “Mastercard SecureCode”. In some implementations it’s optional – the customer can skip it – because not all cards are required to be authenticated like this, but a Maestro payment just won’t complete successfully without a 3D secure authentication.

Before you go down this route it might be worth weighing up the pros and cons of 3D secure:

Pros

• Lets you take Maestro payments
• Can give customers more confidence in buying stuff online (in theory)
• Reduces chargebacks
• It could be that, as people in the UK have become more used to it, they have come to expect it

Cons

• Yet another step in the process of checking out – this is known to lead to a reduction of sales
• Restricts your options for e-commerce software and payment gateways
• From a development point of view, implementing 3D secure can be tricky, depending on the kind of solution you choose

With Website Payments Pro, it’s up to you to set up 3D Secure. Seemingly as an afterthought they recommend using some kind of product from a company called Cardinal, but I’ve no idea what Cardinal actually provide, and their web site is no help. I sent them an enquiry but received nothing in response. This didn’t give me much confidence, so I looked for some other solution, i.e., a payment gateway that had embraced 3D Secure and knew how to implement it.

Turns out there is one that’s popular in the UK: RBS Worldpay. They offer three account options, including one that – like Payments Pro – bundles together the IMA and the gateway. The crucial thing is that WorldPay implements 3D Secure.

RBS WorldPay

At the time of writing I haven’t implemented the gateway, so I can’t say too much about it now. I also won’t say anything about WorldPay’s interesting (in the loosest sense) recent history, but you should check it out because it puts things in perspective.

What I’ve discovered so far is that there are essentially two ways of implementing WorldPay: HTML redirect (or Hosted Payment Pages, which appears to be the newer name) and XML Direct.

HTML Redirect is the easier option. When the user checks out they’re redirected away to the WorldPay site, where all the hard stuff is taken care of. You may be familiar with HTML Redirect from seeing this beautiful interface:

Yes, I know: there’s no way you’re having this ugliness anywhere near your lovely design. But according to the WorldPay site, you can customize this page, in line with your own design. I don’t yet know just how customizable it is: looking at what documentation there is (not an easy task in itself), I’m sceptical how far you can go with it.

I must warn you that browsing the WorldPay site for good information is a painful experience. It has the feel of a collection of documents assembled from disparate sources by a robot with little sympathy for, or understanding of, the requirements of human beings. I suppose this is fairly common with such massive corporate organizations, but it’s a real shock when you’ve been used to dealing with small, focused companies with a dedication to openness on the web.

It took a couple of messages to WorldPay technical support to find out that there was another way: XML Direct. This allows a more complete integration with your site, because all the processing is done in the background, rather than with a redirection. The implications are that you’ll need to make damn sure you’ve got a secure system. RBS have a process for assessing your set-up, and further procedures for testing and activation.

For an alternative to RBS WorldPay that also supports 3D secure, check out SagePay

PCI Compliance

Aside from any particular requirements of your gateway, generally a secure setup is one that is PCI compliant.

With HTML Redirect and the hosted solutions (option 1 below), you don’t need to be PCI Compliant at all, because you’re not actually storing credit card numbers on file or in a database. It’s your payment gateway (eg. WorldPay or Website Payments Pro) or shopping cart service that has to be PCI compliant. However, your gateway may still require you to pass some level of security-checking.

With XML Direct, you do need to pass some PCI Compliance tests, and that’s a whole other can of worms I’m not going to open right now, because I’m still not sure exactly what’s inside. Worms, yes, but what kind? And do they bite?

SSL/HTTPS/Encryption

You might not be storing credit card numbers, but you may need to transmit them securely, in which case you will need to buy an SSL certificate and have it installed on your web server. Basically, if your checkout page resides on your web server, as part of the site, you will need SSL. But if it’s hosted, as it is with WorldPay HTML Redirect or one of the complete hosted shopping cart solutions (option 1 below), you’re not transmitting anything sensitive so SSL is not required.

Software Option 1: Hosted

Examples: Ecwid, Foxycart, Shopify

With hosted e-commerce, you sign up to a service that provides a cart and checkout process, all hosted on their servers. General shop settings – shipping, taxes, coupons and product types – are all administered on their site.

The big advantage is that they they’re the ones who need to have the SSL encryption and security, and they wrap up the whole checkout process very nicely, so there’s not much in that area you have to worry about.

I’ve set up a shop with FoxyCart and MODx, and while there’s a lot of work to be done if you need product management built into your CMS, setting up the customized cart and checkout pages is a breeze.

• They handle SSL and PCI compliance
• No 3D secure/Maestro (because they’re responsible for implementing it and Ecwid, Foxycart and Shopify haven’t done so yet, although they’re working on it)
• Your data is on their servers
• Separate place to manage shop (ie. separate from the rest of the website)

Ecwid (Free)

• Very fast, very easy, and very tempting
• But no API and not very configurable
• Depends on JavaScript

Foxycart ($19/month)

• Very configurable
• Total control over the design of the hosted pages
• API and XML feed to get data from their servers to yours (allowing you to integrate this data with your database and CMS)
• Cart depends on JavaScript

Shopify (From $29 to $699/month)
Look at that price-range. That indicates a pretty mature kind of product, and I get the feeling this might be one of the best hosted solutions out there, but I don’t know much about it. But it’s built with Ruby on Rails so it must be super cool, right?

Software Option 2. Self-hosted / Integrated

Examples: WordPress plugins WP e-commerce and Shopp

I recently split off a WordPress site into two development branches, one with Wp e-commerce and the other with Shopp. I’m very impressed with this type of software solution. I don’t see any reason why you couldn’t implement any size of shop, and there’s loads of CMS features covering just about everything. I went with Shopp in the end, but they’re pretty similar in functionality.

• You have their software installed on your server (ie. the server I’m hosting the site on)
• You keep all the data on your server
• You don’t have to log in to another site to manage your shop
• Shop CMS for product, order and customer management etc., integrated into your site CMS
• Configurable
• Often requires an SSL certificate
• Could require PCI Compliance, depending on how you implement it
• Maestro support is possible (if using RBS WorldPay or something similar like SagePay)
• But no WorldPay gateway option out-of-the-box

WP e-commerce (free, or $195 for the business gold cart)

• Open source
• Configurable, customizable
• Ability to create a custom gateway integration
• The free version withholds some functionality that in most cases is a necessity, such as multiple images per product. You could probably code this in yourself if you wanted to avoid the cost.

Shopp (One-off $55)

• Open source
• Configurable, customizable
• Ability to create a custom gateway integration

Software Option 3: Self-hosted Standalone CMS

Example: Magento

• You have their software installed on your server, alongside your web site CMS
• All-in-one shopping cart, product and order management package
• You keep the data on your server
• Open source
• Separate dedicated CMS for your shop (not integrated into your site CMS)
• Tonnes of features
• Potentially difficult to configure and integrate
• Requires an SSL certificate
• Could require PCI Compliance, depending on how you implement it
• Maestro support possible (if using RBS WorldPay or SagePay)
• But no WorldPay gateway option out-of-the-box

Something like Magento should be seriously considered. It’s very featureful, and I hear they recently implemented 3D secure. However, setting up and configuration could be time-consuming, and systems like these can be pretty bloated, slow and clunky.

Conclusion

If you need to take Maestro payments – and there are still a lot of those cards around – you can’t really go with option 1. The other options all have built-in support for several gateways, but unfortunately not WorldPay. This means some unavoidable work on your part to hook things up. Whether you go with option 2 or 3 depends solely on the needs of the client and your own development decisions.

In the forthcoming weeks I’ll be implementing the gateway – perhaps developing a WorldPay module for Shopp in the process – so I’ll be posting about that. If the Guardians of the Sacred Secrets of E-commerce haven’t tracked me down, that is.