The main difference with this new project is that I've decided to do it alone. No other developer, no designer, no marketer, no seller, etc. I have to do it all by myself.

While it has a lot of advantages, doing things alone require a lot more time. Especially when I'm doing tasks in areas where I'm less comfortable, like marketing. Also, because I'm alone, it's tougher to make decisions, because I'm the one asking and answering the questions.

As it's still a side project, and freelancing is still where my income comes from, I have to minimize distraction in order to keep things moving.

Selling My Shares of Signsquid

As you may know, I'm one of the cofounders of Signsquid, an electronic signature SaaS. We first created that application around 2011. We were 6 cofounders with a lot of enthusiasm. To this day, I still had my shares in that project, which means that I had to do some level of support.

Lately, it required more and more support time, and I did not believe that the team, as it was before I sold my shares, would be able to bring that SaaS to a level where we can say it succeeded. I did not consider that the time I put in support was an investment. So, I decided to sell my shares to the other guys. Which in turns, motivated other cofounders to rethink their involvement in the project and also sell their shares. The new team is now much leaner and in a better place to succeed.

On my side, I feel really relieved of what became a burden.

I can now focus on my current projects, which have more chances to succeed, and are also much more fun.

The Client That Wanted an MVP

A few months ago, a prospect client asked for an estimate to build an MVP. That MVP would have access to users bank accounts and be able to perform real ACH transactions on their behalf.

The client wanted to build something extremely quickly to be able to prove the market fit of his idea, which really does make sense. So I crafted an estimate that would include the desired minimal feature set. He was surprised by the amount of time needed to build that MVP (a few weeks).

For many reasons, I think he is a smart guy with a great business idea. However, he had some misconceptions about what it means to build a web application.

For a lot of people, the difference between a website and a web application can be unclear, and that's really understandable! I have to explain it from time to time to clients, friends and family members.

So, that client asked for some explanation. Why would it take so much time to develop his app?

A Website vs An Application

First of all, let me explain some differences between a website and a web application. You can use both through your web browser, like Chrome, Safari or Internet Explorer, but conceptually, they are not the same.

A Website Is…

… content that you consume, that you read. Here are some examples:

• A restaurant's website where you can lookup the menu and the phone number.
• A furnitures factory's website where you can see a list of what they make, how to apply for a job, phone numbers of each departments, etc.
• A blog or a newspaper's website, where you can read articles.

A Web Application Is…

… software that you can use to do things. In most cases, you can log in. Here are some examples:

• A timesheet software where you can enter the time you worked at your day job.
• A to-do application, where you add tasks that you have to do, and check tasks that you have done.
• A bank's web application where you can log in, see your latest transactions, send money to someone, pay invoices.

The Line Is Blurry

It's not always super clear. Here are some examples where the line is blurry:

• You can reserve a table at a restaurant using their website, this is handled by software.
• Some administrator has to log into a content management system (CMS) to add/edit/remove content from the furniture factory's website.
• A user can log in the newspaper's website and post comments to articles.
• The bank has a regular website, where it showcases it's product. On the top right corner of the page, it says that "You are logged in as Antoine Leclair", which has to come from the application.

Developing a Web Application Is Not Developing a Website

The lifecycle of a website is not the same as the lifecycle of a web application.

The Lifecycle of a Website

The lifecycle of the code base of a website is similar to a brochure. Some process (meetings, design, production, etc.) leads to a final product. The brochure is printed, ready to be read. The website is online ready to be read.

At some point in the future, it will get too old. A new one will be produced. The same process starts over (meetings, design, production, etc.). The new one is ready, the old one is discarded.

The code base of the website during his whole life will not change much. A new social network appears, we must add an icon. The client did not realize he needed a page that let people know how to get to the factory, a new page must be added, with a new link to that page on the home page. If the content has to be changed (more articles in a blog, new products in the catalog), the code base will rarely change. The changes will rather be done in the CMS without touching the code base.

You rarely find bugs in a website because the code is simple: it gets data from a database and shows it in a page, it affects how things are presented (the color for example), it performs simple animations. If there are any bugs, they are generally really simple to fix and deploy. Even someone unfamiliar with the code base will be able to do it easily.

The lifecycle of a Web Application

The lifecycle of the code base of a web application is extremely different. It really is a software dressed like a website.

You can rarely foresee every aspect of a software before actually using it. There's no solution, it's like that. Software is intangible. Talking about it without touching it only make things worse. So, the way successful software is generally done is by delivering the minimal feature set that has value for the users, and then augmenting/changing that feature set as time goes.

It is rarely a better idea to create a new version of a software and discard the old one. For example, if you discover that your timesheet application would also need to do some accounting and that users should have the ability to add details about the hours worked, those features will be added to the existing application, instead of creating a new application.

This requires a code base that is evolvable.

Some Considerations When Building a Web Application

Automated Testing

In order to make the code base elvolvable, the first thing that comes to my mind is automated testing. There are many types of automated testing: unit testing, functional testing, integration testing… I won't go into the details of each type. Some types of testing may be more appropriate than others depending on the project.

An automated test is code that will reproduce a use case in an automated manner and verify the result is correct. It could be code that tests that when feeding 10 to the function compute_taxes, it actually returns 1.52. Or it could be remote controlling a real browser to simulate that a user logs in, fills his daily timesheet then logs out, making sure that the changes were saved correctly and the side effects (update a report, etc.) were done properly.

To someone not familiar with writing software, it may seem like trivial test examples that do not really add value. I'll give you some examples where those tests will come in handy.

• Three years after the software has initially been deployed, and the person that wrote that code does not work there anymore, the business expands and the function compute_taxes must be updated to compute the taxes according to the state where the product is sold. Instead of receiving only 10, it will also receive NY or CA, and the function should return the tax amount according to the state. You can update the existing tests to include the state that was implied before (for example, CA), that make sure the working case still works after the modification, then add new tests to also test other states. The developer does not have to manually test each cases. He just runs the tests, and that way, he is sure all the application still works correctly.

• A bug report comes in. The taxes for NY are not computed correctly. If the code base is not covered by tests, the developer will look at the code, fix the bug, and test the application manually to see if the bug from the report he received is now fixed. It seems to be fixed, so he pushes his change in production. But by fixing the bug, he also introduced another subtle bug, that made the tax calculation wrong for every other state. If the code base is covered by test, the developer can run the test suite and catch the error right away. Also, since there was a bug in the first place, the developer will write a new test to make sure this bug never comes back.

• A new requirement is that the timesheet includes a new box to enter the overtime, separated from the regular time. The existing browser test should still work. It may look like a very simple change, but it's often in those unexpected cases that something breaks somewhere else. In this example, loading the reports could make the application crash. The browser test will find it really fast.

As most bugs are discovered as the code is being written, instead of receiving bug reports, fixing them as they are introduced is much faster. Also, the test suite is there to catch other bugs that the fix could introduce.

To summarize, automated testing really helps to keep a code base evolvable.

Pre-made modules

When building a website, pre-made modules/plugins are often useful. Need to create an animated slideshow? There's a jQuery plugin for that. Need a private section on your website where only authenticated users can upload files? There's a WordPress plugin for that.

When building a web application, using those pre-made modules is often not the optimal solution for many reasons.

1. While it's normally accepted to adapt the requirements of a website according to what the existing plugins can do, it's normally not the case for a web application. For example, it could be important that a user provide his mobile phone number while signing up, but the existing modules do not ask for that information.
2. The requirements of a website will rarely change over time. If they change, there are good chances that a new website will be built to replace the old one. The requirements of an application do change. If a pre-made "users" module was used, to allow users to authenticate and access a private area, when the requirements change (they will), it could put you in a place where you can't modify the application to add the new behaviors. For example, when first writing the application, we did not need to ask for the phone number during signup, but two years later, we need that information to send an SMS near the end of the signup process, but the module that was used does not allow it.
3. It's often really hard to incorporate the pre-made modules into a complete test suite. They generally lack the access points that would be needed to test correctly, in the context of your application.
4. Using pre-made modules is often more about hacking the module to make it work in your context. Hacking code together to ship a website is perfectly acceptable. Doing the same for an application will make very brittle code, with really big consequences.

When building a website, it makes sense to use those modules because it makes it so much faster to ship! Not using it could mean 10x more time needed. Since you don't need to maintain that code, you would rather code 9 new websites over time than investing time in the first one to make it really maintainable by writing custom code. The first one will be thrown away in 2-4 years anyway.

When building a web application, you can't afford to lose the flexibility that pre-made module will take away from you. It could be acceptable in some cases, but probably never in core parts of the application. For example, if there is an animated slideshow in the application, it's probably OK to use that jQuery plugin. Replacing it with another one in the future will be easy since there is nothing that really depends on it.

However, using pre-made "users" modules, or "administration area" modules, is often a really bad idea when building software, because it will lock you in using those modules forever, with their limitations.

APIs and Mocking

When building websites, we sometimes have to access external APIs (data access points that software can use). Most of the time, for a website, it is very straightforward:

• Fetching the latest tweets of a user and displaying them on a page.
• Getting the latest value of a stock and displaying it on a page.
• Getting the weather forecast of a city and displaying it on a page.

You get the pattern: read data from one access point, output it on a page in the desired format. Even better: there are often libraries or code examples that do the data-fetching for you. Copy-paste and tada!

When building a web application, the consumption of those APIs are generally not that straightforward.

• There are often many calls to perform in a sequence.
• Data has to be saved in the database.
• That data in the database has to be architected to fit the exact business needs of the current application. For example, if a user enters a credit card (that is submitted to an external API), in your exact business needs, that credit card could be associated to a group of users, even if in the external API it's associated to only one user.
• The API is only a data-access layer. It's very easy to do one API call. But it requires time to code all the meat around it. For example, if a user entered a credit card to subscribe to your service, it would probably mean that you have to keep track of the payments received for that user, and keep the user account active as long as the payments allow it. That means doing something special when the account becomes inactive. Also, what do we do when that card has expired? What do we do when a payment fails?
• Since we are generally also sending data, there are generally error cases that have to be handled. The user entered a wrong credit card number, we have to tell the user that something failed.
• External APIs will also want to tell your application about events that happen. You will have to write code that listen to those events and do something about them. For example, a user subscribed to your services using his credit card. You'll have to listen for successful payments to increase the duration of the subscription of the user.
• A call to an API could fail because the external provider is down, or there is a network problem. You generally have to handle it in more complicated ways than "just hide the tweets if you can't read them", as you would do on a website.

Also, since those APIs are external services, it can be hard or impossible to write automated test suites (see above) that incorporates them. There are few reasons for that:

1. Every time you begin each of the automated tests, you have to be able to reset the environment to some initial known state. For example, if you had a test where a user logged in, added time to his timesheet, then looked at the updated report, you have to be able to reset the hours to some initial state before the test. Otherwise, every time you run the test, one more entry would be added, and the updated report would be unusable for testing because you will not be able to make sure that the last x hours have been entered, because they are already there 240 times from the previous runs of the tests. You need to be able to reset the state. External APIs often do not have that ability. They sometime only offer a sandbox where you can't reset the state to an initial condition.
2. Some API providers don't even offer a sandbox. It means all the calls that your application will do, even during testing, will be made against the live systems. It's impractical for some, if not most, testing. For example, you would have to submit a real credit card number and do real transactions on it. Every time you run the tests. Which is many times a day.
3. The external API sandboxes are extremely unreliable. They are often offline, the provider closes your sandbox account or changes your settings without telling you, they are testing new features that breaks the old ones, etc.

Since you can't include them in your tests directly, you have to create something that behaves like them for the context of the test cases. It's called a "mock". During the tests, your application will talk with that mock instead of talking with the real API endpoint. It's a mini-version of the external service. For example, if you mocked a payment provider, it could accept all credit card numbers except for one, that will be used for testing when the user enters an invalid number. The payments could always work for all credit cards but one number, where the payments always fail. Or it could refuse payments of 29.99 on one specific card number. Or it could only allow for 2 payments of 29.99 on one specific credit card number, then bounce the next ones.

Since you wrote that mock and you host it, you have complete control over it:

• It will always be online when you need it.
• You can reset it when you want.
• The reset you apply brings the state to the exact case that you need.
• You can simulate specific use cases e.g. two payments that works, then a third one that fails, etc.).

Those mocks are really needed and are really useful, but they also take time to write.

Deployment

Deploying a website is generally done by uploading the new files to the server that hosts those files. The developer sees that it looks good on when he worked on it on his laptop, then he uploads it. Or even faster, some directly edit the files on the production server.

Deploying a web application is generally more involving. You push your work in a source code repository (think GitHub), a continuous integration (CI) server builds the source (more on that below), runs all the tests, and allows you to deploy if all the tests are green. Doing the setup of the CI server (how it pulls the code from the repository, how it builds, how it runs the tests, how it deploys) requires some time.

Also, there's often a staging server to maintain, where a copy of the application is running fake accounts. That way, you can demo new changes for approval without deploying to the production server.

You must use those good practices when developing an application because otherwise, bugs that did not occur on the developer's laptop emerge in production. The exact source of the bugs is always a surprise, but it includes:

• The timezone was not the same on the developer laptop and on the production server.
• The developer runs Mac OS X or Ubuntu, but the production server runs another operating system, like Debian.
• The developer has a library installed on his laptop, but it's not installed on the production server. This allowed the developer to scale an image on his laptop, but doing the same thing on the production server crashes.

Database Migrations

When a website has been deployed, it rarely happens that the database schema will change. For example, it will rarely happen that, suddenly, you need captions in the pictures of the slideshow, while you did not need it when the website was first published. Also, when that happens, you can connect to the database, write the command that adds the new column, and execute it.

For an application, a change to the database schema is everyday life. But, unlike the website, you can't just connect to the database and write some commands. It needs to be executed in a reproducible manner. Some reasons are:

• More than one person may work on a project. Everyone has to keep their schema up-to-date.
• There are more than one server to apply the changes: production, staging, etc.
• The changes has to be tested before they are executed in production. Some changes may fail or have unwanted behaviors. This has to be correct the first time it runs in production.
• Many different features may be developed at the same time in parallel. The database schema has to match the current branch of the code. In order to do that, you have to apply changes and also be able to apply the reverse of those changes easily when you change branch.

This is why we need database migrations: a system to do database schema changes in a reproducible manner. The simplest form could be a folder containing SQL files. But it's generally in the form of code that apply those migrations for you when you ask for it.

Doing the setup of the migration system and writing proper migrations is another task that requires time and is not needed when building a website.

Compiled and Minified Resources

When building a website, you may want to minify your resources. That means that you take your JavaScript and CSS (style sheets) and use a minifier that will reduce the size of those files by using many strategies, like change long variable names like numberOfChildren to b and removing line breaks. Also, there are new languages that developers sometimes prefer to use instead of JavaScript and CSS, that will be compiled by a compiler to become JavaScript and CSS. CoffeeScript and SASS are two examples. When you build a website, building those resources is something that could be done manually. For example, just before you deploy, you minify all your resources by using the command line.

When building a web application, that process has to be automated. Some reasons are:

• You do not simply copy your code to the server when you want to deploy (see deployment above). Some other server will deploy for you.
• The code is kept under source control. When the code is compiled and minified, it's generally on one single line, which means that the whole file that contains all of your JavaScript changes every time you change one single character in one of your file. This result if huge changes in every commit, which makes the repository bigger (takes longer to clone) and harder to understand.
• The minified version also needs to be tested by the CI server.
• The version deployed has to be the one tested on the CI server.
• The code changes a lot. Building the minified files manually every time would not make sense.

Doing the setup of the automated minification/compiling is something that also requires time and could be skipped when building a website.

The Discardable Prototype To Test Market Fit

Until now, I talked about a website vs a web application. I used the website comparison, but it could probably also be valid with a prototype application that you only want to use to discover if your idea has a market fit.

If you intend to throw away your prototype application right after you discovered if your idea has, or not, a market fit, you could consider prototype applications to have the same characteristics that a website would have. In theory that is true. You will never need to maintain the code, so you don't have to bother about making it evolvable.

However, there are some more considerations before you can go that path.

First, it's really easy to fall in the trap of thinking that you will discard your prototype application really soon. In practice, the code base rarely gets thrown away. A total rewrite is extremely rare. The shortcut you take now will stay with you for a really long time. It will cost you money and agility.

Second, it's not always possible to do a quick prototype like that. I'm going back to the example I started with: the client wanted an application that would have access to users' bank accounts, see transaction history and other data and perform ACH transactions. This is a case where the application will store really sensitive data and would be able to perform really sensible operations. A bug or a security breach would be a disaster for the users, the client, the developer and everyone in between. In those cases, you have to write reliable software. That means using the best practices mentioned above, like automated testing.

Back to Grawl

Like I was saying in the introduction, I'm currently pushing Grawl to bring it to an MVP level. In this case, I'm my own client. I would really like it to get there faster.

I'm currently writing the code that handles the credit cards. It seems quite simple at first, but a lot of use cases have to be covered by the automated tests. Otherwise, I could be charging too much to someone, leak private data, allow someone to use the application without paying the correct amount, etc.

It takes time.

But building that part properly is required. That's not an optional feature.

In Summary

This was a super long article, but I think writing a shorter one would not have covered enough to be taken seriously.

Building software the right way requires time, but everyone wins in the end.

If you don't have time to read, the tl;dr is that it gives three examples of people who succeeded using Ramit Sethi's course.

By reading in diagonal, it looks great. But looking closer, they all became coaches and that's how they make money.

Sounds like:

— Give me 1000$ and I'll teach you how to make money.
— OK, here's your 1000$.
— See, that's how I make money.

What Have YOU Done?

When you start a business, one of the things people do is networking. Not like Linkedin networking, but meeting real people in cocktails and events. You soon realize that some people play the game of looking successful in the hope of bringing success.

This type of people, I'm sure there's a name for them, are also often involved as judges and mentors in startup competitions, hackathons, etc. They can be very convincing when giving tips.

But, how can they teach something useful if they have never been successful themselves?

If you are starting in business, it's important to be aware of that type of people. They may look like they can help you, but in reality, they will just consume your time.

Asking the opinion of prospect clients is probably 1000x more productive than to ask those coaches.

To be clear, this is not a rant against Ramit Sethi. I have not read his book nor taken his course. I don't even know how successful he was before being coach. As far as I know, his book is about personal finance and not directly about entrepreneurship.

A Two Year Pause

I've been extremely busy over the last two years. I have been freelancing with 10x Management. They bring me a lot of fun projects to work on, they are awesome! I also had a baby, she turned 2 a few days ago. For those who have had babies, you understand how time consuming it can be! I also spent about a year in Spain, which implies first moving to Spain, and then moving back to Canada.

During those two years, I have put aside pretty much every other projects I had. Building robots and electronic projects, playing music, doing kung fu, building startups.

But now! It's time to get back to building startups!

As some of you may know, in the past, I cofounded Signsquid, an electronic signature web application. I was also part of the early days of Snipcart, an HTML and JavaScript shopping cart, which is really taking off!

I'm a software developer, so when I was part of those projects, I was mostly concerned with the technical aspect. I've never spent much time doing the business part: marketing, closing deals, etc. I've always lived with the idea that I could not do it myself, that I needed other cofounders to do it. But these days are over!

Now, I'm not saying that I'm not welcoming a business guy as a cofounder! I'm saying that I don't want to wait for the perfect cofounder.

Grawl, The Project That Needed Some Love

Last week, I've read a blog post by Nathan Barry. It's called Knowing When to Quit. To summarize a little bit, he says that when a project is just sitting there and doing nothing, you should either kill it or give it the effort it deserves. It's really a great blog post that inspired me, because I have this project I did back in 2012-2013.

Grawl.

It was never quite the MVP I needed to test the market fit. It was abandoned. Here's a screenshot of how it looks like today (February 2016) and have been for the last few years:

Grawl, at its heart, is a SaaS that will find broken links on your websites. For example, it will find links that point to 404. The main difference with existing tools, like Google Webmaster Tools, is that Grawl will keep the list of websites you have and alert you by email when it finds broken links. It crawls all of your websites every months and it sends you an email when it's done. Also, since it's a specialized tool for broken links, it opens the door to cool features, like providing an actionable checklist of all the links to fix, etc.

It's not revolutionary, but I'm sure it has value for someone. Agencies? Big website maintainers?

The 100 First Paying Customers

At the time of writing, there are 11 users in the database. Mostly people that I told about the project in 2012-2013. We could say 0. Also, since there is currently no way to get a paying account, they are not paying customers.

Let's change that.

My first few steps toward my goal of getting 100 paying customers will be:

• Update the code base and infrastructure to 2016 (more on that below).
• Add the ability to charge customers: plans, credit cards, etc.
• Build a real MVP website.

After that, I will be ready to attack the market.

Regarding updating the code base to 2016, I don't plan to rewrite anything. It's easy to fall in the trap of changing framework for the goût du jour framework. Here I'm talking about fixing stuff that broke over the few years the project was abandoned, updating libraries version, having proper backups, etc.

In Summary

This post is more or less the kickoff of my efforts to bring a new SaaS to its first 100 paying customers. I'll try to post a lot more often, about the progress, the strategies, the success and failures. Stay tuned!

So I bought a few ATmega168's on ebay and an usbasp. Well, a cheap Chinese version of usbasp but it feels solid and is pretty small.

Then I decided to try my new things. I connected all the wires properly and plugged the usbasp in my laptop. I run Ubuntu 14.04 as I do this if that might help others in the future. First thing is to try to speak to the ATmega168:

avrdude -p atmega168 -c usbasp

I got this error message:

avrdude: Warning: cannot query manufacturer for device: error sending control message: Operation not permitted
avrdude: Warning: cannot query product for device: error sending control message: Operation not permitted
avrdude: error: could not find USB device with vid=0x16c0 pid=0x5dc vendor='www.fischl.de' product='USBasp'

avrdude done.  Thank you.

I guess I can change permissions later, but for now:

sudo avrdude -p atmega168 -c usbasp

Little bit better, but still not working:

avrdude: warning: cannot set sck period. please check for usbasp firmware update.
avrdude: error: programm enable: target doesn't answer. 1
avrdude: initialization failed, rc=-1
         Double check connections and try again, or use -F to override
         this check.


avrdude done.  Thank you.

I looked all over the web trying to figure out what was going on. Turns out the cheap Chinese usbasp's have a firmware that does not allow to set manually the clock speed. It's automatic. So when avrdude tries to set the clock speed and the usbasp does not change it, it complains. But since it's automatic, it should not cause problem. You can always try to update the firmware of your usbasp, which should let you change the clock speed manually, but you'll loose the automatic and will always have to specify it.

Since the clock speed was not the problem, I double (triple, etc.) checked the connections. I found other diagrams for the pinout of the usbasp, where every pin is reversed, mirror style. But this is just because some diagrams show the pins as they come out of the usbasp and others show it as they come out of the ribbon cable connector. After trying to reverse a few times, I found out there was a red wire in the ribbon to indicate Vcc. OK, some might think I'm stupid but when you don't know... Anyway, all my connections were right in the first place and it was not the problem.

Then I remembered that Arduino chips have some fuses set so they use an external oscillator for their clock, instead the internal clock. When the fuses are set to use the external oscillator, the chip is dead without it. So I had a look at the ebay auction where I bought the ATmega168's, and it says they are preloaded with the Arduino bootloader. Crap. And I don't have an oscillator at hand.

By searching more, I found this forum post. If you run in the same problem, I suggest you read it.

Basically, I guess the best way would have been to use an oscillator, but he suggests a few other ways to hack something to replace the oscillator. He suggests using a 555 timer at around 1 MHz. I have all that is needed, but I would have to select the proper RC components and wire all that together. Time consumming and too demanding at 2 AM. The other solution, which is nice, is that you can use another AVR chip as a clock. All my ATmega168 have an Arduino bootloader, so I can't use them right now. But I have an Arduino Uno that is very easy to program. Just plug it in USB and use Arduino IDE to upload.

So the code looks like this:

#include <avr/io.h>

int main(void) {
    DDRB = 0xFF;
    while (1) {
        PORTB ^= 0xFF;
    }
    return 0;
}

I didn't translate it to Arduino, since it compiled and uploaded without complaining. You can connect any pin of PORTB of the Arduino (that is 8-13) to XTAL1 of the ATmega168.

I took the default fuses for ATmega168 of this calculator.

sudo avrdude -p atmega168 -c usbasp -U lfuse:w:0x62:m -U hfuse:w:0xdf:m -U efuse:w:0xf9:m

It worked and I can now program my ATmega168's. Hurray!

Idempotence is a mathematical concept that should be understood by all developers.

An operation is considered idempotent when doing it more than once is the same as doing it once.

For example, multiplying by 1 is idempotent.

x * 1 == x * 1 * 1

Multiplying by 0 is also idempotent.

x * 0 == x * 0 * 0

The key concept to remember is that applying the operation once can have side effects, but applying it more than once will not do anything more that what was already done the first time.

Assignment is idempotent.

x := 4

You can assign 4 to x as much as you want, x will still be 4. But assigning 4 to x one time is different than zero time.

HTTP Verbs

HTTP verbs can be classified as idempotent or not.

DELETE is an idempotent verb. No matter how many times you do it after the first time, it will give the same result as the first time. For example, DELETE /users/4/contacts/3 could remove your contact with the ID 3. If you call it again, that contact has already been removed and nothing more should happen.

GET is also idempotent. In fact, it's more than idempotent. It is considered a safe method. Safe methods can be compared to multiplying by 1. Doing it zero, once or more times should have the same effect. All GET does is get a resource. For example, you should never use normal links to delete resources.

POST is not idempotent. Every time you do it, you can expect a side effect to happen. For example, every time you POST a contact form, an email is sent.

When it comes to APIs, that concept is well understood by consumers and providers. Designing around it will result in least astonishment.

See Wikipedia: Safe Methods and Idempotent methods.

Message Queues

Let's say you build a web app to manage events. You can add people to the invitees of an event. In other words, an event has many invitees. For quicker response time, you decided to send all emails from a worker. So when a user finalizes an event, a message is queued. A worker gets that message and sends the invite emails for that event.

This is a really common pattern, and if you are not doing it that way, you should really start to.

You realize that you had an SMTP problem and all your emails have not been sent for some time, and it's not even showing up in the logs! You think "oh, well, I'll just call the function to send the emails again", but you don't know for which events you should call that function.

Here comes idempotence.

When you execute a task in a worker, always make sure it's idempotent

For the email example, every time you send an email to an invitee, you can keep the datetime in the database, in the event invitee row. If it has already been sent (sent datetime is not NULL), don't send it again. As easy as that. Also, you might want to check if the event is finalized. If not finalized, do nothing.

More generally:

• Check that your job is ready to be executed (e.g. event finalized). If not, do nothing.
• Check that the job has not already been done. If done, do nothing.
• Do job, keep datetime or something else in the database, log that it has been done (e.g. "INFO Invitation email for event 234 has been sent to john.doe@example.com").
• Keep jobs granular. You have 5 emails to send? Queue a job (e.g. send_event_emails(event_id)) that will queue the 5 other jobs (e.g. send_event_email_to_invitee(event_id, invitee_id)).

You realize something went wrong? You can always call your function to send emails on all events. Still crashed when half of the emails were sent? Fix what was wrong and just call it again. Also, it's easy to inspect what emails have not been sent yet. Bonus, you can do some intelligence with the datetimes (how many emails a day do we send? what are the peak hours?).

Also, some message queues don't garantee that a message will be delivered only once. Amazon SQS is that way. Your workers should really only do idempotent tasks.

SQL Migrations

In the same spirit as the worker example above, when you do an SQL migration, do it in an idempotent way when possible.

For example, you decide to split the user table in two tables. One for basic informations (users) and one for all details that are not always important (profiles). You put a foreign key user_id in the profiles table. You have a migration that takes every row in users (SELECT * FROM users) and inserts a row in profiles with user data. You run it, and well, it crashes midway after 1 hour, because of some NULL value you didn't think about. You fix it and run it again, but you then realize that some users have already been processed and have two profiles.

The idempotent solution: instead of SELECT * FROM users, you can just select the users that don't have a profile row. That way you can run it as many times as you want. It will only process the few users that have not been processed yet. A big advantage of that method is that you can leave your app running in production while you do the migrations. When you are ready to deploy the new code that uses the profiles table, you can call your function again to make sure the latest users that signed up are also migrated. That example is not so great, because a user could change some information in the users table during the migration, but I guess you get the point.

Denormalized Data

You have an application where each user has many documents. They can search their documents by tags. Tags come from many sources. The title of the documents, the folders, actual tags, the names of the authors, etc. You decided to keep a table named tags where you keep all the tags for every documents, it looks like this: id (hash), tag (actual text of the tag), document_id (foreign key to the document). When you add an author to a document, it does an insert in the tags table. When you remove an author, it finds the good tag and removes it.

Someday, you see in the log there was an error every once in a while when inserting a tag because of an obscure character encoding issue. You fix the issue and deploy the new code. However, there are a lot of missing tags and you have no easy way to fix it manually.

Instead of just having functions of the type add_tag_for_new_author, you should have a function of the type update_tags_for_document. When you call that function, instead of just adding a tag for the author that was just added, it checks all the document, rebuilds the tags list and makes sure that the correct data is in the database. That way, the tags table is really managed as it should be: a cache. You could delete all rows from that table and just call update_tags_for_document on every documents. It takes 2s to update the tags for a document? Let the worker do it, queue a message.

Conclusion

If you were not aware of idempotence, I hope I convinced you to use it. Also, please note that I kept the examples simplistic for educational purpose.

On the VPS, the app was running using Waitress behind nginx.

I had to make some minor modifications to the code to make it run on AppFog, which were not really documented anywhere. I decided to post it here, in the hope someone finds it when needed.

Configuration File

First of all, I created a new configuration file. I named it appfog.ini, but name it whatever you like. I removed the key sqlalchemy.url, since it is set in environment variables by AppFog. Also, for the app to know that it has to retreive the environment variables instead of using sqlalchemy.url, I added the key/value appfog = true.

SQLAlchemy Connection

In main function of __init__.py, I replaced the line where the engine is created from the configuration file to take into account that the connection information could come from environment variables. So I replaced:

engine = engine_from_config(settings, 'sqlalchemy.')

with:

if settings.get('appfog') == 'true':
    engine = appfog_engine(settings)
else:
    from sqlalchemy import engine_from_config
    engine = engine_from_config(settings, 'sqlalchemy.')

And I defined appfog_engine later in the same file:

def appfog_engine(settings):
    from sqlalchemy import create_engine
    import os, json
    all_config = json.loads(os.getenv("VCAP_SERVICES"))
    config = all_config['postgresql-9.1'][0]['credentials']
    connection_string = ('postgresql+psycopg2://%(username)s:%(password)s'
                        '@%(host)s:%(port)d/%(name)s')
    engine = create_engine(connection_string % config)
    return engine

Requirements

Like when you deploy on Heroku, you have to provide a requirements.txt file, that you generate by running:

$ pip freeze > requirements.txt

WSGI App

You then have to create a wsgi.py file that you place at the root of your project, beside setup.py and appfog.ini. In wsgi.py, you put:

import os
from paste.deploy import loadapp

os.system("python setup.py develop")
path = os.getcwd()
application = loadapp('config:appfog.ini', relative_to=path)

That's It

Your app is ready to upload using the regular instructions, using the af commmand line tool provided by AppFog.

Useful Stuff

Since my app was already in production for about a year, I had data in my PostgreSQL database that I had to migrate from my server to AppFog. One nice feature they provide is the ability to make a tunnel to your services. You can find the doc here. With the help of the tunnel, it was easy to use pg_dump and psql to migrate the database.

<option value="US">United States</option>
<option value="CA">Canada</option>

Paypal already gives this list.

Problem is that it's only in English and all in caps. What I needed is a French version and an English version, not in caps.

Geonames.org just happens to have what I needed. This Python script is what I coded to generate what I needed:

import urllib, json, unicodedata

def get_list(username='demo', lang='en'):
    """Fetches the json of all countries from geonames.org."""
    params = urllib.urlencode({'lang': lang, 'username': username})
    url = 'http://api.geonames.org/countryInfoJSON?%s' % params
    f = urllib.urlopen(url)
    response_text = f.read()
    return json.loads(response_text)["geonames"]

def country_list_generator(country_list, func):
    """Returns a generator of countries.
    They are sorted alphabetically,
    and the func passed as argument is applied to each of them."""
    ordered = sorted(country_list,
        key=lambda k: strip_accents(k['countryName']))
    return (func(country) for country in ordered)

def country_to_option(country):
    """Transforms a country dict to an html option tag,
    The country code is used for the value."""
    return ('<option value="%s">%s</option>\n' % \
        (country['countryCode'], country['countryName'])).encode('utf-8')

def country_to_csharp_dict_pair(country):
    """Transforms a country dict to a C# Dictionary pair."""
    return ('{"%s", "%s"},\n' % \
        (country['countryCode'], country['countryName'])).encode('utf-8')

def strip_accents(s):
    """Removes the accents from a unicode string.
    This function is used for sorting."""
    return ''.join((c for c in unicodedata.normalize('NFD', s) \
                        if unicodedata.category(c) != 'Mn'))

if __name__ == '__main__':
    country_list = get_list('demo', lang='en')
    with open('output.txt', 'w') as f:
        gen = country_list_generator(country_list, country_to_option)
        f.writelines(gen)

Hope it's useful to someone else.

The code is on Github.

It's really not that complex and we use it often for client sites.

First thing to do is create the JSON to use for the style. You can use this wizard, it's a great tool for that.

Then you use the JSON in your Javascript like so:

// your JSON
var mapStyle = [
  {
      featureType: "landscape",
      elementType: "all",
      stylers: [
      { hue: "#ff4d00" },
      { lightness: -72 },
      { saturation: -74 }
    ]
  }
];

// usual Google Maps code, plus styling
var map;

$(function () {
    var mapOptions = {
        zoom: 12,
        mapTypeId: 'MapStyle',
        mapTypeControlOptions: {
            mapTypeIds: ['MapStyle', google.maps.MapTypeId.SATELLITE]
        }
    };

    var styledMapOptions = { name: "Map" };

    map = new google.maps.Map(document.getElementById("styled-maps-example-1"), mapOptions);
    map.setCenter(new google.maps.LatLng(46.8,-71.25));
    var mapType = new google.maps.StyledMapType(mapStyle, styledMapOptions);
    map.mapTypes.set('MapStyle', mapType);
    map.setMapTypeId('MapStyle');
});

It's even in the docs. I don't see any reason why there's so little use of this feature.

Some basic explanation of a quadratic Bézier curve. You have a start point P1, an end point P2 and a control point C. Both the x and y of each point on the path is dependant on t, which will vary from 0 to 1. t=0 at the begining of the curve and t=1 and the end of the curve.

You draw an imaginary line between P1 and C (P1C line) and another one between C and P2 (CP2 line). For each value of t, you mark an imaginary point on P1C and CP2. The point on P1C is at t of the line, starting at P1, and the point on the CP2 line is at t of the line, starting at C. Let's call those points C1 and C2.

Then, you draw an imaginary line between C1 and C2, and you mark a real point at t of the C1C2 line.

You repeat the procedure for every value of t, where t varies from 0 to 1. All the points you marked on the line C1C2 are showing the quadratic Bézier curve.

Here's a little animation I made to explain it better:

t = 0

You can have a look at the source for a complete understanding, but here's the important part, edited for clarity:

// constants
var CANVAS_WIDTH = 301;
var CANVAS_HEIGHT = 301;
var p1x = 20;
var p1y = 200;
var cx = 140;
var cy = 20;
var p2x = 280;
var p2y = 280;

// basic setup
var $t = $('#bezier-example-1-t span');

var animationCanvas = $('#bezier-example-1 .animation').get(0);
var animationContext = animationCanvas.getContext('2d');
animationCanvas.width = CANVAS_WIDTH;
animationCanvas.height = CANVAS_HEIGHT;

var curveCanvas = $('#bezier-example-1 .curve').get(0);
var curveContext = curveCanvas.getContext('2d');
curveCanvas.width = CANVAS_WIDTH;
curveCanvas.height = CANVAS_HEIGHT;
curveContext.strokeStyle = "#777";
curveContext.lineWidth = 2;
curveContext.beginPath();
curveContext.moveTo(p1x, p1y);
curveContext.stroke();

setInterval(updateDemo, 1000/30);

var t = 0;
var d = 1; // direction

function updateDemo() { // called 30 times/second to animate
    if (t > 1 || t < 0) {
        d *= -1; // change direction
        curveContext.clearRect(0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
        curveContext.beginPath();
    }
    t += 0.01 * d; // continue moving
    $t.html(Math.round(t*100)/100);
    // update values
    var c1x = p1x + (cx - p1x) * t;
    var c1y = p1y + (cy - p1y) * t;
    var c2x = cx + (p2x - cx) * t;
    var c2y = cy + (p2y - cy) * t;
    var tx = c1x + (c2x - c1x) * t;
    var ty = c1y + (c2y - c1y) * t;
    
    animationContext.save();
    // clear old sketch
    animationContext.clearRect(0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
    // draw new line
    animationContext.beginPath();
    animationContext.strokeStyle = '#aaa';
    animationContext.lineWidth = 1;
    animationContext.moveTo(c1x, c1y);
    animationContext.lineTo(c2x, c2y);
    animationContext.stroke();
    // draw points on lines
    drawPoint(animationContext, c1x, c1y, 2, '#0f0');
    drawPoint(animationContext, c2x, c2y, 2, '#0f0');
    // draw point on curve
    drawPoint(animationContext, tx, ty, 3, '#f0f');
    animationContext.restore();
    
    // draw the new Bezier curve segment
    curveContext.lineTo(tx, ty);
    curveContext.stroke();
}

The code is on Github.