smart code

Null, NaN, Infinity in JSON, CouchDB

2011-10-14T13:27:00.001+01:00

If you're getting null values where you don't expect them in CouchDB docs, note that JSON doesn't allow NaN or Infinity: these are converted into null.

console.log(0/0 + " !== " + JSON.stringify(0/0));
NaN !== null

console.log(1/0 + " !== " + JSON.stringify(1/0));
Infinity !== null

Monitoring Apache httpd Error Logs

2011-01-15T10:39:00.001Z

Monitoring HTTP error logs can often help identify website issues you may not notice otherwise, from broken links to browser incompatibilities. They can also fill up with junk from script kiddie attacks.

I tend to have cron run a simple script every night after the logs have been rotated, and email me the results:

#! /bin/bash
date=`date -d yesterday +%F`
grep -i -v -f ignore_errors error_log.$date

Where the ignore_errors file is a list of things I really do not care about. For example:

ignore_errors
.anti-sec:)$
/3rdparty$
/RAHAT$
/README$
/SSLMySQLAdmin$
/Tools$
/WS_FTP.ini$
/ZopeAttributionButton$
/_admin$
/a$
/access_logs$
/adm$
/admin$
/adming$
/administrator$
/apple-touch-icon-precomposed.png$
/application$
/ascils$
/b2b$
/bbs$
/blog$
/build$
/cart$
/catalog$
/config$
/core$
/cpanelsql$
/cpdbadmin$
/crossdomain.xml$
/database$
/db$
/dbadmin$
/default.php$
/ecommerce$
/eshop$
/extlink$
/extlink,
/fastenv$
/fckeditor$
/genimg.php$
/hello$
/home.php$
/horde$
/horde-3.0.9$
/horde2$
/horde3$
/html$
/iphone$
/jc$
/labels.rdf$
/lamp$
/list$
/lists$
/loader$
/logs$
/m$
/mail$
/mobi$
/mobile$
/myadmin$
/mydatabase$
/mydb$
/myphp$
/mysql$
/mysql-admin$
/mysqladmin$
/mysqladminconfig$
/mysqlmanager$
/negozio$
/os$
/osc$
/oscommance$
/oscommerce$
/p$
/pda$
/php-myadmin$
/phpadm$
/phplist$
/phpma$
/phpmanager$
/phpmy$
/phpmy-admin$
/phpmya$
/phppgadmin$
/phppma$
/pma$
/pma2005$
/pma2006$
/pma2007$
/pma2008$
/pma2009$
/pma2010$
/pmaadmin$
/pmadmin$
/pmy$
/program$
/proxy$
/public$
/qql$
/rc$
/roundcube$
/scripts$
/shop$
/shopping$
/shops$
/sitemap.xml$
/sl2$
/sql$
/sqladmin$
/sqldatabase$
/sqlmanager$
/sqlweb$
/store$
/text$
/typo3$
/user$
/web$
/webadmin$
/webdav$
/webdb$
/webmail$
/websql$
/wp-content$
/wp-login.php$
/xampp$
/xmlrpc.php$
/~$
FCKeditor
Invalid URI in request GET HTTP/1.1
Invalid URI in request GET HTTP/1.1 HTTP/1.1
MSOffice
RFC2616 section 14.23
_vti_bin
actionkit
apple-touch-icon.png
client sent invalid HTTP/0.9 request: HEAD HTTP/1.0
noexist_82ef98c2405c8c5e.html
php-my-admin
phpMyAdmin
phpadmin
phpmyadmin
proxy_test.php' not found or unable to stat$
request failed: error reading the headers
tinymce
wp-login.php' not found or unable to stat$

Feel free to copy this list if you want to use it as described above. It can also be an indication of names not to call files and folders on your server.

DG834G Negative Connection Time

2010-08-04T16:46:00.000+01:00

Checking the status of a Netgear DG834G wireless router today, I saw it had a negative connection time...

Connection Time: -11230:00:-52

... which I'm sure is a very long time. But it can't be that long. The guys at Netgear must have either been seriously stingy with the storage allocated to this, or perhaps they never expected one of their routers to stay up for more than a few months!

Disconnect/Connect, and all's back to normal...

Firefox Profile for Web Work

2010-02-11T21:23:00.002Z

If you're like me you often find new and exciting ways to crash your browser when developing websites. I use mostly Firefox for development, given the wealth of useful plugins available (Firebug, YSlow, HTML Validator, Tamper Data...). With Firefox, crash one page, and every window in the browser pukes: bummer. Yes, there's restore functionality after the crash, but with repeated crashes that gets really boring.

However, If you run Firefox with a special "profile" for development, a crash will only take down windows in that profile. To bring up Firefox's profile manager:

$ firefox --no-remote -P

Create a profile, call it something like "web_dev". Then to launch Firefox with this profile, call:

firefox --no-remote -P web_dev

Actually I use a bash alias in ~/.bashrc so I just have to type "firefox_web_dev":

alias firefox_web_dev='/usr/bin/firefox --no-remote -P web_dev > /dev/null 2>&1 &'

Now when you crash Firefox in the web_dev profile, your regular instance of Firefox won't be affected.
But there's more.
Each profile has its own set of add-ons, search engines, etc. So you can set up the HTML Validator add-on in your web_dev profile, and Adblock Plus in your default Firefox profile: you won't slow down your regular browsing by validating the whole web's HTML, and you won't have to deal with Adblock hiding the ads you do want to see on the sites you're working on.
Win-Win?

Colour Your Shells

2009-11-16T20:10:00.005Z

You're doing some work on a database driven website...
You need to do some tests locally on real data from the production site's db...
No problem: just copy the latest db dump backup from the live server!

Copying...

Now just drop the db and reload the backup: dropdb my_precious_data; reload...

Wait, you're still logged in on the production server... which means: BAM! you just dropped the live database. Oops.

Perhaps that scenario seems far-fetched given the way you develop and deploy your site. But when working on a command line, you'll still find it helpful to know at a glance whether you're logged into your development machine or a production server. I do this by having black text on a white background for my local machine, and light text with a black background for remote machines:

I generally use gnome, but you should be able to find an equivalent on whatever system you use. Here's how I set things up in gnome:

Create one gnome-terminal profile per remote machine. Set the profile's background colour to black, the text colour to something very light; I use a different text colour for each remote machine as an additional visual cue. Set the profile's command to "ssh" to the remote machine, e.g.: ssh www.blackbox.com

Add a custom application launcher for each remote machine to the gnome panel. Give the launcher an icon you'll associate with the remote server, and a "command" to launch gnome-terminal with your new profile, e.g.: gnome-terminal --window-with-profile=BlackBox

That's it! Now, whenever you want to do something on the remote machine, just click the application launcher. When switching between consoles you'll immediately know which machine you're on just by glancing at the colour of the background and text.

Website Deployment

2009-10-07T15:29:00.005+01:00

Deploying updates to your website is nothing like a military deployment, except for the fact that a little planning and automation can help you not make a complete ass of it.

Say you have a one-server website, and a set of tested updates on your development machine you want to deploy to the production server.

When are you going to deploy the changes?
During your website's quiet time, when it has the least number of users. If you don't know when this is, find out.

Will you have a partially working site during deployment?
Hell no, that's worse than having no website. You could bring the website down during deployment and ho hum, but that has the tendency to cause users not to come back. Instead block off the relevant areas with an "undergoing upgrade" page, and stop people logging in whist you deploy.

Which files are you going to deploy?
Don't try to remember which set of files you've changed for the update, let your computer do the work for you. Here's a bash script that will do just that:

#! /bin/bash

# Configuration
# The local dirs and remote dirs we need to sync
LOC_DIR[0]=/work/websites/dev_site1/ # Local development dir
REM_DIR[0]=/websites/site1/          # Corresponding production dir
LOC_DIR[1]=/work/libs/common/
REM_DIR[1]=/libs/common/
# Add more items to LOC_DIR and REM_DIR as required.

REMOTE_HOST=yourhost.com
REMOTE_PORT=22
REMOTE_USER=yourusername
OPTS="-a --delete --exclude '*~'"
# Done configuration

deploy=0

for i in $(seq 0 $((${#LOC_DIR[@]} - 1))); do
  RES[$i]=`rsync -ni -e "ssh -p ${REMOTE_PORT}" $OPTS ${LOC_DIR[$i]} ${REMOTE_USER}@${REMOTE_HOST}:${REM_DIR[$i]}`
  if [ "${RES[$i]}" != "" ] ; then
    deploy=1
    echo "Files from local ${LOC_DIR[$i]} to ${REMOTE_USER}@${REMOTE_HOST}:${REM_DIR[$i]}"
    echo "${RES[$i]}" # Double quotes needed to preserve newlines
  fi
done

if [ $deploy = 0 ] ; then
  echo "Nothing to deploy"
  exit 1
fi

This uses rsync to display the differences between your local website files and those on the production server. Now you know what you need to deploy!

How are you going to deploy the changes?
By manually copying the files with your SFTP client? I think not, as that would be tedious and error-prone. Instead take the above bash script, and append the following:

echo -n "Above changes will be synced to production, proceed? [y/N]: "

read -n 1 proceed

echo

if [[ $proceed == [yY] ]]; then
  for i in $(seq 0 $((${#LOC_DIR[@]} - 1))); do
    if [ "${RES[$i]}" != "" ] ; then
      echo "Syncing from local ${LOC_DIR[$i]} to ${REMOTE_USER}@${REMOTE_HOST}:${REM_DIR[$i]}"
      rsync -v -e "ssh -p ${REMOTE_PORT}" $OPTS ${LOC_DIR[$i]} ${REMOTE_USER}@${REMOTE_HOST}:${REM_DIR[$i]}
    fi
  done
  echo "Done uploading"
else 
  echo "Operation Aborted"
fi

This will prompt you whether you want to upload the changes, and "rsync"hronise the lot for you: job done. Well, you might then want to do some testing...

Thanks to SmokedMidge and jai for contributions to this post.

Enterprise Pattern Syndrome

2009-09-24T23:22:00.015+01:00

“Let’s have a dinner party!” your wife informs you. “It will be wonderful and fun and I want to invite all of our friends! This time I’ll even do the cooking! All you have to do is sort out the shopping.”

“A piece of cake” you reply. “I can manage that”.

So your wife hands you a list of ingredients and supplies to get: a joint of beef, some fresh baby carrots, new coffee coasters, that kind of thing. Some of it is open to interpretation eg ‘snacks + sauce for afters... pos salsa?’ and some of it, eg the Shaoxing Wine, you’re not sure even sure where to get it.

“Whoa! Stop right there lady!” you exclaim.

“What’s the matter honey?” she replies.

“Well, you haven’t told me where to get all this stuff. And for some of it you haven’t said how much you want. Also we don’t have an approved suppliers list drawn up yet. We can’t start without this infrastructure in place.”

“Well just go down to the shops, and if you can’t find it then give me a call back. No problem”.

“Sorry”, you reply. “But that’s not the way to do it. I need to examine all the stores first, and then draw up an approved suppliers list of stores that I deem reputable. That way, I can minimise your chances of getting faulty goods, and also provide you with pre-defined returns policies. After this is in place, you will need to specify precisely what you want, assuming it is available for the stores. But don’t worry about which store, I can abstract that for you, providing we both agree on a store-to-item mapping configuration (in XML of course!) which I can draw up and we can both share via use of the NoticeBoard pattern in the front room”

Your wife looks at you completely baffled.

“That’s the way we do it at the office honey” you say. “It works there!”

What you have here is a chronic case of Enterprise Pattern Syndrome. This is the notion that a particular design or process that provides benefits to large scale systems should be applied like a brick to the face to solve smaller scenarios as well.

The pattern of an approved suppliers list is a familiar one for large companies. And yes, when you are large company, especially if you are reselling what you buy as part of a larger product, then for sure, you want to be careful whom you buy from. After all, if your diode supplier goes bust mid-way through your production cycle and you have designed your circuit boards specifically for their diodes, then, wow, that’s a problem. This happened on a project I worked on, diodes and all. We learned it the hard way.

But do you need an approved suppliers list for going to the shops? No.

Recently I was asked to do a software review of a mid-tier web application being built by one of the project teams in our large software house. This application would run on a single mid-range server and handle say, 500 users tops. The whole stack, database through to presentation layer was being developed by a small team of Java devs who had recently rolled off a much larger Enterprise banking system. All the system had to do was read data from the database, maybe change a few fields here and there, and then display it. Read, Update, Display. Here is what they had come up with:

Can you spot the problem here? The whole top two bespoke layers are doing absolutely nothing. Nada. Zilch. Or to use Enterprise speak, they are correctly implementing the SFA pattern.

Now, there is nothing inherently wrong with having four layers of abstraction. The VO layer, (Value Objects for those that don’t speak Java) is a valuable pattern. It is used in much larger applications when your ‘nouns’ of the system (Entity Beans), like Customers, or Hedge Funds are huge bloated objects perhaps with large memory footprints, or perhaps partially obscured (for example for security reasons). Either way, you don’t want to be creating copies of them, so you make a light weight access class called a VO which is mainly used for examining the properties of the class it refers to. You can then juggle these smaller fellas around like a circus clown. But here there is no need for such abstraction. There is 500 Users objects each with a password and a few fields. Are you really going to run out of memory when making a copy of these?

I asked the lead developer to walk me through the process of adding a new database table to the application, one that could be seen on the GUI. He started to build it from the bottom up, beginning with Hibernate and 12 classes later, he had finally reached Struts. 12 classes! And that’s before he even started programming any display logic. I asked him what benefit he was getting from having 12 classes here instead of, say 2. Or, lets go way into left field, maybe just 1. “It’s cleaner this way”. He said.

Quick pub quiz: What’s the best design pattern there is?

Yep, you heard me, the best one. No, not the best for abstracting business logic. No, not the best for providing safety critical functions. I mean the best pattern full stop.

Yes. In case you weren’t listening at the back, here it is again:

K.I.S.S.

“Things should be made as simple as possible, but not simpler”. And that is the truth of the matter.

Every time you use an Enterprise pattern (and by Enterprise pattern, I chiefly mean a pattern that adds complexity to the design), you should know why you have chosen to implement this pattern. What benefit to the design is this added complexity providing? You should be able to explain this, clearly and succinctly, to both your colleagues, and most importantly to yourself.

And if you can’t explain it, you probably don’t need to use it. It’s as simple as that.

Security Through Asceticism

2009-09-23T23:22:00.000+01:00

There's a recent story about several thousand websites revealing their source code due to version control directories being present along with the web content files. For example, if you use subversion to manage your web content, it creates .svn directories in each and every subdirectory of your project. If you do a fresh upload of your site and forget to remove these directories, or have had the not so bright idea of having your production site directly under version control so you can update it with "svn update", then you're revealing the source code to your site.

Maybe that's not a big deal. Everyone can see your HTML source anyway... But it could be a big deal if you're running a dynamic site: revealing your PHP code or your database schema can make it much easier for a malicious hacker to successfully attack your site. Either way, if you include world-readable version control directories in your web space as a matter of course, you're breaking one of cardinal security rules: "Have Nothing More Than You Need On Your Server".

If there are any files on your production server that aren't meant to be accessed: remove them. If there's software installed on your production server that you don't run and never will: uninstall it. Even if you can't think of any way the extra files or programs could compromise your server, chances are someone else can.

So, what should you do about .svn directories? You could have a look at the "svn export" command. You could use an rsync command to update your site, telling it to ignore .svn/*. Or you could use a version control system like git which only creates one .git directory at the root of your project, instead of spreading itself around all your subdirectories like a plague.

Compliant Code is Easier to Debug

2009-09-03T17:24:00.006+01:00

Debugging JavaScript can be much nicer if you're coding for a site that is W3C compliant. There are other reasons to always write valid HTML, like making your site accessible to the greatest number of people and software clients. The case has also been made for the business value of web standards. Note that the detecting and fixing invalid HTML is unlikely to change the appearance of your site in your browser. The reason there is so much dodgy HTML in the wild is because browsers are extremely permissive and try to compensate for most HTML errors. You'll find cases where a page "works fine" up until you try to make it a little easier to use with a touch of JavaScript magic, and get some pretty weird behaviour. Chances are, broken HTML has been used for years and nobody noticed... then it suddenly becomes a problem. Your problem.

I'll give a couple examples of how generally having valid HTML can help debug JavaScript, but first I need to introduce an excellent Firefox add-on: HTML Validator. This add-on displays a small icon to tell you whether your page is standards compliant, and can pinpoint exactly where in a page's source any nasty non-compliance is going on. The HTML Validator also supports site filters: you can set it up to only validate automatically the sites you work on.

So how does this help debug JavaScript? Here are a couple scenarios...

Duplicate IDs

The HTML specification requires that each tag's "id" be unique. HTML Validator will warn you if you've inadvertently used the same "id" twice; it's icon will change to a warning symbol, and clicking on it will show you an error something like this:

If you spot this, you'll not spend time banging your head on your desk trying to figure out the element returned by document.getElementById() isn't giving you the result you expected...

Broken DOM

All kinds of interesting JavaScript behaviour can occur if you've managed to seriously break the Document Object Model, e.g. with <div>s that are never closed, or code outside the <html> tags. This can be surprisingly tricky to spot if you're not checking the HTML for compliance, especially if the DOM-breaking code has been around for years: you might be working on a completely different part of the page in question. However, if you're paying attention to the HTML validator, you'll notice right away something is wrong with the page, and go straight for the fix, rather than trying to debug you own, perfectly correct, code!

Write valid HTML: some of your users will thank you, and you'll save yourself a great deal of hassle in the long run.

JavaScript Snippets for Slicker HTML Forms

2009-09-01T21:30:00.012+01:00

Some simple JavaScript can make your website forms seem slicker, and easier to interact with. Here are a few examples to build from...

You can use the onfocus and onblur attributes to guide users, for example in a search box where you have no room for a label:

<input type="text" value="Search terms..."
  onfocus="if (this.value == 'Search terms...') this.value = '';"
  onblur="if (this.value == '') this.value = 'Search terms...';"
>

This will give you an input like this, try clicking in and out of it:

The onclick attribute can prevent accidental duplicate form submission, say if a user tends to double click everything:

<input type="submit" onclick="this.disabled=true;">

This will give you a submit button that can only be clicked once, try it:

Using onchange to show a different input box depending on what a user selects from a drop down box.

<select onchange="
var val = this.options[this.selectedIndex].value,
     en = document.getElementById('en'),
     fr = document.getElementById('fr');
if (val == 'English') {
  en.style.display = 'block'; 
  fr.style.display = 'none';
} else if (val == 'French') {
  en.style.display = 'none';
  fr.style.display = 'block';
} else {
  en.style.display = 'none';
  fr.style.display = 'none';
} ">
<option>Select your language...
<option>English
<option>French
</select>
<input id="en" style="display: none;" type="text" value="Do you like cats?">
<input id="fr" style="display: none;" type="text" value="Aimez-vous les chats?">

Will give you:

These are just a few building blocks to make interacting with HTML forms a bit less painful. The less a user has to look at and interpret, and the less opportunities they have to screw things up, they happier they will be.

Develop Good Coding Habits

2009-08-26T17:19:00.011+01:00

It's worth developing good coding habits, as they will make your life easier in the long run. You'll end up writing code that easier to read and maintain, that runs slightly faster and is more secure.

Legibility

To make your code easy to read, one good habit is to pick a format for your code and stick to it. Use indentation, braces and variable names in the way that suits you best, but stick to the same format across your whole project.

Another good habit is to always perform equivalent operations in the same way. It might be obvious to you that all these do Perl statements do the same thing:

$huh or die "no!";
if (!$huh) { die "no!"; }
die "no!" unless $huh;

... but it might not be so obvious to your future newby apprentice: indeed if you use these interchangeably, he'll probably think the statements are different for a reason! You care about this, because you don't want to have to get out of bed one night to fix your apprentice's mistake, which was due to you showing off your knowledge of Perl syntax variations...

Also, if you think you've just done something really clever, stop. Write a comment about it. Half the time, you'll realise the code is actually not that clever by writing the comment. If it really is clever, then the comment will help the next person who comes around to look at the code (which could be you, in a year's time).

Speed
How do you choose a set of standards for a new project? You could pick whatever you're most familiar with. Or if you have no strong preference, you might as well pick whatever works fastest, if there is a difference to be found.

For example, in PHP, you could write either of:

print "Welcome " . $name;
echo 'Welcome ', $name;

If you select the second form as standard, you'll get a small speed boost each time you display a string, in three different ways:

echo is faster than print, as print returns a value but echo doesn't;
Single quotes are faster than double quotes, as PHP parses double-quoted strings to expand any variables therein;
Note the comma rather than the period in the second statement: rather than appending strings to display, the comma causes echo to just output one string, then the next. This is slightly faster.

Note, however, that the speed increase is going to be tiny. It would never make sense to re-write existing code with optimisations like these, but if you're starting from scratch this isn't a bad way to go.

Security
When coding, you should always have a process running somewhere in your brain called "How could I hack this?". If you can think of a way you could hack your application or site in a damaging way without any of your privy information, you have a problem. To reduce the CPU usage of this thought process, try to develop habits automatically rule out some kinds of attacks.

For example, one essential habit is to always sanitise your inputs and outputs no matter how benign they might appear. If the PHP statement above was to be used on a website, to protect against XSS attacks it should really have read:

echo 'Welcome ', htmlspecialchars($name);

Another obvious good habit that will keep your site secure is to always make sure you're using the latest stable version of any programs or libraries you are using. Strange that so many people run versions of software with known vulnerabilities...

It's Boring
You got this far, but found all the above kind of boring? Well, it kind of is. It's not really helping you make your code do things. But if you try to develop some good habits and set standards at the start of a project, your job will be so much easier updating and maintaining the project down the line.

Furthermore, if you're lucky, any other programmer that joins you on working on the project will notice the care and consistency that's gone into the code, and try to match it!

Be the First to Catch Your Mistakes

2009-08-24T21:37:00.005+01:00

To: web developer
From: sales
Dear developer, please see message below from our top customer.
--

To: sales
From: top customer
John, I've been trying to log in to your website all day. It's seriously broken. I'm fed up - and leaving!
Ciao, ex-Customer

As a web developer for a small business, that kind of email ranks pretty high on the list of emails you don't want to see. Of course you might think people should contact you earlier about this kind of issue. Of course you can fantasize all you like.

So, you rush to your development machine and test the login code you updated and deployed yesterday: it works fine. You login with a test account on the live server: that works fine. You start thinking the top customer's "Internet is broken".

Hmm...

You check the PHP error log (of course you log errors to file on the production server, and don't display them in the browser, right?).

# less /web/logs/php/php_error_log

Warning: pg_query(): Query failed: ERROR: syntax error at or near "cret1" LINE 1: ... WHERE login = 'customer1' AND password = md5('ze'cret1' || ...
Warning: pg_query(): Query failed: ERROR: syntax error at or near "cret1" LINE 1: ... WHERE login = 'customer1' AND password = md5('ze'cret1' || ...
Warning: pg_query(): Query failed: ERROR: syntax error at or near "cret1" LINE 1: ... WHERE login = 'customer1' AND password = md5('ze'cret1' || ...
...

Bingo! There's an apostrophe in the customer's password, and that's breaking the login SQL query.

Somehow the recent code update removed a call to pg_escape() for the password field. That's not a particularly easy error to catch. You curse yourself for not using pg_query_params() or a prepared statement in the first place, as both automatically escape variables.

However, even if you made that mistake, you still should have caught that error much earlier. In fact, you should have been alerted after the first failed login attempt. A simple bash script run with cron could have done the trick:

# cat /web/tools/rotate_php_log.sh:

#! /bin/bash

PHP_LOG=/web/logs/php/php_error_log

if [[ -f $PHP_LOG ]]; then
  cat $PHP_LOG;
  mv $PHP_LOG /web/logs/php/old/php_error_log_`date +%F_%H-%m-%S`
fi

# crontab -l:

MAILTO=youremail@....com
# Rotate and output any PHP logs
*/4                * * * * /web/tools/rotate_php_log.sh

Setting up this simple cron job would have alerted you of the login problem in under 5 minutes with an email from the server. If you think you'll get too many emails because your logs fill up with notices and warnings... go fix your code! You'll make it so much easier to debug and so much more portable...