When I asked what task he would have me to do, Gitsvn bestowed upon me two invisible golden tablets, with ten sacred commandments written in a strange script that only I can read. He instructed me to spread these glorious words among his chosen people.

And so I present the holy commandments of source control, with my understanding of their meanings.

Commit bad code by proxy

Did you write some questionable code yourself? Then the best course of action is to “work” with one of your team mates. Email the code in question to them and ask them to take a look at it, them make them check it in “since you already have the project open”.

This will divert the attention (and ire) when it inevitably breaks.

Craft evasive and mystifying check in notes

Don’t directly say what you did in the check in notes, it makes it to easy for people to follow your trail. Instead of simply spelling out what changed and why, attempt to enter as many words as possible without actually saying anything. A good technique is to try injecting bizdev terms like “synergize” and “vision”.

If all else fails, just reference a task ID that does not exist.

Do not commit small change sets

When someone checks in small changes they are easy to identify and read, this lets anyone reviewing the history know exactly what changed, as well as give them the power to easily revert your changes. You should not give away such an advantage to parties unknown. It is better to make innumerable changes and/or combine the work of many tasks into one check in.

When in doubt, change things for no reason at all

It is usually advantageous to pad your check-ins with extra “work”. Especially when it is difficult to see exactly what changed, and completely impossibly to identify why. Get creative, it’s amazing how much you can change without changing anything. Swap around the conditionals, reorder switch statements, mix up some associative arrays.

This will serve to confuse and disorient your enemies.

Do not fix a an error until all know who wrote it (unless it was you)

So you’ve diagnosed the problem, and have located the exact block of code responsible, great! But you cannot proceed with the fix until you have sifted through the history and identified the individual who wrote that code. It is now your sacred duty to call said person out and make sure everyone on the team knows where the fault rests!

Change tabs to spaces, or spaces to tabs

When you have a file open, swap the whitespace around. One of two things will happen, either diffs will ignore the whitespace and see no change, so the next person to get the file will be taken completely by surprise when they start coding. Or, the diffs will see the white space and mark every line as changed, essentially marking your territory as the alpha of its change history.

Leave no rarely used project in a buildable state

Sure, you’ve got access to lots of legacy projects running around the system. Start working on them, checking in as you go, then abandon the project, leaving only your commits behind. If you did this right and obey all of the commandments then the project is essentially lost to the ages. A long forgotten un-reconstructable quagmire of history, broken structures, and sin, like ancient Greece, or New Orleans.

If you find a bug you are responsible for, ignore it

When you find a bug of your own making, for heaven’s sake, do not acknowledge it! Hell, don’t mention it, don’t have it open on your screen, do not even think about it. You may be tempted to check in a fix, but by doing so you will have admitted that you screwed up, and that is not something you should ever do.

Such things are best left forgotten and buried. With any luck nobody will know about it until long after you’ve moved on.

Rebase, reset, and filter often

They say that history is written by the victor, and that’s almost right. In reality history is rewritten by the victor, and that is exactly what you are. So take every opportunity to rewrite history.

It doesn’t even have to be big stuff, some of the greatest rewrites of history were minor. Squash a few unrelated commits together, rename a file from the beginning of time, rewrite some commit messages. Little things can make all of the difference in the world.

Commit often, preferably before any actual code has been completed

There are many people evangelizing the benefits of committing often, but you need to step it up a notch. Commit very often, say, every 10 minutes or so. Any wild tangent that your code is taking should be prompty checked in.

Not only does this impress the management with your amazing productivity and work ethic, but it also leaves half completed work in source control, guaranteeing your position for another day.

Conclusion

I have given you this divine knowledge. Now it is your noble task to spread the gospel of Gitsvn Hgtfs to the world. Always hold these truths in your heart and let no coder go ignorant of the sacred rules.

If you are setting up a web server then you are probably best off running Apache. There are other web servers that will serve your needs and maybe even serve them faster but Apache is the default, for good reason.

There are arguments to be made for the merits of nginx and lighthttpd, but I get everything I need out of Apache and enjoy using it. I really cannot say enough good things about Apache, I love it and it is my web server of choice. I recommend it as the starting point for all web servers.

About Apache

The internet runs on Apache, simple as that. There are other notable web servers that have been gaining popularity over the years, and they do offer some performance benefits, however Apache is the benchmark by which they are all judged.

In my opinion you should run Apache unless you have a specific reason to use one of the other servers.

Why use Apache

Apache is the stalwart web server, it has been the de facto web server software since I first started playing with the internet. But age alone does no great software make (though it sure can help). So here are the reasons to use Apache.

• Unfathomable age. In software terms, Apache is older than time. Calling this software “time-tested” is like saying that Egypt has “been around for a while”. The Apache server has been running the internet so long that it has seen, been the victim of and subsequently patched just about every web server issue ever known.
• Stable. As a system administrator this should be your number one concern. Would you rather serve your content 10ms faster or know that your server is extensively tested and nearly impossible to break (if it’s set up properly)?
• Default web server. Basically every mainstream web application that you can download (blogs, forums, CMSs, etc.) was built using Apache and comes specially setup to run on Apache (most notably .htaccess files).
• Exceptional (community) documentation. The official Apache documentation is very good, but is difficult for most people to process. However, the wealth of tutorials, walk-throughs, guides, FAQs and forums is unrivaled. If you have a question about Apache it has been asked at least 1000 times, and been answered at least 2000 times.
• Extensibility. The Apache mods system is great. You can enable or disable features very easily and they perform very well.
• Well maintained. Since most of the internet runs on Apache any issue that comes up is a very big deal. Problems are found and patched very quickly.
• Performance. Although you may have heard otherwise, Apache is fast. If properly tuned it is as fast as any other server on the market.

Why not to use Apache

There are times when Apache is not the best choice.

• Performance. Yes, Apache can be as fast as any other server on the market, but it takes a little more tweaking to get it there. The default configuration seems designed for heavy hitting dedicated machines, and will bring most VPS instances to their knees if they get significant traffic. If you screw up the configuration at all then your sites will go down the first time you get linked on reddit.
• Memory footprint. Apache tends to be more memory heavy than the newer servers. This is what usually causes issues on VPSs.

The other popular web servers

Of course this wouldn’t be an Apache web server article if I didn’t at least point out the other options.

• nginx – Probably the best alternative web server. It is very fast, memory efficient and configuration is very intuitive. Speed is the top priority here.
• lighthttpd – I don’t have much first-hand experience with this one, but I can tell you that it is very efficient and scales well. Configuration can get just as complicated, if not more-so than Apache.

Apache Multi-Processing Modules

Before I dive in to the install and configuration guide I do need to touch on one of the more advanced considerations, MPMs and their affects. This is the most advanced option with Apache, you do not need to know this to get a server up and serving hundreds of thousands of pageviews per day. You probably do need to know this to serve millions of pageviews a day.

You have a choice of several Multi-Processing Modules (MPMs) for your web server. These are the internals of how Apache works and have some very important considerations.

MPM Prefork

This is a non-threaded system that launches a separate Apache process for each connection. This is the setup that just always works, there are no special considerations for your web applications and no restrictions on modules. You will be running mod_php5 for this MPM and each process will have its own instance of PHP loaded. As a result MPMs based prefork takes a bit more memory than the threaded MPMs based on worker.

Configuration is based on limiting the total number of server processes spawned and their lifespan.

Prefork is the default apache2 package on Ubuntu.

MPM Worker

Worker is a system that uses separate threaded Apache processes. In this system you will be running FastCGI with mod_fastcgi or possibly mod_fcgid to execute PHP. This is significantly more memory efficient than the prefork system, but the overall performance difference is debatable. The speed at which the server works may or may not be improved, but FastCGI is, in general, faster for PHP scripts.

However, if you want APC (which you do) you should know that there are caching issues with APC and and mod_fcgid that we cannot work around right now, so the best option is to use the slightly older and slower mod_fastcgi with php-fpm. In addition to the threading and APC issue with fcgid, some older PHP code will not play well with a FastCGI environment (e.g. older mediawiki versions). Modern PHP code should not have any issues in this environment but it may be something you need to consider.

Moving forward I suspect that MPM worker with php-fpm will probably become the default setup because of its speed and memory efficiency.

Configuration is based on limiting the number of threads and their lifespan.

MPM ITK

This is an MPM designed for web hosts that want to be able to isolate virtual hosts with user accounts. It is based on prefork and runs the same way.

Don’t use this one unless you know you need to. I only mention it because there is the apache2-mpm-itk package available to you.

MPM Event

Do not use this one, Apache calls it experimental for a reason. Event is a modified version of worker that has special reserved processes to handle listening and such.

In the future this may be awesome-sauce, but it is not ready for production.

Installing Apache

The basic Apache install really could not be simpler on Ubuntu. Just execute the following command:

sudo aptitude install apache2

Alright, you now have Apache 2 installed. That’s it.

Configuring Apache

Now this part is a bit more complicated than the install. We need to configure some basic global variables and set your MPM limits to values that make sense for your server.

First off lets set the basics, open up your apache2.conf file which is located in /etc/apache2.

sudo nano /etc/apache2/apache2.conf

Now lets do some configuration. I’m not going to list every option you can configure, just the ones you need to change. You should take the time to read over all of the configuration directives in Apache.

Basic configuration

• ServerName – Used in server signatures and some server generated links. If you are only going to run one web site on this server then set this to that sites URL (e.g. www.exampe.com). If you are going to run more than one site then set this to the server’s IP address (e.g. 12.34.567.890).
• Timeout – How long someone can hang a connection open before requesting a page. I set this to 30 seconds, which I feel is generous. The default config value of 300 is outrageous.
• KeepAliveTimeout – How long to hold connections open for followup requests. My recommendation is to set this short, I would recommend 2 to 5 seconds.
• ServerTokens – What the server should tell the world about itself in the response header. The default setting is to send the full version information with PHP and mod versions. You might as well print out a list of every hack your system is vulnerable to in the headers. Set this to Prod.
• ServerSignature – What kind of signature to show on server generated pages. I personally set this to Off.
• TraceEnable – Whether or not to allow TRACE requests. The default is to allow them, but there is absolutely no reason to so you should set this to Off.

MPM Prefork configuration

The default values for MPM Prefork are fine, if you’re running a dedicated server with 4 gigabytes of memory. For us normal people we need to tune these way down or the server will light on fire (or start thrashing like crazy) when you get a significant amount of traffic.

• StartServers – The number of server processes to spawn when Apache starts up. The server will almost instantly change the number of server processes so it’s irrelevant, but for the sake of consistency set this to your MinSpareServers value.
• MinSpareServers – Minimum number of active server processes to keep up waiting to handle new requests. This has nothing to do with how many requests the server can accept, it’s about reducing the delay users would see because of the start-up time needed for new processes. This should be a relatively small number, I set it to 10.
• MaxSpareServers – Maximum number of waiting server processes. Apache will slowly kill these off when the server isn’t seeing much traffic. The point here is to not have a lot of pointless processes hanging around holding memory. Generally speaking this should be double your MinSpareServers.
• MaxClients – This is the important one, how many servers can we have up and actively talking to users. This does not mean how many connections the server will accept, it’s how many processes it can have running concurrently doing the work of processing requests, taking uploads, executing PHP, and serving data out to users. When the server reaches the MaxClients number further incoming requests will be queued and processed in order. The default in the config is 150, you want to turn this way down. For a server with about 1GB of RAM I would recommend setting this in the range of 15 to 50, depending on if you are running other services on the system (MySQL = set this low), how memory efficient your web apps are and how fast your server serves up requests.
• MaxRequestsPerChild – This is a safeguard to prevent server processes from leaking memory and never returning it. The default config value is zero, which means that child processes never die. You should set this to a large number, say 1000 to 5000.

Example config for a 1GB VPS with well tuned MySQL:

<IfModule mpm_prefork_module>
    StartServers            5
    MinSpareServers         5
    MaxSpareServers        10
    MaxClients             25
    MaxRequestsPerChild  1000
</IfModule>

Save your config and reload Apache

Once you’ve got all of your basic global settings where you want them then exit nano by hitting Ctrl+X and Y to save.

Now reload the Apache server:

sudo /etc/init.d/apache2 reload

Your web server is now running with your new settings.

Virtual hosts

Name based virtual hosts are how we do multiple web sites on one server. You have the DNS records for the sites pointing at your web servers IP address and virtual hosts for each domain set up on the web server. Setting up a virtual host in Apache is very simple indeed.

You do this by creating a virtual host config file, these will be located in /etc/apache2/sites-available.

Default virtual host

If you hit the IP of the server in your browser (no host name passed) you will get the default virtual host. This default web site is the one that will be shown when Apache cannot match the requested host to any named virtual host.

The first virtual host that does not specify the ServerName in its configuration will be the default virtual host. Apache comes with a default virtual host (called “default”) already set up and running in the sites-available folder, but if you set the ServerName on default and add another virtual host that starts with a number or a letter combination that would sort it before the word “default” then that will become the default virtual host.

If you need to add such a domain then I recommend changing the file name of “default” to something like “000-default”. This is how you do it:

cd /etc/apache2/sites-available
sudo a2dissite default
sudo mv default 000-default
sudo a2ensite 000-default

Adding a new virtual host

Alright, let’s create a new virtual host. Start by creating the directories for your new site. You can put the directories anywhere you want, but the convention for Ubuntu is that web sites belong in /var/www, and I recommend following that convention. Let’s create a folder for the domain that contains an htdocs folder for the web site and a logs folder for the logs.

sudo mkdir /var/www/yourdomain.com
sudo mkdir /var/www/yourdomain.com/htdocs
sudo mkdir /var/www/yourdomain.com/logs

Now, create a new file in the sites-available directory:

sudo nano /etc/apache2/sites-available/yourdomain.com

And paste in this virtual host config:

<VirtualHost *>

        ServerName yourdomain.name
        ServerAlias www.yourdomain.name
        ServerAdmin webmaster@yourdomain.com

        DocumentRoot /var/www/yourdomain.com/htdocs

        <Directory />
                Options FollowSymLinks
                AllowOverride All
        </Directory>

        <Directory /var/www/yourdomain.com/htdocs>
                Options FollowSymLinks
                AllowOverride All
                Order allow,deny
                allow from all
        </Directory>

        CustomLog /var/www/yourdomain.com/logs/access.log combined
        ErrorLog /var/www/yourdomain.com/logs/error.log
        LogLevel warn

</VirtualHost>

Exit nano (Ctrl+X to exit, and Y to save). Now you just enable the site:

sudo a2ensite yourdomain.com

And reload Apache:

sudo /etc/init.d/apache2 reload

That’s it. You can do all kinds of crazy configuration in the virtual host, but that template is enough to get your site up and running. If your DNS is set up right you should be able to hit your domain in a browser and see whatever is in /var/www/yourdomain.com.

Web optimizations

Now is the time to do some tuning for the client side of your web server. If you’ve ever used the YSlow or PageSpeed addons for Firefox (must have for web developers) you’ve probably seen them complaining about sites that don’t gzip, or that use ETags. Well let’s make sure your sites are not guilty of those sins.

These changes are made by adding these configuration values to the bottom of your config file (it can go anywhere in the file, I just prefer to keep these kinds of customizations all in one place)

To edit your apache2.conf again just enter this command in the console:

sudo nano /etc/apache2/apache2.conf

After you’ve made your changes you exit nano (Ctrl+X to exit, Y to save), and reload Apache with this command:

sudo /etc/init.d/apache2 reload

gzip static files

This is the single best server side optimization you can do, it costs you almost nothing in terms of performance and greatly reduces the over-the-wire footprint of files, and thus response time.

First, enable mod_deflate:

sudo a2enmod deflate

And add the following to your apache2.conf:

# mod_deflate - add gzip compression
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript

You are now serving static text, HTML, XML, CS and JS files gzipped.

ETags

File ETags are used to compare a cached file against the servers version of that file and determine if the client needs to get the latest full version of that file. However they are generally not recommended because they are usually unique to the machine that sent the file so you can’t use them if you want to use multiple machines to serve front-end content.

That probably won’t be a problem for you, but since I am a whore for YSlow scores, and YSlow recommends against them, I always disable ETags.

This is the code to disable ETags:

FileETag None

Far-future cache expiration dates

To get the most out of client-side caching it is recommended that you set the cache expiration into the distant future for all static files. This will ask clients to hold onto your static files for a long time so that they do not have to regularly re-download them, wasting time and bandwidth.

However, I do not recommend this for most people because you are forced to implement versioning on all of your resource files (e.g. stylesheet_v1.2.css). If you want to modify a CSS file you have to make a new copy of that file and give it a file name and/or path different from the original because anyone that has that file cached will not receive any of the changes.

If you are willing to deal with this hassle then this is a nice boost to the effectiveness of caching.

Enable mod_expires:

sudo a2enmod expires

And add the code to set up far-future cache expiration dates on static files:

# Add far-future expiration dates
ExpiresActive on

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
	ExpiresDefault "access plus 10 years"
</FilesMatch>

Further reading

I recommend you get both the YSlow and Page Speed addons and test your sites with them. These really are invaluable tools for improving your web site performance.

Also, I highly recommend taking the time to read through the official Yahoo performance best practices and Google performance best practices documents. They are filled with excellent tips for both the server and the web site.

Other useful mods

This article covers the basics of getting Apache up and running. From here on you can keep it simple or get as complicated as your heard desires. There are at least a few other mods that you should probably be aware of to help you on your journey:

• rewrite – The Apache URL rewriter, industry standard, much beloved and keeping the internet working. This will alow you to do URL rewriting, which is a huge article unto itself. Google to learn about Apache URL rewriting.
• auth_basic and auth_digest – These are the authentication modules that are used to require a username and password to access a file or directory on the server. Very useful, I personally like to protect admin folders (e.g. wp-admin for WordPress) with authentication, just as an extra layer of inconvenience to would-be hackers. Always use digest, don’t bother with basic.
• status – Useful for seeing what your web server is doing at any particular moment. This will give you the Apache status page that you might have seen before. Also, required for some monitoring tools (i.e. munin). Remember to require authentication on that page as well or anyone can see what sites/files you are sending and what IPs are accessing them.

Conclusion

Apache is an awesome web server for any and every purpose, and is very easy to get up and running. It takes some more tuning to get it right, but once you’ve got it all setup and running smoothly it is almost impossible to bring down (from normal usage).

If you have any questions or notice that I forgot to cover something important then please leave a comment below and I’ll get back to you as soon as I can.

Luckily for the CodeIgniter crowd, the framework comes with one of the fastest caching systems possible, Output Caching. Short of writing static HTML files or output caching to memory there is no faster way to serve pages.

However, if you have any degree of interactive or dynamic content, total output caching can be painful (if not impossible) to implement. Now there are other options, such as database caching and third party caching libraries, but none of them will be quite as fast as full output caching. So if at all possible that is what you should use.

Let me provide you with an overview of the caching systems available and a few of my tricks.

CodeIgniter Output Caching

CodeIgniter’s output caching system will take the completely rendered output of your views and save them to disk. It’s a very simple idea, and simple is good for performance. When a page is cached there is no need to talk to the database or process anything.

Output caching is actually built directly in to the CodeIgniter life cycle. Every time CodeIgniter starts up it checks to see if the current URL has a cached version on disk. If it finds it then CodeIgniter will completely skip everything and throw the cached version out to the browser.

It’s also incredibly easy to turn on, just call the cache() method anywhere in in your controller to output cache the page.

$this->output->cache(MINUTES);

Overcoming the problems of output caching

There is one major downside to output caching, when a page is cached you cannot talk to the database or process anything. The greatest benefit is also the greatest curse. This means that anything the least bit dynamic or user derived cannot be cached this way.

Unfortunately CodeIgniter does not come with partial caching, support for dependencies or even a way to evict items from cache. To me at least, this is the single biggest hole in the awesomeness that is CodeIgniter. Just a little bit more love for the output caching system would make the most scalable PHP framework vastly more scalable.

But we can work around these issues with a little bit of extra thought and design.

What output caching does parse

It’s worth mentioning that there are two functions/strings that output caching will parse, even after the file is cached, the benchmark class’ elapsed_time() and memory_usage() functions. These functions will insert text markers ({elapsed_time} and {memory_usage}, respectively) that are parsed by CodeIgniter when it sends the output to the browser.

This always makes for some interesting figures so it’s nice to include a HTML comment in your pages with these functions to watch just how fast the output cache is working.

<!--
	Time: <?php echo $this->benchmark->elapsed_time(); ?>
	Mem: <?php echo $this->benchmark->memory_usage(); ?>
-->

It’s quite common to see these values go from 0.07 seconds with 3.5MB of memory to 0.006 seconds with 0.75MB of memory. Output caching really does work that well!

The small things

For most pages on most web applications there are just a few small items that need to by dynamic, such as Login/Logout links, recently viewed items and pretty dates (x days ago). All of these things can be effectively done via JavaScript without losing any significant functionality.

So build a simple API to make the functions available via JSON. For example this is a basic controller that would allow a JavaScript to get the users current status (logged-in, username and group).

class Api extends CI_Controller
{
	public function user_status()
	{
		$logged_in = $this->user_model->logged_in();
		$data = array(
			'isLoggedIn' => $logged_in,
			'userId' => $logged_in ? $this->session->userdata('user_id') : '',
			'userName' => $logged_in ? $this->user_model->get_user()->username : '',
			'userGroup' => $logged_in ? $this->user_model->get_user()->group : ''
		);
		$this->output->set_header('Content-type: application/json');
		$this->output->set_output(json_encode($data));
	}
}

Now with that data exposed via an API you can use JavaScript to show the user controls you find at the top of many web applications. Here is an example script using jQuery that would do just that.

$.getJSON('/api/user_status', function(data) {
	if (data.isLoggedIn) {
		$('#user-controls').html('Logged in as <b>'+data.userName+'</b> | ' +
			'<a href="/logout">Log Out</a>');
	} else {
		$('#user-controls').html('<a href="/register">Register</a> | ' +
			'<a href="/login">Log In</a>');
	};
});

On an average web app this is still at least an order of magnitude faster than not using caching at all since you are only making one query to the database. The page load will finish much faster, then when the page is ready the supplementary content will be loaded.

Now obviously there is a significant down side to this solution; users with JavaScript disabled will not see the login and register links. You could simply show those links by default and overwrite them with script, but for most applications it is a non-issue since many features that require a user to log in will also require that JavaScript is enabled anyway. It’s a judgment call for you to make.

The big things

For entire pages that are filled with content that changes regularly you will need a different technique. The best solution that I have found is evicting the cache for a page when you change the data. So when someone posts something you will call a function to clear the cache files for any pages affected.

The preserves the massive performance benefit of keeping output caching enabled but still makes the page completely dynamic.

However, as I said earlier, CodeIgniter does not give you any function to delete cache files so you have to do it yourself. Output cache files are saved in your cache folder (config item) as an MD5 hash of the URI they represent. This is the exact code CodeIgniter uses:

$path = $CI->config->item('cache_path');
$cache_path = ($path == '') ? APPPATH.'cache/' : $path;
[ ........ ]
$uri =	$CI->config->item('base_url').
		$CI->config->item('index_page').
		$CI->uri->uri_string();

$cache_path .= md5($uri);

Note: Actually this is for the latest CodeIgniter 2.0. For 1.7 replace APPPATH with BASEPATH.

So you have to search for the cache files you want to purge using that algorithm and unlink them. Here is a simple little helper I wrote to do just that.

<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/**
 * Delete Cache File
 *
 * Evicts the output cache for the targeted page.
 *
 * @author	Steven Benner
 * @link	https://stevenbenner.com/2010/12/caching-with-codeigniter-zen-headaches-and-perfomance/
 * @param	string	$uri_string	Full uri_string() of the target page (e.g. '/blog/comments/123')
 * @return	bool	True if the cache file was removed, false if it was not
 */
if ( ! function_exists('delete_cache'))
{
	function delete_cache($uri_string)
	{
		$CI =& get_instance();
		$path = $CI->config->item('cache_path');
		$cache_path = ($path == '') ? APPPATH.'cache/' : $path;

		$uri =  $CI->config->item('base_url').
			$CI->config->item('index_page').
			$uri_string;

		$cache_path .= md5($uri);

		if (file_exists($cache_path))
		{
			return unlink($cache_path);
		}
		else
		{
			return TRUE;
		}
	}
}

You may want to just download my complete Cache Helper from GitHub. Place cache_helper.php in your helpers directory and you’re good to go. Now you can evict cache files like this:

$this->load->helper('cache');

delete_cache('/blog/comments/123');

CodeIgniter Database Caching

Another caching system that comes with CodeIgniter is the database caching system. This system, as it’s name implies, only caches database responses from your queries. Leaving your PHP code to do all of the dynamic stuff you want without needing to requery the database for every page load.

Implementing query caching

Database caching will greatly reduce the load on your database and increase the performance of your application as a result. It’s also easy to turn on. There are two settings in the database config file you need to modify. Just flip the cache_on switch and set a cachedir that is writable. With those options set you are up and running with database caching.

Once you have query caching up and running you will need to clear the caches whenever you do an update. Thanks to the convenient cache_delete function this too is very easy.

$this->db->cache_delete('controller', 'method');

Possible issues

Database caching has a few quirks that may or may not be a problem for you:

• Uses an odd organization system. Query cache files are organized into “controller+method” folders. This can be helpful, but if you have a single function in a controller that does some heavy lifting or runs different queries for different situations you may build up a very large pile of query cache files in short order.
• You have to specifically exclude queries from cache. You can’t tell the system to only cache queries that you specifically want to cache, you tell the system to cache all queries then you can exclude individual queries with the cache_off() function.
• Can get messy easily. You need to be very conscious about disabling cache for some queries to avoid a pile of useless cache files. It seems that if you’re using CodeIgniter’s DB sessions you will have a session query cached for every end user in every controller method folder, unless you extend it with custom code.
• Can’t purge cache for individual queries. You have to evict cache files for all queries in a controller function if you want to delete a cache. Again if you have a few controller methods doing all the heavy lifting this can cause a problem. But hey, at least query caching has a delete function.
• No expiration/TTL. Cache files can not be aged off. If you want a cache updated you must delete it.

The biggest problem is that the database caching simply isn’t sharp enough for many situations. The all-or-nothing structure can get annoying really quickly. If you’re going to use database sessions then you’re probably going to want to run a cron job to completely purge the cache every night.

Third Party Caching

Several other more advanced and more flexible caching systems have been created by other CodeIgniter developers around the world. For some situations you may find them to be far better alternatives than the caching systems that CodeIgniter offers.

• Phil Sturgeon’s Cache Library
  An excellent and very flexible caching system that can be used with just about any object you would want. This system uses disk caching just like the output and database caching systems. You can cache model returns and library results in a serialized format that will greatly reduce the amount of database queries you need to run.
• Jelmer Schreuder’s MP_Cache
  Another take on flexible caching, MP_Cache is a similar idea to Phil’s Cache Library but adds some more nice little features that you might enjoy, such as dependencies and group tags.

Things I wish CodeIgniter supported

One day I’ll probably stop being lazy and extend the caching system to add these features, but until that day these are the biggest things that I wish EllisLab would implement in CodeIgniter’s caching system:

• Partial output caching. This would be a huge, massive, zomgwtfwin improvement. Even if I could just use partial caching on views. Having main un-cached template view that calls individual cached content views would be immensely awesome. Cache just the content blocks with lots of heavy lifting beneath them and leave the lightweight user-specific stuff completely un-cached.
• Native output cache eviction. One little function, say clear_cache('controller/function') would be very helpful. It is really troublesome to have to find cache files by their full URL. Any little modifier, say page-numbers, greatly complicates a search for cache files to delete. Not to mention that it just plain doesn’t make sense that the application has to know all of it’s possible URIs to purge cache files.
• Output cache groups. Similar to the last idea, it would be nice to group related caches somehow, even just being able to specify a sub-directory to save them in would be enough. One example is paginated content. If I want to evict all caches for every page of a category I have to determine number of pages and the URIs for every page to delete all of the affected cache files. It would be nice to purge all of them as a group.
• Support for common caching back-ends. It’s something I don’t really need to see in the library, but it would be nice if there was support for caching objects with the more popular PHP caching schemes such as APC.

Conclusion

The caching systems in CodeIgniter have their pros and cons, and quirks, but the simplicity of the functions is nice in it’s own way, almost zen like in their simplicity. Of course I wish EllisLabs would improve and expand on this excellent base. Fortunately, thanks to the work of other developers we have some options for more complex caching needs.

So did I miss anything? Do you know of any other CI caching libraries that are worth mentioning here. Or do you know of any other techniques to squeeze some more flexibility out of the CI caching systems? I’d love to hear how you made caching work for your needs. Please add a comment below.

All Valve multiplayer games run off of the same base server software, Source Dedicated Server (SRCDS). They build a version for Windows, Linux and Mac so you can run it on whatever system you happen to have available. In my case I had Ubuntu 10.04 LTS, which is a great operating system for any server.

This article will give you a walk though guide for installing and configuring srcds on Ubuntu. I am writing specifically about TF2 here, but much of this information will apply to other Source games such as Counter Strike: Source, Left 4 Dead, and Half-Life 2: Deathmatch.

Basic server information

First off a few notes about the software:

• The first thing that I should point out is the the srcds software is only available in 32-bit. So if you are setting up a server just for Source games then be sure to use the 32-bit version of Ubuntu. You will not gain anything by having the 64-bit version.
• You may see references to “hlds” in places, hlds stands for Half-Life Dedicated server. This was the name of the old version of srcds (aka Gold Source) that ran the original Half-Life and Counter Strike. Don’t let it confuse you, the name is still used in places but (in this article) it is the same thing as srcds.
• Depending on what game you’re running and what settings are applied srcds can use a good amount of processor and RAM. You really need at least 512 MB of RAM and a good modern processor to run a TF2 server properly. A gig to two of RAM will be optimal for running a couple instances. To my knowledge the srcds software is not multi-threaded so you wont see any benefits from a dual processor machine.
• The server will require a huge amount of bandwidth. For a 24 player TF2 server with voice enabled running at max capacity 24/7 expect 800+ gigabytes of bandwidth per month. I regularly see the upload reach 3 Mb/s and download hovers at around 1.5 Mb/s.

Setting up Ubuntu

For a basic system set up walk through please use Part 1 of my guide to setting up an Ubuntu server. Everything in that article is relevant to this guide. The only thing that I should add is be sure not to run SSH on port 27015, since that is the port your Source server will be running on.

One extra step that you might want to do is to create an account for running the srcds process. This isn’t at all required, but it is good practice. The standard practice seems to be creating and account called “hlds” and using that account to run the process.

Also, if you’re running a good firewall (as you should be) remember to poke a hole for port 27015.

Installing Source Dedicated Server

The installation of srcds is handled entirely by the hldsupdatetool created by Valve. This simple little tool will handle the download and installation of your base Source Dedicated Server. The first step in this process is to get the hldsupdatetool.

wget http://www.steampowered.com/download/hldsupdatetool.bin

This will download hldsupdatetool.bin to your current directory. Next we need to give this file execute permissions and the run it.

chmod +x hldsupdatetool.bin
./hldsupdatetool.bin

This will run the tool. It will present you with a license agreement, type “yes” and hit enter to proceed. It will install steam into the current directory.

Now we have to run the steam client, which will check to see if it is currently up to date and download a whole bunch of updates. This will take some time.

./steam

Now that steam is up to date and ready to run we are ready to install the Team Fortress 2 server. We do this via the steam client by giving it the following command.

./steam -command update -game tf -dir .

The dot after the -dir flag tells it to install the game server based in the current directory. You can tell it to install wherever you want, but most people just install it in home directory.

If you thought the steam update took a while, you might want to find something else to do for a while, the initial install of the Team Fortress 2 server requires downloading about 3.5 gigabytes of files. This will take a lot of time.

Once it has finished downloading everything it will return you to the console. The game server has been completely installed.

Configuring your server

There are several files that you will want to edit for configuring and setting up your TF2 server. The big one is the server.cfg file located in the orangebox/tf/cfg folder. This file is where you set all of the server variables that control how your server runs.

There are literally hundreds of possible variables in srcds, many of which have no effect on a TF2 server. To simplify this guide I’ll provide you with a very simple sample server config file and talk about some of the more important variables. For a more complete list see the complete list.

Basic server configuration file

// General server settings
hostname "Payload/Control-Point Rotation [Fremont, CA]"
rcon_password "myunguessablepassword"
sv_password ""
sv_lan 0
sv_region 1

// game settings
mp_timelimit 30
mp_maxrounds 4
mp_winlimit 0

// voice settings
sv_voiceenable 1
sv_alltalk 0

// bandwidth rates/settings
sv_allowupload 1
sv_allowdownload 1
sv_minrate 10000
sv_maxrate 0
sv_maxupdaterate 100
sv_minupdaterate 66
sv_mincmdrate 66
sv_maxcmdrate 100

// run in pure mode
sv_pure 1
sv_pure_kick_clients 1

My server config is significantly more complicated than that, but trying to mention every possible variable that you can tinker with would be a huge article unto itself! This basic server config should be enough to get your server up and running with the best practices and and smoothest game play.

Any variable that you don’t explicitly set in the server config file will run it’s default value, which is exactly what you want for 99% of the variables.

General server settings

Most of these variables are pretty self-explanatory. The hostname is the server name that players will see in the server browser, rcon_password is the password for the remote console (make this a strong password), sv_password is the password to join the server (leave blank for a public server), and sv_lan 0 sets the server to run as an internet server (instead of a LAN server).

sv_region

The sv_region variable is quite important. This tells the master list server what region your server is running in so players can filter out servers not in their region. Here are the possible values.

Game settings

There are three variables in the game setting section, mp_timelimit, mp_maxrounds and mp_winlimit. These are easily confused and mean different things in different types of maps. They are also the ones that new admins are most likely to screw up.

mp_timelimit

The maximum time a game is allowed to be played (in minutes). This one is easy to understand, what is the longest time that you want players to be stuck on a single map? This doesn’t count in the tie-breaker or sudden death round (which defaults to 2 minutes long).

mp_maxrounds

The maximum number of “rounds” that one team can win before the server changes maps. A round is an entire game where the teams win or loose the game. For example, in a payload map a round is everything from the gates opening to the cart being pushed to the last point. In a CTF map a round is from the gates opening to the the point where the winning team has captured the intelligence three times.

mp_winlimit

This is the one that can really screw things up. How many “game points” can one team earn before the server changes the map. In CTF maps a point is 3 intel captures, in payload maps a point is earned every time the cart is pushed to a control point, or a single point is given to the defending team if the round ends without the cart reaching the end, and in control-point maps a point is earned when one team captures the final control point.

Setting this to zero (no limit) works best, especially if you have different maps in your rotation. If you set it to a real number, like 4, then you will find the server changing maps in the middle of a payload game.

Voice settings

These are again self-explanatory, sv_voiceenable is whether or not to support voice chat, and sv_alltalk defines if AllTalk is enabled. If AllTalk is set to 1 then everyone can hear everyone else, if it’s set to 0 then only team mates can hear each other over voice chat. Generally, players use voice chat for tactical information, so set this to 0 unless you have a good reason.

Running the server process (the right way)

Now that your Team Fortress 2 server is installed, configured and ready to go it’s time to start it up. You can do so by running the following command.

./srcds_run -game tf +map ctf_2fort

However, you probably want to be able to log out of your SSH session from time to time. So you will need to run srcds as a service. To do this we will use a shell script and the Screen utility. Screen is a command line multitasking tool that will allow you execute the process and detach it from your current session. Start by installing screen.

sudo aptitude install screen

Now we’re going to create a script called “srcds” in the /etc/init.d folder that will let you start, restart and stop the server with a very simple command.

sudo nano /etc/init.d/srcds

Now paste the following script in the text editor. I cannot take credit for this script, but it works well. I got this script from the Linux SRCDS server article on FreeNerd.net. It’s worth a read since it covers a couple things I wont bother with in this article.

# replace  with the user you created above
SRCDS_USER=""

# Do not change this path
PATH=/bin:/usr/bin:/sbin:/usr/sbin

# The path to the game you want to host. example = /home/newuser/dod
DIR=/home//orangebox
DAEMON=$DIR/srcds_run

# Change all PARAMS to your needs.
PARAMS="-game tf +map pl_badwater"
NAME=SRCDS
DESC="source dedicated server"

case "$1" in
	start)
		echo "Starting $DESC: $NAME"
		if [ -e $DIR ]; then
			cd $DIR
			su $SRCDS_USER -l -c "screen -d -m -S $NAME $DAEMON $PARAMS"
		else
			echo "No such directory: $DIR!"
		fi
		;;

	stop)
		if screen -ls |grep $NAME; then
			echo -n "Stopping $DESC: $NAME"
			kill `screen -ls |grep $NAME |awk -F . '{print $1}'|awk '{print $1}'`
			echo " ... done."
		else
			echo "Couldn't find a running $DESC"
		fi
		;;

	restart)
		if screen -ls |grep $NAME; then
			echo -n "Stopping $DESC: $NAME"
			kill `screen -ls |grep $NAME |awk -F . '{print $1}'|awk '{print $1}'`
			echo " ... done."
		else
			echo "Couldn't find a running $DESC"
		fi
		echo -n "Starting $DESC: $NAME"
		cd $DIR
		screen -d -m -S $NAME $DAEMON $PARAMS
		echo " ... done."
		;;

	status)
		# Check whether there's a "srcds" process
		ps aux | grep -v grep | grep srcds_r > /dev/null
		CHECK=$?
		[ $CHECK -eq 0 ] && echo "SRCDS is UP" || echo "SRCDS is DOWN"
		;;

	*)
		echo "Usage: $0 {start|stop|status|restart}"
		exit 1
		;;
esac

exit 0

UPDATE: I have created a repository for this script on GitHub. Please check there for the latest version. I would also appriciate it if you would submit issues and pull requests to help improve it!

The highlighted lines are the important ones that you will need to modify. As the instructions say, replace <newuser> with the username that you will be running the service as. This will probably be your username unless you have taken the time to setup a hlds user (a good idea).

The SRCDS_USER variable is the appropriate username, the DIR variable needs to be the path to the orangebox folder and the PARAMS variable are the parameters that will be used when running the server.

Once you have added this script save it by pressing CTRL+X (quit nano) and then Y to save changes.

You can now control the server via the following commands:

/etc/init.d/srcds start
/etc/init.d/srcds restart
/etc/init.d/srcds stop

Depending on your setup you may need to sudo these.

Installing mods

For the most part, I do not recommend installing any game-play mods. Aside from the massive headaches and maintenance issues with outdated mods breaking servers (especially after game updates), Valve put a lot of thought in to TF2 game pacing and balance, and judging by their wild success they seem to have gotten it right. So, if it ain’t broke don’t fix it.

However, the rcon admin system is just plain bad. As a TF2 server admin you will at the very least want to run SourceMod. This srcds addon is a great administration tool and framework for other mods, you will find it to be an invaluable tool for managing your TF2 server.

SourceMod itself is a mod for the srcds mod framework Metamod:Source. So we will have to install that before anything else. This is pretty simple, just download the package and extract the contents to your game directory (orangebox/tf). This will add metamod to the addons folder. Metamod does not really require any configuration, once you have the files in the correct location it’s ready to go.

For a slightly more detailed walk-through check out the official guide: Installing Metamod:Source.

Now you just need to install SourceMod on top of Metamod:Source. This is the same basic process, download the archive and extract the files you your orangebox/tf game directory. You should read the official guide for this as well: Installing SourceMod.

Once you have all of the files in their correct locations then you have successfully installed SourceMod, the base admin tool and the framework for any other mod you could ever want. Restart the server and it should all be up and running.

Adding SourceMod admins

Before you start adding mods you should add yourself (and anyone else you want) as a SourceMod admin. This is quite easy. I personally use Steam IDs to authenticate people as administrators, this is reasonably secure so long as you are running the server with Steam enabled (which you should be).

Start by grabbing your Steam ID. You can easily get the Steam ID of anyone you want from the Steam ID Finder if you know their name.

Now add them to the admin file. Open up orangebox/tf/addons/sourcemod/configs/admins_simple.ini and add the following to the end of the file:

"STEAM_0:1:1234567"    "99:z"

Of course replace the Steam ID with your real Steam ID. This will grant the person with this Steam ID all admin permissions with an immunity value of 99. The next time you join the server you will be able to use the sm_admin command in the console which will bring up a simple little admin menu in-game. You will also be able to use any other the other Admin Commands.

It’s that simple, but it can get as complicated as you want. For more information on the admin format read the Adding Admins guide.

Adding mods/addons/plugins

The AlliedModders forums have basically every mod in existence. It would take another full article to explain the important ones and walk you though configuration, so I’ll just provide you with the basics.

Browse the SourceMod Approved Plugins forum for mod you want to install. The thread will have all of the information you need to configure the mod.

Once you’ve found a mod you want download the mod and install the files in the appropriate locations. Generally the smx files belong in the orangebox/tf/addons/sourcemod/plugins folder and the configuration files belong in the orangebox/tf/addons/sourcemod/configs folder.

Creating a MOTD

TF2 MOTD Screen

The Message of the Day (MOTD) is the intro chalkboard screen you see when you join a server. It is an HTML document that runs in WebKit, so you can do just about anything you like here. Personally, I find the chalkboard look to be the most professional looking MOTD.

The Message of the Day page is located in the orangebox/tf folder as motd.txt. Here you can create your HTML page that will be shown to players who join your server.

I’m not going to try to teach you HTML or CSS, so I’ll simply provide you with the markup for a simple Message of the Day sample page.

Example page markup

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Welcome to the Server</title>
    <style type="text/css">
        html, body {
            height: 100%;
            margin: 0;
            padding: 0;
            overflow: hidden;
            color: #ECECEC;
            font-family: Verdana;
        }
        img#bg {
            position: fixed;
            top: 0;
            left: 0;
            width: 100%;
            height: 100%;
        }
        #content {
            position: relative;
            z-index: 1;
            margin: 0 auto;
        }
    </style>
</head>
<body>
    <img src="http://stevenbenner.com/misc/blackboard.png" id="bg" />
    <div id="content">
        <h1>My TF2 Server</h1>
        <p>
            Welcome to my Team Fortress 2 server.
        </p>
    </div>
</body>
</html>

This is a very basic motd screen. I don’t remember where I got the blackboard graphic, but kudos to whoever did it because it works on just about every resolution. Replace the “Welcome to My Team Fortress 2 server” text with whatever you want.

For more advanced layout you’re going to have to learn some HTML and CSS, Google is your best friend here. Don’t get too crazy though, if your motd is too long then no one will read it.

Updating your server

You will have to occasionally update the server software, whenever Valve releases updates to Steam or the game itself. This is a very simple task, in fact you’ve already done it before. We will be running the same two commands that we used to install the server.

The first thing you need to do is shut down the srcds process.

/etc/init.d/srcds stop

Next you’ll want to execute the Steam software, which will automatically check for updates, and update itself if it needs to.

./steam

Now you’ll need to have srcds check for TF2 updates.

./steam -command update -game tf -dir .

That’s it! Your Steam software and TF2 server software have been fully updated. Now you can start the srcds process again and get back to gaming.

/etc/init.d/srcds start

You’re server is now up and running with the latest updates.

Attracting players

TF2 Server Browser

This is a tough topic to address, getting players to join your new server can be very difficult, and frankly random. I don’t have a recipe for success to offer you because there just isn’t one. But here are some of the top things that will make your server more desirable to other TF2 players.

• Have a good ping (<50ms). This is one of the most important things that players will be looking for, a server with low latency will attract many more players than a server with a marginal ping. Of course a server with a terrible ping will never attract anyone because the experience of TF2 is lost on a laggy server.
• Choose a good name. It should stand out some, but be readable and advertise what makes your server worth playing on. Are you running special mods? Advertise them, not only to tell players what to expect, but to get people who are looking for those mods to notice. If you’re running custom maps you might want to say that as well. But you’re going to have to keep it fairly short, long names get truncated in the server browser.
• Find a good map rotation. It’s hard to define this, different players are looking for different things. Even today 2Fort seems to attract players (even though I find it fairly boring), but this is more of a personal choice. Pick maps that are actually fun to play on.
• Set the region code properly. This has become far more important with a recent TF2 update. Setting your region code correctly will allow players to filter by the right region to find your server.
• Get friends to help jump-start your server. The catch-22 of populating a server is that players tend to only join already populated servers. After all, who want’s to join a server with no one playing? If you have a few people (4-10) that you can coerce into playing on your server at the same time (during peak hours for your region) then you will see random people join. If your server keeps its low ping and stays lag free then it will fill up in no time.
• Participate, and foster a sense of community. Last, and possibly the most importing item, play on your server. Talk to people, make friends, encourage players and make them feel welcome. Kick and/or ban people who mic-spam (too much) or who stack teams. Make the server fun! And show players that there are admins who will keep the game experience smooth and entertaining.

Finding admins

It is a simple fact, you cannot be on your server 24/7/365. You can make the server more self-sufficient with mods, but you will need other admins to help keep the peace and kick/ban offensive players. I can’t really help you find admins, but I can give some some bullet points to think about.

• Do not give anyone rcon access. Only the system administrator (you) should know the rcon password. Anyone with the rcon password can do terrible things to your server.
• Only give full admin permissions to someone you really trust, and only via the SourceMod admin feature. The SourceMod admin has everything necessary to be an admin.
• The type of people you accept as admins will define the attitude of your server. Make sure their personality is the kind that you want to reflect your server.
• Bad admins can literally ruin your server. They can ban everyone from the server, change the maps randomly and kill players on a whim.
• Real life friends are almost always better choices for admin roles than online friends. But there are plenty of good internet people out there.
• Look for players who frequent your server. The best admins will be the admins who spend a lot of time playing on your server.
• Do not give admin roles to anyone under 21 years old. Yeah, there are good kids out there, but generally this is an excellent rule.

Basically, be careful about giving out the admin role. Get to know someone (over voice chat) before you offer them an admin role. Good admins will be the single greatest addition to your server, bad admins will spell irrevocable doom.

Conclusion

It’s quite easy to setup a Team Fortress 2 server and keep it running. The hardest and most expensive part will be finding the right host or paying for the colocation at the right data center. If your server takes off and becomes successful you will be able to enjoy a gaming experience custom suited to your tastes. If you’re really lucky you can even offset the costs with donations.

If you’re planning on setting up your own TF2 server then hopefully this article has offered some helpful information and give you some food for thought. If you have any questions please leave a comment below and I’ll respond back asap.

Despite the many criticisms levied against the Windows operating system (many completely unfounded, many are very true) it is an excellent OS for casual users, home users, business users, gamers, software developers, graphics artists, etcetera, etcetera. Unfortunately most of the point is lost if you don’t know what software to install.

This is my personal list of must have software for your Microsoft Windows computer. When I set up a new computer every item on this list gets installed, and I can do everything that I want to do. This list is valid for Windows XP, Vista or 7.

Internet applications

	Mozilla Firefox (web browser) www.mozilla.com/firefox/ In my humble opinion Firefox is still the best web browser available, for numerous reasons. It is a hugely popular and well supported browser, backed by a huge community of users and developers. The Mozilla team works very hard indeed to keep this browser updated with the latest technology and security. Second choice: Google Chrome / Third choice: Opera
	Mozilla Thunderbird (email client) www.mozillamessaging.com/thunderbird/ Personally, I think the days of desktop email software are coming to an end, email web applications are getting better and better and don’t tie your mails to a single computer. However that system really only works well if you have to monitor one or two email addresses. Since I have several email accounts that I need to monitor I need an email reader that shows them all. Thunderbird does this quite well. It’s not as smooth and refined as I would like, but it is still the best for this purpose, even when compared to commercial software like Outlook. Second choice: Microsoft Outlook
	Trillian Astra (instant messeger) Trillian from Cerulean Studios was the original all-in-one instant messaging software. It is powerful and flexible, but it has some quirks. Luckily Cerulean has a record of being a great company and the community for Trillian is excellent. I find it to be the best instant messenger client available. Update: Trillian 5 requires a pay subscription or it spams ads in your conversations. I am dropping this from my list. Trillian 4 is still good but unmaintained. I’m looking for a better IM client now. Second choice: Pidgin
	FileZilla (FTP and SFTP client) filezilla-project.org I finally made the switch to FileZilla a couple years back when I needed a competent SFTP client and haven’t looked back. Nothing beats good old WS_FTP for pure simplicity and ease-of-use, but FileZilla has it beat by far once you need to use secure FTP. Second choice: WS_FTP
	PuTTY (SSH terminal client) www.chiark.greenend.org.uk/~sgtatham/putty/ The one and only shell client worth mentioning. It is hardly user-friendly, but if you need to use an SSH terminal there is no better choice. It is old, but very powerful and has a nice set of support utilities that enhance it even further.
	mIRC (IRC client) www.mirc.com Yes, people still use IRC, and if you are one of those people then you already have mIRC installed. Listed because it is the obligatory IRC client.
	µTorrent (torrent client) www.utorrent.com If you use torrents, then µTorrent is the software you should be using. It is by far the most advanced torrent software available and has every feature you could want. It also happens to be the lightest and fastest.
	Dropbox www.dropbox.com File synchronization and sharing over the internet, it’s a simple idea, but for some reason Dropbox is the only one that seems to have gotten it right. I love this simple little tool. The free version is limited to 2GB (3GB with viral marking game) but I have yet to even come close to that limit.

Multimedia software

	CCCP w/ Media Player Classic Home Cinema (video player) www.cccp-project.net No, this isn’t some bolshevik plot, it’s the end all be all of video playback, Media Player Classic (MPC) with the Combined Community Codec Pack (CCCP). This one sleek little package includes everything you need to play (almost) any video file. I haven’t found one file in the last 3 years that MPC w/ CCCP can’t play. It has excellent support for just about every codec you’ll find, great support for subtitles, powerful rendering options and playback tweaks that can fix videos that got screwed up somewhere. Second choice: VLC
	foobar2000 (audio player) www.foobar2000.org I have grown to love foobar, but I will say it is not for everyone. Foobar2000 is a slimmed down audio player with a minimalistic interface. The lack of feature clutter makes it more powerful to me. But it is missing things that most normal people will want, like library sorting. If you want a more feature rich audio player then I have to recommend MediaMonkey, which I also love. I use foobar2000 for my FLAC collection and MediaMonkey for my MP3 collection. Alternate choice: MediaMonkey
	Adobe Photoshop (image editor) www.adobe.com/products/photoshop/ The one and only. Adobe Photoshop is the only image editor you should be using if you can help it or consider yourself at all professional. There have been many attempts made by other companies to compete with this juggernaut, but they have all failed because Adobe just keeps making this product better and better. Second choice: Paint.NET

Software development tools

	Visual Studio 2010 www.microsoft.com/visualstudio If you intend to do any kind of Windows software development then Visual Studio is pretty much your only option. But just because it is hopelessly proprietary does not mean it’s bad, in fact Visual Studio is the single best IDE I have ever used for anything, and at this point I’ve used just about every popular IDE ever made. Microsoft Visual Studio is so fluid, so intuitive, and so advanced that it simply sets the bar that all other IDEs have been have been hopelessly trying to reach. Once you get the rhythm, VS development is a real pleasure that nothing else can match.
	phpDesigner www.mpsoftware.dk/phpdesigner.php I was so happy to find a PHP IDE that doesn’t suck. I had basically given up all hope of finding a decent (non-Java) PHP development tool. Then I discovered phpDesigner from MPSoftware and I suddenly found new faith in man kind. This is an excellent IDE that almost compares with Visual Studio. It is smooth, fast and pretty intuitive. It has a primitive version of intellisense which isn’t terrible (unlike Aptana/failbeans/etc), and has reasonably smart formatting. There is some room for improvement but it blows all other PHP IDEs out of the water. Reasonably priced too.
	XAMPP www.apachefriends.org/en/xampp.html WAMP stack in a box with a simple Windows interface. I use this as a simple PHP/MySQL development environment and it does a magnificent job at it. Simply install and go, there is little to no configuration necessary to get a running dev server. Now I wouldn’t recommend it as a production server because server administration is supposed to be complicated (make the admin think about the config), however for a Windows/Apache/MySQL/PHP development environment, nothing competes with it’s simplicity.

Text tools

Notepad++ (text editor) notepad-plus-plus.org The single greatest text editor ever made, in my humble opinion. Notepad++ is aptly named for having all of the features of notepad, plus more, plus more than that. It has integrated code highlighting for just about every language ever made, collapsible code blocks, smart formatting, encoding selection, macros and support for custom plugins. It has an amazing feature set, lots of customizations and is extremely fast and stable. The only features I’ve ever missed from Notepad++ is support for #region tags and the ability to open gigabyte files.

WinMerge (text diff and merge) winmerge.org WinMerge is the best text diff and merge software I’ve found yet. It’s difference mapping is amazing and very smart. I can tell when lines have moved in the document and even when pieces of text have moved within a line. The merge feature is very intuitive and easy to use. Alternate choice: WinDiff

System protection and maintenance software

	NOD32 Antivirus (anti-virus software) www.eset.com In my opinion NOD32 is the best anti-virus software available, period. It is fast, reliable and takes a fraction of the memory that other AV software require. The virus definitions are updated regularly and it’s web/email scanning is excellent. The combination of fast and extremely low miss rate make NOD32 a great choice for AV software.
	MyDefrag (defragmenter) www.mydefrag.com Windows XP had a great defragmenter included with it. It wasn’t particularly feature rich, but it worked great and didn’t require any additional software. Unfortunately Microsoft ditched that app in Vista and it will never return. The defrag tool included with Windows Vista and Windows 7 is frankly terrible. It doesn’t give you any idea of how far it has progressed or what it is doing. So, lets replace it with something that is actually good, MyDefrag. This utility gives you a lot more than just defragmenting file, it optimizes the data on the drive. Moving the the data to the outside of the disc where the rotational velocity is higher and transfer rates are fastest and sorting files on the disc. Suffice it to say that this is a very advanced defrag tool, and you should be using it. Be sure to disable the default defrag tool (a scheduled item in Vista/7) because it will un-optimize the disc.
	CCleaner (drive cleaner) www.piriform.com CCleaner (a.k.a. Crap Cleaner) is a hugely helpful tool for deleting all of the crap that you don’t need. It will clear out all of the regular crap that builds up in Windows as well as the crap that builds up in various applications that you may have installed. It has an excellent registry cleaner that will clear all of the crap registry entries that build up over time. And it give’s you tools for even more advanced maintenance.
	KeePass (secure password storage) keepass.info Secure passwords are too important to let your your brain create and store them, you will end up creating weak passwords and reusing passwords. That is where KeePass comes in to play. KeePass is a very secure password storage system and it has a very powerful password generator. I use it create and store all of my passwords. Just make sure to keep a backup of your database somewhere.
	TrueCrypt (encryption) www.truecrypt.org Encryption, the right way, TrueCrypt is an incredibly powerful and well thought out data encryption utility. Is can encrypt individual files, folders, virtual drives and even whole hard drives. It can use several different encryption algorithms and even layered combination of those encryption algorithms.

Miscellaneous

	Microsoft Office 2010 (word processing & spreadsheets) office.microsoft.com The standard business productivity suite, Microsoft Office is something everyone involved with any kind of business simply must have. Documents, spreadsheets, presentations and email. Some people say OpenOffice is just as good, but that simply isn’t true. While it’s close it lacks complete compatibility with the Microsoft proprietary formats. If you need to be able to share documents with people who use Microsoft Office then you need MS Office. After all, Microsoft Office apps are really good at what they do. Excel spreadsheets and Word documents are powerful, easy to create and can look really good if you know how to do some layout. They print very well and integrate with Windows.
	7Zip (archiving and compression) www.7-zip.org The free file archiving software. At this point all of the popular zip file tools are pretty much at the same level as far as features, file compatibility and interface, so just use the free one. WinRAR has an old clunky interface, but is still quite good. And of course WinZIP is still the stalwart in the industry, but aside from the nicer interface doesn’t have any features to set it apart from the rest. Alternate choice: WinRAR
	Rainmeter (resource monitor) rainmeter.net This software has almost nothing to do with rain, although it can display the weather. Rainmeter is an awesome system monitor and feed consuming piece of software. It has support for skins (of which there have been countless masses created) and is extremely customizable. I run a very minimalistic resource monitor setup on all of my systems. It gives me at-a-glance status of CPU, RAM, network traffic, drive utilization and system uptime.
	Input Director (software KVM) www.inputdirector.com For those of us who have multiple (Windows) computers at our desk Input Director is the single greatest app ever made. It allows you to move your mouse across to the next computer and completely interface with it, without the need for a second keyboard and mouse. This tool gives you amazing multitasking power. Second choice: Synergy
	Process Explorer technet.microsoft.com/en-us/sysinternals/bb896653.aspx You will probably never have a need for this tool, but the one time that you do need it Process Explorer is the only tool that will work. Process Explorer is part of the Windows Sysinternals suite from Microsoft and gives you detailed information about the various processes running on your computer.
	Core Temp www.alcpu.com/CoreTemp/ If you want to check the temperature of your CPU outside of the BIOS you will need a temperature probe or Core Temp. Well there are actually several software options, but I prefer Core Temp because it is the most compatible (even working on my old FX-51) and seems to have the most accurate guesses about Tj.Max (the rarely disclosed max temp processors report against). Update: Be sure to download the standalone version! The normal installer uses the InstallIQ software, which is so well known as being used for adware that NOD32 will flag the file.

Am I missing anything?

This is my personal software list, and it’s ever changing. But maybe I’m forgetting something important.

What’s on your must-have software list?

I have heard several companies bragging about having interview scripts like this, saying that this kind of testing is the most effective way to filter out unqualified applicants. However I think that the post by Slava perfectly demonstrates just how flawed this concept is. If there is one thing you should have learned in school, it’s that standardized tests are a terrible metric for gauging skill or intelligence.

But more to the point, I think that this is the wrong way to define a real programmer. Allow me to present you with my definition of a real programmer.

What is a real programmer?

This comes from one of my favorite quotes in all of history:

“Real knowledge is to know the extent of ones ignorance” – Confucius

Having an encyclopedic knowledge of a college curriculum is not required to be a great programmer. Real world software development isn’t about memorizing low-level programming techniques, or math formulas. These are great tools to have at your disposal, but are certainly not the end-all-be-all of development. It’s all about problem solving, in the most efficient and elegant way that circumstances allow.

As a programmer you will be supplied with problems every day, and it’s your job to figure out how to solve these problems. To do this you need to know what your language of choice is capable of, this comes with experience. You will also need to be able to look at the possible solutions and pick the best one for the situation at hand.

Now here’s the fact that breaks the machine, you do not know every possible solution for every problem, and you never will. I don’t care how experienced you are, how much education you have had, or how many millions of lines of code you have written, because there are so many possible ways to approach the practical problems you find in software development that it is literally impossible to know everything. So you need to have a mix of creativity and resources that let you learn how to solve new problems.

Real programmers are the ones who can learn fast, and learn by doing. These are the people who constantly strive to keep up with the technology they love. They have ample real world experience building, learning and growing their craft, but know that they still don’t know everything.

In short the defining characteristic of real programmers is that they never stop learning.

My experience with real programmers

I’ve worked as a peon coder, link-in-the-chain guy, I’ve worked as a manager who did the hiring and firing, I’ve worked as a one-man-shop serving milti-billion dollar corporations, and I’ve worked as a solo freelancer. This experience has taught me two things:

1. I am not the greatest programmer in the world, despite what my website says, and
2. That you can rarely tell a good programmer from a bad one by the resume, or the interview.

I’ve seen guys with terrible resumes turn out to be so far above my level that I still have them on my guru list, and I’ve seen guys with amazing resumes who couldn’t grasp even the most basic concepts.

How to find a real programmer

It is true that there is often a gap between what candidates say they can do and what they’re really capable of. That is why testing candidates is still a very good idea, but you need to test for higher level skills and abilities. Testing a candidate’s knowledge on obscure techniques and patterns means passing off great people for not remembering stuff that they simply do not need to be top notch coders.

In my experience, it is very easy to separate the wheat from the chaff: go through the normal interview process to see if this is a person you can work with (and let’s face it, that’s all that the interview is really for). Then give them a small project, something realistic that resembles what their real work would be like, and send them home. Not a knowledge test, but a real “build something that does this” task.

When you get the finished result of their labor you will know if you’ve got the real deal. Is it done the way you would have expected (or better)? Is it high quality, showing some love some thought? Did they get it back to you reasonably quickly? Did they do anything cool to show off? If so, congratulations! You’ve just found a great programmer!

code_by_google != bad_programmer

Yes, they could have just coded by Google. But you know what? That shows they can learn and adapt. I have plenty of respect for a person who doesn’t immediately know how to accomplish a complex task, but can quickly find out how and do it on their own without bothering the other programmers on your team.

Remember the key point of my real programmers definition, they never stop learning. Google is the greatest programming resource that has ever existed. You cannot look down on someone for using the most powerful learning tool computer science has ever known. I would go so far as to say knowing how to code by Google is the single most important skill in a programmers tool box, because if you want to grow as a programmer this is going to be the tool to use.

Someone who knows how to search for code examples and how to learn from the work of others will be more or less self-sufficient. They can learn and grow their skills on their own without needing someone else to do it for them. The ability to learn and grow your knowledge is the single most important skill for any developer. Without the ability to grow you will find yourself quickly deprecated.

I do expect people to know how to use the language and/or framework they were hired to work in, but I judge them primarily based on the work they submit. A guy who can figure out how to do things that he doesn’t know how to do, on his own, on the fly, is a real programmer.

The catch-22 of impossibly high standards

Of course it’s pretty absurd to require knowledge that 99% of programmers have absolutely no need for in real life. But perhaps you don’t entirely know why. If you have very strict hiring standards then you are only going to find people who match your specific model, and your code will suffer because of this.

Computer science grows at a fantastical rate, every day there is another small innovation that someone came up with. This small innovation will slowly propegate out to the rest of the programming community and become part of everyone’s toolbox. This happens because some creative programmer tried some alternative idea and found a new way to solve a problem.

If your team consists entirely of people with the same background, skills and knowledge then your creativity bucket will be quite small. This means you will not see as much innovation as a varied team with people asking questions that wouldn’t normally be asked, and people offering solutions that wouldn’t normally be offered.

Most alternative idea’s will, of course, be immediately shot down. But that one in a hundred that actually sticks will give your team an advantage. This alternative idea has given a level of innovation to your project that would not have existed if you didn’t have a radical element thinking in a unique way.

Get real

This was a rather long-winded rant, but let me sum it up in a few nice bullet points.

• People who are more interested in the buzz words and CS theory than actual experience and a history of practical application of skills are a perpetual thorn in the side of the programming industry.
• Don’t look down on other programmers because they don’t fit your model definition of a programmer, they just might be better than you.
• If 99 out of 100 candidates fail your interview then you are looking for something that doesn’t exist.
• Without a varied group of developers in your team you will suffer from a lack of creativity.
• Don’t test a CS curriculum, test the ability to create.
• If you ever finish learning, then your career as a programmer is over. Go study law.

In short

Look for someone who truly understand the concepts, who can offer creative and alternative ideas, and who shows the ability to grow as a programmer. Then you will finally find a real programmer.

However they all have their place in the pantheon of software development. Without a healthy mix of these different programming styles you’ll probably find your projects either take too long to complete, are not stable enough or are too perfect for humans to look upon.

The duct tape programmer

The code may not be pretty, but damnit, it works!

This guy is the foundation of your company. When something goes wrong he will fix it fast and in a way that won’t break again. Of course he doesn’t care about how it looks, ease of use, or any of those other trivial concerns, but he will make it happen, without a bunch of talk or time-wasting nonsense. The best way to use this person is to point at a problem and walk away.

The OCD perfectionist programmer

You want to do what to my code?

This guy doesn’t care about your deadlines or budgets, those are insignificant when compared to the art form that is programming. When you do finally receive the finished product you will have no option but submit to the stunning glory and radiant beauty of perfectly formatted, no, perfectly beautiful code, that is so efficient that anything you would want to do to it would do nothing but defame a masterpiece. He is the only one qualified to work on his code.

The anti-programming programmer

I’m a programmer, damnit. I don’t write code.

His world has one simple truth; writing code is bad. If you have to write something then you’re doing it wrong. Someone else has already done the work so just use their code. He will tell you how much faster this development practice is, even though he takes as long or longer than the other programmers. But when you get the project it will only be 20 lines of actual code and will be very easy to read. It may not be very fast, efficient, or forward-compatible, but it will be done with the least effort required.

The half-assed programmer

What do you want? It works doesn’t it?

The guy who couldn’t care less about quality, that’s someone elses job. He accomplishes the tasks that he’s asked to do, quickly. You may not like his work, the other programmers hate it, but management and the clients love it. As much pain as he will cause you in the future, he is single-handedly keeping your deadlines so you can’t scoff at it (no matter how much you want to).

The theoretical programmer

Well, that’s a possibility, but in practice this might be a better alternative.

This guy is more interested the options than what should be done. He will spend 80% of his time staring blankly at his computer thinking up ways to accomplish a task, 15% of his time complaining about unreasonable deadlines, 4% of his time refining the options, and 1% of his time writing code. When you receive the final work it will always be accompanied by the phrase “if I had more time I could have done this the right way”.

Where do you fit?

Personally, I’d have to classify myself as the perfectionist. So, which type of programmer are you? Or perhaps you know another programming archetype that is missing from my list? Post a comment below and I’ll add it to a new updated list.

Once you have your Ubuntu operating system installed there are still a few things you need to do to get it ready for prime time. A fresh install of Ubuntu will have a root account (you chose the password during installation) and the SSH server running. At the moment my recommendation for version is 10.04 LTS.

Without further ado, lets get this show on the road. Start by connecting to your system via SSH and logging in to the root account. (For Windows users I recommend PuTTY as an SSH client)

Create your account

The first thing to do is to create your account. Logging in as root is bad and you should never do it after you create a personal account. Create a new account with the adduser command.

adduser steve

Of course use whatever name you like for your account. It will ask you for various pieces of account information, all of which is optional except for the password. Be sure to use a strong password, this is the single most important piece of security for the entire system. If you choose a dictionary word for your password then it is almost inevitable that sooner or later, someone will find it and destroy your server.

Give your new account sudo access

The reason you never have to log in as root is because your account will have super-user access via the sudo command. Sudo will let you execute other commands at nearly the same level as the root account. Open the sudo configuration file with the visudo command.

visudo

This will open a text file, at the bottom of that file add the following line (replacing “steve” with your account name).

steve ALL=(ALL) ALL

Now save the file and exit the text editor. Chances are this file was opened in nano, so you exit by hitting CTRL+X on your keyboard. It will ask you if you want to save the changes, say yes by typing “y” and hitting the enter key.

Configure the SSH server

We need to add your new account to the SSH server so you can log in under that name. We will also do a couple minor tweaks to make the server much more secure. Nothing too advanced, just enough to get your SSH of the easy attack list. Open the SSH server configuration file in nano.

nano /etc/ssh/sshd_config

Here are the thing we need to do in this file:

• Change the port to something other than 22. If you leave SSH running on port 22 it will constantly be under attack by botnets and hackers that try to find SSH logins for every IP on the internet. This little trick will save you from millions of automated hack attempts.
• Disable root login. We do not want anyone (even you) to be able to SSH in as root. It is a huge security risk because root can do anything to the system.
• Disable X11Forwarding. X11Forwarding is another security risk for numerous reasons. You simply don’t need it so disable it.
• Add your account to the list of users that can SSH in. The SSH server will only allow the log in of accounts that are added to the configuration file.

First and foremost, lets make sure your new account can log in via SSH. Add the following line to the file (I usually just add this at the bottom).

AllowUsers steve

Again, replace “steve” with your account name.

Now for the security stuff. Find the following config statements and update them accordingly:

Port 12345
PermitRootLogin no
X11Forwarding no

Remember your port number! You will need it to SSH in to your server again.

Use any port number you like, I would recommend 4 or 5 digits because port sniffers start at 1 and scan up from there. So a large number will take much longer to find. This will thwart most lazy port sniffers. Do not use any of the standard service ports (e.g. 23, 24, 25, 80, etc). For a list of standard port numbers check out this Wikipedia list. Basically any port number greater than 1024 should be okay to use. For better security use something 10000 or above.

Now reload the SSH server by entering the following command:

/etc/init.d/ssh reload

Now you can logout and log back in using your new account.

logout

SSH back in to your server and log in with your new account. You should now be logged in as yourself in your home directory. For now on when you need to execute something that requires super-user permissions you will do so via the sudo command.

Update all of the installed software

The next step is to get all of the software on your system up to date. This is one of the best things about running Ubuntu, updates are incredibly easy using the aptitude tool. Frankly, this is even easier than Windows Update, and far more reliable.

Update the packages list with the following command:

sudo aptitude update

This grabs all of the latest package information from the Ubuntu servers for everything you have installed.

Note that when you use sudo it will ask for your password. This is just the password for your account login. It will save your sudo session for a few minutes, so if you need to execute several sudo commands it will only ask for your password the first time you run it.

Now you can upgrade everything that needs updates with the safe-upgrade command.

sudo aptitude safe-upgrade

It will list all of the packages (programs) that it wants to update and ask if you want to proceed. Enter “y” for yes and hit enter. The aptitude program will automatically download and upgrade everything that needs to be updated.

You should do this as often as possible, once or twice a week is more than enough if you’re anal about keeping everything updated with the latest fixes and security updates.

Switch to key-based SSH login

It’s always a good idea to setup SSH to use key pairs to handle logins. This is a huge boost to security because it basically becomes impossible for an attacker to brute-force their way in via SSH.

The idea here is that instead of using a password to login to your account on the server your SSH client will prove to the server that you are in possession of the private key. The way this works (in SSH2) is that the SSH client will cryptographically sign a challenge from the server with the private key, then the server will verify that message with the public key.  If you’re interested you can find a more detailed and technical explanation in RFC 4252.

Note that these instructions are targeted towards Windows users, because that’s what I use.

Generate the public/private key pair.

For Windows users you will use the PuTTYgen utility to generate you keys. Open up PuTTYgen and create a new SSH-2 RSA key pair. The key can be however big you want, but 1024 bits is fine. If you feel compelled to increase it then change it to be a 2048 bit or 4096 bit key. Make sure you set a password on it, and make sure that it’s a password that you will never forget. Then save the public and private keys to your system.

DO NOT LOSE THE PRIVATE KEY! If you lose your private key after switching to key-based logins then you will never be able to login to your server again, ever. There is no way to recover from this unless you have physical access to the machine or your host has a back-door in.

Make sure it is backed up in a safe place. Make sure it is backed up in multiple safe places.

Save the public key on the server

Now you need to give the server your public key. This public key will need to be stored in the ~/.ssh/authorized_keys file. This file will not exists in a freshly installed system so you will need to first create the file, then add your private key to that file (do not sudo these commands!).

mkdir ~/.ssh
nano ~/.ssh/authorized_keys

Now you are viewing the new authorized keys file. Here you will enter “ssh-rsa”, then paste the private key string, then add “user@hostname” to the end. It should look something like this:

ssh-rsa AAAAB3NzaC..........lots more...........6URYr15Xapm2+4sU8= steve@hostname

Now you need to make sure the directory and the file have the correct permissions. SSH is very picky about the user and permissions on the file and directory. The .ssh directory and the authorized_keys file must be owned by the user they reference, and the permissions have to be 700 for the folder and 600 for the file.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Setup PuTTY to use the private key to login

Now you need to tell PuTTY about your private key. This is actually really simple, go to the Connections->SSH->Auth options and select the private key file.

VERIFY THAT YOU CAN LOGIN WITH ONLY THE KEY! Now make sure that you can successfully login with the private key. You should have to enter a password for the private key, but not to log in.

Disable password-based logins.

Now that you have verified that you can login with your private key and everything is working it’s time to shut down the ability to login with a password.

sudo nano /etc/ssh/sshd_config

Now change the PaswordAuthentication and UsePAM settings to “no”.

PasswordAuthentication no
UsePAM no

And then restart the SSH server.

sudo service ssh restart

And that’s it. Once you log out you will never be able to login again without the private key in your possession (unless you re-enable password logins).

Install the base software

This is an optional step, but it is worth adding because a majority of people will need these packages anyway. There are a couple software packages that you will almost certainly need at some point, no matter what you do with your server. So let’s go ahead and install those now.

Lets start with wget, the tool you will use to download files from the web to your server. It’s just a simple file download tool that you will be using.

sudo aptitude install wget

The next item on the list is the build-essential package. It includes all of the software you need to build Debian/Ubuntu software packages on your machine. Most of the software you will be using will be downloaded pre-built and auto installed by the aptitude software, but if you need to build Apache modules or any other non-standard pieces of software you will require all of the suff in the build-essential package.

sudo aptitude install build-essential

This will download and automatically install everything in this package.

Changing the system name

The name of your sever (aka “hostname”) is not only useful for keeping track of what server you’re looking at in a console window, but it is actually used to identify your account in a couple places (e.g. the authorized_keys file). If you’re setting up a VPS or dedicated server then you probably didn’t install the operating system yourself. Which means it probably isn’t named what you want it to be named.

First off pick a name for your server. Some people use the names of greek gods, some use simpsons characters and some use the names of minerals. There are countless naming conventions so pick whatever you want.

Of course the simplest way is to name your server based on it’s role. For example: www1, sql1, memcache1, etc. If you do use generic names like this you probably should add a number to the end in case you add more servers (e.g. www1, www2, www3…).

Now there are two places we need to enter your name to get the system to use that name, /etc/hostname and /ets/hosts.

sudo nano /etc/hostname

Enter the name you have chosen for the computer. There should be nothing else in this file.

sudo nano /etc/hosts

You will see the first line maps the 127.0.0.1 loopback IP to localhost. Just after that line you will add the name of your server, using the same format.

127.0.0.1 localhost
127.0.0.1 myservername

Once these files are save you will have to restart the server for the setting to take effect. Now when you login you should see your new server name in the console prompt.

Useful commands you should know

Here are some of the most useful commands you should know for managing your system.

sudo !!

This is one of the best little tricks to know. When you try to run a command but get a permissions error because you needed to sudo it, then you can run “sudo !!” which will execute the last command under sudo. This will save you lots of typing.

free -m

The free command will display current memory usage information. This will be an invaluable tool for optimizing as well as diagnosing problems. The -m flag tells it to display the numbers in megabytes, instead of just bytes.

df -h and du -h

The df and du commands report disk usage information. The df (disk free) command reports how much disk space is used and available on each partition. The du (disk usage) command reports how much space all of the files and folders in the current directory are taking. Both commands should be run with the -h flag which tells it we want human-readable (xxxKB, MB, GB) numbers.

top

Use the top command to find out information about the processes that are currently running. This app shows a constantly refreshed list of the processes with memory utilization, CPU usage, time running and such. It’s a very useful tool for finding out which processes are taking the most CPU and memory. There are a lot of feature in this one so read the man page. Press “q” to exit the app.

man

Of course real men don’t read instructions, so I don’t know why I’m listing this here. Perhaps it’s all of the times other real men have told me to RTFM. Anyway, man is the manual tool. For example entering “man top” will display the manual for the top application.

Conclusion

That’s it. You now have an Ubuntu server running the latest software with the basic best practices for accounts and SSH security. You can go anywhere from here, set up a LAMP server, a mail server, a game server, whatever. Look forward to the next part of my Ubuntu server series where I’ll discuss installing and configuring the web server.

Allow me to broaden your optimization toolbox with the ultimate HTTP connection killer, the data URI scheme. Most web developers have never heard of data URIs but they can dramatically reduce the number of HTTP connections required to download your web site.

This article will explain what data URIs are, how to use them, and how to properly implement them.

What exactly are data URIs?

The data URI scheme (aka data: URLs) is a method to include data in-line in a URI.

I don’t suppose that makes much sense, let me elaborate; Simply put, data URIs allow you to include a file (or multiple files) inside of another file. The most obvious and practical use is to embed background images inside a CSS file. You do this by base64 encoding a file and embedding the contents, with some basic meta data, into a URI.

It is important to note that while the data URI scheme was created in 1998 it never reached the status of Standard Protocol. It is still on the list of “Proposed Standards”. However, all of the most popular modern browsers have implemented the data URI scheme.

Due to the IE problem (explained further down) data URIs can only be fully supported in CSS files.

Data URI scheme

The format is pretty straight forward, the official data URL scheme as defined in the RFC is:

data:[<MIME-type>][;charset=”<encoding>”][;base64],<data>

Okay, let me break that down for you and provide some real world examples.

Like any URI it starts with a protocol identifier, data. Just like http, ftp, mailto or gopher this tells the browser how to use the following information.

The first piece the browser needs to know is the MIME type of the data you are including (e/g image/png or text/html). Over normal HTTP transfers the server identifies the MIME type from the file extension and sends it back with the file in the response headers. Since we are not transferring the data through an HTTP connection there is no automatic MIME-type data, and we have to manually identify the type of data we are giving to the browser.

Next is the charset identifier, which is exactly what it sounds like. If you are passing any kind of text document (e.g. plain text, html, xml) then you should tell the browser what kind of character encoding your are using for that text (e.g. UTF-8 or US-ASCII).

And the last piece of possible meta data is the base64 keyword. This tells the browser that the following data is base64 encoded.

All of this meta data is optional depending on the data you are embedding. If you are passing an image you won’t include the charset information, likewise if you’re passing a text document then your probably wont be encoding it and will omit the base64 keyword.

Useful examples

Here are a few examples of data URIs at work.

CSS background

In the real world, CSS backgrounds are about the only place where data URIs can be used (see: IE problem below). In my opinion this is the single best use for this technique anyway. We can embed all of our icons and backgrounds inside of the very style sheet that defines the rules that use those images.

#logo {
	background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEkAAABJCAYAAABxcw[*about 2000 more characters*]AAAABJRU5ErkJggg==');
}

JavaScript

If you don’t have to worry about old versions of IE then you can use data URIs anywhere, even in JavaScript for any kind of URL destination. If you need to open a new window with a basic layout but don’t want to host an otherwise useless file then embed the layout in a data URI.

window.open('data:text/html;charset=utf-8,%3C%21DOCTYPE%20html%3E%0D%0A%3Cht' +
	'ml%20lang%3D%22en%22%3E%0D%0A%3Chead%3E%3Ctitle%3EEmbedded%20Window%3C%' +
	'2Ftitle%3E%3C%2Fhead%3E%0D%0A%3Cbody%3E%3Ch1%3EDATA%3C%2Fh1%3E%3C%2Fbod' +
	'y%3E%0A%3C%2Fhtml%3E%0A%0D%0A', '_blank', 'height=300,width=400');

Embedded images

If you don’t need to support IE 6/7 then you can embed an image inside of the image tag itself with a data URI. This probably isn’t very useful, but it is a good demonstration of the technique.

<a href="http://www.linkedin.com/in/stevenbenner" target="_blank">
	<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEkAAABJCAYAAABxcw[*about 2000 more characters*]AAAABJRU5ErkJggg==" />
</a>

Embedded links

Even Internet Explorer 8 won’t let you embed data in links because of security considerations, but it is another good demonstration of the technique.

<a href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEkAAABJCAYAAABxcw[*about 2000 more characters*]AAAABJRU5ErkJggg==">
	Link to Twitter logo
</a>

The Internet Explorer problem

As usual, Microsoft Internet Explorer screws up everything, data URLs are not supported in any version prior to Internet Explorer version 8. As much as I may want to add this to the hate list, I honestly cannot fault Microsoft for not supporting this until version 8. As I said earlier the data URI scheme is still on the list of Proposed Standards. So the fault rests entirely on the shoulders of the W3C for not looking at this RFC for the last 10 years.

In addition to the problem of old versions, Internet Explorer 8 has some additional restrictions.

• data must be smaller than 32KB
• data URIs cannot be used for JavaScript
• only object, img, input and link HTML tags can use data URIs

However, you can use data URIs in any CSS url statements, so there is still plenty of use for data URIs in the latest version of Internet Explorer.

Graceful degradation options for older versions of IE

There are two options for graceful degradation:

• Make regular images available and use regular url links as a fall-back.
• Or, use the MHTML technique.

Personally, I’m not a fan of the MHTML technique because it will greatly increases the size of your CSS file (doubling the number of embedded images). In my opinion, this is just too high of a price to pay. It is far simpler to make the images available on your server and just link to them with regular CSS url statements. At least it is for me, if for some reason you would rather take the file size hit instead of publishing the image files then there is a good MHTML how-to article here.

For the this article I will use CSS url fall-backs for graceful degradation.

IE 6-7 specific CSS degradation

There are two ways to implement your CSS fall-back statements to target IE 6 and 7:

• Use Internet Explorer revision targeting with conditional comments
• Or, use the asterisk CSS hack.

Both of these are good choices, with simple pros and cons. IE revision targeting with conditional comments will require an additional CSS file specifically for older versions of IE, but in return it allows you to craft a style sheet that validates. Whereas the asterisk hack saves you the trouble of having an extra file but creates an invalid CSS statement.

Personally, I am anal about writing valid CSS and often have a separate CSS file for IE anyway (for IE PNG fix behavior statement) so I use IE revision targeting statements.

I’ll leave the choice up to you, so here are examples of both techniques:

Internet Explorer revision targeting

I’m sure you seen this technique countless times before, but here it is again. Internet Explorer supports version targeting statements in HTML comment blocks. These conditional comments are quite smooth because they produce valid markup and guarantee, for all time, that your code will only be parsed by the versions of Internet Explorer that you target.

The following snippet demonstrates how you include a CSS file that will only be seen and downloaded by Internet Explorer versions 7 and older:

<link rel="stylesheet" type="text/css" href="base.css" />
<!--[if lte IE 7]>
<link rel="stylesheet" type="text/css" href="old_ie.css" />
<![endif]-->

In this example base.css is seen by all browsers but old_ie.css is only seen by IE 7 and older. Place this code in your HTML head block and the statements you place in old_ie.css will override the rules from base.css in old versions of Internet Explorer. For example:

base.css

#logo {
	background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEkAAABJCAYAAABxcw[*about 2000 more characters*]AAAABJRU5ErkJggg==');
}

old_ie.css

#logo {
	background-image: url(/images/logo.gif);
}

Now when a modern browser visits the page they will properly download and use the embedded background image but when an old version of IE visits the site it will use the rule from old_ie.css.

The asterisk hack

A less well known technique for targeting old versions of IE is the so called Asterisk Hack. This is a nifty little trick that someone, somewhere figured out. If you prepend an asterisk to a CSS rule then that rule will only be processed by Internet Explorer versions 6 and 7.

Here is an example of this hack:

#wrapper {

	/* Processed by all browsers */
	background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEkAAABJCAYAAABxcw[*about 2000 more characters*]AAAABJRU5ErkJggg==');

	/* Only seen by IE 6 & 7 */
	*background-image: url('/images/graphic.png');

}

Simple! Unfortunately this is invalid CSS, which irks me, but it is a clean and efficient hack. Only IE 6 and 7 will see the second background statement, and since it comes after the default data URI statement, it will override it with a link to the image.

How to get encoded files for data URIs

I built a little tool that allows me to quickly encode files on my computer. Here is the download if you would like to use it. This isn’t my finest work, it is just a basic tool to do the job in 20 lines of code, don’t expect a lot of bells and whistles.

Steve’s Encode64 Utility

Click here to download

• This tool is for almost any version of Microsoft Windows (sorry Mac users).
• You can get the source code here: Steve’s Encode64 Utility Source Code (GitHub project)

How to generate base64 strings in code

If you need to dynamically generate base64 strings then you’re in luck, this task is quite simple. Here are a couple code examples to get you on your way:

PHP

$encoded_string = base64_encode(file_get_contents('logo.png'));

C#

private string Base64Encode(string fileToEncode)
{
	byte[] EncodeBuffer;

	using (FileStream fs = new FileStream(fileToEncode, FileMode.Open))
	{
		int Length = Convert.ToInt32(fs.Length);
		EncodeBuffer = new byte[Length];

		fs.Read(EncodeBuffer, 0, Length);
	}

	return Convert.ToBase64String(EncodeBuffer);
}

Conclution

I think data URIs are an awesome, and powerful optimization trick. You can save many extra HTTP connections, and the associated latency and overhead for all of your users who are running a modern browser. This will make your site much snappier, especially if you use a lot of icons.

Using these techniques you can have a fully compatible web site that will gracefully degrade to traditional external resources if you can accept the following restrictions:

• Only images
• Only in CSS files
• Less than 32KB each

I can see this working well for background images and for icons. With data URIs you can embed all of your presentation resources into the same CSS file that uses them. This file will grow much larger than normal but it will cache well and save in overall bandwidth.

For at least a decade the very idea of filtering information was something first world countries laughed at as a tactic only dictatorships and xenophobes would seriously consider, now it seems the stage is changing and governments and ISPs are becoming more open to the idea of national censorship.

This is a terrible, and dangerous trend. I for one hope the aussies block the forced internet filtering law and I hope the whole world sees ACTA as the scam that it is. There is a quote that I often like to cite when the topic of internet filtering comes up, “The internet sees censorship as damage, and automatically routes around it”. Unfortunately, when censorship happens at the ISP level, that automatic routing is broken, so you have to do some manual routing to circumvent the broken tubes.

I think it is important to know how to get around censorship schemes and I believe that the greater the number of people with this knowledge the better off the internet, and the world will be. So here’s how to circumvent internet censorship, the VPN.

In several countries it is illegal to even attempt to circumvent government authorized internet censorship. You can and will go to jail if you get caught bypassing official filters in such countries. As much as I want to promote internet free speech I do not want any of my readers to go to jail. Please read up on your countries laws regarding net filtering and, of course, I do not advise you to attempt anything illegal.

Disclaimer: This site is not intended for distribution to, or use by, any person or entity in any jurisdiction or country where such distribution or use would be contrary to local law or regulation.

VPN tunnel through a country that supports internet freedom

The idea is simple, do your internet surfing through a VPN tunnel that ends in a country that has a good record of supporting a free internet and protecting privacy (such as Sweden).

What is a VPN?

A VPN (or Virtual Private Network) is a way to securely extend a private network across the internet to another location. The client computer (in this case your computer) makes an encrypted connection to a server which acts as a normal network connection. This technique is commonly used at tech companies to allow employees to connect to their work network from anywhere in the world so they can access files on the corporate network or even open a remote desktop connection to their office computer.

This system is very secure. Since VPN users don’t want any nefarious types or compromised systems along the way to be able to sniff out what files you are reading or what information you are sending, hence “Private”. When a VPN connection is open it is just like having an Ethernet cable connected to the other machine, just slower since it is going over public networks.

Sounds great! So how do I get a VPN?

It’s not free, but it is cheap. There are actually several companies offering this service already, such as PRQ which offers an OpenVPN tunnel service for as little as $10 per month. There, your VPN server is all ready to go! I mention PRQ only because they are famous for protecting their customers, for more VPN providers check out the VPN list on the Internet censorship wiki.

You can also set up a VPN server of your very own anywhere you like, but you will need to find a host to house the system and keep it online. I highly recommend a VPS (Virtual Private Server) system for this because it is cheap, fast and you don’t need much computing power to run a VPN. Of course a dedicated server is fine too, but a lot more expensive. No matter what you will need a system that gives you root access so you can install and configure the VPN server.

Alright, I have my VPN server! Now, how do I use it?

Making a VPN connection is easy with OpenVPN, and even easier with a Windows Server VPN. For this article I’ll assume you are running OpenVPN because a Windows Server VPN would be prohibitively expensive for a home user who just wants to browse the internet in peace. I will also assume you are using the Microsoft Windows operating system (because that’s what I use). If you are a Mac or Linux user you’ll find similar guides with a quick Google search.

The first thing you need to do is to download a copy of the OpenVPN GUI for Windows. This is a free application and the Installation Package includes everything you need, even the base OpenVPN software.

If you are getting your VPN access from a company that provides OpenVPN services then you should receive all of your configuration information from them. If you’re setting up your own VPN server then you will need to read the OpenVPN documentation for your particular system.

The guide for setting up your VPN will be the topic of another article, I’ll write up a nice how-to with screen shots and such eventually. For now, if you’re looking fo a good walk-through on setting up the OpenVPN GUI check out How to configure OpenVPN.

The other option

I believe that a VPN system like what I have described is the best solution because it is fast and nearly impossible to eavesdrop on, however it isn’t the only solution.

The simplest option is the good old fashioned proxy server. A proxy server, as it’s name implies is a server that bounces internet traffic from your system, through the proxy server, to the destination. Setting up your computer to work with a proxy server is easy, every web browser has a configuration section for proxies. They are free and readily available, however they have some serious drawbacks. Proxies are notoriously slow, often have injected ads, offer little to no security and do not encrypt the connection.

There are more advanced proxy systems such as Tor and picidae that aim to remedy some of those problems. However in general proxies are designed to enhance privacy, not circumvent filtering.

In completely cut off countries there is the sneakernet phenomena. A sneakernet is literally a network of people who pass media around by hand, such a CDs and flash drives. This is the last line of defense for free speech because it doesn’t use any computer network to transmit data. However I understand that the latency is quite poor.

Why ACTA is bad for the internet and you

Since I’m American and the only thing really threatening American internet is the upcoming ACTA treaty. So I feel that I must add a note here about ACTA.

ACTA (The Anti-Counterfeiting Trade Agreement) is an international agreement being pushed through many nations. There are some parts of ACTA that are good, basically the anti-counterfeiting portions, however most of ACTA is about intellectual property. In fact ACTA, if it is accepted, will put forth some of the most sweeping international copyright policies ever made.

Primarily backed by media giant and organizations such as the infamous RIAA and MPAA, ACTA will force countries to enact legislation very similar to the American DMCA (Digital Media Communications Act) except with even more stringent rules and harsher penalties.

Quick ACTA facts:

• ACTA will require ISPs to permanently sever users internet connections if they get 3 copyright infringement complaints.
• Complaints need not have any evidence nor see a court of any kind, all it takes is a complaint.
• Several sections of the ACTA draft show that rightsholders can obtain an injunction just by showing that infringement is “imminent,” even if it hasn’t happened yet. (source: Ars Technica)
• ACTA will allow border agents to search and seize any equipment that they believe may contain copies of copyrighted material. (e.g. laptops, iPods, hard drives, etc.)
• ACTA would ban “the unauthorized circumvention of an effective technological measure.” No-CD cracks will become illegal.

Basically, ACTA is a very bad thing. It sacrifices a great deal of your privacy and threatens to get you permanently disconnected from the internet without any evidence or legal action.

Stop ACTA, support the EFF and countless human rights organizations.

Support free speech, privacy and net neutrality

Now that you have the power to get around censorship I want you to help prevent censorship. Take action, join free speech groups, donate to influential organizations, write your government representatives and vote for candidates who support free speech. It is important to know how to get information even when the powers that be try to stop you, but it is even more important to work with those powers to make the world a better place.

Support the watchdog organizations

Here are some influential organizations that are fighting for internet freedom:

• Electronic Frontier Foundation (EFF)
• Center for Democracy & Technology (CDT)
• American Civil Liberties Union (ACLU)
• Electronic Frontiers Australia (EFA)
• La Quadrature du Net

Learn more

These are a few informational sites on topics of freedom of speech and internet censorship:

• Internet censorship article on Wikipedia
• Index on Censorship (UK)
• Digital Due Process (US)
• European Digital Rights (EU)
• Reporters Without Borders
• OpenNet Initiative
• Google government requests tool