I logged into the server to find out there was zero disk space left (I know, I need monitoring on it!).  After clearing down all of the old builds (most were for testing the configuration of the build job anyway) and a few obvious log files, I’d cleared up a huge 400MB.  Nothing on a 50GB disk.

After spotting a jenkins folder within /var/logs, I checked that out.  Two files over 20G in size for logs.  Less than ideal!  One of them was easy enough to remove, but the active log file would show as removed, but not free the disk space.  A quick restart of the jenkins service sorted that out.

Now to check the logging settings of jenkins to see what we can limit!

word-break: break-word

PhpStorm didn’t like it, as it didn’t fit with the CSS specification.  There were no real adverse effects, as it still did what it was supposed to, but the options for word-break are as follows:

• normal
• break-all
• keep-all

However, break-word is available for the overflow-wrap property, so I thought I would give it a change and see what happens.  My original test was to use a repeating paragraph of “The quick brown fox jumps over the lazy dog”, and see what the output was, shown below:

The property for the “title” class is shown as an invalid property value, so could be ignored by some browsers.  After the change to make the code:

overflow-wrap: break-word

The output became the following:

The visual output is the same, and this time the developer tools doesn’t show the property as being invalid.

I’ve submitted this as a pull request, and wait to see how it goes.

As part of builting my (other) website, I came to realise that there’s a lot of things which would need to be set up as repetitive tasks, and thought about these from the start.  Other jobs were things which needed to be coded, but as I was building the code in PHP, it was going to be inefficent to write the code in that.  This is where we have to pick the right tool for the job.  In my case, it came down to some simple Python scripts.

As the other website is (in time) going to have a lot of pages, I needed to set up something to build a sitemap, and update this regularly.  To do this I get a list of articles out of the database and use the imported ElementTree to build the XML before outputting.  Simple really, and here’s basically what it does to build the site map:

def add_sitemap_element(loc, mod, change, priority):
    url = et.Element('url')
    # build the sub-elements
    # add loc node
    loc_node = et.Element('loc')
    loc_node.text = loc
    # add last modified
    modified_node = et.Element('lastmod')
    modified_node.text = mod
    # priority
    priority_node = et.Element('priority')
    priority_node.text = priority
    # change frequency
    change_frequency_node = et.Element('changefreq')
    change_frequency_node.text = change
    # add nodes to URL level
    url.append(loc_node)
    url.append(modified_node)
    url.append(change_frequency_node)
    url.append(priority_node)
    sitemap_root.append(url)

An area where PHP isn’t particularly great is with image manipulation.  It can be done, and it’s fairly trivial once you get into it, but I’ve always found the amount of code needed for it was huge.  I decided that to create thumbnails I was going to use Python for that too.  Every few minutes I check to see what images don’t have thumbnails linked to them and create them.  I use the PIL library and import Image, but resizing an image is as simple as follows:

image = Image.open(input_image_name)  # full path
new_image = image.resize((int(new_width), int(new_height)))
new_image.save(output_image_name)  # full path

Okay, so there’s other processing in between the lines, like calculating the new sizes, but there’s no need to worry about what type of image it is.  If it’s a PNG or JPEG and you want to output it as a bitmap or GIF, then you can.  I also found the performance of the re-size in Python much better than in PHP.  The right tool for the job (even if there’s probably faster libraries out there).

There’s other jobs I have automated, and more which I need to automate, but when something needs automating do it.  Some things might be part of an application scope to do (such as create thumbnails) but you’re better off creating a background job to do it rather than keeping the user waiting for them to be created upon save.

A little thought and a little code can help in many ways!

I chose to use Let’s Encrypt for setting up the SSL certificate as I’ve always had trouble in the past when I’ve tried it (admittedly under Apache on Windows) but this made it as simple as possible.  Just get their ‘certbot’. run a command in the terminal and away we go.  I did have to set an additional IP address on the server, and then set this site to use that IP address to listen on, but once that was done, it was very straight forward.  The command I ran was:

./certbot-auto --apache -d garybell.co.uk -d www.garybell.co.uk

and then follow the prompts on screen.  It took care of the new vhost in Apache, setting up the certificate and restarting apache.  Done! Simple.

With all that in mind, it amazes me that there are websites out there which will take payments and they aren’t loading their page over HTTPS.  Sure they will load the payment section in a HTTPS iframe, but it’s often difficult to tell that the payment part is an iframe, and that it is secure.  After going through the process of setting up the SSL certificate on this and another domain I own, I certainly won’t buy from a company which serves their payment page in a HTTPS iframe, but fails to serve their website over HTTPS.

How much does it cost to get an SSL certificate?  It depends.  If you go via one of the major routes, then anything form around $30 right up into the hundreds.  It depends on what you want and where you shop.  With let’s encrypt it costs nothing.  Zero! I have yet to pay for the SSL certificates I have from them, and I don’t have to pay if I don’t want to or can’t afford it.  That said, I will be donating something to them for providing the service.  They esitmate it costs them just over $2m a year to run, and they have over 5m unexpired certificates.  If each one of those certificates contributed $1 each, then they would get enough to keep the service running for 2 years.

Given the cost of the certificates are free, and the barrier to getting it is essentially nil for anyone with some technical knowledge, there’s really no reason not to encrypt!

Charon is named after the ferryman who took souls across the river Styx in Greek mythology.  In this case, it takes files and other information across the internet whilst the rest of the web page continues as normal.

It’s still early days in development, but the majority of the workings are there.  It uses javascript alone to do what it needs to and doesn’t rely on jQuery.  I might look to build a jQuery version at some point, but I wanted to keep it basic for now.  It’s all available in GitHub for use and modification.

Documentation is available as part of the GitHub page, and there’s a minified version included which reduces the footprint required to use it on a production server.  It’s available under the MIT licence, so do with it what you will, but I’m not responsible for anything that goes wrong with it.

I’ll be dogfooding it soon enough as I overhaul the gallery section of this site, and modify my other projects to use it too.  If there’s issues, raise them on GitHub, and if I find any, I’ll work to fix them too.  If I’m going to use it for things, it’s going to be looked after going forward.

If you’re wondering about the use case, consider the following scenario.  You’ve got a website (like this one) and you have a gallery on it (like this one).  For web best practices, you should have an alt-text for your images.  It’s something which is better to store with the image at the time, but file uploaders tend to only look after files, not other data.  A traditional web form might be too large to post with several images on it, so you want to do the uploading at the time.  This is where charon comes in.  You can select an image, set the alt text and upload to a script of your choice, and send back any information of your choice as easy as the following code

var charon = new Charon({
    url: '/upload.php', // upload location
    file: document.getElementById('image1'), // file field
    additionalData: { altText: document.getElementById('altText1').value }
});
charon.sendFile();

If you then need to change the file and additional data before sending a new file, the following can do that :

charon.file = document.getElementById('image2');
charon.additionalData = { altText: document.getElementById('altText2').value };
// then send the next file
charon.sendFile();

With the custom callback functionality added in, you can get it to run your own functions as it’s uploading (for managing progress bars), for when it’s completed (for letting you know it’s uploaded or for processing any return information) or for any other XMLHttpRequest event.

Um… mysql doesn’t connect over http… nor to a directory

Neither should it be an url. Usually it’s localhost or the IP address of the remote mysql server

The first of these is, to my knowledge, correct.  However the second is absolutely wrong.  A connection to a database, or at least MySQL, does not have to be via IP address specifically.  Let’s break down the claim a little further:

Usually it’s localhost

This is often the case for development purposes as developers tend to run a small development database locally, allowing them to work whilst on the go and without an internet connection.

Some hosts will also tell you to use localhost and have a database server running on the platform you are deploying to.  Similarly, some websites run their databases on their webserver (I’ll go into why that’s not always a good idea another time).

I guess this part of the claim is true.

or the IP adress of the remote mysql server

I’ll start by saying this part is also true, as it’s perfectly possible to replace ‘localhost’ with, say, ‘192.168.0.7’ for things on a local network, or ‘81.81.81.81’ if that’s your MySQL server somewhere.

Neither should it be an url

This is where the remainder of the comment loses all credibility, and unfortunately it’s at the beginning of the comment.  It’s perfectly possible to establish a database connection to something like ‘database01.myhost.com’ or, a more real example, ‘mysql-master.garybell.co.uk’ (though that doesn’t work, so save your time).  That leads me on to why you might want to use a URL over an IP address:

Company X has a SaaS application which stores data in a MySQL database.  They have multiple servers and their database is on a large distributed cluster and/or has master-slave replication, so if one of their servers needs maintenance or dies, they can switch over easily.

Their connection configuration files are set up on each application as follows:

// database connection information
$database['host'] = '10.17.83.123'; // master server
$database['user'] = 'web_app_user';
$database['password'] = 'w3B4pPU53R!';
$database['schema'] = 'client1';

If their master server blew up, the data center suffered a catastrophic failure or any other reason why a connection into that IP was stopped, then they would need to correct the configuration files of all of their clients.  If that is only 2 or 3, then no problem.  When it runs into tens, hundreds or thousands it’s just not practical.

They decide the sensible thing form the get go is to use a similar config file:

/ database connection information
$database['host'] = 'mysql-master.companyx.com'; // master server on 10.17.83.123
$database['user'] = 'web_app_user';
$database['password'] = 'w3B4pPU53R!';
$database['schema'] = 'client1';

If the same disaster is thrust upon them they can open up their DNS manager, point the domain ‘mysql-master.companyx.com’ to one of their slave servers and, with a quick change of settings on the slave server have their hundreds of customer systems running with minimal interruption, all from using a URL.

A URL is a Uniform Resource Locator, and in this case the resource is MySQL, so it’s perfectly possible to use a URL as a database host name and, in some cases, actually preferential.

The reason, I found, was because a lot of the assets for the styles and functionality just weren’t being loaded.  Couldn’t be found.  Vanished.  I worried about the state of the server for a moment, but then realised if someone had compromised it, they wouldn’t just boot off a couple of assets; they would be making a lot more serious changes.  Nope, this one was user error on my part.

I recently moved servers.  Same provider, better spec box.  As part of that, there was obviously a migration of the setup (mostly) and it all seemed to work fine, until I moticed it today.  The issue was that I was sending requests for some of the loaded assets off to a faux-CDN; basically a different subdomain of my site to let them load alongside one another.  The trouble was, when I moved everything I didn’t set those up, so I was effectively seeing a cached version of those files.  When accessing the site from a different machine, there was no cached version to be had, and therefore the issue was all too clear to see.

This brings me to the title of the article, and how this could have been avoided.  I should have gotten everything set up, checked it was working and then checked from somewhere different; or even simply clearing the browser cache to check everything was as it should be.  Testing is something which does take a lot of time to do right, and if it’s not done right is fairly obvious.  With something as important as my own personal website, it should have been far higher on my list of things to do; but it wasn’t, I messed up and I look bad for it being that way.

It’s all fixed now, briefly, but I’ll have the faux-CDN to reconfigure to improve load speeds.

• Fix root url sometimes returning empty in canonical URLs
• Fix having to log in twice when user session expires
• Some SMTP emails could no longer be sent due to a bug in the SMTP email server setup
• Session restarts caused PHP Errors
• Fix insertid() returning 0 for the PDO MySQL driver
• Fix the Empty trash and unarchive button not existing for com_banners

The new version comes with an e-mail update plugin to allow site admins to know when there are patches and bug fixes; hopefully they will then patch their sites to the benefit of everyone.

A run down of the listed major changes are:

• Download system & environment information: gives users the option to download system/environment information for support purposes.
• Ability to add a user CSS file to Protostar: currently it is impossible to use a custom.css file when using the Protostar template. This update adds a check to see if the file user.css exists and loads the file to allow for user customisation.
• Added site and admin links to module user: adds switches to the mod_status module to show/hide links to the front-end and back-end of the site.
• Article counts: a set of updates that allows the visual presentation of published, unpublished and trashed articles in the Category Manager for articles, banners, contacts and newsfeeds.
• Random category blog and list order: this update answers a common user request to add a random ordering option for articles in the blog category and category list view.
• Editor Buttons added to the Toolbar: You no longer need to scroll down the page to find the “readmore” or “insert image” buttons. Editor extension buttons that used to be placed below the text area can now be found in the toolbar where they belong.
• Easily Insert Modules in Articles: allows users to easily add a module into an article with a button directly on the editor toolbar. Users don’t need to learn any syntax or remember the module details as the user interface does it for them.
• Drag and Drop Images: Adding an image is now as easy as dragging and dropping it from your computer directly into the content. This works anywhere that you are using the default TinyMCE WYSIWYG editor.

It started like any audit I’ve been involved in, get it set up and then log in.  Usually that means asking for a password after checking the database to be sure they aren’t stored in plain text.  Nope, these were plain text.  Ok, so we can get in, and chalk 1 up in the bad developer column.  After a small poke around to see what it was doing, it became apparent that every page was its own contained file.  There were some includes for navigation, but most of it all stylesheets, javascript includes and other common resources were written in each and every file.  I wouldn’t want to be the one to have to update the version of jQuery on that one!

When we discovered this, we found that the queries being ran on each page were not sanitising any of the user input.  They were putting it blindly into queries and using old mysql_* functions; long since deprecated.  The bad developer count just hit 2, 3 and 4 from one simple check.  There were some instances where the developer was checking for one result being returned and failing if there was more, but that only gets a fraction of a percent off, when I could simply do a ‘limit 1’ via the GET variable.  Still, this was in the application where there users probably aren’t technical enough to try, they wouldn’t have that sort of code on a login page, right?

I can’t even just add 1 to the bad developer for this.  It’s a minimum of +10 for allowing such simple “hacks” to let someone into the system.  Below is an example of literally what I could put as the username:

‘ or user=’admin’#

No password needed.  No sanitisation done.  I might forgive them if it was a 12 year old system, but the earliest entry in the database (the developer’s username) was last year.  It used jQuery 1.10 and bootstrap 3.3.8, which only came out in November.  Bad person hitting a keyboard (no longer afforded developer status).

Ok, so what can be done to prevent these things?  Firstly, stop using mysql_* functions.  They were deprecated in PHP 5.5, and there’s other alternatives around which are better, like PDO and mysqli_*.  That said, in the given example code, I could have done the following with mysqli_query functionality and potentially had worse consequences:

‘;drop table users;#

Which reminds me so much of the following XKCD comic:

Aside from using current database interaction functionality, you should NEVER put data sent from a browser into a query.  Always sanitise it in one way or another.  Ideally use parametised queries which will take care of sanitisation for you, and also will build the query in a much safer manner (PDO is my friend, it can be yours too).

There’s millions of web pages out there on how to prevent SQL injection, how to sanitise data input, how not to code, and why not to trust users.  Okay, so there’s also a lot of bad resources out there just the same, but there’s no reason anyone should be passing anything from the browser straight to the database.

Developers like to blame users when something in the system goes wrong.  Sure it would be a user issue if I logged in in that manner, but it’s more of a developer issue for allowing it in the first place.  Those things aren’t limited to just logins either though.  If you delete data using the following manner:

delete.php?id=2

Without input sanitisation, I can most likely do:

delete.php?id=2+or+id+>+0#

Wave goodbye to your data or, best case, enjoy finding everything being flagged as deleted in the database.

NEVER trust your users to input things correctly, and never trust their input.