Matt's Blog

@Nullable and @NotNull Annotations in IntelliJ

2017-04-21T06:58:00.002+10:00

Every time I have to configure a new instance of IntelliJ I have to hunt for these settings to ensure that inspections catch possible NPEs to code annotated with @NotNull, so I'm writing the steps here,

The inspection is under Java -> Probable bugs -> Constant conditions & exceptions.

You will also probably need to add the javax validation annotations, which can be done by clicking Configure annotations.

Is this code functional vomit?

2017-04-12T08:29:00.003+10:00

Hi. My name is Matt, and I have pull-request-criticism-ophobia. I die a little inside every time someone tells me what I noob I have been when writing code.

I am also a Java 8 developer, and have developed an interest in functional programing. One day I found the Optional class, and my life has been a living hell ever since. Here's why.

I was tasked to write the following code:

The method must convert an object of type ObjectA to an object of type ObjectD
The conversion will be done via a number of service objects that will do the hard work of converting ObjectA to ObjectB, ObjectB to ObjectC, and ObjectC to ObjectD.
These services do not throw exceptions, or throw only runtime exceptions.
These services return a mixture of null or empty Optional objects to indicate that no conversion could take place.
The method should return an empty Optional if the supplied ObjectA was null, or if any conversion returns null or an empty Optional.

Because I have read no less than 3 blog posts on Monads (so many cute pictures of cats in boxes) and have spent several hours reading up on functional programming, I feel like I am a functional programming boss, and pump out the following code:

public class MyClass {
@Autowired
private ConvertAToB convertAToB;

@Autowired
private ConvertBToC convertBToC;

@Autowired
private ConvertCToD convertCToD;

public Optional convertAToD(ObjectA objectA) {
return Optional.ofNullable(objectA)
.flatMap(convertAToB::convertToB)
.map(convertBToC::convertToC)
.map(convertCToD::convertToD);
}
}

The code is clean, declarative and functional. I’ve neatly dealt with nulls and empty optionals without nested if blocks. I’m feeling pretty good at this point. Until I go and listen to a talk by Stuart Marks, one of the guys who designed the Optional class, who very clearly says this:

Optional is intended to provide a limited mechanism for library method return types where there is a clear need to represent “no result,” and where using null for that is overwhelmingly likely to cause errors.

He goes on to list a number of rules around the use of the Optional class:

Never, ever, use null for an Optional variable or return value.
Never use Optional.get() unless you can prove that the Optional is present.
Prefer alternatives to Optional.isPresent() and Optional.get().
It’s generally a bad idea to create an Optional for the specific purpose of chaining methods from it to get a value.
If an Optional chain has a nested Optional chain, or has an intermediate result of Optional, it’s probably too complex.
Avoid using Optional in fields, method parameters, and collections.
Avoid using identity-sensitive operations on Optionals.

My heart sinks. It seems like my clever method chaining with map and flatMap just violated the intended purpose of the Optional class. Given my fear of criticism in pull requests, I quickly refactor the code to only use Optional as a return type and remove all method chaining:

public class MyClass {
@Autowired
private ConvertAToB convertAToB;

@Autowired
private ConvertBToC convertBToC;

@Autowired
private ConvertCToD convertCToD;

public Optional convertAToD(ObjectA objectA) {
if (objectA != null) {
Optional objectB = convertAToB.convertToB(objectA);
if (objectB.isPresent()) {
ObjectC objectC = convertBToC.convertToC(objectB.get());
if (objectC != null) {
ObjectD objectD = convertCToD.convertToD(objectC);
return Optional.ofNullable(objectD);
}
}
}
return Optional.empty();
}
}

Great. I’m pretty sure I’m no longer violating the intended use of the Optional class here. It is used as a return type and nothing more, and I am no longer chaining methods to get a value.

But, try as I might, I just can’t see this refactoring as better code.

Help me internet. Is Optional map and flatMap chaining an example of clean code, or is it functional vomit that abuses the limited scope of the Optional class?

Configuring Wildfly for HTTPS in a post Poodle world

2015-08-17T10:11:00.002+10:00

If you have ever run into “ssl_error_no_cypher_overlap” errors trying to configure Wildfly to use HTTP then you have probably cursed the lack of decent documentation for configuring Wildfly now that browsers have disabled a lot of insecure SSL cyphers. This is how I got around the problem.

First you need a self signed key. This can be created with the command:

keytool -genkey -alias mycert -keyalg RSA -sigalg SHA256withRSA -keystore my.jks -storepass secret -keypass secret -validity 9999

Then you need to configure Wildfly to accept a list of known cyphers. Mozilla has a nice list of cypher codes for high security, compatibility etc at https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility.

The problem is that this list has the OpenSSL key names, and Wildfly needs the RFC names. So you need to map one to the other using the table at https://testssl.sh/openssl-rfc.mappping.html.

What I ended up with was this list, defined in a enabled-cipher-suites attribute. This list is the Modern Compatibility list. This is added to the <subsystem xmlns="urn:jboss:domain:undertow:2.0"> element.

<https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA" />

You then need to configure the self signed key with this. It goes under the <management><security-realms> element.

<security-realm name="UndertowRealm"> <server-identities> <ssl> <keystore path="my.jks" relative-to="jboss.server.config.dir" keystore-password="secret" alias="mycert" key-password="secret"/> </ssl> </server-identities> </security-realm>

Authenticating via Kerberos with Keycloak and Windows 2008 Active Directory

2015-07-15T07:09:00.001+10:00

The following instructions show you how to configure Keycloak with Windows AD in order to use Kerberos authentication.

Assumptions

The Kerberos realm is VIRTUAL.LOCAL
The hostname used to access Keycloak is virtual.local. This just means we are running Keycloak on the domain controller. In production virtual.local will be replaced with something like keycloak.dev.virtual.local or something like that, giving you a SPN of HTTP/keycloak.dev.virtual.local@VIRTUAL.LOCAL

Configuration

Create a windows domain account called Keycloak.
Run the following command to assign a SPN to the user and generate a keytab file:
ktpass -out keycloak.keytab -princ HTTP/virtual.local@VIRTUAL.LOCAL -mapUser Keycloak@VIRTUAL.LOCAL -pass password1! -kvno 0 -ptype KRB5_NT_PRINCIPAL -crypto RC4-HMAC-NT
Verify the SPN has been assigned to the user with the command:
setspn -l Keycloak
Configure the LDAP settings in Keycloak like this. Since we are running Keycloak on the domain controller, we reference LDAP via the local loopback address. Obviously this would change in most production environments.
Configure the Kerberos integration like this:

Configuring Java

You will need to install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for your version of Java. Download them for Java 8 at http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html, or Java 7 at http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html.

Notes

Don't use port names in the SPN. By default Keycloak will run under port 8080, but this must not be added to the SPN, even though SPNs can contain port details.

To make use of Kerberos authentication from a Windows client, the Keycloak server has to be in a Internet Zone that has User Authentication -> Logon -> Automatic logon with current user name and password enabled. Chrome and IE both respect this settings.

To enable Kerberos authentication in Firefox, you need to add the Keycloak domain to the network.negotiate-auth.trusted-uris setting in about:config.

http://sourceforge.net/p/spnego/discussion/1003768/thread/960ba7ad/ has some useful information that can be used to debug exceptions.

Testing

This page can be used to quickly test the ability for a client to log in.

Get the keycloak.js file from the Keycloak JavaScript Adapter download.

Get the keycloak.json file from Keycloak itself.

<html>
<body>
<script src="keycloak.js"></script>
<script>
var keycloak = new Keycloak('keycloak.json');
keycloak.init({ onLoad: 'login-required' }).success(function(authenticated) {
keycloak.loadUserProfile().success(function(profile) {
alert(JSON.stringify(profile));
});
});
</script>
</body>
</html>

Some notes on Java 8

Builds of Java 8 above 1.8.0_31 have a bug that will throw the Defective token detected (Mechanism level: GSSHeader did not find the right tag) exception. See http://sourceforge.net/p/spnego/discussion/1003769/thread/700b6941/#cb84 for details.

Remember me not - avoiding the Australian metadata dragnet with Tor and Asus

2015-04-09T08:13:00.002+10:00

So it is official. As an internet user in the great country of Australia my actions online are now tracked and recorded by the government. And that doesn't sit so well with me.

But rather than complain, I decided to take action and install an Asus RT-N66U router as the gateway on my home network. The router had generally positive reviews online, but I was mostly interested in the fact that it supported third party firmware, like the popular ones provided by a developer calling himself Merlin.

One of the big benefits provided by third party firmware releases is that you get early access to some cool new features. One such feature that caught my eye was the introduction of Tor into the router.

I have used Tor sporadically in the past. While I have to commend the Tor developers for making it easy to install Tor and browse anonymously, the reality is that running an additional piece of software was kind of a pain. It was a mental jump to go from "always online" to "first make sure you have opened the Tor software, wait for it to connect, then go online", especially when the process was different depending on whether you are on a desktop PC or a tablet.

The end result was that I usually spend a few hours on Tor only to eventually put it in the "too inconvenient" basket and give up.

Which is why having Tor enabled at the router level is so nice. It's a one off setup, after which you never have to think about Tor again.

So how well does Tor on the N66U work?

First of all, the configuration is a breeze. Well, a breeze for someone who is knowledgeable enough to be flashing their home router with custom firmware. The process is not going to appeal to those who don't know what a router is, but anyone with a bit of technical knowledge, or a nerdy friend, can be up and running without too much trouble.

As you can see by the screenshot above, enabling Tor is a matter of selecting "Enable" in a drop down list, and deciding on whether to protect the entire network, or just a few select devices.

Once Tor is enabled, there is nothing more that needs to be done on the devices that you have selected. There is no software to be installed, and no process required to enable Tor.

This convenience comes at a cost though. Tor is only as secure as those applications whose data it is transferring, and one of the benefits of the Tor bundle is a browser that has disabled a number of plugins that are known to leak identifiable information.

However, my goal is to frustrate the government metadata dragnet that has been cast over this country's internet rather than achieve anonymity. Now that the average Australian citizen has to treat their ISP as a Man-In-The-Middle attacker, I'm willing to trade a little less anonymity for the convenience of having the packets that make up my web browsing obscured in a way that makes the government's logging efforts mostly meaningless.

Performance wise, there is a noticeable lag as you open web pages, and complex pages (i.e. 90% of the pages you'll view online) take noticeably longer to load. You'll also notice that the browser continues to show pages as still loading long after the page has been displayed to you. This is most likely because background scripts loading analytics and advertising scripts are still being download after the main content of the page has been displayed. Initially it is a little disconcerting to see pages taking so long to completely load, but you quickly learn to ignore the spinning icon on the browser tab and jump in once the page is rendered.

Overall, while this performance lag is noticeable, I never felt that it prevented me from doing what I needed to do online.

The biggest downside to using Tor is the number of Capchas you'll need to fill out to visit every day sites. Google is especially bad, depending on the Tor exit node you happen to be assigned.

You may find that Google's over zealous bot protection renders the search box on your mobile device useless. Apparently there is no facility on the native Android search widget to prove that you are in fact a human and not a bot scraping Google's search results.

You'll also notice that the ads you are served are now probably in another language. Given that I have clicked on maybe 4 ads in my entire life, this is amusing, and not particularly troublesome.

I even managed to stream videos at 780p. 1080p was asking just a little to much, but if I'm being honest, I don't see much difference between 780p and 1080p anyway.

There are a few things that I didn't try to do over Tor.

The first was streaming Geolocked content (i.e. Netflix). I have a dedicated media PC that uses a VPN (the VPN is also configured through the the N66U) rather than Tor. I imagine that it would be quite difficult to use Tor with something like Netflix, given that you don't have a lot of control as to where your traffic exits the Tor network. One minute you may appear to be in the US, the next in Europe.

I don't torrent, but no one should be using Tor for torrents anyway.

I also don't use applications like Skype. Actually, Skype uses UDP, and won't be routed over Tor anyway.

Even though Tor on the N66U is still in beta, configuration is relatively easy, and I never found myself having to reset the router or perform any kind of troubleshooting. Performance is adequate for day to day browsing, and the ability to only route traffic from specific devices over Tor means that your more bandwidth sensitive devices (like your media PC) are not affected.

Overall, I am quite impressed with how Asus and Merlin have integrated Tor into the N66U. A Tor enabled router is a great way to effortlessly opt out of the Governments metadata dragnet.

Fixing OpenVPN "Authenticate/Decrypt packet error: cipher final failed"

2015-03-08T10:38:00.000+11:00

When connecting to a VPN I was constant getting the error

Mar 8 09:29:27 openvpn[1696]: Authenticate/Decrypt packet error: cipher final failed

I had imported the supplied ovpn file and had followed all the other configuration steps, so this was quite frustrating. Then I saw this in the logs:

Mar 8 09:31:07 openvpn[1790]: WARNING: 'cipher' is used inconsistently, local='cipher BF-CBC', remote='cipher AES-256-CBC'

Changing my client to use "cipher AES-256-CBC" instead of the default (which apparently was cipher BF-CBC) fixed the issue.

A XSS filter for Java EE web apps

2014-12-07T09:17:00.002+11:00

Cross Site Scripting, or XSS, is a fairly common vector used to attack web sites. It involves user generated code being redisplayed by a website with all the privileges and security rights that a browser assigns to code originating from the current host. If the user code is something like <script>doEvil();</script>, then you have a problem.

OWASP is an organisation that provides guidance on web security, and they have a page that provides a suggested method for avoiding XSS in JavaEE web app. You can read this document at https://www.owasp.org/index.php/How_to_add_validation_logic_to_HttpServletRequest.

The library being demonstrated here is based off the ideas presented in that article, but fleshed out to be more flexible and easy to deploy. We call this library the (unimaginatively named) Parameter Validation Filter, or PVF.

PVF is implemented as a Servlet filter that intercepts requests to web pages, runs submitted parameters through a configurable sequence of validation rules, and either sanitises the parameters before they are sent through to the web application, or returns a HTTP error code if validation errors were detected.

We have made the following assumptions when developing this library:

Client side validation will prevent legitimate users from submitting invalid data.
The PVF library should prevent further processing if invalid data is submitted in the majority of cases.
Occasionally it might be appropriate to sanitise submitted data, but any sanitisation should be trivial (like the removal of whitespace).

To make use of the PVF library, you’ll need to add it to your project. This artifact is currently in the Sonatype staging repo, so you'll need to add that repo to your Maven config. See http://stackoverflow.com/questions/13945757/how-do-you-import-a-maven-dependency-from-sonatype-org for details.

<dependency>
  <groupId>com.matthewcasperson</groupId>
  <artifactId>parameter_validation_filter</artifactId>
  <version>LATEST</version>
</dependency>

The filter then needs to be added to the web.xml file with the following settings. You may want to configure the url-pattern to match the pages that you actually want to protect.

<filter>
  <filter-name>ParameterValidationFilter</filter-name>
  <filter-class>com.matthewcasperson.validation.filter.ParameterValidationFilter</filter-class>
  <init-param>
    <param-name>configFile</param-name>
    <param-value>/WEB-INF/xml/pvf.xml</param-value>
  </init-param>
</filter>
<filter-mapping>
  <filter-name>ParameterValidationFilter</filter-name>
  <url-pattern>*.jsp</url-pattern>
</filter-mapping>

Finally you need to create a file called WEB-INF/xml/pvf.xml. This file defines the custom validation rules applied to the parameters being sent to your web applications.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!-- ParameterValidationChainDatabase is always the document element -->
<ParameterValidationChainDatabase>
   <!--
        Enforcing mode needs to be set to true to return a HTTP error code if validation fails.
        If set to false, validation errors are logged but ignored.
    -->
    <EnforcingMode>true</EnforcingMode>

<!-- We always have a single ParameterValidationChains element under the parent -->

    <ParameterValidationChains>

        <!-- Each chain of validation rules is contained in a ParameterValidationDefinition element -->
        <!-- 
            This chain apply some global validation rules. If anyone supplies encoded or params with HTML
            characters, it will fail.
         -->
        <ParameterValidationDefinition>
            <!-- This is the list of validation classes that should be applied to matching parameters -->
            <ParameterValidationRuleList>
                <ParameterValidationRule>
                    <!-- This is the fully qualified name of the class used to apply the validation rule -->
                    <!-- All input fields are to be trimmed of excess whitespace -->
                    <validationRuleName>com.matthewcasperson.validation.ruleimpl.TrimTextValidationRule</validationRuleName>
                </ParameterValidationRule>
                <ParameterValidationRule>
                    <!-- No parameters are expected to already be encoded -->
                    <validationRuleName>com.matthewcasperson.validation.ruleimpl.FailIfNotCanonicalizedValidationRule</validationRuleName>
                </ParameterValidationRule>
                <ParameterValidationRule>
                    <!-- No parameters are expected to contain html -->
                    <validationRuleName>com.matthewcasperson.validation.ruleimpl.FailIfContainsHTMLValidationRule</validationRuleName>
                </ParameterValidationRule>
            </ParameterValidationRuleList>
            <!-- This is a regex that defines which parameteres will be validated by the classes above -->
            <paramNamePatternString>.*</paramNamePatternString>
            <!-- This is a regex that defines which URLs will be validated by the classes above -->
            <requestURIPatternString>.*</requestURIPatternString>
            <!--
                 Setting this to false means the paramNamePatternString has to match the param name.
                 Setting it to true would mean that paramNamePatternString would have to *not* match the param name.
             -->          
            <paramNamePatternNegated>false</paramNamePatternNegated>
            <!--
                 Setting this to false means the requestURIPatternString has to match the uri.
                 Setting it to true would mean that requestURIPatternString would have to *not* match the uri name.
             -->
            <requestURIPatternNegated>false</requestURIPatternNegated>
        </ParameterValidationDefinition>        
    </ParameterValidationChains>
</ParameterValidationChainDatabase>

The XML has been commented to make it easier to understand, but there are a few interesting elements:

paramNamePatternString, which has been configured to enable the validation chain to match all parameters
requestURIPatternString, which has been configured to enable the chain to match all URIs
The three elements called validationRuleName, which reference the full class name of the validation rules that will be applied to each parameter passed into our web application

Although this is a simple example, the three validation rules that have been implemented (TrimTextValidationRule, FailIfNotCanonicalizedValidationRule and FailIfContainsHTMLValidationRule) are quite effective at preventing a malicious user from submitting parameters that contain XSS code.

The first rule, TrimTextValidationRule, simply strips away any whitespace on either side of the parameter. This uses the trim() function any developer should be familiar with.

The second rule, FailIfNotCanonicalizedValidationRule, will prevent further processing if the supplied parameter has already been encoded. No legitimate user will have a need to supply text like %3Cscript%3EdoEvil()%3B%3C%2Fscript%3E, so any time encoded text is found we simply return with a HTTP 400 error code. This rule makes use of the ESAPI library supplied by OWASP.

Like the second rule, the third rule will prevent further processing if the supplied parameter has any special HTML characters. If you would like your customers to be able to pass through characters like &, this rule is too broad. However, it is almost always valid to block special HTML characters.

If you want to see how effective this simple validation chain is, check out the live demo at http://pvftest-matthewcasperson.rhcloud.com/. You may want to take a look at https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet to find some XSS patterns that are often used to bypass XSS filters.

Moving forward we will be looking to implement more targeted validation rules, especially those that can’t be easily implemented as regex matches (like making sure a date if after today, or that a number is between two values etc).

If you have any suggestions, or find any bugs, feel free to fork the code from our GitHub repo . We do hope to get some public feedback in order to make this library as robust as it can be.

Three Daily Things

2014-07-19T15:50:00.003+10:00

Three Daily Things is a new, free website that I have put together based on a motivation app that I have had great personal success with.

The reason why I wrote this app is best explained with a story about myself.

Like most people, I want to be fit an healthy. To achieve this, I sign up to my local gym. But it doesn't take long for the unknowns to start rattling around in my brain. How many reps and set should I do? Should I do cardio before or after resistance training? Should I be taking supplements? Is it best to work out in the morning or afternoon? Which exercises are best?

All these unknowns start to weigh heavily in my mind. I begin to wonder if I am wasting my time. I skip a few sessions, and before I know it I haven't visited the gym in weeks.

So many beneficial aspirations in my life have followed this path. I like the idea, I try it out, I get overwhelmed by the unknowns and eventually I give up.

So I asked myself, what was it that I really wanted out of my gym membership? The answer is right there in the first sentence: to be fit and healthy. And I realize that I can achieve this goal without actually knowing the best ratio of sets to reps, the most effective exercises and without taking supplements. All I need to do is walk through the gym door. Simply being in the gym day after day and doing some sort of exercise achieves my goal of being fit and healthy.

In a nutshell, my problem is that in seeking perfection I often end up with nothing. The solution was to convince myself that continuing to focus on these beneficial activities, even if I do them imperfectly, will eventually get me the result I am looking for.

That is why I created Three Daily Things. It is a way to remind myself that any time I spend working on the three things that I feel would most improve my life is worthwhile.

The app is deliberately simple. You list the three things that you feel would most improve your life, and when you spend any amount of time working on them, you give yourself a tick. The report allows you to quickly gauge how often you have made time (any time) for these three things over the last week.

For such a simple idea, my personal results were quite incredible. Prior to writing this app, I would estimate that I would have spent time on my three most important things maybe six or seven times a week, giving me a weekly score of around 30%. Using the app daily, my weekly score usually sits between 60% and 80%. That means I have easily doubled my focus on those three things that I know are improving my life.

This started out as a personal motivational tool, but it has worked so well that I decided to make the web site free to everyone in the hopes that others would find Three Daily Things as simple and effective as I have.

RHEL 7 VirtualBox Guest Additions Patched

2014-06-13T15:47:00.004+10:00

If you have tried to used RHEL 7 in VirtualBox, and run into the issue with the guest additions not compiling (see https://www.virtualbox.org/ticket/12638 for details), you can download this tar file, extract it, and run

sudo ./install.sh

The tar file is just the guest additions pre patched to work with RHEL 7.

Scroll to the bottom of log files in web pages

2014-01-24T11:57:00.000+11:00

We use supervisord on our systems, which has a handy browser based 'tail -f' feature. The problem is that the end of the log file appears off the bottom of the screen, and the browser won't scroll to the bottom automatically. This bookmarklet will keep the end of the page in view.

javascript:scroll=function(){setTimeout(function(){window.scrollTo(0,document.body.scrollHeight);scroll();},100);};scroll();

If you are a knowledge gatekeeper, the game is changing

2014-01-11T08:46:00.000+11:00

"We take comfort from Charles Darwin's observation that it's not the strongest species that survives, nor the most intelligent, but the ones most responsive to change. We just need to be adaptable." - Gary Pruitt, then the CEO of McClatchy Newspapers and now CEO of the Associated Press (http://www.slate.com/blogs/future_tense/2012/11/12/google_ad_revenue_tops_entire_us_print_media_industry_chart.html).

I recently spent some time looking into the stats for web content we were responsible for and noticed two things. The first was that the majority of visitors were viewing chunked versions of our books, where each individual web page aligned roughly with an entry in the table of contents. The second was that most visitors were spending only a couple of minutes reading the content in any given session.

This observation aligned very closely with my own information gathering process. Once or twice a year I'll make time to sit down and read a technical book from start to finish. This is usually because I am studying for a test or need to familiarise myself with a new technology. This compares to the dozens of web searches I do every single day to find answers to various issues that I run into as part of my daily job.

My productivity depends in large part on my ability to find answers to specific and usually isolated problems. I imagine a lot of information workers are in the same boat.

Right now the process of finding answers is to do a search in Google and then skim over the resulting links to try and find an answer. Or in other words, Googling.

The fact that Googling involves viewing the original content is very important to those that create, collate and profit from factual digital content (think wikis, knowledgebases, books, articles, blogs etc), because it means that they are the gatekeepers to their digital kingdoms. The answers people seek can be presented, restricted and sold in any way that best suits the knowledge gatekeepers.

That is about to come to an end (http://slashdot.org/topic/bi/ibm-investing-1-billion-in-watson-business-unit/):

If everything goes according to IBM’s plan, Watson will help kick off what CEO Ginni Rometty refers to as a third era in computing. The 19th century saw the rise of a “tabulating” era: the birth of machines designed to count. In the latter half of the 20th century, developers and scientists initiated the “programmable” era—resulting in PCs, mobile devices, and the Internet. The third (potential) era is “cognitive,” in which computers become adept at understanding and solving, in a very human way, some of society’s largest problems.

Knowledge is about to become commoditized. Since most people want answers to questions, this third era of computing is going to demote knowledge gatekeepers to funnels into systems like Watson. At best your original content will be presented as a footnote against the answer provided by Watson. At worst, no will will care where the answer came from, only that it is accurate.

Today it takes teams of highly skilled employees to create, curate, manage and navigate digital knowledge repositories designed to support vertical markets. Stick a support contract in front of that and you have the makings of a billion dollar business. In the not too distant future, some kid in a dorm room is going to be able to scrape every forum, wiki, book, bug report, chat log and code base that relates to your business or product, funnel it into a system like Watson, and produce a “Ask Watson” style web page that renders your once valuable digital knowledge repositories obsolete.

And if you think that your content is so engrained, valuable or unique as to be immune from disruption from this third era of computing, I have shares in Encyclopaedia Britannica, print based news media and bricks and mortar book shops that I would like to sell you.

Adaptation is the key to surviving the transition from valuable knowledge gatekeeper to anonymous information source. Because make no mistake, change is upon us right now, and history is littered with the corpses of business that missed the signs.

Your grandkids are going to laugh at the notion of "searching the web"

2014-01-05T10:23:00.003+11:00

In the next 5 - 10 years our good friend and faithful digital companion, the search bar, is going to ride off into the sunset.

Googling has forever changed how we find answers to almost anything we could ever want to know, because the answer is most likely there buried in the first 10 links brought up in response to anything you type into the search bar.

When you think about it though, having to trawl through those 10 links is actually not a great way to answer a question. If you were to ask your mechanic what was making that noise in your engine, you want the answer, not rambling descriptions and discussions on engine design and maintenance. But that is exactly what you get if you were to ask Google the same question: pages of forum posts when you only want the last one that actually has the answer; a blog post where that engine noise is coincidentally mentioned in the comments; or a 10 minute YouTube video where the information you want is 9 minutes in.

That is all about to change. In the not too distant future you will type your query into a question bar, and the result will be a plain English answer. No more "this search returned 10000000000000 results"; no more clicking through web pages to find the information you are looking for; and, if Google isn't careful, potentially no more Googling.

We have seen a glimpse of this move from searching to asking with Wolfram Alpha. You enter some search criteria, Alpha identifies real world entities or concepts in the search query, and then displays a bunch of cross referenced information. It's a great tool if you are looking for something that is easily identifiable, like the manufacturer of your car, and possibly even your car's model. But Alpha is not going to tell you what that sound in your engine could be.

That's not to say that Wolfram is giving up. He has made some vague reference to a new knowledge based programming language:

We call it the Wolfram Language because it is a language. But it’s a new and different kind of language. It’s a general-purpose knowledge-based language. That covers all forms of computing, in a new way.

Apple has made some bold moves away from searching with Siri. Granted, Siri is often just a speech to text converter for plain old web searches, but it is not hard to imagine Siri connecting the dots between the questions you asked about "that noise in my engine" and post number 13 in a forum conversion that mentions a fan belt with the same ease with which she connects you request to set an alarm with the settings in your iPhone alarm clock app.

Microsoft is also making a move towards asking questions:

"Our vision of search is 'just ask'. Search is the ultimate human interface. It should be able to cope with any input."

And then there is IBM, which bested some of the brightest minds ever to grace the game show Jeopardy with their system Watson. Watson is probably the best public example of a system that can answer general questions with plain English responses, and that power is going to be available to developers:

Developers who want to incorporate Watson’s ability to understand natural language and provide answers need only have their applications make a REST API call to IBM’s new Watson Developers Cloud. “It doesn’t require that you understand anything about machine learning other than the need to provide training data,” Rob High, IBM’s CTO for Watson, said in a recent interview about the new platform.

Google has been fairly quiet about this post web search world, probably because it is of no benefit to them to suggest that searching is going anywhere, but their acquisitions of data sources like Freebase, the sidebar that displays relevant information about any entities in your search query and Google Now do indicate that they are looking to move beyond just plain old web searches too.

So when my grandkids ask me what the internet was like when I was their age, I will tell them of the great information hunts I used to go on, carefully crafting my search queries and then staking the answer through pages of unrelated and often unhelpful information. And they will laugh.

Three.js Tutorials

2013-11-15T16:38:00.001+11:00

A tutorial series looking at the JavaScript 3D engine three.js.

Three.js - 1 - Getting Started Part One
Three.js - 2 - Getting Started Part Two

Three.js - 2 - Getting Started Part 2

2013-11-15T16:19:00.000+11:00

Three.js - 2 - Getting Started Part Two

THREE.JS TUTORIALS

VIEW THE DEMO

DOWNLOAD THE SOURCE CODE

Introduction

In the last tutorial we got a very basic three.js application up and running. This tutorial will continue on from that, adding a stats counter and dealing with window resizing.

Stats Counter

A number of the example three.js applications have a neat little stats counter showing the number of frames per second, and displaying a graph of the frame rate over the last few seconds. This information is invaluable when testing the performance of your 3D application.

The JavaScript library responsible for this stats counter is not actually part of three.js. It is a separate library that can be found on GitHub at https://github.com/mrdoob/stats.js/. The file that we need to include in our project can be found at https://github.com/mrdoob/stats.js/blob/master/build/stats.min.jshttps://github.com/mrdoob/stats.js/blob/master/build/stats.min.js. This JavaScript file is included like the others in the <head> element.

 <head>
  <title>Three.js - 2 - Getting Started 2</title>
  <link rel="stylesheet" type="text/css" href="css/style.css">
  <script type="application/javascript" src="javascript/stats.min.js"></script>
  <script type="application/javascript" src="javascript/three.min.js"></script>
  <script type="application/javascript" src="javascript/threejs-2.js"></script>
 </head>

If you look at the code above, you'll notice that we have also included a link to a CSS file. The DOM element created by the stats.js library has the ID of stats. Knowing this, we can create a CSS rule to position the stats counter in the top left hand corner of the page.

#stats { position: absolute; top:0; left: 0 }

With the stats.js library loaded and the DOM element styled, we now create the stats counter and add it to the HTML page in JavaScript. The following lines are added to the end of the init() function. Note that we are appending the stats element directly to the body of the HTML page, just like we did with the renderer in the last tutorial. Because we specified absolute positioning in the style that is applied to the stats element, it will not interfere with the position of the renderer.

 stats = new Stats();
 document.body.appendChild( stats.domElement );

The final step is to update the stats counter every time a frame is rendered. This is done with the following code added to the end of the animate() function.

 stats.update();

Window Resizing

Take a moment to run the example from the last tutorial, and after it has been loaded, resize the window. Notice that the 3D cube does not change its position relative to the top left hand corner of the screen. Obviously this is not ideal.

Fixing this requires the camera's aspect ratio and projection matrix, and the renderer's size, to be updated if the window is resized. We can specify a function to be run in the event that the window is resized by adding an event listener to the window's resize event.

 window.addEventListener( 'resize', onWindowResize, false );

The onWindowResize() function is where we update the camera's aspect ratio and projection matrix, and the renderer's size.

function onWindowResize() {
 camera.aspect = window.innerWidth / window.innerHeight;
 camera.updateProjectionMatrix();
 renderer.setSize( window.innerWidth, window.innerHeight );
}

When running the example supplied with this tutorial, it is possible to resize the window and still have the cube placed in the middle of the screen.

Tips

If you click on the stats counter, you will switch between two views. The first displays the frames per second, while the second displays the milliseconds per frame.

MinHash for dummies

2013-11-14T14:36:00.002+11:00

Duplicate document detection is becoming increasingly important for businesses that have huge collections of email, web pages and documents with multiple copies that may or may not be kept up to date.

MinHash is a fairly simple algorithm that from all my Googling has been explained very poorly in blogs or in the kind of mathematical terms that I forgot long ago. So in this article I will attempt to explain how MinHash works at a practical code level.

Before I start, please take a look at http://infolab.stanford.edu/~ullman/mmds/ch3.pdf. That document goes into a lot of theory, and was ultimately where my understanding on MinHash came from. Unfortunately it approaches the algorithm from a theoretical standpoint, but if I gloss over some aspect of the MinHash algorithm here, you will almost certainly find a fuller explanation in the PDF.

I'll also be using pseudo Java in these examples instead of traditional math. This means when I use terms like Set, I am referring to the group of classes that implement a Set in Java. Of course a set in Java shares a lot of properties with a set in maths, but keep in mind that this is written from a developers point of view.

What does it mean when we talk about document similarity?

The whole point of MinHash is to efficiently answer the question "How similar is this document to that document".

Before we can answer that question, we need to look at a more simple example of two sets of strings and determine how similar they are.

Here are our two sets:

Set a = new Set(["chair", "desk", "rug", "keyboard", "mouse"]);

Set b = new Set(["chair", "rug", "keyboard"]);

A measure of how similar these two sets are is known as the Jaccard Coefficient. It is calculated as number of common elements / (total number of elements - number of common elements).

In our case we have 3 common elements: "chair", "rug" and "keyboard". We have a total of 8 elements. So the Jaccard Coefficient = 3 / (8 - 3) = 0.6, or 60%. http://www.planetcalc.com/1664/ is an online calculator that you can play around with to determine the similarity of two sets.

Documents as sets

While a document can be though of as a giant set of words, we don't just break down a document into individual words, place them in a set and calculate the similarity, because that looses the importance of the order of the words. The sets

Set a = new Set(["I", "went", "to", "work", "today"]);

Set b = new Set(["today", "I", "went", "to", "work"]);

would be considered 100% similar, even though they clearly are not.

Instead we break a document down into what are known as shingles. Each shingle contains a set number of words, and a document is broken down into total words - single length + 1 number of shingles. So if a document contained a single sentence of "The quick brown fox jumps over the lazy dog", that would be broken down into the following 5 word long shingles :

The quick brown fox jumps
quick brown fox jumps over
brown fox jumps over the
fox jumps over the lazy
jumps over the lazy dog

So our document as a set looks like:

Set a = new Set(["The quick brown fox jumps", "quick brown fox jumps over", "brown fox jumps over the", "fox jumps over the lazy", "jumps over the lazy dog"]);

These sets of shingles can then be compared for similarity using the Jaccard Coefficient.

Optimizing the process

So we now have a way to compare two documents for similarity, but it is not an efficient process. To find similar documents to document A in a directory of 10000 documents, we need compare each pair individually. This is obviously not going to scale well.

What we can do to reduce some cycles is compare sets of randomly selected shingles from two documents. So for a document that is 10000 words long, we break it down into 9996 shingles, and randomly select say 200 of those to represent the document. If the second document is 20000 words long, we boil that down from 19996 shingles to another set of 200 randomly selected shingles.

Now instead of matching 9996 shingles to 19996 other shingles, we are are comparing 200 shingles to another 200 shingles. We have reduced our workload at the expense of accuracy, but this is still good enough for most scenarios.

NOTE

So just how much accuracy did we sacrifice by comparing only a small subset of randomly selected shingles? I actually don't know, but I assume it sits on a bell curve where the number of random selections improves the accuracy to a point where more random selections won't provide any meaningful increase in accuracy. If you know the math behind this, please leave a comment.

I chose 200 random selections after reading http://blog.cluster-text.com/tag/minhash/.

Computing and saving the random shingle selections

So this is where MinHash comes in.

We could save the 200 randomly selected shingles, but storing variable length strings kinda sucks. It complicates database design and comparing strings is slow.

The clever part about MinHash is that it allows us to save integers instead of strings, and also takes the hassle out of randomly selecting shingles. It works like this.

1. Break down the document a set of shingles.

2. Calculate the hash value for every shingle.

3. Store the minimum hash value found in step 2.

4. Repeat steps 2 and 3 with different hash algorithms 199 more times to get a total of 200 min hash values.

At this point instead of 200 randomly selected shingles, we have 200 integer hash values. This is effectively a random selection of shingles, because a good hash algorithm is supposed to generate a number that is as likely to be large as it is likely to be small. This kind of distribution of hash codes is what hashing is all about, so selecting the smallest hash is, for all intents and purposes, a random selection of a shingle.

Um, 199 more hash algorithms? WTF!

So you see the String.hashCode() method, but where do these other 199 hash codes come from? This is the step that most explanations on the MinHash algorithm conveniently leave out.

The short answer is that you XOR the value returned by String.hashCode() with 199 random numbers to generate the 199 other hash code values. Just make sure that you are using the same 199 random numbers across all the documents.

If you want some more detail on why this works, see http://stackoverflow.com/a/19711615/157605.

Great, but I still have to compare every document to every other document. Can this be optimized?

This is where the second neat trick of MinHash comes in.

Locality sensitive hashing (LSH) involves generating a hash code such that similar items will tend to get similar hash codes. This is the opposite of what .hashCode() does.

LSH allows you to precompute a hash code that is then quickly and easily compared to another precomputed LSH hash code to determine if two objects should be compared in more detail or quickly discarded.

We don't actually calculate a LSH hash code as such, but the idea of a boiling down a complex object (like our collection of minhash codes) into something that is quickly and easily compared with other complex objects is still applicable.

What we do is take the collection of minhash values, and categorize them into bands and row. For the sake of simplicity we'll assume each document has 20 minhash values. We will break them down into 5 bands with 4 rows each. Also for the sake of simplicity we'll assume that .hashCode() (and any number XORed with it) results in a number between 0 and 9.

So conceptually the minhash values from 5 documents in the first of the 5 bands and its rows looks like this.

Band One

	MinHash Algorithm One	MinHash Algorithm Two	MinHash Algorithm Three	MinHash Algorithm Four
Document One	1	3	6	0
Document Two	2	3	1	0
Document Three	1	3	6	0
Document Four	2	1	3	1

What we are looking for are rows within a band that are the same between documents. Document One and Document Three in this case both have rows with values 1, 3, 6, 0. This tells us that these two documents should be compared for their similarity.

Although it is not shown here, you should look through Bands Two - Five looking for documents that share rows. Any documents that share rows in any bands should be compared for their similarity.

This is useful when you have a document, and you want to know which other documents to compare to it for similarity. I'll assume that the minhash values are stored in SQL, so to get out candidates for comparison you would write:

SELECT DocName

FROM Documents

WHERE (

BandOneMinHashAlgorithmOne = 1 AND

BandOneMinHashAlgorithmTwo = 3 AND

BandOneMinHashAlgorithmThree = 6 AND

BandOneMinHashAlgorithmFour = 0

) OR
(

BandTwoMinHashAlgorithmOne = # AND

BandTwoMinHashAlgorithmTwo = # AND

BandTwoMinHashAlgorithmThree = # AND

BandTwoMinHashAlgorithmFour = #

) OR (

BandThreeMinHashAlgorithmOne = # AND

BandThreeMinHashAlgorithmTwo = # AND

BandThreeMinHashAlgorithmThree = # AND

BandThreeMinHashAlgorithmFour = #

) OR (

BandFourMinHashAlgorithmOne = # AND

BandFourMinHashAlgorithmTwo = # AND

BandFourMinHashAlgorithmThree = # AND

BandFourMinHashAlgorithmFour = #

) OR (

BandFiveMinHashAlgorithmOne = # AND

BandFiveMinHashAlgorithmTwo = # AND

BandFiveMinHashAlgorithmThree = # AND

BandFiveMinHashAlgorithmFour = #

)

This explanation completely glosses over why this works and what combination of bands and rows you should construct when looking for similar documents. If you want to get into that detail, take a look at http://infolab.stanford.edu/~ullman/mmds/ch3.pdf.

Three.js 1 - Getting Started

2013-11-10T16:02:00.003+11:00

Three.js - 1 - Getting Started

THREE.JS TUTORIALS

VIEW THE DEMO

DOWNLOAD THE SOURCE CODE

Introduction

After Apple's war on Flash, relying on plugins in your web pages is just not a viable option. Flash (and maybe, thanks to Netflix, Silverlight) were the last of a dying breed of browser plugins. In their place are a host of new technologies being standardised as part of HTML5.

WebGL is one of these technologies, and is supported by all major browsers. Microsoft is even getting on board with IE 11. Combine that with the ever increasing performance of JavaScript engines and it is now possible to create some impressive 3D applications in a web browser without the aid of plugins.

Three.js is one of the more powerful and mature JavaScript 3D engines available. This article will show you how to get a simple 3D application running with three.js.

What to download

Using three.js requires only a single JavaScript file called three.min.js. You can grab the latest version of file from GitHub.

The HTML page

A three.js application is a DOM element that can be appended as a child of an element in your HTML page. This means that you can embed a 3D application in a web page like it was an image or video file. This gives you a great deal of flexibility when presenting your 3D application.

However, for this example we will create a very basic HTML page and add the three.js application as the only child. While basic, this configuration is quite suitable for full screen games or simulations.

As you can see, this is a very simple HTML page. The only code of importance here are the referenced to the two JavaScript files: the three.js library (three.min.js), and the JavaScript file that will hold our application (threejs-1.js).

<html>
 <head>
  <title>Three.js - 1 - Getting Started</title>
  <script type="application/javascript" src="javascript/three.min.js"></script>
  <script type="application/javascript" src="javascript/threejs-1.js"></script>
 </head>
 <body>
  
 </body>
</html>

The Components of a 3D Application

There are 4 important classes in three.js that make up a basic 3D application: the camera, the scene, the meshes, the geometries and the materials.

The Camera

This is exactly what it sounds like. The 3D world created by three.js is viewed as if through the lens of a video camera. In fact the camera has methods like .setLens() to set properties like focal length and frame size, which relate to a camera's field of view.

The Scene

The scene is the world that you will populate with 3D objects, lights and cameras.

The Meshes

A mesh defines the individual charictaristics of a 3D object. These include things like the position, rotation and scale of the object. A mesh will reference a geometry to define the shape of the 3D object, and a material to define the appearance of the 3D object.

What is important to note about a mesh is that they can share geometries and materials. For example, to render a forest with a thousand 3D trees, you would need a thousand meshes, but potentially only 3 or 4 geometries (to represent the different types of trees) and 1 material.

The Geometries

The shape of a 3D object is defined by a geometry. Geometries define the various vertices and faces that make up a 3D model. Simple applications can make use of the build in geometry classes that describe a sphere, cube, cylinder and other common shapes. More complex 3D applications will load geometries created in 3D modelling applications.

The Materials

While a geometry defines the shape of a 3D object, the material defines the appearance. Materials can be a simple as a plain colour, or they can be complex shaders with reflections or animated appearances.

Putting it all Together

We start by assigning a function to the window.onload event. This is good practice when writing any JavaScript that manipulates the HTML DOM, as it is possible to start playing with the DOM before it is completely loaded. If you use jQuery, the ready() function is even better as it will allow your code to manipulate the DOM without having to wait for images to download. Since our page has no images, and we are not using jQuery, the window.onload event will work just fine.

The function assigned to the window.onload event will call two additional functions: init() and animate().

window.onload = function(){
 init();
 animate();
};

The init() function is where we setup the components that make up the 3D application.

We create the camera, which is an instance of the PerspectiveCamera class. The 4 parameters supplied to the constructor are

fov - Camera frustum vertical field of view.
aspect - Camera frustum aspect ratio.
near - Camera frustum near plane.
far - Camera frustum far plane.
We then set the position of the camera down the z axis. This will allow us to position a 3D object at the origin and be able to view it.

camera = new THREE.PerspectiveCamera( 75, window.innerWidth / window.innerHeight, 1, 10000 );
 camera.position.z = 1000;

Next we create some geometry. Three.js provides a number of geometry classes that define a number of common shapes. Here we use the CubeGeometry class to create the geometry for a cube. The supplied parameters define the size of the cube along the three axes.

 geometry = new THREE.CubeGeometry( 200, 200, 200 );

In order for the 3D object to be visible, it needs a material. The MeshBasicMaterial class allows us to create a simple material without the use of any external image files. The MeshBasicMaterial constructor takes an object whose properties define the appearance of the material. In this case we have set the colour to red, and specified that the material should display as a wireframe.

 material = new THREE.MeshBasicMaterial( { color: 0xff0000, wireframe: true } );

The mesh uses the geometry and material to create the 3D object that we will see on the screen.

mesh = new THREE.Mesh( geometry, material );

Now that we have a 3D object to view, we need to create the scene that will hold it. The mesh is then added to the scene.

 scene = new THREE.Scene();
 scene.add( mesh );

Three.js can use a number of technologies to draw the 3D application within the browser. Each is provided by a renderer class. These different renderers are provided because not all browsers support all the HTML technologies that are available to libraries like three.js. For example, Internet Explorer won't support WebGL until version 11.

We have used the WebGL renderer here. This code will fail if it is run on any version of IE before 11, and future tutorials will provide workarounds for older browsers. For now though make sure you run the demo on Firefox or Chrome.

Because our application will take up the full browser window, we set the size of the renderer to the size of the browser window.

 renderer = new THREE.WebGLRenderer();
 renderer.setSize( window.innerWidth, window.innerHeight );

The final step is to attach the renderer to the HTML DOM. In this case the renderer is attached directly as a child of the HTML body.

 document.body.appendChild( renderer.domElement );

At this point we have created all the objects required by the 3D application. The next step is to animate the scene.

Just like a flip book, animation of 3D applications quickly taking snapshots of the 3D objects and displaying the snapshots on the screen.

HTML pages used to be fairly static, and only updated in response to some kind of user interaction (like clicking a button or submitting a form). This model doesn't work well for applications like the one we are creating, which expect to be able to update the browser page with each frame, independently of any explicit user interaction.

Modern browsers support this kind of application through the requestAnimationFrame function. This function takes another function as a parameter, and will call the supplied function when the browser page is available to be refreshed. The requestAnimationFrame function is not supported by all browsers, but three.js includes a shim (or workaround) for older browsers, saving us from having to deal with these incompatibilities.

So the animate() function is first called when the window is loaded, and is then scheduled to be recursively called each time we render a frame.

 requestAnimationFrame( animate );

So we have something to look at, the cube is rotated slightly with each frame.

 mesh.rotation.x += 0.01;
 mesh.rotation.y += 0.02;

Finally, the renderer takes the scene and the camera, and renderers the current frame to the screen.

 renderer.render( scene, camera );

Final Words

The end result of this tutorial has been to display a single spinning 3D cube in a web page. While this is quite a trivial example of the three.js library, it does cover all the basic elements of a 3D application, and provides a useful starting point for building more complex 3D applications.

Stopping emails in the new Gmail tabs from showing up in new mail counts

2013-09-23T07:47:00.000+10:00

I love the new tabbed interface in Gmail. To be blunt, email clients have done a very poor job of dealing with the increasing volume and variety of email that people get nowadays, and it is nice to see Gmail implementing simple and effective solutions.

The one problem I have with the new interface is that email that was simply going to spam is now being categorised and filed under these tabs, and their existence is contributing to the number I see in my Chrome Gmail monitoring extension. It became impossible to know when I have mail that I was actually interested in, because the little icon in my browser was always showing 100+ new emails.

I initially tried to setup a filter, but nothing I could type into the new filter dialog would allow me to set emails sent to the categories to be marked as read.

Nothing you can type in here will let you create a filter based on the categories.

But I soon tweaked that you can create a filter from a search, and you can search based on categories.

Do a search with the categories: prefix.

Drop the search menu down again, and the Create filter with this search link is now active!

And with that, any email sent to one of the new tabs can be marked as read, leaving your mail monitoring extensions showing only the actual inbox count.

Exporting from MySQL to H2

2013-07-05T16:05:00.000+10:00

When it comes to testing, you can't beat the convenience of an embedded database. It can be created, populated and cleaned up without any dependencies on outside infrastructure, making it ideal for self contained test suites.

However, there is a very good chance that you don't use an embedded database in your production environment. In my case, I have a MySQL database in production, and want to use H2 in testing.

Getting data from MySQL into H2 is not as straight forward as you might think. You might have some luck with CopyDB, Scriptella, or OpenDBCopy. But in the end I simply used mysqldump and a couple of hacks to get data from MySQL to H2.

Dump the data

Dump the data from MySQL with the following command:

mysqldump --compatible=ansi,no_table_options,no_field_options,no_key_options --hex-blob --skip-opt -uroot -p Database > importansi.sql

This will strip out most of the MySQL specific syntax and give you something that you can clean up for H2 to import.

Fix up single quotes

MySQL will escape a single quote with a backslash, like \'. H2 expects to have single quotes escaped with another single quote, like ''. There has been a request to have this kind of escaping included as an option in MySQL, but it has not been implemented. So you'll have to do a find and replace for yourself.

Fix up hex numbers

H2 doesn't import hex numbers properly. To fix that up, you'll need to find any hex numbers and replace them with strings.

If you have a text editor that can find and replace with regular expressions, this can be done by finding 0x([A-F0-9]*) and replacing it with '$1'.

Fix up bits

The default value for a bit in MySQL will be represented as b'0'. Replace this with a plain 0 for H2.

Don't include ranges in keys

MySQL has the ability to define a range on a key, like KEY "ATextFieldKey" ("ATextField"(255)). This doesn't work in H2, so replace it with KEY "ATextFieldKey" ("ATextField").

Remove character sets

Remove any character set information on a field like CHARACTER SET latin1.

Remove COLLATE settings

Remove any collate information on a field like COLLATE utf8_unicode_ci.

Remove indexes on BLOBS, CLOBS and TEXT fields

H2 does not support indexes on these fields. Queries on these fields will be slower, but you should be able to take that into account in your tests.

Make all index names unique

MySQL only requires that an index name be unique within a table. H2 will require that it be unique within the database. So change any duplicated index names before importing into H2.

Use the MySQL compatibility mode

Even though we have dumped the MySQL database using the ANSI compatibility mode, we still need to enable the MySQL compatibility mode in H2. This is done through a connection URL that looks like jdbc:h2:~/test;MODE=MySQL.

Reorder the table creation sequence

MySQL will dump tables in alphabetical order. When it imports a database dumps, some commands in the comments instruct MySQL to ignore relationships, allowing MySQL to create tables with foreign keys to tables that have not yet been created.

H2 does not have the same ability to ignore relationships (I tried SET REFERENTIAL_INTEGRITY FALSE, but it didn't work), and so you'll have to do some copying and pasting to make sure that any table that is pointed to in a relationship exists before the relationship is created.

If you have circular dependencies, you'll have to remove the foreign key creation from the CREATE TABLE command, and create the foreign keys only after both tables have been created.

For those that want to automate this process, fork the repo hold this script. It has saved us quite a few hours worth of work.

Setting up a MariaDB Galera cluster in Centos 6

2013-07-03T15:16:00.001+10:00

Synopsis

One of the big draw cards of MariaDB over MySQL is open source support for clustering. While the open source version of MySQL allows for replication amongst a number of read only nodes, MariaDB has true multi-master replication.

If you have read the documentation at https://kb.askmonty.org/en/getting-started-with-mariadb-galera-cluster/, you could be forgiven for believing that setting up a cluster is a trivial process. Unfortunately those instructions leave out a few vital details that will prevent your cluster from working. These are the steps I used to get a MariaDB Galera Cluster running in Centos 6.

Configure the YUM repo

Go to https://downloads.mariadb.org/mariadb/repositories/ and select Centos as the OS, and 5.5 as the version. Galera (the clustering component of MariaDB) is only available with version 5.5 at this time.

I have the 64 bit version of Centos, so the resulting file looks like this:

# MariaDB 5.5 CentOS repository list - created 2013-07-03 04:40 UTC
# http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

The file for the 32 bit version of Centos looks like:

# MariaDB 5.5 CentOS repository list - created 2013-07-03 04:46 UTC
# http://mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos6-x86
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

You'll want to save this file into /etc/yum.repo.d/MariaDB.repo.

Install MariaDB

MariaDB is a drop in replacement for MySQL. If you have installed the desktop packages for Centos, chances are you have MySQL installed already. To uninstall it, run the following command:

yum erase *mysql*

Installing MariaDB Galera is done with the following command:

yum install MariaDB-Galera-server MariaDB-client setroubleshoot

You will notice that we are also installing the SELinux Troubleshooter. This will come in handy later.

Initial Database Configuraion

We need to start the MariaDB service. Because it is a drop in replacement for MySQL, you use the same command to start the server as you would to start MySQL.

service mysql start

Upgrading the databases fixes an issue where MySQL Workbench will simply say "fetching..." and will never list the tables available in the database.

mysql_upgrade

It is good practice to secure the new installation. MariaDB provides a script that will run you through the process.

mysql_secure_installation

We also need to setup some passwords. To do that we load the mysql command line client, and log into our database.

mysql -hlocalhost -uroot

The following commands will set passwords on the root account, both when accessed locally, and when accessed from a remote system. Obviously you can change the password in these commands to something a little more secure, but for this tutorial we'll use 'password'.

GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;

FLUSH PRIVILEGES;

exit

The last step is to shutdown MariaDB

service mysql stop

Open the network ports

A MariaDB cluster requires ports 3306 and 4567 to be open. If you use rsync to do the state transfers, you'll also need port 4444 open. You can do by running the command:

system-config-firewall-tui

In the other ports section, you need to open up 3306, 4444 and 4567, all with the tcp protocol.

Fix SELinux Issues

The quick way to fix this is to disable SELinux with the command:

setenforce 0

If you want to leave SELinux enabled, you'll need to restart Centos to allow the SELinux Troubleshooter we installed earlier to pick up on any violations.

When you attempt to run MariaDB in the steps below, SELinux Troubleshooter will report three errors.

Clicking the Details button on each of these alerts provides the commands you need to run to allow MariaDB to run with SELinux enabled.

Start the cluster

MariaDB requires 3 nodes to effectively provide fault tolerance. We'll get two running, but the process is the same for any additional nodes.

Start the first server with the command

mysqld --wsrep_cluster_address=gcomm:// --user=mysql --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_sst_auth=root:password

The --wsrep_cluster_address=gcomm:// option defines this as the initial node in the cluster.

The --user=mysql option tells MariaDB to start under the mysql user

The --wsrep_provider=/usr/lib64/galera/libgalera_smm.so option tells MariaDB where the Galera library is.

The --wsrep_sst_auth=root:password option provides the credentials that MariaDB requires when syncing the initial state of the cluster to newly joined nodes. Change the password to reflect the password you set earlier.

Remember that if you have SELinux enabled, it is at this point that the SELinux Troubleshooter should be notifying you of issues. If not, make sure you have restarted Centos to allow the troubleshooter to catch violations.

Joining the cluster

With the first node up, run this command on the second node to join it:

mysqld --wsrep_cluster_address=gcomm://192.168.0.1 --user=mysql --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_sst_auth=root:password

The only change here is the --wsrep_cluster_address=gcomm://192.168.0.1 option. We have assumed that the first node has an IP address of 192.168.0.1, and have requested that this second node connect to the first node.

NOTE
One way to avoid editing my.cnf in order to remove gcomm:// is to start all cluster nodes with the following URL: gcomm://node1,node2:port2,node3?pc.wait_prim=no&...

(note the 'pc.wait_prim=no' option which makes the node to wait for primary component indefinitely.) Then bootstrap the primary component by running SET GLOBAL wsrep_provider_options="pc.bootstrap=1"; on any one of the nodes.

Testing the cluster

At this point you should be able to make changes to the database tables on one server and see them reflected on the second server. Your cluster is now up and running!

Making the changes permanent

All the configuration has been defined in command line options, which means that if one of the servers went down, it would not reconnect to the cluster.

To save the settings we supplied on the command line, edit the /etc/my.cnf file, and add the following lines to the end of it. This configuration allows the first node to reconnect to the second.

[mariadb]

wsrep_cluster_address=gcomm://192.168.0.2

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_sst_auth=root:password

wsrep_node_address=192.168.0.1

log-error=/var/log/mysql.log

The wsrep_node_address option defines the IP address of the local host. This is optional, but when I ran MariaDB on my VM, it reported that it could not automatically determine the IP address, which is why it is hard coded in this option.

The log-error option defines the log file that MariaDB will write to.

All the other options are exactly the same as what was supplied on the command line.

With this configuration in place you can kill the mysql instance that was started from the command line. The second node is still up, and so the cluster is still up. The first node can then be restarted with the command:

service mysql start

This will start MariaDB and reconnect it to the second node, and thus to the cluster.

We can then follow the same process for the second node. Of course you'll need to change the IP addresses from those listed above. The configuration to add to the /etc/my.cnf file for the second node looks like:

[mariadb]

wsrep_cluster_address=gcomm://192.168.0.1

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_sst_auth=root:password

wsrep_node_address=192.168.0.2

log-error=/var/log/mysql.log

All we have done is changes the node it is to connect to, and the IP address of the local host.

The mysql instance started from the command line for the second node can then be killed and restarted as a service. It will then rejoin the first node.

Restarting the cluster

As long as one node is up, the cluster will remain active, and the node that is down will have something to connect to when it starts up. But what happens when all nodes go down?

At that point you'll have to restart the cluster on the node that has the version of the database you want to use. So if node 1 went down, and then a day later node 2 went down, it is most likely that node 2 will have the copy of the database you want to keep, and so node 2 will be used to restart the cluster.

Restarting the cluster is best done from the command line. Since all our settings are now stored in the /etc/my.cnf file, restarting the node as the initial node in the cluster can be done with the following command:

mysqld --wsrep_cluster_address=gcomm:// --user=mysql

Just like before, an empty address in the wsrep_cluster_address option defines this node as the initial node in the cluster.

The node one can then be started as a service, and it will connect to node 2, and the cluster will be running once again.

You probably don't want to have a terminal window open with node 2 running, so once node 1 is running you can kill node 2, and restart it as a service. Node 2 will connect to node 1, and the cluster is back in action.

Troubleshooting

Name Resolution

If the node trying to connect to an establish cluster is reporting the error:

130307 15:14:52 [Warning] IP address '192.168.0.1' could not be resolved: Name or service not known

And the node it is trying to connect to is displaying the error:

'wsrep_sst_mysqldump --user 'root' --password 'password' --host '192.168.0.2' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --gtid '4c754641-e45a-11e2-0800-425dfc14f8f4:390'' failed: 1 (Operation not permitted)

The try adding the following option to the nodes configuration files:

skip-name-resolve

Be aware that by doing this, MariaDB will no longer use credentials with host names. That means you'll have to configure the password on root@127.0.0.1 instead of root@localhost.

Credentials

If the node trying to connect to an establish cluster is reporting the error:

WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.

And the node it is trying to connect to is displaying the error:

[ERROR] WSREP: Try 1/3: 'wsrep_sst_mysqldump --user 'root' --password 'password' --host '192.168.0.1' --port '3306' --local-port '3306' --socket '/var/lib/mysql/mysql.sock' --gtid '4c754641-e45a-11e2-0800-425dfc14f8f4:420'' failed: 1 (Operation not permitted)
ERROR 1045 (28000): Access denied for user 'root'@'192.168.0.2' (using password: YES)

Make sure the account on the node trying to connect to the cluster is correct. The established node that is being connected to is trying to send the database state to the connecting node, and is failing to do so because of an authentication error.

Rsync

If you see the error on the established node of

rsync: failed to connect to 192.168.0.2: No route to host (113)

It means that the default state transfer method is Rsync, not mysqldump. To use mysqldump, add the setting

wsrep_sst_method=mysqldump

Or you can open firewall port 4444.

Rsync Wan

If you see the error

sh: wsrep_sst_rsync_wan: command not found

in the log files of the node trying to connect to a cluster, run

cd /usr/bin
ln -s wsrep_sst_rsync wsrep_sst_rsync_wan

File Access

If you get the error File '/var/lib/mysql/aria_log_control' not found make sure that the file is owned my the user mysql. I ran into this issue running mysql_install_db as root, which meant that the file was owned by root.

Tips and Tricks

Initial Setup

You can find out how large your SQL database is with the command:

SELECT table_schema,
sum(data_length) / 1024 / 1024 "data",
sum(index_length) / 1024 / 1024 "index",
sum( data_length + index_length ) / 1024 / 1024 "total"
FROM information_schema.TABLES
GROUP BY table_schema;

This value includes all the indexes, and represents the amount of data MariaDB will transfer to an empty node when it joins the cluster.

To save yourself some time, initialise the nodes that will join the cluster by dumping the SQL data into a file with the command:

mysqldump -u root -ppassword --all-databases > all-database.sql

And then restoring it on the node to be connected with

mysql -u root -ppassword < all-database.sql

Because the backup will not contain the indexes, and can be compressed, you'll find that you'll be able to get a remote node up to a relatively recent initial state far more quickly than attempting to sync the entire database through the MariaDB state transfer. In my case a 5GB database could be backed up into a 600 MB compressed SQL file.

It doesn't matter if the database changes between when you backed it up and when the node connects, because the state transfer will take care of that for you.

Save Bandwidth

Use the setting

wsrep_sst_method=rsync_wan

to save some bandwidth for state transfers over a WAN connection.

Emscripten - the saviour we have been waiting for?

2013-06-24T07:45:00.000+10:00

Turn the clock back a few years and listen to the promise of this new language called Java. "Write once, run anywhere" was going to change the way developers wrote applications. It didn't matter what OS, browser or device you were using, Java was going to be there.

It didn't quite work out that way. Most mobile devices don't run the garden variety of Java (Android certainly relies on Java, but much to Oracle's disgust, it is not their Java), I haven't seen an applet based website in years, and even desktop Java applications are hard to come by if you are not a developer.

It was JavaScript that won the battle for the ubiquitous development platform. Speed and functionality still varies between devices, but if you want to write code that at least has a chance of running everywhere, JavaScript is your best choice.

The problem with JavaScript is that it lacks a lot of the libraries that other languages have had years to build up. This void is something that emscripten hopes to fill by giving developers the opportunity to compile their favourite C/C++ apps into JavaScript.

I recently used emscripten to compile two XSLT libraries into JavaScript. XSLT on the client is quite limited by a number of longstanding and low priority bugs (https://bugzilla.mozilla.org/show_bug.cgi?id=69799 and https://code.google.com/p/chromium/issues/detail?id=8441 combine to make any serious XSLT transformations impossible with JavaScript), so I had high hopes that I emscripten would allow me to run either xsltproc or xalan on the client.

The good news is that I got both application compiled and running in the browser. The code for xalan can be found at https://github.com/mcasperson/xalan.js and xsltproc can be found at https://github.com/mcasperson/xsltproc.js.

The process for getting both to compile was quite manual and time consuming. You are building in clean room environment, which means you have to manage all dependencies beyond the few provided by emscripten (like libc or sdl). I imagine the standard libraries provided by emscripten will grow, but at this point it is up to you to know what your application depends on and include it in your build environment.

I also found that optimization was a case of start with everything and work back until you find what options break the application you are compiling. Neither xsltproc nor xalan would work with the recommended setting of -O2, and removing the optimizations one by one, recompiling and testing was very time consuming.

And unfortunately the end result was not all that useful: both applications resulted in 20 MB+ JavaScript files; a bug in chrome (https://code.google.com/p/chromium/issues/detail?id=252492) prevented xsltproc from running in a web worker; Firefox was slower than Chrome when running xsltproc by a factor of aruond 6; and xalan took upwards of 90 seconds to transform a simple XML document (compared to the less than 1 second for the native build).

As cool as emscripten is (take a loot at some KDE applications running in the browser), it seems that it will be some time before it is possible to take any existing C++ application and have it compiled reliably into JavaScript and run consistently in the major browsers.

The day I left "mainstream" news forever

2013-03-21T07:57:00.001+11:00

news.com.au (NCA) is a news website run by News Limited (part of the same group of companies now well known for hacking famous voice mails), and was my last tie to the world of traditional news.

It was a little guilty pleasure. Their focus was solely on catchy headlines and simple pieces of reporting that worked well for those like me looking for a five minute break. And their penchant for attaching bikini shots to any story even remotely related to the human body was also quite entertaining.

Then one day NCA decided to fill the space that bordered the central main content column with ads. That in itself was not unlike a lot of big web sites. But they also chose to open the ad when you clicked anywhere in that border, even if you had scrolled down well past the actual ad content and the border just displayed white space. It was a devious tactic, as I found that I usually switched back to a browser window not by clicking the window's title bar, but by clicking in some white space in the page that was being displayed.

After being caught out more than once with a flashy pop up ad, I decided to remove NCA from my browsing vocabulary. And with that, I'd like to welcome news.google.com (NGC) into my life. Yes Rupert, you are actively pushing people into the arms of your arch nemesis.

It was something of an adjustment. NGC presented headlines that I had to think about. There were no boob shots. I couldn't read the first two paragraphs of the stories and safely ignore the remaining text as SEO filler. And I feel so much better for it.

So it is with no great sadness that I say goodbye to traditional media, and fully embrace the brave new world of content aggregation and tick'n'flick news consumption. NCA, you shall not be missed.

Ubuntu phone - the first truly unified platform?

2013-01-03T11:22:00.000+11:00

The Ubuntu phone OS has been announced, replacing the old "Ubuntu for Android" initiative that seems to have been all but abandoned. Frankly I'm not surprised that Ubuntu for Android didn't gain much traction, as the concept had a number of limitations that made it a burden instead of adding value. But the Ubuntu phone OS is another story.

From the information made available in the Canonical press release, it seems that the Ubuntu phone OS is a fully fledged version of Ubuntu that can be installed directly onto a smartphone. It is a single set of apps and single configuration that presents a tailored interface depending on whether your phone is docked or standalone.

It's no stretch to say that the current computing experience provided across desktops, laptops, tables and smartphones is disjointed at best. What you do on a particular device is still determined more by the OS it is running and the configuration that you have taken the time to (re)implement than the practicalities of the device itself. Try to do you everyday tasks on a portable device (even with a dock) and you'll quickly find that compatibility between common file formats is spotty, access to corporate networks is difficult, most of the apps you use for work can not be installed on your portable devices, and even synchronising common settings like bookmarks and passwords is tiresome.

The real value of a platform like the Ubuntu phone OS is not so much that it allows you to have one device, but that it allows you to have one configuration. Whether you are on the go or at your desk, everything is consistent: the data you access; the apps you use; the toolbars, icon placements, keyboard shortcuts, templates, favourites and shortcuts you have configured in your apps and the commands you run are the same.

I'm not saying that all your favourite Linux apps will be a pleasure to use on a smartphone interface. Just because you can install LibreOffice, GIMP or Eclipse doesn't mean you'll be able to use them without a keyboard, mouse and a large screen. But the issue here is more that it is simply not practical to type up a document, edit some graphics or do some development on a smartphone rather than the fact that the word processing app on your phone is not compatible with your document's format, that it can not access the network drive that contains the image you want to edit, or that a compiler simply does not exist.

Finally we are getting to a point where the physical limitations of the device you are using become the primary factor in deciding what you want to do on the device. This has been one of the selling points behind Windows 8, although I don't expect to see Windows 8 (not phone or RT) running on a smartphone any time soon.

For the Ubuntu phone OS to be a roaring success it does need to do a couple of things:

Provide images for current phone hardware. Canonical needs to exploit the legion of fans it has and the phones and tablets they already own to get a grassroots movement started. Some kind of a website where you can register interest in the Ubuntu phone OS and the mobile hardware you already have would be a great start.
Allow Android apps to run in Ubuntu. Bluestacks proved that it can be done on Windows, and Canonical was working on something similar (although the effort appears to have been abandoned).
Promote touch friendly apps in the Ubuntu marketplace.

Canonical has a real opportunity with the Ubuntu phone OS to provide a genuinely unified experience, and I for one look forward to seeing it in action.

Windows 8 isn't so bad, but the damage has been done

2012-12-02T10:21:00.000+11:00

I have used Windows 8 for a couple of weeks now on a desktop PC, and I have to say that I don't know what all the fuss is about. The start menu has been been replaced with a start screen, but once you get past the fact that it now takes up the whole screen, the functionality is pretty much the same: you organise your apps, search for them, and launch them. The charm bar is a little awkward on a multi-monitor setup, but I only use it to shutdown the PC, so it's not a big problem. The selection of Windows 8 apps is uninspiring to say the least, but since so many of these apps are just pretty wrappers around web pages that work fine with a large monitor, keyboard and mouse, I don't really feel like I am missing out on anything.

The real problem for Microsoft is not that Windows 8 is bad at what it does, but that the rise of Apple and Android have dispelled the notion that Microsoft is the PC.

For the majority of people, there was no distinction between the PC and Windows. Macs were in people's peripheral vision, Linux was unheard of, and the concept of not using Windows on a PC was like the idea of not using a steering wheel in a car.

Then came the iPhone, followed by the rise of the tablets. And suddenly everyday people were breaking the link between Microsoft products and "computing". Email no longer means Outlook. Web browsing no longer means IE. Word processing no longer means Word. People now realise that they could use a computer in an enjoyable and productive way without relying on Microsoft applications and operating systems. So much so that it gave rise to the idea of "bring your own device".

Microsoft held the mind share of its users to a large degree because people didn't realise there was an alternative, or had no reason to switch even if they were aware that alternatives existed. Apple (with Android riding shotgun) not only made people aware of an alternative, but also made people want to switch.

And with that, the Microsoft spell was broken.

The downside to having a software ecosystem so intricately entwined is that once the dominoes start falling, they don't stop. If you have no need for IE and Office, you have no need for Windows (Windows RT especially). If you have no need for Outlook, you have no need for Windows Server. If you can bring your own device to work, you have no need for a desktop.

I'm not suggesting that Microsoft is going anywhere. But it will have to work hard to reverse the increasing notion that its products are something that you are forced to use, as opposed to the products that you want to use.

Increasing Ubuntu Wubi disk image

2012-10-19T11:22:00.002+11:00

If you have tried Ubuntu using Wubi, you may have noticed that the maximum disk size you can specify is 30GB. There are a number of ways around this limitation, but the easiest is to resize the disk within Windows using the resize2fs application that is bundled with the Wubi executable.

Open up the wubi.exe file with 7ZIP, and extract the resize2fs.exe and cygwin1.dll files from the bin directory to a convenient directory. Then run the command

resize2fs.exe -f d:\ubuntu\disks\root.disk 70G

Substitute d:\ubuntu\disks\root.disk with the location of your disk, and change the 70G to the size that you want.

Free yourself...

2012-10-11T17:06:00.004+11:00

The network is the computer. It was a visionary statement, but one that is yet to really play out.

If the network is the computer, then the computer has been a slow and unreliable device that can only be used inside metropolitan areas. It wasn't that long ago that dial up was the standard way to connect to the internet, and even now high speed mobile internet access is spotty at best. You can't even expect to get a reasonable connections at major inner city hotels. I remember spending 10 minutes trying to establish a VPN and download my emails from a business oriented hotel.

However, things are getting better. 4G is finally offering the kind of bandwidth and latency that you need to run real time network applications. ADSL and cable internet access is now common place, and rollouts of high speed fibre networks are slowly spreading across various countries (unless you happen to live in South Korea, which has led the way for high speed internet access).

The incremental improvements in internet access have been matched by an ever expanding variety of computing devices. It is not uncommon for a household to have a Windows or Mac desktop, an iPad and an Android phone. These devices still tend to fulfill different roles, but what if the network was the computer, and these devices simply became different ways to access the same information?

Of course, with browser based email clients and productivity suites, this is already possible to some extent. But web based apps still have a long way to go to match the functionality and compatibility of the desktop applications they seek to replace.

The holy grail for me has been to have a desktop that I can access anywhere and from any device. In practical terms this means having a remote desktop accessible from a browser.

My requirements were simple. First, the whole setup had to cost less that $500 a year. Since this would be a work expense that could be claimed on tax, this would amount to about $1 per day, which was a comfortable price point. Second, I needed enough computing resources to do my daily work. This meant at least 4GB of memory and enough CPU cycles to run around half a dozen applications simultaneously. Finally, I needed to be able to easily access the desktop through a browser using standard HTML5 technologies.

Surprisingly few VPS (Virtual Private Server) providers could provide the resources I was looking for with the budget I had allowed. I could have maybe come in under budget with Amazon EC2, as long as I shut everything down at the end of the day and was willing to use their spot pricing system. But that seems like a lot of work. Most other VPS providers would charge at least $100 a month for a system with more than 2GB, which seems ridiculous to me given how cheap RAM is these days.

After much frustration I finally found A2 Hosting. They provided Linux based VPS systems with 4GB of memory for just under $500 a year (as long as you use their coupon to get 5% off).

Getting the VPS setup was a breeze. It was pretty much point and click as far as getting the Centos 6 OS up and running, and from there you have root access via SSH.

I installed the Gnome desktop, and then installed VNC using the instructions provided on the Centos web site. Once that was running, I installed NoVNC and set it up as a service using supervisord. And that was it - a fully fledged Linux desktop that was accessible via a web browser.

Performance is quite good. The tasks I run on a daily basis are at least as fast as they were on my Lenovo T500 laptop, if not slightly faster. I could have used another 2GB of memory, but unfortunately 4GB is the limit on VPS services provided by A2.

Visually there is some lag when accessing the desktop through VNC. You can drop the color depth down to increase the responsiveness over slow connections, but there is a limit to what you can do to reduce latency on a server on the other side of the planet.

I also had to hack up a small Javascript bookmarklet to remove the HTML elements from the standard NoVNC login page in order to get true full screen desktop experience (the code is included below). NoVNC is designed to work with custom pages though, so if you are really keen you can just make your own NoVNC portal.

javascript:(function(){var overflow="hidden"; var display="none"; document.body.style.overflow = overflow; var hideElements = ["noVNC-control-bar", "noVNC_status_bar", "noVNC_screen_pad", "noVNC_logo"]; for (i = 0; i < hideElements.length; ++i) document.getElementById(hideElements[i]).style.display = display;})()

Unfortunately NoVNC doesn't work very well on Android devices. I had to install a virtual keyboard in Gnome to get any access to the system, since the native Android keyboard displayed by the NoVNC portal didn't seem to work.

Thankfully it is possible to connect to the remote desktop using a standard VNC client, and Android / iOS have plenty of VNC clients available. However, because A2 uses a non-standard SSH port, you will need some workaround to tunnel your VNC connection over SSH. The following command will establish a VNC connection to a non-standard SSH port (1234 in this example).

VNC_VIA_CMD='/usr/bin/ssh -p 1234 -f -L "$L":"$H":"$R" "$G" sleep 20' vncviewer -via user@123.123.123.123 localhost:2
On Android I used ConnectBot to establish a connect and forward port 5902, which allowed me to connect to the remote desktop with RealVNC (which is like $7 from the Play Store).

Overall I'd consider the experiment to be a success. I can log into my desktop from pretty much any device and pick up where I left off. It sounds simple, but being able to do some work at home on your desktop and then pick up exactly where you left off when you get into work and fire up your laptop is a huge time saver. It has also allowed me to squeeze some more life out of an old desktop PC, because it no longer does anything other than open a web browser or VNC client.

Looking forward, I can see that prices for VPS services will only come down while performance goes up. There is no need to worry about upgrading my own hardware to keep pace, and I have the piece of mind that the hardware is being maintained by dedicated professionals.

So if you are looking to rid yourself of the bond that an OS configuration on a specific device has over you, check out the A2 VPS services. If you are willing to work around a slightly laggy interface (and even that might not be an issue if you live closer to the A2 installation than I do), you can free yourself from the burden of being tied to a specific computer.

UPDATE
I found that MyHosting offers an 8GB Linux VPS for under $500 a year (as long as you use a discount code). I could really have used that extra 4GB... Maybe next year I will swap.