Uncle G's Rant

So HBO delivered a present this week with the return of both...

Fri, 29 Apr 2016 22:23:59 +0300

So HBO delivered a present this week with the return of both Game of Thrones and Silicon Valley. In the later as a startup founder and developer I’m amazed that anyone outside the “bubble” understands what the heck the characters are talking. However especially hilarious was just prior to watching the opening episode of Silicon Valley I witnessed our team spend an entire lunch giggling over snapchat’s ability to transpose faces.

Apparently in the list of incentives to entice people to your start up the ability to a slap moustache on peoples faces is second to changing the world.

Fatherhood

Fri, 15 Apr 2016 09:01:49 +0300

My last post goes back nearly a year and pretty much coincides with not the birth of my son but when he started to have difficulties sleeping. Its been one huge effort to finally get him to the point where he will sleep through two nights in every three.

I’ve never been a good sleeper myself. No doubt not helped by adopting the “night owl” lifestyle from early adulthood that is rife with software developers. However the body’s ability to cope with sleep deprivation can only go so far and I’m painfully aware of how far its diminished as I’ve grown older. Trying to function at work and at home at the level of expectation that I put on myself has been impossible.

Therefore I’m in awe of parents that can balance home, work and professional development. I don’t know how they manage it.

I’m lazy so I want apt-get package installs for the init.d scripts but why oh why do 99% of packages...

Thu, 07 May 2015 15:24:44 +0300

I’m lazy so I want apt-get package installs for the init.d scripts but why oh why do 99% of packages start any service daemon by default? Don’t friggin do that!

Upgrade time

Thu, 26 Feb 2015 14:07:12 +0200

I’ve finally had to bite the bullet and upgrade. My current Mac Book Pro is being returned and I’ve been given a new one as part of a company wide upgrade. I’m usually very slow to apply software upgrades these days because of the inevitability of it breaking something in my development environment.

It seems even trivial maintenance patches always trash my python virtual environments. I develop using a mix of IntelliJ, Sublime Text and Vim. IntelliJ for Java and Python. Sublime for html and Javascript and Vim when wanting to do anything that is short and quick. The big catch ya with Yosemite is that it doesn’t ship with a Java runtime.

Realising this I thought “no problem” Cask for Homebrew (http://cask.readthedocs.org/) will get that solved in about 10 seconds. Now I do vaguely remember IntelliJ popping something up mentioning Java 1.6 but who pays attention to installation instructions? Two days of random application crashes later there is no escaping it… RTFM.

When I say random it wasn’t IntelliJ that was crashing most of the time, it was other things. Start IntelliJ bang Chrome blows up… hmm weird. Start IntelliJ later… bang Evernote blows up WTF? Finally IntelliJ stopped launching at all and I had to take to Google. I found postings suggesting altering the Contents/Info.plist file and warnings not to do it… tried it didn’t work.

The only thing that worked was installing the Apple JDK 1.6 as advised here.

AFK

Mon, 26 Jan 2015 12:25:43 +0200

So I’ve been largely AFK recently having not only been tied up allot more with non coding activities within my company but also becoming proud Dad for the first time. I hope to get back on the posting band wagon soon as things return to some form of normality. As I write that I know I’m only fooling myself :)

My company is looking to hire a Lead Developer. If you know a...

Mon, 26 Jan 2015 12:20:23 +0200

My company is looking to hire a Lead Developer. If you know a software developer who has strong python and analytics skills then share these details.

Flat batteries included

Thu, 04 Dec 2014 17:31:24 +0200

Python date time handling is a mess for the unwary. Between the datetime, time, pytz and calender libraries how is anyone supposed to know where to begin? That someone hasn’t taken to them in the same way as requests to urllib2 is surprising. But to be fair to python allot of other languages suffer from the same problem in that timezones were an after thought to date time types.

What caught me out this week was the expectation that date time arithmetic in python was safe for standard time/daylight saving time boundaries. It isn’t… silly me for not reading the documentation in pytz correctly:

But since I needed to interpolate missing dates into a time series data set as milliseconds from the unix epoch I was in for a further surprise. When converting from UTC date times you need to use calendar.timegm and when converting from local data times you need to use time.mktime. When converting from any possible timezone then neither of these are going to work. You need to perform the conversion yourself:

However in doing so we have reintroduced the problem when crossing DST boundaries because python date time arithmetic doesn’t account this (remembering what pytz told us). You can see below that our times are no longer aligned to midnight.

Worse when going from ST to DST you will skip a day in the sequence. We need to manually correct again by adjusting for the DST.

Regular Expressions

Thu, 27 Nov 2014 11:27:06 +0200

There are a few things in computer science that you end up doing just infrequently enough that you have to relearn them every time. Regular expressions falls in that category for me. XSLT used to be number two but fortunately JSON came along to save us.

One day I hope to work with someone who can pull them out of thin air so I never have to go and read up on them again. This regular expression visualizer might help in the mean time http://jex.im/regulex/

EDIT 7th March 2015 : the excellent https://regex101.com/ site also does a good job

Apache Spark with a RDBMS

Mon, 24 Nov 2014 12:48:29 +0200

Last week I spent a number of hours looking at Apache Spark. We currently have a problem with one of our computations at EzyInsights and I was interested to know if Spark could be a solution.

I’ll avoid getting into a description on what Spark is and isn’t instead I’m assume that you can read up on it yourself. But I’ll talk abit about all the things that tripped me up trying to get it to integrate with a relational database.

Spark is built with strong interoperability with Hadoop and allot of the other libraries tied to that ecosystem. At its core is concept of the RDD (Resilient Distributed Dataset) and this is where it gets difficult if you’re not sourcing the data for your RDDs via HBase, HDFS, Cassadra etc.

Sourcing from relational databases or other data sources is only briefly touched on in the documentation. Unless you want to use the new experimental Spark SQL then you need to use a Hadoop InputFormat derived type. Please note that Spark SQL provides sql like querying capabilities and allows you to construct typed RDDs but is not an interface to RDBMS sources.

I quickly identified that I could use the org.apache.hadoop.mapreduce.lib.db.DBInputFormat class but beyond that there was very little information on how to wire everything together in Spark. There was no other choice but to pull the sources for Spark and Hadoop and starting working through the code using a debugger. Especially as any misconfiguration results in extremely uninformative NullPointerExceptions right, left and centre.

If you want to listen (or attach) to the Spark shell and executor then you will need to specify the SPARK_JAVA_OPTS environment variable for your IDE’s remote debugging settings not JAVA_OPTS. You’ll probably be wanting to do this allot. There are some other options you will be supplying but I’ll come back to those later.

I was specifically looking at Spark’s Python language bindings (e.g. pyspark) since we use Python allot. In addition we’re trying to connect to an Postgres database. So lets look at what ended up working for me.

The newAPIHadoopRDD method of the spark context must be called to acquire the RDD instance.

In this instance the DBInputFormat class provides an implementation of the InputFormat abstract class. We will supply a number of settings to the DBInputFormat using the conf parameter.

When using the DBInputFormat type you have no choice but to specify the key parameter as a java.lang.Long. It took me a long time to figure out what was going on here. Embarrassingly so since the java doc on DBInputFormat actually says “DBInputFormat emits LongWritables containing the record number as key and DBWritables as value”. Its a pity this can’t be specified to what ever we like.

I ended up specifying java.util.List as the value type but was able to use a Map as well but this required a different org.apache.hadoop.io.Writable implementation and would presumably be more memory intensive. I’ll talk about the Writable soon.

The value converter is required as we must tell spark how to convert the Writable to the value type specified (e.g. java.util.List). Note since the key is a primitive and the LongWritable emitted by the DBInputFormat is a core Hadoop type Spark doesn’t require us to supply a key converter.

The conf parameter is required to supply to the DBInputFormat all the details on where, what and how it can obtain the data for the RDD. This can be defined as something like:

The name of the jdbc driver we’ll be using must be provided. Below I will show how we must specify the path of this to pyspark.

The org.apache.hadoop.io.Writable that we will be using by DBInputFormat to read the data is also required. Rather than supply my code see http://technologywithvineet.com/import_export_hdfs_rdbms/ for an example on how to write.

There are different ways of specifying the query but I ended up using these options since the DBInputFormat does something stupid with aliasing the table that breaks for Postgres schemas. See org.apache.hadoop.mapreduce.lib.db.DBConfiguration for the options.

Now if you just copied these settings into the pyspark shell they won’t work without modification. However you would still be missing the value converter to take a com.kampiki.analytics.spark.FooBarWritable and convert it to a java.lang.List. I had allot of trouble with this as I just couldn’t get it to work using a converter written in Java. I ended up creating a mixed language jar and writing my Writable in Java and my converter in Scala.

Now we’re getting close to being able to get a RDD from Postgres but we still need to know how to specify to Spark where to find the necessary Jar files for the driver and the writable and converter.

bin/pyspark –jars mywritableandconverter.jar –driver-class-path postgresql-9.3-1102.jdbc41.jar

Now everything should work… as long as your query doesn’t timeout.

Endless choice does not lead to simplicity it leads to complexity

Mon, 17 Nov 2014 18:52:02 +0200

Scala is a language I badly want to like but try as a might I can’t sell myself on it. I’m all for being succinct in code but offering endless choice of how you can express intent does not make a language simple.

.NET OSS

Fri, 14 Nov 2014 10:48:30 +0200

Its been a couple of years since I spent any significant time on .NET. So the recent announcement is not as ground shattering as it might be for those locked in Microsoft’s enterprise development eco system.

Announcing Open Source of .NET Core Framework, .NET for Linux/OSX, and Free Visual Studio Community edition: https://t.co/VlDKM9rFu7
— Scott Guthrie (@scottgu) November 12, 2014

While there is still allot of work they need to do the thought of one day getting the certifiable stack on top of Linux is appealing to me. I think C# is an awesome language that gets the OO functional hybrid approach far better than the likes of Scala.

Saltstack

Thu, 13 Nov 2014 22:44:00 +0200

So I’ve been spending allot of time recently working on our Saltstack states. I have a love hate relationship with Saltstack. Not of a love to hate variety but a genuine love it one day hate it the next.

I chose to adopt it as our deployment/configuration platform largely because:

The amount of antidotal stories I was hearing on the difficulty of Puppet and Chef
Its written python
I hadn’t heard of Ansible

The issue with SaltStack is that its not a trivial piece of software to have to learn. Of course it can be argued that deployment/configuration is not a trivial problem.

However when you want to maximise the amount of time you spend coding additional customer value into your product having to spend a number of weeks getting to grips with a piece of infrastructure software hurts.

Welcome Back

Thu, 13 Nov 2014 22:05:25 +0200

Its been about 9 years since I last decided to blog about anything. So since around 2005… oh dear feeling old. Back then I found it impossible to keep my posts short and hence I found that the whole endeavour was taking way too much time.

This time round I hope to keep it short and sweet with a focus less on trying to write informative articles and more like a daily update on what I’m looking into and working on. The kind of stuff I don’t want to bore my friends on facebook with and struggle to fit on twitter. We’ll see how it goes.

Its taken me nearly three years to come to the realisation that life in a startup is about doing the...

Thu, 13 Nov 2014 18:01:00 +0200

Its taken me nearly three years to come to the realisation that life in a startup is about doing the worst possible job you can get away with at the last possible moment.