Irish Android

Building a choropleth map for Irish agricultural data

2018-06-16T07:01:00.000-07:00

I wanted to create a choropleth map using Irish data. The Irish Government has made a large quantity of data available at data.gov.ie. The set I chose was for Bovine TB across Ireland over a number of years.
You can get geojson for a variety of countries and Ireland's county boundaries are available. All I had to do was get the data labels to match. In the data some counties are split into 2 parts. I merged and summed these data points to get the choropleth to work with normal county boundaries.

You can see the results here. A link to the github is included at the top of the notebook.

Early Stopping with Keras

2018-05-06T04:31:00.001-07:00

Steps to train a model

There are a number of things you have to do when working on a Machine Learning problem.

Import your data and transform it as appropriate
Define your model using something like Keras - this includes your chosen hyperparameter values
Run the training phase
Evaluate the model

If at the end of this process the evaluation shows that your model is not sufficient for your needs you will need to make some changes and rerun the training.

Ideally you should have a way to spot that training is not going as you want and stop it. This is one of the uses of early stopping. The other is if your model has reached a sufficient quality standard for you to use and further improvements have slowed, stopped altogether or an unnecessary.

Early Stopping

Keras provides a way of doing this using the EarlyStopping callback. There are a range of callbacks in Keras and you can define your own. Details are here.

The EarlyStopping callback works by choosing a metric that you want to watch. Indicating what level of change in it is unacceptable and how many epochs you are prepared to tolerate that performance before the training should be abandoned.

Sample Code

First the import

from keras.callbacks import EarlyStopping

Then create an instance of the EarlyStopping class. In this case we are monitoring the accuracy. The metrics available to you are just those you defined when you compiled your model. patience is the number of epochs to tolerate a below standard change in the metric.

early_stopping = EarlyStopping(monitor='acc', patience=10, verbose=1)

Note that you can also define a min_delta, which is the amount of difference that the callback should notice, but the default of 0 is fine for our needs.

Finally add the callbacks parameter when you are fitting your model. Note that it takes a list so that you can pass other callbacks too.

model.fit(X, Y, epochs=50, batch_size=5, verbose=1, callbacks=[early_stopping])

So in this case if the accuracy of your model has not improved for 10 consecutive epochs, training will be abandoned. Setting verbose to 1 is a good idea so that you get a better message that the callback ended training. Otherwise everything will just appear to stop.

A more sophisticated use of this callback could adjust a hyperparameter and try again. However a better way to achieve this may be with grid search.

Preventing Mass Assignment Attacks in Spring

2016-03-25T08:07:00.001-07:00

If you have a wizard to collect data from users in a set of steps you can open yourself up to a mass assignment attack. OWASP as always gives great information on this.

How it works

You Bind an object to the model and you want to populate the fields in that object as your user progresses through your wizard. So on the first screen you might collect their name, second screen email and so on. Between each of these steps you perform validation on the just entered values.

However a hacker exploiting mass assignment can bind data from step 1 during a subsequent step just by putting the variable and value in the URL (assuming GET, although obviously this can be done with POST also).

So for the first step you are after this:

https://mywebsite.com/myapp/action?email=anthonynolan@somemail.com

And the second step, this:

https://mywebsite.com/myapp/action2?name=Anthony

Instead the hacker supplies this which is fine and passes our validation:

https://mywebsite.com/myapp/action?email=anthonynolan@somemail.com

Followed by this. Which is not fine as the email is not validated on this step:

https://mywebsite.com/myapp/action2?name=Anthony&email=bad_guy_email@dodgy.com

Because Spring binds all params that it can match in the URL the email address will be bound in the last case. Your code may not have included validation of email (restricted domains for example) in that step of the wizard.

There are a number of approaches to fix this.

Repeat the validations on each step

Therefore the validation is accretive. Something like this.

Step 1

Validate condition 1

Step2

Validate condition 1 again
Validate condition 2

and so on.

This leads to some duplication of code - but it is very minimal - only method calls. I plan to put together a neater way of achieving this accretion, but for now this will do.

Isolate each wizard step in its own controller

This will allow you to use the OWASP recommended method of using a binder (this is the whitelist version, there is blacklist too):

@Controller
  public class UserController
  {
     @InitBinder
     public void initBinder(WebDataBinder binder, WebRequest request)
     {
        binder.setAllowedFields(["userid","password","email"]);
     }
  
     ...
  }

However if you have the steps in the same controller (a better design in my opinion), this won't work. You will end up blocking the params at every step, not just the prohibited ones.

Blacklist params in handler

You can however use the Params attribute on your RequestMapping annotation to disallow certain parameters. This one will disallow role in the params list. If role is supplied the user will get a 404.

@RequestMapping(value = "/bitofurl", params = "!role")

All this is a little awkward and removes some of the elegance of a Spring solution. But mass assignment attacks are serious stuff and can result in all manner of nastiness. Better slightly less elegant code than a compromised user DB.

Encouraging people to do things with VR

2016-02-20T08:47:00.001-08:00

I read an article a few years ago about a study to see if people contribute more to their pension fund if they have seen a future representation of themselves. Not surprisingly they do. I have been bouncing the idea around in my head, so I decided to look up the study. You can read it here.

They used some clever software to produce a 3D model of a person's head which they then aged. The participant was then given a VR headset inside which they could see their older self. They could interact to a limited extent with themselves, so it was better than just a photo of 'you old'.

The experiments were very well run, but basically the outcome was that people can relate to their older self if they see the image and pension contributions go up.

I think this idea is generally fascinating, beyond pension contribution. For example encouraging people to adopt healthy lifestyles is very difficult. Often the pull of pleasure now far outweighs some notion of protecting your future self. If people could see themselves sick, fat and maybe with a few limbs missing they might rethink their choices.

Radical stuff, but really no more than letting people make their own minds up based on what they can see will happen.

Continuous Learning for Developers

2016-02-18T12:16:00.000-08:00

Continuous learning is very important in IT. A break from learning new things for a couple of years can put you in the dark ages.
I have always known this and done my best to keep upskilling, but a few things I have read recently have reminded me how important this is.

Leading Change

John Kotter's Leading Change is a book about managing big change in organisations. It gives a set of steps that should be followed based on the author's experience with change efforts in large numbers of big companies. The most surprising thing about the book is a section at the end where he says that the thing that unites very successful people is lifelong learning.

Big Data

O'Reilly publish a weekly Data Science newsletter. It is well worth a spin if statistics and analysis are your thing. I think that an interest in data is in every developer's interest.
They published an article this week entitled Is your development team ready for Big Data?
Again one of the key skills that a developer should have to be able to handle the challenges of Big Data is continuous learning.

Values in Action

A few years ago I read Authentic Happiness by Martin Seligman. The author had researched the things that make us happy. He analysed world religions to find commonalities which he called Values. Then he devised a questionnaire which determined which of these virtues were 'yours'. The idea being that if you concentrated on pursuing your virtues you would be happier. The book does a much better job of explaining this and gives solid reasoning too. You can find the survey and others like it here.
Why this talk of happiness and values in a technology blog? Whenever I do this survey 'Love of Learning' comes up at the top of my virtues. Lucky me.

Using Statistics to Hire Developers

2016-02-17T12:16:00.000-08:00

The Interview

I was interviewing a developer for a mid level position today and he turned out to be not up to scratch. The CV looked pretty good, but when I dug in with some fundamental Java questions he was found wanting. Not an unusual occurrence, but still costly. It took 30 minutes of my time, his time and a number of other people were involved to get the process to this stage.

I wondered if there might be a way to remove some of the candidate CVs from the batch before we do the costly part of interviewing.

Spam or Ham

Classifying CVs could be considered like the Spam/Ham problem. Apache SpamAssassin is a good example of a spam classifier. It can probably be made to classify CVs - or at least try, but it is written in C, I think and that is not my thing any more. This looks interesting, so I forked it, converted it to eclipse and started the process of making it build with Maven.

First I am going to try this out with spam and ham email, then will try it for other text passages. Maybe CVs if I can get my hands on some. I am not expecting a huge amount of success with this, but I want to try out some Machine Learning techniques in my work. I did this last year and don't want to let all that hard work be for nothing.

Java Interview Questions

If you are going to find yourself on the receiving end of an interview for a Java job take the time to lookup 'Java Interview Questions'. Spending an hour refreshing (or learning) these top items are likely to add considerable polish to your presentation. You can be good, but interviewers have a hard time seeing through rusty skills in an hour.

A Philosophy for Modern Applications

2016-02-16T13:18:00.000-08:00

I listened to a webinar with Pivotal today and they spoke about some of the thinking behind their Cloud Foundry platform. One of the guides they use to build this is called 12 factor.net. It is a set of principles devised by the people behind the Heroku cloud platform.

I won't go into the details here, you can read the brief web page yourself, but they cover a broad range of the factors that we often don't think about when we develop large scale enterprise systems.

One such concept is the avoidance of Software erosion. This is the gradual degradation of software running in a changing environment. For example as security holes in the OS are discovered your OS is eroded. A service on which you rely fails and is not restarted. If not managed continuously items like this can result in your application going out of service. The costs of getting it back may be too great to pay.

A related concept is Technical debt. This is work within a system which needs to be done in order for some other work to progress to completion. If you don't attend to the debt it grows as the cost of doing the work gets greater.

Taken together these items are reasons to factor continual maintenance into the costs of software ownership. Using techniques like DevOps in a managed cloud environment - which is what Pivotal were explaining in today's webinar is a way to keep these time growing problems under control for less cost.

Tips for passing Oracle Certified Associate in Java 8

2016-02-15T12:13:00.001-08:00

I decided to do this exam a few months ago. I have been a Java developer for over 10 years, so this was overdue.

There are a few simple steps and quite a lot of work.

First get a book

I used this one on Kindle. Kindle is the perfect format for this stuff as you can read snippets while you wait at the bus, checkout etc.
Read it through cover to cover. I did not do the exercises on this pass, I was just making sure that I could understand almost everything in there.
Next you should either do the end of chapter exercises, or write small programs to test your understanding. Don't start taking mock exams too early.

Mock exams

These are absolutely required to test your knowledge. I used these ones. I (like most of the people whom I spoke to about this) found I did not know the material as well as I thought. The huge plus for the Enthuware mock exams is that they explain in detail why you were wrong. Take your time understanding why you made the mistakes you did. Don't rush the exams. There are a limited number of them in the download and you can get used to how to answer that set without being prepared for the real exam.
The Enthuware software also builds up a picture of where you are weak and where you are strong. After a couple of mock exams you can check where you are weakest and concentrate on those areas.

The exam

Once you are passing by a decent margin you can go ahead and take the exam. There is plenty of advice on the immediate preparation for the test in the book I recommended.

Good luck!

A tool for plotting networks - Cytoscape

2014-10-26T10:17:00.001-07:00

The networks referred to here are more general than computer networks. Social networks for example. Cytoscape is a great free and flexible way to produce nice graphical output from 'linked node' data. On a recent project I had a requirement for this and Cytoscape did the job well. I would not describe the tool as all that easy to use, but the task it is carrying out is not easy. Once you take some time to familiarise yourself with the basic operation it works very well.

Full details are available here.

A nice feature of Cytoscape is the ability to tie a property of the data to a visual property. For example you can make the thickness of connecting lines very with a property of the relationship - weight. You could also use things like colour.

There is a large set of plugins available. Many focus on biology which as the name suggests is the typical domain for this tool, but it is not limited to that.

Cytoscape presents a steep learning curve, but this type of work was formerly the domain of experts only. Now amateurs can have a go too.

Population of Ireland on a 3d scatter plot

2014-09-19T01:44:00.002-07:00

The Central Statistics Office in Ireland has published this dataset:

http://www.cso.ie/en/census/census2011griddataset/

And here is the user guide:

http://www.cso.ie/en/media/csoie/census/documents/census2011griddataset/1,Km%C2%B2,Grid,dataset,User,Guide,2011.pdf

It gives the population of each 1km square of Ireland on census night 2011. It has various buckets, but I have just looked at total population. I used the 3dscatterplot package in R which is very easy to use.

Here is my code:

cen<-read.csv("COP2011_Grid_ITM_IE_1Km.csv")

install.packages('scatterplot3d')

library(scatterplot3d)

scatterplot3d(cen$CENT_EAST, cen$CENT_NORTH, cen$TOT_P)

and here is the resulting plot

You can see the big cities jumping up there. It needs a lot of polish to avoid things like Cork getting swamped in the 3d projection, but that's it for now.

Using the new pipe feature in R

2014-08-31T06:15:00.002-07:00

The magrittr package has introduced the pipe operator to R. It looks like this:

%>%

You use it the way you use any pipe. First operations result passed to next one and so on.
Makes for more readable code instead of nested functions.

To use it you need to install the magrittr package. In addition to get the nice easy to use filter, group_by etc. functions used below you need to install the dplyr package. In fact dplyr depends on magrittr, so install it first and magrittr comes along for the ride.

install.packages('dplyr')
library('dplyr')

You'll need the nycflights13 dataset for this example, so do this:

install.packages('nycflights13')
library(nycflights13)

Here is the example which points the finger at airlines with longest delays from NYC:

filter(flights, !is.na(dep_delay)) %>%
group_by(carrier)
%>%summarise(delay = mean(dep_delay))
%>% merge(airlines)
%>%arrange(desc(delay))

I really like these functions like the filter one. I find subsetting the 'normal' way in R to be v tricky. This makes is a lot easier - to my mind.

Additional stuff I stumbled upon while looking at and getting the nyc data were these bits:

Find the datasets in your installed packages:
data()

In a specific package:
data(package = 'nycflights13')

Here are the dplyr docs - 63 page monster pdf.

This is a bit more manageable.

magrittr github with nice examples is here.

Using R to analyse data from the Central Statistics Office in Ireland

2014-08-13T08:34:00.000-07:00

I started a Coursera course this week called Getting and cleaning data. I was looking at some data from the CSO and realised that I needed to clean it up. The course is good, but quite difficult. It assumes you have not forgotten everything you learned in the previous course, R Programming.

Mid way through week 2 (I am playing a bit of catchup) I stumbled across this package

http://pxr.r-forge.r-project.org/

px is the format that the CSO releases its 'raw' data in. This package puts that into a data frame which is more amenable to analysis. I reckon there has to be something lurking within the CSO dataset which is one of my main motivations for getting up to speed on R.

Nothing to declare yet, but hopefully I will find something interesting soon :)

Get a list of your functions in mysql

2013-09-16T09:08:00.003-07:00

If you want a list of functions (as opposed to procedures) use this:

select * from information_schema.routines where routine_schema = 'your_schema_name' and routine_type != 'PROCEDURE'

Douglas Crockford Video 5 - 'The end of all things'

2013-08-09T00:42:00.001-07:00

No let up in quality in this video. Here are my notes for the final video in the original series:

Cross site scripting (XSS) is a big problem. Huge privs accorded to a successful attacker.

Caja and adsafe - make js safer.

Don't confuse a variable and a value.

How does an object get a reference:

By Creation

By Construction

By Reference

David Parnas:

http://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf

Lazy programmers guide

http://www.youtube.com/watch?v=eL5o4PFuxTY

Keep performance delays below 100ms - provide some sort of immediate feedback.

Don’t fiddle with code. Measure first. Use PageSpeed

Arrays can be slow in older versions of ie. No hashmaps.

Don’t add unnecessary chrome. Takes time.

Don’t tune for quirks. Keep code clean and readable. Future versions of JS engines will be much faster. Your quirk optimisations may cause trouble.

jslint.com

Avoid global variables.

Avoid ++ - too easy to mess up
Use jslint

Douglas Crockford's JavaScript Video 4 - AJAX

2013-08-07T01:58:00.003-07:00

These videos are remarkable in the packed world of IT training videos in that they are clear and enjoyable to watch. The fourth instalment is about Ajax, but goes into plenty of detail that I didn't know about where the DOM came from and some info about the famous browser wars. Here are my notes so that you can see what is in there before you invest 90 minutes.

Markup languages

RUNOFF
GML - generalized markup language
SGML
HTML - simplified SGML
Latex

Angle brackets came from Scribe.

HTML

Does not fail on errors - allowed innovation. Otherwise the web would have frozen.
2 types of outlines - H1 - not nested and p type which are. Yuk.

CSS

Not modular - clashes can wreck your page.
difficult to manage selectors - classitis and iditis.
None of the browser vendors ever got it implemented!

The DOM

Brendan Eich - Netscape

Browser workflow

url -> Fetch -> cache -> Parse -> Tree -> Flow -> display list -> Paint -> pixels

Comments around script tags just protects users of ancient browsers from seeing the script. Don't bother with this.

document.write

Very bad. Don't do it.

For performance improvement of scripts

minify
gzip
Reduce number of script files (concat at deploy)
Use something like Chrome PageSpeed to test

Javascript uses camel case for style properties. CSS uses hyphens - incompatible with JS - in fact incompatible with most languages. Done on purpose. Source of annoying bugs.

InnerHTML

Nice and fast, but dangerous. Developed by ms - all browsers support it.

Always err on the side of understanding and clean code over performance unless performance is a serious problem.

Events

Bubble up through the DOM - use stopPropagation to deal with this.
Allows attaching of a single event handler to a container. The container then dispatches the event to the appropriate element. Faster to set up.

Use good speed testing tools - Chrome best.

Server vs browser

Neither side should dominate. A balance is the best. The server is not a filesystem and the browser is not a dope that just displays returned content.

Douglas Crockford's JavaScript Videos from his time at Yahoo

2013-08-01T09:05:00.001-07:00

He has now moved on to other things, but these videos are still around. They are like reading a novel - long form is still best. No sound bites here. Each video is over an hour long and there are 8 of them. I am only on the third at the moment, but am getting a huge amount out of them. If you are like most js developers and me you will be bludgeoning your way through whatever tasks you need to complete without knowing the details. JQuery et al insulate us from having to know this stuff right? Afraid not. There is no substitute for knowing the javascript in detail. These videos manage to do that.

Useful CSS Wisdom

2013-07-30T01:00:00.000-07:00

I came across this in the most recent Web Design Weekly. I have called it wisdom as it is full of the type of advice that people only have the balls to give you when they really know what they are talking about. Why IDs in CSS should be avoided is detailed. A lot of people rant about this, but there is something about this document that got me sifting through my CSS to find all the # marks.
The document is short and I am going to look back at it a few times over the coming months to make sure I have squeezed the benefit out of it.

Web Design Weekly

2013-07-29T01:09:00.000-07:00

Web Design Weekly is a great resource for front end developers and wannabes. Once you subscribe it arrives in your inbox once a week with a selection of articles, news, tools etc. that you can use in your work. It is just like subscribing to RSS but I find that getting news on a specific area like this once a week means that I set aside some time to at least browse it and find out what is interesting. The material is often of a high quality and where I don't manage to get stuff working it's probably down to me not persisting with it long enough.
If this is the type of work that you do I recommend signing up for this. The link is at the start of this post.

Mozilla Developer Network JavaScript Resources

2013-07-26T02:01:00.000-07:00

When I am looking for anything to do with js, html or css on the web I tend to put the letters mdn into the search line along with whatever else I am looking for. This usually pushes me to the pages on the Mozilla Developer Network. They are consistently of high quality - unlike for example W3Schools and are clearly written by the type of people who possess the knowledge that I want.
Mostly it is just a dip in to find our a syntax here and there, but they do offer comprehensive book style start to finish information. The JavaScript Guide for example is here.
I have been programming in JavaScript for quite a while now, so I am not reading this line for line, but a scan through it is clearing up some long held confusions for me.
I reckon getting proper skills into your head and hands is a slow process and the way there is to persist and add new pieces every day. Going through the MDN docs is an example of this.

Book about Design Patterns in JavaScript

2013-07-25T04:49:00.001-07:00

This book by Addy Osmani goes into depth about the various design patterns that you can use in your JavaScript. The main advantage that I am getting out of it is going through the examples and reading the references to see how to do 'complex' JavaScript. Singletons always come in handy in Java, now I can do them in JS. A host of other patterns are covered. I will be taking the time to go through this book in more detail over the coming weeks.

Some Resources to Help You Get Started With Grunt.js

2013-07-24T03:15:00.004-07:00

I have been a bit worried about deploying cleartext javascript to my production environment for a while now. The hassle of obfuscating always put me off:

Use something like Google's closure service
Change all of the references in my html to point to the generated files
Make sure the clear text dev versions of the js are not deployed and finally
Do the deploy

Grunt.js looks like it might be my saviour here. Having come from (and still being in) the Java world I think of this as being Ant for javascript. Just what I was after.
I am just doing a bit each day so as not to interfere too much with the day job, but so far I have added these tasks (in development at least, I am not doing anything with grunt on live code yet):
Remove white space from js files with uglify
Compress javascript to gzip
Run my existing QUnit tests
Lint my javascript

Although it is early days this looks like having the scope to make everything a bit easier.
Here are some resources I have used or am in the process of working through:
http://blog.mattbailey.co/post/45519082789/a-beginners-guide-to-grunt
http://flippinawesome.org/2013/07/01/building-a-javascript-library-with-grunt-js/

As usual in things front endy Addy Osmani's website is a useful resource.

Feature removal from mysql following Oracle takeover?

2013-07-22T01:00:00.000-07:00

I am using MySql 5.5.20 (select @@version). There does not appear to be any difference between explain and explain extended in this version. See here for the expected output from extended. I don't know if this is a feature removal that happened after Oracle took over. They seem to be pushing people to buy the enterprise tools. Not an option for some (me included) as I am using Google's Cloud SQL service. I don't think it allows this type of monitoring and even if it did it is unlikely that I have access to the knobs to turn to make good on its recommendations.
My experience of enterprise monitoring is limited as I mainly do dev, not admin. I am usually not around when the dust settles and people start to worry about stuff like monitoring.

I was at a mysql talk given my Oracle last year in Dublin. It was an impressive morning spent explaining the things they had improved in mysql since the purchase. They did not mention removing any handy admin features which might push someone to buy Enterprise Manager, but it would not surprise me.

Falling off a performance cliff with group_concat in Google Cloud SQL

2013-07-19T09:05:00.001-07:00

I started to write this article as the performance of some of my mysql queries had gone off a cliff. This was related to my use of the GROUP_CONCAT function. This rolls up a set result set and shows the passed in row values concatenated. Handy for reporting, but not much else. In the highly memory constrained world of Google Cloud SQL using this on a well indexed set of tables (albeit it with multiple left outer joins in place) pushed a regularly executed query for a few hundred milliseconds to 6 to 9 seconds. Disaster.
I pulled the db over to my dev machine (copy of prod) and ran the query there. No problem. I am concluding that the GROUP_CONCAT was the bad guy here. I found another way to achieve my goal, but this cost me an afternoon and a lot of fretting before I got out of the woods.

MySql is not bad, but occasionally it bites.

Good Video on Creating Large JavaScript applications

2013-07-18T00:35:00.001-07:00

My JavaScript application started out small, but is now getting very big. The main js file is about 3k lines long. Performance is still very good, but I am getting concerned about maintainability. I had been stumbling around in the alphabet soup of javascript tools and libraries for a while and was looking for somethign which would make sense of this.
I found Addy Osmani's video on structuring large JavaScript applications very good. I have just watched the first part (23 mins) so far, but it is pitched at my level, so looking forward to the next ones.
The video is here:
http://addyosmani.com/scalable-javascript-videos/

Addy's slides are here:
https://speakerdeck.com/addyosmani/large-scale-javascript-application-architecture

He mentions at one point that he is writing a book on backbone.js development. Here is a link to the book
https://github.com/addyosmani/backbone-fundamentals
Its free at this link, but you have to build. I will give this a go at some point. You can buy the book from O'Reilly too.

He mentions Angular.js a couple of times. I am going to take at look at this next. There is a video on Angular at this year's IO:
http://www.youtube.com/watch?v=HCR7i5F5L8c

Addy goes through the different types of architectural models MV* an all that.

This is definitely giving me a steer on putting some structure around some rambling javascript code.

I don't want this post to be just a rehash of what Addy said, but I hope the links I have included will add something. The main thing though is to watch the video!

Good Design Learning Resource

2013-07-16T07:32:00.001-07:00

I signed up for HackDesign about a week ago and they sent me an email with some links in.
It took a bit of time to get through it all, but it was worth it. I am always learning new bits of stuff - CSS, Java, SQL you name it, but am rarely satisfied that I am following the right 'path'. Not being thorough enough. Resources like HackDesign are the solution to this I reckon. Nice and deep view of a subject - design in this case. There is enough scope for you to click plenty of links and generally learn by doing something other than just reading.
These Yahoo design patterns for the web were a useful find. I found them through one of the links in one of the articles. And this is only week one.

Now to find similarly deep resources on the other things that I am learning.