CTOIsrael

As if I haven't written enough about HR

2012-02-05T12:26:00.000+02:00

Readers of this blog will know I have a huge axe to grind about bad HR, particularly here in Israel (here, here, here and here). Bad interviews, bad interview questions, not bothering to filter candidates properly. As my company grows and I start to look for new people, I strive to find better techiques for interviewing and hiring, that I have mentioned as potential solutions in my earlier posts, to getting that perfect match. I have just found Skills, a company that may go a long way to ensuring that the right CVs land on my desk. I look forward to using them the next time I am hiring.

I got to them from browsing the jobs on another site that I am excited about Gigantt. Not that I am looking for a job, when I seen new companies I always like to look at what kind of people they are hiring. It gives me a wider view of what they do and how they go about doing it. Gigantt looks like an interesting and very userfriendly project planning tool and maybe just what I need right now. Waiting to hear when I can start beta testing.

My new super duper server

2012-01-16T19:24:00.002+02:00

For the hardware geeks among you lets start with the specs.
Dell R810 platform
4x Intel Xeon E7-8837 2.67GHz 8 core processor
128GB RAM
6x300GB 10k RPM SAS drives (RAID5 using 1 as a hot spare)

This server is for running simulations. It is not serving up webpages, or in fact for any sort of internet application. I needed plenty of parallel power and this is what would fit the budget and requirements best.

So what did I do with it?

Before I start, one of the reasons I am writing this blog is to get input. If while reading this you see something that looks wrong, or a mistake, or that there is a better way to do it then feel free to drop me a line. Let me know where I have gone wrong, and feel free to suggest improvements.

This beast chews a large amount of electricity and makes a lot of noise. It also pumps out a huge amount of heat. My office is small so we elected to host it at a server farm near by and let them worry about all that. When it arrive I went down there and installed it. Spending a day in a server room is not fun if you don't have warm enough clothes, I can also recommend some fingerless gloves.

Annoyingly when I started to install the OS I noticed something wrong with the Harddrive space. I stopped the install and rebooted in the RAID management system. Someone hadn't seen my note about having one hot spare. I had to rebuild the RAID which took a while because the initialization took forever. Having sorted that little problem out I proceeded to install Ubuntu Server 11.10.

The simulation program that I have created is a written in Java and has a GUI front end for configuring the specifics of each individual simulation. That meant some sort of graphical window manager. I installed unity-2d ( I am pretty sure that this was a mistake). Once installed I added GNU Screen, the Oracle version of Java, Eclipse, Firefox, Webmin and FreeNX.

Reasons for my decisions:

Ubuntu
I am most familiar with this distribution, I use it at home and for other systems at work. I feel comfortable with it and under the time constraints felt I needed to chose something that required no time for me to learn. Debian could have worked too as they are pretty similar and both use apt. Given that they are similar I reasoned that there would be no difference either way, so I stuck to Debian. My Other reasonable option was CentOS. Not being familiar with that or with yum, I decided that I did not want to spend the whole day looking up how to install and operating system that I had never used before. What I should have done was done some test installations on VMWare instances on my old server.

Java
The simulations are optimization and testing of a live/ real time system that runs elsewhere, ie not on this particular server. The simulations are far more computer intensive that the live system. The original system that I created depended on a Java API. There were no other options at the time. As a result the software that I have written has remained in Java. As time has gone on we have moved away from the original API and now we primarily use QuickFIX/J. Another Java API. QuickFIX has other APIs but considering how far I had come with the software already written in Java it seemed like it would take too much effort on my part to rework everything into another language like C++. I picked the Oracle JRE as I have had some trouble in the past with the ICEDTEA version.

Unity-2d
Unity itself is a controversial window manager, Ubuntu created it as an alternative to Gnome3. As a change from Gnome2 the previous default window manager its pretty drastic, and there are many people left feeling that the choices that they used to have with Linux are disappearing in a world of constrained window managers. To be honest I do not agree, Ubuntu is aimed at the entire desktop market, their goal is to create an easy to use Linux desktop (or laptop) that you don't have to be a Linux guru to understand. There are options if you want them chose something else. I digress, I chose Unity-2d as there is only a standard graphics card in the server and I really don't need the full version. I believe this was a mistake. I should have opted for something even more lightweight and simple. I am not sure whether this will cause me any problems but I could certainly have used something else. I did however have to chose something. As I said the application that I will be using is controlled by a GUI written in Java, I therefore must have X and a window manager.
My major annoyance with unity is that when I have many windows open I just want a regular taskbar. The unity panel is annoying and can really slow me down. Simple solution to this was using gnome-panel. I now have a taskbar at the bottom which makes life much easier switching between running applications. There was another options tint2 but I prefered the more simple UI of gnome-panel.
For those of you wanting to know more about configuring unity in general have a look here. And more on using unity's features here.
Everything is working fine now as far as the window manager is concerned. I still have my doubts though, it is possible that unity is slowing things down a little. Possibly there is no escape while I need a GUI I must use something and this might be just as good or bad as any.

Eclipse
I use eclipse for my Java coding, so it is useful to have it available should I need to do something directly on the server. I don't envisage needing it very often at all but better to have it available just incase.

Webmin
I love Webmin, it makes many sysadmin tasks very easy. Additionally I have made it available over the internet so that I can view it from my office. I especially like if for configuring the firewall. Iptables can be a bit fiddly and Webmin makes life much easier.

FreeNX
I am using my FreeNX a free port of NoMachine NX server for my remote desktop session. The advantage of this compared to exporting the display is that it is easy to keep the current session alive, like most remote desktop software. I have had trouble using VNC on Linux in the past and I had heard about FreeNX before. Having searched around the internet I found that this was recommended. An additional bonus is that it opens a separate session to that which is viewable directly on the server. Instructions for Ubuntu installation can be found here. I have not been able to set it up avaiable over the web on a browser but there are instructions here. So far so good, configuration was easy and the NoMachine NX client for windows connects, disconnects and reconnects to existing sessions with no problems. The interface is nice and fast over my Internet connection and so far I am happy with the results. I can use my software in a nice full screen GUI over the Internet as if I was sat in front of the server myself.

Thats my setup. I will follow this post up with any gripes and problems that I have with my system as I begin to put it through its paces. Let me know if you have any questions about what I did what I installed and if you have problems with any of these bits of software let me know, maybe I can help.

Interview questions - some ideas and why they are helpful

2012-01-04T11:00:00.000+02:00

I have been away on vacation and I got some time to read, so I dug in to Black Swan by Nassim Nicholas Taleb. He is an excellent and thought provoking writer. I am reading this book extra slowly as I like to stop after every page or so and think about what he has written.

In the book he poses some interesting questions and I think some of them may be interesting to use as interview questions. So that I don't leave you in suspense I will put the questions here and allow you to think about them while I give an further introduction to what I want to achieve, which is not necessarily the point that Taleb was making.

A) If you had no restrictions on folding a piece of paper and could physically fold it 50 times. How tall approximately would the folded paper be?

B) Given the number sequence 2,4,6. Find the rule that determines the sequence. You give three three number sequences that I will answer yes or no to if then conform to the rule of the sequence.

C) If I flipped a fair coin 100 times and it came up with 99 heads, what is the probability that the next flip will be tails?

I have talked before about stupid interviews that waste my time as well as the time of the employees of the company that has been interviewing me. I should note that I have been asked interesting questions and puzzles in some interviews and one or two silly ones too. The one that springs to mind here is when I got asked if I could have any superpower what would it be. If you are an interviewer asking that question be damn sure you have a good interpretation of the responses to this question. Some people, even geeks, are just not in to comics and superheroes. My answer to that question was that it is very hard to give an answer that couldn't be sinister. Flying is probably pretty harmless, but mind reading/ control, invisibility, super strength are all things that could be used for bad just as easily as for good. I didn't think it was a good question, but I tried to answer it as best as I answer good questions, by showing that it was flawed. I then asked the interviewer the same question. He gave me an unsatisfactory answer. I should have asked him how he interpreted my answer, I don't think it made any difference to the result of the interview. I will concede here that if the future employer is a huge comic book fan as are the team that the position is for, then it may be a useful tool for assessing how good a fit it will be for the future employee.

Additionally mathematical and logic puzzles may not give the best metrics for good employees when used incorrectly. Know what you want to get out of the answers, it is not good enough to find a question on the internet and ask it hoping to determine that the answer will give and indication of whether the potential employee is capable of thinking outside the box or has above average logical reasoning skills.

Looking at the questions in a little more detail:

A) Firstly the answer I don't want to hear is that a piece of paper no matter what the size cannot be folded more than 7 times (This is the current accepted number for a regular piece of paper, though apparently a girl managed 12 times and came up with and equation to prove it). I am not testing knowledge, and no one likes a smart ass. Additionally I asked the question stating there were no physical restrictions.

I want to see the reasoning and thought process. First state your assumptions. A piece of regular printer paper is clearly not 1cm thick. I would accept the candidate making the assumption 1mm, though that is a factor of 10 out. It is much closer to 0.1mm, however I am not testing knowledge here just reasoning. Lets assume it is 1mm we can always divide by 10 to get the real answer later. It should be fairly clear that the answer is 50^2 * thickness. I do not expect the exact answer here. But I would like an order of scale. I have a short cut. 10^2 is 1024. So for every ten folds the height of the paper goes up by a factor of 1000, this means that the answer is 10^15 times the thickness. If the thickness is 1mm then the length is approximately 1,000,000,000 km. The real answer given the actual thickness is 100,000,000km, about two thirds the way from hear to the sun.

I have a confession, I am horrible at mental arithmetic. I could train myself to be better at it, but I guess I just got lazy. If I need to add something up accurately I usually have a computer, calculator or even a pen and paper handy. What I am good at is estimating. This is phenomenally powerful. The ability to this well and accurately enough can be more useful than just plugging the numbers into a calculator. If I look at some calculation and estimate the answer first, it is easy to see errors if I make a mistake when plugging the actual numbers into a computation device. I attribute this lesson to Mr Roger Hand my A-level physics teacher, thank you so much for this it has been an invaluable tool throughout my life.

I like to know the scale of big numbers. It puts things in perspective, it is not the most important thing here but you can expand on the estimation aspect by equating a large number with something that it represents. 100 million km really doesn't mean that much to me. The distance to the moon is 400 thousand km, so it is considerably more than that and as I said two thirds the distance to the sun. If someone said its about the distance to the sun, then that would be satisfactory. It's about the right order of magnitude.

Enough about estimation. The other skill this highlights is knowledge and understanding of powers of two. I honestly wouldn't care if the candidate had to write down the powers of two, as long as they stopped at 10. It should be obvious at that point how to short cut to the answer from there.

I will highlight why these two things are necessary for a candidate that I would employ. We deal with large numbers. Millions and tens of millions of dollars. We multiply that by fractions of pennies. If we are producing results that are factors of 10 out is should be obvious from an estimate done before that calculation. This is pretty critical as we are talking about real money transactions and position management. The powers of two should be second nature to any Computer Science graduate. We recently had to send out messages to a system in byte codes. I spent far too much of my time explaining the binary, decimal and Hex representations of these strings to my employee. It was immensely frustrating for both of us, but we got through it. In a world of high level languages, it is too easy to forget the roots of computing lies in bits and bytes.

B) What three sequences did you think of? 1,3,5; 8,10,12; 20,22,24. The answers would be yes, yes and yes. Your conclusion would be increments of 2. You would be wrong. That answer is clearly a possibility but not what I was looking for. You may argue that it fits your sequences. Of course so does the correct answer. You may get defensive, and say how was I supposed to determine that. To which I would answer that you must try to find a sequence that doesn't fit. Using the following 3 sequences: 1,3,5; 5,10,15; 6,5,3, would result in yes, yes and no. 1,3,5 confirms your initial assumption that the number increment by 2. 5,12,15 disproves that the increments must be 2 and that they must be regular. Finally 6,5,3 shows that the sequence cannot be descending. The answer is that the sequence must be ascending, and that is all.

It can only be shown by finding counter examples as well as examples. If you just go about proving yourself right with each suggestion you will get locked in a corner. A candidate able to simply solve this should be good a bug testing. A successful bug test is one that fails, any other bug test does not show evidence of a bug. This must not be confused with show evidence of no bugs. A semantic difference for sure, but a very important one.

C) This takes me back to school. The concept of a fair coin and a fair die. That means that it is not unfairly weighted, there is no bias. It should conform to probability. Of course if the question was I flipped a fair coin 5 times and it was heads. The probability of a tail would reasonably still be 50%. I would not expect anything else. However as the numbers grow the statistics should tend towards conforming to the true probabilities. I would reasonably expect around 60/40 one way or another for 100 tosses and maybe 55/45 for 1000 flips. To have 99 out of 99 flips come up the same, you would be a little foolish to assume that I was telling the truth about it being a fair coin. So OK, its a trick question of sorts, but it is more of a real world problem. I have been told something, but I have evidence to prove otherwise, now I must draw a reasonable conclusion. I am not trying to trick you, I am trying to get you to solve a problem of two conflicting bits of information. I want to here that it should be 50/50 if its a fair coin, but it doesn't seem to be a fair coin. Equally I do not want to hear that it's clearly going to be a 100th head. You would have ignored my statement that its a fair coin, and assumed that just because something has happened one way 99 times in a row then it is going to be the same on the 100th time.

Here we see that that real world problems do not always fit into classroom style examples. The world fits the normal distribution less than we think it does. It also gives an example of how probability and statistics fit together. A reasonable candidate should understand and show this in their discussion of the question.

I hope this has given and potential interviewers some ideas on what to ask and how to understand the answers they get. Please comment and let me know what you think about this, and if you have any other suggestions.

More on Passwords

2011-08-10T10:58:00.000+03:00

Following my last rant about passwords. Today's XKCD is right on the money. However it requires sysadmins to change their silly requirements about having between 6 and 8 letters, with one capitalization and one numeral. It is clear that the longer a password is the harder it is to guess by a computer. However if you pick words from around your office (like they do in the movies) you could be susceptible to a really good human guess.

Advantages of pair programming and or good code review

2011-07-19T17:00:00.002+03:00

I had a little problem with some code that went live an through a NullPointerException, causing various problems in the production system. It was far from a disaster but not great that such a thing should happen. Additionally The error was hard to track down to its source. When I did track it down I found the problem described in this post. As soon as I saw it I knew that was contributing not only to the issue but causing additional problems.

The key here was that in testing which I reviewed with the author of this code everything was OK. I would like to have unit tests for everything testing every possible issue. But we don't. Its not always practical, its not always easy to know the issues ahead of time. We should, I should insist on it. Would projects that are running late run later, most definitely. Would we have less problems once they went live, almost certainly. I just don't have the man power right now to get everything done, and remember I am not the boss. My superiors are all non technical. They want things done yesterday. They don't understand why it isn't ready just that it isn't. As long as I can rule out critical bugs it works in my favour to get things rolled out and running, and solve the minor issues as the come up. Its inefficient and its bad practice. Solution, if you are in the same position as me then get a bigger budget and hire and extra programmer or two.

With or without the extra man power there are two possible methods that can be used to avoid at least some of the issues that come up when going live with new code. Pair programming and code review. At my previous place of employment we used both to ensure as few mistakes as possible. Pair programming is an interesting method, because it requires one computer and two people. One is in the driving seat and the other sits next to them and contributes. Anyone familiar with Agile or Extreme Programming will be familiar with pair programming. When it works its great. The time lost by having to programmers work on one piece of code is gained back by having higher quality of code, that is easier to maintain, and with less bugs. Even things like the write / compile / test cycle is improved by having less syntax errors (due to them being spotted by the one not doing the driving). Ideas are generated and turned over faster. When it comes to maintenance or fixing the bugs that do get through having two people that worked on the code originally can be very helpful. If one is off on vacation or sick, or worse has left the company, the other is likely to be around. This extra ownership of the code is so helpful in these situations.

An old colleague of mine envisioned people rotating the pairs reasonably frequently, lets say one person worked with one pair in the morning and another in the afternoon. The more people you have the more possible pairs. This could mean even more that two people have some ownership of the code. Switching pairs around will give people the opportunity to take a break from each other, which is often needed in some cases, and to find really constructive and destructive pairings, to be either encouraged or discourage in the future. An additional benefit of pair programming is the lack of other distractions, you cannot be sitting in front of some code when working with someone and then suddenly check your facebook/gmail/twitter. You will certainly need more breaks, but you wont spend real coding time wasting it on the internet.

Where it breaks down is when you do not have many people it can still be done but without the variance of a large group. The number of possible pairings is n(n-1)/2 (triangular numbers), so in smaller groups this is limited. Additionally not all possible pairs work. I was often paired with weaker coders which meant I was doing most of the thinking. It was also frustrating for me not driving because the driver was working so slowly. In some projects I was working with one other programmer and we would spend some time working on the same code and other time working separately. The other coder would frequently get stuck and ask to do some work in a pair. This was code for either me working and explaining what I was doing to his code as I went along, or sitting behind him growing ever more frustrated as I told him what to do. That was a particularly nightmare case. Other pairs that I worked with were great. We thought alike, solved problems quickly together and saved time exactly in the ways mentioned above.

Code review is a lot more annoying but is a great final barrier before moving to production. I would often work with another coder on some code, then later go to someone more senior than me to be my pair for the installation. This meant that I had to explain all changes to the senior person, who would look over the code for any glaring errors, and understand the changes that I had made. Installation was made safer by having someone look over my shoulder making sure that I didn't do something stupid in production. The reason its annoying to do is again because it means taking the time of someone senior to go over something you know well again. If you want to get something in to production quickly it can be really annoying having someone asking you to justify every line of code. However in the long run this is the safest way to do things.

If you haven't tried pair programming, try it, even for a couple of hours a day.
Happy coding.

Passwords and Security (cont)

2011-07-18T10:54:00.002+03:00

And just like that here we have it
https://browserid.org/
an attempt by Mozilla at unifying your login. This is exactly whats needed. One login for all sites, cue Lord of the Rings reference.
There are many sites like banking sites that may not work with this straight away, especially as my bank likes me to change my password every 2 minutes. see previous post.
But with initiatives like this I can see things moving in the right direction.
Google goes a long way towards this, once you are signed in to Gmail you are signed in to all Google products. However that is limiting in that you may want to use something other than Google on the web.

Safe surfing...

Passwords and Security

2011-07-09T13:27:00.000+03:00

I have too many passwords, way too many. Its dangerous I am signed up to many websites, often using my email as a username. I am careful though not to use my the same password for my email as I do for the websites where my username is my emails address. http://bit.ly/rqYTu0 XKCD sums this up quite nicely. I am a big fan of using google account /facebook /twitter logins for other sites. This makes perfect sense to me. I only need one strong password for my gmail account and google with authorize my login to other sites. I really hope many sites pick this up, the internet will become a much safer place, though could force some people to get accounts with services they do not want. The other day I almost signed up to facebook as it was the only way to log in to a site. I didn't in the end so I never got to use that site, but doubt many people will be in this situation.

This was not the main purpose of this rant. There are some sites, mainly banking, and also at a previous company where you have to change your passwords every 3 months. I just think this is totally excessive. No one takes security seriously at this point. Every time I have to change my password one of two things happen. I forget my password and I get locked out or I have to write it down on paper and leave it next to my computer. Additionally in order to try to remember it I have to pick something easy to remember. I am not the only one that does this. That being the case the very process used to create more security is actually creating less security. So Sysadmins I beg you, stop this there are better ways to increase security. Insist on very secure passwords that never change or use something like and RSA secureID key. I admit that they are not always practical measures but at the same time they are better than this pseudo secure method of changing passwords every 3 months.

/rantover

JVisualVM - Java's hidden monitor and profiling tool

2011-07-08T16:39:00.000+03:00

There are a few different ways of profiling your Java application. One is using the -prof switch when calling running a Java app. This will output a profile file when the app is terminated. Another is a third party app like Yourkit. Yourkit is great, but you have to add a something to the commandline and after 30 days full free trial is its super expensive.

Finally in recent editions of the JDK there is jvisualvm.exe. Make sure you have the JDK and not just the JRE. locate the install directory and find the bin directory. There you should find the executable. Once you open it up you can go to tools and add some additional plugins. On the left hand side you will see the available JVMs that are connected. The good thing about this profiler/monitor is that you don't need to add anything to the commandline like you do with Yourkit. In your kit you have to add something to connect to that yourkit agent. Here it just connects and you can see all the JVMs running. Each instance of Java should run in a new JVM so you can run and view many different programs at once. It gives you a basic view of the memory and processor usage along details about calls to GC and the number of threads and classes being used. Additionally you can run a profiler to which will sample the application and give details of either cpu or memory usage. This is extremely useful in finding memory leaks or in my case optimizing your code. I do a large amount of number crunching which tends to use a lot of memory and cpu, using jvisualvm I have reduced the amount of memory used and made the whole process more efficient. There are other features that let you view information about classes.

I came across and a problem when running jvisualvm the other day. The executable ran and but it did not detect my running application. I ran an old version of my application and found that it was detected. In the end I have discovered that for some strange reason the java executable has to be running with code located on the same drive. Its very strange but seems to be the only solution I am aware of. If anyone else has seen this problem let me know, especially if you solved it in another way. As usual drop me line if you have questions that I haven't answered in this post.
Happy optimizing.

Excpetions - if you haven't caught on then you should try to

2011-07-06T02:54:00.000+03:00

I am sure this has been said before but I came across an error that occurred yesterday. I was told the code was working. For the most part I could see that in the staging environment it was working just fine. Sunday comes around. My install day. I check the staging environment. I find that there is a null pointer exception. The code crashed at a strange point. With some deduction I realize where the error is coming from. A method was called on an object that was clearly null, hence causing the exception. The object that caused the exception was instantiated on the line before. So no doubt there. I the method called on the line before returns the object in question so I look at the method, which happened to be in a different class. I see:

public SomeObject badMethod(InputObject obj){
try {
<code some of which that could cause an exception..
including declaration of SomeObject
and return of the instantiated object>
}
catch (Exception e) {
return null;
}
}

Two big problems with this, possibly three. The minor of all of these is that so much code is in the try block. I am not sure if this is wrong, but at the same time it doesn't look right mainly from a maintenance point of view. I had to delete the try and catch lines to find out which line of code require the exception handling, thanks to Eclipse that bit was simple.

I found out that the exception was actually a ParseException.

TIP #1: When using try catch blocks to catch exceptions, catch the specific exception. Always be more explicit when you can. Code is more understandable and you have the specific exception object available to you in the catch block.

The really big problem was how this was handled. Theoretically I have no problem with returning null from a method. However if you use a method that could return null that value must be handled. Sometimes it is better if the method throws an exception instead.

TIP #2: If the method could return null you must test for it. Otherwise you will end up with a nasty NullPointerException

In my case I ended up with an unhandled NullPointerException instead of a ParseException. I had no further information about this as well. This brings me to another point, at the very least you want to have the line

e.printStackTrace();

in the catch block. At least then you can track the exception properly. Better still use a logger, and add a little message of your own perhaps with some variable values printed so that you have an idea of what went wrong in your log or on your console. With most loggers you can add the the exception itself as an argument and the logger will handle outputting the relevant information from it.

TIP #3: Use a logger to print a sensible message from the catch block.

In this example the there was no real exception handling in the code. The catch block just returns a null which is not handled in the calling code. My solution in this case was to add a sensible logging line and throw the exception again. This way the exception is passed up the stack. The key here is that it forces the calling method to handle this exception. In this case that works, I needed an extra try catch block, but the error will be handled better. Now the parse exception will be passed up through the calling stack and the method will now handle this error correctly.

TIP #4: Handle the exception, its not enough just to print the stack trace or log the error.

Conclusion:
The tips I have presented here are very important when dealing with exceptions in Java. It is too easy, especially in Eclipse which does so much for you, to leave the printStackTrace() in there and not do anything else. But they must be handled. Learn to use them problem and exceptions will be your friend. Errors will be handled correctly and the code will run more smoothly and hopefully be a little more readable.

Happy coding.

Problem with reporting of disk space

2011-06-30T07:54:00.000+03:00

I ran out of room on a drive. It was reporting that there was 0 space left on the drive. I delete some files and then after running df -h it was still reporting 0 space left on drive. I found this
http://serverfault.com/questions/158524/disk-space-not-freed-on-ext3-raid1-after-deleting
which explains quite nicely the problem.

In short the solution was to type

sudo tune2fs -m 0 /dev/<partitionname>

and all was solved.

Start up pitches

2011-06-26T10:55:00.001+03:00

http://mashable.com/2011/06/24/startup-pitch-presentation/

Whether you are just getting started or about to raise capital this is a really good breakdown of what to present to investors. It is good to be thinking about this from the beginning. As you develop you company and get ready to present it might change, but by the time you are presenting have it down pat. And don't forget to have an elevator pitch too. Sometimes you might only get 30 seconds with someone, and you better have a good answer for the inevitable "and what do you do?" question. Get it right and make it count.

happy pitching

Automating password authenticated commands

2011-06-22T10:59:00.002+03:00

On my linux server I have an IPSEC VPN set up to an external connection. This was a stipulation from the counterparty to allow us to send secure data, from a verified user. I set that up before I started this blog, so at some point I will do a retrospective blog on that. For now I want to talk about something a little more general, I am just framing the problem.

I have a FIX connection running over this VPN, using the QuickFix/J API. For some reason that I cannot quite figure out I lose the session and it will not automatically reconnect. This is usually solved be resetting the VPN. I wrote a short script to take down and bring back up the connection, called restartVPN:

#! /bin/bash
sudo ipsec auto --verbose --down Test
sudo ipsec auto --verbose --up Test

This works fine. As you can see I use sudo to give me admin rights to do this. This is fine for a manual operation. However I have had this problem occur and not been notified quickly enough to do this manually without any consequences therefore I wish to detect when my session has been down for too long and automatically reset it.

This should be no problem I make a call from my code that is watching the session status to the bash script above and hey presto. However this will not work, sudo requires password authentication for non root users be default. A call from another system will get stuck waiting for the password and as its supposed to automated there isn't much I can do about that.

WARNING: Messing around with the sudoers file is dangerous and should be done very carefully and allowing only the least possible permissions to make your scripts run automatically.

The solution is allowing sudo to run ipsec without requiring a password to get the admin rights. Once done ipsec must still be run with sudo, so there is a small layer of security but the password layer is gone, so if you do type sudo ipsec make sure you know what you are doing.

Before you do this it is worth checking out the sudo and visudo man pages.

Bear in mind that my user is called ctoisrael and the command I am running is ipsec. You will need to replace this with your own specifics.

First run visudo to edit the visudo file.

~/sudo visudo

It will open in your default editor. Make sure you are familiar with editing in which ever you have chosen.
add a line like this at the bottom:

ctoisrael ALL = NOPASSWD: /usr/sbin/ipsec

the first ALL means for all servers, in this case it doesn't matter because I am only setting this on this server and not copying it to other servers. If I wanted to be specific ie that it can only be done on one server I would replace this first ALL with the name of the server. The rest should be self explanatory. I have specified that no password is needed when running the ipsec command, but only for this user.
Save the file. It will be default save sudoers.tmp. The visudo program will check to make sure it can be saved at the real sudoers file and then save it.

Now when I run restartVPN it does not prompt me for the password which means it can be run from a script that will automatically reset my VPN when it goes down.

I found a pretty good guide to setting up IPSEC VPNs if you are interested, here.

Feel free to write something in the comments if you have any questions about this or anything else I blog about, happy to help.

Fixing errors you cannot see and how I tried {to} catch (them){}

2011-06-14T22:49:00.001+03:00

For those of you following my twitter @ctoisrael then you will know I was bug fixing today. My system had a bug, the bug produced a tremendous amount of output. I can only presume that what ever went wrong produced some sort of uncaught exception. Uncaught exceptions are not written to the log, this is not usually a huge problem. I run my system using multiple "windows" on a gnu screen. If you use the linux commandline a lot, particularly remotely, this is one of the most useful tool there are out there. I will save screen for another post. I actually have it set to scrollback a large amount of lines, but today even 10000 lines was too much. There was just too much output. Anything output by System.out or System.err was gone for good. The log was not revealing. I could see that there must have been an error from a malformed log line. It didn't look right but I couldn't tell where it came from.

Some of the most critical errors in Java are NullPointerExceptions. They are not caught by default, and the IDEs obviously do not enforce you to surround code with try and catch blocks. So I am guessing that somewhere outside of my scrollback was a NullPointer or ConcurrentModification exception that was not caught and though did not crash the whole system, resulted in some malformed data that produced a glitch that got me in to a big mess, but thankfully not a costly one. The symptoms were resolved and no long term damage done. I am still not further on with the root issue.

I cannot find the root issue it is impossible, I checked the code where it could have happened but I cannot see anything wrong. I have been through the log, I can see the result but still not the cause. The next time this happened I must be prepared. In order to do this I have to catch all possible exceptions. This it turns out is not as easy as you might think. Ideally I need to catch all exceptions, handle them if necessary and write them to the log.

I have looked in to this in the past. I have found two suggested solutions:

Surround everything with try/catch blocks, catching Exception, so that all unhandled / uncaught exceptions will caught, and then handled.
Create a Class that implements Thread.UncaughtExceptionHandler and add it to the Thread

The try catch method is a simple solution but a flexible one. One must remember that every thread needs to have a try catch block around it. My suggestion would be to put it in the run method.

public void run(){

try {

} catch (Exception e){

}

}

This can be a huge pain to implement if your existing code has made different threads. Additionally I find that too many try catch blocks can look ugly and longer than it should be. The one advantage of this is that it allows the coder to have control over each potential exception in every place that it could occur. Clever use or more specific exceptions that can go uncaught can allow handling in different ways.

Despite this the second method is far better. I created an abstract class:

public abstract class Log4JUncaughtExceptionHandler implements Thread.UncaughtExceptionHandler {
private static Logger log = Logger.getLogger("log");
}

because it is abstract and subclass must implement the method uncaughtException(Thread t, Exception ex);
I created a subclass:

public class Log4JCatchAllExceptions extends Log4JUncaughtExceptionHandler{

@Override

public class uncaughtException(Thread t, Exception ex){

log.error(t.getName(),ex);

}

I used them as follows. In any class with a main method I use the class to catch all exceptions with the line

Thread.setDefaultUncaughtExceptionHandler(new Log4JCatchAllExceptions());

This only needs to go in once for the entire program. All uncaught exceptions will be caught and handled by the Log4JCatchAllExceptions class. Its pretty neat, no need to add lots of try catch blocks to the code and has pretty much the same effect. What you don't have is the possibility to handle exceptions differently in different cases. The reason is because it has been added as a static property of the Thread class, hence every new thread will have this property. This can be overridden in the thread instances by using:

Thread t = new Thread();

t.setUncaughtExcpetionHandler(new Log4JUncaughtExceptionHandler(){

@Override

public class uncaughtException(Thread t, Exception ex){

log.error(t.getName(),ex);

}

};

t.start();

This will set the handler of the specific thread to the anonymous class that is declared here. So any time that there needs to be a specific handling of and error it can be done like this. Here I have just printed a log in the same way that I did with the default class. But in my real code I restart threads so that if they die thanks to a null pointer exception then they will respawn, reducing the effects of the error while recording it at the same time.

And don't forget that there are times where going back to regular try catch blocks will fill in any holes that are not handled by these two classes.

Programming tests: is there a shortcut

2011-06-11T12:58:00.000+03:00

For those of you not following my tweets, I tweeted the other day about http://www.codeeval.com/. This is a great service that basically allows you to create job applications online (no big deal) with the bar to sending it in as passing the programming problem that it presents (very big deal). This could be a big help in theory. You can see how organised they are about coding, for a good enough problem it will show their problem solving skills and their thought processes.

I have a problem with this though, there seems to be no way to block the applicant looking the answers up. There are timed problems, but copy paste is pretty quick once you have found the algorithm on the web. In addition it has a plagiarism detector so you can see if candidates are sharing their answers. This could actually be used by submitting answers that you find on the web. Once a candidate uses the same answers from searching the web the plagiarism detector will flag them.

Considering all of this I would still want to do a reasonable live test as well. My main reason being efficiency of programmers is quite important to me. I cannot stand sitting with someone who is repetitively doing the same slow process over and over again. I saw recently a description of programmers as being "Inherently lazy with just the right amount of motivation." This description presents us with someone that is too lazy to keep doing something that takes 2 minutes over and over again, but will be motivated to spend an hour finding or writing a solution that will make this process automatic. I love this, its true about me, and about many other coders. I remember hearing a quote by Richard Stallman, Founder of the GNU project, who said that he would have never got it done if he wasn't lazy. I cannot find any reference for that right now so don't take me at my word. But I am pretty sure that I read it in Rebel Code. I digress...

I want to see not just efficient solutions to problems but efficient working method to get there. There is nothing more painful than watching someone click on the file menu and select copy and paste, even using right click in an editor, everyone should know ctrl^c ctrl^v. Thats ok, I doubt you would come across many programmers that don't, but there are more examples of things that people can do to work quicker. Selecting, single words and lines are very easy to select by double or triple clickling, no need to be exact about highlighting all the letters. The less a mouse is used the better. Users of VI and EMACS will be proficient at this and will probably be mouse free in other environments too. Additionally X windows users will know that if you highlight something it is copied to the clipboard and can be pasted using a single middle click. Saving too, most of the time its just ctrl^s but not doing it will mean you get prompted to save a few seconds are required of a mouse click, these things do add up. It sounds like I am talking about some pretty irrelevant or small things but trust me when you watch some one who doesn't use these things its slow.

I am not the best at these things, but I definitely notice when they are not being done. When I spent more time in EMACS and VI I taped a list of common commands on the wall behind my monitor. The more I used them the less I had to refer to the list and the faster I got at using these wonderful tools.

One weird thing that I have is about searching through code. I get it if you don't know the code, but there are many shortcuts especially in the IDEs to make it quicker. I much prefer ctrl^j to ctrl^f in Eclipse, but what I prefer the most is being familiar enough with the code to know where to go. Don't search just go straight there. Someone who can type quick enough and knows the shortcuts to search should be able to beat me in a large file. However its the familiarity with the code that I am looking for. If you are my programmer and you don't know the code well enough how can you solve bugs or make feature changes. I am not talking here about new code, I am talking about code the programmer has written themselves. Searching is fine, I would never say to someone that they cannot do it, but I want to see some indication that the code they have spent a few weeks writing is somehow bouncing around their heads.

I love whiteboards

2011-06-08T23:10:00.000+03:00

... but not as much as these guys. http://bit.ly/jAxznB

I love this idea, I think its great. I have had whiteboards from my undergraduate degree and my post grad. My supervisor meetings revolved around us working through ideas on the white board together. If it was important and I didn't have time to write it down I would bring my camera and take a picture of it to refer to later.

I cannot recommend them enough. I have one white board in my office, its full there is almost no space left for temporary things. I got it when I was lost in my notepads; ideas, todo lists, diagrams, passwords and notes. Things were getting difficult to keep track off. Now I have my whiteboard, it has not just everything I need to do, but everything that needs done. My task, my staff's tasks, even my boss's tasks. Its colour coded too, to indicate priority, that way I can add things to lists without worrying about the other. Larger tasks are broken down in separate sections in to a group of smaller tasks. Where possible time estimates are written next to tasks to give an idea of the scale of what needs to be done.

This is great when things get done they are erased, when things do not get done they stay, much better than being lost in the depths of early pages of a notepad. My notepads are for doodles and daily tasks, or notes from the evening to remind me to do something the following day. Once the page is turned in the notepad I need not worry about the previous pages, all the important stuff is on the whiteboard.

Its great for planning and working in a team. Particularly if anybody can easily see who is assigned what. My bosses know that I have plenty to do and can see by the turn over of tasks that I am at least getting something accomplished every day.

The problem I have is that I only have one right now. Its almost full of tasks and plans for future projects. There is little room for diagrams, pseudo code and doodles. I find that I spend some time every day explaining something to someone, this is best done with a space on the whiteboard and a dry marker in hand.

I must add a task to the whiteboard:
Get more whiteboards CTO 2hours

More on Hiring: or how to avoid moron hiring

2011-06-06T01:00:00.000+03:00

I have written about hiring before and I felt there was more I could say about it. I do not have a huge amount of experience of searching for jobs elsewhere but here in Israel I feel the whole process is lacking. I am originally from the UK and though I spent most of my time since high school studying at university I did apply for some jobs. One experience jumps to mind. I had a phone interview with a big consulting firm. At the time I was not so confident with my programming ability. I was young and naive and I guess what ever I said came off as, "I don't like programming, I don't want to do it." I was told later by HR from this company that I did not progress because I said that I did not like programming. (things have changed considerably for me). There was no attitude of 'lets interview him some more and see how it goes.' Later interviews, here in Israel, with other big companies have resulted in my not progressing for good reason, failing to pass whatever puzzles, tests and problems that were set out for me (Google, Bloomberg, and Brevin Howard), and being offered a job by Intel because I did pass these tests. These are big companies, with HR that has been developed over many years. They are efficient and are able to take on the best candidates.

So why here do we have this attitude of wasting interviewers' and candidates' time by interviewing them when they are inappropriate. Additionally at my previous company, for all the positives that they gained from taking a chance and hiring me, there were many people who were hired who were let go in 3 months or less because they were just unsuitable for the job. The turn over of employees in my time there was ridiculous. I suggested that they paid more to new candidates in order to attract better people, they didn't believe me and stated that there just isn't anyone out there. This I find hard to believe.

Testing is the way forward. As I stated in my last post on this topic it should be not just one question but a number designed to demonstrate many different abilities and not to prejudice a decision by the candidate failing at one thing. There are certain things that I want to know from a candidate that I should implement for the next interview process. I work with linux and the commandline a lot. I use Cygwin on my windows machines to interface with my linux boxes and more importantly to gain the power of bash to manipulate files in windows. Where am I going with this. If a candidate claims knowledge of linux I have an easy question to test something basic. Given the directory C:\src\ (or whatever) find all the java files in that directory and its sub directories that have classes that implement a certain interface. Lets say the action listener interface. I could even sit them at my keyboard and get them to do it. If the candidate has no understanding of linux this would be impossible. If they do but are unfamiliar with the syntax of grep or find for example I would be happy for them to spend 5 mins reading the man pages. I would expect an answer like:

grep -Ri ActionListener * | grep -v svn | grep implements

I would expect to see them remove the svn directories from the result and refine the use of ActionListener to only when it is implemented in the class and not used as an anonymous class.
This is an ideal list of things a candidate should know. Its use is explained in a link near the bottom. A good score would certainly make a an ideal sounding candidate. But these things especially the in depth knowledge of computer science things like sorting and search algorithms do not always translate well into the best employees. Discussed in a further article about the hiring process, it seems different approaches can be taken and are valid.

Another thing that I would like to see is their participation in something like StackOverflow, Experts exchange or some other forum of this type. Alternatively involvement in an open source project.

Regular expressions are such a useful tool. Everyone should know about them and how to use them. Again its easy to set up a simple test to discover the number of words of a certain type in a give text file. I am reminded of this wonderful xkcd cartoon.

Finally here is another article on some positive traits to look out for in a new candidate.

Happy hiring!

Congratulations Linus - 20 years of a job well done

2011-06-01T14:55:00.000+03:00

Linux kernel 3.0 released

Very amusing comments from him too.

Keep up the good work.

Shutdown Hooks Eclipsed by terminate button

2011-06-01T13:46:00.001+03:00

We are creating a new module for our system to receive data from another source. Everytime we terminate the test application in Eclipse it causes and unexpected termination at the suppliers end. I get notified and they assume something is wrong. Solution add a Shutdown hook to end the connection gracefully.

Nothing is ever that easy.

Something About J: Shutdown Hook and Eclipse

turns out that a known bug in Eclipse is that clicking on the red terminate button does not kill anything nicely.

No solutions only workarounds:

1. Run in a console. A bit annoying when working in dev, you need to constantly stop and start the app and it can slow you down a bit.

2. Relatively safe workaround if you are using threads. It works well but you have to remember to use it and not click the red button!

Thanks to both for providing the most concise and useful explanations and suggestion.

Java TimeZones (Daylight savings time not included)

2011-05-31T11:52:00.000+03:00

Quick background. We receive data with different timestamps. The easiest way to work with the data is to use SimpleDateFormat and work with Date. Date holds a reference to the date as number of milliseconds since the Epoch (Jan 1 1970).
For some reason our suppliers give us formated dates form different timezones. No problem we just set the dateformatter to use a different timezone when parsing the date string. The API doc states that it "also figures out daylight savings". However it does not state that you must use the long code ("Europe/London") for the timezone and not the 3 letter code ("GMT"). List of short codes. Now we are using the long code we are getting the correct dates with daylight savings taken account of.

Even I have a boss

2011-05-30T12:12:00.002+03:00

I am a bridge between two worlds. I manage the technology of the company. Everything that is not financial or administrative I am responsible for. However when it comes down to it I have to do what I am told. The good news, it just has to get done. The bad news, they don't understand what needs to happen to get it done, and usually they think that it is always easy and could be done in a matter of minutes. In practice what this means is that some times as a programmer you can do something simple that the non techie people think is amazing and should have been time consuming. Other times the opposite is true.

Either way I'm screwed, if I do something quickly that they think should take a long time, it raises their expectations for the next time. If I have something that the bosses think should be straight forward, then they simply do not understand why it is that its taking so long.

This gets in the way of my work, I have to either spend time explaining the reasons why things take time or not get them done properly. In the past when I have been required to get out a new product or even a big feature I work hard on it to make sure its done and out there working, even if its missing a few things. Then I let smaller tasks take longer as I use the time to add in all the features left out of the bigger project. In the end most things get done, other things fall by the way side, usually by that time they don't matter anyway.

My bosses aren't bad bosses, in fact quite the opposite. They just don't always get it. Looking back on things I wish I had time to build the infrastructure we work on before going live. I have built it as needed and gone live as quickly as possible. Over time new versions and bug fixes have resulted in better software. I am left wondering whether this has worked out better than creating a whole system over a long period, testing, retesting, and then going live. Would it have just taken me longer to make the same mistakes.

As my staff grows and the codebase gets larger, I can implement better development standards. Better testing, better design, better implementation. I have more man hours available to me through my team, I can get things done faster and better at the same time. That means I will spend less time coding and doing other techie things and more time managing the staff and being a bridge from the bosses to the techies. I am their protector. I know how long it will take, I have been there, when the programmers say its going to take 2 weeks, I know it will take 4. But I know that I will put the right pressure on to get it done, and my bosses will be kept at bay knowing that I am competently managing the project. They will bug me, I will be the one disturbed leaving my staff to get on with it without the added pressure. Hopefully resulting in better products at the end.

Hiring right

2011-05-25T11:58:00.003+03:00

I saw this the other day:
http://www.inc.com/ss/7-unconventional-ways-hire-best-tech-talent#0
While I like the idea, it was truly impractical for my needs. Having said that I do feel that I could have been more thorough in my interviewing process.

Lets start at the beginning. Over the past few years I have been to a few interviews. I have had some tough questions. I have done OK in some and not so great in others. Three companies, two of which I have had multiple interviews, where I have basically been told at the end of it, "we like you but you don't have any experience in Java (or what ever it was at the time that I didn't know)".

I have to say this is particular annoying and a huge waste of both my and their time. If they were looking for a Java programmer why were they interviewing me. I didn't have it on my CV. Ok I have a few interesting things on my CV but at the time Java wasn't one of them. I said during the interview that I didn't know Java right now but I could learn. In fairness most people when they are looking for a Java programmer they are looking for someone with 3 to 5 years experience that can hit the ground running. But again they shouldn't have looked at me. So the conversation continues. "Really you can just pick it up, huh. What evidence can you present us with that would convince me?" Well for my current (at the time) job I learnt Perl Javascript and SQL and became proficient enough at them to run my own two man projects within a month. "Oh thats interesting, what about a real language?" Harsh! Though again I get it, no proof that I have an understanding of OOD, but having said that I believe it is possible to learn the basics in a week reading on the internet. The fact that this guy was so conceited that he thought learning Java was harder than the 3 languages I had to get to grips with prior to this, was a little narrow minded. Aside from reading about Perl in the week running up to the start of that job everything else was done offline. We were on a closed system with no internet in the office. That meant man pages (perldoc to be specific). Javascript and SQL had to be learned from reading pre existing code and talking to others in my office. All done with Emacs, no fancy IDE like Eclipse which does so much for you. So a little context helps when responding to answers.

So I finally landed a job, this job, I didn't know Java but I had a month to read about it before I started. I played around a little and the rest I learned as I went along. Its been two years. And you know what, I rate myself pretty highly. How do I know this. I hired someone. Someone I expected to blow me out the water in terms of coding. You know what, she has 9 years of coding experience, c, c++ and Java. As I understand it 3 years of each. She doesn't know much about Threads or Swing or Sockets. She is proficient at what she does know. But somehow after all her time doing this, I still have a greater breadth of knowledge.

(Additionally I was on track for becoming Rookie of the Year in Java on Experts Exchange but my responsibilities took up too much of my time to actually continue doing all this work for free. I will post more about EE and Stackoverflow later.)

I am the CTO, I need to know everything (not saying that I do, just that I should have a wide range of skills and knowledge). I am confident that my employee will be fine, it will start off slow and she will improve, but there is a lesson. Assuming that someone who has the experience you think is necessary is not necessarily the best indicator of whether they will do well in your company. I believe that had these companies taken a chance on me then they would have been pleasantly surprised by what I could have achieved. They weren't willing to take that chance, and why should they they have I am sure they could have found plenty of other coders out there.

This is the lesson of hiring, you never know what you are going to get. You want some one great, with experience, broad knowledge, in depth understanding and the ability to pick up things they don't already know. You try to test for this. But its still hard to know for sure.

Lets step back again, I have had some interviews where I have been asked to solve some problems, or give examples of code. Generally these have gone badly for me. Personally this is where I crack. Its not the pressure I just don't think well like this. And worst of all I don't end up thinking quickly, which is the worst part, as they not only want to see the correct solution but they want it done quickly. So I decided that I would not ask these types of questions, firstly I don't know how to interpret the results, and secondly if they know how to do this already then it doesn't show me where they are lacking, and if they don't know how to do it then it doesn't show me other areas where they are positive.

Back to my recent hire. I have spent too long going over basic Hex and Binary. This stuff should be second nature to a true CS grad, granted many coders might not be CS grads, but I have a CS grad and I wanted one specifically. The reason: because they should be able to solve problems like this. I am beginning to see the logic in some of these harder interviews and the fact that (big headedly) I may have "slipped through the net", anyone that answers the type of problems I was getting from Google, Bloomberg and Brevin Howard deserves to be there or at least get another interview. It shows a basic level of knowledge/understanding/problemsolving skills.

So for the next interview I will ask the following:
given the a byte array which has the following bits:
sfff ffnn nnnn nnnn nnnn nnnn nnnn nnnn
show me how to extract the one bit sign s, the 5bit denomenator reference f, and the 26 bit numerator n.
the answer would be pseudo code that shows a bit mask for n
a shift and a bit mask for f
and a shift for s.

In fairness to my employee she did solve the big problem that we were having that the byte array we were receiving looked different from the one we were trying to send. The problem was that we were receiving the byte fields as little endian but reading them as big endian.

So now one gets in the door with out understanding these concepts. I am going to keep track of these problems for the future. If any more crop up I will be noting them here and using them for interviewing. I propose a system by which any one incorrect answer will not be the end of the world. A scoring system will be used to decide if the candidate has a satisfactorily broad knowledge and with good problem solving skills.

More on this as time goes on.

The benefits of getting things right in OO

2011-04-13T12:27:00.002+03:00

For anyone experienced in writing good OO code, the design is critical. Planning ahead is essential in creating reusable objects. I have been working on some old code that I wrote when I first started my current job and at the same time was very new to Java. Its horrible. Everything thrown in together. Control, view and models all in one class; kludges, hacks and workarounds thrown in instead of proper refactoring. Right now I am adding a new feature to the system. I had two choices glue another piece of code to the mess or refactor the whole code base while adding in this change. The result an extended project but more maintainable code, better code generally. Additionally many things that I am aware could be added or an issue have been taken care of, so that they should be easier to deal with later.

So why was it such a mess in the first place?

There are many reasons, but it comes down to perceived amount of time to finish a task. If I think I can get something done with a little hack or a quick kludge then thats the way its going to get done. I know its wrong and I know in the future it might cost me but my environment can sometimes dictate that it must be done now. I am the CTO, I am the bridge to the technical staff. My position is to get the technology working based on the requirements of my non technical bosses. The situation where my bosses think something simple in their heads should be easy and quick to implement happens time and time again

The non technical management want working systems or products. My products are two separate systems. One system is an offline system. They run it, it does something, and produces a result. The other is a live realtime system that runs 24/5 communicating with the outside world. That is what they want to see, it is my job to make sure they are produced, working and reliable. Occasionally they request changes or we have routine upgrades. This is where things get complicated. They want it done. They want it done soon. They have no understanding of unit testing, refactoring and documentation.

What is the solution?
Get it done. Many could argue getting it done means unit tests and refactoring as you go. They would say that doing testing and keep code maintained is actually efficient in the short run as well. This maybe the case but some times I am just too short sighted to see it like that. I am under pressure, I have to get it done, why would I write some more code that isnt really "relevant" right now.

I am in a good position right now. I have two working systems and though there is pressure to get this new feature into production I have more time, and I am able to fend off my bosses so long enough to refactor a large chunk of my system.

Conclusion. My code looks a lot nicer. I have implemented the MVC pattern nicely, well considerably better than before. I have reusable objects that have meant creating the two different view layers considerably easier. Small changes to the model in the future will require only making changes in one and not two spots. Its obvious now, but I am glad to see how much I have learned since I started.
So I got it done and now I have got it right. It would have been nice to get it right first time, but they next project will benefit from what I have learned. I will not necessarily get it right but it wont be as wrong as the code that I have just rewritten was.

Edit: This post was written concerning refactoring the offline system. A change was made to the live system recently that used the refactoring of the offline system. The change was smooth and easy because it simply reused previously tested and working components from the offline system. go me!