Luke Chadwick | vertis.github.com

Rails 3.2 Precompile error

2015-02-12T00:00:00+00:00

Every so often I come across an exception that doesn’t turn up good search results as a solution. This time it was:

$ rake assets:precompile --trace
...
NoMethodError: undefined method `empty?' for nil:NilClass
  (in <snip> /app/assets/stylesheets/application.css.scss)
<snip>/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/sass-3.1.16/lib/sass/tree/root_node.rb:23:in `render'

The hint to the problem is actually in the above error, but without context would be difficult to spot (I didn’t see it with context)

The app in question is now a rails 3.2 app. However it didn’t start it’s life that way, it was slowly upgraded from 2.x, and had it’s own asset handling. The error occurred during a migration to the standard asset pipeline.

Sure enough I had gem 'sass-rails', '~> 3.2.0' in one area of the Gemfile and gem 'sass', '3.1.16' in another.

Things I Want To Do This Year (2015)

2015-01-01T00:00:00+00:00

This will be my second year documenting things that I want to do this year and giving myself a mark on the results. I found it to be a valuable exercise last year, though my priorities did shift over the course of the year.

Continued / Extended goals

Read 2 books a month at least 50% non-fiction (This is now based on my velocity from last year)
Collect another large dataset
Obtain a healthy BMI
3D print and build a 3D printer
Publish usable libraries in Elixir and Golang

New goals

Speak publicly about technology at least twice
Write at least 12 blog posts for the year
Write a library for use in ML or AI
Write an automated trading system that uses AI
Fly a drone in an automated way
Create a full game using VR
Learn Italian
Learn how to make a quantum

Scorecard for 2014

2015-01-01T00:00:00+00:00

Wow! 2015 already. 2014 certainly went by in a flash. At the start of the year I documented everything that I wanted to get done in 2014. Now I’d like to do a quick retrospective on those items.

I’ll start by going through each of the items listed and giving myself a mark out of 10 and then I’ll document why I gave myself that mark.

Mark	Item	Comments
5/10	Create a computer game of some type (mobile or desktop)	While I did a bunch of work in Unity (and for Oculus Rift), I did not launch a game to market
5/10	Commit code to an open source project every day. If you forget, just begin again (my streak so far is 50 days)	My streak was about 170 days by the time I got sick of it.
0/10	Take those rollerblades you’ve had in the cupboard for 6 years and actually use them	They’re still in the cupboard
0/10	Finish the personal training app you were working on last year	This has been shelved indefinitely
10/10	Read at least 1 book per month	In total I finished 25 books in 2014. There was a good mix of novels and non-fiction
7/10	Learn another programming language till you can use it proficiently (I’m working on learning golang)	I have dabbled with both Golang and Elixir this year, but have not produced anything substantial in either
0/10	Get the RHoK project FGIS out into some fire trucks for testing	Perhaps one of the saddest of my non-completes for 2014, this failed to progress both from lack of interest on the problem owners side and also lack of pushing on my part
10/10	Design something from scratch and 3d print it	I have designed several items from scratch and 3d printed them this year
Silly	Walk 100km in one go (as of last year I can comfortably walk about 20km in one day, I want to improve on this	I walked consistently for most of the year, but I’m ruling this a silly goal and not scoring it
0/10	Build muscle (Last year I lost about 15km, going from 97kg-83kg)	Weight is back, and I haven’t put on any muscle
7/10	Collect and publish a large dataset under creative commons	I have collected the dataset but have not yet published it
10*/10	Build a quadcopter (carried over from last year)	Ended being smart and buying an almost completed quadcopter. I have been flying it a lot and introducing other people to it. Have managed to get the ball rolling on my employer investing in the technology as well
10/10	Visit a country you haven’t been too before	Visited Italy for work, and went for a day trip to Switzerland while there. Also spent a bunch of time in San Francisco and LA, which while I’d been to the US before (NY) were both new as well
10/10	Meet/befriend 100 new people (more than just friending on twitter)	Can’t say that I collected 100 business cards and that I’ll stay in touch with them all, but I have met and got to know a LOT of people this year

Final Score: 74 / 130

Thoughts

A passing grade but not as good as it could have been if we count the things I intended to do at the start of the year and didn’t end up doing.

On the other hand there are many many things that I accomplished this year that are not on the list.

The only one that I’m particularly unhappy about is my lack of progress on the fitness side. This is truly and area where I need to do better.

Convert an AWS Route53 Resource Record Set to point to an Alias Target

2014-06-19T00:00:00+00:00

Occasionally, you’ll want to change a Route53 Resource Record Set from one type to another. In the case of going from a traditonal ‘A’ record to an Alias Target the following code (with the aws-sdk rubygem) should do the trick.

The gotcha with this process is the need to unset ttl and resource_records as a part of the update.

zone = AWS::Route53.new.hosted_zones.select { |z| z.name == 'example.com.' }.first # find zone by name
rrset = zone.rrsets.create('jedi.example.com.', 'A', { :ttl => 60, :resource_records => [{ :value => '127.0.0.1' }] })

# Now lets update it to be an alias target
stack = AWS::CloudFormation.new.stacks['jedi-stack'] # assuming the stack exists

lb_resources = stack.resources.select {|x| x.resource_type == 'AWS::ElasticLoadBalancing::LoadBalancer' }
lb_resource = lb_resources.first

lb = AWS::ELB.new.load_balancers[lb_resource.physical_resource_id]

rrset.alias_target = { dns_name: lb.canonical_hosted_zone_name, hosted_zone_id: lb.canonical_hosted_zone_name_id, evaluate_target_health: false }
rrset.ttl=nil if rrset.ttl
rrset.resource_records=nil if rrset.resource_records
rrset.update

You should now have a RRSet pointed at your ELB.

Mount an S3 bucket using IAM roles

2014-06-17T00:00:00+00:00

After much searching around I discovered that newer versions of s3fs support IAM roles (I’m using 1.77). Unfortunately, this functionality is barely documented.

I ended up using the following line to mount my directory:

s3fs bucketname /mnt/bucket -o use_cache=/tmp,allow_other,iam_role=`curl http://169.254.169.254/latest/meta-data/iam/security-credentials/`

NB: s3fs does not need s3://

If the role you’re trying to use does not have access you’ll get something like this:

$ ls /mnt
total 20
drwxr-xr-x 5 root root 4096 Jun 16 23:39 .
drwxr-xr-x 7 root root 4096 Jun 16 23:39 ..
d????????? ? ?   ?       ?            ? bucket

The other error message I’ve seen is:

touch: cannot touch ‘/mnt/bucket/test.txt’: Transport endpoint is not connected

This happened when I specified the role incorrectly (it does not need to be the full arn).

Awesome Links #10

2014-01-31T00:00:00+00:00

I’ve promised myself that I’m going to do a lot of reading this year. To date I’ve done quite well, and the books have been amazing.

“How to Create a Mind” by Ray Kurzweil was an excellent read. It seemed a great idea to follow up with more AI books, I read two William Hertling novels, Avogadro Corp and A.I. Apocalypse. They were both great, but I enjoyed the second more.

Not that I have a heap of time for it, but reading “How to Create a Mind” by Ray Kurzweil got me thinking about Machine Learning again. Today I stumbled upon (Machine Learning in Javascript)[http://burakkanber.com/blog/machine-learning-in-other-languages-introduction/], which is an interesting series of posts.

Things I Want To Do This Year

2014-01-01T00:00:00+00:00

While I was on my flight back from visting family in Sydney for Xmas, I started writing a list of everything I want to get done this year.

It’s quite a long list, and it will be a very full year if I manage to do half of it. But it’s worth writing down:

Create a computer game of some type (mobile or desktop)
Commit code to an open source project every day. If you forget, just begin again (my streak so far is 50 days)
Take those rollerblades you’ve had in the cupboard for 6 years and actually use them.
Finish the personal training app you were working on last year
Read at least 1 book per month
Learn another programming language till you can use it proficiently (I’m working on learning golang)
Get the RHoK project FGIS out into some fire trucks for testing
Design something from scratch and 3d print it
Walk 100km in one go (as of last year I can comfortably walk about 20km in one day, I want to improve on this)
Build muscle (Last year I lost about 15kgm, going from 97kg-83kg)
Collect and publish a large dataset under creative commons
Build a quadcopter (carried over from last year)
Build a for the REA hackday in Feb
Visit a country you haven’t been too before
Meet/befriend 100 new people (more than just friending on twitter)

As you can see. I have my work cut out.

Awesome Links #9

2013-12-23T00:00:00+00:00

Now that the end of year is upon us, it seems like an opportunity to get some reading done & work on a few small projects.

I just finished The Last Firewall by William Hertling, which was a very good read. I haven’t read his previous two books, but I’ll have to pick them up now. I partically liked the (brief) reference to a Puppen (half dog, half cat).

I’ve been dabbling a bit with Golang. It’s a fantastic language and I’ll be spending more time learning it over the break.

An update on my AWS bill

2013-12-17T00:00:00+00:00

The last 48 hours have been surreal. When I typed up the experience up yesterday morning, I was not expecting it to get more than a handful of votes on Hacker News.

As I watched the discussion yesterday I was warmed that people both cared about my plight and found it useful to have a reminder.

All that while, the case with Amazon slowly progressed. They first answered and asked me terminate the stopped instances. They indicated that once this was done they would put in a request to waive the fees (subject to approval).

Overnight (I’m in Australia), this has been approved. I still can’t see it from the Billing Console, which I’m told is normal, but I’ve been informed in the support case that Amazon will be crediting the entire bill. Yay!

There was part of me, when I saw this bill, that knew what Amazon’s response would be. I don’t deserve that kind of relief after leaking my key, but Amazon has always been a cut above.

In my interactions with Amazon Account Managers and Architects (work account), Amazon has consistently cared about people being efficient in their instance usage. They have gone so far as to suggest changes that would wipe thousands off a monthly bill.

With this experience they have reinforced, to me a least, that they value me as a customer for the long term. I will one day be paying Amazon $3000 dollars a month, but I hope that it will be the bill for, an as yet non-existent, startup that is skyrocketing to success :D

This is all without mentioning that Werner Vogels stopped by the Hacker News thread (The ticket was already on it’s way to being dealt with by that point, but it was amazing).

Finally, thanks again to all the supportive people over the last 2 days.

Discuss it on Hacker News

My run in with Unauthorised Litecoin mining on AWS

2013-12-16T00:00:00+00:00

Update: You can read an update to this story here

Normally I’m a big advocate of open sourcing projects both current (and old) on GitHub. Today though, I wish that I wasn’t.

On sunday night I received an email from Amazon saying that they’d detected my Amazon key on one of my repositories. This was a little bit of a surprise, because I’m usually so diligent about not saving credentials into repositories.

After a brief search I found the key buried in an old project that I’d just decided didn’t need to be private.

That wasn’t the end of the matter, I was in for a rude shock when I logged into my Amazon account to check for unauthorised usage. $3000+ in pending charges. Woah!

It didn’t take long to find the source of the billing. Twenty cc2.8xlarge instances humming along in the us-east region for the last two days.

By this stage I’d already revoked the key (as suggested in the email). So I quickly shut the instances down, while I would have liked to preserve them for forensics, I just couldn’t afford to leave them running while waiting for Amazon support (I do not pay for support, since this is just my private account that I dabble with).

After taking stock for a few moments, I detached one of the volumes and attached it to another instance. Having a poke around confirmed what I had already guessed. The unauthorised user had been mining litecoin with the mining pool pool-x.eu.

I’ve emailed pool-x.eu asking them to suspend the account, but I’ve yet to receive a reply.

What have I learned from this experience?

Enable billing alerts

Given I spend about $60-80 a month with Amazon usually, I could have been warned MUCH earlier. Needless to say, now that the horse has bolted I’ve enabled the horse bolting detector.

Check GitHub

It’s not really that hard to do a regular search of GitHub for keys and passwords in your repositories. Check your friends repositories as well…many eyes.

Audit code before open sourcing

Always a good rule, but be especially careful flicking the switch on repositories that you’ve had as private for a long time.

Update: @joneaves suggested either using something like checkstyle (java) and/or a pre-commit hook. Good advice.

Use IAM Keys

Quite a few people have pointed out on twitter and hacker news that the other thing you should be doing is using restricted IAM keys.

More tips on Amazon

A friend pointed out that Amazon has a good security blog post that deals with this and other risks to your account.

Discuss it on Hacker News

Awesome Christmas Gifts 2013

2013-12-05T00:00:00+00:00

> Pebble Smartwatch

In a special edition of ‘Awesome Links’ I share what you should be buying your geeky friends for Christmas (assuming they haven’t already bought it for themselves)

We may all use smartphones for everything now, but if you asked the futurists of the 1980s what we would be using now, they probably have described something like Pebble. This Smartwatch took Kickstarter by storm, and continues to impress. At USD$150 it may fit into your budget for a loved one.

For all those geeks with kids, the Leap Motion is the perfect gift. There is nothing quite like watching a kid discover magic. And at USD$79 it has a much better time to entertainment ration than the movie gift voucher you were thinking of giving them.

If you’re looking for something with a smaller budget the Raspberry Pi ($35) or Beaglebone Black ($45) may fit the bill. One of the benefits to giving one of these as a gift, is even if they already have one, a second or third is always useful (they tend to get used in projects). The Raspberry Pi tends to be a bit friendlier for getting started, and the Beaglebone Black fits into the slightly more advanced category.

Need more ideas? Hacker Things has an impressive list of awesome gadgets. Also browsing Adafruit, Sparkfun and Tindie should help you find that perfect geek gift.

Update: opensource.com also has an awesome gift guide.

Awesome Reading: Open Companies

2013-11-22T00:00:00+00:00

> A rare archaeological find at GitHub by fumi (http://www.flickr.com/photos/fumi/9994096423/)

I’ve been reading about a couple of open companies, GitTip and Balanced Payments.

Both of these companies have taken what might be considered extreme approaches. Open sourcing just about everything required to run their companies. But it’s more than that. All the issues and discussions about running the companies take place in public as well. That gives an amazing level of transparency, and encourages so much community participation.

I found about about Balanced Payments through Steve Klabnik’s announcement that he would be partnering (employed by) Balanced Payments. This led to a bunch of other urls.

The first, was a pull request that Balanced Payments made on GitTip. Solving their problem regarding the need for a new payment provider. This is a powerful action to promote both Balanced Payments, but also a statement about how well being open is working for GitTip

Moving on from there, I decided to look into the how and why of Balanced Payments decision to be open. The CEO, Matin Tamizi, has shared a lot of his reasoning in a article on Fast Company: Labs and also in the GitHub produced OctoTales.

There is more information in the blog post announcing the decision, and also a page dedicated to explaining the ‘Open Company’ philosophy.

Of course, a lot of the thinking that has gone into making Balanced an open company comes from interacting with Chad Whitacre, the founder of GitTip, so it’s worth delving into his thinking on the subject. Which you can do by reading his excellent posts “The First Open Company” and “The Second Open Company”

I’m not sure about the assertion that GitTip is the first(or second) ‘Open Company’, but it’s certainly and interesting concept to explore further.

Of course you couldn’t finish a blog post like this without talking about GitHub itself and the influence it has had on the community. Both by being very transparent (if not entirely open), and in the building of tools that allows this kind of openness and collaboration.

Links for Good

2013-11-11T00:00:00+00:00

> Typhoon Haiyan as seend from the ISS

It’s easy to get caught up in making money, but there are things that you can do that will help the social good as well:

Random Hacks of Kindness - The next global hackathon is the 7th - 8th of December. I will be attending RHoK Melbourne

CrisisCamp - A global organization dedicated to helping after Earthquakes, Floods, and major storms. This is timely given the recent Typhoon in the Philippines

Awesome Links #7

2013-10-14T00:00:00+00:00

Awesome links for the 14th of October 2013:

> Edward Snowden courtesy of The Guardian

Cypherpunks 2.0 - Interesting talk by Andy Isaacson from Noisebridge. Videos are split into a bunch of sections Part 1, Part 2 & Q & A

Spark Core - This tiny embedded board looks very promising. Originally a Kickstarter campaign. Delivery is currently behind schedule. Most importantly though, it’s an open hardware project.

redecentralize.org - The site seems to be just getting started, but already contains some excellent interviews from people working on projects to “take back control of the internet”. Well worth the time to watch.

Awesome Links #6

2013-10-08T00:00:00+00:00

Awesome links for the 8th of October 2013:

Dashing (by Shopify) - Easy to use dashboard building goodness

Rickshaw Graphs - I wasn’t aware that shutterstock contributed to open source, until I stumbled across this graphing library in Dashing (by Shopify). It uses D3 under the hood, but provides a simple interface for the more common graphs (with bells and whistles).

Big list of D3 examples - Speaking of D3, this popped up on Hacker News this morning. I find when making graphs, having a wealth of examples is the easiest way to get started.

Polymaps - I’m sure that I’ve seen Polymap before, but I ran across it again this morning. Will have to have a play with it one evening.

Awesome Links #5

2013-10-03T00:00:00+00:00

Awesome links for the 3rd of October 2013:

Getting Press for your product launch - An interesting use of the Google News API and Mechanical Turk. Not exactly scalable though.

180 sites in 180 days - 3 months to form a solid base to continue learning development.

Launch day promotion - Hacker News list of sites to tell on launch day.

Awesome Links #4

2013-09-30T00:00:00+00:00

Python flavoured awesome links for the 30th of September 2013:

Getting started with Django - Clear and helpful videos/text for getting started with Django. Helped me gain at least a little bit of an understanding of django over the weekend.

Learn Python the Hard Way - Great way of learning (or brushing up, like me) on your python skills.

Crispy Forms - Crispy Forms is to Django, what Simple Forms is to Rails.

Awesome Links #3

2013-09-25T00:00:00+00:00

> Nathan Gould - Crysis 2

Awesome links for the 25th of September 2013:

Disqus - Scaling Django to 8 billion page views. Good reminder on the importance of mastering caching.

fesplugas/rbenv-installer - Nice automated way of installing rbenv and getting Ruby up and running.

camlistore - a new content management system written in go-lang. I’ve been playing with this for a few weeks now, and even though it’s still very raw, you can see the potential.

Heroku Roadmap (circa 2010) - Interesting look back at all of the things that Adam Wiggins and Heroku was thinking about back in 2010. They’ve built an amazing platform. Doing some archeology on how they went about it is a great idea (Might follow up on this further).

Awesome Links #2

2013-09-24T00:00:00+00:00

> Microduino: Arduino in your pocket, small, stackable, smart

Awesome links for the 24th of September 2013:

Pop! - An awesome way to securely fill forms (Disclaimer: I’ve just started at Pop!)

Analysis of NuoDB - Don’t claim to have solved/refuted the CAP Theorem, you probably haven’t

Microduino - Lots of potential applications for this Kickstarter project. Good collection of ‘shields’ as a starting point as well. I like my lazy electronics.

Awesome Links #1

2013-09-23T00:00:00+00:00

> The UNSC Infinity (From Halo 4)

Awesome links for the 23rd of September 2013:

Boxen - GitHub’s new machine provisioning framework gets your new Mac off to a flying start. Heaps of libraries to install things you’d usually have ot install by hand.

bamboo-vagrant-install - Automates the installation of Atlassian Bamboo on a Ubuntu Vagrant box so you can get on with the business of delivering your software. You will still need to sign up for a trial or pay for a license and enter it.

yadr - Yet another dotfile repo, this one continues to be my favourite way of getting an awesome set of aliases and defaults for my shell. I’ve yet to work out how it interacts with Boxen

How to build an RPM using Mock

2013-09-19T00:00:00+00:00

Building an RPM can be a somewhat annoying, painful experience. There are plenty of pitfalls, but the most annoying, for me at least, is building a package and then trying to install it and realising that it doesn’t work because of some incompatibility in the packages that just happened to be on your build box.

One solution to this problem is to use Mock. Mock does a great job of building up an standalone environment, so that you can ‘Fail early’, if you haven’t written your spec file correctly.

Start by installing Mock and other required packages. For example:

$ sudo yum install mock autoconf automake bison bzip2 createrepo \
                   gcc gcc-c++ iconv-devel libcurl-devel libffi-devel \
                   libtool libxml2 libxml2-devel libxslt libxslt-devel \
                   libyaml-devel make openssl-devel patch readline \
                   readline-devel rpm-build zlib zlib-devel rpmdevtools

Mock won’t allow users other than root by default. This is easily fixed by adding your build user to the mock group:

$ sudo usermod -G mock bamboo

From there I usually create an rpmify script. This example comes from packaging a rails application:

#!/usr/bin/env bash
set -x

# Remove previous results in case of build environment not being clean
rm -rf result

echo $BUILD_NUMBER > /tmp/build_number

bundle package

export PACKAGE_NAME=example-rails
rm -rf /tmp/${PACKAGE_NAME}
mkdir -p /tmp/${PACKAGE_NAME}
cp -rf ./* /tmp/${PACKAGE_NAME}/
mkdir -p ~/rpmbuild/SOURCES
tar zcvf ~/rpmbuild/SOURCES/${PACKAGE_NAME}.tar.gz -C /tmp/ ${PACKAGE_NAME}

rpmbuild -bs ${PACKAGE_NAME}.spec

export MOCK_CONFIG=epel-6-x86_64
export MOCK_CONFIG_ROOT=/var/lib/mock/$MOCK_CONFIG-$PACKAGE_NAME-$BUILD_NUMBER

/usr/bin/mock -r $MOCK_CONFIG --uniqueext=$PACKAGE_NAME-$BUILD_NUMBER --init
/usr/bin/mock -r $MOCK_CONFIG --uniqueext=$PACKAGE_NAME-$BUILD_NUMBER --copyin /tmp/build_number /tmp/build_number

/usr/bin/mock -r $MOCK_CONFIG --uniqueext=$PACKAGE_NAME-$BUILD_NUMBER --installdeps ~/rpmbuild/SRPMS/${PACKAGE_NAME}-${BUILD_NUMBER}-1.el6.src.rpm
/usr/bin/mock -r $MOCK_CONFIG  --uniqueext=$PACKAGE_NAME-$BUILD_NUMBER --no-clean ~/rpmbuild/SRPMS/${PACKAGE_NAME}-${BUILD_NUMBER}-1.el6.src.rpm

# Artifacts
rm -rf result
mkdir -p result
cp $MOCK_CONFIG_ROOT/result/*.log result/

if [[ -f $MOCK_CONFIG_ROOT/result/${PACKAGE_NAME}-$BUILD_NUMBER-1.el6.x86_64.rpm ]]; then
  cp $MOCK_CONFIG_ROOT/result/${PACKAGE_NAME}-$BUILD_NUMBER-1.el6.x86_64.rpm  result/
  cp $MOCK_CONFIG_ROOT/result/${PACKAGE_NAME}-$BUILD_NUMBER-1.el6.src.rpm result/
  exitstatus=0
else
  exitstatus=128
fi
/usr/bin/mock -r $MOCK_CONFIG --uniqueext=$PACKAGE_NAME-$BUILD_NUMBER --clean
exit $exitstatus

The script is designed to be used within bamboo, so it does some handling around the artifacts that it produces

Mock looks in /etc/mock for the config that you specify, in this case /etc/mock/epel-6-x86_64.cfg. It is from here that it finds information on how to deploy the dependecies. If you have custom repositories, you’ll want to create your own file.

And voila, your very own professionally built RPM.

My Post-PRISM checklist

2013-06-11T00:00:00+00:00

As an Australian, i.e. Non-US-Resident Non-US-Citizen, I've decided to take certain actions to change my online behaviours after recent events. While I have nothing of particular interest in my site usage/cloud/servers, I feel like the behaviour of the US government should not be ignored, and since US companies must to comply with US laws, they cannot be deemed safe.

WHY: Edward Snowden via The Guardian leaked numerous documents, including information about the PRISM program which suggests there is extensive surveillance and interception of foreign citizens' data without a court order.

NB: This is a work in progress. Suggestions welcome. If nothing else this is an indication of how annoying this is going to be, and how many US based services I currently rely on.

% Complete	Company	Service	Actions & Comments
10%	Facebook	Social Media	Backup and close facebook account. I don't really use this anyway.
10%	LinkedIn	Social Media	Backup and close LinkedIn account. Make details available using website with some form of Authentication.
10%	Twitter	Social Media	Backup and close twitter account. Self host a stream?
80%	Google	Gmail	Redirect domains and migrate email off the service.
1%	Google	Analytics	Search for a non-US replacement.
5%	Linode	Servers	Find another provider and migrate applications/data to new provider
10%	Amazon	EC2	Investigate data sovereignty for ap-southeast-2 region, and either move instances there, or completely off Amazon
10%	Amazon	S3	Investigate data sovereignty for ap-southeast-2 region, and either move data there, or completely off Amazon
1%	Heroku	Web Apps	All the little apps need to be migrated. Probably to a VPS (see above)
1%	GitHub	Repositories	Find another provider (or self host) and migrate repositories
1%	Dropbox	File Storage	Probably just self-host, and do without the functionality. Move data out of Dropbox, close account.
1%	iPhone	Mobile Device	Move back to Android, install ParanoidAndroid
1%	MacBook	Laptop	Switch to Linux instead of MacOS X

This is Bullshit!

2013-05-08T00:00:00+00:00

tl;dr

My colleagues made a funny video when there was an incident
Cake makes everything better

Upgrading Github Enterprise

Yesterday, there was a new release for our Github Enterprise installation that mitigated a fairly serious vulnerability. This was announced quite late in our work day, but given the severity of the issue I decided to upgrade the appliance before heading home.

The upgrade took a little longer than expected, but it eventually finished successfully and I headed out the door.

I didn’t realise it at the time, but due to a mandatory password change my email wasn’t coming through to phone. When I finally sat down at my computer after dinner a few hours later, I discovered that all was not right in the world. Our Github Enterprise hadn’t been able to accept pushes for the last two hours.

This is bullshit!

In fact the entire Xi’an office had gone to the trouble of making me (and GitHub) a youtube video.

<p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/DmC4eaaGmTw" frameborder="0" allowfullscreen></iframe></p>

Thankfully, the issue didn’t take too long to fix. The team at GitHub had already created a new package that fixed the issue.

Some context on ‘bullshit’

Our team in Xi’an put up with all kinds of frustrating problems. The internet and VPN(s) have been known to be unreliable. They occasionally bear the brunt of being a satellite office (getting forgotten when changes are made). There was at least one incident where the team in Xi’an waited through an entire standup without telling the team here in Australia that the sound wasn’t working.

They bear all these frustrations as professionals and with a happy attitude.

A little while ago our CIO Nigel Dalton went across to the Xi’an office. One of the sessions he held was how to express their displeasure in an “Australian” way. Which was basically a lesson in saying “This is bullshit”

While I would have preferred not to be the source of their frustration, I am so very happy that they have learned to express their displeasure.

Sorry cake

Now, hopefully it’s time for me to share something I have learned over many years. Cake makes everything better. If you take nothing else from the blog post it should be this.

I’m trying to work out how I can have cake delivered to a team on the other side of the world. Hopefully when I do they will film themselves eating the ‘sorry cake’.

Update: I’ve successfully had two huge cakes delivered. The team in Xi’an made me a nice video to say thank you.

Rails3 Asset Pipeline On Heroku

2013-03-03T00:00:00+00:00

Asset compilation doesn’t run on heroku when deploying a Rails 3.x application

Another of my ‘I just wasted a bunch of time trying to work out why something was broken’ blog posts.

This morning I was having a problem where the rails asset compilation wasn’t triggering when deploying an application to heroku. Googling for the problem resulted in a lot of people that were having errors, but no information about a complete lack of running.

Thankfully the code that does the setup is all open source, so I spent some time digging through heroku-buildpack-ruby on GitHub to work out what was going wrong.

It didn’t take too much digging til I realised that the task was being wrapped in a check:

if rake_task_defined?("assets:precompile")

This results in a --dry-run call. As seen below:

# detects if a rake task is defined in the app
# @param [String] the task in question
# @return [Boolean] true if the rake task is defined in the app
def rake_task_defined?(task)
  run("env PATH=$PATH bundle exec rake #{task} --dry-run") && $?.success?
end

Unfortunately, this also fails silently if there is a problem.

Heroku provides a run command. So we can check directly what is wrong:

heroku run env PATH=\$PATH bundle exec rake assets:precompile --dry-run

In my case it was complaining about a lack of nokogiri during the compilation (even though it was a dry run).

Creating a Vagrant base box from an existing Vmdk

2012-11-02T00:00:00+00:00

Some background.

The work I’ve been doing recently involves creating a standard platform for our applications. Part of the process of creating this platform involves spinning a a machine image for use within our production, staging and development environments.There is ample material in our current build pipeline for a dozen blog posts. Maybe I’ll get around to writing about it at some point.

Our “machine image pipeline” begins by using Koji to create a CentOS image. The image is then transformed for use in EC2 and VMWare.

In addition to using the machines within Amazon EC2, many devs would like to be able to use the machine image locally. Since Vagrant is a popular option for running machines locally, it’s an ideal choice to add to the pipeline.

There doesn’t seem to be a lot of information about the process of converting an existing machine image to work with Vagrant (in an automated fashion). I’ve compiled the steps I had to take and provided links back to where I sourced the information.

Caveats and disclaimers

I don’t pretend to be an expert at VirtualBox (I may be doing things the hard way)
The code is very “brute force”, it doesn’t check very much between each step

Step 1 - Import/Create the VM

After lots of searching I came across some VirtualBox tips and tricks including how to create a machine from scratch. It wasn’t the only blog post I found with these commands, but it was a good complete script from which to hack.

The VBoxManage command will create a directory under the base folder and store a .vbox file of the same name inside.

The --ostype will vary depending on the distro you’re trying to use. You can get a complete list with VBoxManage list ostypes.

Step 2 - Create a storage controller & copy/attach the VMDK

From there on in, I departed (slightly) from the script I’d found, and copied my existing .vmdk into the same directory as the .vbox.

I have an additional problem. The VMDK only has a 1GB of space. After a bit of searching I found a stackoverflow question that helped me resize the disk. The above code changes to:

Step 3 - Provide a NAT Port Mapping

To setup the items required for Vagrant, we need to ssh into the machine. The first part of that is creating an NAT Port Mapping. Another post provided the commands necessary to map the ssh port.

I initially thought to make the port mapping 222 because vagrant defaults to 2222. Annoyingly this failed silently. Though the reason should be familiar to anyone with basic Linux administration knowledge – using ports less than 1024 requires root access.

Step 3a - Stuff that didn’t belong anywhere else

Step 4 - Booting and waiting for SSH

With that done, the next step is to setup the user/access that vagrant expects. To do that we need to SSH into the box.

I experimented with a few ways of watching for ssh, but in the end none of them worked very well. So instead, just sleep for a while.

Step 5 - Setting up the vagrant user

Creating the user is fairly trivial, editing the /etc/sudoers file not so much. I downloaded a copy and then made the modifications.

Step 6 - Shutting down and packaging

The final step, is to shutdown and package the box for distribution

Finally the complete script

For ease of use, here is the complete script.

An awesome collection of ruby tricks from JEG2

2012-10-23T00:00:00+00:00

I very much enjoyed this awesome slide deck from JEG2 from his talk at the Aloha Ruby Conference, in Honolulu, HI in October of 2012.

It’s covers a large collection of tricks you can do with Ruby.

const_defined? wrong constant name Yum.Key (NameError)

2012-01-29T00:00:00+00:00

I hate obscure errors. Google wasn’t the least bit of help.

The chef solo run was breaking with:

[Sun, 29 Jan 2012 07:22:15 -0500] INFO: *** Chef 0.10.8 ***
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Setting the run_list to ["recipe[java::openjdk]", "recipe[mysql::server]", "recipe[jenkins::default]"] from JSON
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Run List is [recipe[java::openjdk], recipe[mysql::server], recipe[jenkins::default]]
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Run List expands to [java::openjdk, mysql::server, jenkins::default]
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Starting Chef Run for li231-195.members.linode.com
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Running start handlers
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Start handlers complete.
[Sun, 29 Jan 2012 07:22:18 -0500] INFO: Missing gem 'mysql'
[Sun, 29 Jan 2012 07:22:18 -0500] ERROR: Running exception handlers
[Sun, 29 Jan 2012 07:22:18 -0500] ERROR: Exception handlers complete
[Sun, 29 Jan 2012 07:22:18 -0500] FATAL: Stacktrace dumped to /var/chef/chef-stacktrace.out
[Sun, 29 Jan 2012 07:22:18 -0500] FATAL: NameError: wrong constant name Yum.Key

Bizarre. Looking at the stack trace provided no more clarity. Why was it looking for a constant called ‘Yum.Key’ in the first place?

Generated at 2012-01-29 07:22:18 -0500
NameError: wrong constant name Yum.Key
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/provider.rb:89:in `const_defined?'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/provider.rb:89:in `build_from_file'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:89:in `block in load_lwrp_providers'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:120:in `call'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:120:in `block (2 levels) in foreach_cookbook_load_segment'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:119:in `each'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:119:in `block in foreach_cookbook_load_segment'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:117:in `each'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:117:in `foreach_cookbook_load_segment'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:87:in `load_lwrp_providers'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/run_context.rb:58:in `load'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/client.rb:195:in `setup_run_context'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/client.rb:159:in `run'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/application/solo.rb:192:in `block in run_application'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/application/solo.rb:183:in `loop'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/application/solo.rb:183:in `run_application'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/lib/chef/application.rb:67:in `run'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/gems/chef-0.10.8/bin/chef-solo:25:in `<top (required)>'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/bin/chef-solo:19:in `load'
/usr/local/rvm/gems/ruby-1.9.2-p290@global/bin/chef-solo:19:in `<main>'

I started digging around in the source (as you do when debugging these type of issues):

def build_from_file(cookbook_name, filename, run_context)
        pname = filename_to_qualified_string(cookbook_name, filename)
        
        # Add log entry if we override an existing light-weight provider.
        class_name = convert_to_class_name(pname)
        overriding = Chef::Provider.const_defined?(class_name)

Having a look into the convert_to_class_name and filename_to_qualified_string methods was a dead end, they seemed to be working alright.

def convert_to_class_name(str)
  rname = nil
  regexp = %r{^(.+?)(_(.+))?$}
        
  mn = str.match(regexp)
  if mn
    rname = mn[1].capitalize

    while mn && mn[3]
      mn = mn[3].match(regexp)          
      rname << mn[1].capitalize if mn
    end
  end

  rname
end
...
 def filename_to_qualified_string(base, filename)
  file_base = File.basename(filename, ".rb")
  base.to_s + (file_base == 'default' ? '' : "_#{file_base}")
end

I then decided to see what values were being provided to build_from_file. Finally, a breakthrough. The file_name being provided looked strange /var/chef/cookbooks/yum/providers/._key.rb.

Where on earth was that coming from!

The culprit turned out to be the code I was using to copy the cookbooks to the server. Which was picking up the invisible ‘._’ file that Mac OS X uses: http://support.apple.com/kb/TA20578

tar cj . | ssh -o 'StrictHostKeyChecking no' "$host" '
rm -rf /var/chef &&
mkdir /var/chef &&
cd /var/chef &&
tar xj &&
bash ./provision.sh'

I worked around the issue by removing those files at the destination with:

rm `find ./  -name "\._*"`

Not an ideal solution, but it fixed the immediate problem and allowed me to get my Jenkins server installed.

git: fatal: index file smaller than expected

2012-01-24T00:00:00+00:00

NB: You should back up your working directory before trying anything in this post.

$ git status 
fatal: index file smaller than expected
$ git reset --hard
fatal: index file smaller than expected

Uh-Oh-Spaghetti-O’s. Not a nice error message to get first thing in the morning.

After much googling I realized that the .git/index is only used to track staged and unstaged changes. So the simplest way
out of this mess is to move the broken index out of the way.

$ mv .git/index .git/index.backup
$ git status

You should now be presented with a rather confused view of the working tree, where it thinks things have been deleted and there is a whole raft of untracked files…

$ git add .

And we’re back to where we should be.

Renaming a Rails 3 app

2012-01-15T00:00:00+00:00

Annoyingly Rails 3 spreads whatever name you give it when running ‘rails new’ over 10 or so files. This makes it hard to setup a base app. The following
shell commands will change the name of the app to the new name:

find . -type f -print0 | xargs -0 perl -pi -e 's/Oldapp/Newapp/g' find . -type f -print0 | xargs -0 perl -pi -e 's/oldapp/newapp/g'
I'm a little ashamed that the snippet uses perl instead of ruby. When I was trying to get the equivalent to work in ruby 1.9 it keep complaining about encoding. For anyone that is interested the ruby version I was attempting to use was:
find . -type f -print0 | xargs -0 ruby -pi -e '$_.gsub(/Oldapp/, "Newapp")'
Which gets an error:
-e:1:in `gsub': invalid byte sequence in UTF-8 (ArgumentError) from -e:1:in `<main>'
Even trying to force an encoding didn't seem to solve the problem.
find . -type f -print0 | xargs -0 ruby -pi -e '$_.force_encoding("UTF-8").gsub(/Oldapp/, "Newapp")'
It is likely this wouldn't occur if used from Ruby 1.8

Update:

While it’s nice to have a one liner that will accomplish the renaming. I’ve seen the above one liner corrupt the .git/index

I’ve written a simple ruby script that is more limited in how it selects the files:

TinyTds::Error: Unable to open socket (Sequel::DatabaseConnectionError)

2011-06-16T00:00:00+00:00

While working on one of our rails apps at work, I got the following error. Since we only recently switched to using TinyTds, I wasn’t quite sure what the cause was.
It turns out that in my case I’d switched the VM I run SQL Server on from ‘Host only’ to ‘Bridged’. So the driver couldn’t connect to the port.

When in doubt:

telnet hostname 1433

Getting the root directory of your gem

2011-06-03T00:00:00+00:00

I’ve been writing a few gems recently. One thing that you inevitably need to be able to do is find the root directory of your gem.

Assuming that you place this in ‘lib/your_gem.rb’, the following code will return the path to the root of your gem:

module YourGem
  def self.root
    File.expand_path('../..',__FILE__)
  end
end

An inefficient MongoDB query

2011-05-25T00:00:00+00:00

Short Version

MongoDB doesn’t support queries using case insensitive regular expressions against indexes (e.g /blah/i).

Long Version

Firstly: A big thank you goes to Jason McCay from MongoHQ for his help on this issue.

Yesterday morning I received an urgent email from MongoHQ regarding one of my databases. According to the email, one of my queries was missing the index. The scale at which these queries were occurring was putting significant load on their server.

The specific query:

Mon May 23 22:47:38 [conn707] query mydb.profiles reslen:441 nscanned:1979415 \
{ address: /^123 Fake Rd, Sydney, NSW 2000$/i }  nreturned:1 44740ms

And the offending ruby code:

Profile.where(:address => /^#{profile.address}$/i).each do |other|
    other[:lat] = lat
    other[:lng] = lng
    other.save!
  end

The reasoning behind my code was to find profiles that had the same address, and update them with the latitude and longitude. I considered preventing another request to the geocoding service to be beneficial due to the restrictive limits.

Since I couldn’t guarantee that the addresses had been entered with the same case sensitivity, I’d chosen to make the regex insensitive. Sadly, I wasn’t aware that MongoDB doesn’t support queries using case insensitive regular expressions against indexes (at this time).

A quick fix was to create another column that had an all lower case copy of the address and remove the ‘i’ from the query. I ran the following script (not necessarily the most efficient way of creating the copy):

Profile.where(:address.ne => nil).each do |profile|
    profile[:lowercase_address] = profile.address
    profile.save!
  end

And changed my code to the following:

Profile.where(:lowercase_address => /^#{profile.address.downcase}$/).each do |other|
  other[:lat] = lat
  other[:lng] = lng
  other.save!
end

It may have caused the ruby code to be a little bit more intensive, but it sped up the processing considerably.

List of Rackspace flavors and images

2011-04-02T00:00:00+00:00

I was recently trying to use the chef command line tool ‘knife’ to provision a new rackspace server. The tool requires
that you specify the flavor (size) and image(distro version), unfortunately I couldn’t find an easily searchable source of this information.

Here is the list of flavors:

1    256 server
2    512 server
3    1GB server
4    2GB server
5    4GB server
6    8GB server
7    15.5GB server

And here is the list of images:

4        Debian 5.0 (lenny)
10      Ubuntu 8.04.2 LTS (hardy)
14      Red Hat Enterprise Linux 5.4
19      Gentoo 10.1
23      Windows Server 2003 R2 SP2 x64
24      Windows Server 2008 SP2 x64
28      Windows Server 2008 R2 x64
29      Windows Server 2003 R2 SP2 x86
31      Windows Server 2008 SP2 x86
40      Oracle EL Server Release 5 Update 4
41      Oracle EL JeOS Release 5 Update 3
49      Ubuntu 10.04 LTS (lucid)
51      CentOS 5.5
53      Fedora 13
55      Arch 2010.05
56      Windows Server 2008 SP2 x86 - MSSQL2K8R2
57      Windows Server 2008 SP2 x64 - MSSQL2K8R2
58      Windows Server 2008 R2 x64 - MSSQL2K8R2
62      Red Hat Enterprise Linux 5.5
69      Ubuntu 10.10 (maverick)
71      Fedora 14
14362    Ubuntu 9.10 (karmic)
187811  CentOS 5.4

While this information may go out of date here is the current (1st of April 2011) list. Hopefully it’s useful

Removing a collaborator from all GitHub repos

2011-03-30T00:00:00+00:00

I help manage the GitHub account for the company I work for. Recently, I’ve been in a situation where I needed to remove a former colleague from about 25 GitHub repositories. Admittedly this is only painful because the account we’re using is an individual account. Frustratingly, “Organizations” an account type introduced to deal with this exact issue, is difficult to justify to management because of the additional cost.

In the meantime I’ve written a quick script using the GitHub Api that iterates through the repositories and removes a given collaborator. You can find the source code below:

#Gemfile
source "http://rubygems.org"

gem 'httparty'

A library to package up some functionality:

#lib/github.rb
require 'rubygems'
require 'bundler/setup'
Bundler.require(:default)

class Github
  include HTTParty
  base_uri 'http://github.com/api/v2/json'

  def initialize(user, pass)
    @username = user
    self.class.basic_auth user, pass
  end

  def repositories
    response = Github.get("/repos/show/#{@username}")
    response.parsed_response["repositories"]  if response.code==200
  end

  def collaborators(repository)
    response = Github.get("/repos/show/#{@username}/#{repository}/collaborators")
    response.parsed_response["collaborators"] if response.code==200
  end

  def remove_collaborator(respository, name)
    Github.post("/repos/collaborators/#{@username}/#{repository}/remove/#{name}")
  end
end

And the actual script:

# remove_user.rb
require 'lib/github'

if ARGV.empty?
  puts "Usage: remove_user <username>"
  exit -1
end

username = ARGV.first

github = Github.new('vertis', 'xxxx')
github.repositories.each do |repo|
  if repo["private"]==true
    puts "Checking: #{repo['name']}"
    collaborators = github.collaborators(repo['name'])
    if collaborators.include?(username)
      puts "Removing '#{username}' from collaborators list of '#{repo['name']}'"
      github.remove_collaborator(repo['name'], username)
    end
  end
end

It’s certainly not a polished script, but it does the job. Hopefully, I’ll eventually be able to convince my manager to upgrade :)

Review: Data Source Handbook by Peter Warden (O'Reilly)

2011-02-01T00:00:00+00:00

When a post about the Data Source Handbook came through my RSS reader today I was excited. I’ve used quite a few webservices, and follow programmableweb closely, so a book about APIs is exactly what I was interested in reading. Later in the evening I caved and bought the ebook version - I should have looked more closely.

At 39 pages I would consider the $7.50 I paid (thanks retailmenot) to be more than the book is worth.

The Data Source Handbook is not substantive, and shares little more than a single request to each of the services it covers, and where to signup for an api key. While Peter Warden has obviously spent quite a bit of time using the different services, as evidenced by https://github.com/petewarden/findbyemail, the level of detail just didn’t impress me.

The book is organised by the categories of the different webservices; Websites, People by Email, People by Name, People by Account, Search Terms, Locations, Companies, IP Addresses, and Products(Books, Films, Music, etc).

Of the 54 APIs covered, I’d used about half already. Which wouldn’t be a problem if the coverage of these APIs had any depth.

All in all, if you’re looking for an advanced and substantive book on APIs…this isn’t it.

Quick Reminder: ssh-copy-id on Mac OS X

2011-01-19T00:00:00+00:00

Mac OS X for some reason lacks the ssh-copy-id shell script. You can install ssh-copy-id on Mac OS X using either ‘homebrew’ or ‘macports’. Since it is a fairly trivial script you could also choose to copy it from various sources manually.

e.g:

curl https://github.com/beautifulcode/ssh-copy-id-for-OSX/ssh-copy-id.sh \
      -o /usr/local/bin/ssh-copy-id chmod +x /usr/local/bin/ssh-copy-id

Switching to RVM+REE+NGINX+Passenger

2010-09-30T00:00:00+00:00

We’re increasing the push toward the release, and I’m now spending precious time looking into the why our Rails Stack is leaking workers. My initial research suggested that these workers could be sent a SIGABRT to get them to throw a stacktrace and exit - this hasn’t worked however.

I’ve been planning to change from the current stack UbuntuProvidedRuby1.8.7+Apache+Passenger to RVM+REE+NGINX+Passenger for a while. With the prerequisite testing done that our app works with the new stack. I’ve decided that rather than attack this problem head-on I’ll use this opportunity to switch and then tackle the problem again if it shows up.

Update: After changing the stack it appears to have stopped leaking workers - that’s a positive outcome to a problem I can’t afford the time to look into in more detail. Additionally the before and after performance tests are in:

Apache:

Server Software:        Apache/2.2.14
Server Port:            80

Document Path:          /
Document Length:        20266 bytes

Concurrency Level:      5
Time taken for tests:   344.008 seconds
Complete requests:      1000
Failed requests:        47
   (Connect: 0, Receive: 0, Length: 47, Exceptions: 0)
Write errors:           0
Total transferred:      20788903 bytes
HTML transferred:       20254908 bytes
Requests per second:    2.91 [#/sec] (mean)
Time per request:       1720.039 [ms] (mean)
Time per request:       344.008 [ms] (mean, across all concurrent requests)
Transfer rate:          59.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    6   2.5      5      37
Processing:   750 1714 267.2   1668    2922
Waiting:      732 1690 265.8   1645    2900
Total:        755 1720 267.3   1673    2927

Percentage of the requests served within a certain time (ms)
  50%   1673
  66%   1778
  75%   1876
  80%   1886
  90%   2071
  95%   2238
  98%   2389
  99%   2546
 100%   2927 (longest request)

NGINX:

Server Software:        nginx/0.7.67
Server Port:            80

Document Path:          /
Document Length:        20266 bytes

Concurrency Level:      5
Time taken for tests:   304.670 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      20798421 bytes
HTML transferred:       20276909 bytes
Requests per second:    3.28 [#/sec] (mean)
Time per request:       1523.352 [ms] (mean)
Time per request:       304.670 [ms] (mean, across all concurrent requests)
Transfer rate:          66.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    7   3.8      6      43
Processing:   866 1513 266.3   1460    3900
Waiting:      848 1490 264.3   1439    3869
Total:        871 1521 266.6   1468    3905

Percentage of the requests served within a certain time (ms)
  50%   1468
  66%   1562
  75%   1651
  80%   1672
  90%   1779
  95%   1901
  98%   2104
  99%   2373
 100%   3905 (longest request)

Daily Stand Up #1

2010-09-28T00:00:00+00:00

I’ve made a commitment with myself that I’m going to write blog posts regardless of whether I feel like it or not. I’m going to start by writing the same kind of content that comes up at “Stand Ups” every morning.

For a start my day is currently sliced into working on taking sliced HTML and working it into the Rails app, that our team in currently working on, as themes. It’s giving me an opportunity to exercise my, somewhat rusted, CSS/HTML/JS skills.

I’ve also been spending some time with my operations/admin hat on. Working through the process of making sure that the actual ops team delivers what we need to launch the product. It’s all pretty run-of-the-mill stuff, a more interesting problem is the reliability of using the legacy SQL Server database over a long period of time.

We’ve been seeing errors like ‘ActiveRecord::StatementInvalid: ODBC::Error: 08S01 (20020) [unixODBC][FreeTDS][SQL Server]Bad token from the server: Datastream processing out of sync’ and ‘ActiveRecord::StatementInvalid: ODBC::Error: S1008 (0) [unixODBC][FreeTDS][SQL Server]Operation was cancelled’

Testing so far has led us to believe that it has something to do with old/invalid connections not being dropped and replaced with new connections. This may or may not have to do with the fact that the connection the SQL Server is done directly on the model(s) rather than through ActiveRecord::Base…which is in turn because we’re storing everything that we can in a MySQL database.

We’ve started working with an already customised version of the activerecord-sqlserver-adapter to improve the way that it handles these errors - by including the error messages in the array of messages that result in a reconnect. Whether this turns out to be an effective solution or not remains to be seen.

Rails security resources

2010-08-30T00:00:00+00:00

As we make the push toward releasing the platform that I’m working on we’ve installed exception_notification in our Rails app. With the increased visibility of all the exceptions it became apparent quite quickly that there were numerous hits against the server from automated vulnerability scanners. These attempts were causing routing errors as they looked for paths like ‘/user/soapCaller.bs’ - thankfully not targeting Rails applications.

The arrival of this sort of scan was not particularly surprising to me as I’ve seen similar scans in the past. I’ve even actively dabbled in some security research by running a few honeypot projects.

Even though these scans usually go after large installations such as Wordpress, Drupal, Joomla and phpMyAdmin, it isn’t stupid to take it as a reminder to keep up to date on security vulnerabilities. In the case of Ruby on Rails the starting point would be the rubyonrails-security google group and the Ruby on Rails blog.

Another great resource is Rails Inside. Rails Inside usually picks up any serious flaws and relays them to the community. In addition to this, they follow new releases of popular plugins/gems that may form part of your app. The site provides an important service because keeping up-to-date is a good way of reducing the risk of being caught by a vulnerability that has been dutifully patched by the maintainer.

The above is certainly not a complete list, so I’d like to hear if there are any other rails/ruby security resources that you find useful.

This week I learned - that bundler needs work

2010-08-01T00:00:00+00:00

One of the challenges over the last few weeks has been switching development environments to phase in the new Macbook Pros (sweet!). One of the problems has been the fact that both our rails projects use bundler.

Before I go on I should point out that @wycats was very helpful in sorting this problem out.

It ‘s not that bundler is bad directly, but when you’re using bundler on Mac but deploying to Ubuntu there are a few eccentricities - particularly with native gems.

If you’ve run ‘bundle package’ at any point in the past then bundler will have started keeping a copy of all the gems in vendor/cache. When you’re using a gem that needs certain compile flags on Ubuntu but not MacOS X that poses a problem.

The way to do this with bundler pre 1.0 is/was to install the gem to the system using the requisite build flags, and then when bundler installs the gem it takes the already installed gem. If the gem is already in vendor/cache however that doesn’t work.

Suffice it to say that after some confusion and removal of the gem from BOTH caches (vendor/cache and ~/.bundle) and then running bundle install will fixed the problem.

Incidentally bundler 1.0.0rc2 takes care of this problem by introducing the ability to add build flags to the .bundle/config file.

Ruby non-greedy matching (OR how interro saved the day)

2010-03-24T00:00:00+00:00

I’ve spend most of the afternoon working on a complex regex in order to parse command line argument forms (for lack of a better term). If you’ve ever run the man command you’ll know what I’m talking about. Take the tar command as an example:

tar  [ - ] A --catenate --concatenate | c --create | d --diff --compare
       | --delete | r --append | t --list | u --update | x --extract  --get  [
       options ] pathname [ pathname ... ]

If you’re already familiar with regular expressions you’ll know that doing something like:

[\[].*[\]]

trying to match:

[ - ]

won’t accomplish what you think. Instead of getting just the first set of braces you’ll end up with the whole remainder of the string. This is because of a feature, we’ll give it that title, called greedy matching. Greedy matching means that it takes the largest possible chunk that your regex will match, which in this case is the ‘]’ on the end of pathname.

I was aware of what was going on, but not being a particular master of regular expressions, I wasn’t sure how to get it to stop being greedy. As it turns out its quite easy.

.* - Greedy matching
.+ - Greedy matching
.*? - Non-greedy matching
.+? - Non-greedy matching

It could not be easier, once you know about it of course.

Why "Mad Libs" style signup forms are just a gimmick

2010-03-07T00:00:00+00:00

The last few days I’ve been seeing a few references to “Mad Libs”. Confused, I quickly punched the term into Google and discovered that it was a reference to something we’ve all done at school, filling in the blanks in a sentence. A couple of sites, like konigi and webmonkey had posts about HuffDuffer using a “Mad Libs” style form as their sign up form. The question was raised by a few people as to whether this was good design.

Luke Wroblewski (in collaboration with Vast.com) did some research and came up with astounding results that the conversion ratio rose some 25-40% when a Mad Libs style form was used. That’s an impressive, but meaningless number. Don’t get me wrong, I’m very happy that HuffDuffer has found something that works for them. But lets step back for a second and analyse why the results that Vast.com got might be this way.

Here’s a thought; maybe it’s just because its a completely novel approach to a signup form. But can you really see this becoming the standard way to do signup forms? How long do you think it will be before this becomes more annoying than amusing? So quit jibber jabbering about how good it is and take a look at the long term picture. Mad Libs style forms are not something you can take and apply elsewhere with guaranteed results.

To be fair, I’m biased against filling in the blanks. I hated the premise at school and I will probably continue to hate the practice until it, or I, ceases to exist.

Security wake up call

2010-03-05T00:00:00+00:00

I don’t think that I’ve had this much adrenalin pumping through my system on a Saturday morning in quite a while. My girlfriend signed into her email (as she does most mornings), to discover an email from eBay Live Support asking for a code. She called me over and a few seconds later the email vanished. What happened after that point is a blur of password resets, both her and the would be hacker trying to gain control of the hotmail and through it the eBay account.

There were paranoid moments when passwords wouldn’t work, but in the end she’s still in control of the accounts. Just about every account she owns has now had the password and details changed to help protect it.

It’s intrusive though. How exactly did the attacker break into the account? Not phishing, Cherie is well aware of those type of malicious emails. Guessing the security questions? Maybe.

That raises other concerns though, email addresses become repositories of knowledge for our online lives. Just about every account you sign up for online has to have an email account linked to it, that means many details about your online life are there in fragments. We’ll never know exactly what the hacker had access to (albeit briefly).

I’m personally going to be reviewing all my accounts to make sure that they’re secure, and I’d advise you to do the same.

Sinatra on Java

2010-02-21T00:00:00+00:00

With JRuby and Warbler it’s possible to get Sinatra, or any WebApp based on Rack, running on a myriad of different Java application servers. There are of course gotchas when it comes to using Warbler with the many different app servers, so this is a definitive guide to everything you have to do to get a simple Sinatra app running on the various application servers.

Why Sinatra?

There are examples of how to get Rails running on Tomcat and Websphere floating around the web, but I find Rails overkill for small projects. With that in mind, it’s worth looking at how to get Sinatra running on java application servers. Besides the weight of rails, Sinatra is a nice, easy to learn framework.

Start by installing Sinatra and Warbler. You don’t have to be using JRuby to install Warbler, the install will download a gem of the jruby jars.

Install Sinatra and Warbler

Lets start by installing the required gems.

$ sudo gem install sinatra warbler haml

Haml isn’t strictly required, but the template I’m going to use has views generated in haml, so if you’re following the tutorial closely you’ll want to install it.

Create a project folder (and structure)

I usually keep a sinatra project template handy. So I’m going to clone my existing template off github. You can create a more minimal example than the one I’ll download, this will get the job done though.

$ git clone git://github.com/vertis/sinatra-example.git deploy_test

TODO: Details about the project

Check that our page is displayed

$ rackup

Go to http://localhost:9292/ and you should see our default page.

Generate the warble config

$ mkdir config && warble config

Lets look at the config file that was generated:

# Disable automatic framework detection by uncommenting/setting to false
# Warbler.framework_detection = false

# Warbler web application assembly configuration file
Warbler::Config.new do |config|
  # Temporary directory where the application is staged
  # config.staging_dir = "tmp/war"

  # Application directories to be included in the webapp.
  config.dirs = %w(app config lib log vendor tmp)

  # Additional files/directories to include, above those in config.dirs
  # config.includes = FileList["db"]

  # Additional files/directories to exclude
  # config.excludes = FileList["lib/tasks/*"]

  # Additional Java .jar files to include.  Note that if .jar files are placed
  # in lib (and not otherwise excluded) then they need not be mentioned here.
  # JRuby and JRuby-Rack are pre-loaded in this list.  Be sure to include your
  # own versions if you directly set the value
  # config.java_libs += FileList["lib/java/*.jar"]

  # Loose Java classes and miscellaneous files to be placed in WEB-INF/classes.
  # config.java_classes = FileList["target/classes/**.*"]

  # One or more pathmaps defining how the java classes should be copied into
  # WEB-INF/classes. The example pathmap below accompanies the java_classes
  # configuration above. See http://rake.rubyforge.org/classes/String.html#M000017
  # for details of how to specify a pathmap.
  # config.pathmaps.java_classes << "%{target/classes/,}p"

  # Gems to be included. You need to tell Warbler which gems your application needs
  # so that they can be packaged in the war file.
  # The Rails gems are included by default unless the vendor/rails directory is present.
  # config.gems += ["activerecord-jdbcmysql-adapter", "jruby-openssl"]
  # config.gems << "tzinfo"

  # Uncomment this if you don't want to package rails gem.
  # config.gems -= ["rails"]

  # The most recent versions of gems are used.
  # You can specify versions of gems by using a hash assignment:
  # config.gems["rails"] = "2.0.2"

  # You can also use regexps or Gem::Dependency objects for flexibility or
  # fine-grained control.
  # config.gems << /^merb-/
  # config.gems << Gem::Dependency.new("merb-core", "= 0.9.3")

  # Include gem dependencies not mentioned specifically
  config.gem_dependencies = true

  # Files to be included in the root of the webapp.  Note that files in public
  # will have the leading 'public/' part of the path stripped during staging.
  # config.public_html = FileList["public/**/*", "doc/**/*"]

  # Pathmaps for controlling how public HTML files are copied into the .war
  # config.pathmaps.public_html = ["%{public/,}p"]

  # Name of the war file (without the .war) -- defaults to the basename
  # of RAILS_ROOT
  # config.war_name = "mywar"

  # Name of the MANIFEST.MF template for the war file. Defaults to the
  # MANIFEST.MF normally generated by `jar cf`.
  # config.manifest_file = "config/MANIFEST.MF"

  # Value of RAILS_ENV for the webapp -- default as shown below
  # config.webxml.rails.env = ENV['RAILS_ENV'] || 'production'

  # Application booter to use, one of :rack, :rails, or :merb. (Default :rails)
  # config.webxml.booter = :rails

  # When using the :rack booter, "Rackup" script to use.
  # The script is evaluated in a Rack::Builder to load the application.
  # Examples:
  # config.webxml.rackup = %{require './lib/demo'; run Rack::Adapter::Camping.new(Demo)}
  # config.webxml.rackup = require 'cgi' && CGI::escapeHTML(File.read("config.ru"))

  # Control the pool of Rails runtimes. Leaving unspecified means
  # the pool will grow as needed to service requests. It is recommended
  # that you fix these values when running a production server!
  # config.webxml.jruby.min.runtimes = 2
  # config.webxml.jruby.max.runtimes = 4

  # JNDI data source name
  # config.webxml.jndi = 'jdbc/rails'
end

If you tried packaging and installing this now, it would fail miserably because, the ‘init.rb’ file would not be included. The generated warble.rb only includes the following

config.dirs = %w(app config lib log vendor tmp)

In addition to this the gems that we installed above would not be install. Here is same config with the lines we need (and the other cruft removed).

Warbler::Config.new do |config|
  config.dirs = %w(app config tmp)
  config.includes = FileList["init.rb"]
  config.gems += ["sinatra", "haml"]
  config.gems -= ["rails"]
  config.gem_dependencies = true
end

Package up the War file

$ warble

From here on in we’ll be looking at any gotchas, when deploying to the different Application Servers.

Deploy to Glassfish and test (effort: moderate - working: yes)

You can get Suns open source application server from http://glassfish.org. The current stable version of Glassfish 2.1.1, though Glassfish 3 is in active development. The installer comes packaged as a jar file. You can run the installer with

$ java -Xmx256m -jar glassfish-installer-v2.1.1-b31g-linux.jar

After accepting the license it should put all the files in a folder called ‘glassfish’ in the current directory.

$ cd glassfish
You need to run the following commands to finish the setup:

$ chmod -R +x lib/ant/bin
$ lib/ant/bin/ant -f setup.xml

Once the software is installed you can start the domain with:

$ bin/asaadmin start-domain domain1

And use either the admin console or the autodeploy directory to deploy the war file.

Glassfish now has a working copy of our application.

Deploy to JBoss and test (effort: n/a - working: no)

JBoss has a community and an enterprise edition. For the purposes of this test we’ll be using the community edition. The current stable version of JBoss AS is 5.1.0 GA. You can get a copy of JBoss from http://www.jboss.org.

Getting started is as simple as unzipping the archive and running:

$ cd jboss-5.1.0.GA

$ bin/run.sh

You can then use the admin console to deploy the application. One gotcha here, the first time I deployed the application using the console I got the following nasty message:

Application initialization failed: no such file to load -- rack
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/vendor/rack.rb:1
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/vendor/rack.rb:22:in `require'
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/booter.rb:22:in `boot!'
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/boot/rack.rb:9
from /opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-rack-0.9.5.jar/jruby/rack/boot/rack.rb:1:in `load'
from &lt;script&gt;:1

Turns out that after some digging there is an open jruby bug about the issue - 63
http://jira.codehaus.org/browse/JRUBY-3935

I also did a bit of digging through the logs and found:

16:27:50,703 ERROR [STDERR] Warning: JRuby home "/opt/application_servers/jboss-5.1.0.GA/server/default/deploy/deploy_test.war/WEB-INF/lib/jruby-stdlib-1.4.0.jar/META-INF/jruby.home" does not exist, using /tmp

I’ve not managed to find a solution to this problem. I will revisit this at some point in the future. After googling a little it may be possible to just revert to a few older versions that seemed to work.

Deploy to Jetty and test (effort: easy - working: yes)

The current version of Jetty is 7.0.1.v20091125, though the version that comes as part of your Linux distro may not be so up to date. You can either install it using your favorite package manager, or if you’re on Windows get it from the homepage at http://www.mortbay.org

Once you’ve installed Jetty copy the generated war file to the webapps folder, and run:

$ bin/jetty.sh

You should be able to go to http://localhost:8080/deploy_test

Congratulations you now have a working copy of your sinatra app on Jetty.

Deploy to Tomcat and test (effort: easy - working: yes)

The current stable version of Tomcat is 6.0.20. You can either install it using your favorite package manager, or if you’re on Windows get it from the homepage at http://tomcat.apache.org

You shouldn’t have to make any changes to get our web app to work on Tomcat. Once you’ve installed Tomcat copy the generated war file to the webapps folder, and run:

$ bin/startup.sh

You should be able to go to http://localhost:8080/deploy_test

Tomcat really is the bread and butter of Java Application Servers, especially outside the Enterprise.

Deploy to Websphere and test (effort: hard - working: yes)

NB: Websphere 6.1.0.11 was the first application server I ever deployed Sinatra too, it failed miserably. I spent a long time debugging and playing with it to make it work properly. The biggest problem stems from the fact that the default way of using rack as configured by warbler doesn’t work.

Start by logging into the administration console, it should be something like - http://localhost:9043/ibm/console

Click on ‘Servers’ and when it expands select ‘Application Servers’. From here you can setup a new server instance that we can use for our testing. Call the instance something like ‘deploy01’. You can follow the default creation steps all the way through.

Once you have a server instance to test on, you can deploy a new application. The big gotcha as mentioned above is that you can’t use filters, the good news is that it’s quite easy to switch out the rack filters for a rack servlet. Rather than duplicate information that already exists, I’ll link to the place I learned to deploy warbler to websphere, http://clint-hill.com/2008/11/26/jruby-on-rails-and-websphere/.

Websphere is not the easiest application server to setup in general, but once you get it all configured it is fairly robust. Worth the effort if you want an application server you won’t have to restart constantly (as can be the case with Documentum on Tomcat).

Deploy to Weblogic and test (effort: n/a - working: no)

Setting up Oracle Weblogic 10.3.2 was nothing short of awesome. The install process is intuitive and speedy, though the size is quite large, at ~600Mb, compared to smaller cousins such as Tomcat. There is a wizard that walks you through the process of setting up your first domain, what Tomcat would call an instance and Websphere would call a profile, once the software is installed. I chose the default options for everything and had a running Weblogic server in about 20mins (including download).

My previous experience with Weblogic, was the version bundled with Documentum D6SP1. I’ve found both that version and the current fully fledged Oracle version to be a joy to work with.

Once the installation process is finished you can find the administration console at http://localhost:7001/console, you can then login using the username/password you picked during installation.

From the admin console it is a simple matter of clicking on ‘Deployments’ on the left and then when the screen loads clicking ‘install’, browse to the directory with the deploy_test.war file in it and start the install.

You should now be able to access the deployed application at http://localhost:7001/deploy_test…

Application initialization failed: no such file to load -- rack
from C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/vendor/rack.rb:1 from
C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/vendor/rack.rb:22:in `require' from
C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/booter.rb:22:in `boot!' from
C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/boot/rack.rb:9 from
C:/Oracle/Middleware/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/deploy_test/qwtgi/war/WEB-INF/lib/jruby-rack-0.9.5.jar!/jruby/rack/boot/rack.rb:1:in `load' from

Clearly thats not going to be the case though. This issue is very similar to the error message received for JBoss.

Conclusion

Not a bad scorecard really. Of all the Java application servers that I tested, only JBoss and Weblogic proved to be a problem. I’ll be retesting these two periodically to see if support has been improved (there are open tickets with JRuby). Until then I hope that this has been useful.

Exploring Documentum RESTful Services - Part 2

2010-02-19T00:00:00+00:00

I originally published this post to the Early Access Area for Documentum RESTful Services.

In the second part of my exploration of Documentum RESTful Services I promised that we’d delve into browsing around the docbase. Rather do that with ruby I thought I’d grab a copy of JQuery and have a look at what it takes to access the services using what is an increasingly popular javascript library. The most important part of delivering on this is the JQuery call:

$.getJSON(path, ...);

The important thing here is that because of concerns about cross-site scripting we can only call local paths. While getJSON allows remote paths now with the help of JSONP style callbacks, those require server side cooperation to work. I don’t know that they’re not implemented in Documentum RESTful Services, but flipping through the documentation I couldn’t find anything of that nature.

Rather than creating a War for one html file, I decided to show off nginx. Nginx is a superb webserver, and more importantly in this case, a reverse proxy. It has gotten a lot of attention in the Ruby on Rails community, which is where I fell in love with it. After grabbing a copy of nginx it’s simply a matter of doing a minor adjustment to the nginx.conf file.

location /resources {
  proxy_pass   http://127.0.0.1:8080/resources;
}

This will mean that anything below the /resources directory on our webserver is passed off to the tomcat instance on 8080. We can start nginx, and discover it works perfectly. Now that we can use just a path to reference the services, lets get started. We start with a fairly blank html file:

 <html>                                                                 
 <head>                                                                 
 <script type="text/javascript" src="jquery-1.3.2.js"></script>         
 <script type="text/javascript">                                        
   // we will add our javascript code here
   $(document).ready(function() {
        // do stuff when DOM is ready
   });  
 </script>                                                              
 </head>                                                                
 <body>                                                                 
   <div id="results"></div>                                         
 </body>                                                                
 </html>

Notice the included jquery.js, the call the $(document).ready, and the results div. We’re going to use the call I mentioned earlier ‘getJSON’ to populate the results div with the results of calling the ‘folders’ resource with no arguments. Lets look at the code.

$.getJSON("/resources/core/repositories/test_repo/folders.json",
       function(data){
       $.each(data.dataPackage.dataObject, function(i,item){
           $("#results").append(item.properties.object_name+"<br/>");
       });
});

If you put the above html file in the right place and run the example you’ll be prompted for the password, just like you would if you actually went to the resource endpoint itself, then you’ll see a nice list of the root cabinets. Suppose we want to more than just display the root cabinets though, that we want to use the returned results to allow us to click around the docbase. Here is an example that does just that:

  <html>
  <head>
  <script type="text/javascript" src="jquery-1.3.2.js"></script>
  <script type="text/javascript">
      function getItems(path) {
           $.getJSON(path,
                function(data){
                  $.each(data.dataPackage.dataObject, function(i,item){
                     $("#results").append("<a href=\"#\" onclick='link(\""+item.relationshipsUri+"\");'>"+item.properties.object_name+"</a><br/>")
                  });
                });
      }

      function getRelationships(path) {
           $.getJSON(path,
                function(data){
                  $.each(data.relationship, function(i,item){
                     $("#results").append("<a href=\"#\" onclick='link(\""+item.target.relationshipsUri+"\");'>"+item.target.properties.object_name+"</a><br/>")
                  });
                });
      }

      function link(url) {
           //Needs to be a local path...this is a hack
           //http://127.0.0.1:8080
           var path = url.substring(21, url.length);
           alert(path);
           $("#results").html("");
           getRelationships(path);
      }

      $(document).ready(function() {
           getItems("/resources/core/repositories/test_repo/folders.json");
      });
  </script>
  </head>
  <body>
      <div id="results">
      </div>
  </body>
  </html>

It’s very rough, and doesn’t take into account many variables, such as relationships that are returned aren’t necessarily folders. But it does illustrate the ability to access Documentum using libraries like jQuery now.

Copy the structure but not the data.

2010-02-09T00:00:00+00:00

I’m working on a prototype at the moment that requires me to insert data into offline tables (offline as far as Documentum is concerned). The examples that I’ve found all resort to specifying the exact structure of the table.

create table DMI_OBJECT_TYPEx (

R_OBJECT_ID VARCHAR2(16) NOT NULL,

I_TYPE NUMBER(10,0) NOT NULL,

I_PARTITION NUMBER(10,0) NULL);

The example above is smaller than most of the tables I have to create. The weakness with this is that you have to look up the table structure. The writers probably chose this method because the much simpler syntax shown below also brings any data that is in the table.

CREATE TABLE DMI_OBJECT_TYPEx AS SELECT * FROM DMI_OBJECT_TYPE;

My initial thought was why not copy the table and then just truncate it, but after a bit of searching I stumbled upon the solution. Essentially adding a WHERE clause to the end of the query that never evaluates to TRUE enables us to take the structure without the data.

CREATE TABLE DMI_OBJECT_TYPEx AS SELECT * FROM DMI_OBJECT_TYPE WHERE 1=2;

Lighter weight deployment with git-deploy

2010-02-01T00:00:00+00:00

Up until now I’ve been using a fairly standard capistrano deploy.rb. The problem is that as we speak I’m trying to deploy a patch from my Windows work machine and it’s not working. The ethics of fixing one of my project from home aside, this is a problem. When I try and deploy the following error comes up

can't convert Net::SSH::Authentication::Pageant::Socket into IO (TypeError)

The change I was trying to deploy stops in its tracks. After quite a bit of searching I found a thread about the error that dates back to Capistrano 2.5.3… from 2008. What a shining example of open source.

To be fair the error isn’t necessarily in Capistrano, it may in fact be in Net::SSH. The sad thing is that we’re over a year later, and nothing has been done to fix the problem. Is the number of people that use Ruby from Windows machines so low that no-one has managed to fix it in a year?

I’m not expecting Jamis to fix it. I get that he has too much to do, and doesn’t have time to give out a bunch of freebie support, but I’m now faced with the choice of either trying to fix the problem, a task I don’t currently have time for, or ditching capistrano.

I’ve considered switching to heroku in the past and just never made the leap. For a start the app in question has to many moving parts for heroku. One thing I did like though was the notion that to deploy all I had to do was ‘git push target master’ and the app would be updated and deployed.

After toying around with rolling my own solution, I stumbled upon mislav’s gem. It lacks some of the features that I’m looking for, but its a good deal closer to the level that I need. It lacks the bloat of capistrano, which is important, because the biggest barrier to me getting in and fixing capistrano would be the size of the library and knowing where to start.

I very quickly migrated my existing application to use git-deploy. It’s not perfect for every problem, particularly if you’re doing multi stage deployments, etc, but at least I’ll be able to do a deployment everywhere I can get access to git now.

RockYou gets rocked by hackers

2009-12-17T00:00:00+00:00

(And I’m hilarious)

Seems that simple lessons don’t get learned. Don’t get get me wrong, its very hard to protect every aspect against hackers who try to pry they’re way into your site. Storing passwords in plain-text is just dumb though. Even if the passwords for your own site are hashed, the proliferation of storing third party login details (which you could still encrypt with a symmetrical key) is a time bomb.

RockYou is just the latest site on the internet to learn this hard lesson. Supposedly the hacker is one of the good guys, but there is no guarantee that someone else didn’t get the information as well. It’s an argument for doing away with passwords altogether, how long will it be until we can use public/private key authentication with websites. It is now accepted best practice with SSH, since the advent of widespread SSH bruteforcing.

Private key authentication solves a lot of the problems with websites storing password information, the hacker would have gained nothing besides the ability to verify users were who they claimed to be.

ActiveDocumentum

2009-11-30T00:00:00+00:00

I’ve finally decided to release ActiveDocumentum. ActiveDocumentum is a Ruby Gem that I created to bring some of the goodness, learned by pulling apart ActiveRecord, to accessing Documentum. Its nowhere near as mature, but I has been doing the job pretty well so far for the scripts and sites I’ve been using it for. It has a dependency on JRuby because it hooks into the DFS client libraries to facilitate connecting to Documentum. I’m going to post some further examples, but for the time being, here it is.

http://www.github.com/vertis/active_documentum

You’ll need a Documentum repository, JRuby and a copy of the DFS sdk to play with it.

Update: I’ve also done up a quick sample which you can find on my github (right next door to the actual library).

http://www.github.com/vertis/active_documentum_sample

Kilroy was here

2009-10-28T00:00:00+00:00

Myth has it that ‘Kilroy was here’ was the mark of a shipbuilder before/during WWII, and that it was picked up by Servicemen who found it amusing that the mark was made in supposedly unreachable places. They then spread the mark elsewhere, as locations around the world were liberated. I love this concept, because I’m not a fan of limitations – I’m an avid gamer, but I rail against games that don’t let you stray from the path. There is no variety, or replay value because each time you have exactly the same sequence of events.

Many hopeful web developers fall into this pitfall as well. I was reading http://www.adaptivepath.com/blog/2009/10/27/desire-lines-the-metaphor-that-keeps-on-giving/, a(nother) look at a metaphor called ‘desire lines’, that has similarly important lessons for the web2.0 community. You can’t force people to do what you want. The lesson is simple and played out all around us, you can set down a path, but you can’t make people want to use that route. A good designer will probably preempt a lot of what people will want, but not everything, and the overnight success that all of us are searching for is a bit hit and miss. Plenty of good ideas are out there, but finding something that people actually want, and then getting the word out there that you’re product is there is tough.

Or something like that.

Getting started with Google Maps

2009-10-25T00:00:00+00:00

My first few searches turned up some fairly unpolished methods of accessing Google Maps from Ruby. I’m definitely using Ruby as a starting point in this challenge, its the language I’m most comfortable with. That’s not to say that I’m not going to do any other languages. I do want to do a Python/Django project at some point, just to get it out of the way.

After bit of digging around I discovered the YM4R gem. YM4R provides access to the Google Maps and the Yahoo! Maps Building Block API’s. Before we go any further lets just get a Google Map on the page and point it at a location. The gem also comes as a plugin for rails, which I’m about to find out has some different capabilities (more on that later).

To do that we need to get an API key here. Making sure to put in whatever url you will be using to access your application, in my case, at least for the example application, that’s http://localhost:4567. If you’re familar with the port number then you know that I’m using Sinatra rather than Rails, this means, is that I can’t use the YM4R/GM rails plugin (at least without hacking it up) & unlike the plugin the gem doesn’t contain the code to actually generate the map code. So I’m back to the drawing board.

Taking a step back I have a look at how difficult it is to just get it onto the page without a helper, which as it turns out is ‘not very’. Lets look at the code to get a minimal example happening:

# app.rb
require 'rubygems'
require 'sinatra'

get '/' do
  haml :index
end

And the view:

-#views/index.haml
%html
  %head
    %script{:type => "text/javascript", :charset => "utf-8", :src => "http://www.google.com/jsapi?key=ABQIAAAAR21mr2vnfC9-sjIojad2WhSmbtbI58sJnUq1AueY0BvTVoVv3BSw-I1OHpTaa0zZiaSEsDrZf9fGWQ"}
    %script{:type => "text/javascript"}
      google.load("maps", "2");

      // Call this function when the page has been loaded
      function initialize() {
      var map = new google.maps.Map2(document.getElementById("map"));
      map.setCenter(new google.maps.LatLng(37.4419, -122.1419), 13);
      }
      google.setOnLoadCallback(initialize);
  %body
    %h1 Google Maps Example
    %p Welcome to the the Google Maps API example application.
    #map{:style => "width: 794px; height: 491px"}

It’s pretty simple to get a nice looking Google Map onto the page. In the next post I’ll go into the details of how to start manipulating the map for our purposes.

Foursquare

2009-10-23T00:00:00+00:00

Apparently the idea for my Google Maps API part of my challenge has already been done by foursquare. Dammit, that was a good idea too. They’ve taken it to the next level as well, making it possible to complete badges (much like Stackoverflow and/or Kongregate). It’s a bit limited, in that you can’t have it in every city right now, but it should be fun to play with and might get a solid following from people with smartphones and people that love the whole geo-tagging phenomenon. Oh well, back to the drawing board, this was probably a bit complex for a week long project anyway.

The API Challenge

2009-10-22T00:00:00+00:00

One of my favorite finds this year is Programmable Web, a site dedicated to APIs and Mashups that are the lifeblood of web 2.0. Having just watched the movie Julie & Julia, which I highly recommend, I think that its only appropriate that I set myself a similar challenge. Take the top 50 apis, and within the next year develop a non trivial application for each of them (in no particular order). I’ve set myself a couple of rules:

You are allowed to use any client libraries if they exist.
Using multiple APIs in the one application is allowed (yay for mashup's)
Any libraries/applications that are developed in the course of the challenge must have tests/specs and documentation.
And of course blog about each api.

First up:

API: Google Maps
Deadline: Thursday the 29th of October

You’ll be hearing from me soon. In the meantime I welcome anyone that wants to participate. Just leave a comment on this post, with your details. If it proves to be popular then I’ll get some kind of signup/participant website going.

Transport out has not been set

2009-09-17T00:00:00+00:00

I’ve was banging my head against the wall all yesterday evening trying to work out why Axis2 was erroring out with ‘Transport out has not been set’. All my searching for similiar problems was returning the same mailing list entries, replicated all across the internet, none of them having anything useful.

For me at least (and some of the problem is that it can be caused by more than one thing), was that I’d commented one URL out when I went home and created my home one, without including ‘http://’ on the front of my URL.

Rack - NoMethodError: undefined method `call' for nil:NilClass

2009-09-09T00:00:00+00:00

Turns out Rack doesn’t read minds. I just spent the better part of the morning troubleshooting a stupid error, convinced there was something wrong with JRuby/Rack/Sinatra. As it turns out the error was caused by having a blank config.ru file, I was editing one that was in a different directory (and believing that it was the correct file). My own stupidity aside, the error message wasn’t helpful, mostly because google didn’t return any results related to the same error. So while this admits my mistake, hopefully the next person who comes across this error will spend less time scratching their heads.

NoMethodError: undefined method `call' for nil:NilClass
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/lint.rb:35:in `call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/showexceptions.rb:24:in `call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/commonlogger.rb:20:in `_call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/commonlogger.rb:13:in `call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/content_length.rb:13:in `call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/chunked.rb:15:in `call'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/rack-1.0.0/lib/rack/handler/mongrel.rb:61:in `process'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/mongrel-1.1.5-java/lib/mongrel.rb:159:in `process_client'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/mongrel-1.1.5-java/lib/mongrel.rb:158:in `each'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/mongrel-1.1.5-java/lib/mongrel.rb:158:in `process_client'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/mongrel-1.1.5-java/lib/mongrel.rb:285:in `run'
C:/development/langs/jruby-1.3.1/lib/ruby/gems/1.8/gems/mongrel-1.1.5-java/lib/mongrel.rb:285:in `initialize'
127.0.0.1 - - [09/Sep/2009 11:24:45] "GET / HTTP/1.1" 500 40410 0.1720

Lotus Notes 8 Development

2009-07-30T00:00:00+00:00

I’m finally finished beating my head against the wall trying to get drag and drop to work in Lotus Notes 8. It turned out to be a good deal less friendly than I expected.

The actual core of drag and drop listening is stock Eclipse RCP. That’ll trigger off the events as expected, but I found that it wasn’t passing anything across. After digging down as far as I could to see if it just wasn’t processing the transfer properly, I concluded that it was a dead end.

The old way of dealing with this would have been to use NotesUIWorkspace to get the currently selected document. That was never implemented in Java (which is probably because it’s not needed). Because of the eclipse architecture, it is strictly possible to access whatever you want using the standard eclipse classes. Thankfully, we don’t even have to worry about going this deep, I stumbled across an open source plug-in called Formul8 by Jeff Gilfelt. Jeff’s plug-in allows you to execute formulas in the context of the currently selected Document(s), which is similar to what I needed.

Under the hood Formul8 is powered by ‘com.ibm.lotuslabs.context’. Armed with that discovery, I headed back to google, finding this post about how to use ‘lotuslab.context’, in addition, it also links to a copy of the source code (yippee!).

Exploring Documentum RESTful Services - Part 1

2009-07-08T00:00:00+00:00

I was chatting to Craig Randall on twitter a little while back and he let me know that they were just about ready to go into Early Access for their new web service platform - Documentum RESTful Services. I was excited then, and I’m even more excited now that I’ve had my first taste.

RESTful web services are hard to explain to people. I usually fall back on explaining them by linking them to Ajax, which is a Web2.0 technology that people are more familiar with. If you wanted to use Ajax to make your Documentum webapps more friendly before Documentum RESTful Services, then you’d be out of luck (short of creating a custom server side component).

RESTful web services are easily accessible because you don’t need any funky client library to access them. Pop open any web browser and type in:

http://localhost:8080/dctm_rest/resources/core/repositories/<RepoName>/folders

And you’ll get back an XML representation of all the root folders (after you’re prompted for a username/password). RESTful web services use only standard HTTP, which means that you’re cutting out the middleman of having to encode requests/responses in Soap (or some other container).

They’re automagically more language agnostic. Gone is the need for a complex client side library. For example, while it should strictly be possible to access DFS in any language, this doesn’t translate into a reality. So Documentum RESTful Services will allow us to step away from relying on Java/.Net and expand into using high level dynamic languages (python, ruby, javascript).

Let’s do a simple example in Ruby to demonstrate:
<pre class="ruby">require ‘rubygems’
require ‘httparty’

class DctmRest
include HTTParty
base_uri ‘http://localhost:8080/dctm_rest’
def initialize(username, password)
@auth = {:username => username, :password => password}
end

def repositories
options = { :basic_auth => @auth }
repository = “test_repo”
self.class.get(“/resources/core/repositories.json”, options)
end
end
d = DctmRest.new(‘username’, ‘password’)
puts d.repositories.inspect</pre>
The example is probably more complex than it even needs to be. If I look at the output:
<pre class="js">{“repository”=>[{“queriesUri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/queries.json”, “repositoryType”=>“managed”, “typesUri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/types.json”, “foldersUri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/folders.json”, “checkedoutUri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/objects/checkedout.json”, “uri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo.json”, “name”=>“test_repo”, “objectsUri”=>“http://localhost:8080/dctm_rest/resources/core/repositories/test_repo/objects.json”}]}</pre>
I can see a very succinct set of other URIs that I can use to get other information. In the next post I’ll go through how to browse around a Docbase using Documentum RESTful Services. For now, take the time to sign up for the early access program, so you can help give feedback on what is a giant leap forward for Documentum in the web2.0 world.

Rails Rumble 2009

2009-07-07T00:00:00+00:00

Unlike last year where I registered at the last minute and went it alone, I’ve been hanging out for this years dates to be announced. Now I’ve got a few days to put together a team and decide on an idea. You can find more information at the Rails Rumble 2009 website.

Installing Documentum Administrator 6SP1 (Patch Release) on IBM Websphere 6.1.0.23

2009-06-12T00:00:00+00:00

I’ve spent the majority of the last 2 days trying to get DA 6SP1 Patch Release to install properly on IBM Websphere 6.1.0.23. It had been a while since I’d last done the install, and I fell into the same traps as last time, forgetting where to set the class loader and so forth. After getting past these I was left with the application unhelpfully still showing a white screen and error messages that revealed (looking back) very little about the true nature of the problem. Having finally conquered the problem, I thought I would provide the steps necessary so that anyone else trying to get a WDK application working on Websphere as at least one more thing to try.

Grab a copy of da.war from the EMC Powerlink Download site.

Unpack da.war and add the following files to a directory
Put dfc.properties into WEB-INF/classes (As described in the deployment guide)
ibm-web-ext.xmi & ibm-web-ext.xmi goes into WEB-INF - https://solutions.emc.com/emcsolutionview.asp?id=esg91105
Find a copy of xml.jar on the Websphere App Server in 'Websphere\AppServer\java\jre' and put in 'WEB-INF\lib' - https://solutions.emc.com/emcsolutionview.asp?id=esg93106
Install the application using the IBM Administration Console.
Change the order the classes are loaded in by expanding the Applications node and click the Enterprise Applications node. Select the application called da_war (or equivalent wdk app). Click the Manage Modules link. Click the da.war link. In the Class loader order drop-down list, select the option 'Classes loaded with application class loader first' - https://solutions.emc.com/emcsolutionview.asp?id=esg93177
You can then save and start the application.

The xml.jar file is appropriate for D6SP1 only, previous D6 versions have a slightly different set of instructions (see linked support note). Any feedback, corrections, suggestions greatly appreciated.

ActiveDocumentum teaser #1

2009-05-07T00:00:00+00:00

Having stepped away from trying to get Documentum running on linux, I’ve been working on bringing the permissions through into our custom Lotus Notes/Documentum client. At the moment when a user imports a document to a given folder it does no permission checking before the fact, just errors nastily when it can’t write the document. A short stop by Prasad’s blog gave me a query to start from.

SELECT i_all_users_names as users FROM dm_group
WHERE group_name IN (SELECT r_accessor_name FROM dm_acl
WHERE object_name IN (SELECT acl_name FROM dm_sysobject
                       WHERE r_object_id = '0c0xxxxx80009e16'))
ORDER BY i_all_users_names

This is superb, but it doesn’t actually go far enough. If I was to attach it to my application and tell it not to let me import anything into the folder if my name wasn’t on the list I wouldn’t be able to copy anything into my home cabinet. It enumerates the members of any group within the ACL but doesn’t deal with named users & also the owner.

This will give me a chance to show off what you can do with my ActiveDocumentum project. ActiveDocumentum is a working name for a JRuby gem written to provide functionality similar to what can be found in ActiveRecord, with the addition of repeating attributes and other DQL specific items. While the final code for the below function may end up being in Java or .Net. Doing it using ruby allowed me to very quickly scope out the logic of what would need to be done to get the permission that a user had on an object (NB: If there is a way to do this simply with DQL I’m not aware of it):

require 'config/environment'

def get_effective_permission(r_object_id, user = nil)
  return 0 if user.nil?
  folder = DmFolder.find(:first, :columns => "owner_name, owner_permit, acl_name", :conditions => {:r_object_id => r_object_id})
  current_level = 0
  if folder.owner_name.eql? user
    current_level = folder.owner_permit
  end
    acls = DmAcl.find(:all, :columns => "r_accessor_name, r_accessor_permit, r_is_group", :conditions => "object_name = '#{folder.acl_name}' ENABLE(ROW_BASED)")
    acls.each do |acl|
      if acl.r_accessor_name.eql? user
        current_level = acl.r_accessor_permit if acl.r_accessor_permit > current_level
      elsif acl.r_is_group.to_s.eql?('1') and not acl.r_accessor_permit.nil?
        group = DmGroup.find(:first, :columns => 'r_object_id, i_all_users_names', :conditions => {:group_name => acl.r_accessor_name})
        current_level = acl.r_accessor_permit if acl.r_accessor_permit > current_level and group.i_all_users_names.include?(user)
      end
    end
  end
  return current_level
end

r_object_id = '0c0xxxxx80009e16'

user = 'John Doe'

puts get_effective_permission(r_object_id, user)

As you can see the above function takes a user and returns the highest permission that they have, by checking if they’re either directly listed on the ACL or in one of the groups on the ACL. The bottom line is that with ActiveDocumentum I can write less lines of code for one off scripts that I have to create, and include complicated logic that is lacking in straight DQL scripts. I will be posting more details on this in the near future.

My ongoing battle to get Oracle working on CentOS 5.3

2009-05-04T00:00:00+00:00

Oracle Base has a fantastic article on how to get Oracle 11g Release 1 working on Red Hat Enterprise Linux. The fact that CentOS is a stripped version of RHEL is a good reason to be using it, because if there is one Linux that’s likely to be used in enterprise situations then it would be RHEL. All up I’m less than happy with Oracle and Documentum’s Linux installers. I know that they’re ports of what are essentially Windows products, but the fact is that if you’ve going to bother writing something in Java(Documentum) then you should at least make sure that it really is portable, otherwise why not just use C/C++ and be done with it.

The main Oracle install I struggle to understand, since the Oracle XE install was so painless, zero to database in no time. It clearly doesn’t fit well into the Linux way of doing things though:

It doesn't install services/scripts to /etc/init.d (or equiv).
It doesn't setup users for you, i.e. install as root, run as oracle (yet it requires root privileges to do two tasks).
Requires SELINUX to be disabled.

Delving into changing memory limits and such is another matter, I think it may be better to leave those to the user, just because they can fundamentally change/break the system. Ah, to go back to PostgreSQL, which I once thought difficult compared to MySQL.

D6.5 Experiments (Part 2)

2009-05-04T00:00:00+00:00

I’ve been trying to do an install of Documentum using the Linux/Oracle downloads without much success. This may well be more on the Oracle side of the fence, but this started when I put Oracle XE(10.2.0.1-1) on a newly installed CentOS 5.3 virtual machine, and then tried to get Documentum (6.5SP1) installed and configured, only to get the majority of the way through and be told that the minimum is Oracle 10.2.0.3, despite what the release notes say.

Since 10.2.0.1-1 is the latest Oracle XE, I concluded that it must not be supported and moved on to trying 11.1 (Standard One). The biggest downside to this is that this Oracle installer is far less friendly than the XE installer, and at the end of it, I am not left with a TNSNAMES.ORA which Documentum needs. I am not an Oracle expert by any means, I would prefer to use PostgreSQL or MySQL, but they’re not a supported configuration. Even ignoring ‘supportability’, since this project is only experimental anyway, there is no information on whether it is even possible to get it running using these databases. Since there is a large amount of mapping/optimization between DQL and the underlying SQL I would tend towards it not being possible.

Does anyone have any advice or instructions for getting D6.5SP1 installed from scratch on linux? Failing that I will persist with my attempts, and let you know how it turns out.

Zillow(?) vs. Social Media

2009-05-01T00:00:00+00:00

If you’d said the word Zillow before lunch today I would have looked at you strangely. At lunch my iGoogle home page revealed a travesty taking place. A user of Hacker News was asking for advice regarding Zillow offering $250 for the his domain (and threatening to cut off API access). It would seem from reading the thread, that Zillow (a real estate site), had decided that Auston Brunson (and his domain iZillow.net) had out lived their usefulness.

iZillow has an iPhone app up on the app store that suddenly became a competitor to Zillow’s own newly released iPhone app. To make things worse there is clearly trademark issues, not to mention that Auston’s app relies on a public API supplied by Zillow. The advice from Hacker News users ranges from “time to play hardball” to “lets start a negative PR campaign on twitter (#zillowfail)”.

Why shouldn’t Zillow shutdown someone that is infringing on their trademark? After all, what right does iZillow.net have to use the name? If it was a clear cut case of trademark infringement, then I wouldn’t be posting this, I’ve had these kind of discussions with Etsy about my former site EtsyTools (the death of which had nothing to do with trademarks). Without going into to much detail, Etsy was outstandingly reasonable in the protection of their trademark, since it’s a case of protect their trademark or risk losing it (so fair enough).

It seems like from Zillow’s perspective at least it is a clear cut issue. A comment from Drew Meyers (representing Zillow) has suggested that this is indeed what the communication is about.

Drew Meyers on April 30, 2009 11:34 pm I've been in contact with Mr Bunson about the issue being discussed on Hacker News. I don't think it's appropriate to discuss our conversation in public, however, I will say that what is described here is not a complete representation of the conversation. Mr Bunson is welcome to continue to use the API as long as he doesn't infringe upon our trademark and use our brand name "Zillow" on his website.

I would agree with this except its taken them a whole year to get around to ‘protecting’ their trademark, and it comes suspiciously around when their releasing their own iPhone app. There is nothing stopping Zillow from using trademark and API access to effectively shutdown their competitor, that doesn’t make it right. Whether he’s right or wrong in his actions, Auston has built up a user base, and invested a large amount of effort into the site.

I can’t help but wonder if going to Hacker News will help or hinder. Certainly the wrath of an incensed mob is nothing to be ignored, but by bringing it out Auston Brunson may have aggravated Zillow. I certainly hope that Zillow will reconsider it’s initial position.

Update:

David and Drew (from Zillow) have been very agressive in maintaining that this is just a trademark issue. They also maintain (this has been confirmed by Auston) that the $250 figure was Auston’s number. According to Auston they offered “a couple of hundred bucks”.

It’s not, and it never will be just a trademark issue, not to Auston anyway. He’s been running iZillow.net since May last year (and all the sweat and blood that comes with that). Has built up a user base & links from the rest of the internet (NYTimes, blogs, etc). Claiming it’s just a trademark issue, will give little comfort to anyone else using Zillows API. The timing is just too close to the release of their own product.

Regardless of who put the figure on the table, I would like to see Zillow rise above and put a large figure on the table. They need to think about the impact that they’re having on an individual.

RE: Rails is not a ghetto, it's a train station.

2009-04-29T00:00:00+00:00

I accept planetmcd’s criticism of my previous post. I’m aware that I’m less than eloquent and my arguments less than logical at times.

I can’t say for sure whether anyone has ever done a presentation like that here. I do know that something like that wouldn’t be accepted not just because of the images (sexual or not), but because it doesn’t conform to corporate standards. Ruby/Rails/Web2.0 has no such standards, more, the culture is one of being risky, on the edge, and of pushing the limits.

There are probably many ways that it could have been done better, but it wasn’t. The problem stems from not going to Matt, and expressing that they didn’t like the presentation. They could have suggested using ‘Fragstar’ next time (via Renae Bair). Instead choosing to drag Rails through the mud publicly, “Here is a professional community that doesn’t respect women”.

I’m aware that Matt has defend his position, and the DHH may have made it worse, but I don’t condone the method this was approached in the first place. It’s sensationalist and unnecessary. Do people actually think they’ve improved the community by acting in this manner?

The dress code is only one facet of what I was trying (albeit poorly) to express. If you asked a programmer whether he would prefer to wear jeans and t-shirt or suit and tie to work which would he pick? What is the dress code at the Web2.0 development houses (not having worked at one I don’t know)? If it is jeans and t-shirts then that workplace is different to mine. My current employers wouldn’t consider them very professional either - This is where it goes to the heart of the community. You can do development the traditional, non agile way, any time you want to put on a suit and tie and forget you know techniques like metaprogramming/bdd/tdd (and don’t forget how to use windows, because that’s what corporate professionals use).

I have worked for a few industrial clients where staff had nudes as desktop wallpapers (we’re not talking partial nudity either), and pinups scattered around the sheds. There were certainly females around, though how they felt about it never came up. They would consider themselves professional, in that they provide top notch solutions to their clients. Warranties & quality assurance, etc. I doubt my current employer would find them very professional either, sweaty, greasy, and not very formal.

One of the things that I’ve heard raised when Australian corporate entities deal with overseas counterparts, is that we’re a good deal less formal and respectful than they are. Socially and culturally Australians are more laid back, some might say unprofessional. Different people are always going to have differing opinions on professionalism, I find it unlikely that Matt felt he was being unprofessional in using the pictures and analogy that he did. I would hazard he still doesn’t feel he was unprofessional, though he undoubtedly realizes that it was a mistake.

Rails is not a ghetto, it's a train station.

2009-04-28T00:00:00+00:00

Before I launch into my response to Peter Szinek, you need to know some background. Matt Aimonetti made a presentation at GoGaRuCa on CouchDB, which included sexual references and supposedly explicit images. I’ve looked over the presentation (which may or may not have the same content as the one at the event) and I have to say that I found it to be pretty tame considering the hype that had spread before it. When I first read the story (found here and here) it was framed in ‘how to scare women away from your development community’. DHH and probably many others, responded strongly to the controversy. I’m particularly fond of Renae Bair’s post.

This morning when I got to work I was greeted by another story…about how rails is still a ghetto. I have to disagree with the argument. Yes if you go to India and go to a Hindu temple you should take off your shoes, and you should be refined enough to KNOW to take off your shoes, either that or the shoes at the door should give you the heads up. This line of reasoning isn’t relevant for a rails conference, because there is no guideline anywhere that says, ‘should not show pictures of that nature’. It may well have been a bad decision, but there without precedent, how was he to know.

This is where the argument that professionalism comes in. I certainly wouldn’t dare using a presentation like that at my work, because they’re card holding accountants and lawyers. The development community (in particular web 2.0) has a reputation for riding on the edge, this is not just rails (which has been pointed out by others), but many other frameworks/languages that consider themselves groundbreaking. Clearly there is a disconnect here, between being passionate and fun, and relaxing from corporate strictures. Professionalism should have had everybody there wearing a suit and tie, but I would bet they weren’t.

There has been plenty of coverage about this less than newsworthy event, so I’ll not waste any more time on it. Another couple of days, and it will have disappeared from reddit, digg, etc & and will be forgotten. The majority of people that use Rails don’t care about posturing and hype that people are putting so much effort into…that’s just a way of big noting blogs and looking important. Rails isn’t a ghetto, if anything it’s a train station, with a lot of people getting on and going places and a few people that feel the need to hang around and cause trouble. I’m not saying you don’t have a right to dislike his presentation, but don’t include the rest of us. Don’t pretend like he did something wrong just because you don’t like it. If you don’t like the way he did the presentation put together a better presentation and you be the one standing on stage.

On that note I’m going back to work.

D6.5 Experiments (Part 1)

2009-04-28T00:00:00+00:00

Spent most of the night downloading Documentum 6.5 components for Linux & Oracle. While I would prefer to use an open source database like PostgreSQL as the backend, the actual database isn’t going to matter much once its installed (they don’t offer it as an alternative anyway). What I’m looking to do is have a D6.5 environment that I can tune to be as fast as possible. This includes doing things that you wouldn’t be able to do on a System that has to be supported, such as using Nginx as the front end instead of Apache Httpd with Apache Tomcat.

I’m still a bit up in the air as to which App Server I’m going to use. Tomcat is the obvious answer and the starting point for clients that don’t want to invest in IBM Websphere, BEA(Oracle) Weblogic, or Jboss. I do think that Apache Geronimo and Jetty are worth investigating as alternatives. One of the complaints that I’ve heard against Tomcat in production environments is that it is ridiculously unstable when running Webtop. With the app servers needing to be restarted every week (or more frequently). IMO there is more likely an issue with Webtop itself rather than Tomcat. I haven’t heard of stability complaints when running other webapps.

One of the big complaints against Apache Tomcat is that it isn’t a full stack App Server like IBM Websphere, for instance architecturally IBM Websphere has a lot more under the covers. I’ve not installed Apache Geronimo before but it’s supposed to be more of a complete App Server. I’ll let you know how my project goes anyway, got to run off to work now.

IS-t in an Economic Crisis

2009-04-09T00:00:00+00:00

Information Solutions - technology, is a simple way of expressing that the IT department is there to serve the needs of the client facing business. The current enterprise that I’m working at is the first massive global organisation that I’ve worked with. 130,000 odd employees throughout the world, with about a quarter of that in the APAC region.

I’m proud to say that the IS-t teams that I work with on a daily basis are excellent people, as are the people from other departments that I’ve had dealings with. Honest hard working people, just trying to do the best they can at their jobs.

Belief and evidence led me to believe enterprises move pretty slowly. Despite that things can change pretty rapidly up here, there is a level of adaptability that I’d not expected to find. A mission that IS-t should be an enabler for change, not an excuse that it is all too difficult. Things have been booming in the last ten or so years. A colleague has a graph on his wall showing the steady climb in the amount of employees that have been added to the ranks in Australia.

There was money, and there probably still is, but something changed a little while back. The negligence of the US property market (and other factors I don’t try to fathom) has sent the world spiraling into a recession. All of a sudden the extravagance of the last decade is thing of the past.

Business class flights are out the window for all but the highest of managers, and furthermore, travel itself only happens as an absolute necessity. Perhaps that’s the way it should have been all along. On Tuesday we had the largest video conference that the IS-t department has ever done. People from the Infrastructure side of IS-t came together in a massive video conference that brought the whole APAC region into the one room. It wasn’t free but it was a damn sight cheaper than flying them here, or having them continue to work on their own and duplicate resources.

After all the scene setting I’ve done above, this is the real meat. There are steps being taken to eliminate the duplicity that has been taking place. No longer will there be 5-10 efforts to manage the desktops and laptop images (the standard set of software installed on machines), and why should there be? Sure it means a lot more work for the people in the Desktop team, moving from managing the computers of 5000 people to 30000 people, but there is an economy of scale. One or two people are assigned to create the image (or fix the image), and the people from other regions that were duplicating this effort can do other things, which does not mean redundancy. There is always too much work and too few people.

Technology has long passed the point where it is possible to work effectively across countries. My own responsibility, the Document Management System, is a part of this. Tools that allow you to collaborate effectively on the creation of documents, and create workflow processes that allow the steps taken when dealing with content, to be distilled.

The global economic crisis has changed a lot of things. Budgets that were a foregone conclusion have been reevaluated. This isn’t a bad thing, good economic times breed weakness. Rather than being smart about things, taking the easy way out because if you throw enough money at the problem will eventually go away.

I say bring on the chance to be leaner. The chance to prune to weak.

Quick Update

2009-03-11T00:00:00+00:00

It’s been almost a month since my last post, disgraceful. While I have the will to maintain a blog, I certainly don’t have the time. We’re currently in the thick of a large Documentum deployment, which means that there are bucket loads of new users that each have their own issues with the system.

I have learned SO much over the last few months, hopefully I’ll have a chance to sit down over the next week or so and distill some of that wisdom. Until then, keep safe.

Falling in with the wrong twrowd.

2009-02-04T00:00:00+00:00

There are several ways to use social networking and social media:

To help you connect and interact with people that you know.
A catalyst to enhance your own thinking.
A way of broadcasting your products and brand to a captive audience.
There are probably other uses as well, but how long do you want this list to be?

I’ll get to what is wrong with users pushing products through social networking in a moment, but there is another group of people that use social networking: self styled experts and consultants. People that claim to know how to utilize social networking and social media, people that would be otherwise unremarkable in their careers but have garnered a following of people.
I started following @keyinfluencer and @skydiver after the former made a comment on twitter that blew up in his face, a horse which has been beaten to death on blogs/comments and twitter. They haven’t had a great amount of interesting updates to read since, but @skydiver aka Peter Shankman has some interesting projects, not the least of which is HARO. So they both stayed on the list of people I was following, even after the drama died down.

Around the same time, my official twitter honeymoon period ended. While people are initially interested in someone new, that doesn’t last long unless you are relevant , such as the work they do or the values they represent, there is very little connection that causes interaction. I was sitting at about 25 followers, none of whom actually interacted with my tweets for the most part. I wanted to be relevant and included, and I really really wasn’t. It’s enough to make you pull your hair out.

Then I added @guykawasaki after @keyinfluencer retweeted something from him. Over the next 15 to 20 minutes I picked up another 5 -10 followers, all of them matching the above stereotype. Lots of followers, about an equal number of people they were following…social media experts. People trying to use the medium to their benefit.

I’ve been to a few presentations in the past that these people remind me of. The sales pitch, the angle:

"You too can be *RICH*, if you do what I do."

With slicked down hair and a suit that they hope I’ll recognize is expensive, (which of course I don’t - I know absolutely nothing about suits), the presentation is a money making effort all on its own. The person up the front has no interest in making anyone else rich, or adding value to your life. And who can blame them, why should they care about you? But why should you respect them either? Aside from the fact that they’re doing well financially (assumedly) there is nothing else outstanding about them. They’re not making the world a better place, just trying to get ahead themselves.

If any of the people above feel that this isn’t fair then speak up. I certainly don’t know everything about you. Peter Shankman certainly has something that helps out a lot of people, what about the rest?

I was having a discussion about SEO and social media the other day with my dentist. We talked about a lot of things, not the least of which was the fact that businesses were increasingly trying to utilise this new platform to push their agenda. That may be effective to an extent - but it’s not real. The essence of social networking and social media is that we don’t want to be spoonfed and forcefed what we consume. Twitter is all about finding people to follow that are interesting and useful, but at what point does it become just another billboard that can be used to get your money?

A perfect example of the corruption of social media is the spamming that goes on. Because of the explosion of available services (there are now several twitter clones for instance). It has become common for people to spam ALL the available services, in an attempt to get your name out as hard and fast as you can. A recent issue on Etsy.com highlights the problem that comes when you force an advertising campaign into a format that is designed for personalities, not products. Timothy Adam Designs was recommending that people use a service called socialmarker.com. SocialMarker is a service which lets you post the same URL to up to 160 social bookmarking sites instantly. Timothy’s instructions were embraced by the Etsy community, and subsequently all linking to etsy from propeller.com was banned as it was considered a spam site. He was at a point the social media expert on the Etsy forums. The kicker is, Timothy’s instructions were posted to the Etsy forums as his method of getting blog hits, which would increase his popularity, which would sell his product. His angle: “You too can be rich, if you do what I do.”

Social media is not designed for products. If you are pushing your products by announcing them on twitter, or bookmarking them on the plethora of social bookmarking sites then you’re not doing the right thing. Another interesting forum post on Etsy deals with the fact that a lot of these sites have rules in the Terms of Service against this very action.

There is a place for /Social Products/, but don’t misunderstand me. Social Products should not be mistaken for self promotion. Social products(or services) are when someone else recommends your product or service. Social products are more commonly known as word of mouth, someone buying your product or using your service and feeling so strongly about it that they have to share it with other people. There is only one way to get into that category; be outstanding in what you do.

Stop falling in with the wrong people. I’m still being ignored on twitter, but I’ve had this realisation that it doesn’t matter. On one level, I’d like to be relevant on twitter and have people read my blog, because that may help me get more consulting work long-term. But what I really have to do to get work is follow my own advice. Be outstanding.

CSS Design: Learning from scratch

2009-01-12T00:00:00+00:00

I’m not a graphic designer. But my use of Twitter has done exactly what I’d hoped and been a focus for my thoughts. As an individual if I want to pump out an impressive web site I have to be able to do every part of it. Unfortunately that means being able to implement a CSS design.

Admittedly you can rely on sites like OSWD, OpenDesigns and OpenWebDesign for a while, or you could buy a commercial design. There is also some good base templates around, but in the end though you’ll need to customise those designs at the very least, and at the very worst you won’t find something you need.

With this in mind I decided that it was high time that I created some simple designs of my own, using the best techniques I can manage. They won’t be perfect, but they’ll be a starting point. I’ll post my first results in a couple of days. In the meantime I’m very interested to talk to people that do Web 2.0 designs and pick their brains as to the right way of doing things.

Getting started Twitter

2009-01-10T00:00:00+00:00

I’ve avoided using twitter for quite a while (for who knows what reason). But I finally relented and signed up for an account while I was looking for good ruby and rails blogs to follow and stumbled across a post that had a complete list of famous Ruby people (Rubyists). I have to say that so far I haven’t been disappointed with it.

It reminds me a lot of IRC but without the having to join lots of channels and check all of them for discussions (when mostly their idle with spurts of discussion). I’ve wanted to be more involved with the ruby and rails social aspect since I took part in Rails Rumble 2008.

I will be interesting to see how my usage of twitter evolves over time.

Boiler updated.

2009-01-10T00:00:00+00:00

A while ago (right after Rails Rumble) I decided to rework Bort to suit my own tastes. I renamed my Bort derived rails base app to Boiler. Yesterday I went through and reworked Boiler from a newer copy of Bort. I added a more inviting default theme and some of the other plugins and gems that I use regularly. You can check it out at www.github.com/vertis/boiler

Back from melbourne and SSH Botnets

2009-01-01T00:00:00+00:00

I’ve been quiet over the Christmas/New Year period, that doesn’t mean that I’ve left my computer alone on the contrary I been working on some projects that have been waiting for quite a while. The most important of which is a tool to fight back against the hackers and bot networks that have been trying (unsucessfully) to bruteforce servers I own or manage for the last 2-3 years. I have seen other posts about the rise of bruteforcing attacks on ssh servers recently, and the added complexity of those attacks coming from multiple locations (botnet coordinated).

Bruteforce attacks on SSH are nothing new, I remember seeing them as far back as 2005. At the time I was managing about 25 linux servers and the bruteforce attacks would cause accounts to get locked out (which I would then have to unlock). For the most part we dealt with the problem by having a strict hosts.allow/hosts.deny setup (and of course auditing passwords for complexity). Beyond that, I was somewhat powerless to do anything about the attempts.

These days I run a dedicated server and a couple of virtual servers, and I was seeing the same kind of attempts in my logs. Unlike managing someone elses servers though I have the ability to actively fight back against the attacks. So I am.

I started using ‘kojoney’, an SSH honeypot, but found that while it was fun watching the hackers login and try and compromise a sandbox, it was not what I wanted. So I modified kojoney to log the password used as well (in addition to the username) and setup a ruby on rails project that would record this information, along with the originating IP address, and attempt to login to the IP address with the username/password combo. I called the project mirror, a sort of if you bruteforce me it’ll bruteforce you kind of thing.

Then I left it alone. I hadn’t expected to see any success. But when I checked the logs a few days later I’d successfully logged into a host in poland. It was a non priviledged account but I backed up everything that the people had uploaded, and changed the password on the account.

A few days later I scored another server, this time a root account. I decided that rather than manually logging in and disabling their access I would go one step further and setup capistrano tasks to secure the box (as much as you can a box that has been compromised at a root level). Not only that but the ssh bruteforcer that had been running on this host had gotten 2 more vulnerable root accounts.

There is typically 2 pieces of software installed on the box. A ssh bruteforcer, and a botnet client, at times there are multiple copies of both, if the host has been compromised multiple times.

So far its ME 4 to Crackers 0. I know the battle isn’t over, and that what I’m doing is somewhat grey, but I don’t know of a more ‘white hat’ way of helping stop hackers. If those four servers are managed by people that a clearly stupid, then someone has to step into the breach. Now if only I could find somewhere to send the invoice for my time.

I will provide anyone that is a legitimate security researcher with more details (upon request), including copies of the botnet and ssh-scan software (which shouldn’t be to hard to get with a regular honeypot anyway).

The Christmas cleanup

2008-12-13T00:00:00+00:00

Over the weekend I went to melbourne for a couple of christmas parties, the work one and the extended family one. I’m into my last week at work before the Christmas break and I must say that I am dying to put up my feet and enjoy the a nice sunny and warm christmas. I know there are people that think christmas should be all white and cold, more power to them, but I’m not one of those people. I like the fact that we can have our social occasions outside at the beach or in the sun.

One of the things that has to happen this week is for me to make sure that the system is in the best possible condition. Create documentation about how to keep the system running for the on call person that will be responsible for dealing with any issues that arise and provide as much knowledge based material as possible. Not to much mind since I don’t want my job to disappear over the christmas break, but enough that when I come back on the 5th of january I don’t walk into a disaster zone.

I’m secretly working on a tool that can be used to monitor each of the Documentum components and send me information when one or more of them breaks. It won’t be mature by christmas but it’ll be good if it notifies me of any irritating problems. I also need to finish up as many of the outstanding tasks as possible. All this should make for a hectic week.

DQL stuff

2008-12-10T00:00:00+00:00

Prasad on his blog (you can find the link on the right), has a really really useful collection of dql that can be used for various admin tasks. I’m going to slowly build up my own list of DQL for much the same purposes. A note on convention: This is standard convention but as a refresher (or for those not aware of that convention) anything within < > is to be filled in at runtime. Remove the < > brackets and replace with appropriate value.

Updates the ACL on all content in a given folder...be aware it won't allow you to replace the ACL on a locked piece of content, so you'll either need to skip it like I have below or make sure that all the content is unlocked before trying to do this query, otherwise the error message is VERY VERY vague.
update dm_sysobject object set acl_name = '<acl_name>', set acl_domain = '<acl_owner>' where folder ('/<folder>', DESCEND) and r_lock_owner = "
Replaces the ACL on all content in a given folder - this is an important change because it only overwrites a specific ACL rather than changing all the content to be under the new ACL, which could potentially be a security problem if the content has been locked down further. It's really only a simple change to the previous query.
update dm_sysobject object set acl_name = '<new_acl_name>', set acl_domain = '<acl_owner>' where folder ('/<folder>', DESCEND) and r_lock_owner = " and acl_name = '<old_acl_name>'

Documentum, old world document management (or why to avoid Documentum 6.0SP1)

2008-12-10T00:00:00+00:00

I’m sick of raising support cases for things that should just work. I’m frustrated that a product as buggy and unstable as Documentum 6.0 SP1 can be proposed as a solution (by EMC). There are a lot of good things about Documentum, but there is a lot of bloat as well. This will be a growing list of stupid problems that I’ve found, I’ll start with one item and go from there.

Clicking on 'Save As' on a permission set with a name over a certain length causes it to bomb out. Why? because it tries to give the permission set a temporary name that is longer than the allowed 32 characters. No chance to override this name, no graceful handling, just a fat error message and the chance to reload DA and try again.
When using LDAPSync (which you probably need to get hotfixed anyway what with it having so many bugs fixed) 'Groups and their member users' does not update the content of mapped attributes even if they change (for instance during a rename).
My favorite of the last 2 weeks: BOCS and DTS are not supported in the same solution. This problem is related to DTS going looking for content that is still parked on the BOCS server. It's been fixed in 6.5...the problem is that it was EMC that built the solution that we're now using. There is some instability in the DTS services, which is something I have to follow up, but EMC/Documentum had better be hoping that the two are not related.

I think part of the problem is that Documentum like a lot of old world enterprise applications has a very tenous hold on some of the techniques that allow for more error free development. I mean was the code even unit tested…

Etsy Api repositories x2

2008-12-05T00:00:00+00:00

For those that are interested I’ve decided to release the work I did on accessing information from etsy’s backend. I have two copies one that is a ruby gem and based on a custom hacked rubyamf implementation (since it doesn’t support being a client) and one written in PHP that relies on SabreAMF (the copy that I gave to juln and that runs the heartomatic), and put that up as well. The url you want for my new code is:

http://www.github.com/vertis/etsy_api

And the one for my old (possibly broken) php code is:

http://www.github.com/vertis/etsy_api-php

If people are interested in collaborating on these projects let me know (sign up for github) and I’ll see about adding you.

Internet Filtering: One ticket to...anywhere else.

2008-11-30T00:00:00+00:00

I’ve been fuming about the internet filtering that the Rudd government is trying to impose on us since I found out about it a couple of months ago. Its frustrating because I KNOW its not practical. I had high hopes for Kevin Rudd’s government, I mean its not exactly the revolution that we’re seeing with the new president that the US has picked, but he was certainly a breath of fresh air after 12 long years of Howard government. But this half cocked internet filtering effort has shattered my belief in him.

I’ve seriously considered just pulling up stake and moving elsewhere in the world, there is always room for one more technical person in most countries. This is not to say that I can’t get around the filter, anyone with half a clue can download Tor, and the rest of us with more technical knowledge can go a lot further than this; tunneling over SSH or creating a VPN to my dedicated server in the US (not that a virtual server wouldn’t work equally as well). But the fact remains that its another step down the road of losing personal choice, I say another and I mean it, this is not the first time that the government has stepped in and stopped something with a heavy hand that should fall under the responsibility of parents (what their children watch).

I’ve been shocked to discover that Australia is a lot more prudish about a lot of things than you might believe. Christians would say that our society has not had its moral fabric eroded as much as say the US (even there the christians exert the same kinds of pressure). The problem is that fundamentally Christians believe that they have a right to shove their beliefs and concerns down everyone else’s throat. Enough about that though, this isn’t supposed to be an anti christian rant (and I’m not anti christian in general). It just frustrates me that people feel they have a right to destroy somebody else’s right to choose.

The last thing that really gets me with this issue is the lies. To steal something from the abortion debate; “Don’t they know that you can be pro choice and still be against abortion”. They equate not wanting the filter with siding with child pornography, and it’s just patently false, I have no interest in EVER supporting child pornography (lock them up and throw away the key). That doesn’t mean that I agree that you should punish(remove the rights of) all Australians to decide what they look at, across a broad range of content, and claim that its all about the children. Hire more technically competent police to deal with cybercrime (an area I’d love to work in), and get a decent technology advisor.

You can sign a petition at http://www.getup.org.au/campaign/SaveTheNet/442

Mongrel service tip

2008-11-12T00:00:00+00:00

If you’re running rails applications on Windows then you’ll be interested in or already use the mongrel_service gem. When I recently (today) tried to install it on a new computer I had trouble because it didn’t want to install the dependencies. Apparently it requires a version of the win32-service gem >= 0.5.2 but < 0.6.0
<div class="CodeRay">

Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp.
C:\Documents and Settings\Administrator>gem install win32-service -v 0.5.2
C:\Documents and Settings\Administrator>gem install mongrel_service

</div>
should do the trick

UPDATE: Apparently using w32-service causes problems whenever you have defined a class named service. Just be aware that its going to cause it to use w32-services service class rather than you’re own, regardless of whether you’re running mongrel as a service (it won’t happen if you don’t run mongrel).

Corkscrew to the Net

2008-11-12T00:00:00+00:00

I’m thrilled to announce I am no longer a prisoner of an oppressive firewall. I’m currently working at a clients site for the next year give or take, which has been great, but one of the things that has really irked is the fact that their firewall is downright oppressive. Corporate policies prevent anything other than 80/443 out onto the net and stop webmail from working (to protect from viruses). This is a problem for me because I keep 2-3 dedictated linux servers for various reasons, obviously I can get to the hosted webapps just fine, but sshing in the change something on them is not possible due to the above mentioned firewall.

Thats where corkscrew comes in. its a tool that allows you to change the clientside settings for OpenSSH to tunnel over the HTTP Proxy. Of course you still have to make sure that the SSH server is listening on either 80 or 443, however this is easy to accomplish if either one or both of those ports are spare or alternately you can assign and extra ip address and avoid apache/etc listening on */0.0.0.0

I’ll leave it to the other corkscrew guides as to the particular setup. And yes its probably possible for them to block this, which is why I won’t scream to loudly. Just remember if you’re one of those people trying to prevent people from using this that I’m not trying to do anything nefarious, i’m simply a geek who needs/loves port 22.

Are Linux users lemmings collectively jumping off of the cliff of reliable, well-engineered commercial software? -- Matt Welsh

2008-11-10T00:00:00+00:00

Update: There has been a bunch of traffic to this page over the last few days. Apparently this quote is showing down the bottom of slashdot pages again. I wrote this post a LONG time ago.

I’m slightly embarrassed that you’re all on this page, however I’ve decided to leave the post in it’s original form regardless. Please don’t judge me too harshly.

There doesn’t appear to be any information available about where this quote came from, and more importantly the context in which it was said – though it would seem likely given other information about Matt Walsh, that I took the quote the wrong way.

I found this question as a quote on slashdot, and couldn’t resist responding.

Lemmings? Lemmings as popularised by the game by the same name are supposedly creatures that follow each other and just keep going regardless of dangers en-route to their destination. Being a lemming is equated to being one of the Herd or not thinking for yourself. This is an unfair and baseless accusation. Linux users by their very minority are not followers, until recently there were no computers that came with linux installed by default. Clearly if you wanted to run Linux you had to leave the Herd.

Well-engineered commercial software? That would be a stretch considering the issues that Microsoft has had over the decades since it started working with operating systems. I’m not saying that Linux has it all worked out, because it most assuredly doesn’t, but the difference is that if you’re not happy with something in Linux you have the power to do something about it. With all commercial software you are at the mercy of the company that owns the software to be diligent.

Now I’m not saying that Windows doesn’t have good things, on the contrary there are features that I miss very much when using Linux, but the downsides are far outweighed by the benefits of an extremely stable OS. More importantly Linux is an OS that given enough time and dedication will run rings around anything Microsoft can field to compete with it.

Turn on Authentication Tracing in Documentum Administrator

2008-11-06T00:00:00+00:00

The following comes in very hand when you’re trying to troubleshoot authentication problems. Go into DA and find the Administrative Methods; there should be one called ‘SET_OPTIONS’ which allows you to put in trace_authentication and set it to true. All authentication attempts will then be logged to the docbase log.

A few notes:

If you have a clustered setup you'll need to make sure it gets turned on for both content servers (you can specify which you're connecting to when you login)
Make sure you turn it off when you're done troubleshooting, it adds a performance overhead not to mention chewing up log space.
Restarting the content server will cause it to turn off...to turn it off without restarting the content server just set it to false in the same place you turned it on.

JRubyOnRails for the enterprise

2008-11-06T00:00:00+00:00

I believe that JRubyOnRails is paving the way for ruby and rails to be integrated into enterprise rather than being the sole domain of cutting edge developers. Perhaps its important that enterprise realises the power of rails or perhaps not (it might be better to keep it for ourselves). Regardless the ability to run ruby on rails applications on Java Application Servers is a big step towards acceptence in the enterprise.

For those that aren’t familiar with this ability google ‘warbler’ and run through the tutorial. It is a little more complex to get a larger project to run on a Java Application Server but not so much that would deterr people. It will likely be a long path before its a mainstream technology but it’s a good first step.

It’s also not necessarily a good thing to be tied to java, but when enterprise is so slow to embrace new technologies and better ways of doing things it is important to compensate to some degree. Not making them completely replace the Java application server that they paid to much for in the first place may make that admission and acceptence a little easier.

UPDATE: I’ve since started following @headius on twitter, the primary force behind JRuby. He is outstanding, and his belief in getting ruby accepted in the enterprise is inspirational.

Rails Rumble 2008

2008-10-15T00:00:00+00:00

I’m participating in Rails Rumble 2008 I got in a bit late because I only came across it the other day. I’m a bit of hermit when it comes to development so its a great idea participating in an event and getting to know some people. The project is currently top secret ;-) but once its finished I’ll open source it.

The other thing I’m going to try and do is record really closely the steps that I take during development. So I can post the videos as podcasts (a la railscasts), but also so I can go back and evaluate my own processes. I haven’t done anything like this in the past but if it goes well then maybe I’ll make it a regular thing.

Word of the Day: Resource

2008-10-02T00:00:00+00:00

This word can mean a lot of things. The reason its currently in my headspace is ongoing discussions/actions to do with the amount of resources that are devoted to the project that I’m currently working on. My background is one of just getting things done, I do things as quickly as possible and then move onto the next item on the list. This occasionally gets me into trouble as I overtax MY resources. Not only this but pushing to hard also causes my colleagues issues.

At the end of the day you can only get so much done, feeling stressed all the time because you’re pushing to hard doesn’t necessarily help you get things done faster it just burns you out. So yeah, make sure that there are sufficient resources allocated to a project if you have to meet a given deadline, and if there isn’t then its important to talk to the client and make sure more resources are allocated.

The pain that is VS 2008

2008-09-22T00:00:00+00:00

I’m very fond of Eclipse. Sadly the code I’m working on at the moment is mostly C# .net (with at tiny bit of VB), that means I have to adapt to working with Visual Studio. Visual Studio has come a long way since I last used it but I still miss the power that you take for granted when you use eclipse. So I was wondering whether anyone had experimented with the couple of C# plugins for eclipse with anything more than hello world applications.

EtsyTools

2008-09-20T00:00:00+00:00

I’ve been doing quite a bit of work to get some tools for etsy back online. The original ‘etsytools’ was forced to close after my girlfriend recieved death threats, not that they could be taken seriously, but it still put a black cloud above an effort that was primarily designed to benefit etsyians. Enough time has passed that I now feel comfortable reviving the site. It won’t come back in the same form, if anything I’ve spent the last six months building tools that will be far more useful than anything I previouslyoffered. As soon as there is a moment to spare from my day job (which keeps me pretty damn busy), I’ll polish off the last of the tasks required to get it ready for use.

Apple iPhone Dev

2008-09-20T00:00:00+00:00

It's rather expensive to get into iPhone development. For a start you need Mac OS X 10.5 and therefore a Mac to do development. It doesn't surprise me that Apple on ly offers the SDK for OS X, but it is a bit disappointing. Its for reasons like this that I can't wait to see the proliferation of open hardware.

Internet Limbo

2008-08-18T00:00:00+00:00

I've been having a nasty time with my internet of late. To start things off my Unwired Wireless Router decided to die a couple of months out of warranty. Unwired has been really useful when I didn't have a permanent address, I've been in serviced apartments for the last year or so. However its relatively slow and has only a small download quota even on the top plan. Definately time to get a nice fast ADSL2+ connection now that I have a permanent address. I began drooling over TPG's 150Gb download limit. I know a lot of its off-peak but thats fine by me, I'm happy to schedule downloads for the wee hours of the morning.

Obviously the first step to getting ADSL2+ would be getting the home phone connected. I know there are Naked DSL plans available, and if I'd taken one of them instead of going after TPGs monster downloads for minimum cost option I wouldn't be writing this post. Naked DSL has it's benefits, but the plans are more expensive and they take longer to get connected. Picking fairly randomly I chose Optus, which as it turns out was a poor roll of the dice. They took 2 weeks or more to get the phone connected (it was finally done today), and afterwards I go to TPGs website and put in my shiny new phone number.

Incorrect.

I'm sorry?

Incompatible.

Why?

Well because optus services that exchange itself and doesn't share its DSLAMs.

I hop on the phone to Optus. After getting through the friendly computer, I talk to a consultant. She transfers me to the right department. I explain the problem. She proceeds to test my line, not failing to mention that Optus has Internet plans (3 or 4 times), I explain to her that Optus doesn't have plans with enough quota. You can refer tomy naked DSL cost comment above at this time and realise that I made the mistake of assuming that I could have the best of both worlds, cost and quota. Realistically I don't care about the cost, if Optus offered the same level of quotas that most of the other broadband providers offered (50Gb - 150Gb) then I'd just hack the extra cost and get on with my day.

They don't however. 30Gb is their biggest plan, and not only is it puny compared to iiNet/Internode/TPG (and i'm not even talking Naked DSL), but it costs more. Additionally the consultant informs me that since the connection report hasn't been filed I should wait till tomorrow to disconnect the phone.

*Sigh*

All this time I've been in internet limbo, and its beginning to wear on me. Going down the road to check my email, etc. The solution for me at this point is to signup for a new naked DSL connection with Internode and just leave the Optus home phone as it is until I can confirm that I have the naked DSL. There is thankfully 2 lines going into my apartment though whether the second line works or not is another matter entirely.

I'm still in internet limbo till the 9th of September, unless of course Internode can somehow fast track my installation. Top lesson of the day though, naked DSL is theway to go. No line rental. No incompatibilites. No phone contract to get out of.

Hopefully someone at Optus will read this. I'm happy to use them as a broadband service provider *if* they can compete, and *if* they realise that people want massive download quotas.

24Mbps/150Gb Today, 100Mbps/1Tb tomorrow. I live in hope.

A little SQL injection to make the weekend more interesting.

2008-06-21T00:00:00+00:00

I’ve been going along mostly unaware of the fact that the internet is currently experiencing a nasty spate of SQL injection attacks. These attacks are being used to infect visitors to otherwise harmless and friendly websites. They’re advanced and complicated attacks that a few years ago the world had never really experienced, thats not the case any more. It is an increasing reality that viruses and worms are no longer the province of bored uni students, but rather done by what can only be described as a technologically aware organised crime. The site I happened to run across the problem at, finally took the site offline and cleaned the fields that had been ‘infected’, and I would hope that they’ve also taken steps to solve the problem.

I’m going to build some more information up on the technical aspects of this attack. So for the moment this is a place holder. I’ll be building up some more information over the next few days.

Passenger, Typo and Sqlite

2008-06-19T00:00:00+00:00

I upgraded my rails sites to use Passenger aka Mod Rails this week. I’d like to say its been all roses, but the truth is that my Typo blog would swear under oath that its not the case. I was quite happy with the out of box Typo + Sqlite combination, and that worked fine under mongrel. Not so under passenger, the website will work perfectly, but I won’t be able to get into the admin section. As you can see I managed to get it working, I switched the database to mysql and its been working fine ever since. I’ll get around to working out WHY it doesn’t work with sqlite at some point (and get around to extracting my lost blog posts).

Other than that small hiccup I’ve been extremely impressed with Passenger. Its certainly a step forward as far a simple rails production hosting. I’m not saying that its time to throw mongrel out the window. But when it comes to trouble free hosting, mongrel is just not there. I want a solution where you can do a simple config and forget about the site. Managing multiple clusters for each of the various sites is something that makes it harder to whip up simple sites.

Passenger allows the user to to configure an apache virtual host with a couple of rails specific parameters (more if you want to take advantage of somecool feature, or do abnormal things), and restart apache and you’re all up and good. Restarting your rails app can be accomplished by running ‘touch tmp/restart.txt’ in your rails root. You’ve then got a very simple solution that will allow you to host multiple sites without doing clustering.

Native Java with GCJ and SWT

2007-06-24T00:00:00+00:00

The ability to compile Java to native code has existed for a while in the form of GCJ, though for the most part it has been overlooked due in part to the incomplete Swing/AWT implementation. Knowledge about how to use GCJ for native compilation is a bit vague. When you first start using GCJ as a real alternative to the Sun JDK it’s difficult to get an idea of exactly what GCJ is capable of doing.

Getting GCJ

If you are a linux user most distributions come with GCJ as either preinstall or one that you can easily install with your favorite package manager. If you are using Windows then your best bet is to use the packages found that http://www.thisiscool.com/gcc_mingw.htm

Mohan Embar, the author of the ThisIsCool website has created several builds of the MingW GCJ that can be easily installed. Additionally he has built in SWT and SwingWT support to some of the builds. Keep in mind that shared builds, and the creation of ‘dll’ files is not supported under the 4.x series of GCJ found on this page.

For the remainder of this build we will be using the latest GCJ found on this page, which includes the eclipse merge branch from the GCC SVN, the best thing about this is that it gives support for Java 5 features.

Basic Native Compilation

The first step is to create a basic ‘Hello World!’ application, to demonstrate compiling an application.

public class HelloWorld {

/**

* @param args

public static void main(String[] args) {

// TODO Auto-generated method stub

System.out.println("Hello World!");

}

Assuming you’ve added the gcc-ecj\bin\ folder to your path, you can then compile the application using the following command:

Gcj –o Hello.exe –main=HelloWorld HelloWorld.java

You can then run the resulting EXE just as you would any other application.

Basic Windows Application

If you tried running this application from windows (and not from the command line) you wouldn’t have seen much, just a black window flashing up and then closing again. Let’s do a basic Windows application so that you can begin making your masterpiece. This is accomplished using SWT. We will start by creating a ‘HelloWindows.java’ file. You can view the source with Listing 1

[gist id=803916]

import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;

public class HelloWorld {
  public static void main(String[] args) {
    Display display = new Display();
    Shell shell = new Shell(display);

    shell.setText("Hello, world!");

    shell.open();
    // Set up the event loop.
    while (!shell.isDisposed()) {
      if (!display.readAndDispatch()) {
        // If no more entries in the event queue
        display.sleep();
      }
    }
    display.dispose();
  }
}

[/gist]

We will then compile the source in two steps.

gcj --classpath .; lib\swt.jar -c -o HelloWindow.o

This creates a object file that can be used in the next step to create a working java application.

gcj --main=HelloWindows -mwindows -o HelloWindows.exe HelloWindow.o lib\swt.a

You will still need the DLL files that came packaged with thisiscool gcj in order to run the application but it should create a small windows with ‘Hello Windows’ in the title bar; not very exciting I will admit, but it demonstrates a working windows application.

Notes about this step:

The ‘swt.jar’, ‘swt.a’ and ‘swt*.dll’ files can be found as a part of the thisiscool download, you either need to move them into the same directory as your project or provide a full directory in the adequate place. The dll files should be placed in the same directory as the build .exe file.

More Complex Projects and Libraries

One of the things you may not know is that GCJ can not only work from source files, but also from class and jar files. This is particularly useful if you want to do more complex projects that use existing libraries such as the db4o-6.0-java5.jar file gotten from http://www.db4o.com/.

First create the object file, using the command:

gcj -c db4o-6.1-java5.jar

This creates a file, db4o-6.0-java5.o; we then use ‘ar’ tool to create a static library:

ar -r libdb4o.a db4o-6.1-java5.o

This library file can then be combined into our future projects much the same way as jar files are used in current java projects.

Using Our Shared Library

Not to put the work to waste lets compile a project using the DB4o library that we created in the last part. The source code is too long to paste into the body of the article so I’ve included it in a zip folder so that you can try it later. Let’s start by compiling the source to object files as before:

gcj -classpath .;lib\db4o-6.1-java5.jar;lib/swt.jar -c HelloDb4o.java Account.java

Then we take the library we created before and combine it with the main project (also adding the SWT library):

gcj -o hellodb4o.exe -mwindows --main=HelloDb4o HelloDb4o.o Account.o lib\libdb4o.a lib\libswt.a

You now have an application that will display the record from the database.

Final Notes

Size is often one of the pet complaints when people compile using GCJ, I would encourage people to check out UPX (http://upx.sourceforge.net/).

The source listings here in were done quite hurriedly; particularly the last application would need a good deal of work before it was functional in a useful way. DB4o is a topic for several books, I strongly encourage you to take it out for a test drive.

In order to catch bugs it is in the interest of the developer to remove the ‘-mwindows’ option from the final build. This causes a command line window to be created behind the main screen to which printed exceptions are sent. Alternately and perhaps more correctly logging to file could be implemented.

Finally, it is possible to use Ant and other build tools to make building projects easier, you will find details in the resources on how to accomplish this. For the purposes of explanation I have chosen NOT to use these tools in this article.

Resources

The following were useful guides/tools as I was getting started with GCJ Native Development.

Create native, cross-platform GUI applications - http://www.ibm.com/developerworks/library/j-nativegui/

Create native, cross-platform GUI applications, revisited - http://www.ibm.com/developerworks/java/library/j-nativegui2/

How to compile Java application to native code, for example to Windows EXE file - http://robohobby.s41.eatj.com/java_to_native_code_exe.jsp

DB4o Object Oriented Database - http://www.db4o.com/

The Ultimate Packer for Executables - http://upx.sourceforge.net/

ThisIsCool GCJ - http://www.thisiscool.com/gcc_mingw.htm

Secure Your Java Source Code

2007-04-19T00:00:00+00:00

I want to quickly run you through the pitfalls of using java as a programming language when it comes to the security of your proprietary source code. One of the problems with Java is the relative ease with which you can reverse engineer the source code. In fact there are numerous tools out there which will very simply and quickly take your project and turn it into a VERY readable document. For the purpose of instruction we will first decompile the example project, comparing the original source with what comes out the other end.

We will repeat this process (if possible with the techniques that can be used to protect your source code).

The Jad Decompiler + Front End Plus

As mentioned above there are plenty of available decompilers, however JAD is a free and open source alternative that is perhaps the most popular option out there. Coupled with Front End Plus, a GUI that is designed to use JAD, you can very easily open and decompile the Java source.

This is a decompile of my own Arms Conversion program, as you can see, while you may have to do a little work if you wished this code to recompile, the intent of can be seen very clearly.

Obfuscation

Obfuscating the source code is one method that is often used to protect Java code. It refers to deliberately making the code harder to read. Rather than a cracker getting back a function called IDoSomethingCool (String name) … they instead might get something back like dadaba3 (String adabc9989).

This doesn’t stop the cracker from reading the code, nor from it working if they chose to recompile it. However it does make it MUCH more difficult to determine the intent of the code. Are the naming conventions that programmers use to determine what information is being stored/parsed/manipulated

I should be clear that this is far from a full proof solution. With time and dedication it is still possible to rebuild the original intent. However Obfuscation is a big step towards making it a waste of time (it may cost less to just write the application from scratch).

There are plenty of tools available that can be used to obfuscate the code.

Native Compilation

By far the best option for protecting the source code is the ability to natively compile the application. While there are potential issues to be addressed in order to implement native compilation it has the benefit of making the reverse engineering infinitely more difficult.

While NO application is truly immune to reverse engineering, going from compiled code back to useable source code requires a very specialized ability, and a depth of knowledge about computer programmers that is possessed by few.

Native code compilation is discussed in more detail in another document, which deals with additional features that are available, and also the potential problems with natively compiling your applications.

The 0th law of security

2006-01-30T00:00:00+00:00

There are supposedly 10 laws of security. Laws that are a firm basis for understanding computer security. They’re obviously not the be all and end all of computing security, but for beginners and those that aren’t going to focus on security they’re an important start.

The Ten Immutable Laws of Security

Microsoft’s Security Response Center Manager, Scott Culp, as a part of his job produced a list He calls “The Ten Immutable Laws of Security.”

They are:

If a bad guy can persuade you to run his program on your computer, it's not your computer anymore.
If a bad guy can alter the operating system on your computer, it's not your computer anymore.
If a bad guy has unrestricted physical access to your computer, it's not your computer anymore.
If you allow a bad guy to upload programs to your Web site, it's not your Web site any more.
Weak passwords trump strong security.
A machine is only as secure as the administrator is trustworthy.
Encrypted data is only as secure as the decryption key.
An out-of-date virus scanner is only marginally better than no virus scanner at all.
Absolute anonymity isn't practical, in real life or on the Web
Technology is not a panacea.

Even without further explanation (which is available from here) it is a fairly straight forward and common sense list of laws.

Law 0

The fact is that these laws don’t go far enough towards describing the problems that are faced by everyday users on the internet. Security people often forget that its not just big companies that are the target of attacks; they may indeed be the target of more personalized attacks

If you can't read the source code for your operating system (and applications) then it's not your computer anymore.

I hate being the open source advocate, but the fact remains that if you and the community can’t get into the source code for auditing and patching purposes then its not your computer its Microsoft’s. You are essentially relying on their good will and the competency of their programmers to protect you against any flaws in the operating system that may let attackers in.

Microsoft has in the recent past finally hopped on the security band wagon, they’re better than they used to be, but its still them against the world, and in practical terms this makes for an impossible situation. The odds are that one of the millions of hackers is going to find it before Microsoft does. Even with their ability to look at the source code they’re still vastly outnumbered.

Open Source

Open Source is not a complete solution to this problem, but its better. The millions of security researchers out there, the developer community and the general public all get the chance to look for flaws in the code. Immediately once it’s discovered a patch is written for it. Unlike a situation where you have to wait for a company to release a patch, you have the ability to patch the problem yourself, its not you against the world though. It’s you and every other technically competent person that uses that particular software against the world.
Yes, hackers have the same opportunity of finding the flaws. But the playing field is more level. Even if they do find a flaw, chances are that it’ll be patched much more quickly than if millions of eyes weren’t looking at the source code.

Open Source vs. the Other Ten

When you look at open source as a solution to the problem above; it puts them in a whole new light. Let’s start with No. 7, not because of the fact that it’s a good number, more the fact that it has long been the belief of the scientific community that closed encryption algorithms are useless.

Encrypted data is only as secure as the decryption key.

While this deals with the key that is used to encrypt the data I would go further and say, that encrypted data is only as secure as the Algorithm and Key that is used to encrypt the data. It doesn’t take genius to work out that even if I encrypt my information using my own proprietary method that doesn’t mean that it’s safe. Unless someone else can test my encryption method, and try and break it, I have no way of knowing whether my information is protected by the encryption; because, I have no way of knowing whether my encryption algorithm is sound, or whether there are fatal flaws in my design.
History is littered with examples of this, and if you look closely at companies like RSA you will notice that they post challenges, trying to get people to break their encryption.More importantly if you can’t look at the encryption algorithm and analyse it for yourself, how are you to know that the creator hasn’t put in a backdoor for themselves, or governments to use.

Watching the Watchers

An out-of-date virus scanner is only marginally better than no virus scanner at all.

Nearly everyone that I know, knows to use a virus scanner now, its slightly harder getting them to workout Spyware and AntiSpyware programs, but here’s the twist, if you can’t look at the internals of the anti virus, how do you know that its doing an adequate job of protecting you.
I’m not trying to say you should be using Linux because of the fact that it is less prone to viruses, the fact is that most viruses are written for windows, and if everyone switched to Linux, then those same people would target Linux. It remains to be seen how well Linux would respond to this kind of problem.
What I am saying is the applications that you pay good money for, you subscribe to a service by Symantec or McAfee, and you really have no idea how well you are being protected. The same goes for firewalls, and any other piece of security software that you use to protect your computer. If you can’t look at the internals then you have no idea what the application is really doing.
You can apply this same principal to at least some of the other laws, and in truth it serves to cement the 0^th law in place.

Regarding Patches

It is not often that I have the chance to talk about security, but one of the things that occurred to me in my day to day work is the fact that Microsoft’s move to allow only ‘Genuine’ users to download patches and applications, most notably SP2 and Microsoft AntiSpyware, was a foolish one.
Regardless, of the fact that most if not all of my clients have legal copies of Windows, it is rare that they keep them patched and up to date (I tend to fix this), but it leads me to believe that there is a large number of legal windows users out there that don’t patch their computers properly.
Now, it’s not overly smart of them, but the fact of the matter is that denying patches and other downloads to ‘non genuine’ users, ends up negatively affecting even those with legal copies in a round about way; look at it like this:
the greater the number of unpatched computers on a given network, the more chance that a bad guy/worm will get in.”
This is easy to apply, but what is more important is that it takes into account not just small local networks, but the internet as well. The more unpatched computers that remain on the internet, the more chance that the bad guy will get control of them; and the more computers that are either part of botnets, or infected by viruses the easier it is for it to spread, or the attacker to use the given host as a base for another attack.

A Note on Piracy

I’m not in anyway condoning & supporting piracy, but there comes a point when you need to accept that a problem isn’t going to be solved and make smart moves. Becoming tight and vindictive about piracy only makes the given company coughSonycough look bad.
Locking your legitimate users out is bad methodology, and putting so many ‘copy protection’ methods into a given technology that it negatively affects it is not healthy either. Security is important, but it needs to protect the interests of the user not the interests of the greedy Mega Corporation.

Why Open Source is fundamentally flawed...

2005-12-27T00:00:00+00:00

I'm a huge supporter of the open source movement. It's the best thing to come along since sliced bread. Many of the tools that I rely on to do my day to day work would not exist if it wasn't for the struggles of the open source movement. This doesn't change the fact that I feel the open source movement is flawed in more than one way; its battle is made more difficult by its very nature.

Enforcing the GPL

Regardless of the good intentions of the GPL there is really no way of forcing companies to abide by it. For example, Company A decides that they want to write a application, they first take a look at sourceforge.net and search for all the various applications that are available in the given category. Company A being lazy decides that its easier to extend Sourceforge Application B than write their own from scratch.

You may well say that even if Company A does this it will be apparent to anyone with 'know how' of decompiling/disassembling. This may well be true, but there are ways and means of getting around these particular problems as well. The same technologies that are used to protect compiled code against reverse engineers can be used to throw open source watchdogs off the trail.

Company A has just benefited from someone else's work, and proceeds to make a handsome profit selling their new product. The open source movement can rant and rave and even try and out market (unlikely without a budget), but at the end of the day they have no proof.

Handing out the source code, regardless of what license it is under is the equivalent of giving away the keys to the kingdom. Licenses have proven time and again that they don't work, and the p2p piracy that is so deeply dug into the internet now is the perfect example of license failure. Even more so the source code for an application, while it is possible to protect a compiled application using anti piracy technology, no such method can be employed to protect the source code of an application.

Development Model

The open source development model has its benefits; one of the biggest arguments that Open Source Movement will throw at you is the fact that allowing open contribution enables projects to evolve at a much quicker rate. What I see is a lot of half finished projects, frustratingly close to complete and treading water for lack of developers. The major projects that are going forward are being driven both by larger organizations, and even companies that have an interest in utilizing said open source project for their own purposes. This is compounded by the fact that rather than getting together with a previous project developer to create a truly powerful tool, quite often a developer will start their own project from scratch and duplicate work that has previously been done.

To be fair the Open Source Movement has had some tremendous successes, that doesn't change the fact that it's flawed, it just means that some of the better developers are able to wade through the weaknesses and produce some world classes tools regardless.

The Misunderstanding of Open Source

Plenty of people flock to the open source movement under the mistaken impression that its free. Hell my first steps into the land of open source were made under this deep-seated misinterpretation, everybody is looking for something for nothing

Thoughts on Java

2005-12-20T00:00:00+00:00

Recently I've been doing quite a bit of development in Java. Of the many languages that I've played with and used I find Java to be one of the most functional programming languages out there. There are plenty of factors that have helped me choose Java as my primary programming language, and while I'm not under the illusion that java is the 'monkey wrench' of the programming world, its still an invaluable tool.

I've been around programming languages for the last 6 - 7 years, in that time a lot has changed. The software industry has matured in its approach to programming, and the fundamental building blocks that make up the software industry have matured as well. Gone is the time when you needed a PhD to understand the complexities of computer programming. This is not an article about the many different languages; this is not even an article about how Java is better than 'x' language. This is simply an article about the features of Java that slipped it into my toolkit.

Native Graphics Support

The first thing that usually pops into mind when you want to convince someone about the superiority of java, is the fact that it's a compile once run anywhere language. Honestly though I find this less useful than you might think. What I do find useful is the fact that with very little research you can write a graphical user interface for an application. I find the graphics support for Java to be logical, where so many other graphics libraries are not. While I'm not overly fond of the layout managers that Java provides, I can appreciate the fact that they're there if I do ever find a use for them.

Excellent Database Support

This was one of my primary reasons for moving to java in the first place. If you've ever tried to get MySQL working with (Visual) C# then you'll know what I'm talking about. The JDBC makes connecting to whatever database a dream, if your program is well written you can even switch database backend without to much trouble.

Eclipse IDE

If you haven't tried Eclipse then you don't know what your missing out on. While its certainly not solely dedicated to Java it's a very flexible and well designed tool that will help you get your work done. In addition to being a useful portal to the Java language, it is also a driving force in making Java into more than just the pet project of Sun Microsystems. Take a look at SWT, which was originally developed by IBM and is distributed as a part of eclipse and you will begin to understand.

Lack of Deployment Options

This is one of the fundamental failings of a lot of programming languages. Running a java program relies on the operating system already having a copy of the Java Runtime Environment (JRE) installed. If its not then you won't be able to do squat, additionally there is no way of knowing whether the version of Java that you're programming in, and that has the features you want is installed on the client.

You fall back into the same rut that the old VB programs had with missing DLL files. There is no simple way of installing both Java and the given Application in one easy step. Additionally you are held hostage by the fact that you require certain environment variables to be set correctly for the java program to run (JAVA_HOME, CLASSPATH). This is all well and good but it's just more overhead for the programmer to worry about when they need to do a large scale deployment of an application that they've developed.

GCJ bridging the gap

One of the baseline flaws with Java is the fact that it is interpreted (from bytecode) at runtime, and as such relies on the JVM. More importantly it means that it runs slower on CPU intensive processes like md5 hashing for example. GCJ is the GNU Toolkit's answer to the fact that Java is not open source. It is in effect a complete replacement for Sun Java. More importantly it provides support for compiling Java directly as an Executable freeing it from the reliance on the JVM.

The speeds generated when running is GREATLY improved, to close GCC/G++ levels, and more importantly the deployment is less of an issue. You do lose the fundamental portability that is gained with Java classes, but there is no reason you can't provide the class files and just install GCJ (rather than Java) to allow the class files to be converted into executables.

THE only place where GCJ is still lacking is the area of AWT/Swing; the developers are still working on adding the support necessary to support the windowing toolkits. Incidentally they are looking for help in this area if any of you programming gurus want to give up your spare time.

Conclusion

The bottom line with Java is that it is a functional and practical language. The fact that Sun is still holding the reins is regrettable because it's inhibiting a language with otherwise HUGE potential from maturing. The compile once, run anywhere is less useful than 'write once / compile (& run) anywhere' this subtle difference is where Sun missed the boat. The matter of compiling different copies for different operating systems is trivial when you consider the fact that doing it the way they are cripples the speed of the language.

Other than that Java is a great choice because it's a language that's flexible enough to be used on plenty of different operating systems. You do lose a certain amount of control over the low level functions; however this particular beef has been adequately dealt with in the form of JNI.

And Ubuntu changes the game.

2005-12-12T00:00:00+00:00

I've had problems with desktop Linux in the past. While I openly promote Linux in the use of servers and the like, my attempts to promote it as an alternative to Windows XP have gotten me burnt. Users who can barely grasp the fundamentals of computing are ill equipped to be thrown into the deep end.

Even I find it hard to commit to using linux as my primary operating system. The majority of my clients use Windows, and compatibility issues have killed me in the past. All this had bred in me a belief that Linux while it was a great tool for the computer geeks of the world, it was never going to cut it as an everyday tool for the masses. I could not have been more wrong.

Not long ago a friend suggested that I take a look at 'Ubuntu'. I hadn't missed its appearance on the now glutted Distro scene, I'd simply written it off as another wannabe. Ubuntu is forging new roads, making steps towards that holy grail of operating systemish glory; conquering the desktop. I didn't know this, so it took a while and a lot of kicking Fedora (and Debian) before I finally downloaded the ISO.

I've found my laptop to be a very good tool for testing the various operating systems and distros. Even Fedora in all its hype fell short of the mark, it took me several recompiles and a couple of double back flips to get both my sound and wireless cards working.

The Installation

Having downloaded the 5.10 (Breezy Badger) ISO, and burned it to CD, I slapped in one of the spare 20GB laptop hard drives I have sitting around and booted from the CD. The Installer is the first place you will recognize the similarity to Debian, I use the word loosely because while it is clearly derived from the Debian installer, it doesn't overload the user with options you may, or more likely, may not understand.

The installation is mostly automated, only stopping a couple of times to ask for vital details. The complexity of the install process falls very close to that of windows, and while Fedora may have a flashier interface, it also gives more options. In some respects, particularly when dealing with inexperienced users overwhelming options can be a bad thing.

So far so good; the installation flies along at a nice pace and I'm left looking at a VERY brown login screen.

First Impressions

Entering the username and password that I setup during the installation process presents me with a fairly standard Gnome interface. Perhaps my only beef with the distro is this standard interface. I'll make no secret that I don't like my menus up the top. I used a Mac for years when I was younger, and if there is one thing I like about Windows its the menu at the bottom.

This is supposed to be be geared at newer users, people that have used Windows will expect the Menus and the like to be in a certain location. It is easily remedied (for someone who knows what their doing), but I would have liked to see this done better.

The applications menu, once I've gotten over my distaste at having to move my mouse to the top left hand corner is a pleasant surprise. It is well organized and contains only a minimum of programs that form what I would consider the star performers of the open source movement. More importantly it includes a simple method of installing/uninstalling applications just in case.

Hardware

Now Linux is great on stock hardware; server hardware. But when I've tried putting Linux, both Fedora and Debian, on my D800 Laptop, I had severe trouble getting the sound and wireless working. If I have to recompile the kernel that's an instant failing grade in the hardware department. I like recompiling the kernel, seeing if I can squeeze a little extra performance from my machine...my computer illiterate windows friends just look at me strangely when I say the word 'recompiling' and lets not even talk about 'kernels'.

To my shock and amazement, my sheer delight. Both the sound and the wireless card (Intel 2915ABG) work by default. I can even press the volume buttons on my laptop and see the volume bar come up on the screen. VERY COOL.

I get instant support for my USB Hard Disk, although I can't write to it (NTFS). When I do manage to track down a disk with FAT32 I can write to it as well, BY DEFAULT from a non root account. If you haven't noticed I put a lot of stock in the 'by default' ability.

On the downside my internal dialup modem doesn't work, this would not be a problem usually in the day of broadband. However having just moved I find myself reduced to dial up. Slotting in a spare PCMCIA card solves the problem.

Another annoying factor during boot is the fact that it sits trying to DHCP the interface(s) even though one or both of them are not plugged in. It slows the boot down, which is a pity, perhaps in future releases they'll address this issue.

Conclusions

Ubuntu has two other flaws that I could pick; It doesn't install a MP3 Compatible music client by default, and the games installed by default lack consistency and are essentially, people really do play the stupid games provided, so WHERE is my Solitaire clone.

I am thrilled with the 'Out of Box' experience that Ubuntu has shown. There are tools, and applications that I would probably use that have not been installed, but then again I'm not your garden variety computer user either.

While there are things that this distro could do better, partially this is taste over necessity (Brown). Overall I would give a 8/10 to Ubuntu as a Desktop Operating System. Of all the things I should thank Ubuntu for; it's my restored faith in Linux, and its ability to become a viable desktop alternative.

The Great Debate: Windows vs. Linux

2005-10-31T00:00:00+00:00

Round 1

Everybody seems to be going nuts over the 'Open Source Movement' at the moment; it's become very hip to cheat programmers out of a living. Linux is maturing nicely and with it comes a wave of wannabe revolutionaries. Yet for all Linux and the open source movement's glory, I'm still primarily using a Windows PC. Sure I'll be the first to admit that I chuck in a Linux HD every so often and tinker around, but for the most part at the end of the day I find myself using Microsoft Windows XP.

Where Linux doesn't measure up

Serious tools, for serious users. The open source movement has scored a couple of really useful and widespread tools, Apache being a perfect example, it has some excellent documentation, but on loading it up on my laptop I still have to deal with a number of issues that I wouldn't have to deal with under Windows. Lets see,

To get a decent video resolution I have to install the NVIDIA driver, which I also have to do under windows, but under linux it doesn't just compile, no I have to download the kernel source for the kernel that I'm currently running. Eventually the driver will compile, once done I go looking for the config file, so that I can change 'nv' to 'nvidia' why it can't do it itself is anyone's guess, my windows driver installs with a minimum of fuss why can't my Linux one do likewise.

Enter my wireless adaptor. Linux does support it, and I did eventually get it to work, but only after I recompiled my kernel a couple of times, doing the Chumpeka mating dance, and praying for divine help; although not necessarily in that order. Functionally this is unacceptable, I'm an advanced user (if I do say so myself), while they have certainly made compiling kernels easier in recent times, there is no way a beginner is going to be technically adept enough to configure the (Intel 2915ABG) wireless card under linux.

Enough with the hardware

The level of configuration tools varies widely between the different distros, regardless of which one you use, the end-user tools you are provided with come nowhere near measuring up to the control panel and administrative tools that are provided (and for the most part taken for granted) by Windows Users. All the important configuration options can be found in one place, and while not perfect they are a damn sight better than the level of configuration that is provided by any of the various Linux desktop environments.

I'm not talking command line tools, nobody but nobody is going to argue which operating system has a superior command line, but the majority of plebs out there run into trouble remembering passwords, let alone a myriad of arcane commands and arguments.

Slack Programmers

There is a fundamental problem with developing end-user applications & tools on your own time, and for little or no money. Where a traditional company has to maintain a level of professionalism, and develop programs that are well rounded, and well supported. The open source community has no such responsibility. While some of the bigger projects DO offer a level of professionalism, time and a again I find tools that are not quite polished enough to be considered 'good', the author has lost interest, and because of the fact that they didn't write the program well enough in the first place no one, will (or is stupid enough too) step into the breach.

Remote X11 Applications on Windows

2005-08-10T00:00:00+00:00

Update: There are now much better ways of installing an X Server on windows (for details start with the XMing project)

The primary focus of this document is setting up Cygwin on a given client computer so that it can see graphics as sent from the server. Most of the configuration is done at a client level, however there are minor modifications that also need to be made at a server level.

Installing Cygwin

For the uninitiated Cygwin is a free application, it can be installed on windows to simulate the Linux OS. Since our server runs on Linux this is a very good thing. You can start by getting the installer from http://www.cygwin.org.

Rather than duplicating an existing document, follow the install procedures found at this site:

http://x.cygwin.com/docs/ug/setup-cygwin-x-installing.html

Configuring Cygwin

The installation will take a while to download the packages depending on your connection speed and the server that you selected to download from. Once installed it is simply a matter of configuring Cygwin to take connections from remote hosts (it defaults to not allowing remote connections). This is file that you want to:

C:\Cygwin\usr\X11R6\bin\startxwin.bat

Right down the bottom of the file you will find the following lines:

run XWin -multiwindow -clipboard -silent-dup-error

REM Startup an xterm, using bash as the shell.

run xterm -e /usr/bin/bash -l

Change it so that it looks like this:

run XWin -multiwindow -clipboard -silent-dup-error

run xhost +

REM Startup an xterm, using bash as the shell.

REM run xterm -e /usr/bin/bash -l

Adding the ‘run xhost +’ line tells it to allow any host to send graphics to your computer. Alternately you can specify an exact IP address after the plus to only allow that computer to send the graphics. Additionally you’ll want to put a ‘REM’ in front of the ‘run xterm’ line otherwise it will pop up a window every time you start it. Save the file and put a copy into your startup directory. If you double click it now you should see a little ‘X’ icon appear in the system tray at the bottom right of the screen. Your computer is now ready to receive images.

Setting up the Server

It is possible that your server is already setup correctly, but we’ll need to check to make sure that it’s got the correct address when its trying to send the images out. You can check this from within Pinnacle, you should see in the top right hand corner an IP address, four numbers separated by dots(e.g. 192.168.0.25) if its there is a number there you can skip right on to testing the images from within pinnacle. If not I’ll need to edit some file. Give me a call

That’s it. Hopefully you have been rewarded with some images. If not give us a call and we’ll work something out.

Hackers: Don't Fear Them...Learn From Them

2004-06-08T00:00:00+00:00

Without debating whether hackers are good, bad or simply a myth created by companies that need extra money, the common garden variety of hacker has a lot to share with those of us that are less computer literate. Hackers spend inordinate amounts of time delving into computers, and more to the point looking at computer security. Rather than fearing the hackers, we can take a page from their book, and focus a little on how to secure your computer.

Antivirus

It sounds simple, and it is, but having a virus scanner on your computer is the first step towards securing it against intrusion. While not all viruses breach security, there are plenty that do, sending documents from your computer to complete strangers, or simply reporting passwords. It’s a good enough reason to download an up-to-date virus scanner.

Its important to note that for a virus scanner to be effective, it needs to be kept up to date. If it hasn’t got the latest patterns/definitions then you might as well not have it at all. There are plenty of virus scanners out on the market, and the one you use is about preference, the URLs below should help you out:

Spyware Removal Tools

One of the most annoying aspects of the internet today is Spyware/Adware. Not only do they cause the computer to stop functioning correctly, various types of spyware also funnily enough, spy on you computer usage. Removing them can be a stressful and time consuming effort, the links below are useful in removing spyware, and protecting your computer against further infection.

Firewalls

Perhaps the most misunderstood piece of technology on the planet, the firewall is a piece of software that sits between you computer and the rest of the network, or in bigger setups, between the local network and internet. Firewalls work by stopping traffic coming into a given area, while still allowing traffic to go out.

For the price of a cheap computer you can setup your very own linux firewall, around your network, customising it down to the finest details as to what you want to let in and out. It is an important step, but one that most people do not need.

The links below are for firewall applications that can be installed on your computer and used to filter both incoming traffic, and outgoing traffic generated by applications. This will help prevent attackers from gaining access to your computer.

Encryption

Less used than even the firewall, encryption is where you encrpt files and text so you can securely pass it through the internet, without the risk of anyone snooping in on your information. There are many tools for encrypting information, perhaps the most well known is PGP or pretty good privacy, which can be used to encrypt both files on your computer and also emails that you send.

Conclusion

The above applications are not the end of your security efforts, to tell you the truth, they are only the beginning. Having a secure computer is about methodology, if you are consciously trying to secure your computer, then you will be aware when you do something that makes your computer less secure. There is not harm in being a little paranoid, and certainly no harm in protecting your assets against unwanted intrusion, on that issue, hackers have their stuff sorted out.