Jason Pollock's Controversial Originality

Adventures in Electricity.

2011-10-15T17:30:00.000-07:00

Our power bill was getting huge. We were paying $350/month, and that was in the summer. Winter bills would be considerably larger. It was well past time to do something about it.

First up, change electricity providers. In New Zealand, Consumer NZ researched all of the electricity providers and put up a web site, Powerswitch. Now, according to Powerswitch, I was with the most expensive power company (Contact Energy). So, I switched to Powershop. They weren't the cheapest, Meridian Energy is the cheapest, but Powershop had amazing reviews. I also wanted a smart meter so that I could easily track our usage, and Powershop makes them available.

So, I switched to Powershop. According to PowerSwitch, I would save $675 on a bill of $4066, or >15%!

Side comment: Make sure to provide a final reading to your provider. They have a habit of over estimating your power usage to keep your as a customer as long as they can - you pay the estimated final reading, not an actual reading.

In New Zealand, there are different ways of saving money when buying electricity. The first and easiest is to sign up for ripple control. When you sign up for this, in times of peak power demand, the power company will turn off your water heater. If you give them this ability, they will discount your power by ~10%. Sounds like a good idea, except for one problem - it doesn't work. I repeatedly had the ripple controller turn off the water heater and then fail to turn it back on. Nothing is more unpleasant than getting into the shower and there not being any hot water, calling a plumber, who tells you the problem is electrical, and then calling an electrician who tells you the problem is the lines company. TWICE.

When that happened, I turned off ripple control, and I will never turn it back on.

Powershop gives you a different way of saving money. They offer time of day charging. This provides a discount on electricity between the hours of 11PM and 7AM. It has the same effect as ripple control, only it's under my control instead of the lines company's.

With the Powershop "All Inclusive" plan, they currently charge 23.93c/kwh. This is for the ripple controlled power. Then there is uncontrolled at 26.23c/kwh, that's ~10% more.

However, if by switching to Day/Night Uncontrolled, I'm paying 24.07c/kwh day and 15.17c/kwh night. That means that as long as I shift a certain amount of power to the night, I end up saving, and don't need to keep ripple control. The magic ratio is:

23.93 = 24.07 * x + 15.17 * (1-x)
23.93 = 24.07x + 15.17 - 15.17x
8.76  = 8.9x
0.98  = x

That means that as long as I move at least 2% to my night plan, I'm better off. Seems strange, the service rep indicated that the magic number was 10%.

I switched plans, and installed a smart meter, which cost $165. The savings from 26c/kwh to 24c/kwh (the minimum savings), with an average usage of 1600kwh/month, I have a payback of:

2c/kwh * 1600kwh = $32/month
$160 / ($32/month) = 5 months.

five months! That's pretty kick-ass in anyone's book.

To recap:

Shop around. You can save a tonne of money with another provider. Switching is free and easy, just head over to Powerswitch.org.nz
Change plans. Investigate day/night plans, particularly if no one's home during the day,
We're at a savings of >25% from where we started, with more to come.

Powersavings and water heater.

2011-09-03T16:36:00.000-07:00

I have switched to Powershop, a New Zealand power company that provides intelligent meters with daily readings. This makes it really, really easy to see the impact of changes you make to your power usage. Not only that, but they provide time of day billing, where power from 7AM to 11PM is 50% more expensive than night time power.

That convinced me to look at how we used power. First, I shifted the laundry and dishwashing. I made it either the last thing I did before bed, or the first thing I did when I got up. That made a moderately substantial change in our bill, but nowhere near what I was hoping for. It was time to track down the big user in our power bill.

We are currently using about 40-50kwh of power per day. In New Zealand, we have expensive power prices ($0.25/kwh), so that's $10-$15 per day. Not cheap. The power meter we've got will tell us how much power we are currently using, which allowed me to zero in on the big users. First, just sitting there doing nothing, we use ~200w. That's for the server (with six HDs), networking and other vampires. Not bad, and not the main power user, they are constant background use, so they only end up being 5kwh/day. 10%, but not bad, bottom of the heap to fix. Next are the heat pumps and dehumidifiers. The dehumidifiers pull about 400w each, so they're a biggy if they were running all day. Heat pumps, also not a biggy, although they can pull about 5kw each. We're going to have to look at passive dehumidifing for the house if we keep it.

The biggest power user in the house is the hot water heater. It looks like it is using about 20kwh per day. However, we don't need a lot of hot water. We don't use it for laundry, the dishwasher uses cold water and our showers all center around 6-7AM.

I looked into ways to save power. The first and easiest is to turn off the hot water heater when you don't need it, so I did that. Our power usage collapsed. From 45kwh to 24kwh. That's a savings of $5/day. Not only that, but we had plenty of hot water for the rest of the day. If we were worried about running out of hot water, we could turn the water heater back on, and it would be at temperature within an hour.

However, there's a big scary monster out there in water heaters - Legionella. The bacteria loves warm, stagnant water between 20C and 45C. That means it loves electric hot water heaters. The recommendation is that water should reach your tap at at least 50C to prevent legionella from colonising the pipes and fixtures.

Oddly, it seems that legionella is found in 40% of electric water heaters, but the disease doesn't seem to be that prevalent (1 case per 100k population/year). It only appears to be a concern for people who are immune compromised. If it does take hold though, it is very dangerous if not caught quickly.

Because of that, we decided to turn the water heater back on during the day. I decided to attack the problem from another direction. Depending on how quickly the water heater loses heat, I can keep it above 50C while still saving on power. If the water heater doesn't drop below 45C during the day, it can be left off safely, bringing it back up to 60C at night, quickly killing off legionella and making use of night rates.

I will conduct some experiments to see:

What the temperature of the water is in the heater.
What temperature water is reaching the taps at.
How quickly the temperature drops off.
How quicly the heater drops into the danger zone.

Map of Earthquake Prone Buildings in Wellington

2011-03-06T15:30:00.000-08:00

On the weekend, I took the PDF that Hon. Peter Dunne (my local MP) provided of the earthquake prone buildings in Wellington New Zealand and put them onto a map.

I initially tried to use Google Docs to do this, because I thought it should be relatively easy to map data from a Google Spreadsheet. I thought that this would be an initial usage scenario. It seems that it isn't. So, instead I used BatchGeo. They have a pretty quick and easy service, complete with Geocoding.

Here is what their online tool created when I gave it the CSV file:

View Wellington NZ, EQP Buildings in a full screen map

The full screen map includes a list of the locations mapped.

Of course, the next morning, I found out that James over at maptd used Google Fusion Tables, which allow you to directly map data from a spreadsheet onto Google Maps. I'll have to look into them next. I would like to be able to keep the grouped view that BatchGeo provides in any reworking.

Cheaper Internet TV

2010-01-17T13:57:00.000-08:00

A while ago, I did some math to show how streaming TV through a US proxy on Amazon was cheaper than torrenting the show, and how it was even competitive with Sky TV if you watches less than 12 hours of TV a week.

That was if you used Amazon as your US proxy. With Amazon, you are paying for both traffic and time, which skews the price considerably. If you already have a US server (several of my friends do) then you already have your US proxy with a huge bundle of data. However, the rest of us need another, easier way.

Of course, the US being the US, and the Internet being the Internet, there is already several groups offering a range of methods, starting from free and getting more expensive from there.

First off, some math:

From last time, we saw the price for Amazon was:

Amazon Cost:
  Instance: US$0.1*(NZ$1/USD$0.67) = NZ$0.15
  Traffic (based on 1 episode of Eureka)
  (304MB/43min)*(60min/hr)*(1GB/1024MB)*(USD$0.27/1GB)*(NZD$1/USD$0.67) = NZ$0.17
  Total   : NZ$0.32/hr
NZ Bandwidth cost:
  (304MB/43min)*(60min/hr)*(1GB/1024MB)*(NZ$1.50/GB) = NZ$0.62/hr
Total cost per hour: NZ$0.94/hr

If we can reduce the Amazon cost, we can substantially reduce our total cost.

My goal is to get the "Amazon" portion of the charge down to 8c/day. That would make the price for a Hulu stream into:

NZ$0.62/hr + $0.08/day

So, the break even point against the Sky basic plan would become:

($11.74 - 0.08*7)/0.62 = 18 hours/week.

Not quite to the 3hrs/day that I have previously boasted about, but close. Within spitting distance, even. If you can get bandwidth for less than NZ$1.50/GB, the break even point will change dramatically. Given that the Southern Cross prices have just dropped by 50%, I hope to see some end-user price changes soon.

So, I went looking for a VPN provider. First I tried free, Hotspot Shield from AnchroFree. I quickly gave up on them. They don't have enough bandwidth, and they seem to be specifically targetted by Hulu.

I thought about using TOR, but quickly discarded that one. It would be just plain rude.

I looked at using the free proxy add-on for Firefox. However, they just aren't reliable enough for the next stage - a set top box.

So, I went looking at commercial offerings. I tried StrongVPN.com, WiTopia.net and HideMyNet.com.

VPN is still pretty new and arcane. There is a lot of magic, and many things that can go wrong. I found it very frustrating to get properly configured, and ultimately had to rely on the support teams at a couple of the providers to get configured. Thankfully, both StrongVPN.com and WiTopia.net offered good support. HideMyNet (and others) were not so good. I had other issues too. Sometimes it felt odd giving these companies my credit card details. So, I only went with ones that used PayPal.

Notes:

Avoid PPTP. It's generally provided by the OS, and is flakey and hard to get configured properly. It may be cheaper, but it's not really worth it.
Go with an OpenVPN based solution. This is available on Windows, OS X and Linux. The benefit is that it is it's own tool, separate from the network stack and therefore a lot easier to diagnose.
CPU, CPU, CPU. My 2.4Ghz AMD single core Athlon 3800+ wasn't fast enough to stream more than 700kbps reliably. My 2.5Ghz Core2Duo is.

StrongVPN.com offers points of presence all over the world. However, they are moderately expensive at US$85/yr, and charge money to change PoPs (Point of Presence). In other words, while they do have PoPs in the UK (for access to BBC iPlayer or Spotify), if you want to switch between UK/US PoPs, they charge money. It really sounds like they should fix their provisioning and access control systems to me.

WiTopia.net is the company that I've ultimately gone with. They offer instant switching between VPN PoPs, UK and US PoPs and a low annual fee of US$60. Even better, they have US PoPs close to the Southern Cross US interconnect points in LA and San Francisco.

Since I've gone with WiTopia.net, Hulu came along and banned them from accessing their streams. However, a quick support message and I was on another PoP, which wasn't filtered (yet). However, I've also got access to BBC's iPlayer and Pandora, which kicks ass.

I just wish NetFlix was supported on Linux, then I would sign up for a NetFlix streaming account.

The final cost analysis works out as:

WiTopia.net Cost:
  Cost: NZ$80/yr
NZ Bandwidth cost:
  (304MB/43min)*(60min/hr)*(1GB/1024MB)*(NZ$1.50/GB) = NZ$0.62/hr

Break-Even point vs Sky: 
  ($11.74 - $80/365)/0.69 = 17 hours.

In other words, if you watch less than 17 hours of Sky specific TV per day (it isn't available on FreeView), you are better off paying for a proxy account and obtaining the content from the US or the UK.

Topic Relevant Comment Spam

2009-11-30T13:22:00.000-08:00

On my previous post "Streaming Audio to your iPhone", I received a very odd comment. It was related to the subject, so it didn't immediately look like spam, but it wasn't topical.

Here's the comment (I've removed it from the original post):

Web casting, or broadcasting over the internet, is a media file (audio-video mostly) distributed over the internet using streaming media technology. Streaming implies media played as a continuous stream and received real time by the browser (end user). Streaming technology enables a single content source to be distributed to many simultaneous viewers. Streaming video bandwidth is typically calculated in gigabytes of data transferred. It is important to estimate how many viewers you can reach, for example in a live webcast, given your bandwidth constraints or conversely, if you are expecting a certain audience size, what bandwidth resources you need to deploy.

To estimate how many viewers you can reach during a webcast, consider some parlance:
One viewer: 1 click of a video player button at one location logged on
One viewer hour: 1 viewer connected for 1 hour
100 viewer hours: 100 viewers connected for 1 hour…

Typically webcasts will be offered at different bit rates or quality levels corresponding to different user’s internet connection speeds. Bit rate implies the rate at which bits (basic data units) are transferred. It denotes how much data is transmitted in a given amount of time. (bps / Kbps / Mbps…). Quality improves as more bits are used for each second of the playback. Video of 3000 Kbps will look better than one of say 1000Kbps. This is just like quality of a image is represented in resolution, for video (or audio) it is measured by the bit rate.

It was posted by "Andy". I was ready to post a comment thanking Andy for the additional information, but I decided to look to see if it was copied from somewhere else.

It seems that the exact same comment is placed on just about any blog post that mentions streaming. On the Google search I used, I found a lot of comments from the past couple of months.

Seems "Andy" has been a busy boy, and is actually a dirty, rotten spammer.

"Andy" (blogger), who also posts as "andylock", or "Andy Lock" (Facebook) is an automated spam program for vsworld.com, and the website is a flash-only website. I didn't stay there long enough to really figure out what they were selling, but it looked like some sort of contracting agency in India.

Still, I found it interesting that it wasn't immediately obvious that the post was spam.

Audio Streaming to the iPhone, Take Two

2009-11-23T16:00:00.000-08:00

I wasn't completely happy with the results of using M3U files. While it did allow me to specify all of the files, I wasn't able to skip forward and back through the song list.

It turns out that the M3U support is there primarily to support live streams. For example, if you have a live stream from a video source (like a TV tuner card), the server records it, splits it into small chunks and converts it to H.264. The iPhone will then use the M3U file as an index to the individual data files, refreshing the M3U file occasionally. I'll have to set that up next. :)

So I went back to Google to see what I could find. I had come across the <OBJECT> tag in my previous searches, but I had discarded it as too difficult. I didn't want to work in HTML, and object embedding just seemed dirty. However, if I wanted to get access to skip forward/back, it looked like I was going to have to use it. I went through and modified the perl program from yesterday to produce an HTML page for each M3U file. It wasn't until I got to the end that I realized that I had done it wrong.

Lots of things don't work with the OBJECT tag, primarily the GOTO command. It seems that the proper way to embed things into a document is with the EMBED tag. That isn't to say that the EMBED tag is prettier. It isn't, they are both nasty. It is platform and player specific. Luckily, this only has to work on the iPhone/iPod Touch, so I'm lucky that way. I would hate to have to support this for multiple clients.

The first nasty surprise is that while you can have multiple songs in a list, you are limited to 255 of them. Even worse, they aren't done through object references in the rest of the document, the entire list is in the single object tag. That makes it harder to do dynamic, on the fly modification of the list. No small cgi to do shuffles here!

Still, 255 songs is enough to cover pretty much all of my artist directories.

It also seems that Mobile Safari ignores GOTO commands. In regular Safari, you are able to loop back around to the start of the playlist by putting "QTNEXT255=GOTO0" into the embed tag. Looks like Apple doesn't want playing loops on the iPhone.

Next, the iPhone ignores the "autohref", "autoplay" and "autostart" parameters. It always waits for user interaction. This is because the object is not really embedded, it takes full control of the screen. If it did start automatically, it would cause problems on many other sites. It's a small pain, but we'll survive.

I still wish it properly supported M3U files. The seamless transitions are nice. With the QTNEXT, there is a definite pause between tracks.

Here is the updated Perl code:

#!/usr/bin/perl

use File::Find;
use File::Basename;

use vars qw/*name *dir *prune/;

my $URL_BASE= "http://10.10.10.5/";

my @M3Us;

sub createPlaybackHTMLHeader {
    my ($filename, $target, $song) = @_;

    my $title = basename(dirname($filename));

    # open the file, 
    open(HTML, ">$target");

    # Write the HTML/Head elements,

    print HTML <<END;
<html>
  <head>
    <title>$title</title>
    <meta name="viewport" content="width=device-width; initial-scale=1.25"/>
  </head>
  <body>
    <p>Play all music in the "$title" directory, click the button below</p>
    <embed src="$song">
      autoplay="true"
      controller="true"
END
;
    # close the file.
    close(HTML);
}

sub createPlaybackHTMLFooter {
    my ($filename, $target) = @_;

    # open the file, 
    open(HTML, ">>$target");

    # Write the HTML/Head elements,

    print HTML <<END;
      qtnext255="GOTO0"
    </embed>
  </body>
</html>

END
;
    # close the file.
    close(HTML);
}

sub createPlaybackAddSong {
    my ($filename,$target, $song, $count) = @_;

    open(HTML, ">>$target");
    print HTML <<END;
      qtnext$count="<$song> T<myself>"
END
;
    # close the file.
    close(HTML);
}

sub convertM3UToHTML {
    my ($m3ufile, $target) = @_;

    print "Converting $m3ufile to $target...";

    if ( ! -f $m3ufile ) {
 print "No m3ufile, returning\n";
 return;
    }

    # Open that m3u file, and convert it to a playback html page.
    open(M3U, $m3ufile) or die "Can't open m3u file";
    my @songs=<M3U>;
    close(M3U);

    if (@songs > 255) {
 print "Too many songs! Oh well\n";
 @songs = @songs[0..254];
    } elsif (@songs == 0) {
 print "No songs, returning\n";
 return;
    } else {
 print "No problems!\n";
    }

    my $first_song = shift @songs;

    chomp $first_song;

    createPlaybackHTMLHeader($m3ufile, $target, $first_song);
    my $count = 1;

    foreach my $song (@songs) {
 chomp $song;
 createPlaybackAddSong($m3ufile, $target, $song, $count);
 $count += 1;
    }

    createPlaybackHTMLFooter($m3ufile, $target);
}

sub entering {
    print "entering Directory boundary ", $File::Find::name, "\n";

    push @M3Us, $File::Find::dir;

    if ($File::Find::name =~ /.AppleDouble/) {
 return;
    }

    return sort(@_);
}

sub leaving {
    print "leaving Directory boundary ", $File::Find::name, "\n";

    my $directory = pop @M3Us;

    # open the file, 
    my $source = $directory . "/PLAY.m3u";
    my $target = $directory . "/PLAY_ALL.html";

    convertM3UToHTML($source, $target);
}

sub wanted {
   print "Checking ", $File::Find::name,"\n";

   if ( $File::Find::name =~ /\.mp3/ ) {
       print "MP3 Found ", $File::Find::name, "\n";
       my $url = $File::Find::name;
       $url =~ s/ /%20/g;
       $url =~ s/\/export\///g;
       $url = $URL_BASE . $url;
       foreach my $dir (@M3Us) {
    my $m3u_file = $dir . "/PLAY.m3u";
    open (M3U, ">>$m3u_file") or die "Boom!";
    print M3U "$url\n";
    close(M3U);
       }
   }
}

sub cleaner {
    if ($File::Find::name =~ /PLAY.m3u/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }

    if ($File::Find::name =~ /PLAY_ALL.html/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }
}

find ({ wanted => \&cleaner},"/export/mp3");

find ({ wanted => \&wanted , preprocess => \&entering, postprocess => \&leaving},"/export/mp3");

open(M3U, "/export/mp3/PLAY.m3u") or die "Unable to open play";

my @main_m3u = <M3U>;
close(M3U);

my @music = grep(!/AudioBooks/, @main_m3u);
my @random = sort { int(rand(3))-1 } @music;

open(M3U, ">/export/mp3/MUSIC.m3u") or die "Unable to open MUSIC.m3u";

print M3U @music;
close(M3U);

convertM3UToHTML("/export/mp3/MUSIC.m3u", "/export/mp3/MUSIC_PLAY.html");

open(M3U, ">/export/mp3/RANDOM.m3u") or die "Unable to open RANDOM.m3u";
print M3U @random;
close(M3U);

convertM3UToHTML("/export/mp3/RANDOM.m3u", "/export/mp3/RANDOM_PLAY.html");

Streaming Audio to your iPhone

2009-11-22T20:00:00.000-08:00

I'm trying to hide the CDs and DVDs that are steadily taking over my house. I've managed to get rid of all of the DVDs, they've all been ripped to the server and are now happily up in the attic. The last problem are the CDs.

Yes, I have them all ripped to the server. The problem is that I don't have any way of getting at them everywhere in the house. Specifically, my wife likes to listen to murder mysteries as she cooks (me, I like cooking to the news). Since I don't have a network enabled stereo, I have to put up with a stack of CDs sitting on the counter.

I have given up waiting for a network enabled stereo system. The squeezebox just doesn't do it for me. I want something small, with wifi and wired, with a radio, clock and a streaming mp3 player. It should also have .m3u support and a "shuffle".

I decided to see what I could do with what we already have...

First realization. We both have iPhones.
Second realization. We have a WiFi network in the house.
Third realization. The iPhone will happily stream music off of a web page.

So the project was formed. First, I tried the SqueezeBox server (previously SlimServer) a server which serves up streams to network attached stereo component equipment. I've got a 1st generation SliMP3, it was a great device. The server just doesn't play well with the iPhone. So, I decided to go the bare bones route. I decided to setup Apache on my file server so that it will serve up music to the iPhones in the house. Then the phones can be used to stream the music/books wherever anyone is in the house. I would finally be allowed to hide all of the CDs! Perfect!

Setting up Apache was pretty easy. I followed the instructions that are easily Googled. I didn't create any HTML files, but I did configure the auto_index module (/etc/apache2/mods-available/autoindex.conf). My changed settings were:

IndexOptions SuppressDescription SuppressSize SuppressLastModified
IndexOptions SuppressHTMLPreamble
HeaderName /include/iPhone_Header.html

NOTE: HeaderName is relative to the DocumentRoot, not the filesystem.

This allowed me to keep the listing to just the filenames, and replaced the standard HTML header with one of my own:

<html>
<head>
<meta name="viewport" content="width=device-width; initial-scale=1.25"/>
</head>
<body>

This header provides a hint to the iPhone of where to set the viewport. It seems to work for my listings, making the file list usable.

Once we have that, we've got a server with a directory listing that we can scroll through and play individual MP3s. We don't have any playlists though, which is pretty unpleasant.

Bring in the Perl!

I've written a small piece of Perl code to iterate over my MP3 tree and create M3U files in each directory containing URLs for all of the MP3s that are children of that directory. Because of how I store my MP3s, that gives me album, artist and full play lists. I then randomize the playlist to give me a shuffled list.

The new iPhone release is able to play M3U files, with one problem. You can't skip to the next track, which is pretty poor. But now I've got a method of delivering music to any room in the house. I just have to get one of those 3rd party iPod speaker systems, and I can get rid of all of the CDs in the kitchen!

#!/usr/bin/perl

use File::Find;

use vars qw/*name *dir *prune/;

my $URL_BASE= "http://10.10.10.5/";

my @M3Us;

sub entering {
    print "entering Directory boundary ", $File::Find::name, "\n";

    push @M3Us, $File::Find::dir;
    return sort(@_);
}

sub leaving {
    print "leaving Directory boundary ", $File::Find::name, "\n";

    pop @M3Us;
}

sub wanted {
   print "Checking ", $File::Find::name,"\n";

   if ( $File::Find::name =~ /\.mp3/ ) {
       print "MP3 Found ", $File::Find::name, "\n";
       my $url = $File::Find::name;
       $url =~ s/ /%20/g;
       $url =~ s/\/export\///g;
       $url = $URL_BASE . $url;
       foreach my $dir (@M3Us) {
    my $m3u_file = $dir . "/PLAY.m3u";
    open (M3U, ">>$m3u_file") or die "Boom!";
    print M3U "$url\n";
    close(M3U);
       }
   }
}

sub cleaner {
    if ($File::Find::name =~ /PLAY.m3u/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }
}

find ({ wanted => \&cleaner},"/export/mp3");

find ({ wanted => \&wanted , preprocess => \&entering, postprocess => \&leaving},"/export/mp3");

open(M3U, "/export/mp3/PLAY.m3u") or die "Unable to open play";

my @main_m3u = <M3U>;
close(M3U);

my @music = grep(!/AudioBooks/, @main_m3u);
my @random = sort { int(rand(3))-1 } @music;

open(M3U, ">/export/mp3/MUSIC.m3u") or die "Unable to open MUSIC.m3u";
print M3U "@music";
close(M3U);

open(M3U, ">/export/mp3/RANDOM.m3u") or die "Unable to open RANDOM.m3u";
print M3U "@random";
close(M3U);

Oracle's BETWEEN keyword

2009-11-15T13:29:00.000-08:00

I came across a new Oracle keyword today, BETWEEN.

At first, I thought it was pretty cool. I would be able to simplify the majority of the range checks that I perform. Before I really started using it though, I decided to look at what it actually did. Ouch!

Google, my ever present documentation source, told me that between doesn't work the way I thought it would. It's inclusive of both ends of the range. Who would want that? You never want a range that is inclusive of both ends! Otherwise, elements in contiguous ranges have indeterminate ownership!

Let's try an example:

We want all rows with a date on the row that has a date of today.

  select id from test_table where my_date between 
    trunc(sysdate) and trunc(sysdate) + 1;

Great you would think, almost too easy!

You would be correct too, it was too easy. It doesn't work. Since it is inclusive of the end of range, you get all values that have a my_date of 12:00:00 tomorrow. What you really want is:

  select id from test_table where 
      (my_date >= trunc(sysdate)) and 
      (my_date < trunc(sysdate) + 1);

When using new keywords and abstractions, you should always know what they are doing.

std::auto_ptr and GDB.

2009-11-02T13:04:00.001-08:00

I needed to gain access to the contents of an auto_ptr inside of GDB. However, GDB doesn't like the overloaded -> operator, so the simple foo->fnImInterestedIn() doesn't work. Here's the simple pattern:

class bar_t {
  public:
     int fnImInterestedIn();
};

std::auto_ptr<bar_t> foo;
(gdb) p ((struct bar_t *)foo._M_ptr)->fnImInterestedIn()

Hulu Proxy Apocalypse

2009-11-02T11:27:00.001-08:00

It seems that the great Hulu apocalypse has hit more than just Witopia.net, Amazon's EC2 instances also seem to be blocked.

Of course, there are a tonne of other cloud providers out there, several of them even cheaper than Amazon. Personally, I'm still with Witopia. Like any good company, they had a new address range up immediately.

Hulu is stuck in the same game as Apple. They're playing whack-a-mole with the hackers. The only problem, every time they want to block an access method, it costs them money. However, it is free for us to invent a way around a block (it's a hobby), and there are a lot more of us than there are developers at Hulu.

This is yet another example of "Don't piss off the nerds".

Personally, here are the lessons that I would take from this.

There is a market for International access. People are willing to _pay_ to get access. I am currently paying US$12/month to access Hulu, Pandora, and TV.com. I would pay that for unfettered access to Hulu.
Hulu isn't going to win with black lists. They're going to have to implement a white list, which is a lot more expensive to maintain.
Not a single person who was using Hulu simply stopped watching TV when Witopia and Amazon were turned off, they just went back to bittorrent.
Don't piss off the nerds. They can out-spend you.

There are plenty of cloud providers out there. The same solution will probably work on other sites. Install Squid and give it a try!

Using Amazon EC2 to access Hulu

2009-09-05T16:58:00.000-07:00

As more and more content moves onto the Internet, it is frequently provided on a region-by-region basis. A lot of people want to have access to Hulu, Pandora, ABC, NBC, Netflix and the US Amazon and iTunes online stores.

I was one of them. S92A also provided an impetus to my research. If downloading US content was going to result in disconnection, I needed another way to get my North American TV fix. It would be even better if it was legal.

A little-known section of the NZ copyright law makes it legal to break DRM if the only purpose is to provide a region lock. To me, that indicates that if I can get around the geographic IP block on these web sites, I am no longer breaking NZ copyright law by watching the shows.

Perfect.

There are a couple of methods to do this. The ultimate method is using an OpenVPN server on EC2. I didn't start there. I started by using Squid.

First, you will need to learn how to construct an Amazon EC2 instance. This requires setting up an account, downloading the tools and starting an instance. Nothing too difficult, and all described by Robert Sosinski.

Since this will be a network proxy, we don't need a fast CPU, or a lot of memory. The smallest EC2 machine image is perfectly usable. I used a Fedora instance, since that was what I was familiar with at the time.

From the steps in Robert's instructions, I would leave out allowing access to port 80 (ec2-authorize default -p 80). We don't need it for this.

Now that you have a working image, we need to get Squid working. A funny aside, you can tell how mature the Internet is getting by the search results for open source project names. It used to be that if you Google'ed "Squid" you got the HTTP proxy. Now you get cephalopods.

I wanted a set of instructions that could be easily scripted so that I didn't have to leave the instance running, or store anything on S3, Amazon's storage system.

export EC2HOST=ec2-xx-xxx-xx-xx.compute-1.amazonaws.com

First we install and start squid.

ssh -i ec2-keypair root@$EC2HOST  "yum -y install squid"
ssh -i ec2-keypair root@$EC2HOST  "/etc/init.d/squid start"

Finally, we setup a local tunnel from our local machine to the Squid proxy.

ssh -i ec2-keypair -N -L3128:localhost:3128 root@$EC2HOST

To make use of the proxy, all we need to do is point Firefox (or your preferred browser) at localhost:3128. Voila we now have access to the US.

However, if we try to use Hulu, only some of the videos work. We don't get the "not available in your region" error message anymore, instead we get a "unable to play the video at this time". Something else is going on.

Hulu is using multiple layers of security. They are not only checking the source of the HTTP stream, the actual RTMPE stream is protected as well. Time to add more stealth.

First, we tell squid to not tell anyone downstream who they are proxying for:

ssh -i ec2-keypair root@$EC2HOST  "echo "forwarded_for off >> /etc/squid/squid.conf"
ssh -i ec2-keypair root@$EC2HOST  "/etc/init.d/squid restart"

However, that doesn't fix all of the problems. Reading up on the protocol that the flash player uses (RTMP), we see that while it will tunnel over HTTP, it will first try to make a direct connection. It is that direct connection which is causing us problems, so we will turn it off.

sudo ipfw add 2000 deny tcp from any to any 1935 out

Now, when we try to use Hulu, we see that all of the videos are working, the RTMP stream is properly using the HTTP proxy, and Hulu is no longer restricting our access.

However, this isn't perfect. Amazon EC2 seems to rate limit the instances. Even though you are paying per byte of transfer, EC2 doesn't let you have more than 1mbps per connection. That means that while we can watch Hulu, we can't get reliable access to the HD content.

So, how expensive is it?

Here's some math....

NZ Sky subscription, basic plan (no movies, no sports). $11.74/week

Hulu Video

Amazon Cost:
  Instance: US$0.1*(NZ$1/USD$0.67) = NZ$0.15
  Traffic (based on 1 episode of Eureka)
  (304MB/43min)*(60min/hr)*(1GB/1024MB)*(USD$0.27/1GB)*(NZD$1/USD$0.67) = NZ$0.17
  Total   : NZ$0.32/hr
NZ Bandwidth cost:
  (304MB/43min)*(60min/hr)*(1GB/1024MB)*(NZ$1.50/GB) = NZ$0.62/hr
Total cost per hour: NZ$0.94/hr

Bittorrent cost per hour

  SeedRatio = 1.0
  (350MB/40min)*(60min/hr)*(2-(1-SeedRatio))*(1GB/1024MB)*(NZ$1.50/GB) = NZ$1.53/hr

Break even point of Squid Proxy:

vs Bittorrent - instant
vs Sky - 11.74/0.94 = 12 hours.

Therefore, to have any value, you need to be watching content on Sky TV which isn't available on FreeView for more than 12 hours a week in order to justify paying for Sky TV.

So, in this case, going legal is cheaper.

The Future of Cable-TV

2009-03-22T22:48:00.000-07:00

There's a bit of a discussion going on between Mark Cuban and Avner Ronen. It started with someone writing an article about boxee, the company that Avner heads.

Mark seemed to take exception to the article, and posted, "Why Do Internet People Think Content People Are Stupid?" Avner then followed up on his blog.

I've been thinking about Cable and Broadcast TV for a while. I find myself agreeing with Avner, that the Internet will allow customers to have a la carte access to shows and channels.

First off, why would this happen? We only need to look at PVRs to see where it is going. PVRs are changing the relationship between broadcast networks, content creators and viewers. Currently, a PVR changes how a viewer makes use of the TV network. They can watch any show they are interested in, regardless of when it was broadcast, or even if they know it is being broadcast. People with PVRs timeshift TV shows, frequently watching a show up to a week after broadcast. This change in viewing habits is showing up in the TV show's ratings, with some shows seeing a drop of up over 30% viewership as their viewers record it for later viewing.

What happens when the majority of your customers have PVRs? They no longer watch "whatever is on". They will watch the shows that they want to watch, when they want to watch them. The broadcast schedule becomes less important. In fact, it doesn't matter when the show is broadcast anymore. Your viewers will automatically follow the show through scheduling changes.

Some consultants say that people will still want to watch the show during the broadcast slot so that they can talk about it the next day at work. That is true, there is some pressure to watch a show as it comes out, but it doesn't need to be watched immediately. It can be watched just as easily +1,+2,+6 hours later. The person could even elect to watch the show the next morning over coffee or on the way to work on the train.

So, if the timeslot doesn't matter, what does that do to a broadcast network? Frankly, it turns them into two things. First and foremost, they are aggregators. They decide what it is that you are going to watch. Second, they are a very efficient one to many data network. Because of their sunk costs in the form of spectrum and transmitters they are able to deliver content to homes across the world cheaply.

Now, remember, in the world of the PVR, it doesn't matter when a show is broadcast - it'll be found and recorded. This lowers the value represented by prime-time slots. Everyone is watching TV at that time, but it isn't needed to show them pre-recorded content. It also increases the value of the 2-6AM slots. The time when a broadcaster would usually send out a test-signal or infomercials are now perfectly positioned to show syndicated content.

Remember, it's all about feeding the PVR. You want to fill their viewing hours with _your_ content, not the other guy's.

So, we're not in a world where insanely popular shows are broadcast to people's PVRs at 2AM on a Monday morning. Just in case you think this is crazy, this is exactly how bittorrent works with background downloaders like TED (torrent episode downloader). TV shows show up on the pirate networks in the middle of the night, and are available for watching on your local PVR a couple of hours later.

So, now that they're feeding your PVR, what happens next? Aggregation. The networks provide aggregation and editing services. They are tastemakers. The only problem? They demonstrate this taste through their line-up and schedule. As I've already shown, the schedule is unimportant - people use PVRs. That leaves only the line-up. Those "up-next" teasers for the next show? Of little value - the viewer can't watch the next show! So, the line-up also becomes unimportant. There is no way to package shows to make viewers more "sticky". They will choose shows from networks a la carte. Again, we can already see this behaviour - both on PVRs, and bittorrent. There are also better "tastemakers" than networks. They're called bloggers. There are a lot more of them, they are the reviewers of the Internet age. Even better, they are perceived as more trustworthy, because they don't review things for a living. True "tastemaking".

In the world of the PVR, syndication is dead. In the world of 500 channels, how many versions of CBS do you need? How many times a day do you need to watch the same episode of "The Simpsons"? If your viewers have a PVR, the answer is you only need 1 instance. All those cable networks that are syndicating your shows to fill in their schedules? If you put those shows on your own network, with your own ads, you can have that revenue too.

So, the value of syndication has just collapsed. There's only room for one copy of each episode per week (perhaps even per year). All that revenue broadcasters get from cable companies for retransmitting their channels? Gone - the cable company only needs one, and they'll take the local one - they can frequently get that for free.

Let's recap. We've killed the schedule, aggregation and syndication. We have just turned a local TV broadcaster into a broadband pipe that specialises in delivery of video and audio content supported by local advertising.

Does that sound familiar? It should. It's an ISP running a rewriting HTTP proxy, that inserts advertisements into pages that its customers view.

At that point, the model shifts. Content producers sell their own advertisements and purchase time on the broadcast networks. This is how I see it proceeding:

Content producers sell their own advertisements.
They put their back catalog up on the Internet at VoD.
They make the content available on a P2P network.
In markets where they have significant penetration, they purchase time from the local broadcasters, and feed that information to their customers.

Now that would be an interesting synergy. You watch a show on Hulu through Boxee. When you are finished, up pops a dialog box "Would you like to schedule this show for recording in your area?". If you select yes, it instructs your local PVR to record that show the next time it comes around.

All of a sudden, there is a continuous flow between Internet VoD, P2P and OTA (or cable) broadcast. Each with their own strengths.

That'll be cool.

New Zealand S92A

2009-02-23T14:44:00.001-08:00

Just to make sure that my stance isn't lost now that the blackout's been lifted.

I would like to see several changes to the TCF, including the right to see the evidence of your accusers.

Holy Cow

2009-02-10T19:29:00.000-08:00

Wow, the price plans are changing. Even the prepaid plans are getting into the whole "flat rate" thing. I love the idea of a plan where if you don't use the phone, you aren't charged for the plan. The addition of unlimited on-net calling? Perfect.

Verizon prepaid pricing changes coming February 11th

(from Engadget Mobile)

Risk Management, are you getting your money's worth?

2009-02-10T12:27:00.000-08:00

For the first time in the better part of a decade, I'm tech lead on a new project for my current employer. For the intervening years I've been working for other employers, other projects.... I wasn't slacking, honest!

Anyways, it gives me a perfect opportunity to compare and contrast the organization of 10 years ago with today.

Holy Cow! It takes them way to long to start up a project. They spend too long deciding what to do, and not enough time actually doing it. Of course, this is all in the name of "risk mitigation".

I'm finding it both extremely frustrating and hilarious. It makes me want to throw things. The company is going to spend well over 30k to make sure that the 150k they spend on the project is a success. Even before writing the SRS. They are going to spend 30k writing a Product Concept Document, Project Definition Workshops, Project Initiation Gate Meeting, and a Product Scoping Document with estimates. I'm even willing to bet that I am underestimating how much they've spent on this.

The funny thing? All that work will be thrown out as soon as the SRS is written.

The next shock was the testing cost. For every day of development, there is a day of testing, more risk mitigation. Then there's another day added for "overhead". So, that small 100 day project? It's actually 300 days.

So, I looked at it from a math point of view. First the project budget:

Pre-SRS work - 45k
SRS - 45k
Development effort - 100k
Testing effort - 100k
Management overhead - 100k

Total Project cost: 390k
Risk of complete failure: 10%

Even worse, the calendar time and effort are unrelated. The calendar time for steps 1+2, 3, 4 are all the same.

Now, let's have a look at a riskier way of doing it:

Pre-SRS work - 15k
SRS - 7k
Development effort - 75k
Testing effort - 50k
Management overhead - 50k

Total Project cost: 197k
Risk of complete failure: 50%

We've gotten rid of all of the risk mitigation. Not only that, we've shrunk the time in steps 1+2 by 75%! That's a huge time to market win.

Let's see if the risk reward makes sense.

The cost of a failure is 50% (odds of failure) * the cost of the project:

  Risky Way: 197k * 0.5 = 95k
  Safe Way: 390k * 0.1 = 39k

Therefore, the amortized cost of a project using the:

  Risky Way: 197k + 95k = 292k
  Safe Way:  390k + 39k = 429k

What failure rate would be needed to justify the extra cost? 70%? 80%? 90%? To justify the extra money spent (assuming 0% the safe way), the failure rate would have to be:

390k - 197k
----------- = 97%
   197k

Failure is the cost of total failure, as in the project has to be thrown away and started over.

Add in the time to market benefits (on the order of 30% for these assumptions), and it starts to look pretty convincing.

Everyone wonders why so many businesses are CMM level 0/1. Have you considered that they might actually be right?

Prepaid is Dead, redux.

2009-01-15T14:10:00.000-08:00

I just saw this in the news today:

Boost sees $50 unlimited plan battling Leap, Metro

Unlimited calling and texting for US$50/month. It's a race to the bottom with all-you-can-eat plans.

If you're charging per minute, per sms, per byte, the question is "why?". Save yourself a lot of money and quit billing for the core service! That Intec Billing Engine you're thinking of buying? Get rid of it, or use it for something other than charging for calls.

Adding value?

2008-11-25T14:50:00.000-08:00

Ask yourself, "Can I be replaced with a shell script?" Specifically:

 % yes 'No'

Then ask yourself, "What value am I adding?"

Google Chrome, a Contrarian View.

2008-09-09T14:05:00.000-07:00

I read the comic about Google Chrome. I love the architecture! We downloaded it here in the office, and sure enough - it forks a new process for every page load. This is the browser architecture that I've been waiting years for, one where a script on one page won't kill performance in another window.

So, I went home and tried it on my single core Athlon at home. Ouch! The performance is terrible. The browser keeps locking up, it stops responding to mouse clicks, keypresses, anything and everything. It acts like alpha quality software, not the "beta" it is supposed to be.

I've got a 2 year old system (Athlon64 3800+), the only difference between it and new ones is that it is single core. It's still fast enough to play new games on, but it can't handle Google Chrome.

I can only assume that Google Chrome has a tendency to starve some of their child processes on single core machines.

In the meantime, I'll stick with Firefox 3. It leaves Google Chrome in the dust.

Duck Typing Sucks.

2008-08-06T14:37:00.000-07:00

I'm working with some code that's written in Python, and makes extensive use of duck typing. In duck typing the premise is that if it quacks like a duck, it is a duck.

Of course, if you've got code that doesn't tell you what it expects to receive, you can't hope to figure out what the type should be.

For example, let's say you've got 3 code blocks:

class A:
    def funkyDuck():
        # random cool stuff

class B:
    def funkyDuck():
        # entirely different cool stuff

class C:
    def funkyDuck():
        # a third entirely cool thing

Now, imagine you're looking at code that goes:

some_random_object.funkyDuck()

Which one gets called? By the way, they're all in different files, so the only way to find them is with "grep". Added bonus, the undocumented hash!

    def addApplications(self, appDict):
        """ Add some application modules to the defaults.
            All binaries and libraries in any of the source dirs will be soft linked
            to a directory with the name of the product.
            @type  appDict: dict of string -> (list, string)
            @param appDict: The source dirs and OS user name for each product
        """

Guess I'm supposed to guess what appDict actually looks like.

Hrm. Perhaps my problem isn't with duck typing itself, but it's use here.

Hint: When your test framework is just as complicated as the program under test, you're doing something very, very wrong.

One in a Million

2008-07-22T03:57:00.000-07:00

I remember discussing multi-threaded programming with a senior engineer the better part of a decade ago. I was reacting rather strongly to his cavalier attitude towards multi-threaded code, deadlocks and race conditions.

"That is such a small hole, it's one in a million that it will happen."

At the time, we were a small company, and I was a young engineer. One in a million, that doesn't sound that big. Of course, one in a million happens more frequently than you would expect. It didn't take very long for our systems to be processing a million transactions a day, and then millions per hour. Every time the holes had to get smaller and smaller.

They still have a fault in there where the entire system crashes when the system goes completely idle for a couple of minutes. It doesn't happen all the time, it's a "one in about 20 million", but it only has to happen once. So, it crashed about once a week. That one took forever to track down. Still haven't been able to fix it entirely, but they know that it's there, and the hole is smaller, probably "1 in a couple billion" now, below the rate that it's restarted for maintenance. Fixed, for now.

It's interesting see that the DNS engineers are learning the same lesson. Compute power has now shifted in favour of the attackers. As we saw with Captcha, it doesn't matter that you only get through rarely - you only have to get through once in a while. Even just guessing a DNS TXID (2^16), you've got a 1 in a 65536 chance of guessing the right answer.

In pure mathematical terms, the likelihood of getting the TXID wrong 400,000 times in a row is 0.2%. In other words, pretty certain.

(1-(1/2^16)))^400000)

Let's look at the 50/50 point, where the likelihood of seeing X number of failures is 0.5.

That turns out to be somewhere around 45000 attempts, simple for loop range. Since I am spoofing packets, I don't care about a response, even better, filling up the pipe will expand the race situation, giving me a larger chance to get through.

Still, it sounds like the attacker has to also be requesting lookups to poison the server. Let's say they take 2ms each, and they're being nice by only doing them one at a time. Even using the large 400k count, it will take at most 15 minutes of concerted effort to poison the cache. The 50/50 point? One minute, 20 seconds.

Look out for those one in a million situations, they happen more often than you'd like.

Naming Documents

2008-07-21T17:48:00.000-07:00

I thought we killed off Hungarian Notation because it doesn't work? Did I miss a memo somewhere? It seems that Hungarian Notation is making a comeback, this time in document names.

There's nothing like having your meta data encoded in the file name, with three letter acronyms for everything. It is impossible to decode, and becomes really fun when the team names change once a year!

"Say where's the SRS?" "Is it in SRS.doc?" "Nope" "How about esgEng_SIT_DR4_SRS.doc?" "Nope" "Oh, wait, Engineering was renamed last year for 6 months.... How about esg_PEN_SIT_DR3_SRS.doc?" "Ah, that's got it."

Encoding meta data in the filename is stupid. How about putting it in the file in the meta data portion where it belongs, and then using a search tool to search it? Or, maybe a directory structure that represents your tag tree.

esgEng/SIT/DR4/SRS.doc

Then, when you need to change the meta data, you rename the tree:

esg/PEN/SIT/DR3/SRS.doc

But Jason, I need to know who produced the document without opening it!

Use the checksum, and search/store that. Anything else can be mistakenly altered or lost. Use the file's checksum. It is a much more reliable descriptor of the file than the filename! I have learned from experience to, never, ever trust a filename, they lie.

Or better yet, you buy a document repository off the shelf for 100k, and shove the problem at them. Personally, I just use the Google Search Appliance. Now that they've got it looking at the correct bits of data, it's very useful. Much better than any of the searches built into the various corporate portals.

Getting Starcraft working on OSX

2008-07-11T15:35:00.000-07:00

I've started playing Starcraft again after ages away, but I found that my brand spanking new MacBook Pro wouldn't work!

The new versions of OSX have an updated video driver which does not support 256 colour modes. This means that Starcraft, Diablo II, or basically any older game will no longer work. Not good when the applications were still sold by Apple as OSX compatible games (they've since been removed).

I have now figured out a simple (albeit more expensive - US$79.99 + Windows license) way to solve the problem. I decided to run the game under VMWare Fusion.

However there were two problems with this. First, when running in full screen mode, the display doesn't stretch to fill the screen. So on my laptop, I end up with a postage stamp display right in the middle of the screen. Second, the mouse is not restricted to that little postage stamp, it moves freely all the way around the display. When attempting to scroll the display in the game, you move the mouse to the edge, since the mouse will leave the bounds, the game doesn't scroll very well.

The first was fixed with a quick google.

In the file "Preferences/VMWare Fusion/preferences" add

    pref.autoFitFullScreen = "fitHostToGuest"

This will stretch the display. If you have a widescreen display, you will still have black bars on the left and the right, but we're making progress.

Now to mouse capture. There doesn't appear to be an option to prevent the mouse from being taken back by the host OS, so I took a more drastic approach. I uninstalled VMWare Tools. This requires a reboot of the VM, but when you are done, you will no longer be able to move the mouse outside of the VM!

To get back to the host OS, press CTRL-CMD. To re-install VMWare Toole, select "Install VMWare Tools" under "Virtual Machine"

You should now be able to play Diablo II and Starcraft in all their 256 colour beauty.

Now, if I can only control my nerves enough to work my trackball....

Keeping Efficiency Gains

2008-07-10T14:39:00.000-07:00

Air Canada has implemented self check-in at Pearson Airport in Toronto. For all flights, you no longer go to an attendant to obtain you boarding pass, you enter all the required data into a computer, and then check your bags.

This was supposedly done for efficiency reasons. However, most of the efficiencies in the system have been lost. Amazingly, when you get to the front of the second line (to check in your bags), the check in clerk does exactly the same amount of work as before. The first thing they do is verify all of the details that you provided the computer!

Let's go over that again. After having provided my information to a computer, I have to provide it a second time to a human who verifies the first data. The time taken for verification was exactly the same as it would have taken for them to key it in in the first time. Instantly, you have a net loss of efficiency. Even worse, they only have a single set of scales for each pair of desks. That means that there is substantial dead time while you wait for the person at the desk next to you to finish weighing their bags. More lost efficiency.

I can see how it happened. Someone checked in to the wrong flight, or their bags went to the wrong place. Perhaps they got to US customs (you go through US customs in Canada when flying to the US), and didn't have the correct forms or all their data provided, and were sent back, maybe they even missed a flight.

So, the check-in clerks, who are also looking to protect their jobs, add in the task of checking customer provided data.

However, they don't check the data of people who don't have checked baggage, they go straight through to customs. This shows the stupidity of the additional check. Is Air Canada saying that people with checked luggage are more likely to enter incorrect data? I doubt it.

If you make efficiency gains in your organisation, make sure you protect them. Guard them jealously. If you don't, you will see them frittered away.

The Truth about Vacations.

2008-06-12T10:29:00.000-07:00

This invariably happens to me. I go on vacation, and then the server goes down. It doesn't go down during the year. It's had an uptime of over a year! However, it's down and stayed down. So, if you've been trying to email me, I apologize, it won't get through. The same goes if you're wondering where blog.jason.pollock.ca went to, or why I'm not on jabber anymore. I'm trying to get it going again, but email will probably be down until I get back home. The eBook Repositories will definitely not be updating, so the Baen Books will slowly rot.

Installer.app and chmod/chown

2008-06-07T17:09:00.000-07:00

How is it that I would go about chmodding the directories created by CopyPath?

It seems that if the .zip contains the permissions already, they are maintained when used by CopyPath.

In other words, if you have:

foo/bar

in your .zip, and do a CopyPath("foo", "~/foo") in the XML document, the permissions are set up correctly. At least I haven't had any problems if my installer looks like that.

However, if you create implicitly create directories by using CopyPath on the individual files, they are created without permissions, and a chmod/chown is needed. For example, CopyPath("foo/bar", "nuts/baz/bar") will create "nuts/baz", with no permissions.

I have to do this with the Gutenberg SciFi and Baen Books libraries, since I don't host the .zips themselves.

To manage the chmod/chown problem, the packages ensure that the BSD subsystem is installed (or Cydia), and then runs them as part of the install steps.

Here is an example:

      <key>preflight</key>
      <array>
        <array>
          <string>IfNot</string>
          <array>
              <array>
                <string>InstalledPackage</string>
                <string>com.natetrue.iphone.iphone_binkit</string>
              </array>
          </array>
          <array>
              <array>
                <string>AbortOperation</string>
                <string>You must install the "BSD Subsystem" package from Installer first!  It's in the System Category.</string>
              </array>
          </array>
        </array>
      </array>

Which checks for the BSD subsystem as part of the pre-install. If the BSD subsystem is not installed, the user gets a pop-up error message and the package will not install.

And then the install

      <key>install</key>
      <array>
        <array>
          <string>CopyPath</string>
          <string>1013.txt</string>
          <string>~/Media/EBooks/The_First_Men_in_the_Moon/1013.txt</string>
        </array>
        <array>
          <string>If</string>
          <array>
              <array>
                <string>FirmwareVersionIs</string>
                <array>
                  <string>1.1.3</string>
                  <string>1.1.4</string>
                </array>
              </array>
          </array>
          <array>
              <array>
                <string>SetStatus</string>
                <string>Changing Permissions</string>
              </array>                              
              <array>
                <string>Exec</string>
                <string>/bin/chmod -R 755 /var/mobile/Media/EBooks</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/chown -R mobile /var/mobile/Media/EBooks/.</string>
              </array>
              <array>                 
                <string>SetStatus</string>
                <string>Running GutenMark</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/GutenMark --profile=en --no-toc --config=/etc/GutenMark.cfg /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013.txt /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013-h.htm</string>
              </array>
              <array>
                <string>SetStatus</string>
                <string>Running GutenSplit</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/GutenSplit --no-toc /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013-h.htm /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/Chapter_</string>
              </array>
          </array>
        </array>

That's not the full install segment, but it should give you an idea. I have to separate 1.1.3/4 from earlier firmwares because they use different users and store files differently.

To help, I've created a little perl library that allows me to write packages without having to mess with the nasty XML. It's part of the iphoneebookrepo project over at google code. Specifically PlistMaker.pm.

In terms of the perl library, it ends up looking like this:

my $gutenberg_html_install =
 p_if(IsFirmwareVersion("1.1.3", "1.1.4"),
      SetStatus("Changing Permissions").
      Exec("/bin/chmod -R 755 /var/mobile/Media/EBooks").
      Exec("/usr/bin/chown -R mobile /var/mobile/Media/EBooks/.").
      SetStatus("Running GutenSplit").
      Exec("/usr/bin/GutenSplit -1 -2 -3 -4 --no-toc --no-skip /var/mobile/Media/EBooks/$BookDir/${BookNumber}-h.htm /var/mobile/Media/EBooks/$BookDir/Chapter_")
 ).
 p_if(IsFirmwareVersion("1.0.0", "1.0.1", "1.0.2", "1.1.1", "1.1.2" ),
      SetStatus("Changing Permissions").
      Exec("/bin/chmod -R 755 /var/root/Media/EBooks") .
      SetStatus("Running GutenSplit").
      Exec("/usr/bin/GutenSplit -1 -2 -3 -4 --no-toc --no-skip /var/root/Media/EBooks/$BookDir/${BookNumber}-h.htm /var/root/Media/EBooks/$BookDir/Chapter_")
 );

my %BookDescriptor = (
 bundleIdentifier => "ca.pollock.gutenberg.$BookNumber",
 name => $BookTitle,
 version => "1.0",
 location => $BookURL,
 category => "Gutenberg SciFi (" .$BookLang . ")",
 size => $BookSize,
 hash => $BookMD5,
 url => "http://jason-pollock.blogspot.com/",
 maintainer => "Jason Pollock",
 contact => 'jason@pollock.ca',
 description => "by $BookAuthor",
 install => $Book_install,
 preflight => $Book_preflight,
 postflight => "" ,
 uninstall => $Book_uninstall,
 update => ""
 );

my $BookPLIST = CreatePackage(%BookDescriptor);

One "Gotcha" to be aware of... The Exec command is actually performing it's own parameter splitting prior to calling "Exec". It does not do any sort of "~" expansion, nor does it understand spaces. So, if you are doing a chmod on a directory with spaces in the name, it won't work, regardless of escapes or quotes.