Stopwords

iChat, Growl, and Lion

2011-09-09T15:08:00.000-07:00

UPDATE: This script has been improved; the new version is now on github.
UPDATE 2: Growl 1.3 was just released, so I've updated the script to work with 1.3. If you were using this script and it broke with 1.3, then just download the updated version from the same place. If you're still using Growl 1.2.2, you'll need to use the old version of this script. All the rest of the instructions are the same regardless of which version of Growl you're using.

Lion is the latest version of OS X. iChat is the chat program that's part of OS X. Growl is a system that lets all sorts of programs post little notification windows on your screen.

For example, if you get a message in iChat while you're in another window, it's nice to have the message pop up on your screen so you can decide whether or not to switch windows to read it. Or if a buddy logs in, it's nice to know about that (which can be done in iChat with an audio notification), and see who it is without having to check your buddy list.

I used to use Chax for this. Chax is a pretty good program, but when a new version of OS X comes out, it takes a little while to catch up. The main reasons I use Chax are:

Accept text chat invitations without being prompted.
Put all my buddies in the same window.
Give me Growl notifications.

Everything but the Growl feature is now built into iChat in Lion. (The auto-accept is an AppleScript that ships with Lion; you can turn that on in Preferences:Alerts. The option for "Show all my accounts in one list" is in Preferences:General.)

Chax isn't yet compatible with Lion, so I searched online and found a little AppleScript program to do a bit of what I need. Using the ideas from that, I expanded it a bit. Goodbye Chax; I'll miss you!

To use this:

Install Growl, if you don't have it already.
Visit https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript in your web browser. (You can use the formatted page instead.)
Open AppleScript Editor (in Applications:Utilities).
Copy and paste the code from the web page into a new script.
Save it to your home directory's Library:Scripts:iChat folder. (I used the filename "Growl".) The iChat folder might not exist; you need to create it. By default, your Library folder is hidden. (Personally, I solved that by saving to the desktop, and then I ran "open ~/Library/Scripts" in Terminal so that the Scripts window opened in Finder, and dragged my script there.)
In iChat, open the Preferences.
Under Alerts, select the "Message Received" event.
Turn on "Run an AppleScript script:", and select the script you saved earlier.
Repeat the last two steps for each event you're interested in.

Note that the "Message Received in Chat Room" (which is called "addressed message received" in the AppleScript) and "Invitation to Share Buddy's Screen" (a.k.a. "received remote screen sharing invitation") aren't currently enabled, since it looks like there may be a bug in iChat that will cause them to be confused with each other.

If you use the Growl script for chat invitations, you can't also use the Auto-Accept script, since you can only select one script to be run. Because of this, I've added an auto-accept capability to this Growl script. By default, it's turned off, but you can turn it on. Near the top of the program, there's a line that says:

property autoAccept : {""}

Change this to:

property autoAccept : {"text"}

Suggestions for improvements are welcome. Share and enjoy!

(If you expected the source code to be here in my blog, I've moved it. See https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript for my latest version.)

Copying a filesystem on Linux

2011-07-05T02:52:00.000-07:00

In the last two days, I've had to copy a filesystem on my Linux development VM four times (to grow it). I eventually made a shell script for it, since I kept forgetting details (like the arguments to cpio). Since I started this blog to have a place to publish random bits of information for future Google users, I'm putting the script here. The script is below; you pass in the old device name (e.g., sda) as the first argument, and the new one (e.g., sdc) as the second.

There's some significant assumptions in this script, and they may not be warranted for your system:

It assumes that you have only one filesystem, and that's on the first partition of the disk. (It's also easy to add a swap partition using mkswap.)
You're expected to use fdisk to set up the new filesystem yourself. (Linux is fs type 83; Linux swap is 82.)
This builds an ext3 filesystem.
It assumes that once you're finished, you'll be putting the new device into the old device's place; in other words, it doesn't try to replace sdc with sda in /etc files. (If you're only using UUIDs in your fstab, this probably doesn't matter.)
It uses a default grub install, more or less. If you're using LILO, or have got something funky going on, it might not carry over. If you've customized your grub.cfg, for example, it will get overwritten. (However, changes in grub.d are fine and will be reflected.)
As used here, cpio will make sparse files out of non-sparse files. Normally, this is a good thing, but some programs (such as BitTorrent clients) will intentionally build non-sparse files. (It's ok to turn them sparse after the torrent completes, and it's only a fragmentation hit if they get turned into sparse before the torrent is done.)
This uses a 64k I/O block size. I'm doing that because that's the block size of the default QEMU backing store under qcow2. (Remember, in my case, I'm doing this for a VM.) I'm not sure if that helps anything or not (since the I/O goes through many layers between cpio and qcow), but it's not going to hurt anything. However, you may want to replace the -C65536 with -B if you're using a normal disk.

This script is better to learn from than to use directly. But if you need to copy a filesystem (to grow it, move off a failing disk, defragment it (rarely necessary on Linux), or whatever), this may give you a good starting point.

#! /bin/sh

in=$1
out=$2
set -xe

fdisk /dev/${out}
mke2fs -j /dev/${out}1
mount /dev/${out}1 /mnt
find / -xdev -depth -print0 | cpio -pdmV0a --sparse -C65536 /mnt
grub-install --root-directory=/mnt /dev/${out}
perl -pi.bak -e s/$(blkid -o value -s UUID /dev/${in}1)/(blkid -o value -s UUID /dev/${out}1)/g /mnt/etc/fstab /mnt/boot/grub/grub.cfg
umount /mnt

A few implementation comments:

This makes a lot of assumptions (listed above) for simplicity. Feel free to improve it.
The -depth is so that find will print the directories after their contents, which will cause cpio to fix the modification time. Otherwise, cpio will create the directory, change its mtime to match the original, and then add contents, thereby changing the directory's mtime to the current time.
The Perl can be done by hand: it's changing the old filesystem's UUID to the new filesystem's UUID in fstab and grub.cfg. You can find out the IDs by running sudo blkid.
Note that the Perl line is long; be careful when copying it. It spans from perl to grub.cfg.

Share and enjoy!

PS: If you're wanting to grow a filesystem on the same disk (i.e., you've deleted the next partition and want to use its free space), there's utilities to do that. This script is about moving to a different disk.

(Disclaimer: I release the above code to the public domain, as if the public actually wants it. It's for education, and should be evaluated and customized before running it in any particular environment. I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)

iPhone location log to Google Earth, all with shell commands

2011-04-26T00:46:00.000-07:00

If you're a privacy geek, then you've no doubt heard people talking about a file on the iPhone called consolidated.db. This file stores all the places your phone has been.

I don't know why everybody is making such a fuss about it. I expect my own computers (including my phone) to store logs of what I'm doing. It's good to be able to reset this information before I give a device to somebody else, and hey, the iPhone has a comprehensive reset function!

I think the concern is that the phone is sending this to Apple periodically. That would be a very serious concern. But until somebody gives any sort of evidence to that effect, I think it's just a badly-managed log file.

Anyhow.

Yesterday, I saw that I had apparently bought a song on my iPhone that I didn't recognize. I listened to it, and it didn't sound familiar. It also didn't sound like something I'd buy. After a couple of blind alleys, I checked the purchase date in iTunes. I then correlated that to Google Latitude's history. Ah, it was a mall in Santa Cruz. Now I remembered: the reason I went into the mall was to get a lemonade at Starbucks, and while I was there my friend gave me one of the vouchers that Starbucks has for a free iTunes song.

While I was relaying this story to another friend, he asked if I'd used consolidated.db to read it. Nope, but not a bad idea. I started wondering what was actually in there. Most stuff I saw to read that requires me to download some stranger's code, and I wasn't too thrilled about that.

OS X and iOS tend to use property lists and SQLite for all kinds of storage, so I'd guessed it was SQLite. That meant that it could be read from the command line using the sqlite3 utility, which is part of OS X.

So, without further ado, here's a quick rundown on how to look at all this stuff straight from the command line. There's one bit of Perl to convert it to Google Earth's KML format, but the Perl really isn't necessary to view the data in lat/long format. Note that making use of the following requires a certain degree of technical savvy; knowing SQL is going to be very useful to fiddle with the data. However, you don't need to know anything about OS X (except how to start the terminal), iPhone development, etc.

   # Get the current time in the format used by the database (Apple's CFAbsoluteTime, which is the number of seconds since Jan 1, 2001 at midnight GMT; we use UTC instead of GMT here, but it's close enough for these purposes):
$ perl -MTime::Local -e 'print time() + timegm(0,0,0,1,0,70) - timegm(0,0,0,1,0,101), "\n"'
325464777
   # Go to the backups
$ cd ~/Library/Application\ Support/MobileSync/Backup/
   # Go to the most recent backup
$ cd $( ls -t | head -1 )
   # The files in here are named (almost) randomly.
   # Get the database's filename by looking at table lists of all of the SQLite databases and finding one that has CellLocation
   # (This prints the filename and leaves it in $db)
$ for db in $( file * | sed -e '/SQLite/ !d' -e 's/:.*//' ) ; do sqlite3 $db .tables | grep -q CellLocation && echo $db && break ; done
4096c9ec676f2847dc283405900e284a7c815836
   # Your filename will be different; it's based on a hash of the data.
   # View the file
$ sqlite3 $db
   -- Take a look at the schema of CellLocation
sqlite> schema CellLocation
CREATE TABLE CellLocation (MCC INTEGER, MNC INTEGER, LAC INTEGER, CI INTEGER, Timestamp FLOAT, Latitude FLOAT, Longitude FLOAT, HorizontalAccuracy FLOAT, Altitude FLOAT, VerticalAccuracy FLOAT, Speed FLOAT, Course FLOAT, Confidence INTEGER, PRIMARY KEY (MCC, MNC, LAC, CI));
   -- (That will also print a few other tables that you can ignore.)
   -- View the contents of the past week (this uses the time computed at the beginning of this walkthrough; 604800 is the number of seconds in a typical week)
sqlite> select latitude,longitude,timestamp from CellLocation where timestamp > 325464777 - 604800 order by timestamp;
   -- Back to the shell
sqlite> .exit
   # Put the data through Perl to convert to KML
$ sqlite3 $db "select longitude,latitude,timestamp from CellLocation where timestamp > 325464777 - 1209600 order by timestamp, horizontalAccuracy" | perl -nw -MPOSIX -MTime::Local -e 'BEGIN { our $LTS=0; print "\n";} next if /^[0.]+\|[0.]+\|/; my ($long, $lat, $tstamp) = split /\|/; next if $tstamp==$LTS; $LTS=$tstamp; my @time = gmtime($tstamp - timegm(0,0,0,1,0,70) + timegm(0,0,0,1,0,101)); my $tstr = strftime("%Y-%m-%dT%H:%M:%SZ", @time); print "$tstr$long $lat 0\n"; END { print ""; } ' > ~/Desktop/locations.kml
   # Open it in Google Earth
$ open ~/Desktop/locations.kml

If you're trying to make sense of the Perl (particularly the $LTS bit), note that each timestamp tends to have several locations with varying precisions. I'm not sure whether that's a triangulation of the user's position, or of the cells the user was able to communicating with, but I suspect it's increasingly precise triangulations of the position that are recorded with the same timestamp. This code will use the first recorded position for each timestamp, which (because of the ORDER BY clause) is going to be the one with the smallest horizontalAccuracy. I'm currently supposing that the horizontalAccuracy column is actually the precision radius.

I also played with using the WifiLocation table (which is similar), but it seems to be pretty unreliable data; it seems to include data based on one database that's about 10 miles too far north. (At least, in my case, it kept showing me jumping abruptly between Santa Clara / Sunnyvale and Fremont. I haven't been on the Fremont side of the bay in over a month.) I read online that Verizon customers need to use CdmaCellLocation instead of CellLocation.

If you're a potential employer, please note that the Perl above was thrown together for my own experimenting. I just put it here on the blog in case somebody else wanted to fiddle with it, and it's not at all what I'd put into production.

Well, that's about all the time I felt like putting into decoding that file. Feel free to post improvements in the comments section below. Share and enjoy!

(Disclaimer: I release the above code to the public domain, if you care. It's for education, and is not intended to run in any particular environment. I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)

Magic GarageBand Chords

2011-03-16T19:53:00.000-07:00

Magic GarageBand Chords

I sat down to practice my guitar by improvising along with GarageBand's "Magic GarageBand" feature. This handy feature lets you put together a five-piece band from a wide selection of instruments and styles, which will play a song in a genre of your choosing. You can accompany them (using a mic, MIDI instrument, or DI from a guitar or other instrument), and record the result. By choosing the instruments and styles, you can get very different-sounding songs.

This is great if you want to jam but don't have a band with you at the moment. But, there's one bit of information that's missing.

In order to accomplish this mix-and-match, each genre (of the nine possibilities) plays the same basic chord changes. The individual parts riff all over those, but since they're following the same basic pattern, you get a good harmony. This is handy information to know if you're trying to play along! All of these have pretty straightforward patterns, but it's a lot easier to know the changes ahead of time.

Regrettably, I wasn't able to keep up with the chord changes when I sat down to play the other day. I went online to find them, but couldn't find anybody who had written them down for the world to see.All Google brought me is other people looking for the changes.

Hence, today's post: the chord changes for all nine genres in Magic GarageBand. If you have any suggestions for improvements, let me know! (I mean, something that makes this list closer to what GarageBand actually plays. I'm not looking for improvements to the songs; talk to Apple about that!)

Blues (E, 90bpm, 4/4)

When you create the GarageBand project, it says the song is in B,but it's really in E. The loops are also in the wrong key, which can lead to surprises if you try to use them in your own project!

Other than that, there's really no surprises here. This is a moderate tempo 12-bar blues, which is probably what you were expecting here. There's not even a real intro or outro.

Intro	1	E				Pickup; can use A as well.
Verse 1	2	E	E	E	E
	6	A	A	E	E
	10	B	A	E	E
Verse 2	14	E	E	E	E
	18	A	A	E	E
	22	B	A	E	E
Verse 3	26	E	E	E	E
	30	A	A	E	E
	34	B	A	E	E

Rock (D, 140bpm, 4/4)

Intro	1	D				Pickup
	2	D	D	D	D
Verse 1	6	D	D	D	D
	10	F	F	D	D
	14	D	D	D	D
	18	F	F	A	A
Chorus	22	Bm	Bm	E	E	Remodulating in the chorus. Can use G♯° instead of E.
	26	G	G	A	A
Verse 2	30	D	D	D	D
	34	F	F	D	D
	38	D	D	D	D
	42	F	F	D	D
Ending	46	D	D	D	D
	50	F	F	D	D
	53	D	D	D	D	Sustained fade

Jazz (F, 164bpm, 4/4)

This is similar to the 12-bar blues progression, but with a more jazzy twist to the chords actually being used.

The basic idea is still the 12 bar pattern F / F / F / F / B♭ / B♭/ F / F / B♭ / C / F / F. However, there's some changes that are often used:

The B♭ (IV) is often played as D° (vi°)
The F is often played as an F7, particularly in long stretches.
Bars 7–8 and 11–12 (of the 12-bar pattern) are usually played to walk into the next phrase (changing chords every two beats) instead of being played straight. In particular, bar 8 is very rarely played as an F, but if you're improvising over an F you'll still be fine.

First, here's the basic pattern:

1	F	F	F	F
5	B♭	B♭	F	F
9	B♭	C	F	F

Now, here's the pattern as I've interpreted it. As I said, there's a lot of possibilities here.

Chorus 3
Intro	1	-				Pickup
Head	2	F7	F7	F7	F7	May substitute F for F7.
	6	D°	D°	F7	C; D	Not sure about this phrase. May substitute B♭ or even Fm6 for D°.
	10	Gm	C	F7; F	B♭; C
Chorus 1	14	F7	D°	F	F
	18	D°	D°	F7	C; D	Some of the accompaniments can get a bit muddied in the chords here.
	22	B♭	Gm; C	F; F7	B♭; C
Chorus 2	Too tired to transcribe these; it's jazz, so just get the idea from the pattern I've done so far.
Head Out

Country (G, 100bpm, 4/4)

Another 12-bar blues progression, played pretty much straight this time. (In case you haven't gotten the picture, 12-bar blues is great for improvising over, so it's used a lot in Magic GarageBand.)

Intro	1	-				Pickup
Verse 1	2	G	G	G	G
	6	C	C	G	G
	10	D	C	G	D	If you want to keep up with the lead-in for the last bar, it's D / C / Bm / Am
Verse 2	14	G	G	G	G
	18	C	C	G	G
	22	D	C	G	G
Verse 3	26	G	G	G	G
	30	C	C	G	G
	34	D	C	G	D	Same lead-in as the end of verse 1
Ending	38	G	G	G		Sustained fade

Reggae (C min, 75bpm, 4/4)

I don't really know reggae, so this may be totally off. I've marked it as C minor, even though when you create the project GarageBand lists it as C major.

Intro	1	-				Pickup
Verse 1	2	Cm	Cm	A♭; Gm7	Cm
	6	Cm	Cm	A♭; Gm7	Cm
Chorus	10	B♭m; A♭	A♭; E♭	B♭m; A♭	A♭; E♭
	14	B♭m; A♭	A♭; E♭	B♭m; A♭	A♭; E♭
Verse 2	18	Cm	Cm	A♭; Gm7	Cm
	22	Cm	Cm	A♭; Gm7	Cm
	26	-				Sustain from last note

Funk (C, 120bpm, 4/4)

Also really not sure how accurate this one is.

Intro	1	Cm	Cm	Cm	E♭
Verse 1	5	C	C	C	C	If it's too monotonous, you can make the first and third bars F; C (two beats each)
	9	C	C	C	C
	13	F	F	C	C
	17	G	G	C	Em7	You can use a G for the Em7
Bridge	21	F	F	F	F
	25	F	F	G	G
	29	G				Fermata from bar 28
Verse 2	30	C	C	C	C
	34	C	C	C	C
Ending	38	G7	G7	C	C
	42	G7	G7	C	C
	46	C	E♭

Latin (Fm, 98bpm, 4/4)

This is pretty simple. There's just one pattern: Fm, E♭, D♭, C,for two beats each.

Intro	1	Fm				Pickup
Verse 1	2	Fm,E♭	D♭,C	Fm,E♭	D♭,C
	6	Fm,E♭	D♭,C	Fm,E♭	D♭,C
Verse 2	10	Fm,E♭	D♭,C	Fm,E♭	D♭,C
	14	Fm,E♭	D♭,C	Fm,E♭	D♭,C
Bridge	18	Fm	Fm	Fm	Fm
	22	Fm	Fm	Fm	Fm
	26	Fm	Fm	Fm	Fm
	30	Fm,E♭	D♭,C
Verse 3	32	Fm,E♭	D♭,C	Fm,E♭	D♭,C
	36	Fm,E♭	D♭,C	Fm,E♭	D♭,C
	40	Fm

Roots Rock (G, 76bpm, 4/4)

Not really sure what distinguishes the chorus from the verse, but it could just be the instruments I picked don't really do much differently. This is a simple pattern, but has a lot of room for improvisation.

Intro	1	G				Pickup
Verse 1	2	G	Am	C	D
	6	G	Am	C	D
Chorus	10	G	Am	C	D
	14	G	Am	C	D
Bridge	18	Em	Bm	Am	G
	22	Em	Bm	Am	D
Verse 2	26	G	Am	C	D
	30	G	Am	C	D
Outro	34	G	G

Slow Blues (A♭, 88bpm, 12/8)

Wow, there IS something other than 4/4 in here! Unsurprisingly, it is still 12-bar blues, although with a IV in the second bar to keep it interesting.

Intro	1	E♭				Pickup
Verse 1	2	A♭	D♭	A♭	A♭
	6	D♭	D♭	A♭	A♭
	10	E♭	D♭	A♭	E♭
Verse 2	14	A♭	D♭	A♭	A♭
	18	D♭	D♭	A♭	A♭
	22	E♭	D♭
Ending	24	A♭	A♭	A♭		The measure breaks are hinky

Imminent Death of the Net Predicted

2011-01-08T19:43:00.000-08:00

I didn't know that it was coming this soon, but in mid-February, the Internet will run out of addresses.

My blog post's title is a bit tongue-in-cheek. People have been talking about the net not being able to handle the growth rate for decades. The address space exhaustion is going to take some work to deal with, but it's not going to cause things to grind to a halt.

There's already some great resources to explain this to people who don't know about the problem:

You can get an app for your smartphone, blog, etc that shows the countdown from Hurricane Electric's IPv6 page. (Hurricane Electric is a very IPv6-friendly ISP. If you're a technical person, they can set you up with a free tunnel.)
ABC wrote a story, "IP address shortage to limit Internet access"
A slightly more technical, but still pretty accessible, FAQ is on Geoff Huston's blog.
Of course, Wikipedia has a page on IPv4 address exhaustion for all the backgroundy stuff that Wikipedia tends to have.

What I couldn't find is an answer to my friend's question, "What will happen when we run out?" Nobody knows for sure, but here's a basic rundown.

Internet addresses are globally administered by IANA. They give blocks of addresses to the regional groups, RIRs. These cover continent-sized areas, like Europe, Asia/Pacific, North America, etc. The RIRs give blocks to the local registries (LIRs), which are usually large ISPs, but may also be large organizations (like Stanford, although that's mostly because they were one of the first places on the Internet).

Once IANA runs out (which is the event in February), then the individual regions can't get more addresses. The first to run out will be APNIC (Asia/Pacific), in about eight months or so (we're guessing; nobody can be sure). Then APNIC won't be able to give out new addresses to its LIRs. When an LIR can't get new addresses, it can't attach new customers. Its customers are mostly small ISPs or companies.

So there's a trickle in the "supply chain" before things get to the end users. You won't see problems until it gets down to this level, although you may see some changes as ISPs hurriedly try to prepare (with IPv6; I'll get to that in a minute).

Once a small ISP runs out, then it can't put new servers on the Internet: that means no new web servers, no new mail servers, nothing. (To a limited degree, it's possible to piggyback these on existing servers in some cases.) The ISP can attach new home customers (who don't use servers) for a little while, but if they attach too many, then people won't be able to connect. (A home ISP tends to hand out addresses for two hours at a time, as long as the customer's router or computer is on.)

At that point, to deal with that, they'll probably start putting large blocks of home customers behind NATs. NATs let several computers use the same IP address. However, you can't have a server behind a NAT, it can be unreliable, and a lot of games, video or voice chat programs, file-sharing programs, etc. don't behave well with NAT. Because APNIC is running out of addresses, much of Russia and China are already behind regional NATs.

There's also a possibility that organizations will start trading small blocks of IP addresses. (Nobody "owns" them; they're allocated by IANA.) However, routers will have a hard time keeping up with that (a router is designed to think in large blocks, and lots of tiny blocks can be a problem); this might make the Internet more flaky.

I don't really know how the situation is going to be for end users. There'll be some hiccups, but it'll probably come down to just a little bit of reconfiguration; your ISP will tell you to do it, and you'll have some notice. If you've got a particularly old router, you may end up having to replace it.

Ultimately, the solution is to go to IPv6. This has a lot more addresses: 3.4x10^28; that's enough for every gram of matter on Earth to have 10 addresses. (The current mechanism, IPv4, has about 4 billion addresses.) However, IPv6 adoption has been slow, because customers aren't asking for it. Most devices support IPv6, but some don't. Not many websites are currently equipped for IPv6; you have to have IPv4.

I migrated my home network to IPv6 yesterday, and it was a breeze. Right now it's just to get a dancing turtle on Kame's website, but I recommend that my techie friends with a home network do the migration. Even though your ISP probably doesn't provide IPv6 connectivity, there's still peers all over the place that'll give you tunnels.

For the technically-minded among us,I actually did the migration twice: once using 6to4 for connectivity with the Anycast2002:c058:6301:: peer for 6to4. Then, I did it again using a tunnel (using FreeBSD's gif interface, which also works on OS X) that I got for free from Hurricane Electric's website. The tunnel is faster (23ms ping time vs the 6to4 peer's 96ms), and their website walks you through the process; it's got easy instructions for Windows, OS X, Linux, BSD, or whatever else you've got.

Dealing With Dashcode, Part 4: Dashcode and Debugging

2010-08-17T20:06:00.000-07:00

UPDATE: As Todd points out in a comment, the bug described here has been fixed in 3.2.4. That's now out of beta, and registered developers can download it from Apple.

(This is a continuation of a previous post. You may want to read the first post in this series for an overview of the architecture.)

The last problem I'm blogging about is actually the first one that I encountered. It's also what first inspired me to write this "Dealing With Dashcode" series of posts.

Early on in my development process, I made sure that I had the latest XCode installed (3.2.3), which is the iOS 4 development environment; my app is targeting iOS 4 on iPhone 3GS and 4. The iOS SDK includes the iPhone Simulator, with environments for iOS 3.2 on the iPad, and iOS 4 on the iPhone 3GS and iPhone 4.

Dashcode comes with a quite handy JavaScript debugger. Since Mobile Safari's built-in developer support is fairly scant (you can globally turn on and off console logging), having a JavaScript debugger is really handy. The JavaScript debugger works against Mobile Safari in the simulator: when you're running your program in the iPhone Simulator, you can use breakpoints, inspection, single-stepping, etc. from within Dashcode's debugger. Nice, huh? Too bad it doesn't work.

Here's what I've been able to work out. Dashcode implements this debugger by loading an extra framework into Mobile Safari when it launches. When you click the "Run" button, Dashcode launches Mobile Safari with some extra arguments to load a framework called CayenneClientDebugger (which is within the Dashcode executable). It looks like Mobile Safari (as well as the Browser Simulator that's part of Dashcode's Safari targeting) has hooks to load this framework for some debugging usage. The debugger UI that's built into Dashcode communicates with the debugger backend loaded in Mobile Safari.

(By the way: I looked online to see if I could find any references to this Cayenne. There's an Apache project named Cayenne, but it seems unrelated. I suspect that Cayenne was the internal codename for Dashcode; it looks like the bulk of Dashcode is implemented in a framework called CYKit.)

Now, here's the catch. iOS 4 uses a different ABI than 3.x did. It looks like the CayenneClientDebugger is built for the old version. Mobile Safari refuses to load it, so instead of a debugger, all you get is a message in system.log.

I tried to cherry-pick a few bits of XCode from a backup I had from before I upgraded it. I tried copying iPhoneSimulator3.0.sdk into the current installation, and although the iPhone Simulator would display the 3.0 version in its Version menu, if I tried to actually select it, then the Simulator would crash. I tried a few other combinations of replacing the Simulator, CayenneClientDebugger, and even all of Dashcode, but couldn't get anything really going. I probably could back out the entire XCode installation, but that's a bit far.

As it was, I never did solve this problem. I just made a lot of use of console.log for debugging purposes. If I needed something more than I could get that way, then I'd use Safari (the OS X one); I could load my webpage there and use Safari's built-in development tools. The most signifiant bit of UI I was using for this app (StackLayout; it's essentially a nested list that you navigate one level at a time) doesn't actually work in Safari — only Mobile Safari — but I could still test little bits in some ways. It was enough for me to finish coding.

I haven't looked into the 4.1 SDK yet (as of this writing it's in beta and has limited availability); I hope that Apple's fixed this problem. Not having access to a debugger makes Mobile Safari development a bit tricky.

By the way: there are a few options out there. The best of these seems to be "Firebug for iPhone". Don't get your hopes up, though: it's not Firebug (the must-have development tool for Firefox). It's not even close. Firebug for iPhone mostly just relays console messages to your server, and lets you send commands to the browser. While this is very useful, it's still not a substitute for a full JavaScript debugger. (I considered writing something similar, using AJAX to relay messages between my code and a program running on my server.)

IBM has an article on using Aptana (a webapp development system for Eclipse) while developing iPhone web apps. It's entitled "Debug iPhone Web applications with Eclipse", but in truth, Aptana doesn't support the iPhone for debugging. Instead, the article recommends you use Firefox with Aptana to debug your application. Uh... no thanks. WebKit and Gecko are different enough that, if I'm developing an iPhone-specific application, I'll want to do my development with Mobile Safari.

Neither of these was really worth the effort of learning; the app in question is simple enough that console.log does everything I needed. But I do hope that Apple has fixed this problem in the newest Dashcode.

Dealing With Dashcode, Part 3: Dashcode, WebDAV, and Trailing Slashes

2010-07-25T16:29:00.000-07:00

(This is a continuation of a previous post. You may want to read the first post in this series for an overview of the architecture.)

At this point, I had a WebDAV-accessible directory that I had configured Dashcode to deploy to. It worked to deploy once, but after that, future deploys would fail. A dialog box would appear with the title, "An error occured while deploying the project to the server." and the text, "The folder “myapp” cannot be created on the server." After that, the status at the top of the Run & Share page would show "Last deploy attempt failed: The folder “myapp” cannot be created on the server."

The problem is that Apple's WebDAVFS will send a WebDAV PROPFIND request for "serv/myapp" (which is a folder), get a 301 redirect to "serv/myapp/" (with a trailing slash this time), and then send a PROPFIND for "serv/myapp/", but using credentials which are only valid for "serv/myapp". Apache doesn't like that, sends a 401 Forbidden response, and WebDAVFS gives up and sends an error to Dashcode. (I'm not sure whether this particular problem of retaining credentials across redirects is specific to WebDAVFS, or if it's actually in CFNetwork.)

The solution is to not have Apache send the 301 redirect. Adding this to the Location section for my DAV URL does the trick:

<Location /myapp-devel/dav/>

# In addition to the DAV-related directives
# I talked about in Part 1, add:
DirectorySlash Off
</Location>

This is probably a good thing to do on all DAV directories, since redirection isn't exactly a great thing to do within a filesystem.

After this, I could deploy repeatedly from Dashcode with no problems.

This is as much as I needed to deploy my app. Debugging it, though, was still of an issue. See the next post!

Dealing With Dashcode, Part 2: Apache, WebDAV, and umasks

2010-07-25T15:11:00.000-07:00

(This is a continuation of a previous post.)

At this point, I had set up a directory that I could publish to using WebDAV and also edit as my regular user id. Since WebDAV works as user www, my normal user id (piquan) wouldn't have access to edit them. So I needed to set up a directory using FreeBSD's ACLs to give write access to both me and www.

$ mkdir serv
$ setfacl -m u:piquan:rwx,u:www:rwx piquan
$ setfacl -d -m u:piquan:rwx,u:www:rwx,u::rwx,g::rx,o::rx,mask::rwx serv

The problem is, when WebDAV created a directory, it created it with mode 755 (rwxr-xr-x). The problem is, if a program that's not aware of ACLs creates (or chmods) a file or directory, the ACLs mask bits get taken from the group access bits. In other words, the permissions that the program gives for the group restrict the permissions of any u:… or g:… entries in the ACLs. The r-x mask that WebDAV created my directory with masked my u:piquan:rwx ACL entry to an effective u:piquan:r-x. (Since www was the directory's owner, its u::rwx entry didn't get masked.)

In other words, I could no longer add, rename, or delete files from the directories that WebDAV created. I also couldn't edit the files, since they were created with mode 644.

After some investigation, I found that WebDAV creates files and directories based on the Apache process's umask.

On one hand, I could edit /usr/local/sbin/apachectl and /usr/local/etc/rc.d/apache22 to set the umask before launching Apache. However, I didn't like that idea, since then any Apache upgrades would blow away my changes. I wanted something a little more lasting.

Instead, I used – or rather abused – a mechanism in rc.d/apache22 that's designed for setting process limits. It includes a bit that runs some shell code and evaluates the output. I added to /etc/rc.conf:

apache22limits_enable=YES
apache22limits_args="echo umask 0002"

After that, I ran "sudo /usr/local/etc/rc.d/apache22 restart", sudo rm'd the files that WebDAV had created already, and published it again. This time, WebDAV created files I could edit. Mind you, after that, Dashcode couldn't delete the directory it had created to republish it, but that's for my next post.

Dealing With Dashcode, Part 1: Architecture and Basic Configuration

2010-07-25T01:54:00.000-07:00

I've been writing a web app for the iPhone recently. The main tool for this type of web app is Dashcode, which is part of Xcode. There's a couple of problems with my setup: one of them I've solved, and the other I haven't. In the hopes that somebody else on the Internets might find this information useful, here ya go.

The solved problem is how to publish this on my own server. Most of this revolved around my Apache configuration. Most of this should be pretty obvious to anybody who's dealt with WebDAV, but one problem with authentication wasn't obvious; that's at the end, but first I'll talk about my setup.

My app involves a combination of three major elements: static HTML documents, dynamically generated data, and the web app that's generated by Dashcode. (The web app is served as a static page by Apache, of course – it's the client that does all the computation there – but it has its own considerations in the Apache configuration.) My web site, therefore, has three sections. (All the project names, domain names, etc. in this are replaced with more generic versions. These are not real URLs.)

http://www.piquan.org/myapp/gen/ : Dynamic content, generated by Python scripts. This is being read by Ajax, so there's some files in here that generate XML, some that generate JSON, and for my own convenience it's nice to have a few static files in here as well.
http://www.piquan.org/myapp/htdocs/ : Static content, just a bunch of HTML files. The web app will sometimes send the user to these files (and out of the app entirely) by setting window.location.
http://www.piquan.org/myapp/serv/ : This holds the app generated by Dashcode. I want Dashcode to be able to blow everything under here away, and replace it.

All of the files are currently in ~/src/myapp, and while I'm just working on the initial version, I want it to be served straight out of there. While I'm doing development, all of this is under http://www.piquan.org/myapp-devel/ , and I'll move to /myapp/ once I've got a releasable version.

To get the static files to be served is pretty easy. I just put this in my Apache config (within the VirtualHost section for www.piquan.org):

Alias /myapp-devel/ /home/piquan/src/myapp/
<Directory /home/piquan/src/myapp>

AllowOverride All
Order allow,deny
Allow from 192.168.42.0/24
</Directory>

(The "AllowOverride All" is to make it easier for me to experiment using .htaccess instead of needing to change my Apache config and restart the server. Of course, .htaccess is somewhat limited in some ways, as we'll discuss later.)

Ok, so what next? Well, I need to serve the dynamic content. Remember that there's both static and dynamic files in there. Ideally, it should be transparent whether the content is coming from a static file or being dynamically generated.

To generate the dynamic content, I chose to use Python with raw WSGI. This is a very simple way to write simple dynamic content. For complex projects, then a web application framework like Django would be more appropriate, but here I'm looking at about 300 lines of code, so Django would be overkill. Also, remember that here I'm only sending either simple XML or JSON. Now, WSGI is actually an interface standard (like CGI), not an implementation; the Apache WSGI implementation is mod_wsgi. I installed this (using FreeBSD's Ports mechanism).

The default for mod_wsgi is to serve data from within the Apache process. On one hand, this is very efficient. On the other hand, it's very inconvenient for development, because if you change your code, you have to restart Apache. There's an easy solution: put the WSGI handlers in a separate group of processes that Apache automatically manages. When you change your code, mod_wsgi will automatically kill and restart those. If you don't understand that, then don't worry: the config file additions are quite simple. I just added to the VirtualHost section:

WSGIDaemonProcess piquan.org display-name=%{GROUP}
WSGIProcessGroup piquan.org

This sets up a simple set of processes to manage all WSGI requests. In this configuration, the same set manages everything across my entire domain. (That's not because of "piquan.org" there; that's just an identifier for the process group. It applies to my entire domain because it's within the VirtualHost section, not within a Directory section.) Once it's time to release, I'll change the name from "piquan.org" to "myapp" or something, tune the WSGIDaemonProcess line to use the appropriate amount of resources (in terms of processes, timeouts, etc), and move the WSGIProcessGroup to within a Directory section so that it only applies to my program.

The mod_wsgi Quick Configuration Guide says to use WSGIScriptAlias to tell mod_wsgi what directory should be considered WSGI programs. However, remember that I have a mix of dynamic and static files, so I took a different tack.

To deal with this, I put the static files in files named things like staticdata.js, staticdata.xml. The dynamic files are named things like dynamicdata.js.wsgi and dynamicdata.js.wsgi. Then, a bit of .htaccess magic, along with the miracle of MultiViews (which I don't know WHY it's not more widespread) lets me tell Apache to send things the right place.

Options All MultiViews
MultiviewsMatch Handlers Filters
AddType application/json .js
AddType text/xml .xml
AddHandler wsgi-script .wsgi

This is actually a bit of overkill. I specified a very widespread Options line. The only option I really needed for this was MultiViews. [Edit: I also need ExecCGI. Thanks, Graham!] It's just more convenient during development to have things like Indexes and FollowSymLinks on. Also, I set the MultiviewsMatch to include Filters, when really I only need it to deal with Handlers. (I'll probably turn on the mod_deflate filter later, since this is an iPhone web app that will sometimes be sent over 3G and even EDGE networks, but that's not something that MultiviewsMatch needs to be involved in.) I generally tend to put in pretty broad web capabilities during dev, and then tighten it when I deploy.

The two AddType directives are because I thought one of the libraries I was using was being a bit picky about the Content-Type it gets back. (I could have named my JSON files with .json instead of .js, since Apache already associates application/json with the .json extension, but the .json extension irks me a bit.) As it turns out, the library wasn't as picky as I thought (for instance, it would be happy with Apache's default of application/xml, which is arguably more appropriate for this purpose; note that both are valid MIME types for XML), but I left them in anyway. Note that the AddType directives only apply to the static data; they don't apply to WSGI scripts, since those send their own Content-Type header.

(By the way: when you're writing apps like this, you can handle REST-style URLs pretty easily. For example, http://www.piquan.org/myapp/gen/person/piquan could be handled by a script named gen/person.wsgi, which can look at environ['PATH_INFO'] to read the "/mary" bit. Note that this is the environ passed to application, not sys.environ. If you really wanted to get fancy, you can probably add some MultiViews magic along with sections to separate this into person.GET.wsgi, person.PUT.wsgi, person.POST.wsgi, etc. Hmmm... maybe I'll write a filter module to let that happen easily.)

Finally, we have the Dashcode-generated content. This is created on my Mac, and I need to send it to my web server. Hello, WebDAV! WebDAV is pretty nice to have on your web server if you use a Mac as your desktop: it lets you keep an iCal calendar shared on a website (without paying for MobileMe!), gives you a pretty convenient and WAN-accessible file storage from the Finder, you can publish from iWeb, and so on. (Ok, blatant advertising done.)

WARNING: Don't configure WebDAV until you have secured your web server! WebDAV lets people write to your disk. That's its point. If you don't have a secure server, then you may find people you don't want writing to your disk.

In fact, don't trust my configs on this. Read over the docs for mod_dav, and you should have a decent understanding of web server security. At a bare minimum, read Apache's docs on Security Tips, and also the docs on Authentication, Authorization and Access Control. You should also know why the latter is insecure. (Hint: Don't use Basic auth; use Digest instead!)

Ok, now that I've made it clear that I don't want you to open your disk to the whole world, here's a bit of my configuration to allow Dashcode to publish using WebDAV. I actually opened up a lot more than I needed to: instead of just the area where Dashcode puts its static files, I have it set up to allow WebDAV access to my entire program. This lets me fiddle with stuff in the Finder if I need to.

Now, here's why that's a really bad idea from a security perspective. WebDAV lets people write files. WSGI lets people run files. That means that with the two together, an attacker could write to, and then execute, a file on MY computer. Bad news. Don't do this unless you're satisfied with your security: in particular, and at least put a reasonable Allow clause in place.

Here's the configuration I used. Again, this is back in the Apache config file, in the VirtualHost section.

Alias /myapp-devel/dav/ /home/piquan/src/myapp/
<Location /myapp-devel/dav/>

Dav On
Options Indexes
Require user piquan
</Location>

And then in .htaccess:

AuthType Digest
AuthName "myapp DAV area"
AuthUserFile /home/piquan/.htdigest

(By the way: everything I put in my .htaccess, you can put in the regular Apache config. I just prefer to use .htaccess for everything I can, since it lets me change options without restarting Apache.)

Finally, I set up my password file by using the htdigest shell command on my web server:

$ htdigest ~/.htdigest "myapp DAV area" piquan

(I already had a ~/.htdigest, and just was adding a new realm+username+password tuple. If I didn't have an .htdigest, I would need to specify the -c option.)

Now, the mod_dav docs recommend using completely different URLs for the WebDAV URL and the "regular" serving URL. That's probably a little bit easier to configure. As for me, I kinda wanted to keep my entire project under the same top-level directory (myapp-devel), so I made the DAV-enabled section a subtree.

Ok, there's one element left: permissions. I needed the web server to be able to write to the serv directory (where Dashcode was to write its files), but I also wanted to be able to – as my normal user on the web server – edit those files (to edit the manifest, or for whatever other post-deploy changes I wanted to make). Since WebDAV accesses files as user www, and I access them as user piquan, I needed to make serv writable by both. But I didn't want to make it globally writable, either.

For this, I turned to FreeBSD's ACL support. I'm not going to go into details on how to configure a filesystem or kernel for ACL support; there's guides for that online already. (If you don't like the Handbook's dry style, try the O'Reilly ONLamp article instead.)

I needed the serv directory itself to be writable by www and piquan, and also I needed any files within that either I or WebDAV create to be the same (unless explicitly changed). This means configuring both an access ACL and a default ACL.

$ mkdir serv
$ setfacl -m u:piquan:rwx,u:www:rwx piquan
# The next command is all on one line.
$ setfacl -d -m u:piquan:rwx,u:www:rwx,u::rwx,g::rx,o::rx,mask::rwx serv

After all this was done, I went back to Dashcode's "Run & Share" section, and added my web server as a new destination. Finally, it was time to publish!

At this point, everything I've talked about so far worked. But soon after that, I started having problems. I'll tackle each of these in a separate post.

Good Businesses

2009-04-18T15:36:00.000-07:00

Here's some businesses I dealt with over the last week that have given me excellent customer service, and I'd like to recognize publicly.

Speakeasy, my DSL provider
The Starving Musician, where I get instruments and equipment. (I don't deal with them online; I go to their Santa Clara location.)
State Farm Bank, my bank (natch)
Audible, an audiobook website

Shouts out to these companies for taking care of their customers!

A few non-secrets

2009-02-19T01:30:00.001-08:00

A month and a half without a post, yow. Ok, so here's the thing. I've had stuff going on in my life, but a lot of it involves friends, and their privacy. I can't talk about the friend getting surgery, because that's medical. I can't talk about the acquaintance who I suspect may be trying to manipulate me, because it's just a hunch right now. There's a lot of stuff I can't talk about going on.

Here's some things I can talk about.

Two of my best friends are getting married in two weeks. Congratulations, Scott and Mili! Scott and I have been close friends since fourth grade, and he and Mili have been crazy about each other for years.

My work has been progressing well. I've still been working on the project I talked about in my last post, but the pilot has been running more slowly than we had expected. There's been some minor technical changes from the description I made (most notably changing from using a magic uid/gid as a marker to using UFS2 extended attributes), but all the details will be out when we open-source it.

As you know, I've occasionally been writing some music. Mostly funny stuff, not good stuff, although I do kinda like "Baby Wombats (House Remix)". I've also been working on a metal song about puppies and rainbows and stuff. I've got it written, but need to re-record it since what I've recorded so far is just draft quality.

I hope I'll have something more interesting to post soon!

What I'm working on at Juniper

2008-12-31T13:18:00.000-08:00

I haven't had much time to update my blog for a while. It's mostly been work, so I think I'll tell you all about it.

I'm loving my job at Juniper, although I must say it's keeping me very busy lately. The project I'm on is pretty close to the wire. In the weeks approaching the Christmas holidays, I was spending every waking minute working on my code to get it ready. That left me able to go to Christmas and enjoy it without worrying about work (more about my vacation in another post, but the basic summary is, it was terrific!). Even though my part is working great, there's some other parts to the project that are a bit tense, so I'm helping where I can. Since I got back from vacation, though, things have been a bit more relaxed for me.

Anyway, I can't go into all the background on the project, but I can tell you about my part. (This gets a bit technical from here on in, so you can skip it if you're not a computer geek.) The goal is to make it possible to compile with source code on NFS servers and object code on local disk. The source then can be edited remotely on workstations, and easily backed up; the objects don't take up expensive NetApp space, and the compile proceeds at local speeds. This mechanism speeds up compiles by a factor of 2-3.

The core of this is a new filesystem layer that I wrote, lcachefs. This is a type of loopback filesystem, similar to nullfs or unionfs. lcachefs does some magic mirroring involving three directories. As with any filesystem, one of these is the mountpoint: where lcachefs shows its results. Unlike most filesystems, though, it involves two sources: one is known as the source directory (which holds the NFS-mounted sources), and the other is a storage directory (local disk).

Ok, let's pretend that you first copy your sources from a source directory to the storage directory. Then you loopback-mount the storage directory over your original source directory. Then, you compile there. (You mount it over your original source directory so that the filenames in the debugging information point to the original sources, instead of to a temporary storage directory.) This looks, to the compiler, just like a regular compile. However, instead of doing the compile on NFS, you're doing it on local disk.

This has two advantages. The one that's easy to explain is that you're storing your objects and products on local disk (which is cheap), instead of on NetApps (which, despite all their greatness, are expensive). Objects can easily be recreated, and take up most of the space in a build tree.

The other part is that you're working entirely on local disk. This is much, MUCH faster than using NFS. It's not because of the bulk data transfer (i.e., reads and writes), so higher bandwidth doesn't help. The problem is with the per-request overhead. There are a LOT of requests going across the wire in a compile.

Let's consider what happens when you build a file. Specifically, we'll talk about if you're just building hello.o. Make will look for Makefile, makefile, BSDmakefile, and .depend. Then it will look at hello.c and hello.o. (If it needs to, it may look for hello.s, hello.S, hello.f, and whatever other potential sources you may have.) It'll look up hello.c.gch (if you have precompiled header support), hello.gcda (if you have coverage support). Then it has to look for stdio.h. Well, let's suppose that you have two directories in your -I path; let's call them foo and bar. Now it needs to check for foo/stdio.h.gch, foo/stdio.h, bar/stdio.h.gch, and bar/stdio.h, all of which are in your source repository (hence usually on NFS) before it can go on to look at /usr/include/stdio.h. Well, the first thing that stdio.h does (on FreeBSD; your OS may vary) is to include sys/cdefs.h. You got it... another four lookups in the source repository to find sys/cdefs.h! Repeat for sys/_null.h, sys/_types.h, and machine/_types.h. In the end, just to compile hello.o, you've done 39 file lookups into your source directory. Of those lookups, 21 are for .h files... and hello.c only includes one file, stdio.h! The average .c file (using the FreeBSD source base as a sample) includes 9 files directly.

That's a lot of work. These lookups are all blocking I/O, so it's one round trip each; from watching network traces, I'm actually very impressed with how fast NetApps can respond, but it's still a non-zero time. Moreover, though, you've got to look at the CPU usage. NFS uses RPC, and the overhead isn't cheap. In my testing, using NFS to compile something requires four times as much kernel processing-- which works out to twice as much CPU overall-- as using local disk! (That's all in the kernel, too, and that means that preemption isn't as easy, so scheduling takes a little hit... and don't forget the new network interrupts!)

Finally, NFS is designed around concurrent use. You can't assume that the contents of an NFS-mounted directory will be the same now as they were 90 seconds ago. While local disk can use metadata caches very, very effectively (and FreeBSD's filesystems do), when you're using NFS, you have to expire caches a lot. (There are also some things that FreeBSD could do better when it comes to its NFS client, but I fixed those and only got a 5% boost.)

Ok, now you should be convinced that the speed hit for NFS is bad, and that building on local disk is much, much, faster. Let's go back to talking about lcachefs.

Earlier, I asked you to think about copying your sources to local disk, and doing the compile there. That's exactly what lcachefs does, but with a twist: it does this lazily, by which I mean it won't copy files until they're needed.

When you first mount an lcachefs directory, it scans just the top-level of your source directory. It then creates stub files for each entry in the local storage. This is just an empty file with a special uid/gid pair that marks it as a stub. If you ls -l that file, then the filesystem will report the correct uid/gid, size, link count, etc, but in reality the locally-stored file is empty.

When you actually try to read (or otherwise use) a file or directory that's a stub, then it will copy it over to local storage before the system call returns to userland. In the case of a file, this means it copies the contents and sets the uid/gid to its real, non-stub values. In the case of a directory, it scans the directory and populates the stubs just like I described in the previous paragraph.

That's the basic overview. The reality is quite complicated because of hard links, the need to watch out for vnode lock order reversal, and other icky stuff like that. You can see all the gory details, though, once we open-source lcachefs. I'm currently awaiting clearance from legal, but I hope to either contribute it to FreeBSD, or release it as an independent open-source project.

Good Friends

2008-12-07T15:59:00.001-08:00

I was thinking that I should make a post today, and was wondering what. I was trying to think of cheery topics, and looked at my blog to get ideas. Then the Google ads were one for losing fat, and one for help with alcohol & depression. Boy, how cheery can you get?

Actually, quite so. In the last several weeks, I've gotten in touch with a lot of friends who are very dear to me. People I haven't heard from in years and years. Some of them contacted me, I contacted some of them. Most of this has been over Facebook or MySpace, one was on this blog which led to some emailing (and there'll be more of that, I promise!), one I got to visit in person while I was in Texas (rock on!), others I've only spoken with on the phone. But it really seems like the stars have aligned for reopening old friendships!

Really, think of everybody who's ever been your life who has their own big mansion in your heart. Then imagine getting to visit with each of them, over the span of a couple of weeks. It's been an incredibly special time for me lately.

I'm gonna be grinning like a fool for a long time now.

Not what I'm doing at Juniper

2008-12-04T01:53:00.001-08:00

I don't often make technical blog posts (ok, lately I don't often make posts), because two of the three people who read my blog don't really care. But sometimes it's nice to be able to brag about what I'm doing at work. Besides, last post I promised to talk about what I'm doing.

Since then, I haven't made any posts, because I didn't have the time to say what I was doing at work. Which is stupid, really. You three people don't read my blog because it's got juicy technical details; that's for the rest of the masses, and they don't care what I might have said I was going to post about until I post it. I've had other things to write about, but since I said I'd write about work, I didn't want to write about anything else. I several times thought, "Ooh, I should blog about this!" and then thought, "No, I said my next post would be about my work at Juniper."

Well, I am hereby renouncing that commitment. What's more, I am advising you three (tonight's loyal fans) to take anything I might tease about as merely speculation on future topics, and not be sure I won't write about something completely different first.

So there. I've just freed myself to write about other stuff. I don't really know why I felt it necessary to maintain an obligation that nobody cares about; why would anybody care if I put some posts in between the previous post (when I said I'd talk about my Juniper work) and the future post when I actually do, is totally beyond me. I just did feel that it's necessary, so now I can go on with more regular posts.

Except it's late and I'm going to bed instead of subjecting you to more of my incredibly convoluted late-night grammar.

Gasp! It's a post!

2008-11-17T17:18:00.000-08:00

So here's the thing.

I was miserable at Google. It's a great company, and I think it's a wonderful place to work. But the position I was in just wasn't right for me. I didn't want to blog at the time, because I didn't want people to get the wrong impression about Google. Seriously, it's a great company; I just wasn't where I should have been.

So at the end of July, I left Google and went back to Juniper. I was happy at Juniper, and wanted to be happy again. Besides that, Juniper offered me a promotion, a raise, my choice of projects, and the opportunity to work under a manager I know I work well with. I'll talk about the work I'm doing for Juniper in my next blog post.

After I went back to Juniper, I've been concentrating on getting into the swing of things, so I still haven't been blogging. Now I'm doing well, and can start posting again.

Now, here's one last note to the blogosphere: you may note that I'm still using a Google's blogger.com, storing the photos using Google's Picasa Web, and displaying Google AdSense ads. If I were upset with the company, I wouldn't be doing these things. I'm strenuously emphasizing that I like the company, because things get taken the wrong way in the blogosphere, and I don't want anybody to get the wrong idea. I'm also free to badmouth the company if I wanted to, since they're no longer my employer and have no control over my actions. But I'm not; I'm talking about how great they are.

Anyway, my point is, hello world, it's nice to see you again!

An Oakland Shade

2007-09-04T21:58:00.000-07:00

Wow, that was a whiny post last weekend! Fortunately, this weekend was much better... after all, Labor Day, right?

My friend Jenny threw a party this weekend, a holiday barbecue. She had a problem: there's been a heat wave lately, and her house was going to be positively sweltering. She came to me, a friendly neighborhood engineer, to build a shade.

The party was Sunday, so on Saturday I went over there, scoped out the project, took some measurements, and made a list of supplies. Then, off we went to the hardware store to get supplies, which naturally deviated considerably from my list. (Anybody who likes going to hardware stores understands what I mean. Computer stores too.) Three canvas sheets, some rope, twine, dowels, and various fasteners. For once, I needed exactly what I got! (Well, I had a small pack of garden stakes left over.) A couple of supports, stitching together the canvas, some guy lines, and the job was done.

It's not the best shade in the world, but for an afternoon job, I liked it. I also had a lot of fun building it; I may have had more fun building the shade than going to the party! It reminds me of something I used to do when I was a kid called Christmas in April. A lot of people would get together and rebuild the house of somebody who really needed it, and couldn't afford it; usually, this was an elderly or severely disabled person on a very minimal income. We'd have people who knew what they were doing well enough to lead our efforts, plus lots of hands helping get the work done.

I like building stuff; I think I get that from my Dad. I think I was first attracted to computers because I could make my creations come to life, from my mind to the computer. Even so, there's something satisfying about building something in the physical world.

Why This Post Isn't About The Javascript DOM

2007-08-27T00:00:00.000-07:00

My original title for this post was "Why The DOM Is The Best Lousy Thing We've Got". I guess I'm not feeling terribly creative with my titles tonight. To be honest, I'm not feeling terribly creative at all tonight. It was a terrific week, but a pretty lousy weekend. I had some unexpected work take up all my time, and it put me in a sour mood for most of the weekend.

It's strange how creativity is so dependent on mood. I'm not just talking about blogging, here; I'm talking about coding, too. When I'm doing well, I can code anything, start to finish, piece of cake. When I'm doing poorly, then even writing hello.c is a huge effort. The work I talked about earlier involved some coding at the end of it— nothing big, just a few dozen lines to shove some data around— but if it weren't for that, I probably would have had a really lousy weekend, instead of the merely lousy one I had.

Ok, now I'm just sounding needlessly whiny. If you start seeing emo bands show up on my weekly top ten chart on the sidebar, send help. Like a box of kittens or something.

Anyway, where was I. Oh, that's right, having a lousy weekend because of work. I really started thinking that maybe I made a mistake taking this job; maybe I was more cut out for the Juniper job than the Google one. Now, here's the irony. When I left Friday, I was writing a document that includes advice to people who feel that way... pretty much saying, "No, it's normal, everybody feels that way, you'll do fine". (This is advice I had previously given Cheryl, and also talked about more in my blog.) And then this weekend I get hit with this wave. Wasn't expecting it.

I did get some useful stuff done for myself this weekend, though. The office is in much better shape, with the bookshelf moved, lighting hung, and desk accessories connected. Even the living room looks cleaner for some reason. At the end of it, I wanted to relax by writing some code. I was going to learn some SVG. Still, as I sat down to work on it, nothing came to mind. I hate hacker's block.

If you haven't read Jhonen Vasquez's "I Feel Sick", it's about an artist who is suffering from artistic withdrawl once she takes a job at a big company. I hadn't read it, and didn't know what it was about, but picked it up tonight to read and forget about my bad mood. (Hint: the "Alice" soundtrack works well with Vasquez.) Y'know, that just didn't help me forget as well as I had expected.

Finally, I decided to write about a blog post I've been wanting to write for a while... I even alluded to it in a post from two weeks ago (when I refer to the "later, more technically-oriented post"). And that's what I've sat down to do here. Hmmm... I seem to have done a great job, haven't I?

Moving

2007-08-17T19:01:00.000-07:00

We moved from upstairs to downstairs in the same building. I like the new crib, although now I'm back in a cube. Still, it's the best cube I've worked in, and there's plenty of opportunity for cube dressing. I'm going to go by the surplus store, hardware store, and toy store to get some supplies for a few ideas I have, and cube decorations from past jobs (which are still in the box I took home from Juniper) will also play a heavy part.

There was some discussion about airspace issues. There's 18' ceilings in this area, so the question of how to handle the airspace becomes relevant. Naturally, the discussion mostly involved us playing with Karl's RC helicopters, and trajectory arcs of Nerf guns.

Anybody have cube decoration ideas they want to throw into the ring?

SRE R00lz!

2007-08-17T16:10:00.000-07:00

This was taken at a company picnic yesterday. We had a tropical theme going... steel drums, flower leis, etc. There was a climbing wall, human foosball, and, of course, volleyball. Since my knee has been giving me problems (I actually had to leave early to go to physical therapy), I stayed away from the volleyball pits and instead played frisbee.

Yesterday's Beans

2007-08-15T16:02:00.000-07:00

Okay, yesterday's post was evidently more tantalizing than I meant for it to be. I started by saying that I wasn't talking about my new friend. Well, that seems to have been raising questions among my other friends, so I've been getting bugged to talk about it.

Well, not from all of my friends. Some of my friends were there, some of them are too polite to pry, some are so busy that we haven't talked since then, but this is not an exclusive partition of my friends. So m3tus, this post is for you.

The thing is, it's not like there's anything scandalous that would make Matt Drudge drool. I just talked to my new friend about some problems she's been going through. I thought those problems might make for an interesting post, but her problems are hers to tell the world, not mine. That's all.

Restraint

2007-08-13T21:38:00.000-07:00

All week now, I've been mulling over a blog post. It's about a new friend I made last weekend, and the interesting circumstances under which we met. But there's a problem: that would be a violation of her privacy. I spent a fair bit of the week thinking about this. For a while, I was considering writing a version that left out some parts. Finally, I decided that I should let her tell her own story to the people she wants to; it's not my place to do so.

This is the sort of thing I have to think about when I'm writing to a public audience. (Okay, that public audience is-- in the last week-- only 56, but ethics is not a numbers game.) I have to do this all the time when I write about work. Obviously, this is partly because I don't want to end up like Mark Jen or something. But moreover, it's because I have a responsibility to my employer, and don't want to disclose information when it might do harm.

This is a sort of self-restraint, which is a good thing. It suppose that in this context, it's similar to the legal concept of prior restraint, which is generally considered a bad thing under most circumstance. However, there's a big difference. Here, I'm avoiding talking about things on my own volition; in the case of prior restraint, it's from a governmental gag order.

This weekend was my friend's birthday. His girlfriend and I had planned out a birthday party for him, culminating with us seeing a performance of The Compleat Wks Of Wm Shakespeare (Abridged). But it started off with us playing Guitar Hero Encore: Rocks the 80s. I've got Guitar Hero and Guitar Hero II, and we have a blast playing them multiplayer. Unfortunately, we realized when we started playing that most tracks in multiplayer aren't available until you unlock them in single player mode. That meant that, for the time being, we only had eight songs available. Luckily, by the time we finished those eight, it was time for us to head up to the city.

Now, that all happened Saturday, so Sunday I had to myself. I took care of some personal errands, like laundry and groceries, but also played Guitar Hero for a while... y'know, just to unlock the songs. I normally play GH on Expert, but this time, I played on Hard. That way, my friend and I would be able to play the Expert level for the first time together. (I did unlock all the songs, by the way. If you want the track list, and have iTunes, I put together an iMix with the songs.)

I thought about some of the stuff I had going on at work. There's plenty that I'd enjoy doing, but I was keeping myself from doing work. I was bound and determined to make the weekend about myself, and NOT the shiny new job that's been occupying me for the last few weeks.

I got pretty bored playing Guitar Hero on Hard, but wanted to unlock the songs. So I kept at it. I played on Hard, and not on Expert. I unlocked songs, instead of doing some interesting work.

Why the restraint there? I have to wonder if at some times, I'm holding back where I shouldn't.

I feel like that at work too. I keep feeling like I'm marking time, slowly learning, waiting until I get to the point where I get to take off the kid gloves and really get to it. Thursday and Friday, I sat down and did some coding. I started on a program to help me maintain my calendar, and another one to give me a cheat sheet for a weekly meeting. The former is pretty mundane, but the latter was certainly fun. I'll talk about why in a later, more technically-oriented post.

For now, I'll have to think about when's the right time to hold back, and when's the right time to let loose.

I've been found!

2007-08-03T21:51:00.000-07:00

Okay, this is a surprise. I've been writing this under the idea that I'd have some small portion of my friends and family reading. And maybe the occasional Google search. I figured that's where the one comment I've had came from. Then, today, I did a little log analysis.

Since I opened the site a couple of weeks ago, I've had 2,490 visits from all over the world, from Malaysia to Tanzania. The visitors are pretty telling of the type of audience: 49% have been using Windows, 29% using Macs, and 21% using Linux. The browser numbers are more optimistic: 69% Firefox, 12% Safari, and only 7.5% using Internet Explorer.

I will point out that these numbers may well be skewed. The statistics were only collected for about 2/3 or less of the page views. In other words, I know that there's been at least 6,000 views, but only about 4,000 of those had statistics logged.

How did all this come about? Well, somebody found my post about Sawzall, and posted it to reddit. There was a little bit of activity before that, but I haven't yet looked at what led them to find it.

Then again, I never did publicize Vigor at all (I just sent Illiad an email), and enough people downloaded it that I was getting dozens of emails a day. I suppose even the most trivial writings are still going to be found by the zillions of eyes on the Internet.

Carving out time

2007-08-03T20:47:00.000-07:00

You haven't heard from me all week, and you're starting to wonder if I vanished into thin air, lost my password, or what. Well, I've just been busy.

All last week, I was wondering why I was having such a hard time getting anything done. I had so much stuff that I wanted to get done, but never seemed to be making any traction on it. Then I looked at my calendar, and it all became clear.

Okay, admittedly, that screen shot is from two weeks ago, when I still had a full schedule of training. But nevertheless, I've been going to a lot of meetings. Some of them are necessary, some aren't. I realized that I'd have to defend my schedule against meetings.

That's what I've been doing the past week: I was declining meetings that I didn't need to go to, and making time to work instead. This involved studying, coding, the usual stuff. This week, I was getting stuff done— but I was staying there until 11:30 PM at the earliest, 1:30 AM at the latest. I wasn't getting nearly enough sleep. On the upside, spending all this time in the office gave me a chance to learn about the laundry facilities at work, but overall, I think it was more of a bad thing.

Next week, I'm going to be defending my personal schedule against my work schedule. I need to make sure that I give myself time to relax, to take care of personal errands, to visit with friends, sleep, and do all of the rest of the things I have to do that don't involve work.

You may have noticed that my "Recent News I Found" sidebar has had a fairly high number of life organization articles. (Remember that it's a separate feed from my blog. To subscribe to that, click the "Read More..." in the sidebar, and subscribe to the feed there.) I suppose that, as I'm finding my feet in my new job, I've been thinking a lot about getting my life better organized.

Welcome to the blog!

2007-07-29T19:18:00.001-07:00

As you can see, the new format is up, up, and away! I've brought my Noogler writings over here, of course, and you'll be seeing more new content as the muse strikes.

Let me start by giving you the executive summary for this post. If you don't care about technical details, or getting things just right for your tastes, then just go to reader.google.com, sign up if you need to, click "Add subscription", then in the box that pops up, type "www.piquan.org". Then, add another subscription, this time for (and you'll need to copy/paste this) http://www.google.com/reader/public/atom/user/10100467309635594692/state/com.google/broadcast
(whew; got that all)? Add subscriptions for all the other sites you might go to frequently to read news (Slashdot, CNN, other blogs, etc). Then, instead of going to each of those sites each day, you can go to Google Reader to see them all together, easily read. There you go, now you don't have to read the rest of this post.

Whew. We've got the type A personalities out of here. Now the rest of us can relax and enjoy the new blog.

Now, if you've taken the time to look over the sidebar, you might think this website is the height of vanity. Besides my last ten posts, and monthly archives, you've also got my bookmarks, my news clippings, my music, my friends, my photos, my art, me, me, me! Yeah, okay. I guess the way I figure, it's my blog; it should have a lot of my personality in it. You've got links to loads of personality, right there in the sidebar. And naturally, you should come back every day to check it!

Okay, no, that's not true. Most of us already have websites we periodically look at; I have about 30 different websites I like to check every day already! (You can click on the "daily" tag in the sidebar under "My Bookmarks" to get a list.) Surely there's some great technology to help us keep up with all the articles from these different sites, and show us just the new ones that we, personally, haven't seen.

Why yes, there is! It's called RSS, and it's all over this website. I'll talk about some of the basics of using RSS now, so later on I can just say that some feature "has its own RSS feed", and you'll know what I mean, and how to use it.

First, I want you to look at the right hand side of the address bar. Yes, the bar at the top where you type in "www.piquan.org". There may be an RSS logo there. Depending on what browser you're using, it might be the word "RSS", or might look like a silly orange logo instead. Chances are, it looks like the silly orange logo. (If you're using Internet Explorer 6, you won't even see the logo; we'll talk about that case in a little bit.)

Now, you could click on that, and I'll talk about how different browsers handle it.

Firefox: If you're using Firefox, you'll be shown the feed (a list of the recent articles) and given a chance to add it as a Live Bookmark, or subscribe (mark it as something you want to read) using one of a number of other possibilities (certain websites or programs). If you add it as a Live Bookmark, then you'll have a bookmark (by default, in your bar) that's actually a drop-down list of articles. When you feel like reading my articles, you can click on that bookmark, find a headline that looks interesting, and click it to go straight to the article. It's a quick way to scan headlines without having to visit the site!
Safari: When you click on the RSS icon, the web page will transform into a feed view. If you drag that to your bookmarks bar, it will turn into a bookmark that will always show how many new articles there are. If, for example, there's two articles since you last looked at the site, you'll see Piquan(2) for the bookmark. Clicking that bookmark will take you to back to the feed. Safari also has a lot of other RSS features, like the ability to show multiple feeds at once (put them in a bookmark folder), show just part of each article in the feed display (the control is on the right side), or turn your RSS feeds into a screensaver (it's in the Desktop & Screen Saver system preference panel, with all the other screensavers).
Opera: When you subscribe to the feed (use the Atom feed), it will be added to your Feeds menu. Within the feeds menu, you can click on an individual feed to read just that, or "Read Feeds" for all your feeds at once.
Internet Explorer 6: Doesn't support RSS. Use Firefox, or look at the section below for alternatives.
Internet Explorer 7: It supports RSS, but I don't really know anything about it. I'm afraid you'll have to play with it yourself, or use Firefox.
iPhone: The iPhone does support RSS, but its support is pretty primitive. The RSS icon isn't displayed, presumably because there's not much screen space. You'll need to click the Atom button at the bottom of the sidebar. You can view the feed that way, and it's easier to read and pick out articles that way, but it doesn't remember which articles are read, or let you aggregate articles. I recommend Google Reader instead; see below.

Lots of different websites have RSS. Pretty much, any website that has short, periodically added content will support RSS. Obviously, this includes blogs (both individual and corporate), but it also includes newspapers like the New York Times and San Jose Mercury-News (yes, even smaller local papers have RSS!), news channels like CNN and the BBC News, and even common social sites like personal photo galleries, music trackers, and even social bookmarking sites. If you have a personal site, such as a team high score list or baby diary, you may want to ask your administrator about enabling RSS, if he hasn't already.

Okay, everybody's using it, but why is RSS so great? It lets you keep up with a lot of different sites easily. Some feed readers (such as the one in Opera) let you read all your feeds at once, so you can read your day's news from a variety of different sites, all at the same time. Some (such as the one in Firefox) don't do that, but do let you quickly scan the article headlines without visiting the website, so you can quickly see if there are any articles you want to read.

There's also programs that let you see RSS news in other ways than your web browser. For example, the Mac ships with a screensaver that will read and display RSS feeds. Google Desktop for Windows has a widget that will display RSS feeds at the side of your screen.

Okay, you remember how I said that my website should have my personality? Apparently, that includes my tendency towards long-winded technical explanations. For just that reason, I'm going to recommend that you stand up, take a break, get a cuppa tea, and come back. But we're getting to my favorite part, so don't go too far!

Better? Okay, then, face front, true believer, 'cause here we go!

I mentioned before that there are third-party programs to view RSS feeds. I don't know what they are; you can search Google for them if you feel the need. (Firefox will automatically find several of these programs, and give you the option of using them to subscribe to RSS feeds.) But I have other ways I prefer to read my feeds.

My favored method is Google Reader. I hadn't really bothered with it before recently. The explanation they gave on the web page was far too vague for my taste, and I didn't really understand what it was. So now, I'm here to tell you.

Google Reader takes all of the RSS feeds you have from every different site and puts them all together. It keeps track of which ones you've read. It lets you mark some of them with a star so you remember to go back to them. It lets you share them with other people. It does all this in a very easy-to-use web interface. It also has a huge list of feeds available, so you can search for feeds on baby clothes, cowboy poetry, or whatever else tickles your fancy. Personally, I'll spend a few minutes each day scanning over my Google Reader inbox, and quickly pick out the articles that are worth reading.

If you use iGoogle, which is Google's customizable portal page, then you can also put a Google Reader widget there.

Okay, I probably sound like I work for Google or something. But seriously, I've really enjoyed using it. Now, maybe you go to Yahoo! every morning. Or perhaps you prefer Ask.com, because it used to be named after an amusing character from Wodehouse books. Well, you're in luck! Yahoo! has a similar service; you can put your RSS feeds on your My Yahoo! homepage. (I don't know how, but it's something they've been promoting a lot lately.) There's also a service called Bloglines that's owned by Ask.com; I don't know anything about that, though.

So maybe, just maybe, you're wondering what interesting stuff I'm finding while I'm looking over all my RSS feeds. Well, you're in luck! You see in the sidebar, where it says "Recent News I Found"? While I'm perusing my RSS feeds, I've got a button in Google Reader to share articles. That will put the article in the "Recent News I Found" sidebar. I've been doing this for several days now, and there's quite a few things I've found in that time.

Of course, the "Recent News I Found" section has an RSS feed. The articles I put in there aren't in the blog RSS feed, and vice versa. That way, you can choose to read just the articles I find interesting (but didn't write), without having to deal with my ramblings, or you can read all my wisdom without having to put up with the rush-a-day tidbits about killer badgers in the UK. The feed for the "Recent News I Found" section is a bit long, though... ready? Here goes: http://www.google.com/reader/public/atom/user/10100467309635594692/state/com.google/broadcast

Okay, that's it for today. That's plenty enough, don't you think?

If You Can Read This, Thank A Sysadmin

2007-07-27T11:21:00.000-07:00

Today is Sysadmin Appreciation Day. Sysadmining, as most of you know, is a rough, thankless job. That's why the last Friday of July is designated Sysadmin Appreciation Day. Do something special for your system administrator, network administrator, sysop, support tech, or (ahem) son today!

If you have no idea what I'm talking about, then visit sysadminday.com.