Sometimes these sketchy ideas are recurring. "Paper Trail" is one of them; it first surfaced circa 2009.

Rather than sketching ideas in private, I thought it'd be interesting to do publicly. Ridicule away.

This idea's motivation is a thought experiment:

What if you could trace every value in a program all the way back to its source at runtime?

This isn't a novel idea. Spreadsheets have been doing this forever, right? For example, assign `1` to cell A1 and `=A1+1` to cell B2. The output value of cell B2 will be `2`, but you can later trace its source back to A1.

It is a foreign idea, though, in general purpose programming languages. For example, assign `a = 1`, then assign `b = a + 1`. The value of `b` is now `2`, but there's no way to know, at runtime, its source is `a`.

Why is this interesting?

Imagine a database-driven website's execution flow. The source of data is now the database, rather than hard coded values. Let's wave our hands with pseudo-code:

# Fetch a blog post
post = Post.first()
# Render template with blog post
render(template, post)
# Print title of blog post in template
<h1>{{ post.title | toUpperCase }}</h1>

The rendered output would be:

<h1>HELLO, WORLD.</h1>

Now imagine a "paper trail" of back references was baked in by default through that program listing. The template engine could just as easily also render (if it wanted to, for content editors):

<h1 class="papertrail-value"
    data-papertrail-href="http://krisjordan.com/posts/1" 
    data-papertrail-property="title"
    >HELLO, WORLD.</h1>

Sprinkle in some JavaScript magic and voilà, front-end content editing becomes possible without the trade-offs it traditionally imposes on front-end developers:

• no constraints on template structure
• no additional work and effort
• no new DOM tags
• no ugly widgets forced in

It'd be powerful.

Interesting follow-on questions and problems fall out quickly:

• What would it take to implement this in user land?
• What about sources that are collections rather than objects?
• What about values that combine multiple source values, like string concatenations?
• How much overhead would this impose? Would it actually matter?
• Is there value in storing the whole trail or just the source(s)?
• Is there any value if the source is read-only?
• What other applications would a reference trail have?
• How much work would a proof-of-concept take?

I'm going to keep sketching on this, at least through the weekend, and worst case when it resurfaces for me again in a few years.

Sound interesting to you? Ping me @KrisJordan.

With `openssl`, it's not too hard. The following tutorial assumes you've setup RSA private/public keys for ssh/git/github/etc.

(Note: If you're on OSX, you should install the latest versions of OpenSSL and OpenSSH with Homebrew.)

First, let's start with our plaintext file:

echo "Hello, world." > plain.txt

Before we can encrypt the plaintext with our public key, we must export our public key into a PEM format suitable for OpenSSL's consumption.

openssl rsa -in ~/.ssh/id_rsa -pubout \
  > ~/.ssh/id_rsa.pub.pem
 
cat ~/.ssh/id_rsa.pub.pem

It should look something like this:

-----BEGIN PUBLIC KEY-----
MIIBIDANBgkqhkiG9w0BAQEFAAOCAQ0AMIIBCAKCAQEAkq1lZYUOJH2Yeq5IG/TfB3vFbRcc6fSxrwuADNuS10ftI9Nd5lsVKiU+T/NkDQ42I8DMVyjrrFS/bfBUoH1DeyhDVMXvCyfRYNtQdhq0zKMs7l1bmmeBoTiXEyOnjst0LTNzdjY6huvWilACCiU+DeRUvZr73VZty/YoAZsHA4GdnTqyLHnusN/k0r6KaTagUxZl26Wkj2J2sIw+3XIMczmPHO0p4bpynEKmKF3tr7bqBPe6s8azQMElibCAA8jTUs45RvHYtdKajmTxfETIQa8a54ZzZ54dApo0yFXOb2LRgk8H5awk5dUNfcX88FoYDWD/RigJEd3F5Y1unaZXJwIBIw==
-----END PUBLIC KEY-----

Encrypt

cat plain.txt \
 | openssl rsautl \
     -encrypt \
     -pubin -inkey ~/.ssh/id_rsa.pub.pem \
 > cipher.txt

The important command in the pipeline is `openssl`. The first argument passed to `openssl` is the OpenSSL command you are running. It has a wide variety of commands covering a wide range of cryptographic functionality. For our purposes, we're doing public/private RSA encryption, so we're using the RSA Utility, or `rsautl`, command. Next, the `-encrypt` key indicates we are encrypting from plaintext to cipher text, and finally the `-pubin` flag indicates we are loading a public key from `-inkey [public key file]`.

Print the contents of the ciphertext with `cat cipher.txt`. You should see fully encrypted gibberish.

Decrypt

cat cipher.txt \
  | openssl rsautl \
      -decrypt \
      -inkey ~/.ssh/id_rsa

"Hello, world."

Boom! We're back to plaintext.

If you actually wanted to trade encrypted messages, PGP is the much "friendlier" and accepted system for doing so. This manual, command-line method of encryption is a neat demo nonetheless.

Memory pressure led to swapping, swapping lead to thrashing, and thrashing led to the dark side where the ready queue was briefly in excess of the number of the machine's cores by a factor of 10. The interruption was brief, but it lead to thinking about low hanging fruit, bottlenecks, and ops.

I've been thinking a lot about twelve-factor app development and deployment lately. The "Twelve-Factors" are a set of principles put together by Adam Wiggins of Heroku fame. Following the methodology takes some additional thought and effort upfront, but has a lot of upside once you're operational. One of the factors concerns logging:

Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including:

• Finding specific events in the past.
• Large-scale graphing of trends (such as requests per minute).
• Active alerting according to user-defined heuristics (such as an alert when the quantity of errors per minute exceeds a certain threshold).

I've been meticulous about the maintenance and archival of HiFi's data logs. We have used them in retrospectives and the occasional in-depth analysis, but have stopped short of putting together a system for harvesting and inspecting the data interactively. Today's incident was motivation to set one up.

The triumvirate of Logstash, Kibana, and elasticsearch came up in a recent Hacker News thread sounded encouraging. Logstash is a flexible event log streamer that makes it simple to extract data from logs and get it into elasticsearch for indexing and analysis (it can do a lot of other neat things, too). Kibana is a beautiful web interface for creating interactive dashboards for your data in elastic search. It's an all HTML5 application running entirely in the browser using elasticsearch's rest API. It's easy to make good looking dashboards:

Getting kibana, logstash, and elasticsearch running only took an afternoon to install and configure after running through logstash's Getting Started Guides. I'll need to come back for some additional work (like purging data) after it's fully proven itself. My initial impression: it's great collection of software.

Ignoring the per-server JVM bloat required by logstash, the whole setup seems too good to be true. Especially for open source software. It's simple to get logstash configured and kibana/elastic search are impressive (and fun!) to work with interactively. Diving in to specific events, isolating classes of events (like 500 errors), and composing dashboards to quickly answer your key questions is powerful.

Warehousing, visualizing, and diving into log data has never been easier. Now I just want to log more.

In the previous post, I decided on an electric kettle to heat the water to the ideal coffee brewing temperature ~195F. I cracked the Kettle's base station open and found two boards inside. One was connected to the heating element with thick, scary wires, the other had buttons, the temperature LED, and a microcontroller. They were connected by 5 wires:

To MacGyver this kettle for PourBot, I needed control over the heating element and the ability to read the water's temperature.

I had five unknown wires and a multimeter. Time to reverse engineer the wires' purposes!

If electronics are also new to you, this is how it feels to open an electronic device, plug it into the wall, and press buttons while trying to discover what the wires do:

Reality looks more like this:

First discovery: the kettle's control electronics are regulated to 5V, which is what the Arduino operates with! This means no extra work to communicate with the kettle, I can jump wires straight to the Arduino.

The Kettle's base has three states: without Kettle, idle, and on.

Without the Kettle, the LED does not display its temperature, so I started measuring the voltages over the wires in this state. From yellow-to-red, they were:

1. 5V
2. 0
3. 0
4. 2.5V
5. 0 (Red)

I was off to a good start, finding the positive/anode wire #1. I was unsure of negative/cathode, somewhat expecting it to be the red wire #5, but it could also be wires #2 or #3. The 2.5v wire #4 was a mystery (and still is, tweet me @KrisJordan if you know the purpose it serves).

Next, I transitioned to the idle state by placing the kettle, with some room temperature water in it, on the base. Here were the voltages of the wires:

1. 5V
2. 0
3. 0.37V
4. 2.5V
5. 0 (Red)

Ah ha! With the kettle on the base, wire #3 had a voltage. Given its fractional value, this must be the temperature sensor. If so, it should increase as the water heats up.

"Here we go," I winced while pressing the Boil button of the high-wattage electric kettle. Once I heard the heating element doing work, I took another reading of the wires while in the On state:

1. 5V
2. 0
3. 0.84V
4. 2.5V
5. 5V (Red)

With the device on, the negative/ground/anode wire turned out to be #2. The red wire, #5, was a full 5V and clearly the wire controlling the relay to the heating element. Finally, an increasing voltage on the thermal sensor wire #3 confirmed it was rising with the temperature of the water.

Now we have a pin layout!

1. Positive
2. Negative (Ground)
3. Thermal Sensor Output
4. 2.5V(?)
5. Power Relay Input (0 off, 5V on)

Mission accomplished: we can read the thermal sensor and control the heating element.

Next, I need to reverse engineer the output from the thermal sensor to know how to convert its voltage reading into degrees fahrenheit. Remember, 195°F is the target temperature for brewing coffee!

TLDR: More Light + Better Sensor = Happiness

Nexus 4 (ISO 1000, f/2.7, 1/20s) - Full Resolution

Nexus 5 (ISO 172, f/2.4, 1/8s) - Full Resolution

The Good News

Optical Image Stabilization (OIS) allows the Nexus 5 to choose slower shutter speeds. Its autoexposure algorithm makes aggressive use of this. Shooting with a phone, hand-held, with a 1/8s exposure, and having minimal camera shake blur is pretty amazing. Almost across the board, the Nexus 5 chose a shutter speed around a full stop slower than the Nexus 4.

The Nexus 5 shoots sharp while wide open at f/2.4. In theory, the Nexus 4 also has an f/2.4 maximum aperture, but in none of my reference shots did it choose to open up beyond f/2.7. Combined with the longer shutter speeds, the Nexus 5 is soaking its sensor with more than twice the light of the 4.

The Nexus 5 optics and sensor are better, too. In only one test shot did the 4 choose ISO 100 and the 5 went higher. The 5's result was still sharper. With a wave of the hands, this is some evidence that the optics are improved. Once shots get beyond ISO 1000, the Nexus 5 is much less noisy than the 4. The sensor is certainly an upgrade.

Image quality is improved. Color looks better than the 4 (I'll come back to this in the bad news). Images are sharper. The front-facing camera improved by an even bigger margin than the back.

The Bad News

Too saturated. All of the shots felt on the saturated site, some especially so. The Nexus 4 tended to do a better job faithfully reproducing the colors of this cold, cloudy day. That said, the saturated pictures overall look better than the Nexus 4's dullness. It'd be nice if saturation could be dialed down.

Shutter release lag is pretty awful. Using the stopwatch on the Nexus 4, I unscientifically measured around 1s to 2s between pressing the shutter release and the exposure actually being taken. Even with focus locked in, it takes a full second. The Nexus 4's reaction was closer real time with focus, and still faster without focus.

The lag should get better with software, but there's a part of me that's nervous it's there to enable the slower shutter speeds (or, perhaps, a period of time in which an algorithm decides if the camera is stable enough for such slow shutter speeds).

Longer shutter speeds are a gamble. A decent number of shots were blurry. I'm guessing my dual-wielding hands were too shaky for the OIS to overcome it. Combined with the shutter release lag, it's quite a long time to maintain a frame. I'm doubtful the Nexus 5 will be able to shoot moving targets.

Auto white balance is slow. Simple solution: don't use auto white balance. Lazy solution: in warm/incandescent environments, give the camera around 5-10 seconds to adjust. Fun exercise: flip to the front camera and back indoors and watch how long it takes to find white balance. There's a good example of what happens when you don't give it enough time, below.

Comparison Shots

All but one of the comparison shots are scaled down from an arbitrary cropping to help highlight differences. These were shot using Android's default camera app. No additional post processing was done beyond crop and scale.

Natural Light

Nexus 4 (ISO 100, f/2.7, 1/25s)

Nexus 5 (ISO 227, f/2.4, 1/50s)

Nexus 4 (ISO 100, f/2.7, 1/25s, 1:1 pixels)

Nexus 5 (ISO 227, f/2.4, 1/50s, 1:1 pixels)

Nexus 4 (ISO 1100, f/2.7, 1/20s)

Nexus 5 (ISO 216, f/2.4, 1/8s)

Incandescent Light

Nexus 4 (ISO 1200, f/2.7, 1/25s)

Nexus 5 (ISO 706, f/2.5, 1/17s)

Nexus 4 (ISO 1600, f/2.7, 1/13s)

Nexus 5 (ISO 665, f/2.4, 1/8s) - White Balance: fail.

Nexus 4 (ISO 1400, f/2.7, 1/20s)

Nexus 5 (ISO 337, f/2.4, 1/8s)

Low Light

Nexus 4 (ISO 1700, f/2.7, 1/13s)

Nexus 5 (ISO 1624, f/2.4, 1/6s)

Nexus 4 (ISO 1600, f/2.7, 1/13s)

Nexus 5 (ISO 1624, f/2.4, 1/6s)

Selfies

(Snapchat friends, watch out. My front-facing camera just got a massive upgrade.)

Nexus 4 (ISO 500, f/2.7, 1/10s)	Nexus 5 (ISO 379, f/2.9, 1/17s)

Conclusion

I'm pleased with the Nexus 5's camera improvements. Sure, it has some limitations, but there's plenty of headroom to take great photos within those limits. The Nexus 5 is capturing 2x-4x the light of the Nexus 4 and with much better image quality. Well lit exposures will be much higher quality photos, and lower light exposures are now possible. My Instagrams of coffee are going to look so sexy.

Always Setup crontab "Variables"

Your crontab file is not a shell script, but it has a few special variables that you can setup the same way you would a shell script. Set these special variables up in every crontab file you work with. Future you will thank you for it.

MAILTO="a@b.com,b@b.com"

When your cron jobs have output, or, more importantly, when they fail, cron will send the output e-mail these addresses.

PATH="/usr/bin:/sbin:/bin"

Logged in to the user account whose crontab you're setting up, go ahead and echo $PATH and copy those contents into the PATH variable of your crontab. Remember, this isn't a real script file, so you can't implicitly append :$PATH.

After assigning PATH and MAILTO, setting up your crontab is much easier.

HOME="/path/to/app/root"

The HOME variable tells which directory cron should execute the crontab commands from. Often times you'll have a user/crontab per project. If so, set the HOME variable to your project's root directory to avoid long, absolute paths to scripts or from having to 'cd /path/to/app/root && ...' for each job.

SHELL="/bin/bash"

Set the default shell to execute your commands from with the SHELL command. Like PATH, a safe bet is making this the same as your user's shell. Logged in as that user, run which `echo $0` to get an absolute path to your shell.

Still Stumped? Dump Your Environment

It's easy to pipe cron's environment variables to a temporary file, as seen on Stack Overflow. Tweak the following to run from your crontab in the near future:

30 08 * * * env > /tmp/cronenv

Once it runs, cat /tmp/cronenv to see your crontab's environment variables. Alternatively, if you're getting cron error messages e-mailed to you after setting up MAILBOX above, look at the raw headers of the e-mail. E-mails from cron should set X-CRON-ENV headers with these environment variables, too.

Happy automating!

I've started thinking about my next iteration of this system for Didsum, and using pushing for more than just deployment purposes. The Push-to-______ pattern is powerful and easy to use, once you know how the pieces fit together. In this post, I'll walk through the setting up Push-to-Deploy from the ground up. 

(I'm assuming a working knowledge of: terminal, git, and a scripting language.)

Preparing our Repositories

Let's keep things straightforward by placing our development git repository, remote git repository, and deploy directory under the same local directory.

mkdir push-to-deploy
cd push-to-deploy
mkdir {development,remote,deploy}

Awesome, now let's setup the remote git repository. With a real project, this would be a directory on your production server. There is a special setup for git repositories whose purpose is to receive pushes from developers, they're called "bare" repositories. You can read more about "bare" repositories, but for our purposes their purpose is just to receive pushes.

cd remote
git init --bare
cd ..

Perfect, now let's setup our development directory with git. (You could also copy one of your git project's contents into the development folder.)

cd development
git init
echo "Hello, world." >> file.txt
git add file.txt
git commit -m 'First commit.'

We now have a development repository with its first commit. Our last preparation step is to register the "remote" repository. If you're fuzzy on git remotes, the official git site has you covered.

git remote add production ../remote
git push production master

Boom. You've just pushed your commit from development to your bare, 'remote' repository. We're ready to setup push-to-deploy.

Set up Push-to-Deploy

Now that our remote repository is setup, we're ready to write a script for what it'll do when it receives a push. Let's navigate to the hooks folder of our remote repository. Hooks are scripts that git runs when certain events happen.

cd ../remote/hooks
touch post-receive
chmod +x post-receive

The hook we care about for push-to-deploy is post-receive. It is run after receiving and accepting a push of commits. In the commands above, we're creating the post-receive script file with `touch`, and making sure it's an executable file.

Next, open the post-receive script file in your preferred text editor. Copy the contents below:

#!/usr/bin/env ruby
# post-receive

# 1. Read STDIN (Format: "from_commit to_commit branch_name")
from, to, branch = ARGF.read.split " "

# 2. Only deploy if master branch was pushed
if (branch =~ /master$/) == nil
    puts "Received branch #{branch}, not deploying."
    exit
end

# 3. Copy files to deploy directory
deploy_to_dir = File.expand_path('../deploy')
`GIT_WORK_TREE="#{deploy_to_dir}" git checkout -f master`
puts "DEPLOY: master(#{to}) copied to '#{deploy_to_dir}'"

# 4.TODO: Deployment Tasks
# i.e.: Run Puppet Apply, Restart Daemons, etc

Let's walk through each of the steps:

1. When git runs post-receive, the data about what was received is provided to the script via STDIN. It contains three arguments separated by spaces: the previous HEAD commit ID, the new HEAD commit ID, and the name of the branch being pushed to. We're reading these values and assigning them to from, to, and branch variables, respectively.

2. Our purpose here is to automate push-to-deploy. Assuming a workflow that keeps production on the master branch, we want to exit this script prior to deploying if the branch being pushed is not master.

3. The first deploy step is to "checkout", basically export or copy, files from the master branch to the directory where our project is deployed to in production. (Remember, in this demo it's the fake "deploy" directory, in the real world this might be /var/www, or wherever your project expects to be in production.)

4. Now that our deploy directory is up-to-date, we can run whatever deployment tasks we need to run. This could be applying Puppet scripts (I'll write a post on this scenario soon), restarting a web or application server, clearing cache files, recompiling static assets, etc. Whatever steps you'd normally need to do manually after updating your project's files, automate them here!

Save your post-receive hook, and let's test it out!

Testing with Pushing

We can test our script manually, by creating a new commit in our development directory and pushing:

cd ../../development
echo "New line." >> file.txt
git add file.txt
git commit -m 'Testing push-to-deploy'
git push production master

In the output of the git push command, you should see lines starting with "remote:". These lines are the output of our post-receive script:

Already on 'master'
DEPLOY: master($TO_ID) copied to 'push-to-deploy/deploy'

The first line is noisy output from the git checkout command in step 3, we can ignore it. The second line, is from the puts command, also from step 3 in our post-receive script.

The directory we're deploying to should now be populated and up-to-date:

ls ../deploy
diff file.txt ../deploy/file.txt

Pretty awesome, right?

Testing without Pushing

When you're working on a post-receive hook, it's annoying to muck up your project's commit history and push each time you make a change. Luckily, because it's just a script, we can fake it from the command-line.

cd ../remote
git log -2 --format=oneline --reverse

First, we need to get the IDs of our most recent 2 commits. The git log command, above, will give us these two IDs in the order you'll want to replace the $FROM_ID and $TO_ID variables with, respectively.

echo "$FROM_ID $TO_ID master" | ./hooks/post-receive

This method makes setting up your post-receive hooks enjoyable, enabling you to quickly iterate on your script and execute it repeatedly.

Next Steps

In this post, we've walked through how to setup the push-to-deploy with git. For a real world project, your 'remote' and 'deploy' folders would usually be setup on a server, not locally. The details of doing that and properly configuring SSH is beyond the post of this scope (note to self: I should write on SSH configuration, too!).

From here, it's up to your project to determine what actions to automate! Happy pushing!

(If the push-to-X pattern interests you, you should follow me on Twitter, or subscribe to my RSS feed, because I've got a few more posts on the subject coming up! Also, feel free to tweet feedback / questions / topics you'd like to read.)

I started running back in June. In the few years past, I haven't been too active. All hack, no play.

I wanted to get in shape, but I also wanted to test out the group tracking feature of an app I've been working on, Didsum. (It's getting close to being open to the public; until then you should sign-up for an invite from that link!) It's been a lot of fun tracking with friends, family, and coworkers. I don't think I would have kept it up without it!

While on my run tonight I thought it would be cool to review my running data from Didsum and see how I've been doing.

41 Runs, 28 #withdog

Since June I've gone on 41 runs. If we toss out the month of June, it's 37 runs. June was a weak month because I was carb-loading in Italy for a couple weeks.  I've taken my dog, Krackle, on 70% of those runs. She loves it.

1 Run Every 3 Days

Over this time period, throwing out an outlier while on vacation, I'm running once every three days on average. There's a good bit of variance here, which I want to work on. My rough goal has been to run every other day. For a week I was on a Hal Higdon 5k training schedule, which bumped me to 5 days a week. I'm pretty pleased with this average, but want to avoid those gaps of 5+ days between runs. I did that 8 times this period.

97 Miles

Being 3 miles shy of 100mi makes me wish I'd waited for one more run! It's pretty cool to see these miles add up. My first 4 runs, recorded in June, were 1.2, 1.6, 1.7, and 1.8 miles long. They all felt like death. I guess runs still feel like that, but I'm running (a little) further and faster now.

My longest run was 4.5 miles at Oak Island. It was my only point-to-point run and was a lot of fun. I ran as far as I could until meeting my friend, Bryan Deaton, who was driving in to the house I departed from. I nerd sniped him with the calculus problem of how far I'd be able to run before he picked me up in his car. He somehow got it right.

The following chart shows a 7-run trailing average plot of distance per run to smooth out the noise. It's a neat graph to see go up, but I need to step it back up! I'm liking the 3 mile range, but I wish I had a better loop around my neighborhood for it. Note to self: find one.

28% Faster

It's crazy how quickly you can pick up your pace when you're getting started with running. On my third run, I ran 1.8 miles in 21 minutes, for a pace of 12:00/mi. On Monday, I ran the same roads for 2.0 miles in 17.5 minutes, for a pace of 8:45/mi. (It might just be the new running shoes I bought on Monday!)

The following pace chart is also a 7-run trailing average.

If you flip between my distance and pace charts, you can see I slowed down as distance went up, but once it came back down I was moving faster than before.

First 5k Race: 26m 48s, 8:39/mi

Thanks to some friends (Thanks Annie and Nathan... and Carmen!), I had the opportunity to run my first 5k race on a whim ahead of training schedule. It was the Chapel Hill Cleft Palate Gallup and it went pretty well: I finished in 6th place amongst females aged 30-39th under Carmen Stanco's bib!

Self-assigned Grade: B-

"80% of success is showing up."

Nice job overall, better than I would have projected myself to do in June getting started. I need to work on consistency and try to shoot for 3 miles per run. 

Goals for the next Running Report Card:

• No periods of 5 or more days between runs
• 100 Miles in less than 35 runs / average 3 miles per run
• Finish averaging an 8:30min/mi pace
• Find a good 3 mile loop
• Read Natural Running cover-to-cover, improve form
• Get my wife to join me on at least 1 run per week*.

*This post edited by Kris' wife. 

Earlier this month, probably after seeing some neat post on Hacker News, I bought an Arduino Starter Kit on a whim. My mission: to hack up a pour over coffee machine (or fail trying).

Why? Well, partly because I love coffee. Especially a good cup made via pour over.

The truth, though, is that I've always been secretly jealous of the kids who broke open electronics and hacked them to do sweet shit. I've been firmly anchored in software¹ since since a middle school buddy gave me a bootleg copy of Visual Basic 6. (Writing code was love at first sight, even despite the ugliness of VB6.) For me, Arduino offers an approachable way to tinker with electronics and hardware, while still having code as a crutch.

For the sake of reference, here's the Arduino Uno board that comes with the kit:

First: What is Pour Over Coffee?

It's a simple, old method of brewing coffee that is making a revival at boutique (hipster) coffee shops. The pour over method produces a single cup of coffee brewed deliciously fresh.

Here's roughly how pour over coffee works, step by step:

1. Water is boiled to 192°F-200°F. This is important. (Most drip machines fail to breach 180°F.) 
2. Coffee is measured and ground fresh, with a burr grinder.
3. The filter is placed in the dripper (the white ceramic funnel).
4. The grounds go in the filter and are preheated by pouring just enough water to soak. Wait 30 seconds.
5. Using a gooseneck kettle, water is poured onto the grounds with a growing spiral starting from center. Add more water as necessary. It should take around 2 minutes to brew.

That's it! So, let's see how much of this process can be automated with little electrical knowledge, an Arduino, and some hacking.

Problem #1) Sourcing a Boiler

Boiling water to right around 195°F, is step one, so it's the first problem I needed to tackle. Aubrey and I have collected a number of coffee machines over the years. My plan: salvage parts from the most promising of them.

I pulled our first coffee machine from the attic, a nice looking, red KitchenAid and set it up. I filled its tank, and set it to brew, without any coffee, to test its water temperature.

It started pumping water into the grounds filter almost immediately, with lukewarm water. Gross. The hottest reading I could get it to reach was 160°F. No wonder this machine made coffee that tasted like wet garbage. The good news is, I wouldn't need to clean the filth left in it from the abuse it endured at NMC's office back in the 605 building days.

Next up: the Keurig. I had seen an Arduino Keurig hack before, and it had a nice looking pump system I thought would also be handy. Unfortunately, the warmest reading from the Keurig I could get was 180°F. It wasn't a completely fair test, I couldn't read directly from the spout without the machine refusing to pump water, but this isn't science, it's hacking. 

For some reason, I had mentally blocked out our Hamilton Beach electric kettle as a possible solution. I guess because it lacked pumps. It could certainly get water to boiling, though.

200°F! Hell yes!

The other cool thing about it, I had forgotten: it has a built-in temperature display. I actually bought it for this feature. Most electric kettles lack this.

New Theory: If I can read the temperature sensor and control turning on/off the heating element, I've got everything I need to boil water to 195°F!

So I opened up the Kettle's "Docking Station". This was annoying because of the cockamamie Tri-Wing(C) screws they used, I assume, to prevent people like me from from opening it up. "Ah, screw it²," I said, forcing a Phillips Head with some pressure to get the bolts out (and strip the heck out of them).

Boom! We're in!

I spent some time looking at the two boards. (Ok, an embarrassingly long time. I'm coming into this project extremely green with electronics.) My conclusion was, "this is really encouraging".

The board attached to the base, with thick wires going to the heating element's connection, seemed to be the "muscle" of the kettle. The breakout board, attached by 5 jumper wires, contained the buttons for control, the LED, and what looked to be its "brains", a microcontroller, on the backside. If I was correct, it meant I should be able to wire up those jumper cables to the Arduino, and have full control over the kettle and read its thermal sensor!

I tried tracing the circuit on the backside of the breakout board to figure out the purpose of the wires, but realized I was slowly getting nowhere. So I stopped and ordered a cheap multimeter to be able to read the voltages of the wires while it was in operation. I had my fingers crossed the breakout board would be operating within the 0-5V range the Arduino does. That'd be great.

Being physically "blocked", because you're missing a tool or a part you can't get today, is a big change from equivalently simple software learning. The upside: it forces you to stop and go to bed.

In the next post, I reverse engineer the Kettle's jumper wires' purposes.

1. This isn't entirely true. During an awesome Freshman Seminar course, "Robotics with Legos" taught by Prof. Gary Bishop (one of UNC CS's finest), I played with hardware at a  simple level. We never worried about voltages or circuits, Legos abstracted those details away. Here's a fun picture of Joel Sutherland and I in that course, with our final project. Even in this project I avoided most of the mechanical and stayed in the comfortable world of quasi-C code. This is the first technical project we teamed up on, and dates back to December of 2003 (crazy):
    
   Our robot was S³ - Super Skittle Sweeper. It used 2 Brick CPUs, one was the "muscle" that powered the motors and took actions, the other was the "brain" which read the sensors sent actions to the muscle brick. Fun little coincidence to see this same concept at play in the electric kettle.
    
2. Second post in a row with a truly awful pun. Pretty soon this will be a streak.

Patrick,

Nice work getting through the section on looping today! Being "a bit shaky with them" is nothing to worry about. The details about loops are hard to remember, especially from tutorials that force feed you details too early.

Covering the full power of loops in a single tutorial is kind of like trying to teach someone new to baseball all of the grips and releases a pitcher could use. Until you need to throw a curve ball, knowing exactly where to hold your fingers around the stitching and how to release it isn't a big deal. 

Once you're writing a program to do something you need it to do, you'll probably hit points where you think "I need to do something a bunch of times" or "I have a bunch of data, I need to do something with every piece of it." When you find yourself thinking that, loops are a powerful tool at your disposal¹.

Once you have a need, and can jot down a basic outline of your strategy for solving it (in English), you can search Google for "types of loops" to decide which kind of loop to use or its specific syntax. Test and tweak until you get it right.

Common Purposes of Loops

After you've used loops for a while², some patterns will start to emerge in how you use them to help you with a particular kinds of tasks. These tasks are not worth trying to commit to memory, but they're common enough that being aware of them should be useful. All of these examples will use for loops over arrays³.

Do something for each element

var numbers = [1, 2, 3, 4, 5];
// For each number in the `numbers` array:
// log it to console
for(var i = 0; i < numbers.length; i++) {
  console.log(numbers[i]);
}

This was the opening demo of your Codecademy tutorial. It loops through each number in the numbers array and takes an action with it. In this example, the action is to print it to console. The action could have been alert(numbers[i]); to get it to show an alert dialog for each number. There will be times when you just need to take an action with every element in an array and this simple pattern will have you covered.

"Change" elements of an array

var numbers = [1, 2, 3, 4, 5];
var numbers_doubled = [];
// For each number in numbers:
// double it and add it to the numbers_doubled array
for(var i = 0; i < numbers.length; i++) {
  numbers_doubled.push(numbers[i] * 2);
}

In this demo, our goal is to double an array of numbers. The doubling isn't important, what is important is that after we're done we have a new array whose elements are "mapped" from the elements of the original array. In this case, that mapping is to double the element. Maybe your array is made up of investments and you want apply interest on each of them.

Note: We could have changed the elements of original numbers array in place, without using the separate numbers_doubled array, and that would have been fine. One upside of creating a new array is that we still have a way to access our original data later in the program. Another upside is that this program will be easier to debug.

Filter elements of an array

var numbers = [1, 2, 3, 4, 5];
var odds = [];
// For each number in numbers:
// if it is odd, add it to the odds array
for(var i = 0; i < numbers.length; i++) {
  if(numbers[i] % 2 == 1) { 
    odds.push(numbers[i]);
  } 
}

In this demo, our goal is to pick out all of the odd numbers in the numbers array and place them in the odds array.

Note: If you haven't seen the % modulo syntax before, it's basically "integer division remainder", i.e. if you divide 3/2 you get a remainder of 1.

When you have a dataset, but only want to focus on a particular chunk of it, you can filter out exactly the data you want using a loop to populate a new, filtered dataset. Creating a new variable to copy the elements you do want into a new array is usually simpler than removing the items you don't want.

Summarize elements of an array

var numbers = [1, 2, 3, 4, 5];
var sum = 0;
// For each number in numbers:
// add it to the sum variable
for(var i = 0; i < numbers.length; i++) {
  sum = sum + numbers[i];
}

In this example, we're adding up all the numbers. We could have done other things, too, like averaged the numbers. The important takeaway is that we initialized a new variable, sum, to hold the results of our computation. Once we've looped through the array, and reduced it down to a single value, we now have their sum.

For a lot of common mathematical summaries (like sum, product, average, stddev, etc.) there will be libraries you can use so you do not have to write these computations by hand. When you find yourself wanting to summarize in your own special way, this is a useful pattern to come back to.

Conclusion

Don't worry about the details of loops until you find yourself needing to do something repeatedly. The important thing to know is that they exist. If you can frame your problem as one or more of the different common types of loops I demoed above: taking repeated actions, modifying a bunch of values, filtering out values, or summarizing values, then you can use those patterns as building blocks.

Once you're more comfortable with functions in JavaScript, it'll be fun to revisit these looping concepts with the full power of functions at your command. Once you've got functions down we could write some that allow you to avoid the hassle of writing loops altogether. Let's save that glory for a rainy day.

Best,

Kris 

1. Loops aren't the only way to do something a number of times or to operate on sets of data, though...
2. What an awful pun.
3. It's somewhat rare to write real programs with loops whose bounds are hard-coded numbers. Looping through arrays is much more common and useful.

Since we moved into our space on Main Street in Carrboro, now nearly two years ago, we've talked about doing window art. Joel put the pieces in place to make it happen this past Friday for Halloween. Lenny Terenzi dropped in to paint life-sized versions of his awesome Lunchbox Monsters. They turned out great.

Making the Time Lapse

We decided to try and shoot a time lapse of the windows being painted. I'd never shot or created a time lapse before, so it was a fun challenge. There are certainly better, more professional ways to put together a time lapse, but for the casual photographer who wants to put one together without super specialized software, here are few tips.

Stay Warm

If the temperature is anywhere below comfortable outside, don't underestimate how hard it is to stay warm while sitting still for a few hours. Luckily NMC had an extension cord and a space heater which helped.

Canon EOS Utility

Canon cameras come bundled with a program called EOS Utility. It allows you to control your camera's settings over USB and shoot photos directly to your computer. It also has a timed shutter release feature that works well for doing time lapses. Its only limitation is you can't use an interval less than 5 seconds. EOS utility worked well enough for our purposes.

Image Quality Settings

I knew there would be a large number of frames, which would consume space fast in my 5dII's RAW format. The time lapse wound up at 2300 frames. If I shot in RAW, that'd be upwards of 50gb of data. I dialed down to the smallest resolution, high quality JPG mode. In retrospect, if I knew I would be able to consistently crop all of the shots of the same composition I would have probably gone to the middle resolution, high quality JPG mode. One composition wound up needing to be cropped below the video's 1080p resolution.

Lightroom Bulk Editing with Sync Settings

If you ever need to apply the same edits to a series of pictures, Lightroom's "Sync Settings" feature is a game changer. I didn't even know it existed prior to doing the shoot. I wish I had. It's like the Copy/Paste Settings functionality for an entire selection of photos. While in the Develop tab, select the batch of photos you want to edit, then focus on the photo with the correct settings, then go to Settings > Sync Settings.

The Magic is in Post-processing

Armed with Lightroom, you can turn average source photos into a pretty good looking time lapse. Because the time lapse already feels disconnected from reality, you can be more aggressive with post-process editing than you might otherwise be with stills. At least, that was the theory I operated under. It's fairly rare to see a photographer's before and after post pictures, so I thought it'd be fun to share a few frames. If you don't like knowing how the sausage is made, look away!

Original

Post-processed

Original

Post-processed

Original

Post-processed

Original

Post-processed

Original

Post-processed

iMovie Time Lapse Tips

iMovie isn't really made to do time lapse or stop motion videos, but it can be done. Dragging in 2,000 images makes iMovie cry and beach ball. Be patient. Once your shots are on the movie's timeline, select all to change the crop away from the Ken Burns Effect Crop and to reduce the duration of the frame. iMovie's lower limit for frame durations is 0.1s. That's OK, because the best way to play with timing is to export this project as a 1080p movie and then reimport that movie into iMovie for the real timing and editing. Trying to do timing/editing with the individual frames would be a world of pain.

Happy Halloween, Folks!

I was pleasantly surprised to get asked this after releasing Wiretap's first demo video. (Embedded below.)

Good demos are hard.

It’s hard not to get lost in features and technical details. It’s hard to give enough context, but not too much. It’s hard to keep a good pace.

I was really afraid of falling into these traps. Wiretap is an HTTP debugger for mobile and web apps integrating with REST APIs. By default, these are the ingredients of a snoozer video.

Here’s how Wiretap’s demo avoided putting (most) people to sleep and some other fun, behind-the-scenes details.

Trust the ole Inverted Pyramid

Put your most important messages in the beginning. It’s all most people will see.

If someone stops watching after 30 seconds, will they know what they need to know?

News writers popularized the idea of an Inverted Pyramid, or, “Bottom Line Up Front”. The idea is readers should be able to leave a story halfway through and have a good enough understanding of it.

• Who is this product for?
• Why does it exist?
• What does it do?
• How is it used?
• When is it useful?

The intro to Wiretap’s demo was something I struggled with. Luckily I got some good feedback on the first take from a friend. More on that in the next section.

In the body of a demo: tell a story.

Walk through scenarios that motivate features. Don’t give a feature tour. Reuse the scenarios you came up with to guide your app’s design and implementation. Heck, maybe even copy them straight from your BDD tests, if that’s your bag.

Don’t expect someone to watch your whole demo, but give them reasons to.

Show Friends, Act on Feedback

After hacking on a project for a while it’s really hard to describe it in a demo context-free.

You need fresh eyes.

Show someone who could be a user of your product but doesn’t know anything about it yet. Take action on their feedback.

The feedback I got on the first version of my demo was:

“The intro was missing the why? Why did you build this? Why is it useful?”

He was right.

(You can watch the original with password 'wiretap'.)

I jumped straight into using the product. So I scratched it, recorded new audio, and rearranged the video clips to match. It wasn’t much work and the improvement was substantial. (It could be better yet: the current intro takes 50 seconds. Tsk, tsk.)

Use Bargain Tools that are Easy

Great recording software doesn’t make a great demo, but it can save you a lot of time. Since folks are always curious about tooling, here’s what I used.

Video Capture - QuickTime / iShowU HD - Yep, the QuickTime Player installed on every OSX can make screen recordings. Just go to File > New Screen Recording. You can record the screen or a specific bounded rectangle. It’s a little clunky, though, and lacks some nice features in commercial software like scaling and a better workflow. I’ve had an iShowU HD license ($30 and worth it) for a few years and used it for the Wiretap demo.

Audio Recording - Audacity - Awesome, free, open-source software for recording a narration track. This was the first time I wrote a script for a demo and recorded the video and audio independently. It allowed me to crop out places where I misspoke or had long pauses. This helped the pace of the demo. The downside was that the edits and scripting were noticeable at times, but it was good enough.

Post-production - iMovie - iMovie makes chopping up screen recordings and aligning them with an audio track easy. With software demos you can speed-up and slow-down screen captures to align events with the narration. I abused this freedom. In iMovie just go to ‘Clip Adjustments’, look for the speed section, convert the video, and enter the desired duration of your clip.

Have Fun, Keep it Light

Sprinkling in some spice won’t take away from your meat and potatoes.

With Wiretap I wanted to walk through a complete Mobile App + Server-side API + 3rd Party API scenario. Doing something that was the “AirBnB of ____” seemed pretty obvious and my dog was laying under my desk looking guilty at the time so I ran with “ShameKennelRnR”. I dropped in a few little easter eggs, as well.

Be excited to show off what you’ve been hacking on and have fun with it.

Shameless plug: Do you hack on REST API integrations or implementations? Wiretap is entering private beta soon. I’d love your help testing it. Sign-up over on Wiretap's demo page.

The Marco Polo Method

SSH in to your new server. You know you need a web server and Ruby or PHP so you sudo apt-get install apache php, or whatever. You copy over an Apache configuration file and enable it as a site. Restart Apache. Marco: directory does not exist. Polo: mkdir /www/awesome-app. Restart Apache. Copy files in. Open up browser. Marco: function pdo mysql driver does not exist. Polo: sudo apt-get install php-pdo-mysql (or whatever). Refresh. Marco: cannot connect to localhost, connection refused. Whoops, forgot to install MySQL. Polo: sudo apt-get install mysql. Eventually, hopefully, it all works.

Deploy by error-Google-fix loop. Everyone has been down this road before.  The Marco Polo setup is painful, error prone, and time consuming. It's a necessary evil when you are new to configuring systems or rarely have to do it. The skills you pick up here, though, are fundamental to being successful in evolving to shell scripts and beyond.

The Scripted Method

At some point it becomes painfully obvious setting up a system is just running a series of shell commands. What better tool to run a series of commands than a shell script? None, right?

Scripting a complete system setup turns out to be painful, too. Scripts are fragile. All it takes is one remote server being down in the middle of a script and dependent steps will fail. So you try running the script again and the steps that worked are rerun. Source downloads again, recompiles happen, directories already exist, and so on. So you open up the script and either comment out the steps that worked or start wrapping steps in conditionals checking for criteria to be met.

Shell scripts are an imperative approach to mutating a system's state. Your commands describe how to reach a desired state, not what the desired state actually is. By describing steps and not final state the scripted method's sophistication and robustness are a product of time spent crafting the script and an author's ability. In practice, this often leads to scripted setups which are not idempotent; running setup scripts multiple times doesn't generally work without thought and effort. This becomes more troublesome as system setups evolve over the lifetime of a product. Commit logs become riddled with comments like "must run setup script 23-install-redis.sh to run and pass tests". We must be able to abstract away these pains, right?

The Declarative Method

Our ideal solution, thus, is automated, declarative, and idempotent. We need a language for describing a system's desired state and a repeatable means for the system to reach that state. Enter Puppet (and other sysadmin automation technologies like Chef).

Puppet is just that: a language for describing a system's desired state and software that can, in essence, diff your system's current state with your desired state and determine the series commands that need to be applied in order to reach that state. It was initially designed for sysadmins managing groups of servers. It's valuable for small custom development projects and single-server deploys, too. For the purposes of this article, we'll only demonstrate its single machine facilities.

Puppet exists in most package manager repositories. On Ubuntu, for example, sudo apt-get install puppet will get you there.

Once Puppet is installed all we need to do is setup a manifest file and run puppet apply. A .pp manifest file uses Puppet's domain-specific language for describing our system's desired state. Let's walk through a simple static HTML nginx server system manifest in Puppet.

# demo.pp
package { "nginx":
    ensure => installed
}

service { "nginx":
    require => Package["nginx"],
    ensure => running,
    enable => true
}

Our demo.pp manifest file opens by declaring the 'nginx' package should be installed and setup as a service. The service requires the package as a dependency and is ensured to be in a running state. Also, it is 'enabled' so that it will start automatically on system boot up.

We can save and apply this puppet manifest file by running sudo puppet apply demo.pp and testing with curl localhost. You should see a dump of nginx's default HTML message. Let's kill the default nginx site with our puppet manifest.

# ...
# demo.pp - part 2 
file { "/etc/nginx/sites-enabled/default":
    require => Package["nginx"],
    ensure  => absent,
    notify  => Service["nginx"]
}

The symbolic link enabling the nginx default site will be made absent by Puppet. If reaching this state requires action, as in the symlink being deleted, Puppet will notify the nginx service to reload its configuration automatically. Run sudo puppet apply demo.pp && curl localhost and it will report nginx is no longer listening; no site is configured. Try running sudo puppet apply demo.pp once more. No actions are taken because the system is already in the desired state. Cool, huh?

Let's setup a new site with a static HTML index file. We'll throw the site in the /www directory.

# ...
# demo.pp - part 3
file { "/www":
    ensure => "directory"
}
file { "/www/index.html":
    require => File["/www"],
    ensure => "file",
    content => "<!DOCTYPE html>
        <html><body>
        Hello, world.
        "
}

Run sudo puppet apply demo.pp && ls /www once more to see that our latest additions to the manifest file are setting up the /www directory and then, once it is ready, ensuring the file index.html is placed in the directory with specific contents. Our last step is setting up the nginx configuration for the site in our demo.pp manifest file.

# ...
# demo.pp - part 4
file { "/etc/nginx/sites-available/puppet-demo":
    require => [
        Package["nginx"],
        File["/www"]
    ],
    ensure => "file",
    content => 
        "server {
            listen 80 default_server;
            server_name _;
            location / { root /www; }
        }",
    notify => Service["nginx"]
}
file { "/etc/nginx/sites-enabled/puppet-demo":
    require => File["/etc/nginx/sites-available/puppet-demo"],
    ensure => "link",
    target => "/etc/nginx/sites-available/puppet-demo",
    notify => Service["nginx"]
}

We've declared an nginx configuration file for our site should be in sites-available and that a symbolic link pointing to that configuration file be placed in sites-enabled to turn the site on. If changes are made to either of these resources nginx will automatically reload its configuration. Run sudo puppet apply demo.pp && curl localhost and our wonderful hello world HTML file is displayed.

Let's have a little fun showing off the value of Puppet's declarative method. Try turning nginx off with sudo service nginx stop. Rerun sudo puppet apply demo.pp. "Service[nginx] changed from 'stopped' to 'running'". It compared the desired system state, where nginx should be running, with the current system state, and made only the change necessary. We could delete the /www directory, uninstall nginx, and so on, and a simple run of puppet apply will bring us back to our desired state. Same is true on a brand new server.

To be suitable for a gentle introduction to Puppet I have avoided discussing any detail of its language. It is robust. It has variables, units of abstraction and organization, the ability to specify file content with ERB templates, and a great deal more capability than what you have seen here. The practice of putting an entire system definition in a single manifest file is a practice you will quickly outgrow; it is great for learning, though.

Stop doing the work. Start describing the target.

Systems can be setup using a variety of methods. From manually pounding away in a shell, to automating repeated installs with shell scripts, to declaring a desired system state in Puppet and letting it figure out what, if anything, needs to be done to get you there. Automating system setups with a declarative configuration tool like Puppet or Chef will change the way you approach both sysops/syadmin work as well as development. With your Puppet manifests stored alongside your projects in source control it is easier to evolve your system setup and your code without getting lost in configuration hell.

Puppet may have been designed for the sysops folks managing farms of servers, but it is also an incredibly valuable technology for smaller teams, single-server projects, and solo devs, who want to abstract away the pains of maintaining system configurations.

"I believe functions are the smallest unit of functionality"

The missing tenet from the Micro PHP Manifesto is: I believe functions are the smallest unit of functionality.

Not classes. Not methods. Functions. This belief fits in naturally with, and is a precursor to, most every other tenet of the manifesto.

"I like building small things with simple purposes." There is no smaller or simpler reusable unit of functionality than a function.

"I like building small things that work together to solve larger problems." Functions and their higher-order friends compose with mathematical simplicity and beauty.

"I want code that is easily verifiable." Pure functions are the easiest unit of functionality to test.

"I need to justify every piece of code I add to a project." Do we really need to subclass Controller to add a route to our app? Nope, not if you're using Slim, a great HTTP micro framework (full disclosure [1], my office mate wrote it). An anonymous function will do just fine, thank you. Senseless subclassing is what got us into this mess to begin with! [2]

When the traditional PHP framework programmer looks at a problem, he asks himself these questions, in order of preference:

1. Can I add a method to an existing class?
2. Can I add a subclass and override a method?
3. Or should I start a new base class?

When the Micro PHP framework programmer looks at a problem, she asks herself these questions:

1. Can I use an anonymous function?
2. Can I use a named function?
3. Can I add a method to an existing class?
4. Can I add a subclass and override a method?
5. Or should I start a new base class?

Namespaced functions and function values being legitimate options in 5.3 enables lighter weight libraries that play together nicely. Combined with jQuery pushing web developers' collective comfort with a more functional style of programming, the micro PHP movement is in a great position to take off.

But, for the love of Rasmus, PHP has avoided solving two fundamental problems that would radically inspire even smaller, even simpler, even more single-purposed, micro PHP libraries.

Damnit PHP, why can't you just...

1. Allow functions to be imported like classes

User defined functions are second rate compared to user defined classes in the PHP language. You can import classes from namespaces but you can't import functions. Read that again. You can import namespaced classes but not namespaced functions. PHP itself doesn't fully believe that "functions are the smallest unit of functionality". Oh, sweet irony. Most of PHP's core functionality (even array manipulation!) is exposed as functions (in a global namespace).

PHP discourages writing plain old functions by making them more painful to reuse than classes. Currently, the best way to write useable "plain old functions" is to write methods on a class you'll only ever invoke statically (i.e., underscore.php's __::each). If PHP respected functions they wouldn't need to be wrapped in bastardized classes to be useful.

2. Indoctrinate a symbol loading algorithm

So it isn't possible to import namespaced functions directly. But it gets worse. Even using what is available (use namespace; namespace\func();), if the function has not yet been defined, there is no way to autoload that function's source like PHP will do for classes. But it gets worse. The class loading algorithm is user defined. This is another historical impediment to "micro" component compatibility in PHP, because every framework has its own class loader [3]. There is hope in PSR-0, but it won't ship with 5.4.

PHP needs to plant a hard foot in the ground and indoctrinate one built-in, automatic symbol loading algorithm that will load classes and functions and constants. The chaos has gone on long enough. If Python and node.js can be bold enough to do it, so can PHP. The flexibility has stunted our growth.

Please, if you are listening PHP Core Team, for the love of the fat blue elephant in the room, do whatever it takes and break whatever you need to get these fundamental language utilities in the next major point release. Or even the mythical 6.0. The temporary pain will be worth the massive, revitalizing shock it will give to a community of PHP developers who would like to share components and code in a simple and standard way.

How should micro libraries share plain functions until PHP fixes importing and symbol loading?

Step 1. Use a PSR-0 class loader There are plenty. The Symfony2 ClassLoader's is good and PSR-0 compliant. It's MIT. Potencier writes great code. Use it.

Step 2. Write plain-old functions as static methods on a bastard, wrapper class. Ugh. Step 2 is the worst, but distributing plain-old functions puts consumers back in require_once hell. If all you are trying to write and share are plain-old functions, use static methods.

Step 3. Write to your local PHP Core politician. Demand importable functions and a native, standard symbol loader for PHP.next. Support this PHP RFC for function autoloading, and this one for PSR-0 inclusion.

Finally, will an isaacs please step up? PHP needs a modern package manager.

If someone really wants to blow oxygen on the Micro PHP fire they should start with a new package manager for PHP. PEAR is far too bureaucratic and, frankly, embarrassing in the face of a modern package manager like node.js's npm. PEAR has been around since before PHP5's release in 2004 and there are 179 modules hosted at pear.php.net for 5.0+. (And only 19 for poor PEAR2 and PHP 5.3) Contrast with npm which has been around since August 2010 and hosts 6,157 packages.

npm hosts over 30 times more packages than PEAR!?! How? You don't have to pass a bureaucratic, written proposal process to publish to npm. Anyone can publish any package they want in under 10 minutes. So do all npm packages have the high quality of PEAR packages? No. Do they all follow the same coding standards like PEAR packages? No. Are these real problems to fear in practice? No. The best packages are naturally high quality and obtain the most attention and dependencies. PEAR is selection by committee, npm is natural selection. The Micro PHP movement really needs a package manager with fewer, smaller rules for publishing and sharing. PEAR may be for sharing small packages with simple purposes, but it is certainly not Micro PHP.

(Update: some promising movements in this direction are Packagist and Phark.)

tl;dr

Functions are the smallest unit of functionality. This is a key missing tenet of the Micro PHP Manifesto. It's a practical precursor to many of the others. This isn't to say object-oriented libraries and designs don't have their place in PHP, they most certainly do, but that if you can solve your design problem with functions, choose functions. Prefer small, simple functions over complex, stateful class hierarchies.

Footnotes

[1] Even fuller disclosure: I've written a "Full Stack" PHP Framework. A lot of what Ed said is wrong with Zend and Symfony was wrong with Recess, which I wrote three years ago. It's intention was to be light weight. In terms of CLOC, it was, under 10k for a RESTful HTTP layer, ORM, "Views", & Annotations. In terms of code written by end-users, it was, comparatively, too. Recess was the first PHP framework to use Annotations. Declarative-style programming requires less code. I don't use it for new projects anymore. It's still too heavy. It's not line counts or ugly code, though. It caries the PHP 5.2 OOP-only baggage, too [2].

[2] Ed's post stops short of tracing the legacy leading to the current state of popular frameworks being un-micro. Here's the story. When all you know are objects...all you see are classes. And methods. And private variables. And design patterns. And Java. When all you know is Java all you see is Java.

PHP did not become be one of the most popular kids in school without having a "hey look, me too" mindset. PHP grew its popularity in the early 2000s by copying everything it could from Java's object model in the move from PHP4 to PHP5. From visibility controls to reflection.

When your language is inspired by Java, your libraries and frameworks will be too. You can't look at Recess' annotations without smelling Java. You can't look at Doctrine without smelling Java. You can't look at Symfony components without smelling Java. This isn't to say Symfony is bad, it's not. It's really damned good, but it's manifesto is different. It's manifesto was written by a Gang of Four. It's lineage predates PHP 5.3. Popular frameworks and libraries originating before PHP 5.3's release carry the burden of pursuing best practice object-oriented patterns at the cost of ignoring perfect opportunities where standalone or anonymous functions would suffice.

[3] PEAR/Zend-style class naming and directory structuring became somewhat of a norm before 5.3, but it sucked. Class names were combined with namespaces which lead to instantiating classes with names like Zend_Db_Table_Definition. Only in recent history, since 5.3 namespaces, has the community embraced a "standard" class loading algorithm with PSR-0. (Design by committee retained PEAR-style support, which is unfortunate, but I digress.) In theory this will make its way into the Standard PHP Library and be bundled with PHP, which will be great.

The quickest way to understand is by looking at a simple, familiar example implemented with a multimethod. So let's write a recursive Fibonacci function.

> var fib = multimethod()
                .when( 0, 0 )
                .when( 1, 1 )
                .default( function(n) {
                    return fib(n-1) + fib(n-2);
                });
> fib(20);
6765

Multimethods are a functional programming "control structure" enabling you to dynamically build-up and manipulate the dispatching behavior of a polymorphic function. Inspired by Clojure's multimethods, multimethod.js provides a functional alternative to classical, prototype based polymorphism. The multimethod.js library is MIT licensed, is found on GitHub and npm with npm install multimethod, and its annotated source code is available here.

Motivating Examples

Let's use the node.js REPL to build a few multimethods and see what they are capable of doing. In this first example we'll create a mulimethod that calculates the area of shapes instantiated with object literals.

> var multimethod = require('multimethod');
> var area = multimethod()
                .dispatch(function(o) {
                    return o.shape;
                })
                .when("square", function(o) {
                    return Math.pow(o.side, 2);
                });
> var aSquare = { "shape":"square", "side": 2 };
> area( aSquare );
4

> var aCircle = { "shape":"circle", "radius": 5 };
> area( aCircle );
undefined

> area.default(function(o) { 
    throw "Unknown shape: " + o.shape;
  });
> area( aCircle );
Unknown Shape: circle

> area.when("circle", function(o) {
    return Math.PI * Math.pow(o.radius, 2);
  });
> area( aCircle );
78.53981633974483
> area( aSquare );
4

> area.remove("circle");
> area( aCircle );
Unknown Shape: circle

Notice how dispatch returns the value we'll match to a "method" registered with when. You can introduce, overwrite, and remove new methods dynamically at runtime. Fallback behavior can be established with a default function called when no methods match the dispatched value.

> var hitPoints = multimethod()
                    .dispatch(function(player){ return player.powerUp; })
                    .when( {"type":"star"} , Infinity)
                    .default(5);

> var starPower = { "type":"star" },
>     mario = { "powerUp": starPower };
> hitPoints(mario);
Infinity

> mario.powerUp = null;
> hitPoints(mario);
5

> var godModeCheat = function() { return starPower; };
> hitPoints.dispatch(godModeCheat);
> mario.powerUp;
null
> hitPoints(mario);
Infinity

In this last example notice how we are matching against an object. Matching is done using deep equality so objects and arrays are valid method matching criteria. Also notice how we can completely override our dispatch function to change the logic with which a multimethod evaluates its arguments for dispatch, or, in this case, ignores them!

Multimethod Dispatch Algorithm

1. User calls multimethod with argument anArgument.
2. Multimethod calls its dispatch function with anArgument. The returned value is stored in dispatchValue.
3. Multimethod iterates through each 'method' registered with when and performs an equality test on the dispatchValue and each method's match value. If a match is found, set matchFunction to the method's function and go to step 5.
4. If no method match found, set matchFunction to the multimethod's default function.
5. Multimethod calls matchFunction with anArgument. The returned value is returned to the user who called the multimethod.

A More Detailed Walkthrough

The Basics

A multimethod is instantiated with the multimethod function.

var stopLightColor = multimethod();

A multimethod has methods. A method is has two parts, its match value and its implementation function. Methods are added using when.

stopLightColor.when("go",    function() { return "green"; })
              .when("stop",  function() { return "red"; });

You can call a multimethod just like any other function. It will dispatch based on the argument(s) passed in, invoke the matched method, and return the results of the method call.

console.log( stopLightColor("go") ); // "green"

When no method matches control can fallback to a default method.

stopLightColor.default( function() { return "unknown"; } );
console.log( stopLightColor("yield") ); // prints "unknown"

A multimethod can handle new cases dynamically at run time.

stopLightColor.when("yield", function() { return "yellow"; });

There is a shorter way for a method to return a plain value. Rather than passing an implementation function to when, pass the value.

stopLightColor.when("yield", "yellow");
console.log( stopLightColor("yield") ); // prints "yellow"

A method can be removed dynamically at run time, too.

stopLightColor.remove("go");
console.log( stopLightColor("go") ); // prints "unknown"

Dispatch Function

Each multimethod call first invokes a dispatch function whose return value is used to match the correct method to call. The dispatch function is passed the arguments the multimethod is invoked with and returns a value to match against.

The default dispatch function is an identity function. The basic stopLightColor examples could have been created with an explicit dispatch function.

var stopLightColor = multimethod()
      .dispatch(function(state){
         return state;
      })
      .when('go', 'green');
console.log( stopLightColor('go') ); // green

The power of the multimethod paradigm is the ability to dispatch with a user-defined function. This gives a multimethod its "polymorphic" powers. Unlike classical, object-oriented polymorphism where the compiler dispatches based on the type hierarchy, a multimethod can dispatch on any criteria.

var contacts = [
  {"name":"Jack", "service":"Twitter","handle": "@jack"},
  {"name":"Diane","service":"Email",  "address":"d@g.com"},
  {"name":"John", "service":"Phone",  "number": "919-919-9191"}
];

var sendMessage = multimethod()
     .dispatch(function(contact, msg) {
       return contact.service;
     })
     .when("Twitter", function(contact, msg) {
       console.log("Tweet @"+contact.handle+":"+msg);
     })
     .when("Email", function(contact, msg) {
       console.log("Emailing "+contact.address+":"+msg);
     })
     .default(function(contact, msg) {
       console.log("Could not message " + contact.name);
     });

// Blast a message
contacts.forEach( function(contact) {
  sendMessage(contact, "Hello, world."); 
});

Plucking a single property from an object is so commonly used as a dispatch function, like in the example above, there is a shortcut for this pattern. The following dispatch call is equivalent to above.

sendMessage.dispatch( 'service' );

A multimethod's dispatch is usually specified when constructed.

var sendMessage = multimethod('service');

Just like methods can be added and removed from a multimethod at run time, the dispatch function can also be redefined at run time. Ponder the implications of that for a minute. It is really powerful and really dangerous. Don't shoot your eye out.

Deep Equality Matching

A method's match value is compared to dispatch's return value using the underscore.js isEqual function. Deep equality method matching enables concise expressivity. Contrast this with a traditional switch statement that is limited by JavaScript's === equality behavior.

var greatPairs = multimethod()
      .when( ["Salt", "Pepper"], "Shakers" )
      .when( [{"name":"Bonnie"}, {"name":"Clyde"}], "Robbers" );
console.log( greatPairs( ["Salt", "Pepper"] ) ); // Shakers

API Recap

• Constructor: multimethod( [fn | string] ): No arg constructor uses an identity function for dispatch. Single arg constructor is a shortcut for calling dispatch with the same argument.
• dispatch(fn | string): Sets the multimethod's dispatch function. String values are transformed into a pluck function which projects a single property value from the first argurment.
• when(match, fn | value): Add a method to be invoked when the dispatch return value matches 'match'. If a non-function value is provided it will be returned directly. Calling when with the same match value twice will override the previously registered method.
• remove(match): Remove a method by it's match value.
• default(fn | value): Catch-all case when no method match is found.

Installation

Install with npm for use in node.js based projects.

npm install multimethod
node
> var multimethod = require('multimethod');

For in-browser use you will need to grab underscore.js and multimethod.js:

• underscore.js
  • Development: http://documentcloud.github.com/underscore/underscore.js
  • Minified: http://documentcloud.github.com/underscore/underscore-min.js
• multimethod.js
  • Development: https://raw.github.com/KrisJordan/multimethod-js/master/multimethod.js
  • Minified: https://raw.github.com/KrisJordan/multimethod-js/master/multimethod-min.js

How-to Contribute

• Submit bugs and feature requests on GitHub Issues page.
• Fork the repository and submit pull requests. Pull requests that update the test suite for coverage on changes will be brought in quickly.

The Accept header allows a single object or resource to have a single URI with multiple representations, such as HTML, XML, or JSON. It is a fundamental component of the HTTP spec and REST style web services. Unfortunately, today's web services rarely use content-negotiation, because browsers have historically chosen bad Accept headers. WebKit's was the last serious offender:

application/xml,
application/xhtml+xml,
text/html;q=0.9,
text/plain;q=0.8,
image/png,
*/*;q=0.5       

Parsed and prioritized:

1. application/xml
2. application/xhtml+xml
3. image/png
4. text/html
5. text/plain
6. */*

WebKit's Accept header preferred XML over HTML prior to my fix landing in trunk today. Web services could not offer both HTML and XML for a single resource and use content negotiation because WebKit based browsers (Safari, Chrome, et. al.) would be served the XML data not the HTML web page. Developers have had to hack around the issue. Twitter's API, for example, uses a common work around that changes the URI by appending a format extension.

http://api.twitter.com/version/statuses/friends_timeline.format
format ::= json | xml | rss | atom

The goal of the fix for WebKit's Accept header was straightforward: prefer HTML over XML. The ultimate solution was to minimize risk and take Firefox's lead. Here is the Accept header, already in order of preference, WebKit-based browsers will be using in the future:

text/html,
application/xhtml+xml,
application/xml;q=0.9,
*.*;q=0.8

Contributing to Open Source Software

The fix may have been committed today, but the initial bug report was filed in July 2009, almost two years ago! What took such a simple, but important, fix so long to be made? I would like to say it was the need for five other people to chime in on the ticket, someone on the WebKit team to notice, or a duplicate bug request to be filed 15,000 tickets later. But no, all the WebKit team needed was for someone, anyone, to submit a patch. So I checked out the subversion repository, made a patch, submitted it, and boom, I have contributed code to WebKit and fixed its Accept header woes. Very cool, but I feel horrible for not doing so with the original bug report in 2009; content negotiation would be more usable today.

Lesson learned: no matter how big or small an open source project is, when you find a bug and know the fix just do it. Do not assume code to be sacred or untouchable; just fix it and let the committers make the call. Bug reports are helpful, but patches and pull requests are gold. Discussions get read, code gets committed. Code speaks louder than words.

var byKey = function (key) {
	... do slow, expensive stuff ...
	return { value: “something expensive” };
};

And later optimized it to feel like this?

var lookup = {};
var byKey = function (key) {
    if ( lookup[key] === undefined ) {
        ... do slow, expensive stuff ...
        lookup[key] = { name: “something expensive” };
    }
    return lookup[key];
};

I certainly have. The lookup table is a trusty pattern for caching the results of expensive calls. With four additional lines of code you can dramatically improve the performance of an often called function. Sprinkle these lines around your most expensive, referentially transparent functions and you have a nicely optimized program, right?

With a functional language this pattern is considered harmful; an anti-pattern to refactor. It may be four lines of code, but it is redundant when used over and over. Your function’s intentions become conflated with performance concerns. It would be best to keep our intentions and optimizations separate.

Can we simply write “cache this function’s return values”? It would feel like this,

var byKey = cache( function(key) {
    ... do expensive stuff ...
    return { name: “something expensive” };
} );

That feels much better. Although, not knowing what the cache function actually does is a little off-putting. Let’s implement it.

If you are unfamiliar with anonymous functions and closures in JavaScript this is about to get slightly hairy. Understanding those concepts will make writing JavaScript much more interesting, and little blog posts like this more palatable. I suggest Crockford, but there are plenty of good on-line resources, too.

We need our cache function to take any function we throw at it and return a new function. The returned function must accomplish what our imperative lookup table code did: remember the arguments of previous invocations and cache their return values.

var cache = function( expensiveFn ) {
    var lookup = { };
    return function(key) {
        if(lookup[key] === undefined) {
            lookup[key] = expensiveFn(key);
        }
        return lookup[key];
    };
};

Let's walk through this. We have defined cache as a function that takes expensiveFn as a parameter. Within this function we setup the variable lookup to be our hash/cache table. Then we return a function that encapsulates the lookup table pattern used earlier. Do you see how the structure has been extracted and the state hidden within the closure?

In computer science this function is conventionally named memoize. It is often demonstrated using a recursive function like factorial. Factorial is a more beautiful application of memoization than basic key-value caching, but this is an anti-patterm you are more likely to find in real code.

An Exercise for the Reader

The most obvious shortcoming to cache, our implementation of memoize, is that it does not generalize to functions of more than one argument. If you have a few minutes, pop open your browser’s JavaScript console in Firebug or Webkit Inspector and try writing it. Hints: 1) The arguments keyword provides an array of all arguments a function is called with. 2) A function can be invoked with an array of arguments by using Function's apply method.

Refactor one-off caching via lookup tables with memoize

Plenty of libraries provide a generic memoize function, including underscore in JavaScript. The next time you catch yourself writing a naive in-memory cache lookup table around a function stop and consider memoizing it instead. Using memoize is faster to write and less bug-prone than a one-off cache, it separates intentional concerns from optimization concerns, and it keeps your code a little cleaner.

While sitting at the bar one of my sisters' friends drew a lot of attention from the gentlemen of Rutherford County. We'll call her Liz (gah, when does 30Rock start up again??). Some would come up and chat, others ask for a dance. The most peculiar case of them all was one who had been ordering drinks from across the bar and finally got up the courage to come over and, without a word, slip Liz a piece of paper with his phone number on it before disappearing into the night.

Yes, this post is about programming, bear with me.

Let's give this guy, say Dennis, the benefit of the doubt and assume that these tactics have worked before with success. On this particular night with a gorgeous girl from out of town: no dice. Dennis wouldn't get the callback.

Let's pseudo-code this scenario:

$liz = new AttractiveOutOfTownGirl();
$dennis = new SmallTownFriendlyYoungMan();
$liz->setDrink($dennis->buyDrink('Vodka Redbull'));
$liz->setDrink($dennis->buyDrink('Sex on the Beach'));
$liz->receiveNumber($dennis->getPagerNumber());
$dennis->disappearIntoTheNight();

Spot any patterns? Thrice Dennis has taken an action that results in some value: he's bought 2 drinks and retrieved his pager number. All of his methods have resulted in some value that is handed to Liz and he's getting no acknowledgement back. He interacted with muted methods. Dennis' shot at success depended on Liz having some internal, finite state machine that transitioned from "stranger" to  "stranger willing to call a stranger" with every additional drink. How could this have gone better for Dennis?

Before we get there let's talk about muted methods. Procedures. Voids. Black holes in programs where input results in no output. Scary stuff! But why, what's so bad about muted functions? Let's nitpick our example. It is so  choppy. Do you feel the choppiness?

Noun verb.
Noun verb.
Noun verb.
Liz set drink.
Liz set drink.
Liz give number.

When your verb is muted, there's nothing more to be said. End of story. Next line. You've got nothing to play off of, no chance to converse. When you're working with muted functions you have no segue. You're giving without receiving.

If muted functions are the problem, how could this scene look a little less ugly? Perhaps Dennis could have conversed with Liz. Interacted with her. Made use of a more fluent interface...

$dennis->saysTo($liz, "Hey girl, from out of town?") // returns $liz
       ->saysTo($dennis, "Yes, how'd you know?") // returns $dennis
       ->saysTo($liz, "Never did seen you before!") // returns $liz
       ->isStillSmiling()
       ? // If she's smiling
   $liz->setDrink($dennis->buyDrink('Vodka Redbull')) // returns $liz
       ->saysTo($dennis, "Thanks, you're kind of cute.")
       ->saysTo($liz, "That's how we're grown here. Can I call you?")
       ->saysTo($dennis, "Sure, my name is Liz and the number is...")
       ->punchesInNumber("919-928-9090") // returns $dennis
       ->says("Great, call you tomorrow!") // returns $dennis
       ->disappearIntoTheNight()
       : // If she's frowning
   $dennis->movesAlong();

When you're not dealing with muted methods you give your code the chance to come to life in a more fluid, fluent way. There's only sentences here, it's just longer, richer, and more involved. There's magical, programmatic banter happening here!

This style of object-oriented interface, where method chaining and thoughtful return values enable a pseudo-domain specific language written entirely in a general purpose language, has been called a "Fluent Interface" by Martin Fowler and Eric Evans. It's an inspiring departure from the days of the void, black hole methods like setters.

So, next time you're writing an interface for some new class, or sitting at a bar wondering how best to approach that attractive member of the opposite sex, avoid muted methods at all costs. They're black holes. They're just awkward, feedbackless, and overall uncomfortable for everyone involved.

When your functions or methods don't return ask yourself "if I were to return something here what would be useful?" Most often, it's a reference to the object itself. Or an immutable clone.

"Oh, god no, not an immutable clone! I thought this article was about bar game?" Hmm... you're right. I'll save the immutable clone wars for another post. You should subscribe to this RSS feed if immutable clones pique your interest. Or nerdy bar game.

• Using virtual machines and shell scripts to achieve the Ultimate Small Team Software Engineering Environment (tm)
• Real Software Engineering in PHP
• Trees, Recursion, Permissions, and Architecture of my company's next Content Management System
• How to simplify internals and increase functionality of the Recess Framework by leveraging PHP 5.3's new features like anonymous functions and namespaces
• A big client integration project that deals with scary acronyms like HIPAA

This space should be more lively than the ghosttown that it has become over the last 8 months. More off the cuff writing on the topic of the hour than the more structured pieces I've been writing for New Media's Web Development Blog or the Recess PHP Blog.

To kick that off the off the cuff writing, a few words on _why. I don't know _why, I haven't read the poignant guide to Ruby (although I've come across it a number of occasions and been intrigued), nor have I used much of his huge body of open source work. None the less his recent disappearance has been sitting really funny on my stomach. Really funny. This guy is prolithic, talented, and obsessed with teaching the art and joy of programming. I didn't fully appreciate _why until watching his recent talk at Carnegie Melon [embedded below]. His excitement and creative knack for opening programming up to a younger audience is energizing. Here's to hoping an even brighter chapter to that story is about to unfold...

Without further adieu... Routing in Recess!

REST is an architectural style, or set of conventions, for web applications and services that centers itself around resource manipulation and the HTTP spec. Web apps have traditionally ignored the HTTP spec and moved forward using a subset of the protocol: GET and POST, 200 OKs and 404 NOT FOUNDs. As we entered a programmable web of applications with APIs the decision to ignore HTTP gave us problems we're still dealing with today. We have an internet full of applications with different interfaces (GET /user/1/delete vs. POST /user/delete {id=1}). With REST we can say /user/1 is a resource and use the HTTP DELETE verb to delete it. For more detail on REST check out wikipedia and "quick pitch".

Tip #1: Using PUT and DELETE methods

In PHP you can determine which HTTP method was used with: $_SERVER['REQUEST_METHOD']; From web browsers this will be either GET or POST. For RESTful clients applications need to support PUT and DELETE (and ideally OPTIONS, etc.) as well. Unfortunately PHP doesn't have $_PUT and $_DELETE variables like it does $_POST and $_GET. Here's how to access the content of a PUT request in PHP:

$_PUT  = array();
if($_SERVER['REQUEST_METHOD'] == 'PUT') {
    $putdata = file_get_contents('php://input');
    $exploded = explode('&', $putdata); 

    foreach($exploded as $pair) {
        $item = explode('=', $pair);
        if(count($item) == 2) {
            $_PUT[urldecode($item[0])] = urldecode($item[1]);
        }
    }
}

Tip #2: Send Custom HTTP/1.1 Headers

PHP's header function allows custom HTTP headers to be sent to the client. The HTTP/1.x header contains the response code from the server. PHP will, by default, send back a 200 OK status code which suggests that the request has succeeded even if it has die()'ed or a new resource has been created. There are two ways to change the status code of your response:

header('HTTP/1.1 404 Not Found');
/* OR */
header('Location: http://www.foo.com/bar', true, 201); // 201 CREATED

The first line is a generic way of setting the response status code. If your response requires another header, like the Location header to the resource of a '201 Created' or '301 Moved Permanently', placing the integer status code in the third parameter of header is a shortcut. It is the logical equivalent of the following example, which is easier to read at the cost of being an extra line of code.

header('HTTP/1.1 201 Created');
header('Location: http://www.foo.com/bar');

Tip #3: Send Meaningful HTTP Headers

Policy for deciding when it is appropriate to send each HTTP status code is a full post on its own and the HTTP spec leaves room for ambiguity. There are many other resources on the net which provide insights so I'll just touch on a few.

201 Created is used when a new resource has been created. It should include a Location header which specifies the URL for the resource (i.e. books/1). The inclusion of a location header does not automatically forward the client to the resource, rather, 201 Created responses should include an entity (message body) which lists the location of the resource.

202 Accepted allows the server to tell the client "yeah, we heard your order, we'll get to it soon." Think the Twitter API on a busy day. Where 201 Created implies the resource has been created before a response returns, 202 Accepted implies the request is ok and in a queue somewhere.

304 Not Modified in conjunction with caching and conditional GET requests (requests with If-Modified-Since / If-None-Match headers) allows web applications to say "the content hasn't changed, continue using the cached version" without having to re-render and send the cached content down the pipe.

401 Unauthorized should be used when attempting to access a resource which requires authentication credentials the request does not carry. This is used in conjunction with www-authentication.

500 Internal Server Error is better than OK when your PHP script dies or reaches an exception.

In the Recess! Framework I use this StatusCodes class to provide named constants for all HTTP/1.1 status codes. Example usage:

header(StatusCodes::httpHeaderFor(StatusCodes::HTTP_NOT_FOUND));

Tip #4: Don't Use $_SESSION

Tip #5: Test with cURL or rest-client

# curl -X PUT http://www.foo.com/bar/1 -d "some=var" -d "other=var2" -H "Accept: text/json" -I

Alternatively, a free GUI to test REST interfaces is Java/Swing based rest-client. rest-client is scriptable and has support for JSON/XML.

Tip #6 - Use a RESTful PHP Framework

Frankly, developers shouldn't have to worry about many of these low-level details of REST when writing PHP apps. REST is based on conventions and conventions, by nature, involve a lot of boilerplate. This is right up a framework's alley (as Rails has shown). What options exist for PHP? CodeIgniter's routing completely ignores the HTTP METHOD so there is serious hacking that needs to be done. Cake has REST support but wasn't designed to make specifying useful response status codes a part of the framework. Konstruct appears to have a very well thought out architecture for a controllers framework built around HTTP and REST. Unfortunately it is not easily approached and lacking (intentionally) many components, like an ORM layer, developers have come to expect in a modern web framework.

(Disclaimer: Shameless Plug!) The lack of a solid, RESTful PHP framework was one of my primary motivations for creating the Recess! Framework. Recess is a full-stack, open source (MIT), RESTful PHP framework. If you're interested in writing RESTful PHP applications check it out and sign-up to be notified of its upcoming release.

<?php
/**
 * StatusCodes provides named constants for
 * HTTP protocol status codes. Written for the
 * Recess Framework (http://www.recessframework.com/)
 * 
 * @author Kris Jordan
 * @license MIT 
 * @package recess.http
 */
class StatusCodes {
	// [Informational 1xx]
	const HTTP_CONTINUE = 100;
	const HTTP_SWITCHING_PROTOCOLS = 101;
	// [Successful 2xx]
	const HTTP_OK = 200;
	const HTTP_CREATED = 201;
	const HTTP_ACCEPTED = 202;
	const HTTP_NONAUTHORITATIVE_INFORMATION = 203;
	const HTTP_NO_CONTENT = 204;
	const HTTP_RESET_CONTENT = 205;
	const HTTP_PARTIAL_CONTENT = 206;
	// [Redirection 3xx]
	const HTTP_MULTIPLE_CHOICES = 300;
	const HTTP_MOVED_PERMANENTLY = 301;
	const HTTP_FOUND = 302;
	const HTTP_SEE_OTHER = 303;
	const HTTP_NOT_MODIFIED = 304;
	const HTTP_USE_PROXY = 305;
	const HTTP_UNUSED= 306;
	const HTTP_TEMPORARY_REDIRECT = 307;
	// [Client Error 4xx]
	const errorCodesBeginAt = 400;
	const HTTP_BAD_REQUEST = 400;
	const HTTP_UNAUTHORIZED  = 401;
	const HTTP_PAYMENT_REQUIRED = 402;
	const HTTP_FORBIDDEN = 403;
	const HTTP_NOT_FOUND = 404;
	const HTTP_METHOD_NOT_ALLOWED = 405;
	const HTTP_NOT_ACCEPTABLE = 406;
	const HTTP_PROXY_AUTHENTICATION_REQUIRED = 407;
	const HTTP_REQUEST_TIMEOUT = 408;
	const HTTP_CONFLICT = 409;
	const HTTP_GONE = 410;
	const HTTP_LENGTH_REQUIRED = 411;
	const HTTP_PRECONDITION_FAILED = 412;
	const HTTP_REQUEST_ENTITY_TOO_LARGE = 413;
	const HTTP_REQUEST_URI_TOO_LONG = 414;
	const HTTP_UNSUPPORTED_MEDIA_TYPE = 415;
	const HTTP_REQUESTED_RANGE_NOT_SATISFIABLE = 416;
	const HTTP_EXPECTATION_FAILED = 417;
	// [Server Error 5xx]
	const HTTP_INTERNAL_SERVER_ERROR = 500;
	const HTTP_NOT_IMPLEMENTED = 501;
	const HTTP_BAD_GATEWAY = 502;
	const HTTP_SERVICE_UNAVAILABLE = 503;
	const HTTP_GATEWAY_TIMEOUT = 504;
	const HTTP_VERSION_NOT_SUPPORTED = 505;
		
	private static $messages = array(
		// [Informational 1xx]
		100=>'100 Continue',
		101=>'101 Switching Protocols',
		// [Successful 2xx]
		200=>'200 OK',
		201=>'201 Created',
		202=>'202 Accepted',
		203=>'203 Non-Authoritative Information',
		204=>'204 No Content',
		205=>'205 Reset Content',
		206=>'206 Partial Content',
		// [Redirection 3xx]
		300=>'300 Multiple Choices',
		301=>'301 Moved Permanently',
		302=>'302 Found',
		303=>'303 See Other',
		304=>'304 Not Modified',
		305=>'305 Use Proxy',
		306=>'306 (Unused)',
		307=>'307 Temporary Redirect',
		// [Client Error 4xx]
		400=>'400 Bad Request',
		401=>'401 Unauthorized',
		402=>'402 Payment Required',
		403=>'403 Forbidden',
		404=>'404 Not Found',
		405=>'405 Method Not Allowed',
		406=>'406 Not Acceptable',
		407=>'407 Proxy Authentication Required',
		408=>'408 Request Timeout',
		409=>'409 Conflict',
		410=>'410 Gone',
		411=>'411 Length Required',
		412=>'412 Precondition Failed',
		413=>'413 Request Entity Too Large',
		414=>'414 Request-URI Too Long',
		415=>'415 Unsupported Media Type',
		416=>'416 Requested Range Not Satisfiable',
		417=>'417 Expectation Failed',
		// [Server Error 5xx]
		500=>'500 Internal Server Error',
		501=>'501 Not Implemented',
		502=>'502 Bad Gateway',
		503=>'503 Service Unavailable',
		504=>'504 Gateway Timeout',
		505=>'505 HTTP Version Not Supported'
	);
	
	public static function httpHeaderFor($code) {
		return 'HTTP/1.1 ' . self::$messages[$code];
	}

	public static function getMessageForCode($code) {
		return self::$messages[$code];
	}
	
	public static function isError($code) {
		return is_numeric($code) && $code >= self::HTTP_BAD_REQUEST;
	}
	
	public static function canHaveBody($code) {
		return
			// True if not in 100s
			($code < self::HTTP_CONTINUE || $code >= self::HTTP_OK)
			&& // and not 204 NO CONTENT
			$code != self::HTTP_NO_CONTENT
			&& // and not 304 NOT MODIFIED
			$code != self::HTTP_NOT_MODIFIED;
	}
}
?>

Re: Oversight #1 - No Data Access Layer

Quality software design and engineering is all about abstracting away details and finding the proper layers of functionality. As Justin points out, if there is no layer of abstraction which wraps data access code then low-level data access code will appear everywhere. PHP projects which interleave data connections and queries throughout their code are difficult to maintain.

The database stack in Recess! has four major components: 1) SQL statement builder, 2) Data Sources, 3) Data Sets, 4) Object-Relational Mapped Models. The notion of a 'data access layer' is taken care of by Data Sources. Data sources wrap properly around PDO and introduce new methods beyond PDO's, for example: getTables and getColumns($table). The implementation of these methods is specific to the RDBMS so Recess! also has the notion of a vendor specific driver called a DataSourceProvider. Currently Sqlite and MySql are supported.

Re: Oversight #2 - Design for Only One Database Connection

Small, greenfield projects often start out with a single database. This leads to naive data access layers which only support connections to a single database. Recess! supports multiple, named Data Sources. Instead of using a singleton pattern it uses a registry to store the Data Sources by name.

Having named Data Sources in an application gets really powerful at higher layers of abstraction, such as at the ORM layer, where we can mark-up a model with a single annotation to define which Data Source the model is persisted in.

Re: Oversight #3 - No Developer Logging

This isn't as fully fleshed out yet as I'd like but the plans and hooks are in place to be able to log queries, their stack traces, as well as their EXPLAIN'ed strategies for execution in the RDBMS. In Recess! there is a distinct difference between Development mode and Production mode. Sql logging will be a toggle option on by default in development and off by default in production.

Re: Oversight #4 - Queries Written in Procedural Processes

My dislike of queries being written in procedural code may run even deeper than Justin's! Recess! abdicates SQL string manipulation to a low-level SqlBuilder class. An instance of SqlBuilder allows SQL query strings to be built incrementally using chained method calls. Let's take a look at some examples:

$sqlBuilder = new SqlBuilder();
$sql = $sqlBuilder
            ->from('people')
            ->like('name','Kris')
            ->orderBy('age')
            ->toSql();
// $sql is now: 
// 'SELECT people.* 
//    WHERE people.name = :people_name 
//    ORDER BY people.age
$args = $sqlBuilder->getPdoArguments();
// $args is now array( 'people_name' => 'Kris' );

// Let's add another criterion
$sql = $sqlBuilder->equal('home_city', 'Charlotte')->toSql();
// $sql is now: 
//  SELECT people.* 
//      WHERE people.name LIKE :people_name 
//      AND people.home_city = :people_home_city 
//      ORDER BY people.age

$args = $sqlBuilder->getPdoArguments();
// $args is now array( 
//                   'people_name' => 'Kris', 
//                   'people_home_city' => 'Charlotte' );

SqlBuilder can do more complex things like joins, as well as inserts, updates, and deletes. Recess! users, though, will likely never use SqlBuilder because it's still too low level of an abstraction. Internally, however, having the ability to incrementally construct query strings has made the framework code quite pretty.

Here's code at the level of abstraction Recess! developers can expect to write:

class Person extends Model { }
$person = new Person();
$person->name = 'Kris';
$person->homeCity = 'Charlotte';
$people = $person->find()->orderBy('name');
foreach($people as $person) {
   echo $person->name, ' ', $person->age, '<br />';
}

This example just scrapes the surface of Recess! object-relational mapping. Relationships, CRUDS, cascading deletes, etc. are all handled as well. In a future post I'll step through the data access stack's capabilities in more detail. Needless to say, in Recess! queries won't be mixed with procedural code!

Re: Oversight #5 - No Separation of Reads and Writes

Once projects outgrow a single server RDBMS the next step is often to do a Master/Slave setup in MySQL. In a Master/Slave setup expensive and less frequent writes are channeled to the Master server while reads are channeled to the Slave server.

In Recess!, by using named data sources as described in #2, we can handle Data Sources on a per model basis. A natural extension of this is handling reads and writes independently on a per model basis. The pseudo-code below shows how this may look on a model:

/** 
 * !Source master, For: Writes
 * !Source (slave1, slave2, slave3), For: Reads
 * !HasMany books
 */
class Person extends Model {}

/**
 * !Source master
 * !BelongsTo person
 */
class Books extends Model {}

In a future post I'll describe the Recess! annotations system which will explain the constructs you're seeing in the DocComments. Recess! annotations give us a way of providing static metadata about a class and are built-in to the framework's core. Recess! annotations should be familiar to Java and .Net programmers who have used annotations.

Conclusion

So that's a quick look at how Recess! handles Justin Carmony's commonly made MySQL/PHP mistakes. These issues are abstracted away in the internals of Recess. If you're a PHP developer check out RecessFramework.com and sign-up to be notified when it is publicly released "very soon now".

Simple, Built-in Dynamic Properties

Check out the following code listing:

class DynamicProperties { }
$object = new DynamicProperties;
print isset($object->foo) ? 't' : 'f'; // f

// Set Dynamic Properties foo and fooz
$object->foo = 'bar';
$object->fooz = 'baz';

// Isset and Unset work
isset($object->foo); // true
unset($object->foo);

// Iterate through Properties and Values
foreach($object as $property => $value)  { 
     print($property . ' = ' . $value . '<br />'); 
}

// Prints:
//   fooz = baz

Using the built-in dynamic instance properties is an order of magnitude faster (30x, by my profiling) than using magic __get and __set methods. Built in dynamic property accesses happen without invoking a method call back to PHP script.

So when does it make sense to use __get and __set? If you need more complex behavior, like calculated properties, you must use __get and __set. Also, as an astute comment points out, if you would prefer not to have dynamic properties on a class you can throw errors from __get and __set.

StdClass: Anonymous Objects

Sometimes all that is necessary is a property bag to throw key value pairs into. One way is to use array, but this requires quoting all keys. Another way is to use dynamic properties on an instance of StdClass. StdClass is a sparsely documented class in PHP which has no predefined members.

$object = new StdClass;
$object->foo = 'bar';
json_encode($object);

Next I'll touch on the SPL's Countable and ArrayAccess as a means of being able to accomplish the following in PHP:

class MyClass implements Countable, ArrayAccess { ... }
$myObject = new MyClass();
// Using array access notation
$myObject[0] = 'hello';
$myObject[1] = 'world';
$myObject['foo'] = 'bar';

Thanks to the folks pointing out that you don't need to extend from StdClass in order to have dynamic properties!

After the elections have taken place the votes on websites will be tallied up and compared to election results. Should be really interesting to see what the outcome is and how strong website quality correlates with campaign success.

Within the first 24 hours over 5,000 votes have been cast. Congrats to the New Media team for getting this out the door so quickly (with a little help from the Mechanical Turk)! Go vote on the websites in your state…

VoteTheSite.com was written (in under 2 days!) on an early version of the PHP Framework, the Recess! Framework, I’ve had my head down plugging away on the past couple of weeks. Expect blog activity to pick up as the Recess! Framework moves closer to a public release. Sign up to be notified of the release at RecessFramework.com.