ElasticDog.com

Use sshuttle to Keep Safe on Insecure Wi-Fi

2011-12-31T10:56:00-08:00

If you’ve ever used an insecure Wi-Fi connection in a public place (think coffee shop, airport, etc.), your personal information could be at risk. Anyone on the local network could potentially be monitoring your traffic and they could even hijack your sessions to control accounts that you’ve logged into.

Every week I meet up with some friends to work on personal projects, and the venue only offers an open wireless connection with no encryption. To mitigate the risk of leaking personal information, I’ve tried a handful of strategies and eventually settled on using sshuttle.

What is sshuttle?

The sshuttle utility is part transparent proxy, part Virtual Private Network (VPN), and part SSH. It solves a lot of common problems with encrypting your traffic, and it does so in a very efficient way.

There’s no need for a complicated pre-existing infrastructure. All you need is Python 2.x, and a remote machine you can SSH to that also has Python installed. You don’t even have to be an administrator on the remote machine. The project’s README has a lot more information on its theory, design, and alternate uses.

Previous Method

As an aside, I had previously relied on creating a SOCKS proxy with SSH, and then tunnelling my web traffic through it…

$ ssh -fqND 1080 example.com

But, you have to adjust your machine’s proxy configuration settings each time you set up the tunnel, and then tear it all down when you’re done. This method will not force absolutely all traffic over the tunnel, as things like DNS lookups and a lot of command-line tools will just ignore the proxy. You’re also encapsulating TCP-over-TCP, which can cause performance problems.

I briefly messed with the Sidestep project to automate these setup/teardown steps on OS X, but ran across repeated authentication issues and wasn’t overly impressed with its developer. Overall, I needed a more elegant solution.

Installation

I wanted to use sshuttle on OS X Lion, and even though it’s primarily a command-line utility, it also comes with a graphical user interface wrapper that will sit in the menu bar. If you just want the command-line script and already use Homebrew, brew install sshuttle is the way to go. If you don’t use Homebrew and/or want the GUI, I’ve got your back…

Building sshuttle from Source

Unfortunately, pre-compiled binaries are not made available for every release, and documentation for building sshuttle is non-existent. The project uses a build system called ‘redo’ instead of the more-customary ‘make’, so the process is a bit different than what most people are used to.

Luckily, the sshuttle repository includes a shell script named ‘do’, which is a simplified version of redo, so you won’t need to install any additional build software. The only real dependencies are Python and two Python modules to generate the man page (feel free to use pip install instead of easy_install, if you prefer):

$ easy_install markdown BeautifulSoup

Once you’ve installed those two modules, you’re all set to build…

NOTE: You may have to adjust paths or permissions slightly depending on your machine’s setup.

$ mkdir -p /usr/local/libexec
$ git clone git://github.com/apenwarr/sshuttle.git /usr/local/libexec/sshuttle
$ cd /usr/local/libexec/sshuttle/
$ ./do all

Once sshuttle is built, we just make symlinks so the binary and man page are in the proper places:

$ ln -s /usr/local/libexec/sshuttle/sshuttle /usr/local/bin
$ mkdir -p /usr/local/share/man/man8
$ ln -s /usr/local/libexec/sshuttle/Documentation/sshuttle.8 /usr/local/share/man/man8

For the GUI app, we need to create an alias instead a symlink, because Spotlight will not index symlinks. Yes, you could just copy Sshuttle VPN.app directly into /Applications, but then you’d have to remember to copy it every time you update/rebuild sshuttle.

Without getting too technical, aliases are a lot like symlinks but they use extra metadata called “resource forks” to follow a file (even when it’s moved) rather then blindly pointing to a path. More importantly, Spotlight will index them. There’s no way to create an alias from the command line, so you have to use Finder:

Open Finder and press Command-Shift-G to open the “Go to the folder:” dialog box.
Type /usr/local/libexec/sshuttle/ui-macos in the box and click the Go button.
Click and hold Sshuttle VPN.app, then while holding down both the Command and Option keys, drag the app over to Applications on the left and release the mouse button.

Configuration

Due to some sysctl changes in OS X Lion, you will have to reboot your machine after you run sshuttle for the first time. An adjustment to the net.inet.ip.scopedroute kernel flag needs to be made, and it can no longer be done during runtime. This is required so forwarding rules will be honored by the firewall.

Everything should be handled automatically by sshuttle, but if your curious to see the change, you can display the boot kernel flags and current settings with the following commands:

$ defaults read /Library/Preferences/SystemConfiguration/com.apple.Boot
{
    "Kernel Flags" = "";
}
$ sysctl -a | grep scopedroute
net.inet.ip.scopedroute: 1
net.inet6.ip6.scopedroute: 1

After you run sshuttle for the first time and then reboot the machine, you can confirm the updated status:

$ defaults read /Library/Preferences/SystemConfiguration/com.apple.Boot
{
    "Kernel Flags" = "net.inet.ip.scopedroute=0";
}
$ sysctl -a | grep scopedroute
kern.bootargs: net.inet.ip.scopedroute=0
net.inet.ip.scopedroute: 0
net.inet6.ip6.scopedroute: 1

Usage

Finally we can get down to actually using sshuttle! It’s flexible enough to do fancier things, but for our particular use case, we just need to forward all traffic over the tunnel.

The basic command to achieve our goal looks like this:

$ sshuttle --dns -r example.com 0/0

NOTE: Sshuttle will honor your ~/.ssh/config connection settings, but you can also manually specify a different username/port, if necessary:

$ sshuttle --dns -r username@example.com:2222 0/0

Run that, and all of your traffic (including DNS requests) will be transparently proxied through an SSH connection to example.com. You can verify this by browsing to http://ifconfig.me.

To stop forwarding traffic, just press Ctrl-c back in the terminal. We can do a bit better though by forking the process into the background so we don’t tie up our terminal session. These are the aliases I use to make setting up and tearing down the tunnel easier:

alias tunnel='sshuttle --dns --daemon --pidfile=/tmp/sshuttle.pid --remote=example.com 0/0'
alias tunnelx='[[ -f /tmp/sshuttle.pid ]] && kill $(cat /tmp/sshuttle.pid) && echo "Disconnected."'

The Sshuttle VPN.app GUI is pretty self-explanatory, just make sure to enable the Send DNS requests through this server checkbox, and select Send all traffic through this server for the “Network routes:” option.

Known Bugs

You may see a bunch of “warning: closed channel …” messages when running sshuttle (either on STDOUT or in your system.log), but these warnings are safe to ignore. The developer knows about the issue and is thinking of the best way to suppress/eliminate the condition.

Also, prior to version 0.60 of sshuttle, there was a bug in the GUI application when disconnecting tunnels, where the controlling terminal would disappear before firewall rules could cleaned up. This would leave the network in an odd state and all DNS lookups would fail. It should be fixed now, but if you experience networking trouble after disconnecting a sshuttle tunnel, you can see the current firewall rules using:

$ sudo ipfw list

…and you can manually flush out those rules to fix connectivity by running:

$ sudo ipfw -f flush

Be aware though, this will flush out all rules, not just the rules set by sshuttle, so it may have unintended consequences. If you’re unsure, you can always reboot the machine to get connectivity back to normal.

Update sshuttle

Since our installation of sshuttle is managed via git, updating to the latest version is pretty straightforward:

$ cd /usr/local/libexec/sshuttle
$ git pull
$ ./do all

Conclusion

Admittedly, sshuttle takes a bit more work than other solutions to get up and running, but the security it provides gives me peace of mind when forced to use insecure Wi-Fi networks.

If you have any issues, there is an active mailing list for the project, or you can always send me a note and I’ll see what I can do to help.

Shelf-Made Standing Desk

2011-01-25T00:00:00-08:00

Since I started working from home, I’ve found that I spend even more time sitting at my desk than I did when I worked at the office. I’ve never had great posture and started noticing that my back and shoulders were pretty achy by the end of the day. Once I got to that point, it was harder to stay focused on work and I was becoming noticably fatigued.

I started researching alternative workspaces, and found ideas ranging from sitting on an excercise ball to walking on a treadmill desk. Having sat on excercise balls before, I knew that I would quickly regress back to slouching; and the idea of working on a treadmill seemed too impractical. I needed something in between…

A Standing Desk

There seems to be a surge of people converting to standing desks recently, and there’s no shortage of personal stories and information on why they can be beneficial. Some very nice commercially-made standing desks are available, but not knowing if I’d actually like using one, I didn’t want to make a big up-front investment.

My Old Desk

For reference, my old desk was simply a cubicle desk surface that had been reclaimed from a dumpster. I always liked the amount of horizontal space it provided, and it was extremely solid, but not completely ideal.

My New Desk

I decided to convert a cheap shelving unit into a standing desk as a way to test the waters. At 48"W x 24"D x 72"H, this $80 rack from Lowes was pretty ideal, and I figured that after a month if I didn’t like it, the shelf could always be used in my garage.

I tried to follow OSHA guidelines as closely as possible when setting up the height of each shelf. The bottom shelf is about 7" above the ground to give my feet ample space below the rack; the upper shelf was placed so the top of the displays are right about eye-level; and the middle shelf was placed just below my elbow height for the keyboard and mouse. I used the extra shelf (which would have been on the very top) as a keyboard “tray” by cutting some notches in the corners to create a ledge that sticks out about six inches in the front.

Some friends pointed out that MDF could potentially contain Formaldehyde in the resin used to bind it together, and that prolonged contact might cause skin irritation. To prevent this, I wrapped the front part of the keyboard tray in faux leather contact paper. It looks nice, and prevents my wrists from getting itchy.

Here’s how everything came together…

So my hands always stay at the proper height, I use Synergy to control both my Linux desktop and my MacBook Pro laptop with the same keyboard and mouse.

The medical community points out that staying in one position for too long (whether that’s sitting or standing) can be bad for you, so there’s also a futon in my office where I can switch to using my laptop with a Logitech Comfort Lapdesk. Eventually, I may order a tall stool, so the option to sit at the desk will also be available; although, I do plan on standing for the majority of the time.

Accessories

To make the standing desk more ergonomic, I purchased a few accessories:

A Rain Design mStand laptop stand to bring my MacBook Pro’s display up to the proper height
A Kensington SoleMate Plus adjustable footrest to help facilitate shifting my weight throughout the day
A 16" Zilotek LED strip light to help eliminate shadows and illuminiate the shelf with the keyboard and mouse

Here’s an idea of roughly how the keyboard tray looks without the LED strip light installed; and if you mouse over the image, you’ll see what it looks like with the LED lights turned on…

Conclusions

My first full day of standing was January 19th, 2011, and so far, I really love the desk. The first week was definitely a little rough, with my feet and back becoming sore by the end of the day, but it’s getting better. I’ve read that it takes a couple of weeks to fully adjust, so after I’ve given the desk a bit more time, I’ll post another update. At this point, I’m optimistic that it will be a permanent change to my home office setup.

Update

It has been almost 8 months since I started using the standing desk, and I don’t think I could go back. My posture has noticeably improved, and I quickly got over the initial soreness. I stand up about 85–90% of the time, and sit down with my lapboard for the rest (mostly later in the day). I’ve only made a few changes to my setup so far:

I switched to an Kenesis Freestyle ergonomic keyboard to help improve wrist positioning and limit overreaching when using the mouse
I added an Imprint anti-fatigue floor mat to make standing more comfortable (even though my office is carpeted)
I added an Apple Magic Trackpad near my mouse pad so I wouldn’t need to reach to the upper level of the desk to perform multi-touch gestures

It’s safe to say that I enjoy utilizing a standing desk, and even though my shelving unit was originally purchased as an inexpensive way to get my feet wet, it has worked out better than expected. For now, it gets the job done and I don’t see the need to purchase a more expensive standing desk.

The only other planned improvements I have would be, to finally cut off the extra portion of each of the metal support rails that stick out from the top, and to build small side shelves that could support a pair of KRK Rokit 5 studio monitors. Once I’ve done those things, I’ll post an updated photo of how it looks.

Since posting this original article, I’ve inspired a handful of friends to try out standing desks, and all of them have stuck with it. If you’ve been on the fence about trying a standing desk, I can easily say that it’s worth the effort to give it a shot. I’d love to hear from you if you end up taking the plunge!

Update²

Well, I finished all of my planned improvements and here are the updated pics…ignore the dust and messy cabling :-)

I found the shelves as a kit from Knape & Vogt for ~$20 each, but had to purchase some additional hardware (bolts, washers, and locking nuts) since they were originally designed for mounting with wood screws. The shelves can supposedly support up to 50 pounds each; they are stable, but I wouldn’t be comfortable adding much more than the current 15 pounds on them.

Cutting the metal support rails and trimming the shelf mounts to fit took a fair bit of time, a steady hand, and a Dremel. If you try it, the one thing I’ll say is to definitely wear safety glasses!

Anyway, I think I’m finally done tweaking things for a bit, but am incredibly happy with the results. Go forth and stand!

Porting a Poker Hand Evaluator from C to Factor

2010-11-07T00:00:00-07:00

To help teach myself Factor, I started solving a lot of the problems at Project Euler and approached each one, not only as a way to learn a new language, but as a means of exploring unfamiliar programming/mathematical methods, structures, and algorithms. This article explains the process I went through to port a fairly complicated poker hand evaluation algorithm from C to Factor.

Problem 54

In order to solve Problem 54, I needed to come up with a way to programatically determine the winner when comparing two poker hands. I had recently read an extensive blog post covering various poker hand evaluators, so I knew that I didn’t want to bother with a naive solution when there were much better options out there. After reading through the different techniques, one stood out to me in particular; it was a relatively small amount of code, it didn’t require a ton of storage space, and it was plenty fast…

Cactus Kev’s Poker Hand Evaluator

Through the power of mathematics (combinatorics in particular), you can ascertain that out of all 2,598,960 possible poker hands (52 choose 5), there are actually only 7462 distinct hand values that you need to be concerned with. The basic idea behind Cactus Kev’s Poker Hand Evaluator is that you can take advantage of this fact by storing a card’s representation in an efficient manner, do some basic bit twiddling, some multiplication (which is computationally cheap), add in a couple lookup tables, and you can determine a hand’s equivalence class value very quickly.

The key concept here is that each of a card’s possible 13 ranks (deuce through ace) is stored as a different prime number, and by multiplying a hand’s five prime numbers together, you’ll get a unique result that can then be used to determine that hand’s overall value. If you handle the special cases of flushes, straights, and high cards, the rest can be boiled down to a simple binary search. You can read the gory details at Kevin Suffecool’s site.

The Porting Process

Bitfield Representation

Each card will be stored as a bitfield, inside of an interger that’s 4-bytes long:

+-------------------------------------+
| xxxbbbbb bbbbbbbb ssssrrrr xxpppppp |
+-------------------------------------+
  xxxAKQJT 98765432 CDHSrrrr xxpppppp
  -----------------------------------
  00001000 00000000 01001011 00100101  =  134236965  =  King of Diamonds
  00000000 00001000 00010011 00000111  =     529159  =  Five of Spades
  00000010 00000000 10001001 00011101  =   33589533  =  Jack of Clubs

x = bit turned off, not used
b = bit turned on depending on rank of card
s = bit turned on depending on suit of card
r = rank of card (deuce = 0, trey = 1, four = 2, …, ace = 12)
p = prime number value of rank (deuce = 2, trey = 3, four = 5, …, ace = 41)

For now, I’m going to ignore how we get cards into this particular format, but I’ll come back to that in a bit.

Constants

Now that we have an efficient means of representing individual cards, we need to establish some basic constants to align with the above bitfield representation and overall hand values…

Constants In C:

#define CLUB     0x8000
#define DIAMOND  0x4000
#define HEART    0x2000
#define SPADE    0x1000

#define Deuce  0
#define Trey   1
#define Four   2
#define Five   3
#define Six    4
#define Seven  5
#define Eight  6
#define Nine   7
#define Ten    8
#define Jack   9
#define Queen  10
#define King   11
#define Ace    12

#define STRAIGHT_FLUSH   1
#define FOUR_OF_A_KIND   2
#define FULL_HOUSE       3
#define FLUSH            4
#define STRAIGHT         5
#define THREE_OF_A_KIND  6
#define TWO_PAIR         7
#define ONE_PAIR         8
#define HIGH_CARD        9

static char *value_str[] = {
    "",
    "Straight Flush",
    "Four of a Kind",
    "Full House",
    "Flush",
    "Straight",
    "Three of a Kind",
    "Two Pair",
    "One Pair",
    "High Card"
};

Constants In Factor:

CONSTANT: CLUB     8
CONSTANT: DIAMOND  4
CONSTANT: HEART    2
CONSTANT: SPADE    1

CONSTANT: DEUCE  0
CONSTANT: TREY   1
CONSTANT: FOUR   2
CONSTANT: FIVE   3
CONSTANT: SIX    4
CONSTANT: SEVEN  5
CONSTANT: EIGHT  6
CONSTANT: NINE   7
CONSTANT: TEN    8
CONSTANT: JACK   9
CONSTANT: QUEEN  10
CONSTANT: KING   11
CONSTANT: ACE    12

CONSTANT: STRAIGHT_FLUSH   0
CONSTANT: FOUR_OF_A_KIND   1
CONSTANT: FULL_HOUSE       2
CONSTANT: FLUSH            3
CONSTANT: STRAIGHT         4
CONSTANT: THREE_OF_A_KIND  5
CONSTANT: TWO_PAIR         6
CONSTANT: ONE_PAIR         7
CONSTANT: HIGH_CARD        8

CONSTANT: VALUE_STR { "Straight Flush" "Four of a Kind" "Full House" "Flush"
    "Straight" "Three of a Kind" "Two Pair" "One Pair" "High Card" }

Here, the only difference I made in the Factor version was to start the hand value types at 0 instead of 1 to eliminate the empty string at index 0…it’s unneccessary.

Prime Number Representations

As mentioned above, each card rank will be represented by its own prime number, so we’ll need a way to reference that…

Prime Numbers In C:

int primes[] = { 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41 };

Prime Numbers In Factor:

CONSTANT: RANK_STR { "2" "3" "4" "5" "6" "7" "8" "9" "T" "J" "Q" "K" "A" }

: card-rank-prime ( rank -- n )
    RANK_STR index { 2 3 5 7 11 13 17 19 23 29 31 37 41 } nth ;

A minor change I made here was to establish a RANK_STR constant to make it easier to develop ways to parse and print card representations later (see the section on printing below for the C equivalent). That said, card-rank-prime in the Factor version has the same purpose as the primes array in the C version, but it performs the lookup for you.

Initializing Cards

Now that we have some basic infrastructure in place, we can tackle initializing cards in the desired format. Cactus Kev’s code glosses over this process and only demonstrates how to generate an entire deck, so I’ll just cover the Factor code here…

Initializing Cards In Factor:

: card-rank ( rank -- n )
    {
        { "2" [ DEUCE ] }
        { "3" [ TREY  ] }
        { "4" [ FOUR  ] }
        { "5" [ FIVE  ] }
        { "6" [ SIX   ] }
        { "7" [ SEVEN ] }
        { "8" [ EIGHT ] }
        { "9" [ NINE  ] }
        { "T" [ TEN   ] }
        { "J" [ JACK  ] }
        { "Q" [ QUEEN ] }
        { "K" [ KING  ] }
        { "A" [ ACE   ] }
    } case ;

: card-suit ( suit -- n )
    {
        { "C" [ CLUB    ] }
        { "D" [ DIAMOND ] }
        { "H" [ HEART   ] }
        { "S" [ SPADE   ] }
    } case ;

: card-rank-bit ( rank -- n )
    RANK_STR index 1 swap shift ;

: card-bitfield ( rank rank suit rank -- n )
    {
        { card-rank-bit 16 }
        { card-suit 12 }
        { card-rank 8 }
        { card-rank-prime 0 }
    } bitfield ;

:: (>ckf) ( rank suit -- n )
    rank rank suit rank card-bitfield ;

: >ckf ( string -- n )
    #! Cactus Kev Format
    >upper 1 cut (>ckf) ;

: parse-cards ( string -- hand )
    " " split [ >ckf ] map ;

To use this code, you’d start with a string representing a card, like "AS" for the Ace of Spades or "7H" for the Seven of Hearts, and then pass it to >ckf to get its integer representation. The parse-cards word simply does this conversion for an entire hand; given a string like "7C 5D 4H 3S 2C", it will output an array of the proper integers:

(scratchpad) "7C 5D 4H 3S 2C" parse-cards .
{ 2131213 541447 270853 135427 98306 }

Due to Factor’s concatenative nature, you tend to write a lot of small words that do one specific thing, and then tie them together. Words are expected to be defined before you reference them (unless they’ve been deferred), so when reading code from top to bottom, you’ll see big-picture words defined after the more-focused words they use. It’s also a convention to name helper words the same as the word from which they’re called, but with parentheses around the name.

One special thing to note is the use of locals in the (>ckf) helper word…if you start a word definition with :: instead of just :, Factor will bind the named inputs (from the stack declaration) to lexical variables from left to right. It’s used in this case because we need to pass these values to the card-bitfield word in a specific order, and shufflers would make this a bit more confusing.

The last thing to explain is the card-bitfield word itself…it looks up the proper bit values for the different parts of our card representation, and then constructs the final 4-byte integer from those values. See the bitfield documentation for more details on its syntax.

Shuffling Decks and Printing Hands

Initializing and shuffling a deck wasn’t strictly necessary for solving the Project Euler problem, but it was a useful abstraction and would be expected in a general poker library. Printing out a poker hand falls under this same category, but I’ve included it here as well since it was part of Cactus Kev’s original code. This is where the languages start to noticibly deviate in their implementations…

Initializing a Deck In C:

init_deck( int *deck )
{
    int i, j, n = 0, suit = 0x8000;

    for ( i = 0; i < 4; i++, suit >>= 1 )
        for ( j = 0; j < 13; j++, n++ )
            deck[n] = primes[j] | (j << 8) | suit | (1 << (16+j));
}

Initializing a Deck In Factor:

: <deck> ( -- deck )
    RANK_STR SUIT_STR 2array [ concat >ckf ] product-map ;

Shuffling a Deck In C:

double  drand48();

shuffle_deck( int *deck )
{
    int i, n, temp[52];

    for ( i = 0; i < 52; i++ )
        temp[i] = deck[i];

    for ( i = 0; i < 52; i++ )
    {
        do {
            n = (int)(51.9999999 * drand48());
        } while ( temp[n] == 0 );
        deck[i] = temp[n];
        temp[n] = 0;
    }
}

Shuffling a Deck In Factor:

ALIAS: shuffle randomize

Printing a Hand In C:

print_hand( int *hand, int n )
{
    int i, r;
    char suit;
    static char *rank = "23456789TJQKA";

    for ( i = 0; i < n; i++ )
    {
        r = (*hand >> 8) & 0xF;
        if ( *hand & 0x8000 )
            suit = 'c';
        else if ( *hand & 0x4000 )
            suit = 'd';
        else if ( *hand & 0x2000 )
            suit = 'h';
        else
            suit = 's';

        printf( "%c%c ", rank[r], suit );
        hand++;
    }
}

Printing a Hand In Factor:

: >card-rank ( card -- string )
    -8 shift HEX: F bitand RANK_STR nth ;

: >card-suit ( card -- string )
    {
        { [ dup 15 bit? ] [ drop "C" ] }
        { [ dup 14 bit? ] [ drop "D" ] }
        { [ dup 13 bit? ] [ drop "H" ] }
        [ drop "S" ]
    } cond ;

: card>string ( card -- string )
    [ >card-rank ] [ >card-suit ] bi append ;

: print-hand ( hand -- )
    [ card>string ] map " " join print flush;

I’m cheating a bit here with the shuffle word, as Factor already implements randomize using the Fisher-Yates algorithm; so rather than re-inventing the wheel, I just aliased the wheel. With Factor, you’ll also notice that count-controlled loops are much less common in comparison to using sequence operators directly on the elements of a data structure.

Lookup Tables

We can almost get to the actual evaluation of hands to determine their relative value, but before that, we’ll need the proper lookup tables. These are all straight-forward arrays, and we have four to consider:

a table for all flushes (including straight flushes)
a table for all non-flush hands consisting of five unique ranks (i.e. either straights or high card hands)
a table of the product of prime values associated with a hand
a table covering the final hand values of all hands not covered by the flushes/unique5 tables

A zero means that specific combination is not possible for that type of hand…

Lookup Tables In C:

short flushes[] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1599, 0, 0, 0, 0, 0, 0, 0, 1598, 0, 0, 0, 1597, 0, 1596,
...
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1
};

short unique5[] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 1608, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 7462, 0, 0, 0, 0, 0, 0, 0, 7461, 0, 0,  0,  7460,  0,
...
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0,  0,  0,  0,  0, 0, 0, 0, 0, 0, 0, 1600
};

int products[] = {
48, 72, 80, 108, 112, 120, 162, 168, 176,  180,  200,  208,  252,
264,  270, 272, 280, 300, 304, 312, 368, 378, 392, 396, 405, 408,
420, 440, 450, 456, 464, 468, 496, 500, 520, 552, 567, 588,  592,
...
66737381,   71339959,  73952233,  76840601,  79052387,  81947069,
85147693, 87598591, 94352849, 104553157
};

short  values[] = {
166, 322, 165, 310, 164, 2467, 154, 2466, 163,  3325,  321,  162,
3324,  2464,  2401,  161,  2465, 3314, 160, 2461, 159, 2400, 320,
3323, 153, 2457, 6185, 2463, 3303, 2452,  158,  3322,  157,  298,
...
1676, 14, 168, 2469, 2468, 1611, 23, 1610, 13, 179, 12, 167, 11
};

Lookup Tables In Factor:

CONSTANT: flushes-table
{ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 1599 0 0 0 0 0 0 0 1598 0 0 0 1597 0 1596 8 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1595 0 0 0 0 0 0 0 1594 0 0 0 1593 0 1592 1591 0 0 0 0 0 0 0 0 1590
...
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1 }

CONSTANT: unique5-table
{ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1608 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 7462 0 0 0 0 0 0 0 7461 0 0 0 7460 0 7459 1607 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 7458 0 0 0 0 0 0 0 7457 0 0 0 7456 0 7455 7454 0 0 0 0 0 0
...
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 1600 }

CONSTANT: products-table
{ 48 72 80 108 112 120 162 168 176 180 200 208 252 264 270 272 280 300 304 312
368 378 392 396 405 408 420 440 450 456 464 468 496 500 520 552 567 588 592 594
612 616 630 656 660 675 680 684 696 700 702 728 744 750 760 780 828 882 888 891
...
64379963 64992503 66233081 66737381 71339959 73952233 76840601 79052387
81947069 85147693 87598591 94352849 104553157 }

CONSTANT: values-table
{ 166 322 165 310 164 2467 154 2466 163 3325 321 162 3324 2464 2401 161 2465
3314 160 2461 159 2400 320 3323 153 2457 6185 2463 3303 2452 158 3322 157 298
2460 2446 152 3292 156 2398 3321 2462 5965 155 6184 309 2456 3320 2439 3313
...
1622 191 3546 2490 2470 15 2600 25 3326 169 24 1612 2479 1677 1621 1676 14 168
2469 2468 1611 23 1610 13 179 12 167 11 }

Hand Evaluation

Finally, we can get to the actual evaluation of hands…

Hand Evaluation In C:

int findit( int key )
{
    int low = 0, high = 4887, mid;

    while ( low <= high )
    {
        mid = (high+low) >> 1;      // divide by two
        if ( key < products[mid] )
            high = mid - 1;
        else if ( key > products[mid] )
            low = mid + 1;
        else
            return( mid );
    }
    fprintf( stderr, "ERROR:  no match found; key = %d\n", key );
    return( -1 );
}

short
eval_5cards( int c1, int c2, int c3, int c4, int c5 )
{
    int q;
    short s;

    q = (c1|c2|c3|c4|c5) >> 16;

    /* check for Flushes and StraightFlushes
    */
    if ( c1 & c2 & c3 & c4 & c5 & 0xF000 )
    return( flushes[q] );

    /* check for Straights and HighCard hands
    */
    s = unique5[q];
    if ( s )  return ( s );

    /* let's do it the hard way
    */
    q = (c1&0xFF) * (c2&0xFF) * (c3&0xFF) * (c4&0xFF) * (c5&0xFF);
    q = findit( q );

    return( values[q] );
}

Hand Evaluation In Factor:

: flush? ( hand -- ? )
    HEX: F000 [ bitand ] reduce 0 = not ;

: rank-bits ( hand -- q )
    0 [ bitor ] reduce -16 shift ;

: lookup ( hand table -- value )
    [ rank-bits ] dip nth ;

: prime-bits ( hand -- q )
    [ HEX: FF bitand ] map-product ;

: hand-value ( hand -- value )
    dup flush? [ flushes-table lookup ] [
        dup unique5-table lookup dup 0 > [ nip ] [
            drop prime-bits products-table sorted-index
            values-table nth
        ] if
    ] if ;

Each language’s implementation checks for flushes first, then for hands with five unique ranks, and if all else fails, multiplies the primes together, searches for that product in a lookup table, and then uses that index to determine the hand’s final value. The findit function in C is the binary search, and once again I used Factor’s built-in libraries to do the same thing with the sorted-index word.

Hand Ranking

We’re basically done at this point…the hand with the lowest value (from 1 to 7462) wins. If the values are the same, the hands are tied. A simple comparison function can tell you which hand is best.

One last function that’s handy though, is the ability to output what that value actually means…

Hand Ranking in C:

int
hand_rank( short val )
{
    if (val > 6185) return(HIGH_CARD);        // 1277 high card
    if (val > 3325) return(ONE_PAIR);         // 2860 one pair
    if (val > 2467) return(TWO_PAIR);         //  858 two pair
    if (val > 1609) return(THREE_OF_A_KIND);  //  858 three-kind
    if (val > 1599) return(STRAIGHT);         //   10 straights
    if (val > 322)  return(FLUSH);            // 1277 flushes
    if (val > 166)  return(FULL_HOUSE);       //  156 full house
    if (val > 10)   return(FOUR_OF_A_KIND);   //  156 four-kind
    return(STRAIGHT_FLUSH);                   //   10 straight-flushes
}

Hand Ranking in Factor:

: value>rank ( value -- rank )
    {
        { [ dup 6185 > ] [ drop HIGH_CARD ] }        ! 1277 high card
        { [ dup 3325 > ] [ drop ONE_PAIR ] }         ! 2860 one pair
        { [ dup 2467 > ] [ drop TWO_PAIR ] }         !  858 two pair
        { [ dup 1609 > ] [ drop THREE_OF_A_KIND ] }  !  858 three-kind
        { [ dup 1599 > ] [ drop STRAIGHT ] }         !   10 straights
        { [ dup 322 > ]  [ drop FLUSH ] }            ! 1277 flushes
        { [ dup 166 > ]  [ drop FULL_HOUSE ] }       !  156 full house
        { [ dup 10 > ]   [ drop FOUR_OF_A_KIND ] }   !  156 four-kind
        [ drop STRAIGHT_FLUSH ]                      !   10 straight-flushes
    } cond ;

: string>value ( string -- value )
    parse-cards hand-value ;

: hand-rank ( string -- string )
    string>value value>rank VALUE_STR nth ;

Cactus Kev’s code didn’t include the code for explicit output of a string, but he did include the value_str array which could be used for this purpose.

Putting it All Together

To use the poker evaluator directly in Factor’s interactive listener environment, you can enter something like these examples:

(scratchpad) "7H 7S 7D AS AC" hand-rank .
"Full House"

(scratchpad) "7C 5D 4H 3S 2C" "KD QS JC TH 9S" [ string>value ] bi@ > .
t

Solving Problem 54

Getting back to our original goal of solving Project Euler’s Problem 54, we can now quite easily determine how many times Player 1 wins out of 1000 random hands dealt to two players…

: source-054 ( -- seq )
    "resource:extra/project-euler/054/poker.txt" ascii file-lines
    [ [ 14 head-slice ] [ 14 tail-slice* ] bi 2array ] map ;

: euler054 ( -- answer )
    source-054 [ [ string>value ] map first2 before? ] count ;

The source-054 word simply converts the provided text file into the format our evaluator expects and the hand comparisons are subsequently handled by the euler054 word and counted up! Let’s just say that Player 2 is much better at poker :-)

Vocabulary Improvements

Since porting the initial version of the poker vocabulary, many improvements have been made to generalize the code for use beyond just solving Project Euler problems. I ended up dropping the binary search in favor of using the Senzee Perfect Hash Optimization, and other Factor contributors have added code specifically for Texas Hold’em and Omaha evaluation. I didn’t want to cover all of that here, as the focus of this article is the direct comparison between C and Factor, but you can view the current poker vocabulary to see all of these enhancements.

As always, feel free to drop me a line if you’d like to discuss things further.

Magnetic-Based versus Flash-Based Storage

2010-11-01T00:00:00-07:00

This article is the answer I wrote to an exam question for a basic cyber forensics course I took back in Feb 2009. I recently ran across this exam in my files and thought others might be interested in a high-level overview of the differences between magnetic-based and flash-based storage technologies…

The Question

“Briefly discuss the differences between magnetic-based storage and flash-based storage. What effect do these differences have on digital evidence investigations?”

My Answer

Between the two types of disk storage technology of magnetic-based and flash-based hard drives (HDDs and SSDs respectively), magnetic-based drives are more common, with flash-based drives quickly gaining speed in development and adoption rates.

Magnetic-based drives use a series of stacked metal or glass disks called “platters”, which store microscopic magnetic information on concentric rings called “tracks”, which are further split into smaller “sectors” of 571 bytes each. A sector only holds 512 bytes of actual data and uses the remaining bytes for syncing purposes, a header with a location ID, and error detection via CRC checksums. A magnetic read head hovers over the metal platters as they spin, and depending on the electrical impulses that motion creates, you get either a one or zero represented as binary data. A “cylinder” is the combined data for a single track on all of the platters in a drive (both sides of each platter). Once the data is read, on-board hardware decodes that information and transmits it to the computer via a standardized interface.

Flash-based drives are quite different. There are no physically moving parts, and they store data electronically instead of magnetically. Using a pair of transistor gates (a float gate and a control gate), flash-based storage sends electrical impulses to logically segmented areas in memory to store the appropriate ones and zeros in binary format (just like magnetic drives). Because they use only electricity and don’t have to wait for actuator arms to physically move heads into position for reads and writes, flash-based storage is typically much faster at I/O operations. Like magnetic drives, flash-based drives also have a built-in hardware controller to manage its operation and to convert the data into a standardized format to be read by the computer.

The effect each one of these has on a digital investigation is wide-spread. First of all, they both retain data for different amounts of time when left sitting. Magnetic data is not very reliable after a year or so, while flash-based data will usually last quite a bit longer. Flash-based memory also has a limited number of erase cycles, after which, whatever data is stored on that segment will not change. This can actually help in an investigation if you can trick the hardware controller into thinking it can no longer use the memory, you could essentially turn it into a read-only device and preserve evidence. Likewise, magnetic disks often have a residual magnetic signature of previous data, even if it has been overwritten. Sometimes these properties can be exploited by an expert lab with the proper equipment to recover data that was intentionally destroyed.

Both storage mediums will usually not overwrite data when it is deleted, but will simply update an allocation table to specify that a cluster or segment is free for writing again.

Keeping in mind how each of these technologies transforms physical properties into a common, logical storage interface can help you to make better decisions on how to preserve data, how to find evidence, and what to expect when doing an investigation. If you don’t pay attention to how these technologies differ, you have the potential to either miss important information, or in the worst case, destroy the information you do have.

Beginning Factor – Shufflers & Combinators

2008-12-04T00:00:00-08:00

In the previous installment of Beginning Factor, we discussed some of the attributes of stack-based languages and the syntax for defining words in Factor. This time around, I’d like to introduce stack shufflers, quotations & combinators, and touch on some more basic data types and their properties.

Up until now, we’ve essentially been using Factor as an overqualified RPN calculator. I just wanted to make sure that you don’t underestimate Factor because of these particular examples; Factor is an extremely capable and modern language that can be used for everything from web applications, to game development, to complex text parsing, and so on. I’m purposefully using over-simplified examples as a means to demonstrate specific points about the language. Stick with me, and I assure you the examples will gradually get more expressive.

Stack Shufflers

Because of Factor’s stack-based nature, you sometimes need the ability to rearrange or copy items in the stack to ensure that they are in the correct position for future use. The way to go about this is a group of words called “stack shufflers”.

Much like the name implies, stack shufflers are merely words that change the order (or number) of the items on the top of the stack. It is said that shuffle words “control the flow of data between words that perform actions.” In fact, you already know one stack shuffler… drop

There are three basic varieties of stack shufflers. Here they are along with the most commonly used word for each type:

Removing Shufflers

drop ( x -- )

removes the top item from the stack and discards it
Duplicating Shufflers

dup ( x -- x x )

duplicates the top item on the stack
Permuting Shufflers

swap ( x y -- y x )

exchanges the positions of the first and second items on the top of the stack

…and here are a couple of word definitions that use these shufflers appropriately:

: sq ( x -- y )
    dup * ;

: neg ( x -- -x )
    0 swap - ;

(scratchpad) 5 sq .
25
(scratchpad) 1 neg .
-1

Caveats

There are many more shuffle words that support intricate rearrangements and the duplication/removal of multiple items from differing locations within the stack. The problem is, code that is full of stack shufflers can easily become confusing. In general, you should try to minimize the use of stack shufflers to keep things understandable.

Of course there are exceptions to that rule, times when shufflers make the most sense, but the preferred alternative to complicated stack shufflers is the use of a “combinator” that fits your use case. That said, combinators will take a bit of explaining…

Quotations

Before we can get to the idea of combinators, we first have to discuss quotations:

(scratchpad) 5 [ "hello" print ] times
hello
hello
hello
hello
hello

In this example, [ "hello" print ] is a quotation.

In layman’s terms, a quotation is a way to encapsulate a snippet of code so it doesn’t get called right away, but can be passed around on the stack and called later. The computer science term for this is an “anonymous function”. It’s referred to as “anonymous” because it is fundamentally a word that has no name.

If we didn’t have quotations, that same example would get ugly real fast:

(scratchpad) "hello" dup dup dup dup print print print print print
hello
hello
hello
hello
hello

Imagine if you wanted to print “hello” 1000 times!

To create a quotation, you just surround your code snippet with square brackets, and it will be pushed onto the stack: [ ... ]. In order to do anything useful with that quotation once it’s on the stack, what you need is a combinator.

What is a Combinator?

A combinator is just a fancy name for a word that takes a quotation as one of its inputs. In the example above, times is a combinator which takes an integer (n) and a quotation from the stack, and it calls that quotation n times.

Factor uses quotations & combinators extensively for conditionals, sequence traversal, namespaces, closures, and more…but that’s jumping the gun a bit. Before we dive into all of that, I’d like to get back to the idea of minimizing the use of stack shufflers by replacing them with appropriate combinators instead.

Combinators That Express Intent

While our code examples thus far have been easy to follow, when you start tackling more realistic problems, stack shufflers will begin to obfuscate your code. If we can represent our intentions consistently with a combinator instead, then code becomes cleaner and you can consequently focus more on your problem domain and less on mentally organizing your stack.

To illustrate this point, I’d like to introduce a couple of simple combinators:

dip ( x quot -- x ): calls a quotation while temporarily hiding the top item on the stack

keep ( x quot -- x ): calls a quotation with an item on the stack, restoring that item after the quotation returns

While both of these combinators have the same stack effect declaration, their usage is a bit different:

(scratchpad) 1 2 4 [ + ] dip

--- Data stack:
3
4

(scratchpad) clear
(scratchpad) 1 2 4 [ + ] keep

--- Data stack:
1
6
4

These two combinators alone can greatly reduce the number of stack shufflers your code will need. If you’re curious about how these combinators work, they both secretly take advantage of an auxiliary stack (called the “retain stack”) to temporarily store items while the supplied quotation is being executed. There are a few other preserving combinators that are worth exploring as well.

Cleave, Spread, and Apply

The cleave, spread, and apply combinators are your best weapons when trying to reduce the use of stack shufflers while simultaneously expressing intent. The key point being that they should express intent…if you find yourself writing code where these combinators don’t fit logically, then try another option.

Cleave Combinators

These are used when you want to apply multiple quotations to the same set of items on the top of the stack.

Let’s say that you want to find the average of a bunch of numbers in an array. The steps are straightforward, you take the sum of all the numbers and divide that sum by how many numbers you have (the length of the array):
```
(scratchpad) { 1 2 3 } dup sum swap length / .
2
```
We can eliminate the need for those stack shufflers and better express our intent by using a cleave combinator to achieve the same thing:

bi ( x p q -- )

applies quotation p to x, then applies quotation q to x
```
(scratchpad) { 1 2 3 } [ sum ] [ length ] bi / .
2
```
The different cleave combinators change either the number of quotations applied to your items (bi vs. tri), or the number of items used as input for your quotations (bi vs. 2bi).
Spread Combinators

These are used when you want to apply a different quotation to different items on the top of the stack. The spread combinators are closely related to dip, but provide a bit more flexibility while also expressing intent.

Let’s say that you have two coordinate positions in the form of { x y }, and you’d like to extract the x-coordinate from the first position, and the y-coordinate from the second position to form a new position with those values:
```
(scratchpad) { 1 2 } { 3 4 } swap first swap second 2array .
{ 1 4 }
```
We can eliminate the need for those stack shufflers and better express our intent by using a spread combinator to achieve the same thing:

bi* ( x y p q -- )

applies quotation p to x, then applies quotation q to y
```
(scratchpad) { 1 2 } { 3 4 } [ first ] [ second ] bi* 2array .
{ 1 4 }
```
When you want to do the same thing with more than two quotations/items, then using spread combinators eliminates the need for nested dips or shufflers and the added clarity becomes much more evident.

The different spread combinators change the number of quotations applied to the corresponding number of items on the stack (bi* vs. tri*).
Apply Combinators

These are used when you want to apply a single quotation to multiple items on the top of the stack.

Let’s say that you have two strings, each containing a name, and you want to see if those names are the same. In order to ignore case when doing the comparison, you decide to convert both strings to upper case before checking for equality:
```
(scratchpad) "john" "John" swap >upper swap >upper = .
t
```
We can eliminate the need for those stack shufflers and better express our intent by using an apply combinator to achieve the same thing:

bi@ ( x y quot -- )

applies the quotation to x, then to y
```
(scratchpad) "john" "John" [ >upper ] bi@ = .
t
```
The different apply combinators change the number of items on the stack your quotation is applied to (bi@ vs. tri@).

The cleave, spread, and apply combinators are all closely related; if you’re having trouble keeping them apart, try to memorize the naming convention:

If there is no suffix, it is a “cleave”
If the suffix is *, it is a “spread”
If the suffix is @, it is an “apply”

Once you learn these combinators, you should be able to express almost any pattern of complicated stack shufflers. Note that there are also generic forms for all of these combinators that can take additional inputs from the stack. If you find that you resort to using the generic forms more often then not, that’s usually a good indication that you should rethink your approach or put your data into a more appropriate structure.

Data Type Details

Before I turn you loose, I wanted to offer a few extra details about some of Factor’s basic data types…

Sequences

In the cleave and spread examples above, I was sneaky and used sequences without explaining them formally. A sequence is a finite, ordered, collection of elements. Any data type that implements the sequence mixin class (meaning a data type that knows its length and will let you set/get an element at a specific index) gains the ability to use the powerful built-in sequence operators. Read through that documentation to get an idea on how to manipulate sequences and their elements.

Factor has many sequence types that you may already be familiar with, such as arrays (fixed-size mutable sequences) and vectors (resizable mutable sequences), but there are also other data types that you might not expect to be sequences, such as strings. In Factor, a string is merely an array of Unicode 5.0 code points.

Using the sequence operators and combinators together, you can create all sorts of powerful abstractions that I’ll talk more about next time. Here are a couple of examples to whet your appetite:

(scratchpad) { 1 2 3 4 5 } [ even? ] filter .
{ 2 4 }
(scratchpad) { 1 2 3 } [ . ] each
1
2
3
(scratchpad) "Hello" [ alpha? ] all? .
t
(scratchpad) "Hello!!!" [ alpha? ] all? .
f

Keep in mind that in Factor, sequences are zero-based.

Numbers

So far, we have only used integers, but Factor also supports rational numbers (fractions), floats (decimal approximations of a number), and complex numbers (imaginary numbers).

(scratchpad) 100 330 / .
10/33
(scratchpad) 5/4 1/2 + .
1+3/4
(scratchpad) 5/4 0.5 + .
1.75
(scratchpad) 2 5 rect> .
C{ 2 5 }

Iota

One last thing that is going to come in handy, is a word for creating sequences from integers:

iota ( n -- iota ): creates a sequence of length n, whose elements are its non-negative predecessors (i.e. 0 to n-1 inclusive)

This word can be very helpful when performing counted loops or other control flow statements. For example, the following two lines are equivalent:

{ 0 1 2 3 4 5 6 7 8 9 } [ . ] each

10 iota [ . ] each

Next Time

Next time we’ll stick with a theme of “flow” and discuss control flow for your words and after that, the typical work flow when developing code with Factor. Hope you enjoyed this installment!

Beginning Factor – Introduction

2008-11-29T00:00:00-08:00

I’ve been involved in the Factor programming language community for about a year now, and am constantly amazed with how productive its contributors are. Large improvements to the language and its libraries are made on a weekly (if not daily) basis, and it’s finally starting to attract some much-deserved attention from the programming community.

The problem is, the language is a huge departure from the norm for most developers and it can be overwhelming to someone just getting started. I would like to help ease that transition by posting on various topics that I know have been confusing to me over the past year.

Note that many topics are covered in the official FAQ — which is well worth a read — and I won’t spend time covering how to install Factor on the 14 or so platforms it supports, but beyond that, I’ll try to give enough information (from basic to advanced) to get you going.

The Basics of the Basics

First of all, Factor is a stack-based language, which means that it uses a stack to store all arguments and returned values from functions (called “words” in Factor). To put that another way, Factor words don’t receive arguments in the traditional manner; all input values that your word needs are expected to already exist on the top of the stack.

What is a Stack?

If you’re unfamiliar with the data structure called a stack, the concept is fairly simple. It centers around the idea of “Last In First Out” (LIFO), meaning the last item placed onto the stack is the first item that will you will get when you remove an item from the stack. You cannot get to the lower items until the all of the items above it have been removed.

I like to picture a stack like a tower of LEGOs. The only thing you can do with it are add another brick to the top (“push” an item onto the stack), or take the top brick off to use it for something else (“pop” an item off the stack). That’s all there is to stacks!

How to Use the Stack

So, now we know that Factor manages its input and output with a stack, but how do we actually use it? Well, when entering data into Factor’s listener, one of two things is happening:

You are pushing a literal onto the stack
OR
You are calling a word which will consume literals from the stack

Things do get slightly more complicated than that, but for the most part those two rules hold true. The most simple example of this behavior is the ubiquitous hello world program, shown entered directly into Factor’s interactive listener environment:

(scratchpad) "Hello world!" print
Hello world!

To understand what is happening, you can just type the string first, and then execute the print word later:

(scratchpad) "Hello world!"

--- Data stack:
"Hello world!"
(scratchpad) print
Hello world!

In this particular case, typing the literal string "Hello world!" will push that value onto the stack; then the word print, takes a single string off the stack and writes it to the output stream.

The same idea can be applied to simple arithmetic. Like strings, numbers are also literals, so you just have to type them in (separated by spaces) for them to be pushed onto the stack:

(scratchpad) 2 3

--- Data stack:
2
3

NOTE: the stack is displayed in the listener upside-down from the way you’d think, so the bottom number is actually the top of the stack.

…and if you want to add the top two numbers together, the + word will simply pop two numbers off the stack, add them together, and push the result back onto the stack:

--- Data stack:
2
3
(scratchpad) +

--- Data stack:
5

A Few More Words

If you start messing around in the listener, odds are your stack is going to grow pretty quickly and become unmanageable. There are a few words that are essential to know in order to keep things under control:

drop ( x -- ): removes the top item from the stack and discards it

. ( obj -- ): takes the top item from the stack and prettyprints it

clear ( -- ): removes all items from the stack

(scratchpad) 2 3 -

--- Data stack:
-1
(scratchpad) drop
(scratchpad) 20 5 / .
4
(scratchpad) 2 3 + 6 7

--- Data stack:
5
6
7
(scratchpad) clear
(scratchpad)

Ramifications of Stack-Based Design

From these simple examples, we can observe a couple of important things about Factor:

Postfix Notation

When using a stack, passing data becomes implicit and we can assume that all input needed by words already exists on the stack. This naturally lends itself to using postfix notation because you have to push your data onto the stack before you can use it.

This also makes words more concise and unambiguous when compared to infix notation and eliminates the need for copious amounts of parentheses used by prefix notation languages, like Lisp or Scheme.
```
POSTFIX:  6 5 4 * +
INFIX:    6 + 5 * 4 =
PREFIX:   (+ (* 4 5) 6)
```
With the infix example above, you’d have to know order of operations rules in order to get the correct answer (or use parentheses to force the matter). When using postfix notation, the fact that multiplication is done first becomes explicit. Prefix notation also gets rid of that ambiguity, but becomes messier and harder to type with the more nesting you add.
Calling Words is Implicit

You don’t have to specify that you’re calling a word, you simply use the word. This, combined with postfix syntax, means that you can easily nest words or cut and paste parts of definitions into new words without disrupting the flow of data. This lends to keeping code modular, short, easily testable, and readable.

Your First Word

If you’ve typed in the examples from above and messed around in the listener, then you might want to know how to write your own word rather than just using ones that are predefined. Drawing on what we already know, here’s how to write your first word:

: plus-two ( x -- y )
    2 + ;

If you copy and paste that into your listener, than you can use the word plus-two anywhere you would like to add two to a number on the top of the stack:

(scratchpad) 15 plus-two .
17

Not very exciting, but it gives us a couple more things to talk about…

Syntax Specifics

If you study the word definition, you’ll see that it’s made up of a few elements:

: plus-two ( x -- y )
    2 + ;

A colon (:) is used to start the definition of a word. This is required and must have a space after it.
Right after the opening colon is the name of your word, also required.
Following the name of your word is its stack effect declaration, which is a list of the word’s inputs and outputs separated by -- and surrounded by parentheses. All words must have a stack effect declaration unless it only pushes literals on the stack. The names of elements in the stack effect declaration don’t make a difference, only the number of elements. That means that ( elt elt -- seq ) and ( x y -- z ) are the same thing. There are some common conventions for these names, but don’t get caught up by it as I’ll talk more about them in a later article and it won’t change how your program runs.
Next comes the word definition itself, in this case 2 +.
And the last item is a semicolon (;), used to end the word definition. This is required and must have a space before it.

The reason that everything must be surrounded by spaces is that there aren’t any syntax-only elements to Factor…everything is a word! The colon/semicolon, the parentheses, etc. are all just parsing words working together in order to create the syntax.

Next Time

Next time we’ll talk more about stack shufflers, quotations & combinators, details about more datatypes, and more…

If something is unclear or if you’re having any trouble, let me know and I’ll try to help out!

Comments Missing in WordPress Dashboard After 2.5.1 Upgrade

2008-04-25T00:00:00-07:00

I just upgraded the WordPress installation for ElasticDog to version 2.5.1 and noticed that the Manage Comments page in the administrative Dashboard was not displaying any comments. I only noticed there was a problem when the interface kept telling me there were 9 comments awaiting moderation, yet it would not let me see them. Everything was displaying correctly on the publicly viewable blog pages, but no comments showed up when trying to access them via the Dashboard.

After searching for a bit and checking my sever logs, I found that the upgrade script did not add the proper index to the “comment_date_gmt” column in the comments table for my WordPress database:

Apr 25 18:22:17 php-cgi: WordPress database error Key 'comment_date_gmt' doesn't exist in table 'wp_comment' for query...

The Solution

If you don’t want to wait until the upgrade script gets fixed in a later WordPress release, you’ll have to add the missing index to the database table yourself to get things working again. I’ll demonstrate how to do that using the MySQL command line interface, however, if you’re more comfortable with using phpMyAdmin, see the steps outlined here.

Check for Proper Privileges

First of all, you must have the proper privileges on your WordPress database in order to add an index. To check that, follow these steps:

$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 59202
Server version: 5.0.51a Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

Specify that you want to access the “mysql” database, then ensure that you do have the privileges to add indexes.

mysql> use mysql
Database changed
mysql> SELECT user, index_priv FROM user;
+------------------+-----------------+
| user       | index_priv |
+------------------+-----------------+
| root       | Y          |
| elasticdog | N          |
+------------------+-----------------+
2 rows in set (0.00 sec)

As long as you see a ‘Y’ next to your username, then you should be good to go.

Add the Index to the Comments Table

Once you have access to an account that can add indexes, then follow these steps to add the missing index:

Specify that you want to access your WordPress database. If you don’t know the name of your WordPress database, run show databases; to see a list of databases you have access to.
```
mysql> use elasticdog_wordpress
Database changed
```
Add the index. The table names for WordPress are fairly standard, but if you don’t have one named “wp_comments”, run show tables; to see a list of contenders and it should be easy to figure out.
```
mysql> CREATE INDEX comment_date_gmt ON wp_comments (comment_date_gmt);
Query OK, 458 rows affected (0.14 sec)
Records: 458  Duplicates: 0  Warnings: 0
```
If want to verify that the index was created, run
SHOW INDEX FROM wp_comments; to list all indexes, and then you can exit out.

If everything went as planned, all will be right in the world and your comments should be viewable through the dashboard once again!

HOWTO: Install WordPress on Nginx

2008-02-27T00:00:00-08:00

A few years ago, I moved all of my sites from a cheap shared host to a shiny new VPS, and have never looked back. As long as you’re comfortable on the command line, running your own VPS gives you a great amount of flexibility since you have full root access on the server and can configure everything exactly how you want it. That said, it also means that if you want to maintain high performance, you have to keep your resource usage to a minimum.

Apache is a very well-establish web server that can handle just about any situation. Unfortunately, that flexibility comes at the cost of size and relatively high demands on server resources. Nginx (“engine x”) is a lightweight web server/reverse proxy that is very efficient and perfect for hosting WordPress. Read on to see how that can be done…

Step One: FastCGI

First off, Nginx does not provide FastCGI for you (FastCGI is what your web server uses to interact with WordPress’s PHP code), so you’ve got to have a way to spawn your own FastCGI processes. My preferred method is to use the spawn-fcgi program provided by the lighttpd project. You can use PHP’s built-in FastCGI manager php-cgi to do the same thing, but it’s not as straight-forward. Plus, if you learn how to use spawn-fcgi, you can easily adapt it for use with other non-PHP web applications that require FastCGI.

Install spawn-fcgi

To download and install spawn-fcgi, either use your native package manager (this would be the recommended approach), or run the following commands to install the package from source. All of the building happens in your current directory and the binary will be installed under /usr/local/bin/ with a man page under /usr/local/share/man/.

$ wget http://www.lighttpd.net/download/spawn-fcgi-1.6.3.tar.bz2
$ tar xvjf spawn-fcgi-1.6.3.tar.bz2
$ cd spawn-fcgi-1.6.3/
$ ./configure
$ make
$ sudo make install

NOTE: If you’re following the steps above verbatim, you will need to have root privileges in order to perform the final instalation step…everything else should work fine as a normal user. To gain root privileges, the program sudo was used in the example above; you may or may not have access to sudo on your machine.

After spawn-fcgi has been installed, you can safely remove the build directory and original source file:

$ cd ..
$ rm -rf spawn-fcgi-1.6.3/
$ rm spawn-fcgi-1.6.3.tar.bz2

Run spawn-fcgi

This part will be fairly distribution-specific, but I’ll provide the basic command that you’ll need. What you want to do is find a way to run this command as part of your init scripts so the processes will be spawned automatically when you reboot your server.

/usr/local/bin/spawn-fcgi -a 127.0.0.1 -p 53217 -P /var/run/fastcgi-php.pid -- /usr/bin/php-cgi

-a <address> → the IP address to bind the processes to; in our case we want the localhost
-p <port> → the port number to bind the processes to; pick whatever you want that won’t cause a conflict (technically it would be best to pick a random number between 49152 and 65535), just make sure to remember the number and use that same port for your Nginx configuration file later on
-P <path> → the location where to save the process id file; you can use this file to easily kill the processes later
-- <fcgiapp> → the absolute path of the fcgi-application to spawn; in our case we want “php-cgi”, which is provided by your distribution’s PHP package. If you don’t know where to find it, try running which php-cgi on the command line.

For better security, you can also spawn the processes as a non-privileged user by specifying the user/group with the -u and -g flags respectively. For more information on all the available options, run spawn-fcgi -h or man spawn-fcgi on the command line.

If you’re interested in seeing a complete example init script that I wrote for use with Arch Linux, you can download it here: fastcgi-php

Step Two: Complete 83.33% of the Famous 5-Minute Install

Next you should download the WordPress files and extract them to their final location on your server. Simply follow steps 1-5 of the Famous 5-Minute Install (the 6th and final step requires that your web server be up and running properly, so we’ll do it later). This guide will assume that you extracted the WordPress core files here: /srv/http/nginx/example.com/

Step Three: Nginx Configuration

To get the web server up and running properly, the file you need to edit is called “nginx.conf” and is installed in different places depending on your Linux distribution. If you install Nginx from source, the default location is /usr/local/nginx/conf/nginx.conf, however yours may be somewhere else like /etc/nginx/conf/nginx.conf.

Once you find that file, open it with your favorite text editor and add a server declaration that looks something like this (I’ll cover what each part means after posting the code):

server {
    listen       192.0.32.10:80;                # your server's public IP address
    server_name  example.com;                   # your domain name
    root         /srv/http/nginx/example.com/;  # absolute path to your WordPress installation

    try_files $uri $uri/ /index.php;

    location ~ \.php$ {
        include        fastcgi_params;
        fastcgi_pass   localhost:53217;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    }
}

Download the Source Code: nginx.conf.txt

You will need to edit the configuration above using your own information. The first part is merely the server declaration where you define what your server’s publicly available IP address is, and what domain name that address is associated with. Next we add some default settings which point Nginx to the absolute path of the WordPress installation on your server.

After that, the try_files directive automatically checks for the existence of files in order, and returns the first file that is found. The key part here is that WordPress uses the “Front Controller” design pattern, meaning that any request for a file that does not exist on the server should be handled by the main index.php file. So that’s exactly what we do…if the requested URI exists on our local filesystem (images, stylesheets, etc.), then we’ll serve it up directly; if the requested URI does not exist on the filesystem, we forward that request on to index.php.

Last, we add a location block that tells Nginx to dynamically forward PHP requests to the FastCGI processes we spawned earlier. You just have to make sure that the port number matches what we did above. That’s it!

Step Four: Finishing Up

Everything should be good to go…all you need to do now is start your Nginx server process (another distribution specific command), then complete the 6th step of the Famous 5-Minute Install and you should have WordPress up and running on Nginx!

If any of this needs further clarification or you’re just having trouble, drop me a line and I’ll see what I can do to help…

Current Status of the Code Viewer WordPress Plugin

2008-02-23T00:00:00-08:00

This site has been neglected for a long time, and that negligence has carried over to the development of the Code Viewer WordPress plugin as well. While I’ve been away, Code Viewer has been picked up/enhanced by a few other developers — namely AJ and Håkan Carlström — and has gained many new features…

Code Viewer now supports syntax highlighting using GeSHi, can display specific line numbers, and has user-configurable scrollbars (among a few other niceties). On top of those changes, the default code path variable can now be configured directly in WordPress’s admin interface rather than editing the plugin code by hand…very nice!

Reactions

I don’t agree with all of the design choices they made with the new changes, but my qualms are minor. Specifically, I don’t like some of the chosen defaults, nor the new format when specifying viewcode tags:

[viewcode ] src=/code/example.txt link=yes[/viewcode]

…in comparison to the original:

<viewcode src="/code/example.txt" link="yes" />

That said, it is a cool feeling to find out that other people appreciate your work and chose to carry on with its development in your absence.

Passing the Torch

With the latest design incarnation here at ElasticDog, I decided to no longer use Code Viewer for displaying code snippets. Instead, I went with the plugin Code Markup, which allows me to easily embed custom markup within code blocks. It’s a completely different way of going about displaying code in blog entries when compared to Code Viewer, but it does the job well and suits my needs for the time being.

Although I haven’t actually worked on the development of Code Viewer since late 2004, I did register to have it hosted at the WordPress Plugin Repository. Unfortunately, I never got around to packaging it up for general consumption.

Trying to not be a slacker anymore, I have contacted Håkan in hopes of persuading him to take the reins and officially manage the plugin at the centralized WordPress repository. If he agrees, you should be able to download the plugin here very shortly: http://wordpress.org/extend/plugins/code-viewer/

In the mean time, you can find the most up to date version of Code Viewer at Håkan’s site. Happy coding!

Optimal Title Functionality Now in WordPress Core

2008-02-20T00:00:00-08:00

I wrote Optimal Title three and a half years ago to help fix a blatant flaw in WordPress’s default usability and search engine optimization (SEO). Since then, the plugin has been downloaded by tens of thousands of people and written about by hundreds of bloggers. Soon, the plugin will become obsolete…

A Change in WordPress v2.5

As of the beginning of this year, a changeset was checked in to WordPress’s SVN repository which adds a “separator location” argument to the built-in wp_title() function. What that means, is that when WordPress v2.5 is released around March 10th of this year, users will be able to change the ordering of their HTML title tags without having to use a third-party plugin.

The developers decided not to change the default behavior of wp_title(), and instead added the additional argument to the function in order to prevent breaking existing themes that rely on the current behavior.

How to Make the Change

If you’re currently using Optimal Title and want to change your template back to using wp_title() after upgrading to WordPress v2.5 (and you should), here’s what you need to do…

Using the simplified example from the original Optimal Title article, you should currently have something like this in your theme’s header.php file:

<title><?php optimal_title(); ?> <?php bloginfo('name'); ?></title>

Just replace the function call optimal_title() with a call to wp_title('»',TRUE,'right'), and you’ll end up with something like this:

<title><?php wp_title('&raquo;',TRUE,'right'); ?> <?php bloginfo('name'); ?></title>

That’s it!

Parameters

The wp_title codex page won’t be updated until the next release, however, here is how wp_title() will work after WordPress v2.5 is out:

<?php wp_title('separator', display, 'seplocation'); ?>

'separator' – string – The text to place between portions of the page title, such as the blog name and the category. Defaults to '»' (»).
display – boolean – Should the title be displayed (TRUE) or returned for use in PHP (FALSE). Defaults to TRUE.
'seplocation' – string – The location of the separator in relation to the title. All values default to the standard left position, with the exception of 'right', which will place the separator after the title rather than before.

I’m Glad

Even though it took so long to get this done, I’m glad WordPress finally came around to the idea and added Optimal Title’s functionality to the core. Regardless of whether or not you were using Optimal Title before, I’d highly recommend that all users make this simple change to their templates in order to take advantage of the many benefits that a proper title tag has to offer.

HOWTO: Have Multiple Users Share an iTunes Library on One Machine

2007-04-29T00:00:00-07:00

I recently purchased a new MacBook (never having owned a Mac before), and have been reading up on keeping things secure. Per one of those security-related suggestions, once everything was set up to my liking, I stopped using the admin account for everyday use and started forcing myself to log in as a regular user. I’ve also created a user account for my live-in girlfriend so she can go wild customizing her desktop with pictures of flowers, Hugh Jackman, Hugh Jackman laying in a bed of flowers, and so on, without making my eyes bleed.

The trouble is, we both buy music from iTunes on a regular basis using our own individual Apple ID accounts, and there isn’t an easy way to share our music seamlessly with one another. We have similar tastes in music and we’d like it so that when one person buys a new song, it is immediately available for the other person to listen to. Ideally, we’d also be able to create our own unique playlists and assign ratings to the various songs to reflect our own personal likes/dislikes. This article lays out the details of how I accomplished that goal…

The Problem

The main problem with assigning two iTunes Libraries to the same location comes down to permissions. By default, when you download music through iTunes, it assigns permissions based on the user that’s logged in. It only allows that user and people in the UNIX group “wheel” to read the music file (if you’re UNIX-literate, that means it has 640 file permissions). Since a different user (in this case, my girlfriend) would not fit into those categories, she doesn’t have the proper permissions to read any of the music I download. To put it simply…music needs to be world readable.

The Solution

These permissions can be easily fixed by running a simple script that uses the chmod command to modify your music so anyone can play it. To give you a breakdown, the script runs a few safety checks to ensure that you have the proper permissions to modify the files, it then checks that your specified library actually exists, and then it recursively looks at all of your music making sure it is world readable.

The trick is getting the script to run only when necessary as not to waste resources. You could set up a cron job, but then it would have to run on a fixed time table. Either it would run more often than necessary, or not often enough…so cron is out. You could set up a launchd/launchctl configuration file and specify it to run when your Library folder is modified, but unfortunately that doesn’t get triggered when a sub-folder gets modified…so launchd is out.

The key ends up being something called a “LoginHook” that is configured in the /etc/ttys file. Creating a LoginHook will ensure that whenever any user logs in to the machine, the script will be run and proper permissions will be set on the entire library.

Putting it All Together

This should go without saying, but I’m saying it…back up all of your music files before doing anything else! Once that’s done, proceed with the following steps:

First you must pick a centralized location that all your users have access to and make sure the directory exists. I picked /Users/Shared/Music/iTunes and created it on the command-line using:
mkdir -p /Users/Shared/Music/iTunes
Next, you must adjust your iTunes settings to point to the new location. You do this by launching iTunes, then navigate to iTunes » Preferences… » Advanced, and under the General tab, click the Change… button next to the “iTunes Music folder location” box. Navigate to your newly-created centralized folder (in my case /Users/Shared/Music/iTunes), and click Choose.
For ease of maintenance and to prevent duplicate files from being generated, make sure the checkbox labeled “Copy files to iTunes Music folder when adding to library” is not selected (this is also under the Advanced section of the Preferences dialog box).
Click OK to accept the changes you made to your preferences.
To copy your current library to the new location, navigate to Advanced » Consolidate Library… then proceed by clicking Consolidate. This step only needs to be performed one time per user to ensure that everything gets moved over to the new location.
Repeat these configuration steps for all users that you wish to consolidate libraries for.

At this point, you’ve done all that is necessary to do within iTunes, and all that is left is fixing the file permissions. To do that, log in as a user with administrator privileges and follow these steps:

Download the script and put it anywhere you want. I made another directory called /Users/Shared/bin/ and put the script there, but the location doesn’t really matter.
Modify the script’s LIBRARY variable (in the CONSTANTS section) to point to the centralized music directory you created above.
Ensure that the script is executable by running:
```
$ chmod 755 /Users/Shared/bin/share-itunes
```
Then edit the /etc/ttys file with your favorite text editor, and change the line starting with “console” to add:
-LoginHook /Users/Shared/bin/share-itunes
right after the “loginwindow” declaration. Make sure you actually point it to the location where you downloaded the script to. The line should have looked something like this before you edited it:
```
console "/System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow" vt100 on secure onoption="/usr/libexec/getty std.9600"
```
…and look like this after you edit it:
```
console "/System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow -LoginHook /Users/Shared/bin/share-itunes" vt100 on secure onoption="/usr/libexec/getty std.9600"
```
Save the file, and close your text editor.

That’s it! Everything should be all set to go and your permissions will be checked each time you log in to the computer.

Caveats

The only thing I wasn’t able to automate was adding new music purchased by my girlfriend in to my library so it actually shows up when I launch iTunes. For all I know, there may be an easy way to automate it, but I haven’t found one at this time. Luckily it’s a fast/easy procedure to do by hand…

To actually have new music purchased by another user show up in your library, all you have to do is launch iTunes, go to File » Add to Library…, then navigate to your shared music folder and click Choose. It will take a few seconds to parse through your library, but after it’s done, all of the music should be available for you to play. You will have to do that every time you know that music has been added by another user, but everything else should be automated for you.

Why it Works

Unless you get tricky, iTunes always stores your album artwork, playlists, and song rating information in a file under your home directory (well, in ~/Music/iTunes, to be precise). It also stores a meta file with information regarding what it thinks should be included in your library. These files will be unique to the user even if you choose to share your actual music collections. So when you update your iTunes preferences to move the location of your Library, you still retain the ability to have all your own custom settings regardless of what other users add to the common music folder.

By periodically telling iTunes to add the common music directory in to your library, you’re simply forcing it to update the library meta file so it reflects whatever actually exists in that directory…regardless of whether or not you were the person to purchase the music.

Hopefully these steps will be able to help someone else out there having a similar issue. When I first started looking in to how to accomplish this configuration, I though surely it would be easy and done before, but alas, things got a bit more complicated than I had hoped. If you know of a better way, please do leave a comment and let us know what works for you. If you have any trouble understanding these procedures or getting it to work, let me know and I’ll see what I can do to help out…happy listening!

New Flat-Panel Displays using Ceramics

2007-03-31T00:00:00-07:00

While waiting for my tax refund, I’ve been checking out flat panel TVs and I ran across this article on a new ceramic flat panel technology in Popular Science:

“Researcher Steve Yando has invented a ceramic panel that converts video signals into moving pictures. Yando did not have to invent a new way to convert electricity into light; flat electroluminescent panels to do that have been on the market for years. The problem was scanning: how to switch individual points of the panel on and off very rapidly to build up the quick-changing light-and-dark pattern of the TV image. …Yando solved the problem by pulling a very cute trick with piezoelectric crystals, which are old and long-used electric devices.

Piezo crystals convert mechanical movement to electricity, or electricity to mechanical movement; they work either way….Yando’s trick was to make a piezo crystal work both ways at the same time — convert electricity to motion, the motion to electricity — and combine that with an electroluminescent panel to get light.

The dinner-plate picture panel is a flat rectangle of piezo ceramic (lead zirconate-titanate) coated with electroluminescent phosphor (a powder similar to fluorescent-lamp coating). Electrodes at the sides of the ceramic bring scanning voltages that determine which point will light up. Electrodes on the back and face of the panel bring TV picture-signal voltages that determine how bright the point will light up. (The face electrode has to be transparent so you can see the picture that is created.)

The scanning voltage causes a mechanical vibration to move across the ceramic in a line – like a long ocean wave. The vibration wave, in turn, generates a piezo voltage in the ceramic. So a line of voltage moves across the panel. But suppose you feed in two scanning voltages simultaneously, one to the top edge of the panel and one to a side edge. This gives you two moving lines of vibration waves and two moving lines of piezo voltage. Where the lines cross, you get a moving point of double piezo voltage. You can control the course of this point – make it scan across the panel – by doing things with the circuits in the TV receiver.

The electroluminescent layer will light up whenever there is a voltage. But you don’t want the moving lines to show at all, and even the moving spot must adjust it’s brightness to suit the TV picture signal. So a varying resistance cuts out all voltage at or below the double-strength piezo voltage – even the moving spot, unaided, won’t light up. But the additional voltage supplied by the picture signal can make the moving spot glow, changing it from dim to bright to create the entire image."

Though the article is a little old and lacking on display specifications, they did mention that a full-scale prototype has already been built. So I may end up waiting another year or so to see if this develops before looking for a new TV.

If anybody has any more info on ceramic flat panel TV technology, please drop me a line at aaron [at] elasticdog dot com.

Update

Whew…talk about an April Fools Day-spurred adrenaline rush! I think we duped literally TENS of people with that one (not much response from Digg, Fark, or Shacknews). This entry was the brain-child of my diabolical work friend James whom found this article in a 1961 version of Popular Science. Here are scans of the original article: page one and page two. Hopefully next year we can get more traffic coming our way if we want to pull a successful prank on the internets. Live and learn…

Upgrade in Progress

2005-06-30T00:00:00-07:00

I’m currently in the middle of upgrading my WordPress installation to 1.5.1.3, and thus things might not be working quite right around here for a bit. I’m not dead, and I will be going back to my original design once I can take to time to develop a theme properly. With all all of the hacking I did to the default installation of WordPress 1.2.1, it could be a little while. Bear with me…thanks

WordPress Design Sandbox

2004-11-10T00:00:00-08:00

When creating a website, it’s best not to make experimental changes to your code directly on your server for the whole world to see. What if you accidentally mess things up beyond all recognition and don’t remember exactly what you did? It’s a good idea to test developmental changes in a protected environment where you know they won’t effect anything that’s mission critical. Typically this is simple to do as you can just edit the files on your local machine before uploading them. Making a test bed for WordPress, however, is a bit more difficult due to the nature of dynamic page generation.

WordPress uses PHP to pull data from a database, and then subsequently creates the needed HTML based off of that data. Without the PHP, your page would merely render as junk. Thus, in order to see a design change made in your CSS, you’re usually forced to upload the file to your server so WordPress has access to its heart and soul. Not anymore…

The Sandbox

In an effort to make things easier on WordPress users wanting to customize the default installation, I’ve created a static version of the default WP template and have placed it in a zip file along with a clean copy of wp-layout.css and print.css. The template includes a sample entry that utilizes many common HTML elements so you can tell exactly what they would look like on your page.

In this developmental sandbox, you can make structural code changes, try out new CSS techniques, whatever you want…all without having to worry about messing up your live site. Once you get things the way you like them, simply update your real files accordingly, and you’re good to go!

Installation

Download wp-design-sandbox.zip
Create a new directory on your local machine in which you’d like to place the test files
Unzip the three files into the newly created directory
Design away

Design by Fire Cleanup

2004-11-05T00:00:00-08:00

Last week, Andrei Herasimchuk of Design by Fire announced a contest to see who could take his site’s current design and “clean it up, optimizing it for best semantic and structural architecture.” Having the final code judged by three industry bigwigs (Dave Shea, Dean Jackson, and Eric Meyer), Andrei is offering the winner their choice of any three Adobe products. It’s a widely known fact that software from Adobe is like crack to designers, so I figured I’d have a go at it.

I by no means would consider myself an “expert” on web standards (at least not in the same league as these guys), but my anal-retentive temperament has provided me with the abnormal patience to sit down and scour the web for details regarding this subject matter. I’d like to think that although I may not have a lot of industry experience, I have read tons about web standards and do enough work on my own to offer advice when it’s needed.

The contest consisted of cleaning up three different pages of code, and also optimizing the CSS involved in creating the look of Andrei’s site. Initially underestimating the amount of time it would take to complete this cleanup, I immediately downloaded the necessary source code and began hacking away.

The Results

After about four days of working on the contest, creating three new images, having two bloodshot eyes, and one neglected girlfriend, I decided my pseudo redesign was complete. I’m sure it’s not perfect, but I can honestly say that it is better.

Overall, my changes make the site much more consistent…not only in its structure, but in the way it’s displayed across different browsers and platforms. It’s also much more accessible, much more semantic, and it tastes like chicken!

The final judging for the contest has yet to take place, but I’m crossing my fingers. Even if I don’t win the software, it has been a good way for me to flex my design muscles — plus there’s the added bonus of three very prominent designers in the industry who will be looking closely at my work.

If any of my readers are interested in seeing the completed code (yes, I’m talking to both of you), the three pages can be found here:

I’ve also created a zip file that contains all of the new code, images, CSS, and source files used in creating my entry. Inside the zip, you’ll also find a changelog that discusses the improvements I made in more detail.

Wish me luck!

Faking Multiple Templates in WordPress

2004-10-07T00:00:00-07:00

One of the things that surprised me when I first switched from Movable Type to WordPress was its lack of multiple templates. In MT you create numerous templates which allow you to change the look of different pages on your site (i.e., the homepage, archive pages, individual entries, etc.). If you want a list of your favorite links to appear on the homepage, but not on the page for an individual entry, you just delete that section of code from the appropriate template.

WordPress handles its pages a bit differently. Everything is based off of a single template, and through the magic of database queries and dynamic page generation, the right content is always shown according to which type of page you’re looking at.

The Problem

By using a single template, the default installation of WP will show the exact same sidebar on every page, and likewise, whenever you view an entry listing, it will always look the same. This one-template structure is viewed as a blessing to some, but an over-simplification to others.

What if you want to show a list of just the entry titles on your archive pages, rather than the entire entry itself? What if you want to hide the sidebar’s calendar on your individual entry pages? There are countless scenarios like these that require a little more flexibility than the default index.php file can give you.

This issue has been discussed many times on the WordPress support forums, but I’ve found most of the proposed solutions to be less than ideal. It’s rumored that the functionality to control these types of things will be built into WordPress 1.3, but for the time being, it seems we’re on our own.

A Common Solution

Many people suggest using .htaccess rules and simply forward visitors to templates other than index.php. I feel this method over-complicates things, and here’s why:

It creates duplicate code when most of the time you’ll have the exact same basic page structure in each template. Not only that, but multiple templates aren’t very easy to maintain if you ever need to change that underlying structure.
There’s no guarantee that all WP users will be able to create, edit, and use .htaccess files. Some hosting companies just don’t allow it.
Writing .htaccess rules can be downright confusing!

My Solution

Using some simple PHP and the functionality that’s already built into WordPress, you can easily control what appears on different page types without having to create multiple templates. The key lies in exploiting a few under appreciated variables…

$single, $cat, $m, and $year are all built in variables used by WP to determine information about the type of page you’re looking at. The $single variable corresponds to individual entry pages, the $cat variable to categorical listing pages, $m to monthly listings, and $year to yearly listings.

When you’re viewing each variable’s corresponding page type, only that variable will be assigned a value. If you’re viewing the main page of your site, then none of those variables will be assigned a value. Thus, a simple test of whether or not each variable has a value can be use to control exactly what is displayed and when. Here’s the basic idea:

if ($single) {
    // individual post contents
} else if ($cat) {
    // categorical listing contents
} else if ($m) {
    // monthly listing contents
} else if ($year) {
    // yearly listing contents
} else {
    // main page contents
}

All you have to do is place the content you want to appear in the appropriate spot. If you don’t want to do anything different in a particular section from how WP handles it by default, just leave out that variable’s test.

How to Apply It

Using one of the scenarios from above as an example, let’s say that you want to show just the entry titles when viewing a categorical listing. All you’d have to do is change the <div id="content"> section of your index.php file to be something like this:

<div id="content">
    <?php
    if ($posts) : foreach ($posts as $post) : start_wp();

    if ($cat) { ?>
        <h3 class="storytitle" id="post-<?php the_ID(); ?>"><a href="<?php the_permalink() ?>" rel="bookmark" title="Permanent Link: <?php the_title(); ?>"><?php the_title(); ?></a></h3>
    <?php } else { ?>
        <?php the_date('','<h2>','</h2>'); ?>

        <div class="post">
            <h3 class="storytitle" id="post-<?php the_ID(); ?>"><a href="<?php the_permalink() ?>" rel="bookmark" title="Permanent Link: <?php the_title(); ?>"><?php the_title(); ?></a></h3>

            <div class="meta">
                <?php _e("Filed under:"); ?> <?php the_category() ?> &#8212; <?php the_author() ?> @ <?php the_time() ?> <?php edit_post_link(); ?>
            </div>

            <div class="storycontent">
                <?php the_content(); ?>
            </div>

            <div class="feedback">
                <?php wp_link_pages(); ?>

                <?php comments_popup_link(__('Comments (0)'), __('Comments (1)'), __('Comments (%)')); ?>
            </div>

            <!--
            <?php trackback_rdf(); ?>
            -->

            <?php include(ABSPATH . 'wp-comments.php'); ?>
        </div>
    <?php }

    endforeach; else: ?>
        <p><?php _e('Sorry, no posts matched your criteria.'); ?></p>
    <?php endif; ?>
</div>

The process would be just as easy to change what appears in the sidebar of your page…just edit the <div id="menu"> section. Once you get the hang of it, these variables can be used to manipulate just about anything.

Taking It a Step Further

Adding in code for each of these page types can get confusing really fast. With that much information packed in to one file, the scanability of index.php drops considerably and it becomes difficult to find what you want when you need it.

To combat this, I’ve taken the fake template concept a step further…rather than listing all of the code directly in index.php, I use PHP includes to import each custom section from an external file. Rather than make this tutorial any longer than it already is, here’s how I set it up:

<div id="content">
    <?php
    if ($posts) : foreach ($posts as $post) : start_wp();
        // Faking multiple templates
        if ($single) {
            require('includes/single.php');
        } else if ($cat) {
            require('includes/category.php');
        } else if ($m) {
            require('includes/month.php');
        } else if ($year) {
            require('includes/year.php');
        } else {
            require('includes/main.php');
    }
    endforeach; else:
        require('includes/nomatches.php');
    endif;
    ?>
</div>

In the external files themselves, simply put what content you want to appear, and that’s it! I hope this helps some of you guys out.

WP Plugin: Code Viewer

2004-09-26T00:00:00-07:00

Code Viewer is a WordPress plugin that pulls source code from an external file and displays it, optionally adding a link to download the code directly. The plugin displays the code with proper indentation, line numbers, and automatic long-line wrapping.

When using Code Viewer, you no longer have to worry about typing countless entity references in order to make your code “HTML friendly” or to simulate tabs…it’s all done automatically with PHP and a little CSS. By inserting line numbers on your page, the plugin also eliminates the problem of long, non-breaking lines screwing up your layout. In addition, line numbers make convenient reference points when discussing code samples.

Last, but not least…because you’re pulling the source code from an external file (rather than having it embedded directly in your entry), the same code can be repeated on numerous pages, and you only have to type it in once. This not only saves you time, but when you update the code in the centralized file, all instances of that code will be updated along with it.

Installation

I am no longer maintaining Code Viewer, please see Current Status of the Code Viewer Wordpress Plugin for more details on where to get the latest version.

Download Code Viewer v1.1
Open the downloaded file in a text editor and configure the $default_path variable (found on line 12), setting it equal to the absolute path of your code folder
Save the edited file
Change the extension of the saved file from .txt to .php
Upload the changed file into your wp-content/plugins/ directory.
Activate the plugin on your WP Admin » Plugin Management page by clicking the corresponding “Activate” link under the Action column.

You’ll also want to add some new classes to your CSS that will control the look of the code listing. Everything (with the exception of auto-indention) works just fine without the CSS; but adding it in makes a huge difference on readability. Feel free to change these values to whatever you like, but this is how I have mine set up:

/* ---( CODE VIEWER FORMATTING )------------------------- */

ol.codelist {
    border: 1px solid #DDD;
    color: #C63;
    font-family: "Courier New", Courier, monospace;
    line-height: 130%;
    padding: 12px 12px 12px 45px;
    margin: 1.5em 0;
    }

ol.codelist li {
    margin: 0;
    padding: 1px 2px;
    }

ol.codelist li.tab0 { padding-left: 2px; }
ol.codelist li.tab1 { padding-left: 26px; }
ol.codelist li.tab2 { padding-left: 50px; }
ol.codelist li.tab3 { padding-left: 74px; }
ol.codelist li.tab4 { padding-left: 98px; }
ol.codelist li.tab5 { padding-left: 122px; }
ol.codelist li.tab6 { padding-left: 146px; }
ol.codelist li.tab7 { padding-left: 170px; }

ol.codelist li.odd { background-color: #FFF; }
ol.codelist li.even { background-color: #F0F0F0; }

ol.codelist li.sourcelink {
    color: #000;
    font: 115% Georgia, "Times New Roman", Times, serif;
    list-style: none;
    margin-left: -32px;
    padding-top: .85em;
    text-align: center;
    }

ol.codelist li code { color: #222; }

Download Source Code: code-viewer.css

Usage

Code Viewer basically searches your entry for a custom XML tag named <viewcode />, that tells the server to look at an external file and parse it line-by-line into an ordered list. It can be placed anywhere a block-level tag is valid, and like any empty element in XHTML, the tag must be properly closed.

To better illustrate its usage, the tag used in this entry (to show the code-viewer.css listing above) was:
<viewcode src="2004/09/code-viewer.css" link="yes" />

Parameters

<viewcode src="URL" link="display" />

The src attribute is required.
- "URL" – string – The URL of the code to display. Note that relative paths are in relation to the $default_path variable set during the installation step.
The link attribute is optional.
- "display" – string – Should the link to the code be displayed ("yes"), or not be displayed ("no"). If the link attribute is left out of the tag completely, the value defaults to "no".

Here are some examples of valid syntax recognized by Code Viewer…note that all of these examples point to the same file (assuming the $default_path variable is set to "http://elasticdog.com/code/"):

<viewcode src="http://elasticdog.com/code/example.txt" link="no" />
<viewcode src="http://elasticdog.com/code/example.txt" />
<viewcode src="/code/example.txt" />
<viewcode src="example.txt" />

Each attribute value must be surrounded with double quotes ("), and not single quotes ('), or the tag will not be recognized.

Version History

v1.1 (10 Oct 2004) – Fixed blank line detection; no longer confused by whitespace characters.
v1.01 (30 Sep 2004) – Update to code-viewer.css; fixed inconsistent font sizing.
v1.0 (26 Sep 2004) – Initial release to the public.

Credits

The idea to use ordered lists in XHTML to present code, originated from Simon Willison’s CSS Numbered Code Listings blog entry. The same idea has been used by other designers, and in particular, a fellow WP plugin author named David House…

Looking at David’s code-snippet system, I fell in love with the idea, but thought his implementation could be improved upon. Code Viewer should be much more efficient in comparison and has the added functionality of optional link inclusion, styling differences for every other line, and better URL & file type support. A big thanks goes out to both David and Simon for their contributions to my latest creation.

Workin’ My Backend

2004-09-21T00:00:00-07:00

I’m under no delusions about the number of readers I have, and yet I feel compelled to write an update on how things are progressing. Call it hope for a self-fulfilling prophecy…after all, if I’m going to start a blog that I actually expect people to read, I have to write entries in it sooner or later, right?

It may look as though not much has changed around here over the past couple of weeks, but I assure you things are moving forward. Admittedly, I have been busy with school and my personal life, and consequently haven’t had much time to devote toward ElasticDog, but I haven’t completely ignored it either.

Not only have I started working on numerous entries, but I’ve also been teaching myself some PHP in an attempt to customize WordPress even further than I already have. Here’s a short list of some of the projects I’ve been tackling:

Convenient Code
A WP plugin that pulls source code from an external file and displays the associated code with line numbers.
Faking Templates with WP
There are about a billion different ways to do this, and I’ve come up with one of my own. With my method, I’m able to control what appears in both the main column and sub column of my site, differing each one for the home page, individual pages, and each of the three archive page types (yearly, monthly, and categorical).
Project JumpDrive
How I found the road to USB flash drive heaven.
Additional CSS Inclusion
A WP plugin that will allow you to include an additional CSS file for a specific entry. That way, you’re not forced to load a bloated style sheet for all of your pages.
Streamlining Code and Planning for Upgradeability
Although this is a never-ending process, I’ve been going through my code, commenting important sections, simplifying structure, and adding more flexibility in preparation for inevitable updates.
Text Editor Showdown
A no holds barred comparison between popular HTML & text editors.

That’s just a sneak peak at what I’ve got brewing over the next couple of weeks. I’ll also be adding the rest of the main site pages (about, portfolio, etc.) as soon as humanly possible. Simply put, there’s much more to come here at ElasticDog, and by the time I’m finished workin’ my backend, even Jennifer Lopez will have something to be envious of!

WP Plugin: Optimal Title

2004-09-08T00:00:00-07:00

This plugin is now deprecated…see Optimal Title Functionality Now in WordPress Core.

IMPORTANT!!!

Due to recent XSS security issues fixed in WordPress 2.1.3, the code for Optimal Title also had to change in order to fix the vulnerabilities. It is recommended that ALL users of this plugin upgrade to version 3.0. Due to changes in how Optimal Title works, that should be the last upgrade you will ever need For more details, see the update details section below…

Optimal Title is a WordPress plugin that mirrors the function of wp_title() exactly, but moves the position of the 'separator' to after the title rather than before. This allows you to have your blog name tacked on to the end of the page title instead of having it appear first.

Having your page information appear before your blog name in the title is advantageous because it provides more meaningful search engine results and browser bookmark names The text that appears between your <title> tags is used to generate both of these things, and will often be truncated when viewed. Because of this, it is more effective to have words directly relating to the content of your page appear before common markers. Not only will the titles be more meaningful, but the they will also be more scannable when being viewed in a list For more information on this concept, see Dr. Jakob Nielsen’s article Microcontent: How to Write Headlines, Page Titles, and Subject Lines.

Installation

Download Optimal Title v3.0 and unzip the archive.
Upload optimal-title.php into your wp-content/plugins/ directory.
Activate the plugin on your WP Admin » Plugin Management page by clicking the corresponding “Activate” link under the Action column.
Follow the usage guidelines shown below…

Usage

In your header.php file, replace the function call wp_title() with a call to optimal_title(). You’ll also want to position your blog name after your usage of Optimal Title in order for this to work as intended. You should end up with a line that looks something like this:

<title><?php optimal_title(); ?> <?php bloginfo('name'); ?></title>

For a more complicated version that checks to make sure the plugin is installed and also includes support for a tagline on the home page, this is actual code I use on this site:

<title><?php if ( function_exists('optimal_title') ) { optimal_title('|'); bloginfo('name'); } else { bloginfo('name'); wp_title('|'); } ?><?php if ( is_home() ) { ?> | <?php bloginfo('description'); } ?></title>

Parameters

Optimal Title uses the same syntax as wp_title, from the default WP installation, and thus functions in the exact same way…

<?php optimal_title('separator', display); ?>

'separator' – string – The text to place between portions of the page title, such as the blog name and the category. Defaults to '»' (»).
display – boolean – Should the title be displayed (TRUE) or returned for use in PHP (FALSE). Defaults to TRUE.

Version History

v3.0 (06 Apr 2007) – Complete rework on how the plugin works; now should be compatible with all WordPress releases no matter the version.
v2.0 (26 Mar 2006) – Updated to be compatible with the latest WordPress release (including WP 2.x version support).
v1.0 (08 Sep 2004) – Initial release to the public.

v3.0 Update Details

Upon the suggestion of Lee Kelleher, the way in which Optimal Title works was adapted in order to be more efficient, future-proof, and secure. The plugin no longer generates the title text itself, but lets the built-in wp_title() function handle the generation, then rearranges things afterward. Because of this change, Optimal Title will have the exact same security implications as the default installation of WordPress itself and should never need to be upgraded again…in other words, if your WordPress installation is up to date, you’ll never have to worry about the security of using Optimal Title. That also means that the plugin should work for ANY version of WordPress, past, present, and future.

A Fresh Start

2004-09-03T00:00:00-07:00

I started the original ElasticDog about a year ago when I became interested in standards-based—web-design. The more I read about the subject, the more I wanted to try things out and to create a standards-based site of my own. It had been years since I had even touched HTML, but I was determined to do things right this time around. I stayed up late nights reading, experimenting, repeatedly failing, and eventually, learning…

Once the initial design had been completed, I was pretty happy with the results, but didn’t know where to go from there. I had successfully created a functional/accessible site using XHTML and CSS; but they say that “content is king,” and I felt that at the time, I had nothing significant to write about.

And so, I kept researching as much as I could on web design, intermittently writing on what came to mind, and fooling around with different ideas that I ran across. In time, I began to develop my own thoughts on the subjects that I was reading about, rather than just taking what others said at face value. It was quickly becoming apparent to me that ElasticDog needed some major improvements.

Repair or Rebuild?

To give you some background info, one of the ideas behind designing with XHTML/CSS is forward compatibility - in theory you shouldn’t have to update your code in order to do a redesign because your content and style are kept completely separate. For a beautiful demonstration of this concept, just check out the css Zen Garden.

With the first version of ElasticDog.com, I had put a lot of thought in to how my code was structured and included many unique features that I really came to like. But on the flip side, I also did some things that were out of the ordinary which prevented me from executing a redesign as easily as I had hoped. For example, instead of having a fixed-width or fluid page, I decided to try my hand at an elastic design, which is a combination of the other two. This added unnecessary complexity to my code and limited some of my future design options. Because of limitations, and others like it, I knew that if I started over again, I could do better.

Around this same time, I was experiencing some reliability issues with my hosting provider, the new licensing structure for MovableType (my former CMS) was causing a ruckus in the web log community, and I had pretty much stopped writing entries all together. It was then that I became determined to mix things up in order to turn ElasticDog into the site that I had always envisioned.

Stepping Back to Move Forward

To kick off this renewal, I went back to the basics by looking at some of my favorite sites and writing down the aspects that made them stand out. How was the site organized? How did the author handle semantics? What characteristics of their design were useful? What features were attractive, or conversely, distracting? Why did the designers make the decisions they made? What things in my previous design did I do right?

Keeping all of those things in mind, I began creating a new design from scratch. I completely restructured my XHTML, cleaned up my CSS, and made everything more user friendly. I paid attention to the smallest of details, and made sure to insert a lot more flexibility into my code so future changes wouldn’t be such a headache.

Once I had an idea of how things were going to look, I went out and read reviews about hosting providers and ended up switching my site over to ICDSoft …a move that I’ve been very happy with.

Now that I had a redesigned look for my site, and a new server to put it on, I needed software to manage it all. Due to a number of reasons, I chose to leave MovableType behind and use the open-source publishing platform WordPress instead. Once the software was installed, I created templates (based on my design) that would automatically generate my entire site using WordPress…and the rest is history!

I Promise, I’m Almost Done

Now that I’ve bored you to death, I can finally welcome you to the new and improved ElasticDog! Not everything is quite polished yet, but I figured it was good enough to get things going again. For the time being, my old entries will not be imported into the new design. Eventually the ideas will work their way back in the archives, but for now you’ll have to do without.

Last, but not least, I’d like to send thanks out to those who gave me support during my initial learning stages and also to those who offered their insight on my latest design. I realize there’s not much here yet, but I now have a better idea of what I want ElasticDog to be, and more importantly, how to get it there.

ElasticDog.com

Use sshuttle to Keep Safe on Insecure Wi-Fi

What is sshuttle?

Previous Method

Installation

Building sshuttle from Source

Configuration

Usage

Known Bugs

Update sshuttle

Conclusion

Shelf-Made Standing Desk

A Standing Desk

My Old Desk

My New Desk

Accessories

Conclusions

Update

Update2

Porting a Poker Hand Evaluator from C to Factor

Problem 54

Cactus Kev’s Poker Hand Evaluator

The Porting Process

Bitfield Representation

Constants

Constants In C:

Constants In Factor:

Prime Number Representations

Prime Numbers In C:

Prime Numbers In Factor:

Initializing Cards

Initializing Cards In Factor:

Shuffling Decks and Printing Hands

Initializing a Deck In C:

Initializing a Deck In Factor:

Shuffling a Deck In C:

Shuffling a Deck In Factor:

Printing a Hand In C:

Printing a Hand In Factor:

Lookup Tables

Lookup Tables In C:

Lookup Tables In Factor:

Hand Evaluation

Hand Evaluation In C:

Hand Evaluation In Factor:

Hand Ranking

Hand Ranking in C:

Hand Ranking in Factor:

Putting it All Together

Solving Problem 54

Vocabulary Improvements

Magnetic-Based versus Flash-Based Storage

The Question

My Answer

Beginning Factor – Shufflers & Combinators

Stack Shufflers

Caveats

Quotations

What is a Combinator?

Combinators That Express Intent

Cleave, Spread, and Apply

Data Type Details

Sequences

Numbers

Iota

Next Time

Beginning Factor – Introduction

The Basics of the Basics

What is a Stack?

How to Use the Stack

A Few More Words

Ramifications of Stack-Based Design

Your First Word

Syntax Specifics

Next Time

Comments Missing in WordPress Dashboard After 2.5.1 Upgrade

The Solution

Check for Proper Privileges

Add the Index to the Comments Table

HOWTO: Install WordPress on Nginx

Update²