chapados.org - Home

Leopard shines the spotlight on xattrs

2007-11-04T19:41:00Z

When Tiger was released back in 2005, Spotlight search technology was all the rage. Several articles appeared showing off the technology and explaining the details. For most people, being able to easily find their files was all they cared about. Others thought it was too slow and immediately disabled it.

If you're a geek like me, you probably got excited when Amit Singh explained how to use the undocumented fsevents API to monitor spotlight events directly. However, although spotlight in Tiger monitors basic file activity, it ignored xattrs (extended attributes). This matters, because extended attributes have major potential to store arbitrary metadata on files. Ever wanted to be able to tag your files? Maybe you've used one of the existing Finder-comment hacks, or a more elegant solution like Tagbot. Perhaps you've experimented with spotmeta, a project started by Ben Summers that made extended attributes searchable through spotlight. In Tiger, this was difficult, since the fs_event system did not monitor changes to extended attributes. A quick look through the bsd kernel source code shows that this has changed in Leopard:

(from xnu/bsd/sys/fsevents.h)

// Event types that you can ask to listen for
#define FSE_INVALID             -1
#define FSE_CREATE_FILE          0
#define FSE_DELETE               1
#define FSE_STAT_CHANGED         2
#define FSE_RENAME               3
#define FSE_CONTENT_MODIFIED     4
#define FSE_EXCHANGE             5
#define FSE_FINDER_INFO_CHANGED  6
#define FSE_CREATE_DIR           7
#define FSE_CHOWN                8
#define FSE_XATTR_MODIFIED       9
#define FSE_XATTR_REMOVED       10

#define FSE_MAX_EVENTS          11
#define FSE_ALL_EVENTS         998

#define FSE_EVENTS_DROPPED     999

Leopard adds the FSE_XATTR_MODIFIED (9) and FSE_XATTR_REMOVED (10) events, making it possible to monitor changes to extended attributes. This might seem like no big deal. However, this opens the door to building custom metadata solutions and integrating them directly into the operating system. I haven't installed Leopard yet, so I'm not sure if Apple is utilizing this underlying technology for anything in Leopard. I checked the online Spotlight metadata attributes, and there is nothing obvious to suggest that Spotlight is making custom extended attributes easily searchable through the MDQuery API. Apple has a habit of adding features to the kernel in preparation for use in later versions of the OS (cough*autofs). This little gem of a feature could bring a project like spotmeta back to life, or inspire a new round of xattr-based metadata solutions. Stay tuned for more information on this topci after I have time to install Leopard.

(07-dec-2007 update: fixed a few typos)

You might need version control if

2007-10-31T16:39:00Z

You're working on a grant or paper with collaborators and your filenames look like this (no, I'm not making this up):

endo07_gaz_D2edits_26oct_2pm_kih_26Oct_700pm_rick_27Oct.doc
The filename would actually be longer, except that then it would exceed the 255 character limit, so you just make it more cryptic instead. Soon you end up with files named:

e07gD2e26o14k26o19r27o.doc

Version control software is not "for programmers only". If you're working on a collaborative writing project, please, pretty please, stop pretending that good version control tools do not exist and learn how to use one of them. While you're at it, ditch word and use a real text editor that supports syntax highlighting and integrates with your version control system. Then you can write with markdown + LaTeX using Pandoc and save yourself and your collaborators from lots of headaches.

ReadyNAS upgrade includes ssh access

2007-08-24T06:22:00Z

If anyone out there is actually checking this site for the latest info on readynas shell access, then know that this will probably be my last post about the topic. ~~Infrant~~ Netgear has finally released a beta version of RAIDiator v4. Highlights include:

RAIDiator 4 is now based on Linux 2.6.
Remote SSH for Support is disabled by default. If you wish to enable SSH access for Support, you’ll need to install the ToggleSSH add-on.
Root SSH support. With the EnableRootSSH add-on, you can now remote login to the ReadyNAS RAIDiator shell as a root user. Initial password for root will be the same as the current FrontView admin password. Please keep in mind that NETGEAR may deny support if you’ve enabled root access.

However, the upgrade is a one-way deal. You can't go back to v3.x with the 2.4 kernel if something goes wrong. Also, keep this in mind:

"NETGEAR Support will in no way provide support for beta releases. All support will come from this beta forum. The Jedi council will try our best to support you here, but please understand that even if have encountered legitimate bugs, bug fixes may not occur in a timely manner, and in some cases if at all. "

Should I upgrade?

It depends... If you're worried about stability, then probably not. Ironically, in terms of stability, I'm pretty sure that the info published here is probably a safer bet than upgrading to raidiator 4 if all you care about is ssh access. They apparently redesigned frontview, but who cares if you do all your administration over ssh. In the end, who do you trust more, me or Netgear? ;-) The choice is yours (see below if you're having a hard time deciding).

I personally have not upgraded my ReadyNAS yet. To their credit, they are already on the 3rd release of the beta. I will probably wait until things are looking stable to apply the upgrade.

IMPORTANT: install the Toggle SSH add-on for 3.x/4.x

One thing you should definitely do is close this backdoor security hole[1] by installing the ToggleSSH add-on. Infrant also announced this vulnerability in their ssh-based backdoor. If you connect your readynas to any kind of network, you should either disable SSH access, or login and disable root ssh login.

[1]: I want to point out that while my name is listed first on the security advisory, practically all of the work was done by Felix. He did some very clever and elegant work to obtain that information, and I was thrilled to be along for the ride.

Take back your readynas

2007-07-24T20:44:00Z

A few people have asked if I’ve been able to compile a working kernel and run it on my readynas. I haven’t attempted this, but the lazy web has come through for all of us. Head over to 360 and learn how to compile a working kernel and install it on the box.

ReadyNAS shell access redux

2007-05-06T04:44:00Z

When I wrote the original article on how to enable shell access to the readynas, I was motivated by the need to solve my backup problems. Given the unfulfilled promises from infrant regarding ssh availability, I was also frustrated, and wanted to share how easy it was to do it yourself. After solving solving the immediate problem, I did not try to develop a more elegant solution. Fortunately, the web is a big place, and I managed to inspire someone else to come up with a better solution that doesn't require removing any drives. This person, whom I will call "D", has asked to remain anonymous.

Protocol

Here is D's method for changing the root password:

This is a simple approach that exploits the ability to create symbolic links (symlinks) while using NFS, and the ability to traverse symlinks while using AFP (Apple File Protocol). It also exploits the fact that /etc/cron.d is writeable by the admin user, which permits arbitrary crontabs to be created. This will probably require a Macintosh, or another platform which can mount AFP shares.

Enable NFS and AFP services (Services -> Standard File Protocols).

Make a share NFS write-enabled and root privilege-enabled (Shares ->NFS)

Make the same share AFP write-enabled for the admin user (Shares -> AFP)

Mount the share using NFS

Create a symbolic link on the share to /etc (etc -> /etc).

Mount the share using AFP, as the admin user.

Create a new crontab file in etc/cron.d/

# example listing for /etc/cron.d/passwd

* * * * * root /usr/sbin/usermod -p '$1$RVWNkJR9$CaniKWqUxyXC3ETsWKrCE1' root

Reboot the device, to restart cron.

Notes

This would not work if the backend software on the readynas was configured properly. It turns out that frontview, which is written in perl, makes system calls directly and executes commands as the admin user. To make life easier on themselves, Infrant allows the admin user to modify key system files such as /etc/cron.d. In fact, the entire frontview interface is owned by admin, so you should be able to mount /frontview that same way that you mounted /etc and modify any of the files that control the web interface. Now that infrant has been acquired by netgear, maybe some of this will get cleaned up. I suspect that is why infrant was promising a 4.0 release of RAIDiator that will include ssh access, and will not be backwards compatible with the current versions of the OS (3.x). Sounds great, doesn't it? Given the amount of time that it takes Infrant to actually deliver on their promises lately, I think that if you want ssh access before 2008, you should probably use the method described above.

Infrant ReadyNAS software

2007-03-23T02:32:00Z

The first quarter of 2007 is almost over and Infrant has still not released any patches that enable ssh access to the readynas. I guess they were probably too busy putting out fires caused by Windows Vista. Those poor bastards.

Judging from the responses to my cross-compiler post (a few spam messages and 1 real response), I think most people would prefer to just download pre-compiled software, rather than go through the hassle of building and using a cross-compiler. In an ideal world, I would bundle up some of the packages I’ve compiled as .deb files that could be installed with dpkg. If there is enough interest, maybe I will make some time for doing that. Until then, I’m just going to redistribute the compiled packages along with the full source code. I’ve read through the license agreements and I’m pretty sure that I’m complying with all the conditions for each package. If I’m not, please tell me. It is not my intention to illegally redistribute this software.

I usually just build things when I need them for some reason. Since I’ve been primarily concerned about backups, I’ve tried rdiff-backup and rsync with extended attributes. I will add new software to this page if/when I end up building it for the readynas. Enjoy!

(Unless otherwise noted, you can install all of the prerequisite software from the debian-sparc repository)

rdiff-backup

rdiff-backup-1.1.9-readynas.tar.bz2 md5: c029e2ab6ad6a99f6b7aec2c7be871ab

This is a great backup program. However, due to the extreme strain it puts on the meager CPU of the readynas, it is not usable on the readynas.

prerequisites: python, pylibacl, pyxattr, librsync

installation

The tar file contains the complete source code, along with the build directory containing the compiled C extensions.

install prerequisites
unpack the tarball
run the setup.py program (as root): sudo python setup.py install

rsync-EA

rsync-2.6.3-readynas.tar.bz2 md5: 1735165cb7ab886a089f0fc42e20766b

This is a patched version of rsync-2.6.3 with all of the patches from apple and lartmaker.nl applied. In addition, I back-ported some very minor changes from rsync-2.6.4 so that I could cross-compile the code. This version has worked well for me so far, and is my primary backup tool. The tar file contains the full source code along with the cross-compiled executable.

installation

unpack the tarball
‘sudo make install’ to install rsync in /usr/local/bin. Alternatively, you can just copy the executable to your favorite place.

Building a compiler for the Infrant ReadyNAS

2007-01-16T20:17:00Z

Having shell access is cool, but the real fun doesn't begin until you can build and run the software that you want on the ReadyNAS. After an embarrassingly long delay, here is the second installment of my series on Infrant ReadyNAS hacks. In this article, I'll show you how to use crosstool to build a gcc compiler on a x86 linux box that you can use to compile software to run on your ReadyNAS.

As opposed to installing a development environment on the ReadyNAS itself, a cross compiler offers a few key advantages:

The ReadyNAS is slow and doesn't have much memory (256 MB by default). Compiling software on the box would take a long time.
It requires minimal modifications to the ReadyNAS. Let's face it, at the end of the day, you probably want to use the ReadyNAS for backups, which means you probably want a stable operating system that is compatible with upgrades released by Infrant. Manually installing the files necessary for a working development environment on the ReadyNAS is neither trivial nor elegant. I'm not sure if it would cause problems with future updates, but it is certainly an ugly solution.

Your first reaction to term "cross compiler" or "cross toolchain" might be that it sounds complicated, especially for an unknown system. I'm not an expert on gcc by any stretch of the imagination. Fortunately, the people who are experts have packaged up their knowledge into a set of shell scripts called crosstool, which mostly automates the process of building a cross compiler/toolchain. In short, Dan Kegel and the crosstool contributors are my heros. They deserve the lions share of credit for their work. I'm just a user reporting results.

Resouces

Access to an x86 linux box with a working build environment.
Crosstool. Go and download crosstool v0.43.
Patches for glibc-2.3.2 [released] by Infrant in accordance with the GPL.
Time. It will probably take a few to several hours to build the tool chain, depending on the speed of your system.

Background

If you haven't already, go read the crosstool howto, I'll wait.

Crosstool configuration for Infrant ReadyNAS

Crosstool knows how to build toolchains for various combinations of glibc/gcc to run on target systems. For the purposes of configuring crosstool, we can assume that the ReadyNAS is just running vanilla sparc-linux. We already know that the ReadyNAS uses glibc-2.3.2. According to the crosstool build matrix, there are only a few versions of gcc that are likely to work.

So, there actually aren't many Infrant-specific modifications that haven't already been provided by Infrant. However, to keep your crosstool directory nice and neat, you might want to set things up as follows:

In your crosstool directory, make yourself a customized copy of demo-sparc.sh

cp demo-sparc.sh infrant-sparc.sh

I was able to successfully build gcc-3.3.6 and glibc-2.3.2, so I recommend using that combination. For example, my infrant-sparc.sh file looks something like this:

#!/bin/sh
set -ex
TARBALLS_DIR=$HOME/downloads
RESULT_TOP=/opt/crosstool
export TARBALLS_DIR RESULT_TOP
GCC_LANGUAGES="c,c++"
export GCC_LANGUAGES

# Really, you should do the mkdir before running this,
# and chown /opt/crosstool to yourself so you don't need to run as root.
mkdir -p $RESULT_TOP

# Build the toolchain.  Takes a couple hours and a couple gigabytes.
eval `cat sparc.dat gcc-3.3.6-glibc-2.3.2.dat`         sh all.sh --notest

echo Done.

Get & Patch glibc-2.3.2

If you just started crosstool at this point, it would download all necessary programs, apply patches necessary for building a cross compiler and build. It really is that easy. In our case, the only thing we want to do differently is use the patched version of glibc-2.3.2 from Infrant. Download glibc-2.3.2 from your favority Debian mirror, and create the following shell script to apply the infrant patches:

apply-patches.sh:

#!/bin/bash

VERSION=2.3.2 # version of glibc

# Patch in the Debian patches:
DPATCHES="$( grep -v '^#' debian/patches/00list | grep infrant )"
for pf in ${DPATCHES}; do
    patchf=debian/patches/${pf}.dpatch
    if [ -s ${patchf} ]; then
      echo "$patchf"
      chmod 755 ${patchf}
      ${patchf} -patch glibc-${VERSION}
    fi
done

Now unpack the tar file, apply patches, and repackage the source:

cd $HOME/downloads
mkdir infrant
cd infrant
wget http://www.infrant.com/download/GPL/glibc-2.3.2.ds1.tar.bz2
tar xjvf glibc-2.3.2.ds1.tar.bz2
cd glibc-2.3.2.ds1
chmod +x prep.sh
./prep.sh
./apply-patches.sh
tar cvf ../../glibc-2.3.2.tar ./glibc-2.3.2
cd ..
bzip2 glibc-2.3.2.tar

These commands download the glibc-2.3.2 from Infrant, unpack the debian distribution, and apply all the patches (prep.sh). The end result is a glibc-2.3.2 directory ready to be built. We then pack up the patched source directory to create a glibc-2.3.2.tar.bz2 file in the download directory. Crosstool will find this file and use it instead of downloading the vanilla glibc-2.3.2.tar.bz2.

Are these patches really necessary?

I'm not sure if this step is necessary, because I never even tried to build the cross toolchain without using Infrant's glibc-2.3.2. I know, I know, I wasn't being very scientific, but I didn't have time to mess around forever with this stuff. Building gcc and friends takes time. If you try it without the patched glibc and it works, please let me know.

Build it

cd $HOME/crosstool
sudo mkdir -p /opt/crosstool
sudo chmod 777 /opt/crosstool
./infrant-sparc.sh | tee infrant-sparc-glibc-2.3.2-gcc-3.3.6.log

You don't need to use tee, but I like to keep a log of the build.

After some time, your build should have completely successfully, and you will have a cross toolchain in /opt/crosstool.

Test it

The right way to test your new toolchain is to use crosstest. Dan Kegel has written up some very nice documentation on exactly how to test gcc using dejavu. I am guilty of not doing this yet. Sorry. The worst part is that I was holding up this article for the test results. Sorry. It seems like there is enough interest in this information to just provide it "as is" for now, with a "works for me" clause. When I get the tests done, I will post another article about testing. Until, then I can only report that it works for the limited number of userland programs that I've tried.

Build stuff

If you're not planning to run the full test suite, you should at least try building a few "userland" programs. You can start with the classic hello world:

/* hello.c : prints "hello world" */
#include <stdio.h>
int main(void) {
    printf("hello world\n");
}

This will at least tell you that you can compile a basic running program. To compile it, save the text above into a file called hello.c, then:

/opt/crosstool/gcc-3.3.6-glibc-2.3.2/sparc-unknown-linux-gnu/bin/sparc-unknown-linux-gnu-gcc -o hello hello.c

Copy the "hello" executable to your ReadyNAS, run and make sure it prints "hello world" on the screen. This works, but it's annoying to deal with the long paths. I like to wrap all the commands necessary for using the cross toolchain into a shell script. Here's an example:

#!/bin/bash

TARGET="sparc-unknown-linux-gnu"
CROSSBIN="/opt/crosstool/gcc-3.3.6-glibc-2.3.2/sparc-unknown-linux-gnu/bin"

CROSS="${CROSSBIN}/${TARGET}-"

PATH="$PATH":$CROSSBIN
export PATH

# build commands below this line
# --------------------------
${CROSS}gcc -o hello hello.c

Unfortunately, according to the crosstool website, the builduserland option is broken for the time being. I've successfully compiled rsync, Python and the C components of rdiff-backup. I promise to post those instructions soon as an example of what is possible.

[UPDATED: 22-March-2007]

I realized that I forgot to include the step for actually applying the patches from infrant. This is now included as the apply-patches.sh shell script.

Infrant ReadyNAS shell access

2006-11-23T08:12:00Z

The Infrant ReadyNAS NV is a great backup server or media server. However, the one critical missing feature that will make any power-user break into a cold sweat is ssh shell/root access. My initial reaction was: Huh!? I’m buying this box to store my precious data and you won’t even tell me the root password or give me shell access? Dubious. I’m sure that this has driven away many potential customers. To be fair, Infrant has promised to add this feature in late 2006, but it’s almost December and it hasn’t happened yet.

As it turns out, gaining root ssh access is trivial, you just need:

Logs from your ReadyNAS
Computer with a free internal SATA port
Knowledge of linux

Don’t try this at home kids

This article is not a step-by-step, copy-and-paste walk-through guide. If you are not comfortable working at a root prompt and have no clue about how linux is configured, then this article will not help you. My intended audience is knowledgeable users who want shell access, but have live data on their ReadyNAS boxes and can’t afford to poke around and screw up their backups.

The system partition

The first thing I did after unpacking the ReadyNAS (no drives installed) was to plug it, connect it directly to my laptop and turn it on. My reasoning was that if the OS runs from a flash memory card, then the system should be accessible even without any disks. This is not the case. Instead, as I had hoped, the ReadyNAS creates a system partition on one of the drives. This means that the problem is essentially the same as that one time when your friend forgot the root password to her linux box and you had to help her “break in”.

Reconnaissance

If the ReadyNAS creates a system partition on a drive, where does that partition live? I’ll give you a hint: Download the logs through frontview and look at them. The file called “partition.log” is a good place to start.

If the ReadyNAS could be booted from a CD and had a monitor and a keyboard, you would just need a linux boot CD and and you’ve have access. It’s not quite that easy, but the drives are very easy to remove. You’ll just need to plug the drive with the system partition into another computer running linux. If you don’t have a linux installed on the system with a SATA controller, try one of the live CDs from Ubuntu or Gentoo. These will even nicely with a PowerMac G5.

Now that you’ve determined where the system partition lives, shutdown your ReadyNAS and remove the drive with the system partition. Plug it in to a computer with an internal SATA controller. Turn on the computer.

Break in (through the unlocked front door)

While you were looking at the log files, you probably noticed that the system partition type is ext3, which is not surprising, since the ReadyNAS runs GNU linux. Mount the partition as ext3. That’s it. You can now modify/create/delete files. However, the engineers at Infrant are clever. Enabling shell access is not as simple as modifying /etc/passwd and putting the drive back in your ReadyNAS.

Don’t steal the marked bills

.enc files

While you’re poking around in /etc, you’ll notice some files with “.enc” extensions. These are encrypted versions of the corresponding files without the extensions. The ReadyNAS updates the .enc files after you make changes to the system through frontview. The catch is that when you boot the ReadyNAS, it apparently compares each normal file with the encrypted version. If they are different, then the encrypted version is used to regenerate the normal file. This means that you won’t be able to modify files that are managed by this mechanism. Trust me, I already tried it. For those following along at home, this rules out:

/etc/passwd
/etc/exports
/etc/sudoers
/etc/inittab

I’m sure we can all dream up a few ways to get around this “security” system. I used the method outlined below.

/root

Anything you add to /root appears to get removed when you put the disk back in the ReadyNAS and reboot the system.

Set a trap

Since the usual targets get reset when you boot the ReadyNAS, one route of attack is to plant a trojan horse that will modify these files after the ReadyNAS boot up. Fortunately, /etc/crontab is not controlled by the security encryption, which makes setting the trap trivial.

Write a shell script to add a user with uid = 0 if the user doesn’t already exist. Add a line to /etc/crontab that executes this script as root every minute or so.

Spring the trap

Once you’re happy with your changes, unmount the partition, shutdown your computer, and transfer the drive back into the ReadyNAS. Turn it on and wait for it to boot up. Wait a few minutes for the cron job to execute, then login as your new root user. You’ll probably want to change the configuration settings so that you can login as a normal user and enable root access via sudo.

Cool… now what?

Well, now you can modify any file you want, install your favorite software, and configure everything exactly the way you want. Slow down. Before you get too excited, let’s think about this for a minute:

The Infrant processor runs @ ~250 MHz. You’re probably not going to want to run your database-backed app off of the ReadyNAS. It can barely handle ssh file transfers without maxing out the CPU.
The OS is a minimal version of Debian Linux. It does not have a working build environment.

Come back and look for the next article, which will cover building a sparc-linux cross-compiler with crosstool.

UPDATE

If you arrived at this page from a search engine and you’re looking for an easier way to enable ssh access that doesn’t require futzing with hardware, read this article.

Infrant ReadyNAS hacks

2006-11-23T07:37:00Z

The Infrant ReadyNAS NV is an extremely capable network attached storage (NAS) device that packs some serious power into a tiny box. If you’re in the market for a RAID storage device or perhaps even a home media server, give it a look. You’ll probably be impressed.

The ReadyNAS supports up to 4 drives in RAID 0,1,5 or Infrant’s proprietary X-RAID configuration. The X-RAID technology allows you to add drives as you go and the machine automagically configures the RAID to give you redundant storage.

The ReadyNAS uses a proprietary SPARC-based CPU/motherboard, optimized for RAID in a small package. The Raidiator OS is a slightly customized version of Debian GNU/Linux. So, when you buy a ReadyNAS NV, you are actually buying a tiny box optimized for life as a home RAID storage server.

Despite the impressive features, I had second thoughts, because the readynas doesn’t provide ssh shell/root access. It also doesn’t support rdiff-backup, rsnapshot, or even a version of rsync that plays nicely with Mac OS X (see this site). Shame on Infrant for thinking that power users will want to use their pre-packaged backup solution. In the end, my need for a small-footprint RAID solution for my music library and online backups won out. I bought a disk-less ReadyNAS from eAegis and one 500 GB Seagate Barracuda ES drive from newegg. I plan on adding a second drive soon, but in the meantime…

Since I was already storing data on a stack of external hard drives, there was no immediate rush to transfer all my data to the ReadyNAS immediately. Thus, I had some time to “play” with it before using it as my production backup system. The result is a series of hacks for the Infrant ReadyNAS that I’ve put together in my spare time over the past couple months.

Outline

In the next few articles, I hope to cover:

The first article covers gaining root ssh shell access. Go read it now.
Building a cross-compiler for the ReadyNAS using crosstool

This process is straightforward using crosstool and building on x86 linux.
Compiling a modified version of rsync that properly stores meta-data from HFS+ filesystems.
Installing Python and the necessary requirements to run rdiff-backup

The ReadyNAS runs Debian Linux, and dpkg is installed. You can download sparc packages and install them. To compile the [latest] rdiff-backup versions required for use with Mac OS X, you’ll need the cross-compiler. This all works really nicely, except that it’s really slow.
For those of you that were [asking], Perl is already installed, so [rsnapshot] shouldn’t be a problem, although I haven’t tried it yet.

Work at home with your SOCKS on

2006-10-22T04:15:00Z

Most scientific articles are published in journals that are not freely accessible to the general public. If you’re in academic science, biotech or big pharma, your respective institutions pay the big bucks to the publishers so that you can access journal content over the internet. However, what about when you want to look something up at home, or access one of these resources from off site/campus? VPN seems to be the typical solution these days, but I honestly think that it’s overkill. Why should you run your entire networking stack through VPN, when you really only want to access a few pages through your browser? The solution: SOCKS.

Publishers usually only allow access to computers within a certain IP range, which includes the computers located at your institution. Therefore, in order to read journal articles at home, we need the publisher’s website to think that we’re accessing their site from a computer at work. All we need in order to pull this off is a program that can implement the SOCKS protocol. SOCKS is a proxy protocol that relays information from a source computer through the proxy to another computer. It effectively allows your computer at home to interact with the internet through your computer at work.

Unless you’re using Windows, you already have the necessary tool installed on your system: ssh. If you’re using Windows, download and install a copy of PuTTY. If you want something closer to *nix, download and install Cygwin. Both of these are free, and IMHO, are essential software on any windows box.

Put your SOCKS on

ssh is one of those extremely versatile programs with so many different features that it’s hard to remember all of them off the top of your head.

To use it as a SOCKS proxy, open a terminal and type to following at the prompt (denoted by $>):

$>ssh -ND 8080 username@host.with.access.com

If you’re on windows using PuTTY, select Start -> Run, then type:

plink -ssh -ND 8080 username@host.with.access.com

In both of these commands, the -N option prevents ssh from executing a command on the remote system, which lets you use this on systems where you can’t access a shell, but can still authenticate via ssh. The -D 8080 tells ssh to act as a SOCKS proxy server, listening on port 8080. You can use a different port if you want.

In your browser, go to the preferences, and find the proxy settings. Scroll down the list and enable the SOCKS server with these settings:

SOCKS server settings
host: localhost
port: 8080
protocol: SOCKS v5

Here are specific directions for commonly used browsers:

Firefox

Open Firefox -> Preferences.
Under the “General” tab, click the “Connection settings…” button. This will bring up the proxy configuration dialog.
Click the “Manual Proxy Configuration” radio button.
Fill in the values for the SOCKS configuration as listed above.

Get the Proxyswitcher Plug-in

This is an extremely useful tool that lets you switch between different proxy settings by clicking a button in your toolbar. I use it all the time when I need to access a journal article from home.

Safari

Open Safari -> Preferences.
Under the “Advanced” tab, click the “Proxies: Change Settings” button. This actually opens the Apple System preferences pane to the proxy settings for the current network setting. Alternatively, you can access these settings, through the System Preferences: Open System Preferences, select “Network”, and click on the “Proxies” tab.
Select the option to “Configure Proxies Manually”.
Select the “SOCKS” proxy.
Fill in the settings as described above. Don’t worry about the protocol, it will automically select SOCKS v5.

Internet Explorer

Sorry, I try to avoid using Windows, and when I do, I use Firefox. If somebody knows what to do, please post instructions and I will update this post. Otherwise, I will update this section next time I’m near a Windows box.

Cleanup

When you’re done, kill the ssh session by pressing Ctrl+C, and reset your browser’s proxy settings to directly connect to the internet.

Before I forget…

In case it’s not obvious, in order for this to work, you need an account with login access to a machine at your institution. If you’re a *nix user, you probably already have an account and know what to do with it. If you’re not sure, think about how you get your email. Do you have a shell account that lets you login to that computer? Yes? Great, use it. Still not sure, contact a systems administrator and ask about getting a shell account on one of the unix terminal servers or file servers. If they refuse, see if you can have access to an account with guest access. Any account that can authenticate via ssh will work (remember the -N option). Strictly speaking, you do not need shell access.

Fun things to do in your SOCKS

Obviously, accessing journal articles from remote locations is only one of several uses for this technique. SOCKS is useful whenever you need to access a service as if you were on a different computer. Utilities such as ProxyChains and tsocks allow you to wrap any network-based utility via a SOCKS proxy. Using SOCKS in this manner turns out to be a convenient way to acheive anonymity on the web. If this interests you, trying searching for Onion routers. After that, get back to work. Isn’t there some article you should be reading?

Upgrade Complete

2006-10-18T04:45:00Z

I just finished migrating my site to mephisto. There might be a few rough spots left to smooth out, but all of the old content that people were actually viewing should available at the same urls, although some things have new homes that should be reflected in the links. If you find a problem please let me know. I’m using a slightly modified version of scribbish, which is one of my favorite basic themes. I had actually ported scribbish for use with Mephisto on my own, but DeLynn Berry released his port yesterday! We used remarkably similar approaches, but since his version encompasses sribbish trunk, I used his version and “back-ported” my changes. Thanks DeLynn!

My motivation for doing this was to make it easier for me to write content, so the plan from here on out is to add some new articles and information that I have laying around in various unfinished states.

Life with Mephisto

2006-09-21T06:08:00Z

No, I’m not talking about the demon. I’m not living in hell, yet. I’m talking about the hot, new blogging cms app that all the cool kids are using. I’ve finally upgraded this site to use some modern technology. If you’re one of the ~100 people who has actually viewed this page in the past couple months, then congratulations, you’re seeing something new! For the rest of you, trust me, it’s better now.

For those of you who don’t know, mephisto is a ruby-on-rails app. Here’s where I stand as far as the hybrid blog+website setup is concerned:

I’m running the trunk version of mephisto, which is upgraded practically everyday - go techno-weenie! To keep up with the changes, I’m using SVK and subversion as described by octopod. Pages are served up with mongrel and apache on a linux vserver. I even setup a one-step deployment procedure using capistrano to move changes from my classic Powerbook G4 Titanium to the vserver. What’s that? I’ve spilled some kool-aid on my shirt? Thanks, I didn’t even notice.

Overall, I’m happy with mephisto for now. I get the following wrapped up in a nice package with a bow:

write text in markdown
do basic version control
easily manage non-blog areas of the site
manage files/images using the assets system

For those interested, in the past I’ve also evaluated radiant, pier, and also xowiki from openacs, but ultimately chose mephisto. Here’s a brief summary:

Radiant seemed like a great option at first, and it powers the Ruby site, so it is clearly capable. However, after some basic testing, reading the mailing list archives, and checking out the “plugin” system, I got the impression that despite the slogan, radiant has not clearly defined what it’s trying to be. I guess the core of radiant is fairly simple, but it’s missing enough features that I wanted (comments, asset management, and basic version control) that I decided to pass for now.

I’m blown away by Pier, and I can envision using it at some point. At this point, I decided against it because it doesn’t support markdown, and the object database persistancy mechanism is still being developed. Still, I think the concept of storing pages as objects is much cleaner than markup text in a database, and will ultimately be more flexible. Also, Pier is written in Seaside, so you can integrate it into an existing Seaside app or embed a Seaside app into Pier. Try doing that with rails. Unfortunately, as stupid as this might sound, one of the things I don’t like about using pier for a CMS app is the external URLs. Since seaside is a continuation-based framework, the URLs store the continuation key. This looks ugly and also breaks the cache/history feature of your browser that denotes which links have already been visited. These are minor concerns for Applications that run in a browser. However, in my opinion , this situation is not ideal for a content-heavy site.

OpenACS is the first web development framework that I ever used. I never used it for a public site, but I was inspired by Phil Greenspun’s book and problem sets. I also followed the success and demise of Arsdigita while I was in grad school. As far as I can tell, (Open)ACS was the original “opinionated” web framework: “use AOLserver+Tcl and Oracle/Postgres and make the db do your heavy lifting, because SQL is easy.” Despite the fact that there are several out-dated packages, the core of OpenACS still has some well-designed pieces. The xowiki package is very cool, but it’s just not as easy to setup, maintain or use as the other two options.

With Mephisto, you know where you stand. Mephisto is a blogging app first, and can also handle simple pages with comments. For the most part it does what you expect it to do, and it doesn’t try to be all things to all people. I give it a thumbs up.

Pier Design Tutorial

2006-05-25T20:55:00Z

Introduction

Lukas Renggli has created an amazing content management system called Pier. It's still in the alpha phase, but the functionality and flexibility make it a truly innovative content management solution. If you've played with wiki software in rails or php, try Pier and Seaside and prepare to experience enlightenment. Currently, the default Pier installation does not take full advantage of the capabilities of CSS. This document describes how to build a simple theme/template for a Pier wiki starting from an existing css framework.

Disclaimer

I'm new to Squeak/Smalltalk, Seaside and Pier. I've learned quite a bit over the past few months, but I'm still a newbie. As a result, my approach might not represent the "best practices" way of customizing Pier. My hope is that in writing this document, people that know more than me will read it and give me some useful feedback.

Pier layout structure

Before we dive in and start writing or editing css code, let's take a look at the style handles that Pier provides by default and how to change them.

A brief introduction to CSS

Cascading StyleSheets (CSS) provide an abstraction layer for specifying the appearance of an HTML document. It's beyond the scope of this document to explain the intricacies of css. Instead, you should check out:

If you collect computer books for a hobby, you should buy Bulletproof Web Design by Dan Cederholm.

What you need to know about CSS

For now, suffice it to say that CSS allow you to specify the appearance of any HTML tag. They also provide "class" and "id" elements that are used to create a hierarchy of style commands. As a general rule of thumb, use an "id" when there is only one occurrence of the object per page. Use a "class" when there are one or more occurrences per page.

Given this general rule, you can get a very good idea of where to start customizing the layout of a page/site by looking through the HTML source for <div id="..."> and <div class="..."> tags.

Pier layout

Pier uses the Seaside web framework. Compared to other web frameworks, Seaside is a web heretic that breaks several conventional rules. For example, most designers are probably used to working with templates containing a mixture of HTML and framework-specific code. Seaside applications use an API to generate HTML. There are no templates. That's right, no templates. This means that as a designer, you only need to worry about the stylesheets. This section provides an overview of the default HTML structure provided by Pier, and how to customize it.

General outline of a page

Pier currently uses classes under the #frameContent id to control the style of different page elements. In css code, '#' is the prefix for ids, and '.' is the prefix for classes.

#frameContent
    .frame
    .header
    .box .views
    .box .commands
    .box .tree
    .content

In Seaside and Pier, the "pages" that are rendered in your browser are made up of several components. In Pier, the main layout components are embedded in a special page component called the 'environment'. It's normally hidden, but you can access it by typing it in the location bar of you browser. In the default case (above), the environment contains header, box->view, box->command, box->tree and content components. The view, command and tree components are actually subclasses of the box component, so they inherit the default css settings from that class.

Changing the css parameters of a Pier component

You can easily change the default values. For example, let's say you would rather specify the header using an id (#header) instead of a class (.header)

navigate to environment/header
click on the 'settings' link under commands
add the text "header" to CSS Name. Note that the CSS Name field controls the <div id> name.
delete the text for CSS Class
click the 'save' button

Where do the default values come from

Pier components are subclasses of PRWidget. Continuing our previous example, here is the class heirarchy for PRHeaderWidget:

ProtoObject
  Object
    WAPresenter
      WAComponent
        MAComponent
          PRWidget
            PRBoxWidget
              PRListWidget
                PRCommandsWidget
                  PRHeaderWidget

and the class methods for PRHeaderWidget:

PRHeaderWidget class
  defaultCommandClasses
  defaultCssClass
    ^ 'header'
  defaultTitle
    label

Notice that defaultCssClass returns the string "header". You can change the default value by changing this string. I'll leave it as an exercise for you to explore the rest in detail. (Hint: defaultCssName appears as a class method of PRBoxWidget.)

Don't work too hard: Get a template

Unless you're a seasoned web designer, there's a good chance that you probably won't create an ideal css template from scratch on the first try. Like the scaffolds created by generators in Ruby on Rails, having a good outline for your css files will get you started down the right path, and help you avoid mistakes. Furthermore, you can avoid several cross-browser css compatibility issues by finding a good starting framework.

Look for design contests

Design contests are a great source for CSS templates. The quality varies, but it's usually high. Many of the designers are trying to make a name for themselves, so their entries are probably worth hundreds to thousands of dollars in development time. Think of design competition sites as a repository for high-quality, open-source css. However, be sure to double-check that you are legally using a theme/design, and give the author credit!

Here are a few design contest sites to get you started:

Typogarden: themes for Typo, a Ruby on Rails-based blogging app.
Textplates: themes for Textdrive, a php-based CMS.

Find a template

One of my favorite design-theme/frameworks is Scribbish, a css framework and typo theme designed by Jeffrey Allan Hardy at Quoted-Printable. Scribbish has a simple, yet elegant appearance, which is mirrored in the css framework. There are separate stylesheets that control different components of the page, making this theme easy to understand and modify to fit Pier, or any site. Before continuing, go read about the css framework used by scribbish, I'll wait.

Sounds good, right? Download the latest version of Scribbish, expand the tar file and take a look at the source code.

Modify scribbish for use with Pier

Before we tell Pier about the new css files, we need to change a few things so that scribbish works correctly outside of its native Typo environment.

Use relative paths

All references to 'url's in all the stylesheets are hard-coded to work with Typo. For example, in scribbish/stylesheets/application.css, the body tag looks like:

body {
    background: url(/images/theme/background.gif) repeat-x left top;
    font: normal 12px "lucida grande", verdana, arial, helvetica, sans-serif;
}

You could hard code a new url, but for now I recommend just using a relative url:

background: url(../images/background.gif) repeat-x left top;

Find all the references to absolute paths change them to relative paths.

Deal with tables and textareas

Pier still uses tables in a few places. Whenever you edit a page, the form generated by Magritte uses tables. I haven't figured out how to change this yet. For now, we'll work around it.

Unless the table width is set to 100%, the width of your textareas will be constrained by the table width, rather than the width of the 'content' div. To ensure that your forms use the full allowable width, and have reasonable height, add the following code to the bottom of content.css:

#content textarea {
    width: 100%;
    height: 300px;
}

#content table.container {
    width: 100%;
}

Configure Comanche to serve files

If you look closely at your default Pier installation, you'll notice that you are using css stylesheets from here. We want to use our own custom stylesheets. There are multiple ways of accomplishing this goal. You can put the stylesheets at any web-accessible URL and reference them. If you running Pier behind a proxy front-end, then you can just put them somewhere in your root directory and refer to them. If you know how to setup Seaside Style Libraries (I haven't tried this yet) then you can just do that. The solution I chose was to configure Comanche to serve files. This allows me to keep the stylesheets outside of my Squeak image. Right now I find this easy since I'm used to working this way.

The default configuration

If you followed the instructions for installing Pier posted on the Smallwiki site or the instructions for installing Seaside from the Seaside home page, then Comanche is not configured to serve static files.

The new configuration

Fortunately, all of the necessary changes are summarized nicely in the Getting Software section of David Shaffer's Seaside tutorial. For completeness, I'm posting the exact same code here, but all credit goes to David Shaffer.

Rather than starting Comanche like this:

WAEncodedKom startOn: 8080.

Kill any existing server instances with this command:

"Kill all existing Kom HTTP servers"
HttpService allInstancesDo: [:each | each stop. each unregister].

And then start Comanche with these options:

"Start a new server on port 8080 servering both static content and seaside apps"
| ma seaside |
seaside := WAKom default.
ma := ModuleAssembly core.
ma serverRoot: (FileDirectory default directoryNamed: 'FileRoot') fullName.
ma alias: '/seaside' to: [ma addPlug: [:request | seaside process: request]].
ma documentRoot: (FileDirectory default directoryNamed: 'FileRoot') fullName.
ma directoryIndex: 'index.html index.htm'.
ma serveFiles.
(HttpService startOn: 8080 named: 'httpd') plug: ma rootModule

Note that the directory 'FileRoot' can be called anything you want. You'll need to put that directory in the same folder as your Squeak image file.

Customize your Pier environment

Although I recommend more changes to the css stylesheets, those minor edits are the only changes that you have to make. To use the new stylesheets, we need to give Pier the path to 'application.css', and modify the environment to reflect the scribbish layout structure.

Use the new stylesheets

To make Pier use the new stylesheets, just change PRStyleLibrary>>style as follows:

PRPierLibrary>>style
    ^ '@import "/scribbish/stylesheets/application.css";'

Note that if you started Comanche prior to adding scribbish underneath the 'FileRoot' directory, you might have to restart Comanche before the stylesheets will be recognized.

Edit the environment

With the new stylesheets in place, the pages will look strange because the Pier layout structure does not match the css design. To fix this, you need to edit the environment. Access it by typing it in the navigation bar of your browser, or clicking here, if you're following this tutorial.

A new environment

<div id="container">
+header++views++path+
<div id="page">
<div id="content">+main+</div>
<div id="sidebar">+search++tree++commands+</div>
</div>
<div id="footer">Pier &mdash; Empowered by Seaside</div>
</div>

Set the default environment

The default environment is returned by the method PRStructure>defaultEnvironment. To add new default components, just add children to the the environment Page (a PRPage instance). To change the environment contents, change the string for the 'contents:' parameter. Here is an example of a new default environment that adds a path component under the header, and a search component in the sidebar.

PRStructure>environmentStructure
    ^ (PRPage named: 'environment')
            addDecoration: PRHider new;
            addChild: ((PRComponent named: 'header')
                    componentClass: PRHeaderWidget;
                    yourself);
            addChild: ((PRComponent named: 'path')
                    componentClass: PRPathWidget;
                    yourself);
            addChild: ((PRComponent named: 'views')
                    componentClass: PRViewsWidget;
                    yourself);
            addChild: ((PRComponent named: 'search')
                    componentClass: PRSearchWidget;
                    yourself);
            addChild: ((PRComponent named: 'commands')
                    componentClass: PRCommandsWidget;
                    yourself);
            addChild: ((PRComponent named: 'tree')
                    componentClass: PRTreeWidget;
                    yourself);
            addChild: ((PRComponent named: 'main')
                    componentClass: PRContentsWidget;
                    yourself);
            contents: '<div id="container">+header++path++views+<div id="page"><div id="content">+main+</div><div id="sidebar">+search++tree++commands+</div>
</div><div id="footer">Pier &mdash; Empowered by Seaside</div></div>';
            yourself

This new environment will be used whenever you create a new kernel. To reset your kernel (this will destroy all data):

"Reset Pier Kernel"
PRKernel reset. PRPierFrame initialize.

Customize components

The new environment defines some new components, and uses familiar components (i.e. views) in a new way. In this scenario, the header, path and view components are intended to be laid out horizontally across the top of the page, before the "page" and "sidebar" content. In my customized scribbish layout, each component has a corresponding #id or .class to provide complete control over the style. The default component settings in Pier use classes, so you'll either have to change the component 'settings' after adding the component, change the code to set new defaults, or change your stylesheet to fit the Pier defaults. For example, the default path widget has a title that gets in the way when the path is displayed as an inline list. You might also want to hide the icons by default. The Pier UI components that you add to each page are subclasses of PRWidget. Thus, to change the default settings of the path component, add class methods to PRPathWidget.

PRPathWidget>defaultTitle
        ^ nil

PRPathWidget>defaultShowIcons
    ^ false

PRPathWidget>defaultCssName
    ^ 'path'

Custom Header component

The default Pier header uses tables.

PRHeaderWidget>renderWidgetOn: html
html table
    class: 'header';
        with: [
            self renderLogoOn: html.
            self renderSpacerOn: html.
            self renderTitleOn: html ]

Let's change the header to look like a simple blog or wiki: get rid of the table, and add a subtitle. On the instance side, add methods to get, set and render the subtitle. Then modify renderWidget on to render both the title and subtitle.

renderWidgetOn: html
    self renderTitleOn: html.
    self renderSubtitleOn: html

renderSubtitleOn: html
    html heading level: 2; with: (self expand: self subtitle).

subtitle
    ^ self propertyAt: #subtitle ifAbsent: [ self class defaultSubtitle ]

subtitle: aString
    self propertyAt: #subtitle put: aString

On the class side, add a method for the description and the default subtitle (a new string).

defaultSubtitle
    ^ String new

descriptionSubtitle
    ^ MAStringDescription selector: #subtitle label: 'subtitle' priority: 120 default: self     defaultSubtitle

CSS class controls for inline lists

Inline lists are a versatile design technique that allow the use of lists to control horizontal layout (see CSS Design: Taming Lists). With this techinque, lists can be used to create menu bars and context bars. To control the style of inline lists, it is often convenient to designate the first and/or last item of the list using a class. This allows these items to be styled differently than items in the middle of list, providing a convenient mechanism for creating a path string separated by slashes, or a menu bar separated by lines.

Although we could make these properties configurable, simply hard-coding them to always be "first" and "last" should work for 99% of all cases. To add this capability, modify PRListWidget>renderItems.

renderItems: aCollection on: html
    html unorderedList with: [
        aCollection do: [ :each |
            html listItem 
                class: (aCollection first = each ifTrue: [ 'first' ]);
                class: (aCollection last = each ifTrue: [ 'last' ]);
                class: (self selected = each ifTrue: [ 'active' ]);
                with: [ self renderItem: each on: html ] ] ]

Add a context bar

Now that we can uniquely render the first and last items of a list, it's easy to add a context bar. Just add a path component to the environment. Under the settings for environment/path, add a custom CSS name. Then, use css to render the path as an inline list, adding a '�' (») or '/' character before each item except the first. To add this change into the scribbish framework, we'll add #path to the layout, and create a new component.css file to handle the the style settings for #path and other Pier components.

Since path is a floating element, we need to make sure that all of the floats are cleared before rendering the page content, by adding 'clear: both;' to the top of the #page style.

Make changes to layout.css

#path {
    padding: 0px;
    margin-bottom: 0px;
    float: left;
}

#page {
    clear: both;
    /* ... page settings here ... */
}

create component.css (based on code from *CSS Design: Taming Lists)

#path {
    /* additional custom path styles here */
}

#path ul {
    margin-left: 0;
    padding-left: 0;
    display: inline;
    border: none;
}

#path ul li {
    margin-left: 0;
    padding-left: 2px;
    border: none;
    list-style: none;
    display: inline;
}

#path ul li.first {
    margin-left: 0;
    padding-left: 0px;
    border: none;
    list-style: none;
    display: inline;
}

#path ul li:before {
    content: "\0020 \0020 \0020 \00BB \0020";
}

#path ul li.first:before {
    content: "";
}

Add a reference to component.css in application.css just above the 'local.css' reference, so that component styles override any default settings.

@import 'component.css';

P is for paragraphs

Lukas recently changed the way paragraphs are rendered in Pier from using <p> tags to <div class="paragraph">. This change first appeared in Pier-lr.83. The monticello comments say something about the <p> element causing problems with nested pages. Unfortunately, the change to <div class="paragraph"> completely breaks the scribbish theme.
Currently, I'm using <p> elements with one level of nesting and I haven't noticed any problems.

To use normal <p> elements, change PRViewRenderer>visitParagraph to

visitParagraph: anObject
    html paragraph with:  [ super visitParagraph: anObject ]

Please note that it might be possible to fix this problem by using specifying properties for <div class="paragraph"> that will make it behave like <p>. If anyone knows how to do this, please let me know.

Get the code

I haven't covered every Pier and css modification in detail. Rather, I tried to provide example cases that highlight the types of changes you can make to add some style to Pier. To provide a finished example, you can download my version of a default Pier installation (Pier-all-lr.87) styled along with my modified version of scribbish (version 1.2).

Files

Pier Design Tutorial image+changes
Squeak (3.8) .image and .changes files loaded with Seaside 2.6a3, Pier-All-lr.87 plus all the necessary dependencies.
pier-scribbish-1.2
Scribbish 1.2 distribution containing changes described in this tutorial and a few other modifications for Pier.