My Geekdom

Disk Crash, Part 3: OS Indepent Configuration

2012-09-03T22:02:00-04:00

NOTE: This was written on September 3^rd and published on September 23^rd.

Managing Configuration

My previous postings about my disk crash and my backup strategy discuss how I keep files in sync across my computers. While I have a 3-tiered system using CrashPlan, DropBox, and rdist to meet different synchronization requirements for different files, this strategy uses tried-and-true technology.

But identical files across machines doesn’t guarantee identical behavior across the machines. At the moment, my computer collection is:

A laptop running Ubuntu 12.04
A laptop double-booted to run Ubuntu 12.04 and Windows 7. I’m not a big Windows fan but sometimes I need to use it.
My media/game server running Windows 7 (good for games for my kids), and Ubuntu 12.04 (best for me).
My MacBook Pro running OS X 10.6.
My VPS cloud server. It has been running Ubuntu 10.04 but I’ve requested an upgrade to the newest version of Debian, I think Lenny. I need the upgrade because Ubuntu 10.04 is 2+ years old. I’m switching to Debian because Debian tends to be more stable though it has a weaker desktop environment than Ubuntu. Being that my VPS is a headless server, the desktop features don’t really matter to me.

Home Directories

Being that I run three different OSs (Ubuntu is built on Debian so I don’t consider it a separate OS), I have different home directories.

Ubuntu/Debian: my home directory is /home/neil.
Mac OS X: My home directory is /Users/neil.
Windows 7: Synchronizing configuration between *nix systems and Windows only makes sense if you are running Cygwin. Unfortunately, Cygwin home directories seem a bit random (info about Cygwin home directories).

While using the $HOME variable to refer to your home directory works in many situations, it doesn’t always help. My solution to this is simple: *nix wins. My home directory is /home/neil on all of my computers. If it is not there by default, I use atypical file mounts or, when needed, symbolic links to achieve conformance.

My $PATH

Maintaining your $PATH settings is a real hassle. Not only are executables in different locations based on your OS, I frequently find that executables are in different locations on the same OS.

Ruby is a good example of a program that tends to be in different locations on different computers. I use Homebrew to install Ruby on my Mac OS X computer, APT to install Ruby on one of my Ubuntu laptops, and rbenv on my other Ubuntu laptop. I don’t bother installying Ruby on my Windows computer as I only use Windows when I have to.

Dynamically Assigned $PATH Variables

As I use a single .bashrc file across all my computers and exectuables are scattered everywhere, I dynamically configure my $PATH variable based on the computer’s file system. I use my a bash function, addpath, to add directories to $PATH:

addpath() {
    if [ -d "$@" ]; then
        PATH="$@:$PATH"
    fi
}

With addpath, I don’t have to worry about adding too many directories to my $PATH. This allows code such as:

1:  addpath /usr/bin        # typical *nix path component
2:  addpath /sw/bin         # used by some Mac OS X package managers
3:  addpath /c/usr/bin      # used by Cygwin on Windows

addpath ensures that only existing directories are added to $PATH.

This meets nearly all of my needs. The one problem I have with this solution is that I find that my $PATH frequently changes. For example, when I recently started using rbenv, a path change was required. These changes aren’t major but sometimes I need a few tries to get it right.

So I have a strategy for repeatedly updating my .bashrc file, executing the .bashrc file, and testing my new $PATH. The first part of this strategy is that, the first time bash is run, I squirrel away the value of $PATH into another variable:

1:  export ORIG_PATH="${ORIG_PATH:-$PATH}"
2:  export PATH="$ORIG_PATH"

For those of you that aren’t fluent in shell syntax, ORIG_PATH=${ORIG_PATH:-$PATH} will set the value of $ORIG_PATH to $PATH the first time it is executed. On subsequent executions, $ORIG_PATH will be assigned the value of $ORIG_PATH. This ensures that $ORIG_PATH always equals the value of $PATH the first time the .bashrc file is executed.

I then use $ORIG_PATH as the basis for all of my $PATH changes.

The final piece of my $PATH management is this function:

1:  reinit () {
2:      . $HOME/.bashrc
3:  }

It is just a convenience function to execute my .bashrc file in the current shell. Being that my .bashrc file is idempotent, I can rerun it as often as I need, confident that my bash environment will be the same no matter how many times I’ve executed my .bashrc.

Miscellaneous Path Differences

Managing `rbenv`

While the above solution handles general $PATH configuration, some special cases still arise. Managing my $PATH to work best with Ruby tends to change across my computers and even change over time on the same computer.

For Ruby, some of my machines have rbenv installed on them and some don’t. On one machine I find that sometimes I use rbenv and at other times I don’t wish to use it. So I have a one-off solution for managing my rbenv environment in my .bashrc file:

if [ -f "$HOME/.rbenv/USE-RBENV" ]; then
    addpath $HOME/.rbenv/shims
    ... other stuff ...
fi

The file USE-RBENV is an empty file that I create when I want to, not surprisingly, use rbenv. To switch between using and not using rbenv I create or delete the USE-RBENV file and re-execute my .bashrc file using my bash reinit function. This strategy leverages the dynamic path settings and idempotency of my .bashrc file.

Managing My `.emacs`

Avoiding most of the details, Emacs needs to find some elisp files as well as some *nix files that are not in the same place on all of my computers. For loading elisp files, I have written a function that is similar to my addpath function that I use in my .bashrc file.

The elisp function, add-load-path, has a bit more smarts than addpath. addpath works by omitting non-relevant directories to $PATH. add-load-path knows how to search through directories that may contain an elisp package and then include that directory. A call to add-load-path is simply:

(add-load-path "jdee/lisp")

There are also some functions of Emacs that behavior better if they know the full path of some executables. For example, Emacs can make use of the *nix man(1) command. While add-load-path finds elisp files, find-first-executable searches for executable files. This is the elisp command I use to find the man executable:

(setq manual-program
        (find-first-executable
         '("/usr/local/bin/man" "/usr/bin/man"
           "/bin/man" "c:/cygwin/bin/man.exe")))

find-first-executable returns the first path that corresponds to an executable. You may have noticed the c:/cygwin/bin/man.exe argument which allows the same .emacs file to run on Windows as well as *nix.

In The Next Installment

The next and final part of this series will delve a bit deeper into my general philosophy of portable and maintainable configuration and code. In general, I try to plan for the unplannable. I try not to let the fact that planning for the unplannable is impossible get in my way.

I try to produce flexible designs that typically include a separation of data and code. Then, when the unexpected occurs, in theory, I can return to work with only a minimal hit to my productivity. In some sense, I use the MVC programming model in my data management and system configuration.

Disk Crash, Part 2: My Backup Strategy

2012-08-21T22:02:00-04:00

NOTE: This was written on August 21^st and published on September 23^rd

My File Distribution And Backup Strategy

In my previous posting I discussed my how I categorize files in terms of importance, timeliness when distributing files, and the platforms I distribute the files to. Here I’ll cover the technologies I use to meet my file management goals.

My Backup Strategy

Making sure your computer is backed up is typically a hassle. Even worse, my experience has taught me that thinking that you are backing up your system is very different from actually backing up your system. Numerous times I’ve been told by a system admin that, unbeknownst to them, the backups are unusable.

Knowing how difficult backing up your computer can be, I outsource it to a company whose only business is backing up data.

CrashPlan

I started using CrashPlan because they have a $25/year backup plan. That is, they are cheap. As their service was simple, reliable, and pretty much configure-and-forget-about-it, I upgraded to their premium plan. It costs $120/year. For that $10/month I get unlimited data storage and can backup as many as 10 computers.

While my household has more than 10 computers, many of them aren’t used very often and don’t need to be backed up so the limit of 10 computers has never been a problem.

I consider the $10/month a great deal as CrashPlan has been a life saver for me more than once. CrashPlan has an application that runs in the background, similar to DropBox’s application, and continually backs up your computer. The initial backup can take a few days but once backed up, the Crashplan database is never more than a few minutes behind changes on your disk drive.

I should mention that the initial backup takes a few days because CrashPlan tries to share the computer with other applications. I can restore all my backed up data from CrashPlan in about an hour.

Not Just Backup, Secure Backup

When configured properly, CrashPlan uses the Blowfish encryption algorithm with a 448-bit key. My encryption key was generated on my computer and has never left it. This means that even if CrashPlan’s servers are compromised and all of my backup data is stolen, it is very unlikely that the attacker could decrypt my data.

Backup Everything

I use CrashPlan to backup everything that I want to backup. I don’t distinguish between more or less important files. I just backup everything I think I may miss if it were to go away. CrashPlan is even kind enough to keep multiple revisions of each file and deleted files available for restoration. What a deal.

On my Mac OS X, I back up nearly everything in my home directory. While I try to keep my application preferences out of ~/Library/..., some applications just won’t cooperate. So I also selectively backup directories in ~/Library/... I also don’t bother backing up ~/Downloads. I don’t currently back up ~/.cpan, the data directory for the Perl package installer. As mentioned in my previous posting, I don’t yet have a working strategy for what applications have been installed by package managers such as CPAN.

My File Distribution Strategy

My file distribution strategy is more complex than my backup strategy. To date I’ve not found any service that meets all of my needs for data distribution. This leaves me using multiple distribution technologies based on the importance and timeliness of the data.

Dropbox

Dropbox and similar services have become quite popular the last few years. I use Dropbox far data distribution continually. In fact, this blog entry is being authored inside my Dropbox directory. Dropbox is the primary data distribution technology that I use for managing the distribution of High Importance, Timely Distribution files. Especially my Org Mode PIM files.

Because of Dropbox’s nearly instantaneous file distribution, I can be working on one laptop, save my Org Mode files, pick up my other laptop to find the updated Org Mode files waiting for me. Wonderful!

Dropbox Security Concerns

While Dropbox is awfully convenient, I’m not terribly comfortable with its security. Dropbox’s security claims state that they use TLS, which is frequently referred to as “SSL”, for network security. But the rest of their claims are intangible. For example, what is a “hardened” website and how is it better than other websites? They flat out state:

Like most online services, we have a small number of employees who must be able to access user data for the reasons stated in our privacy policy (e.g., when legally required to do so).

Without disputing Dropbox’s claim that most online services have a means to allow their employees to access your private data, I use CrashPlan for my backups and know that they can’t access my private data.

Even if Dropbox’s employees are 100% trustworthy, if their employees can access your data, so too can a hacker.

Because of my security concerns, I avoid using Dropbox for my more sensitive data. For example, I have an Org Mode todo list related to financial matters such as taxes and bill paying. The files that have the details about those tasks are not in my Dropbox directory.

If you are interested, here’s a nice article covering some of Dropbox’s security problems as well as strategies to make them less of a problem.

rdist + ssh

rdist and ssh are two tried and true Unix programs. rdist is used to keep two directories synchronized. The directories can be on one computer but are typically on two different computers (rdist = Remote Distribution).

rdist and Dropbox are similar in that they both keep directories across multiple machines synchronized. However, unlike Dropbox, rdist must be manually run. It doesn’t watch specific directories watching for a file modification and then automatically synchronize the files across all of your computers.

I use rdist to sync files to my Ubuntu VPS server that I rent, quite happily, from RoseHosting. The reason for synching to my VPS is that it is my only computer that is always accessible from the internet:

Rosehosting keeps my server running.
It has a DNS entry so it can be found from any computer.
Unlike my personal computers, especially my laptops, my VPS is never turned off or not online.
Unlike computers on my home network, the firewall my VPS is behind allows incoming connections. My home network firewall is significantly more restrictive.

In short, my VPS is the only computer I have that is available 24x7 from any internet connection. This means that every computer can sync with it at any time. While it would be faster and use less network bandwidth if I synched the computers in my home to each other rather than a VPS somewhere out in the cloud, I have more than enough bandwidth to use.

Next Up

Keeping files in sync is great but identical files don’t always behave identically on different systems. For example, there tends to be lots of file paths in my .bashrc and my .emacs files. I utilize a series of tricks to allow many of these files to behave identically on different machines.

Disk Crash

2012-08-21T22:02:00-04:00

Ouch! My Disk Crashed

My MacBook Pro became unusable and required a complete repartition and several reformats before I could resuscitate it. Recovering from the disk crash was surprisingly easy, there were a few hitches. Being that, without a duplicate computer, the only time that you can see how well your backup strategy works and correct it, I’ve been using this disk crash as an opportunity to bolster my backup strategy.

While I had some gaps in my backup strategy, within 2 hours of my disk crash I had my backup computer running, complete with my data files, shell scripts, config files, etc…

File Distribution

In my mind, file backup and file distribution across multiple machines are intimately related. File backup is, well it’s file backup. I use file distribution to keep various files synchronized across my devices. It should be noted that while the goal of file distribution is synchronization of data across computers, distributing files to other computers effectively backs them up.

My primary computers are my MacBook Pro and an HP laptop running Ubuntu Linux. My mobile is a Samsung Infuse running CyanogenMod 10, a customized version of Android 4.1 Jelly Bean. I also rent a VPS cloud computer running Ubuntu. Lastly, I have an Asus running Windows 7. I primarily use te Windows computer for streaming media (I don’t dig Windows).

I use file distribution to achieve two goals. I like to keep my configuration files consistent across my MacBook and two Ubuntu computers. This includes my .bashrc and my .emacs site startup files. While I want those files to stay in sync to make the management of them simpler, there’s generally no rush in synchronizing them.

Another example of my file distribution involves my PIM. I use Org Mode as my PIM and want the data files to be synchronized across my MacBook, two Linux boxes, and my Android. I want those files synchronized immediately as I frequently move from computer to computer and always want my PIM data to be up to date.

Prioritization Of File Backup And File Distribution

My backup and distribution strategy categorizes files using two metrics:

File Importance: This describes how critical each file is. Files in ~/Downloads are typically not very important while the text for this blog entry is very important (at least to me).
Synchronization Delay: As I move between multiple computers, my productivity is related to how frequently some files are synchronized across the computers. Some I want optimized as soon as possible while other files can take hours or even days to get synchronized without affecting my productivity. Some low priority files, such as those in ~/Downloads don’t ever need to be synchronized across machines.

In practice this leads to four types of files:

High/Timely: Files that are very important and need to be synchronized in a timely fashion. My PIM files fall into this category.
High/Delayed: Important files that don’t need to timely synchronization. Configuration files such as my .bashrc are High/Delayed.
Low/Delayed: Files that are convenient to have across my machines but aren’t essential to my immediate productivity. For example, I frequently download manuals for various appliances that I own. I like to have them handy but it’s fine if they aren’t synched immediately. As I can easily download them again, they’re of relatively low importance.
Everything Else: These are files that are of significantly low importance that I don’t worry too much about them.

My backup and distribution strategy varies by which quadrant (I’m excluding the N/A boxes) a file is in.

There’s Data That Should Be Backed Up That Is Not In A File

My first lesson was that not all data is backupable via file backup and distribution processes. The best example I have of this is the set of installed applications. Applications are typically large files and frequently require complex installations. While many Mac OS X applications can be installed by copying the file into the /Applications directory, others require the use of package installers. This is similar to Windows applications that store data in the Windows Registry.

Knowing that applications were cumbersome and frequently needed special installation scripts led me to decide to not back up my applications. After this disk crash, I continue to think that it makes no sense to backup your installed apps. That said, I have sorely missed having a list of the applications that I had installed before the disk problems.

Monitor Installed Applications

My initial response to the lack of a list of installed apps was to start recording apps as I installed them. After the first half-dozen or so apps I decided that this was a bad idea. I could see that the list was going to be incomplete, outdated, and generally not very helpful.

So I took a few minutes to whip up a trivial shell script that I run once a day via crontab. The shell script attempts to record the names of installed applications rather than the actual applications themselves. The version of the shell script that I’m currently using is:

#!/bin/sh
shopt -s nullglob
shopt -u failglob
ls -dlrt /Applications/* /Library/PreferencePanes/* ~/Library/PreferencePanes/* > ~/Dropbox/Backups/InstalledApps.txt

An abbreviated sample of its output is:

drwxr-xr-x   3 root  wheel  102 May  5  2009 /Applications/Font Book.app
drwxr-xr-x   3 root  wheel  102 May 19  2009 /Applications/Stickies.app
drwxr-xr-x   3 root  wheel  102 May 19  2009 /Applications/Chess.app
drwxr-xr-x   3 root  admin  102 Sep 19  2009 /Library/PreferencePanes/RCDefaultApp.prefPane
drwxrwxr-x   3 root  wheel  102 May 24  2011 /Library/PreferencePanes/Growl.prefPane
drwxr-xr-x@  3 neil  admin  102 May 24  2011 /Applications/HardwareGrowler.app
drwxr-xr-x@  3 neil  admin  102 May 24  2011 /Applications/GrowlTunes.app
drwxr-xr-x   3 neil  staff  102 Jul 25 18:27 /Applications/BetterTouchTool.app
drwxr-xr-x@  3 neil  admin  102 Aug 13 22:51 /Applications/Google Chrome.app
drwxr-xr-x@  4 neil  admin  136 Aug 18 08:19 /Applications/FirefoxAurora.app
drwxr-xr-x   3 neil  staff  102 Aug 18 11:52 /Applications/GrowlSafari.app

As this script is run once a day and stored in my Dropbox folder, I always have an up-to-date copy of the file.

I still have some changes planned for the script:

The script only works for typical Mac OS X applications. It doesn’t check for command line applications as well as applications installed by a package manager such os Homebrew (Mac OS X package installer), CPAN (Perl package installer), or rbenv (Ruby package installer - sort of).
Perhaps I had a good reason to put the data file into my Dropbox folder but I can’t recall it now. This data seems a perfect example of High/Delayed data as discussed in the table above.
The script misses some peculiar applications. Most of these applications are from Apple. Xcode is a good example of a “peculiar” application. Xcode includes a GUI application, command line programs, and runtime libraries. These get scattered across the machine. I suspect I’ll just hardcode these applications into my script rather than trying to create the logic to find them. For example, I’ll have my script add a line to the installed applications that reads Xcode from https://developer.apple.com/xcode/. That seems good enough.

Other Missing Data

Missing the list of installed apps had a quick fix but other data isn’t so readily available or I don’t know how to access it. Some examples of missing data are:

Customized icons for folders: On Mac OS X you can associate a small icon to each folder. On the directories that I use routinely I have customized the icons so that I can find them quickly.
The settings of some Chrome extensions did not get restored.
My cron jobs were only stored within cron’s database.

The cron jobs problem was another one that I was able to fix with a simple shell script and a minor modification to my workflow. Instead of editing my cron jobs directly via crontab -e, I now keep them in a file in my bin directory and modify that file as needed. When I modify my cron jobs I can either run crontab to immediately update my cron jobs or wait until I run my nightly cron job as that auto-updates my cron jobs:

#!/bin/sh

# Run my apps lister
$HOME/bin/installed-apps-lister.sh

# Run other stuff that I've removed from this example.

# Update my cron jobs in case I've modified the cron file.
crontab $HOME/bin/crontab-jobs

The $HOME/bin/crontab-jobs file gets automatically backed up with the rest of my bin files.

Other Backup And Distribution Strategies

I’ll discuss the rest of my backup strategy in a subsequent posting. As a sneak-peek I’ll tell you that the technologies I use include:

Time Tracking, Part 2: Org Mode File Format

2012-08-05T18:38:00-04:00

A Brief Introduction To Org Mode File Syntax

I’ve decided that my time tracking solution requires some basic understanding of the file format of Org Mode as Org Mode syntax pervades the system. I’ve done my best to cover the smallest amount of Org Mode syntax that is needed to understand the examples I’ll of my time tracking system.

It is frequently said that when you buy a house the three most important things to look for are location, location, and location. When it comes to Org Mode, the three most important things to remember are Org Mode file format is plain text, Org Mode file format is plain text, and Org Mode file format is plain text.

By plain text I mean:

You don’t need a GUI or any other program to read an Org Mode file.
Any text editor can be used to create Org Mode files.
Org Mode files are highly compatible with standard text file utilities such as grep(1).

While Org Mode’s use of plain text has many advantages, the files tend to look a bit intimidating to a newcomer.

While this posting only discusses Org Mode’s file format, I use Org Mode for its incredibly rich organizational functionality.

Hierarchical Outlines

Org Mode encourages you to store your data in an outline. A level one headline is simply a line of text that begins with a single asterisk (*). A level two headline begins with two asterisks (**), etc…

* Work
** Work hard.
Finish my project ahead of schedule.
** Ask boss for raise.
Remember to compliment him lots when asking.

* Personal
** Clean up after my kids.
Grumble while cleaning up.
** Tell my kids that they should clean up after themselves.
Laugh quietly to myself as I know it won't help.

Adding Tags

Org Mode allows you to further organize your tasks by adding tags to your todo list. Being that Org Mode is plain text, tags are words wrapped in colons (:).

* Work                                                              :work:
** Work hard.
Finish my project ahead of schedule.
** Ask boss for raise.
Remember to compliment him lots when asking.

* Personal                                                          :personal:
** Clean up after my kids.
Grumble while cleaning up.
** Tell my kids that they should clean up after themselves.
Laugh quietly to myself when they say they will but I know they won't.

Tags in Org Mode are inherited with the hierarchical layout of the outline. In the above example, only the two level 1 headlines have tags on their line but their children inherit these tags. For example, the task

** Clean up after my kids.

inherits the :personal: tag.

Using Tags To Describe Multiple Facets Of A Task

The above use of tags was rather dull. The tags repeated data that was already in the outline hierarchy. The real power of tags becomes apparent when you use them to classify different facets of a task. For example, a GTD style of tagging can include context tags such as :@home:, as well as priority tags such as :next_action:. Adding these tasks, our example becomes:

* Work                                                              :work:
** Work hard.                                                      :@work:@home:%next_action:
Finish my project ahead of schedule.
** Ask boss for raise.                                             :@boss:%someday:
Remember to compliment him lots when asking.

* Personal                                                          :personal:
** Clean up after my kids.                                         :@home:next_action:
Grumble while cleaning up.
** Tell my kids that they should clean up after themselves.        :@home:%why_bother:
Laugh quietly to myself as I know it won't help.

In Org Mode all tags are created equal. For example, priority tags aren’t different than location or topic tags. Keeping with Org Mode’s plain text file format, I decided to distinguish my priority tags with a percent sign (%), changing GTD’s common :next_action: tag :%next_action:.

More Info

If you are interested, you can read more about Org Mode on its home page, from any one of the vast collection of tutorials on the Org website, or one of the links from Google search.

Time Tracking

2012-08-04T19:29:00-04:00

The Quantified Self

A couple of months ago I heard about this ~~nerdy~~ cool movement called Quantified Self. Its goal is improving life by leveraging organized data about yourself. The ideas behind Quantified Self are far from new. Recording your caloric intake with the aim of reaching a goal, ie: counting calories, has been around for… well for a long time.

The Quantified Self movement takes old principles, such as calorie counting, and improves upon them in two key ways:

QS is more of an atitude than an activity. Tracking calories, cholesterol levels, blood pressure, etc… are generally targeted at a very specific goal. For example, a baseball player that records his batting performance against different pitchers is looking to be a better batter. The Quantified Self movement encourages people to look for new means of measurement, as wel as to use existing means of measurement differently, with no single goal. If you will, QS is an umbrella encompassing both old and new tools.
Technology is a game changer. Almost overnight it became possible to measure and record huge amounts of data about yourself. This ranges from relatively low-tech activities such as “checking in” to a social network during the day or using an online to-do management system to multipurpose devices that collect fairly complex data.

My Project

Beinng the type of person who likes ~~nerdy~~ cool things, I got all excited the moment I heard about QS. After toying around with some ideas I decided to monitor my daily activities with the goal of using my time more wisely.

I am not referring to the type of activity data you can get from your Filofax or the detailed location data that a service such as Google Latitude provides. I am going to sample my activity several dozen times a day.

I have a living document and code repository that discusses my goals in more detail. I also have a snapshot taken at the time of writing this.

Besides documenting my goals, I have tried, for the first time, to keep a journal of the problem solving process as I was going through it. To reduce the overhead of maintaining a journal I have left it pretty raw so please forgive its informal presentation.

Implementing Time Tracking

The broad solution I’ve implemented in my time tracking experiment is documented in the living document and journal referenced above. To save time, I’ll not duplicate what is documented there. What I will discuss is my first solution, perhaps my first prototype of the time tracking system.

The goal is to track my activities via my ever-handy Android phone and record that data in a structured manner on my computer. Due to my personal preferences, I’m storing the data in Org Mode but the concepts are not limited to Org Mode.

Being that I’m storing my data in Org Mode and using Org Mode as my task manager, the solution has a very Org Mode’ish feel. I’ll try to explain enough about Org Mode so that you can follow.

In The Next Posting

I’m going to post this blog entry in two parts because my time tracking system just interrupted me, asking for a sampling of my activity. When I responded to the request I saw that I’ve used up the time I’ve allotted for blogging today so I’m putting this aside for now.

PS: I don’t think I’ve mentioned it before but my time tracking system provides instantaneous feedback. Every time it requests a sample of my activity it tells me how long I’ve been doing that activity as well as all the other activities I’ve done during the day. This allows me to immediately alter my behavior without waiting for longer-term analyses. Definitely cool (OK, and nerdy too).

Blurring Static And Dynamic Blogs

2012-07-22T18:23:00-04:00

Static Blogging

I’ve discussed the pros and cons about static blogging many times before. In summary, the cons are that, well that your blog is static. No surprise there.

Fortunately, there are ways to push the limits of static blogging so that it is not quite so static.

Website Hosting

Unless you are self-hosting your Octopress or Jekyll blog, you are using another site to host your blog. I host mine on Github. They have a web server running 24x7 to host it. But that’s not my problem. I just use their (free) services.

I understand that website hosting isn’t a terribly exciting example of stretching the bubble of static blogging, but it is an example nonetheless. Dynamic blogs require specialized software running on their hosting computer to work. Static blogs tend to be able to run on generic web servers but there still must be a web server or there’s no blog.

JavaScript Helps

Out of the box, Octopress comes with some features to show your most recent tweets as well as recent activity within your Github repositories. This is dynamic data that Octopress implements with some JavaScript.

Just to be clear, by dynamic data I mean that the data displayed on the page may change without a republication of the blog. I am not necessarily referring to a scrolling widget or something. Just dynamic data.

Conversely, JavaScript can be used to dynamically display static data. That is, data that is embedded in the source of the web page. Being that the data is embedded in the page’s source at or before publication time, it is assuredly static data.

But client-side JavaScript (ie: JavaScript running in the user’s browser) has its limits. For example, if you want to send a tweet whenever you add a new blog entry, client-side JavaScript is going to have a hard time at it.

Pushing Beyond Web Hosting JavaScript

The trick behind adding a much wider array of dynamic features to your static blog is:

Integrate with external web services and let them do the work for you.

It is even better if you can let other sites do the hard work for you free of charge. For example, Github is hosting this blog at no cost to me. Thank you Github! But Github won’t tweet about my new blog postings or search through my website. Fortunately, there are services that will happily do that for you.

Site Search

This blog’s search functionality is provided by Google. Dynamic blogging systems such as Joomla have a large array of search extensions that you can run on your server to provide various searching features. I’m sure that they provide a much larger set of functionality than my site’s Google search provides.

Richer functionality or not, each extension needs to be installed, configured, and maintained. With Octopress, you never need to think about your site’s search functionality. The site comes preconfigured for Google site search. That works for me.

Automated Tweeting

There are many ways to tweet via an email message but there is no way, at least none that I’ve bothered to think through, to get Github to send emails formatted for a tweet whenever you post. But, there are many sites that will watch your blog for change and perform some activity when your blog changes.

Exactly which site you use depends on what your needs are. For automated tweeting I decided to use FeedBurner. If you’re unfamiliar with FeedBurner, it provides a web service that takes your site’s RSS/Atom feed as input and adds value to that feed. My site’s RSS feed is http://feeds.feedburner.com/MyGeekdom. Assuming you don’t have uncommon browser settings, if you visit that URL you will see something that looks like:

Amongst the multitude of options that FeedBurner provides is the ability to subscribe to a site by email. So I subscribed myself, using a unique email alias, to FeedBurner emails about updates to my site. I then created an email filter that automatically forwards the FeedBurner emails to my tweeting email address and voila, whenever I post a new article I automatically produce a tweet like this one.

Tracking “The Buzz”

“The Buzz”, also known as “Trending” or simply “current events”, is a term to describe what is popular on the internet at any given moment. While tracking all of the internet’s news doesn’t seem appropriate for a blog, tracking news related to your blog posting may be desirable.

Courtesy of SocialMention, many of my blog postings have current news in the sidebar (see example blog posting). In many ways this is similar to tracking a Twitter #hashtag. But SocialMention gathers data from many dozens of web sites providing a richer and more diverse collection of data than Twitter can.

For example, by tracking several popular bookmarking sites such as Delicious, SocialMention can tell you what pages people are bookmarking that are related to your blog posting. Most people, thankfully, don’t send a tweet whenever they bookmark a site.

While it may be redundant to state,

Displaying current events on your blog postings means that they will always have current information.

This means that there’s no need to update your blog posting with links to relevant websites as new information becomes available. That is clearly a task that would become unmanageable as your blog grows. Instead, your static blog will always have dynamic data on it.

The Age Of Cloud Computing

While my blog is far from a large operation, the advent of cloud computing has made these dynamic features possible. From tiny systems like my blog to large systems like Google’s search engine, cloud computing is about wiring together the most appropriate services in a manner that produces your desired functionality with a minimal amount of setup, cost, and maintenance.

While I don’t have exact dates, my automated tweeting on new blog posts has been working maintenance free long enough that I forgot that I had implemented it until I noticed that I had just tweeted about my previous blog posting. I believe that is the very definition of “minimal maintenance”.

Given the large and growing number of web services in the cloud and my limited time to explore them, I’m sure that this is but a small sampling of the different ways to use dynamic data in static blogs.

Desktop Time Tracking

2012-07-21T20:17:00-04:00

Motivation

Recently I’ve been trying to improve my time management skills. This is further discussed in this Github README file. I have also been keeping a journal my activities and thoughts as I work on this project. You may also wish to read up on the Quantified Self.

Basic Activity Tracking

While I had not initially planned to focus on Mac OS X time tracking, it turned out to be easy. For a couple of weeks I’ve been playing around with two Mac OS X applications that track your computer activities: Timing (actually Timing Lite) and Time Sink.

Time Sink integrates with Mac OS X to track the active or foreground application and provides a variety of reports to display that information. In addition to tracking the current app, Time Sink tries to track what you were doing in that app. It achieves this by using a relatively low-tech process of collecting the active window’s name.

While a window name does not seem like a lot of information, it turns out to be surprisingly useful. From the window name Time Sink distinguishes, for example, the amount of time you spent browsing Facebook from the amount of time you spent browsing Github.

Time Sink can harvest this data from any app that includes information about the current activity. This includes editors that put the file being edited in the window name, media plays that put the title of what is currently being played, etc…

Your activities get displayed in a nice graphical format:

Due to browser page size I’ve excluded information such as how long I did these activities were as well as when I was doing them.

In addition to the activity tracking performed by Time Sink, Timing has some application-specific functionality that allows it to better track what you are doing inside an app.

From Activity Tracking To Category Tracking

The real power of these apps is that they allow you to categorize your activities. For example, you can specify that time spent browsing Facebook is socializing while time spent on Github is developing and exploring open-source.

My open source development category includes:

Time spent browsing Github.
Time spent editing files in $HOME/coding, the root of all of my open-source Github projects.
Time spent in a shell while the current directory is in $HOME/coding directory or one of its subdirectories.
Other stuff not worth describing here.

Due to its fine-granularity and inability to associate activities across apps, the activity tracking data is not that useful. But, once it is categorized, called Pooling by Time Sink, it starts to provide much more useful information. Consider this screen snapshot:

In it you can see that I was spending time on time management. This included time spent in three different apps.

Both Time Sink and Timing have a convenient, drag-and-drop mechanism for adding activities to categories.

Choosing Between Timing And Time Sink

Today, the end of my 2-week trial period, I decided to purchase Time Sink because of its ability to automatically export data into a CSV file. Besides the automatic scheduling, Time Sink allows you to specify what data is exported as well as whether you want the export to include all data that Time Sink has stored or only data since the last automated export. Being that I want to use Org Mode as my data repository, this data exporting feature sealed the deal for me.

That said, I feel that Timing and Time Sink are on par with each other. Each app costs < $10US so cost is not a factor. Each app has its own pros and cons. Time Sink just fit my specific needs.

Analyzing The Data

To state the obvious, collecting time usage data is only useful if you can analyze it such that it improves your time management skills. This is where Org Mode comes into play. With a bit of tweaking, the CSV importer from Memacs will be able to import this data into Org Mode.

Org Mode provides a rich set of features regarding scheduling and time management:

Standard task/to-do list management features.
Comparing actual time spent v. estimated time.
Identifying if you are falling behind on a long-term task.
Summarizing your time usage.
Sophisticated tracking of repeating tasks.
An integrated archive management framework that allows you to focus on the most pertinent information while maintaining all historical data.
Tons more.

Early Payoffs

Being as I’m planning on using Org Mode to perform data analysis and I am not yet able to import Time Sink data into Org Mode, one would think that there would be no immediate benefit of having this system working. I thought that would be the case. But I was wrong.

For some time I’ve used a timer that pops up every 15 minutes just to remind me that time is passing. I call it my Focus Alert timer as it reminds me to make sure I’m doing what I should be doing. When I’m notified I can stand up and stretch, check the clock to make sure I’m not late for something, make sure I’m working on my highest priority task, etc…

Today I modified my Focus Alert (a small shell script) so that it helps me track my time with Time Sink. Before today’s modification, my Focus Alert simply popped up a notification box that I had to click to close. As you may imagine, clicking quickly became second-nature.

Starting today, my Focus Alert interrupts my work and displays the Time Sink application. While I can immediately switch back to whatever I was doing before the notification, it is rather difficult not to notice Time Sink’s data. For example, about two sentences ago my Focus alert went off. As you can see in this screenshot of the relevant part of the Time Sink window:

I’ve spent about an hour on this blog posting. As Time Sink is kind enough to distinguish wall-clock time from actual activity time, that hour is a very reliable statement about my activity.

Being that I don’t think this blog posting is worth more than an hour of my, I would have allocated 30-45 minutes top, I’m ending the posting here.¹

I hope to follow-up this posting with more blog postings as well as journal entries as I try to understand what I’m doing and then make myself more efficient.

UPDATE 2012-07-22 Sun 15:50

Sacha Chua (blog, twitter) responded to one of my tweets about integrating Org Mode and Quantified Self. She pointed me towards this blog posting. In it she discusses how she uses Org Mode to track her blog writing time. Not only does she use Org Mode to track her activities, she uses it to track her WPM while writing blog posts. She averages 16 WPM while blogging.

Footnotes:

¹ After reading Sacha’s post, I decided to calculate my WPM for this blog post. It turns out that I typed 15 WPM (actually 14.79729729…). I’ll have to readjust my estimates for blogging as I don’t think I can hope to outperform the industrious Sacha Chua. Especially being that she types 110 WPM at full speed!

Bookifying News

2012-07-15T10:09:00-04:00

Bookifying News

I frequently find it convenient to take a website on the road. That is, I want a website or part of a website in a convenient, portable, easily accessible format on my Asus Transformer tablet. The Transformer has WiFi but no mobile data. Without a portable means of accessing a website, the lack of data plan would dramatically limit the usability of my tablet.

I have taken to bookifying websites that I have an interest in staying up to date with (eg: XKCD). For most sites, I only care about the newest postings. I convert the newest posting into an epub book, load it onto my tablet, and then read it using one an ebook reader.

Besides bookifying the newest articles on LifeHacker, I have also bookified entire websites.

The cool part is, this process requires a one-time configuration and then works automatically. That is, the entire web sites sit on my tablet and old website articles are automatically replaced by the new ones.

The Process

The bookification process utilizes the following tools:

The FOSS epub creation and library management tool Calibre.
The Cheetah Sync file-synchronization application for Android.
The Moon+ Android book reader.

Calibre comes packed with features for downloading new postings on a website, converting them to an epub book, and then managing the library it creates. It has nice extras such as updating sites on a customizable schedule and allowing you to specify how many ebooks of a site you wish to keep (eg: just today’s publication or an entire week’s worth).

Cheetah Sync provides WiFi folder synchronization between your desktop (Mac OS X or Windows) and your Android tablet. This synchronization can happen on a schedule or manually to reduce battery usage.

Moon+ is one of a large number of Android book readers. It seems to facilitate being used on a rotating library such as Calibre’s by monitoring a directory on the file system and responding appropriately to changes.

The end product of this is the LifeHacker website on my Transformer:

As a bonus, Calibre comes with a built-in epub book reader that you can run on your desktop:

The output is less-cluttered that the site; seeming like a Readibility or InstaPaper version of the all new content on the site.

Books And Websites

Besides using Calibre to create epub books, it also does a good job of managing existing ones. For example, SapphireSteel Software (whose name is an esoteric reference to a late 1970’s British TV show) have published an epub book called The Little Book Of Ruby. I’m learning Ruby and sometimes it is nice to be able to look up something when I’m not near a WiFi connection and don’t want to use my mobile WiFi hotspot as that tends to get costly quickly.

Calibre also provides a means for bookifying entire websites or a specified portion of them. This works by creating a temporary copy of the website on your computer and then converting it to epub format with Calibre.

Calibre provides its own web2disk utility that can download a website to your local disk. Sites such as this one cover the details of this conversion.

I happen to be familiar with the more fleible but more complex Linux program curl(1) and tend to use that to download an entire website. This Google search has numerous references on using curl to download a website.

While Moon+ is my preference for ebook news due to its responsiveness in dealing with an externally managed library, I prefer uploading my books to Amazon’s Kindle library. For me, the Kindle’s synchronization of bookmarks, notes, and current reading location are of much greater importance when I’m reading a book than when I’m reading a website. I use the Kindle for Android app on my tablet and smartphone and the Kindle for desktop or browser from my computer.

While Kindle doesn’t directly support the epub format, there are simple directions for converting epub ebooks to Kindle format using Calibre. This is a page from the Org Mode website ebook on my Transformer Kindle:

Yahoo! Forgets To Proofread Emails?

2012-07-01T21:33:00-04:00

Update: 2012-07-04 Wed Today iGoogle had a notice at the top saying tat Google was discontinuing the service in November 2013! The cancellation notice had this Learn more link. Yahoo!’s cancellation notice looks even worse when in the shadow of Google’s.

Yahoo! Notification

Yahoo! Groups is Yahoo!’s version of public groups or forums. I used to think that it was one of the better group sites so I tended to use it. Over time I thought that it fell behind the pack and I stopped using it. I never deleted any of my groups. They had no confidential data and they were ocassionally handy to have around for historical purposes.

On June 29 I received an email from Yahoo! informing me that they are shutting down some of their group features.

Is There A Proofreader In The House?

I frequently notice errors in emails I get but I know that I frequently send emails with errors in them. Most emails are just not worth the trouble of iterating on until it is perfect. But I typically expect large companies to send professional looking emails. The email Yahoo! sent me was anything but professional.

Rather than list the problems, I think looking at my annotations of their email is simpler:

5 Days – You Gotta’ Be Kidding

Ignoring the formatting mistakes, Yahoo! gave me only five days to backup my data before the service was to be shutdown and all data erased. Five days? If you don’t count the weekend and the July 4 US holiday, then they gave me two days to get my data.

Furthermore, while they state that I should backup my data, they don’t provide any directions or pointers telling me how to do such.

Conclusion

This was one email regarding one of the ?hundreds? of services Yahoo! offers. As such, I don’t want to make broad generalizations about Yahoo! based on this email. That said, for several years I have been concerned with relying too much on Yahoo! services because I feel the company is not doing well. In particular, I have avoided any serious use of Yahoo! Pipes, a tool that allows you to quickly create implement and deploy data processing services.

Being that Yahoo! seems to feel that five days is enough warning to shut down a service, I’ll now be extra certain not to use Yahoo! for anything that I care about losing. This just reinforces my decision to only use Yahoo! Pipes for quick prototyping. It is a shame because Yahoo! Pipes are a wonderful tool.b

The New, Privacy Invading, Facebook Like Button

2012-06-14T10:51:01-04:00

(Background information about online advertising.)

Facebook Keeping Tabs On You

Facebook has announced that it will sell user tracking information to its advertisers. Facebook, following in the steps of other large web advertising publishers such as Google, claims that targetted advertisements are a win-win deal. Advertisers win because they are able to provide ads that have a greater chance of being clicked and, hopefully, higher profit for the advertiser.

Users win because they see ads that they are more pertinent to their interests. For example, a user who frequents technology websites may not have a strong interest in an ad for laundry detergent. But that same user may appreciate ads about the newest gadgets.

So, win-win! Everybody’s happy.

Even I Can Count To Three

The phrase win-win implies that there are two involved parties. Facebook, Google, and other companies that track users for targeted ads seem to forget to mention that there are three involved parties:

The advertiser
The user
The publisher, that is the company selling the advertising

Perhaps win-win-win is a more apropos statement than win-win. The advertiser wins because their ads have a higher click-through rate, users win because they see ads that they are more interested in, and the publisher wins because they can charge a premium for targeted ads. For years Google has been running an instantaneous, automated auction so that Google gets the highest price possible for its ads.

A higher click-through rate is especially important for Facebook. Besides Facebook’s new obligation to its shareholders, Facebook has a notoriously low click-through rate. From a 2010 TechCrunch article:

Biggest [advertising presence] doesn’t mean most profitable. Facebook might be serving up more ads than almost anyone else, but they are still selling at very low ad rates because they perform poorly for the most part. If Facebook can figure out a way to make the ads on its site become more relevant and useful, it has a lot of room to boost its ad rates.

Facebook’s Dilemma

While Facebook is forced to look for ways to increase their revenue via targeted advertising, users tend to feel that having their internet usage being tracked is a privacy invasion. As user tracking becomes more accurate, and hence more valuable to Facebook and other advertising publishers, it is also viewed by users as a greater invasion of privacy.

For years Facebook has been tracking users’ activities as they roam the pages of Facebook. Facebook has also been gathering information about what non-Facebook pages users like. These two types of tracking have a common property:

Facebook only tracks users when they interact with Facebook.

Whether the interaction is on the Facebook site or on another site where a user does a Facebook share or like.

Facebook’s Bait And Switch

The change in user tracking that Facebook announced is that they will track users even if they do not interact directly with Facebook. Facebook will now gather users’ behavior whenever a user visits a site that has any Facebook code on it. That includes every page that has the ubiquitous: button.

For years, Facebook has been facilitating websites to use its Like button as well as social widgets and other means of Facebook integration with your site. Facebook has discussed the many benefits this brings to website publishers.

Facebook’s change in user tracking has many ramifications:

After years of facilitating Facebook integration with websites, Facebook has unilaterally decided to make every one of those sites a data collector for Facebook.
Once Facebook starts gathering this data, Websites with Facebook integration will be providing Facebook with valuable information without receiving any compensation for that information.
Short of not using Facebook, users cannot effectively opting out of this data tracking.
Some sites will have to update their privacy policy to reflect Facebook’s new data collection.

Furthermore, Facebook says that this change will happen within the next few weeks, leaving users and websites little time to react to the change.

Alternates

Even if a website decided to make changes to their site, there are few options to the Like button that do not involve severing all Facebook integration. One solution would be to replace the Like button, along with badges from other sites, with a generic social bookmarking widget.

That said, only five of the top ten social bookmarking widgets do not add tracking to their own widgets.

The bottom line is that Facebook, likely due to pressure from their recent IPO and their plummeting stock price, has decided to enlist, rather, to draft millions of sites into providing Facebook with valuable information at the cost of invading the users of those sites.

As this change in policy seems to leave Facebook as the only clear winner with users and other websites as losers, I think the phrase win-lose-lose is more appropriate Though probably not a good sales pitch.

Update 2012-06-14 Thu 12:39

Just saw that Facebook allowed its users to vote on the changes to its privacy policy. That said, comparing an old and a proposed privacy policy and understanding what the ramifications are is a very difficult task. Especially when there is little publicity about the voting process.

If if you heard about the voting, waded the through the privacy policy’s legal jargon, understood how the change would affect you, and decided to vote, your vote wouldn’t have counted.

Facebook said that they would consider any vote on their privacy policy as advisory unless a third of their 900M active users voted. The lack of publicity around the privacy policy change and serious questions about the veracity of Facebook’s claim of 900M active users all but guaranteed that, whatever the votes were, they would be advisory.

SPG: The Secure Password Generator

2012-06-10T16:48:00-04:00

As Users, We Must Expect Our Passwords To Be Compromised

It is an unfortunate fact that, if you create accounts on websites, some of your passwords will be stolen. For example:

There has been a lot of recent press about LinkedIn’s recent 6.5M stolen passwords.
In the week since LinkedIn’s attack became public, eHarmony lost 1.5M passwords in an unrelated attack.
Earlier in the year, an undisclosed number of Zappos 24M users had their passwords compromised.
The attack on Gawker Media’s collection of popular sites (Gawker, Lifehacker, Gizmodo, io9, Jalopnik, Kotaku, Deadspin, and Fleshbot) in late 2010 that compromised 1.3M accounts shows that this is not a new phenomenon.

Cascading Damages

Having your password stolen from a website is a problem. But if you use the same password for many sites, the problem becomes a nightmare. I know that everyone tells you to use strong passwords and to use a different password for each website – but few people do. A quick look at these Google search results should be convince you that people just don’t follow that advice.

The Reason Behind The Mega-Password Thefts

Despite every assurance made by websites (eg: LinkedIn’s recent message), the only way a website can lose millions of passwords is if they just didn’t bother to follow standard security procedures for storing your password. So LinkedIn, Gawker, Zappos, etc.., didn’t care about your password’s security until a breach made it into the headlines. As revolting as that fact is, it is accurate.

The trick to securely storing a user’s password is rather simple: don’t store the user’s password as it can’t be stolen if it is never stored.

Huh?

You heard me correctly. Websites should store a <geek vernacular warning> salted hash of your password rather than your password. Understanding the math and computer science behind salted hashes is unimportant. All you need to know is that the technology exists. If you are curious, you can read this OWASP article about salted hashes.

An Ounce Of Prevention

The only way to protect yourself against these thefts is to, as everyone will tell you, use a unique, strong password for every site. Don’t worry, I won’t preach to you. You know that this is what should be done and I know that you know that this should be done.

There are two problems with this:

It is difficult to keep track of so many passwords.
People are astoundingly bad at creating secure passwords.

Other than that, how was the play, Mrs. Lincoln?

Tools And Reprogramming Are The Answer

It turns out that the best solution to managing your passwords involves the use of tools and reprogramming. The tools are used to help you create unique passwords as well as remember them. The tools can be simple. For example, a handful of dice can be used to generate unique and secure passwords.

The reprogramming part is the more difficult. It involves you reprogramming your brain so that you protect yourself. Think of it as safe surfing.

The Answer Is…

The answer is going to be in my next blog posting. I’ll present several tools and strategies, including one that I’m currently developing, that work better than carrying around a handful of zocchihedrons :-)

My First Ruby Project

2012-06-01T19:18:00-04:00

The Project

Typically, even my pet projects have a specific goal in mind. I don’t write a spec for my personal projects, there’s just a, maybe slightly fuzzy, goal. Generally I want something that will be useful. I’ll work on a project until:

The project does everything I hoped for. Maybe even more!
I run out of spare time to work on it. This doesn’t happen as often as you may imagine. I really enjoy coding. So, after a tiring day of coding at the office, I come home to relax and work on one of my pet projects. If it was any different, then I have no right to call my blog My Geekdom.
I decide that I did a poor job picking a pet project as they’re supposed to be fun and this projects isn’t. So long!
I get stuck such that proceeding will require more effort than I feel it’s worth. That said, hopefully I learned something working on the project.

But, unlike my typical pet projects, my goal with my new project is rather amorphous. I want to analyze some data. At time of writing this, I have just shy of 5K records totally 181MB of data.

Leaving the details for a later posting, each record is a snapshot of the state of my Mac OS X computer. I’ve been collecting, and am continuing to collect, samples every five minutes since the beginning of March. At least every five minutes that my computer isn’t sleeping or hanging.

I decided to collect the samples because I was getting downright frustrated with the performance of Mozilla’s Firefox browser. For reasons that are off-topic for this posting, I feel a strong interest in making sure that Firefox continues to be successful.

For years I thought Firefox was the bomb. Starting about 9-12 months ago I started to notice that Firefox was routinely using all the resources on my computer. This concerned me because Google’s Chrome browser was starting to have enough functionality to be a primary browser.

Being that I wanted Firefox to stay competitive, I decided to try to stay with it. As Mozilla changed their release model, I started to use Aurora and Firefox Beta and provide lots of feedback. The newer versions also seemed to consume a more reasonable amount of my computer’s resources. Being that the Firefox team was working hard at fixing performance problems, especially memory leaks, I stayed with Firefox. (See Reference: Articles About Firefox’s Memory Leaks And Repair Efforts for information about FF’s memory leaks.) Around the beginning of this calendar year I started to think that Firefox wasn’t getting faster and may have been getting slower. That said, I can be a pretty heavy user of my computer. Running FF with a couple of dozen add-ons is typical. Also, my experience with performance tuning has taught me that I don’t know enough to make accurate guesses. Guessing can be used to lead your performance tuning investigations, but you always need hard data.

After a bit of thought, I decided to collect some data and see if I can convince myself that Firefox is or is not the problem. There are, of course, simpler solutions to tracking a performance problem than sampling your entire process every five minutes for months. But this is a pet project. It wouldn’t be any fun if I just switched from Firefox to Chrome to see if I notice a difference.

Now that I have accumulated a few months of the data, I need to try to analyze the data and see what, if any, conclusions I can make from it. I have some ideas about where to start my data analysis and will just need to follow the path they outline. In short, I’m prepared to MacGyver the analysis as I go along.

My new pet project is the exploration of these data samples.

The Programming Language

Forensic performance analysis is never simple. For this project I anticipate that the difficulties will be in determining what types of data analyses are required. I’m hoping the actual coding will be relatively simple as there is no need for network interaction, provide real time responses, or even have a GUI. As long as I don’t need to do very complex statistical analyses, the choice programming language won’t likely matter to much.

So I have chosen to implement this project in Ruby. I’m certain I could implement it faster in any of a few dozen other programming languages but I chose Ruby for two reasons:

This is a personal project. One of my main goals is to have fun. Being that I don’t know Ruby, learning it will be more fun. Especially being that I haven’t learned a new language in a year or so.
Ruby is a new-ish language, growing in popularity, likely to be around for awhile, so it seems like something I should learn or at least be familiar with.

Ruby isn’t wholly unfamiliar to me. I’ve made a few small (very small) changes to the org-ruby gem. Based on that experience, I think that the syntax of Ruby is kind of strange but the functionality it provides seems comparable to other languages such as C and Java. I just have to learn the new syntax. Unlike C and Java, Ruby has dynamic typing. But languages that I’m fluent in, such as Lisp, Python, Bash, and AWK have dynamic typing. Doesn’t seem a problem.

But the structure of a Ruby application, at least the structure of the org-ruby gem, feels different to me than the structure of programs I’ve seen in other languages. Sure Ruby has classes and modules and functions and inheritance as many other languages do. But the design of the org-ruby gem feels a bit unusual to me.

I think that part of the atypicality of the design of the org-ruby gem is that it is a parser that is written in a manner that is not familiar to me. I have a strong background in designing and implementing programming languages, their associated runtimes, IDE’s, and such. In my world, there are only a few ways to implement a parser and org-ruby doesn’t use any of them.

That said, I don’t think the feeling of unfamiliarity stems from org-ruby’s parser not being a recursive descent parser or implemented with a parser generator from a formal grammar. I’ve seem plenty of really terrible parser implementations that left queasy but didn’t feel unfamiliar.

Not only do I not think that org-ruby gem is terrible, I think it has a certain elegance to its design.

Having eliminated other possibilities, I’m left to think that the feeling of unfamiliarity must stem from Ruby itself.

For example, C and Pascal have a similar feel when you use them. So too do object oriented languages such as Java and C++. While not OO, C and Pascal are somewhat similar to Java and C++. But a Scheme application doesn’t feel anything like a C application (at least not if you understand how to program in Scheme). While I don’t have a word for it, I think that Ruby leads to a different way of writing applications than languages that I’m familiar with..

While I’m not sure what that difference is, I hope to figure it out as I move through this project. In the end, I don’t want to have a great “Java program” that happens to have been written using Ruby syntax. I want to have a great Ruby program – period.

The Project’s Current Progress

“Not much” seems to sum up my current project. While I’ve been collecting data for months and easing myself into Ruby by working on the org-ruby gem, I just started coding for this project a few hours ago. All I’ve done so far is write a tiny class to hold a line of data output by the ps(1) program:

 1:  class PSDataLine
 2:    attr_accessor :pid
 3:    attr_accessor :stat
 4:    attr_accessor :time
 5:    attr_accessor :elapsed_time
 6:    attr_accessor :virtual_size
 7:    attr_accessor :rss
 8:    attr_accessor :percent_cpu
 9:    attr_accessor :percent_mem
10:    attr_accessor :command
11:  
12:    # Create the object
13:    def initialize(line_string)
14:      line = line_string.split(' ')
15:      @pid = line[0]
16:      @stat = line[1]
17:      @time = line[2]
18:      @elapsed_time = line[3]
19:      @virtual_size = line[4]
20:      @rss = line[5]
21:      @percent_cpu = line[6]
22:      @percent_mem = line[7]
23:      @command = line[8]
24:    end
25:  end

I know it isn’t much but I still learned writing it. For example, when I was tinkering with the org-ruby gem, I just copied and pasted code, looked up some details in the Ruby library, etc… Being that I was modifying existing code, I never learned even the basic syntax.

For example, in order to write my PSDataLine class, I needed to learn how Ruby uses syntax to determine if an identifier is a local variable, an instance variable, a global variable, or a constant. (My thanks to the Rubyist for the good reference.)

I’m pretty busy this week so I don’t think I’ll make much progress on it. I was only able to get this work done by staying up till midnight on a Friday night and squeezing in time for a proofreading today.

Process Data

While I’ve only begun writing my program and haven’t reached any conclusions about anything yet, I do have lots of data. If you have an interest in seeing what my computer’s been up to you can view the process data, read about my sampling mechanism, and copy the data from my process-data Github repository.

Reference: Articles About Firefox’s Memory Leaks And Repair Efforts

A blog posting posting by Mozillian Nicholas Nethercote. A second posting of Nicholas about weeks 49 & 50 of the Mozilla effort to reduce the memory footprint.
A different engineer’s blog posting discussing the hunt for memory leaks in Firefox.
Another posting from Nicholas regarding the great [[http://bit.ly/LsNjF0][performance improvements in Firefox V7. - A September 26, 2011 article in CRN discussing the lack of progress Mozilla has been making. Even if you ignore any opinions in the article, it has a lot of facts and links.
A Firefox Forum posting expressing user frustration.

Org-Mode + t Makes Tweeting Easy

2012-05-27T18:23:00-04:00

Update 2012-06-04 Mon 00:57

You can download the code from its Github repository.

Motivation

After yesterday’s ranting about Octopress, it occurred to me that while I now own the content of my blog postings, I don’t own the content of my tweets. I’m sure that there’s some way to convince Twitter to give me my tweets, but why should I rely on them to store my tweets.

The obvious answer is that my tweets should be created on my computer and then sent to Twitter.

It should be noted that I’m not interested in implementing a full Twitter client. There are more than enough of those. I’m interested in having my tweets, or at least the bulk of my tweets, in a simple and easily accessible format.

The Answer Must Be Cheap

While I want to author and store my tweets locally, it is not a high priority task for me. Whatever solution I come up with must add little or no overhead to my tweeting workflow, must be simple to implement, and cannot require ongoing tinkering to keep working.

Being that my relationship with Org Mode is rather simple: The more I learn about it the more of my life I want to be based on Org Mode. Org Mode just seems to make everything easier. So I decided to use Org Mode for the author, publication, and storage of my tweets. The system I’m describing has these properties:

It took me just a few hours to design, implement, test, and begin using it for tweeting.
Besides not adding overhead to my tweeting, it has made tweeting dramatically simpler.
The entire implementation is a single file of elisp that is 120 lines long. 50 of those lines are standard Emacs header comments that are automatically created. The implementation, along with its documentation, is only 70 lines long. While a small code base isn’t one of my goals, not needing ongoing tinkering is. It is hard to imagine that 70 lines of code will require too much tinkering.

Advantages Of Org Mode

Before continuing, you should know a few facts about Org Mode:

At its core, Org Mode is an outliner on steroids. This works great for me as I tend to organize my thoughts as trees. I’m obvious not alone as the outliner page on Wikipedia has a list of many outliner applications along with references to many outliners that are designed to implement specific functionality.
Org Mode augments its outlining functionality with a task management system (details).
Org Mode has a template based data capturing system that allows for the easy capturing of data into a consistent format.
Org Mode is implemented in GNU Emacs. Besides the fact that I spend nearly all of my non-browser computer time in Emacs, Emacs has a rich extension mechanism

The Storage Structure Of Tweets

Each tweet would be an outline entry whose title would be a small description of the tweet plus some other information to be discussed later. The body for each outline entry would be the tweet. In Org Mode this would look like:

* My First Tweet
  This is my first tweet using Org Mode.

While this works fine for one tweet, after 50 or 100 tweets, things are going to get pretty messy. But, as Org Mode is an outliner, it supports the nesting of outline entries via both direct data entry as well as through Org Mode’s capturing mechanism. You just need to provide Org Mode with a string that describes the data you want to capture and the outline format for that data.

While, for the moment, sparing you the ugliness of the capture configuration string, I’ll tell you that Org Mode supports a date tree outline format.

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday
******* My First Tweet
        This is my first tweet using Org Mode!
***** 2012-05-27 Sunday
******* My Second Tweet
        This is now my 2nd tweet using Org Mode!

Being an outliner, Org Mode supports the folding of the outline. That is, you can collapse the outline to hide parts of it that you are not immediately interested in. For example, let’s say that, being today is Monday, I don’t want to be bothered looking at my tweets from Saturday Or Sunday. No problem. When you fold the outline headings for Saturday and Sunday, the list of tweets will look like this:

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday ...
***** 2012-05-27 Sunday ...

The “…” is Org Mode’s way to let you know that this entry has been folded.

Keep in mind that I get all of this storage mechanism from Org Mode for free. All I needed to do was create a single configuration string for it.

Capturing Tweets

A simple yet highly functional storage mechanism is fine but what I’m looking for is to reduce my tweeting overhead. This means that the workflow is what I care about.

The first part of the workflow is the “capturing” of my tweets. Org Mode’s capturing mechanism is rather rich. It allows me to capture data (ie: to tweet) from Emacs with a couple of keystrokes. It also allows me to capture data, including an URL and highlighted text from my browser by clicking on a bookmarklet.

When I initiate a tweet capture, I’m initially prompted for a “title”. That is the descriptive text for the tweet. In the above example, “My First Tweet” is a title. After entering my title, I’m given an Emacs buffer, basically a new Emacs window, in which to type my tweet.

When I’m done typing my tweet, a couple of keystrokes tells Org Mode that it should add the tweet to my tweet file and save the file.

Tweeting

Now that I’ve captured my tweet, I need to actually tweet it. This is where some of the wonders of Emacs come into play. Emacs has lots of hooks that allow you to insert code into the workflow. Org Mode adds many hooks of its own.

Without boring you with the exact details, by using the appropriate hooks I have set up Emacs to take special actions whenever a file is saved that ends in “-update.org”. I used the file naming convention of “-update” instead of “-tweet” in case I decide to add other types of status messages at a later date.

Being as I was going for simple, I decided that Emacs would not talk directly to Twitter. Instead, I have Emacs use the wonderfully intuitive, functional, and efficient t application. For my purposes, t’s CLI is perfect. Once you’ve signed into Twitter with t, it will tweet for you with the simple command line of:

t update This is my tweet text.

That seems about as simple as it can get.

Tracking Tweet Status

Being that I’m storing more than one tweet in a file, it is important that I know which tweets have been tweeted and which ones haven’t. This is another place where I get to leverage the wonders of Org Mode.

Beyond outlining, Org Mode supports a simple yet highly functional and customizable task management system. The customizability allows you to define task states well beyond the typical “open” and “closed” states of many other task managers.

I used the task management to teach Org Mode about status update tasks. A status update task can be in one of two states: POST and DONE. When it is in the POST state, it needs to be posted. The DONE state means it has already been posted. Org Mode’s slogan is

Org: Your Life in Plain Text

and they mean it.

To indicate a tweet’s, ie: a task’s, state, you simply add the state to the outline header line. For example, using the example from above, if I had already posted my first tweet but not my second, the outline would look like this:

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday
******* =DONE= My First Tweet
        This is my first tweet using Org Mode!
***** 2012-05-27 Sunday
******* =POST= My Second Tweet
        This is now my 2nd tweet using Org Mode!

When I use Org Mode’s capturing mechanism to add my new tweet to the file, it automatically adds the state of POST to each new tweet. As mentioned above, when the tweet file is saved it executes my code. While I have 70 lines of code, only two functions consisting of a total of 45 lines is needed to post my tweets. The remaining 35 lines are configuration that, via Emacs and Org Mode’s hooks, makes sure that my code is called whenever the tweet file is saved.

The first function, all 9 lines of it, simply asks Org Mode to loop through all tweets (ie: outline headings) in the POST state and call my second function on each of those tweets. The second function calls the CLI t to execute the tweet. This 28 line function that actually does the tweeting is really rather simple. Only 10 lines of it are related to actually tweeting. The remainder of the function is error handling in case, for some reason, my tweet fails.

After successfully tweeting a status update, I tell Org Mode to change the tweet’s state from POST to DONE. That’s all there is to it.

Some Nice Freebies

Besides the data capturing, the ease of calling an external program, the automatic outline maintenance, the automatic state tracking (ie: POST and DONE), and the automatic execution of my tweeting code whenever the tweet file is saved, the Emacs and Org Mode’s richness provide other nice features:

Twitter’s 140 character limit per tweet is easily eyeballed while you are typing in Emacs as the editor tells you how long a line is. I have, of course, implemented a character count error check before tweeting, but it is nice to catch the problem when you are typing.
Automated highlighting of tweets.
A common problem with the 140 character limit on tweets is that URLs can be very long. Emacs helps here with its Bitly library (courtesy of me :-). This shortens URLs in Emacs, using Bitly, with two keystrokes. (Bitly happens to be my current URL shortener of choice.)
Org Mode has the ability to automatically record a variety of times associated with each task. I have set up Org Mode to record the time the tweet was entered by me as well as the time that it actually got tweeted.
Org Mode allows you to tag (aka: “label” or “categorize”) each outline item. I have leveraged this in anticipation of extending the tweeting system to include status updates to other services than Twitter. Also, in the future I may want to be able to manage multiple Twitter accounts. I’m not sure that I’ll ever implement that functionality, but thought I’d throw it in as preparing for it is essentially free. I currently label each status update with two labels. The first label describes the type of status update. In this case it is “TWEET” to designate that I want to update Twitter. The second label is “NS” to indicate that I want to update the status of my @neil_smithline Twitter account.
Emacs is my tweet editor. For those of you who don’t know Emacs as well as those of you that know Emacs and hate it, I’m sure you wouldn’t want to edit tweets in Emacs. But I have been using Emacs for 25+ years and still think it is pretty slick. Besides the functionality discussed above, Emacs gives me spell checking/correction, the ability to easily change the case of something I’ve mistyped, functionality based on English grammar (eg: the ability to move, delete, or even transpose words or sentences), etc…

A Sneak Peek

After all my blathering, I thought that a screenshot of my tweets would be nice. This screenshot shows tweets for the past three days. I have folded the tweets from Saturday so that they are easy to ignore. For the sake of this picture, I have unfolded my tweets from Sunday so that it shows what a posted tweet looks like. Finally, I have a tweet for today that is queued up to announce this blog posting.

As you look at this screenshot, remember that almost all of this is automatically generated. For example, for the final tweet I had to enter the title “Tweeting From Emacs” (this can actually be blank but I think the title makes it more useful) and the actual tweet. Everything else, including the Bitly URL shortening, comes for free.

For Technorati: PHTDYXEZKM3Q

Octopress Is Pretty Great

2012-05-26T13:32:00-04:00

Jekyll Overview

I have simply fallen in love with Octopress. I think I mentioned previously that Octopress is an auto-configuration tool wrapped around Jekyll. And Jekyll is the technology behind Github pages. As such, if you use Jekyll for your blogging platform, you can preview your blog locally, push it to Github, and let Github do all the page serving.

This isn’t actually as much work for the Github servers as you may think. Jekyll is a static blogging system with no CMS support. A static blog needs to be compiled whenever content is modified, but once compiled, the only thing the server needs to do is serve the pages. Dynamic blogging systems such as Wordpress and Joomla are programs that run on the server and work interactively with the web server to produce each page.

Configuration

Octopress’s biggest addition to Jekyll is that it automagically configures Jekyll for you. I have found that configuring Jekyll is a bit of a pain. Jekyll decided to trade easy configuration for flexibility. This is a common decision in applications and one I can’t really argue with. But Octopress packages a mostly configured Jekyll installation in a neat little package. The introductory paragraph on Octopress.org states:

Octopress is a framework designed by Brandon Mathis for Jekyll, the blog aware static site generator powering Github Pages. To start blogging with Jekyll, you have to write your own HTML templates, CSS, Javascripts and set up your configuration. But with Octopress All of that is already taken care of. Simply clone or fork Octopress, install dependencies and the theme, and you’re set.

While that may sound a bit intimidating to the less-computer literate, the Octopress site carefully steps you through the process. I’ve installed and used Wordpress, Joomla, Drupal, Nanoblogger, and many other blogging systems. I can say that Octopress is, by far, the easiest.

The above quote states three steps for installing Octopress but there are actually a few more steps:

Install Git
Clone or fork Octopress
Install dependencies
Install theme
Configure your blog
Configure deployment

Git

If you don’t have Git installed, you need to install it. This was a trivial task for most users but, thanks to the clever folk at Github, has become idiot-proof. Github created a Mac OS X and Windows application that allows easy use of Github. These make installing and using Git user-friendly for those who prefer to avoid CLIs.

Clone Or Fork Octopress

Cloning Octopress, simpler than forking it, is a single command line

git clone git://github.com/imathis/octopress.git octopress

It takes only a few mouse clicks if you are using one of Github’s applications to do the clone.

Install Dependencies

Installing the dependencies is, yet again, a few simple CLI commands described in an easy-to-read style on Octopress’s setup page.

Install The Theme

Installing the theme is a single CLI command:

rake install

There are additional theme installation options but you can simply ignore them as the defaults work well. In a subsequent blog post I’ll discuss how I have made a minor alteration to the Octopress installation that allows me to entirely change my theme as often as I like without losing any data. Simple theme changes (eg: page colors) are easy to make at anytime.

Configure Your Blog

Configuring your blog is perhaps the most complicated step in the entire installation process. You’ll need to open a text editor such as Textedit on a Mac or Notepad on Windows.

The configuration file is written in a format called YAML. That sounds a lot scarier than it actually is. Octopress’s configuration page states that you just need to edit the existing _config.yml, and modify the following lines:

url:                # For rewriting urls for RSS, etc
title:              # Used in the header and title tags
subtitle:           # A description used in the header
author:             # Your name, for RSS, Copyright, Metadata
simple_search:      # Search engine for simple site search
description:        # A default meta description for your site
subscribe_rss:      # Url for your blog's feed, defauts to /atom.xml
subscribe_email:    # Url to subscribe by email (service required)
email:              # Email address for the RSS feed if you want it.

Several of the configuration options have default values or can be left empty.

This configuration step in Octopress differs from other blogging systems only in that you have to use Textedit or Notepad to enter the data. Many other systems allow you to enter the data into a GUI or a web page.

Configuring Deployment

Configuring your deployment, the final setup action, involves telling Octopress what computer is going to host your blog. While you can run your blog right on your desktop or laptop, it will only be visible by other people in your home or office. This is helpful if you want to preview your blog before “publishing” it so the rest of the world can see it but when you want to publish your blog you will need to have access to a hosting computer (AKA: a server) that is on the internet 24x7.

Getting access to a server can be less than straightforward and almost always costs money. But this is rather simple with Octopress because:

Octopress is a static blog. This is compared to a dynamic blog that requires programs to interact with the server. Once Octopress has done its job, you just need to put the blog into the right location on your server and everything will magically work.
Octopress, the blogging system that tries to make configuration easy, has carefully documented, in easy-to-read text, three different ways to host your blog on a server. Two of them are absolutely free: Github itself (the one I use) and Heroku. The third strategy using rsync is a bit more complex and typically costs some money to use. That being said, the documentation guides you to a hosting company that charges only $5 a month for server access. Unless you know that you need a server for something besides hosting a blog, I strongly recommend going with one of the other options.

As I said, I am using Github to host my blog. It takes exactly one line to configure your blog to use Github. I haven’t used the other deployment mechanisms but Heroku seems to be of equal complexity while rsync definitely requires more work and a greater knowledge of computers.

Configuration Summary

While there is a lot of descriptive text above, the entire configuration process took me a couple of hours the first time and about 20 minutes the second time. Unless you are like me and enjoy trying stupid things, you’ll only need to install it once :-) I would imagine that a non-technical users could easily install Octopress in half a day.

The only exception that I can think of would be if you’re on a Windows computer and have lots of difficulties installing the dependencies. Mac OSX and Linux come with most of the dependencies pre-installed. I don’t think that Windows does.

It’s Blogging Time!

Before you can actually start to blog, you need to choose an editor and a markup language. While you can change these at anytime, you need one of each to blog. I’ll discuss this in a subsequent blog posting where I discuss my personal Octopress configuration.

Now that everything is configured, it’s time to blog. While this involves using the command line, it is a trivial usage. When you want to create a new post you execute the command:

rake 'new_post[Name Of My Post]'

The reason why that is the magical incantation to create a new post is unimportant. All you need to do is remember (or be able to lookup) that command.

When the rake command finishes, it will tell you the name of the file for your new posting. Just open the file and start typing. If you want to see a preview of your blog before publishing it to the server, run the command

rake preview

Then you can open your web browser to http://localhost:4000 to see your blog. I strongly recommend that you copy-and-paste that URL and then bookmark it.

Once you are ready to publish your blog to the world, execute the command

rake gen_deploy

And voila! You have updated your blog.

Octopress Summary

Cons

Configuration

Configuring Octopress is definitely more complex than configuring a blog on Blogger.com or a similar service. If you are looking for the easiest solution, no-frills solution, go with one of those.

Single Computer Publishing

While I haven’t seen it explicitly stated in any documentation for Octopress or Jekyll, they really are intended to publish new posts from a single computer. You can author the posts on any computer or network service. The first option that jumps into my head is Google Docs. But to publish (ie: to deploy) your site from more than one computer will definitely require extra configuration, computer skills, and tinkering. If you absolutely must publish from multiple computers, the online blogging systems may be a better choice.

Pros

Configuration

While configuring a blog on a site like Blogger is definitely easier than configuring Octopress, if Blogger doesn’t meet your needs, I think Octopress is easier to manage than all Joomla, Drupal, and all of its competitors – at least the ones I’ve seen.

Content Possession

When you use a site like Blogger.com, the content you have posted is in the possession of Blogger. They determine how they will store your content, how they will let you export it, modify it, delete it, etc… But with Octopress, you are always in possession of your content. In order to have your blog visible on the web, you must deploy it to a server, but that is just a copy of your content. You are always in possession of the original content.

Content Accessibility

Whether you are using an online blogging service such as Blogger.com or hosting your own blogging application such as Joomla or Drupal, your postings are typically not stored in a convenient manner. Even if the blogging system lets you get access to all of your content, the content is frequently meaningless or difficult to use.

But with Octopress, the content for each posting is in a file. One file for each posting. While I’ve not discussed choosing an editor or a markup language in this posting, it is your choice of editor and markup language that determine what you can do with your content.

For example, I was blogging with Nanoblogger for a few months. As it required continual tinkering, I moved to Octopress. Like Octopress, Nanoblogger lets you choose your editor and markup language. Because both blogging systems give you accessibility to your content, I was able to convert about 20 Nanoblogger postings into Octopress postings in about 20 minutes. I imagine it would have taken less computer-savvy users as long as 2-3 minutes a posting.

The one-file-one-posting principal of Octopress, and Nanoblogger, make life so much simpler.

And It Looks Good Too!

Even after customizing them, I find that Wordpress blogs tend to be drap and all look like this and that Blogger.com blogs require lots of tinkering but still tend to look like this. On the other hand, Octopress blogs tend to look like the Octopress homepage or my blog.

IMO, Octopress looks great out of the box. I am partial to using dark themes where black, white, and shades of gray are the main colors. It took me about 10 minutes to change the default colors for Octopress to use my preferred color pallette.

It Will Grow With You

I think that Octopress’s greatest features are the above mentioned ease-of-configuration and the fact that, while it is hidden from you, it is built on a very strong blogging engine, Jekyll. So, if you wake up one day and say “Gee. I wish my blog could do XYZ.”, the odds are that it can. Just google “Jekyll blogs XYZ” and, at least for reasonable XYZs, you’ll find that instructions about reconfiguring your blog so it can do XYZ.

There are also many discussion groups, forums, and helpful Jekyll users in the blogosphere that can offer suggestions, provide pointers, or even make a small modification to Octopress/Jekyll so that XYZ is easy to do.

Future Postings About Octopress

This is now my second posting about Octopress. My } discussed my initial opinions about Octopress. They were largely positive but so too were my } of Nanoblogger.

At the time of writing this blog posting I’m certain that I’m going to love Octopress. (Though, if I change my mind, I am in possession of my own content so it is easy to switch to another platform.)

My next posting about Octopress will explain the minor modifications I have made to Octopress. I made these modifications to accomplish two goals:

Allow me to easily use Org Mode as my markup language and Emacs as my editor. While I simply love this setup, I imagine that it will only be of interest to preexisting Emacs users.
Keep my content totally separate from the Octopress’s content. By this I mean that if, you were to burn my computer as I’m typing these very words, I could have Octopress up and running on a new computer with all of my content, including postings, images, etc.., in about 20 minutes. IMO, this an improvement on Octopress’s configuration and should probably be changed in Octopress.

YANBT: Yet Another New Blogging Technology

2012-05-14T12:36:00-04:00

NanoBlogger Was Killing Me

NanoBlogger really looked cool to me (my postings about why I liked it):

Small
Most functionality provided via plugins
CLI only
It is FOSS.
Produced a static site (ie: only serves static pages)
It was built on Bash – I thought this was the coolest

I liked it so much that I wrote a CLI for it that was better suited to my personal workflow. I also spent some time integrating it with Emacs. But it had endless problems.

It was slow, it did not do anything automatically, frequently required rebuilding of the entire blog, was hard to view on my local machine, isn’t really being maintained, doesn’t have a real templating system, breaks the RSS feed on special characters such as etc…

Where Next

Over the past few weeks I’ve been looking around for a new blogging system. The only real contender seemed to be Jekyll. It is written in Ruby and used by Github to power Github Pages. It has so many contributors to it that its Github repository has, at the moment, 50+ pull requests.

I had nearly chosen Jekyll when I moved to NanoBlogger. I found that the discussion about moving to Jekyll on the 0xdecafbad.com blog really aligned with my thoughts.

But, after playing with it a bit, I found that Jekyll was difficult to configure. And I just so liked the Bash thing that NanoBlogger had that I decided to go with NanoBlogger. But now I’m wasting too much of my time fighting with Nanoblogger and I should reconsider Jekyll. This time, when I was researching Jekyll, I found Octopress. Octopress is Jekyll on steroids. It provides an automated configuration and maintenance system that takes the pain out of using Jekyll.

So now I’m giving Octopress/Jekyll a try. I’ll see how it goes. Configuring a site with a hello world blog entry only took a couple of hours. Most of that was just downloading Octopress and reading the docs to figure out what to do. It turned out that configuration was running three rake commands.

Converting From Nanoblogger To Jekyll

Because NanoBlogger blog entries are plain text with markup, converting my entire old blog to Octopress was fairly easy as Octopress also uses plain text with markup for blog entries. I ran into two snags. The first one was in Emacs’ Org Mode, my authoring tool. It broke when you had a “+” in the path name of your input files (in my case, in the path of my blog). It took me several hours to find the bug. Once I found it, it was easy to patch and I sent my fix to be included in subsequent Org Mode releases.

The second problem was a bit trickier. Octopress was breaking on some of my blog’s generated HTML files. The error message gave me the line number and the character number in that line that was causing the error but it omitted to give me the file name. I believe the technical term for that is “Oops!”.

To find the file that was casing the problem I started to remove blog files using a binary search-ish style to track the problem file down. This proved more difficult than I thought it should. After much frustration, it occurred to me that the problem was in many blog entries so the binary search wasn’t so helpful. So I wiped my blog and started adding one posting at a time.

Even after I found the first posting that caused Octopress, actually Jekyll, to crash, I couldn’t figure out what was wrong with the file. The error message said something about non-ASCII characters but I only had ASCII, or so I thought. It turned out that I had copied and pasted some text from my browser and it had inserted a hyphen (-) that was not the ASCII hyphen. To me, they looked identical.

Despite knowing what the problem was, I couldn’t automate Emacs or grep to find the offensive hyphens. In Emacs, the search treated ASCII and non-ASCII hyphens the same. I forget the problems I had with grep but I couldn’t get it to work either. This meant I was going to have to add one entry at a time to my blog to track down the other problems. Thank you – NOT!

But Jekyll is written in Ruby and Ruby is a scripting language so I have the source code for Jekyll. Being that the error message included a full stack trace, I figured I could update the error message. In the end I just ended up making Jekyll a bit more verbose: it announces which file it is about to process with the message: :Processing file myfile.

Then I added all my blog entries, ran Jekyll, got the same error message with only the line and character position, but now had the name of the file it was processing when it had a problem.

Plain Text Blogs Rock!

After correcting a couple of entries I noticed that the problem only occurred on hyphens (-) and single quotes (‘). Being that both NanoBlogger and Octopress/Jekyll use plain text files for their input, I had a whole bunch of tools to help me fix the problem.

In the end I ended up turning Emacs failure to distinguish between the ASCII and non-ASCII versions of these characters to my advantage. I did a search-and-replace on all my blog entries replacing hyphens with hyphens and single quotes with single quotes. While Emacs’ search found both ASCII and non-ASCII characters, it always replaced them with what I typed as the replacement, ASCII.

Then, voila, there was my blog. And my blog looked way better in Octopress than NanoBlogger. Check out a NanoBlogger sample blog and an Octopress sample blog. Huge difference. Also, excluding the weak error message from Jekyll (I’ll file a bug report on that and I suspect someone will fixed it as Jekyll is so heavily used), I didn’t need to do a thing to build and publish the blog. Octopress even pushes the blog to Github automatically. I had written a script to do this automatically for NanoBlogger, but I’d much rather use someone else’s perfectly good solution rather than writing my own.

I think I’m going to be very happy with Octopress.

SparkleShare Doesn't Sparkle

2012-05-09T23:12:00-04:00

Based on a LifeHacker posting, I previously blogged about giving SparkleShare a try.. It was cool. A free Dropbox with private hosting. So I installed, configured, and started using SparkleShare. While I was quite excited when I embarked upon my journey using SparkleShare, things didn’t work out as well as I had hoped.

I quickly found out that SparkleShare doesn’t handle symbolic links. Not at all. There’s no workaround. The problem is that SparkleShare relies on Git for synchronization and Git doesn’t support symbolic links. At least for me, this rendered SparkleShare useless.

When I moved to Dropbox directory, what I had been using for synchronization before SparkleShare, had no files in it, just a bunch of symbolic links. But with SparkleShare, you have to move all your directories into your SparkleShare directory and then scatter symbolic links around your system where the you want the directories to be. This caused a variety of minor but really annoying problems.

But SparkleShare totally fails if you want to synchronize a directory that has symbolic links in it. SparkleShare’s forum was covered with complaints about the symbolic link problem.

I am currently using the following scheme for file distribution/backups:

I’m using rsync to keep my media, bin directory, dot-files, and the like in sync on all my computers.
For actual backups, I’ve been using Crashplan for years and couldn’t be happier.
And, in the past week or two, I’ve put Dropbox synchronization back into the mix to manage synchronization between Org Mode files on my computers and MobileOrg files on my Android. Despite the strong cult following that Dropbox has, synching with my mobile seems to be the only effective use I can find for it.

Blackberry's Manufacturer RIM Nearly RIP

2012-03-31T20:35:00-04:00

RIM No Longer A Viable Company

RIM, arguably the manufacturer of the first successful smartphone, the Blackberry, has not only lost its market lead, but is facing a rather unpredictable future. RIM’s new CEO, Thorsten Heins, has stated that RIM is open to acquisition. At least for the short-term, his statements have led to a one-day, 5% bounce back of the stock price which was down 70% in the 12 months.

RIM’s History

For those of you that are unaware of RIM’s history, it got its start in Canada as a 2-way pager provider. In 1999, RIM moved into the smartphone market with its first Blackberry and immediately achieved market dominance.

Factors that led to their dominance included:

Intelligent Marketing: One example was a giving away pre-1.0 Blackberries to attendees of the 1999 JavaOne conference. I imagine that there wasn’t a better set of beta testers than the gadget geeks that attended JavaOne (myself include).
Sales Channels: RIM was selling its 2-way pagers to professionals who had a need to be in constant communication such as doctors, attorneys, business executives, and law enforcement. With the introduction of the Blackberry, RIM just added a better product to offer those same professionals. Sure it was more expensive, but 2-way pagers were already fairly costly. They were the perfect audience to market smartphones to.
First To Market: In 1999, most mobile providers were focused on upgrading their networks from analog to digital. Once the digital networks were in place, new functionality such as internet access would be easy to providing. But upgrading networks takes time. It involves men climbing towers, FCC regulations, local zoning laws, etc. But RIM didn’t have to do this.

RIM’s Brilliance

In what I consider a spectacular marketing and technological move, RIM started providing internet service over their existing 2-way pager network. While only a short-term solution as the 2-way pager network was designed for low-bandwidth usage, it was instantaneous. The first Blackberry was basically a 2-way pager with a good sized screen and full QWERTY keyboard.

What Went Wrong

RIM’s mistake was that they didn’t learn from prior boom-to-bust technology companies. RIM seems to have perfectly followed the (mis)steps of Digital Electronics. While IBM has survived, it lost its market dominance in a sector, twice. The first time as a manufacture of mainframes in the 1970’s and then again as a PC manufacturer in the 1980’s. Apollo Computers, SGI, Xerox, all leaders at one time, are either gone or no longer of interest.

All of these companies suffered from a similar set of mistakes including:

Overconfidence: Each company thought that they were invulnerable. IBM is perhaps the best example. In the 1970’s, when an IT manager had to make a decision about what gigantic computer to buy, they often said “Nobody ever got fired for choosing IBM.”.
Lack of Foresight: Technology and markets change and so too must your company. Clearly RIM, along with its arch-rival Palm, both missed the boat. Phones went from being personal communication devices to be tiny, fit-in-your-pocket, computers.
Unwillingness to Self-Cannibalize: In order to stay ahead of the technology market, you must be willing to come out with products that are better and cheaper than your existing products. HP and Goodyear are two companies that get this. Within a product line, HP routinely comes out with a better and cheaper printer than the one they currently have on the market. They know that it is going to kill sales of their existing printer. But HP also knows that when their customers need a new printer, it will be an HP printer. I know that Goodyear isn’t considered a technology company, but their product, tires, involve high-tech materials and manufacturing processes. The CEO of Goodyear once asked how do you keep a company in business when you know that your next product is going to be cheaper, better, and last longer than your current product? While I think that HP and Goodyear would both prefer to keep selling their old products, they know better. It seems RIM didn’t.

The Final Outcome

In the end, RIM, the makers of the Blackberry that was a status symbol for business executives in the 1990’s, has all but been put out of business by Apple, a waning computer company in 1990, and Google, a non-existent company in 1990.

The trick to surviving in the high-tech marketplace is simple. All you need to do is realize that your market share is never safe, that there is always another company working on something that is better than what you have, and that you must act if you are fighting for your life, all-day, everyday.

Automated Gvoice And Gmail To Gcal

2012-03-31T16:14:00-04:00

I seem able to hold incredibly large amounts of technical details in my head. In contrast to this, I’m excited when I can remember what year my kids were born in. From my viewpoint, this is just the way I have always been. From the viewpoint of people in my personal life, it can be frustrating.

Being the technophile that I am, I’ve incorporated many electronic assistive technologies into my life. In 1989 I got my first PDA. It was about the size and weight of a half-brick. But I could carry it wherever I went and it would ring to remind me to do things. What a deal!

My newest electronic assistive technology is a rather complete phone and SMS management system. Its features are:

I have a permanent record of every incoming or outgoing call or SMS.
My mobile phone rings whenever I get an incoming call or SMS.
My home phone rings whenever I get a call.
My mobile rings and gives me a notification whenever I get a call or SMS.
My computer rings and gives me a pop-up whenever I get a call or SMS.

This blog entry is unfinished. It will be updated or followed up with a second blog entry shortly.

Cloud Computing: How Expensive Is A Rainy Day?

2012-02-29T11:42:35-05:00

According to The Register (via Slashdot), Microsoft’s cloud computing platform, Azure, has been down, and I mean kaput, for 8+ hours. Worse yet, Azure’s status page that tells its customers about outages and expected repair times, to which I won’t provide a link, has now been Slashdotted and is down.

Less than a year ago, AWS suffered a significant, though less serious, outage.

While highly-distributed, cloud-based computing is clearly the future of IT, how can an organization understand the costs and prepare for the unavoidable outages? With the relative newness of 3^rd party cloud services, what statistics are useful. For example, let’s consider a fictitious service cloud service Neil’s Cloud Computing or NCC. NCC is a more established cloud computing service. Its detailed records shows that it has been open for 1,701 days (approximately 4.7 years). In that time it has steadily grown to here it now runs tens of millions of instances of virtual computers, services tens of billions requests each day, and has never had a serious outage.

To be generous, let’s define a serious outage as any outage that renders key parts of the site unusable to a significant portion of the site’s users and lasts a minimum of five minutes. We’ll consider any outage less than that a tolerable outage.

Even with all of this information, how can you calculate the cost of moving your enterprise to a cloud computing platform in general or NCC’s in specific (let’s assume that NCC is the best service you can find - after all, it is named after me). Even if your is willing to ignore all costs associated with the generously named tolerable outages, how can you know NCC’s 1,701 days of uptime mean that it will have another 4.7 years of uptime? How can you know it will be up tomorrow?

The answer is simple: You can’t! Just as it says on the bottom of financial documents, past performance is no guarantee of future results.

But, when it comes to managing an enterprise, “can’t” means things like “can’t go faster than the speed of light”. Everything else is just “How much?”. How much does our current IT solution cost? How much would a cloud computing solution cost? How much do our current outages cost us? What are the odds of having a serious outage if we moved to the cloud and how much would that cost us? Etc…

Smartphones Everywhere

2012-02-22T13:38:14-05:00

Interpreting this chart from Nielsen (via Wired) does not require any great amount of effort.

It can be summarized by any of the following sentences:

The greater your income the more likely you are to have a smartphone.
Young adults are more likely to have smartphones than older adults.
Smartphones are only for the young and get outgrown by the old. (Nah! But it is a valid interpretation of the chart.)

But I think the graph can best be summarized by:

Smartphones are where it is at today; and tomorrow, even more so.

Perhaps not the best grammar but I think it gets the point across. It becomes even more dramatic when you look at the number of people who have acquired smartphones in the past three months:

As Robin would say: Holy ubiquitous computing, Batman!

It seems clear that this is where the money is. This means that some subset of technologies such as HTML5, virtual computing, Scala, Hadoop, AWS, and CouchDB, to name just a few, are going to be the technologies of tomorrow.

In my lifetime, I have seen computers revolutionize society in several different ways.

Back office: Back office computerization led to the popularization of ATMs, credit cards, Fed Ex and its overnight delivery service, and set the stage for the online marketplace we are familiar with today.
Personal computers: Personal computers introduced computing to the public. When I was in school, I would get bonus points for typing my homework assignments. Today, homework assignments must be printed from a word processor if not submitted electronically.
The web: The web, for the first time ever, connected the back office directly to consumers in the comfort of their own home. ATMs allowed you to get 24x7 access to your bank account, but you needed to be at an ATM. PCs gave you access to many digital technologies, but having multiple computers interact was difficult, slow, and frequently expensive. But throw in the web and suddenly: BOOM! You have the online shopping, video gaming, and video on demand that we live in today.

As a diehard nerd, I’m thrilled to have lived through these technical revolutions. Not only have I lived through them, but I’ve had the honor of having been an early consumer and even helped develop some of them. So, with great joy, and without trying to sound too greedy, I am joyous that the next revolutionary wave is coming to shore. And it is called ubiquitous computing.

I don’t claim to be the prognosticator of prognosticators. Wikipedia’s page on ubiquitous computing was created two years ago. I’m just happy to be around to be part of it and hope that I get a chance to surf the wave.

My Geekdom

Disk Crash, Part 3: OS Indepent Configuration

Managing Configuration

Home Directories

My $PATH

Dynamically Assigned $PATH Variables

Miscellaneous Path Differences

Managing rbenv

Managing My .emacs

In The Next Installment

Disk Crash, Part 2: My Backup Strategy

My File Distribution And Backup Strategy

My Backup Strategy

CrashPlan

Not Just Backup, Secure Backup

Backup Everything

My File Distribution Strategy

Dropbox

Dropbox Security Concerns

rdist + ssh

Next Up

Disk Crash

Ouch! My Disk Crashed

File Distribution

Prioritization Of File Backup And File Distribution

There’s Data That Should Be Backed Up That Is Not In A File

Monitor Installed Applications

Other Missing Data

Other Backup And Distribution Strategies

Time Tracking, Part 2: Org Mode File Format

A Brief Introduction To Org Mode File Syntax

Hierarchical Outlines

Adding Tags

Using Tags To Describe Multiple Facets Of A Task

More Info

Time Tracking

The Quantified Self

My Project

Implementing Time Tracking

In The Next Posting

Blurring Static And Dynamic Blogs

Static Blogging

Website Hosting

JavaScript Helps

Pushing Beyond Web Hosting JavaScript

Site Search

Automated Tweeting

Tracking “The Buzz”

The Age Of Cloud Computing

Desktop Time Tracking

Motivation

Basic Activity Tracking

From Activity Tracking To Category Tracking

Choosing Between Timing And Time Sink

Analyzing The Data

Early Payoffs

UPDATE 2012-07-22 Sun 15:50

Footnotes:

Bookifying News

Bookifying News

The Process

Books And Websites

Yahoo! Forgets To Proofread Emails?

Yahoo! Notification

Is There A Proofreader In The House?

5 Days – You Gotta’ Be Kidding

Conclusion

The New, Privacy Invading, Facebook Like Button

Facebook Keeping Tabs On You

Even I Can Count To Three

Facebook’s Dilemma

Facebook’s Bait And Switch

Alternates

Update 2012-06-14 Thu 12:39

SPG: The Secure Password Generator

As Users, We Must Expect Our Passwords To Be Compromised

Cascading Damages

The Reason Behind The Mega-Password Thefts

An Ounce Of Prevention

Tools And Reprogramming Are The Answer

Managing `rbenv`

Managing My `.emacs`