My Geekdom

Blurring Static And Dynamic Blogs

2013-03-12T16:51:00.000-07:00

Static Blogging

I've discussed the pros and cons about static blogging many times before. In summary, the cons are that, well that your blog is static. No surprise there.

Fortunately, there are ways to push the limits of static blogging so that it is not quite so static.

Website Hosting

Unless you are self-hosting your Octopress or Jekyll blog, you are using another site to host your blog. I host mine on Github. They have a web server running 24x7 to host it. But that's not my problem. I just use their (free) services.

I understand that website hosting isn't a terribly exciting example of stretching the bubble of static blogging, but it is an example nonetheless. Dynamic blogs require specialized software running on their hosting computer to work. Static blogs tend to be able to run on generic web servers but there still must be a web server or there's no blog.

JavaScript Helps

Out of the box, Octopress comes with some features to show your most recent tweets as well as recent activity within your Github repositories. This is dynamic data that Octopress implements with some JavaScript.

Just to be clear, by dynamic data I mean that the data displayed on the page may change without a republication of the blog. I am not necessarily referring to a scrolling widget or something. Just dynamic data.

Conversely, JavaScript can be used to dynamically display static data. That is, data that is embedded in the source of the web page. Being that the data is embedded in the page's source at or before publication time, it is assuredly static data.

But client-side JavaScript (ie: JavaScript running in the user's browser) has its limits. For example, if you want to send a tweet whenever you add a new blog entry, client-side JavaScript is going to have a hard time at it.

Pushing Beyond Web Hosting JavaScript

The trick behind adding a much wider array of dynamic features to your static blog is:

Integrate with external web services and let them do the work for you.

It is even better if you can let other sites do the hard work for you free of charge. For example, Github is hosting this blog at no cost to me. Thank you Github! But Github won't tweet about my new blog postings or search through my website. Fortunately, there are services that will happily do that for you.

Site Search

This blog's search functionality is provided by Google. Dynamic blogging systems such as Joomla have a large array of search extensions that you can run on your server to provide various searching features. I'm sure that they provide a much larger set of functionality than my site's Google search provides.

Richer functionality or not, each extension needs to be installed, configured, and maintained. With Octopress, you never need to think about your site's search functionality. The site comes preconfigured for Google site search. That works for me.

Automated Tweeting

There are many ways to tweet via an email message but there is no way, at least none that I've bothered to think through, to get Github to send emails formatted for a tweet whenever you post. But, there are many sites that will watch your blog for change and perform some activity when your blog changes.

Exactly which site you use depends on what your needs are. For automated tweeting I decided to use FeedBurner. If you're unfamiliar with FeedBurner, it provides a web service that takes your site's RSS/Atom feed as input and adds value to that feed. My site's RSS feed is http://feeds.feedburner.com/MyGeekdom. Assuming you don't have uncommon browser settings, if you visit that URL you will see something that looks like:

Amongst the multitude of options that FeedBurner provides is the ability to subscribe to a site by email. So I subscribed myself, using a unique email alias, to FeedBurner emails about updates to my site. I then created an email filter that automatically forwards the FeedBurner emails to my tweeting email address and voila, whenever I post a new article I automatically produce a tweet like this one.

Tracking "The Buzz"

"The Buzz", also known as "Trending" or simply "current events", is a term to describe what is popular on the internet at any given moment. While tracking all of the internet's news doesn't seem appropriate for a blog, tracking news related to your blog posting may be desirable.

Courtesy of SocialMention, many of my blog postings have current news in the sidebar (see example blog posting). In many ways this is similar to tracking a Twitter #hashtag. But SocialMention gathers data from many dozens of web sites providing a richer and more diverse collection of data than Twitter can.

For example, by tracking several popular bookmarking sites such as Delicious, SocialMention can tell you what pages people are bookmarking that are related to your blog posting. Most people, thankfully, don't send a tweet whenever they bookmark a site.

While it may be redundant to state,

Displaying current events on your blog postings means that they will always have current information.

This means that there's no need to update your blog posting with links to relevant websites as new information becomes available. That is clearly a task that would become unmanageable as your blog grows. Instead, your static blog will always have dynamic data on it.

The Age Of Cloud Computing

While my blog is far from a large operation, the advent of cloud computing has made these dynamic features possible. From tiny systems like my blog to large systems like Google's search engine, cloud computing is about wiring together the most appropriate services in a manner that produces your desired functionality with a minimal amount of setup, cost, and maintenance.

While I don't have exact dates, my automated tweeting on new blog posts has been working maintenance free long enough that I forgot that I had implemented it until I noticed that I had just tweeted about my previous blog posting. I believe that is the very definition of "minimal maintenance".

Given the large and growing number of web services in the cloud and my limited time to explore them, I'm sure that this is but a small sampling of the different ways to use dynamic data in static blogs.

Octopress Is Pretty Great

2013-03-12T16:48:00.002-07:00

Jekyll Overview

I have simply fallen in love with Octopress. I think I mentioned previously that Octopress is an auto-configuration tool wrapped around Jekyll. And Jekyll is the technology behind Github pages. As such, if you use Jekyll for your blogging platform, you can preview your blog locally, push it to Github, and let Github do all the page serving.

This isn't actually as much work for the Github servers as you may think. Jekyll is a static blogging system with no CMS support. A static blog needs to be compiled whenever content is modified, but once compiled, the only thing the server needs to do is serve the pages. Dynamic blogging systems such as Wordpress and Joomla are programs that run on the server and work interactively with the web server to produce each page.

Configuration

Octopress's biggest addition to Jekyll is that it automagically configures Jekyll for you. I have found that configuring Jekyll is a bit of a pain. Jekyll decided to trade easy configuration for flexibility. This is a common decision in applications and one I can't really argue with.

But Octopress packages a mostly configured Jekyll installation in a neat little package. The introductory paragraph on Octopress.org states:

Octopress is a framework designed by Brandon Mathis for Jekyll, the blog aware static site generator powering Github Pages. To start blogging with Jekyll, you have to write your own HTML templates, CSS, Javascripts and set up your configuration. But with Octopress All of that is already taken care of. Simply clone or fork Octopress, install dependencies and the theme, and you're set.

While that may sound a bit intimidating to the less-computer literate, the Octopress site carefully steps you through the process. I've installed and used Wordpress, Joomla, Drupal, Nanoblogger, and many other blogging systems. I can say that Octopress is, by far, the easiest.

The above quote states three steps for installing Octopress but there are actually a few more steps:

Install Git
Clone or fork Octopress
Install dependencies
Install theme
Configure your blog
Configure deployment

Git

If you don't have Git installed, you need to install it. This was a trivial task for most users but, thanks to the clever folk at Github, has become idiot-proof. Github created a Mac OS X and Windows application that allows easy use of Github. These make installing and using Git user-friendly for those who prefer to avoid CLIs.

Clone Or Fork Octopress

Cloning Octopress, simpler than forking it, is a single command line

git clone git://github.com/imathis/octopress.git octopress

It takes only a few mouse clicks if you are using one of Github's applications to do the clone.

Install Dependencies

Installing the dependencies is, yet again, a few simple CLI commands described in an easy-to-read style on Octopress's setup page.

Install The Theme

Installing the theme is a single CLI command:

rake install

There are additional theme installation options but you can simply ignore them as the defaults work well. In a subsequent blog post I'll discuss how I have made a minor alteration to the Octopress installation that allows me to entirely change my theme as often as I like without losing any data. Simple theme changes (eg: page colors) are easy to make at anytime.

Configure Your Blog

Configuring your blog is perhaps the most complicated step in the entire installation process. You'll need to open a text editor such as Textedit on a Mac or Notepad on Windows.

The configuration file is written in a format called YAML. That sounds a lot scarier than it actually is. Octopress's configuration page states that you just need to edit the existing _config.yml, and modify the following lines:

url:                # For rewriting urls for RSS, etc
title:              # Used in the header and title tags
subtitle:           # A description used in the header
author:             # Your name, for RSS, Copyright, Metadata
simple_search:      # Search engine for simple site search
description:        # A default meta description for your site
subscribe_rss:      # Url for your blog's feed, defauts to /atom.xml
subscribe_email:    # Url to subscribe by email (service required)
email:              # Email address for the RSS feed if you want it.

Several of the configuration options have default values or can be left empty.

This configuration step in Octopress differs from other blogging systems only in that you have to use Textedit or Notepad to enter the data. Many other systems allow you to enter the data into a GUI or a web page.

Configuring Deployment

Configuring your deployment, the final setup action, involves telling Octopress what computer is going to host your blog. While you can run your blog right on your desktop or laptop, it will only be visible by other people in your home or office. This is helpful if you want to preview your blog before "publishing" it so the rest of the world can see it but when you want to publish your blog you will need to have access to a hosting computer (AKA: a server) that is on the internet 24x7.

Getting access to a server can be less than straightforward and almost always costs money. But this is rather simple with Octopress because:

Octopress is a static blog. This is compared to a dynamic blog that requires programs to interact with the server. Once Octopress has done its job, you just need to put the blog into the right location on your server and everything will magically work.
Octopress, the blogging system that tries to make configuration easy, has carefully documented, in easy-to-read text, three different ways to host your blog on a server. Two of them are absolutely free: Github itself (the one I use) and Heroku. The third strategy using rsync is a bit more complex and typically costs some money to use. That being said, the documentation guides you to a hosting company that charges only $5 a month for server access. Unless you know that you need a server for something besides hosting a blog, I strongly recommend going with one of the other options.

As I said, I am using Github to host my blog. It takes exactly one line to configure your blog to use Github. I haven't used the other deployment mechanisms but Heroku seems to be of equal complexity while rsync definitely requires more work and a greater knowledge of computers.

Configuration Summary

While there is a lot of descriptive text above, the entire configuration process took me a couple of hours the first time and about 20 minutes the second time. Unless you are like me and enjoy trying stupid things, you'll only need to install it once :-) I would imagine that a non-technical users could easily install Octopress in half a day.

The only exception that I can think of would be if you're on a Windows computer and have lots of difficulties installing the dependencies. Mac OSX and Linux come with most of the dependencies pre-installed. I don't think that Windows does.

It's Blogging Time!

Before you can actually start to blog, you need to choose an editor and a markup language. While you can change these at anytime, you need one of each to blog. I'll discuss this in a subsequent blog posting where I discuss my personal Octopress configuration.

Now that everything is configured, it's time to blog. While this involves using the command line, it is a trivial usage. When you want to create a new post you execute the command:

rake 'new_post[Name Of My Post]'

The reason why that is the magical incantation to create a new post is unimportant. All you need to do is remember (or be able to lookup) that command.

When the rake command finishes, it will tell you the name of the file for your new posting. Just open the file and start typing. If you want to see a preview of your blog before publishing it to the server, run the command

rake preview

Then you can open your web browser to http://localhost:4000 to see your blog. I strongly recommend that you copy-and-paste that URL and then bookmark it.

Once you are ready to publish your blog to the world, execute the command

rake gen_deploy

And voila! You have updated your blog.

Octopress Summary

Cons

Configuration

Configuring Octopress is definitely more complex than configuring a blog on Blogger.com or a similar service. If you are looking for the easiest solution, no-frills solution, go with one of those.

Single Computer Publishing

While I haven't seen it explicitly stated in any documentation for Octopress or Jekyll, they really are intended to publish new posts from a single computer. You can author the posts on any computer or network service. The first option that jumps into my head is Google Docs. But to publish (ie: to deploy) your site from more than one computer will definitely require extra configuration, computer skills, and tinkering. If you absolutely must publish from multiple computers, the online blogging systems may be a better choice.

Pros

Configuration

While configuring a blog on a site like Blogger is definitely easier than configuring Octopress, if Blogger doesn't meet your needs, I think Octopress is easier to manage than all Joomla, Drupal, and all of its competitors – at least the ones I've seen.

Content Possession

When you use a site like Blogger.com, the content you have posted is in the possession of Blogger. They determine how they will store your content, how they will let you export it, modify it, delete it, etc… But with Octopress, you are always in possession of your content. In order to have your blog visible on the web, you must deploy it to a server, but that is just a copy of your content. You are always in possession of the original content.

Content Accessibility

Whether you are using an online blogging service such as Blogger.com or hosting your own blogging application such as Joomla or Drupal, your postings are typically not stored in a convenient manner. Even if the blogging system lets you get access to all of your content, the content is frequently meaningless or difficult to use.

But with Octopress, the content for each posting is in a file. One file for each posting. While I've not discussed choosing an editor or a markup language in this posting, it is your choice of editor and markup language that determine what you can do with your content.

For example, I was blogging with Nanoblogger for a few months. As it required continual tinkering, I moved to Octopress. Like Octopress, Nanoblogger lets you choose your editor and markup language. Because both blogging systems give you accessibility to your content, I was able to convert about 20 Nanoblogger postings into Octopress postings in about 20 minutes. I imagine it would have taken less computer-savvy users as long as 2-3 minutes a posting.

The one-file-one-posting principal of Octopress, and Nanoblogger, make life so much simpler.

And It Looks Good Too!

Even after customizing them, I find that Wordpress blogs tend to be drap and all look like this and that Blogger.com blogs require lots of tinkering but still tend to look like this. On the other hand, Octopress blogs tend to look like the Octopress homepage or my blog.

IMO, Octopress looks great out of the box. I am partial to using dark themes where black, white, and shades of gray are the main colors. It took me about 10 minutes to change the default colors for Octopress to use my preferred color pallette.

It Will Grow With You

I think that Octopress's greatest features are the above mentioned ease-of-configuration and the fact that, while it is hidden from you, it is built on a very strong blogging engine, Jekyll. So, if you wake up one day and say "Gee. I wish my blog could do XYZ.", the odds are that it can. Just google "Jekyll blogs XYZ" and, at least for reasonable XYZs, you'll find that instructions about reconfiguring your blog so it can do XYZ.

There are also many discussion groups, forums, and helpful Jekyll users in the blogosphere that can offer suggestions, provide pointers, or even make a small modification to Octopress/Jekyll so that XYZ is easy to do.

Future Postings About Octopress

This is now my second posting about Octopress. My {{{relative_url(2012/05/14/new-blog-tech, First Posting, first posting)}}} discussed my initial opinions about Octopress. They were largely positive but so too were my {{{relative_url(/blog/2011/12/07/new-blog-tech, Nanoblogger Posting, initial impressions)}}} of Nanoblogger.

At the time of writing this blog posting I'm certain that I'm going to love Octopress. (Though, if I change my mind, I am in possession of my own content so it is easy to switch to another platform.)

My next posting about Octopress will explain the minor modifications I have made to Octopress. I made these modifications to accomplish two goals:

Allow me to easily use Org Mode as my markup language and Emacs as my editor. While I simply love this setup, I imagine that it will only be of interest to preexisting Emacs users.
Keep my content totally separate from the Octopress's content. By this I mean that if, you were to burn my computer as I'm typing these very words, I could have Octopress up and running on a new computer with all of my content, including postings, images, etc.., in about 20 minutes. IMO, this an improvement on Octopress's configuration and should probably be changed in Octopress.

My First Ruby Project

2012-06-01T16:55:00.000-07:00

The Project

Typically, even my pet projects have a specific goal in mind. I don't write a spec for my personal projects, there's just a, maybe slightly fuzzy, goal. Generally I want something that will be useful. I'll work on a project until:

The project does everything I hoped for. Maybe even more!
I run out of spare time to work on it. This doesn't happen as often as you may imagine. I really enjoy coding. So, after a tiring day of coding at the office, I come home to relax and work on one of my pet projects. If it was any different, then I have no right to call my blog My Geekdom.
I decide that I did a poor job picking a pet project as they're supposed to be fun and this projects isn't. So long!
I get stuck such that proceeding will require more effort than I feel it's worth. That said, hopefully I learned something working on the project.

But, unlike my typical pet projects, my goal with my new project is rather amorphous. I want to analyze some data. At time of writing this, I have just shy of 5K records totally 181MB of data.

Leaving the details for a later posting, each record is a snapshot of the state of my Mac OS X computer. I've been collecting, and am continuing to collect, samples every five minutes since the beginning of March. At least every five minutes that my computer isn't sleeping or hanging.

I decided to collect the samples because I was getting downright frustrated with the performance of Mozilla's Firefox browser. For reasons that are off-topic for this posting, I feel a strong interest in making sure that Firefox continues to be successful.

For years I thought Firefox was the bomb. Starting about 9-12 months ago I started to notice that Firefox was routinely using all the resources on my computer. This concerned me because Google's Chrome browser was starting to have enough functionality to be a primary browser.

Being that I wanted Firefox to stay competitive, I decided to try to stay with it. As Mozilla changed their release model, I started to use Aurora and Firefox Beta and provide lots of feedback. The newer versions also seemed to consume a more reasonable amount of my computer's resources. Being that the Firefox team was working hard at fixing performance problems, especially memory leaks, I stayed with Firefox. (See Reference: Articles About Firefox's Memory Leaks And Repair Efforts for information about FF's memory leaks.)

Around the beginning of this calendar year I started to think that Firefox wasn't getting faster and may have been getting slower. That said, I can be a pretty heavy user of my computer. Running FF with a couple of dozen add-ons is typical. Also, my experience with performance tuning has taught me that I don't know enough to make accurate guesses. Guessing can be used to lead your performance tuning investigations, but you always need hard data.

After a bit of thought, I decided to collect some data and see if I can convince myself that Firefox is or is not the problem. There are, of course, simpler solutions to tracking a performance problem than sampling your entire process every five minutes for months. But this is a pet project. It wouldn't be any fun if I just switched from Firefox to Chrome to see if I notice a difference.

Now that I have accumulated a few months of the data, I need to try to analyze the data and see what, if any, conclusions I can make from it. I have some ideas about where to start my data analysis and will just need to follow the path they outline. In short, I'm prepared to MacGyver the analysis as I go along.

My new pet project is the exploration of these data samples.

The Programming Language

Forensic performance analysis is never simple. For this project I anticipate that the difficulties will be in determining what types of data analyses are required. I'm hoping the actual coding will be relatively simple as there is no need for network interaction, provide real time responses, or even have a GUI. As long as I don't need to do very complex statistical analyses, the choice programming language won't likely matter to much.

So I have chosen to implement this project in Ruby. I'm certain I could implement it faster in any of a few dozen other programming languages but I chose Ruby for two reasons:

This is a personal project. One of my main goals is to have fun. Being that I don't know Ruby, learning it will be more fun. Especially being that I haven't learned a new language in a year or so.
Ruby is a new-ish language, growing in popularity, likely to be around for awhile, so it seems like something I should learn or at least be familiar with.

Ruby isn't wholly unfamiliar to me. I've made a few small (very small) changes to the org-ruby gem. Based on that experience, I think that the syntax of Ruby is kind of strange but the functionality it provides seems comparable to other languages such as C and Java. I just have to learn the new syntax. Unlike C and Java, Ruby has dynamic typing. But languages that I'm fluent in, such as Lisp, Python, Bash, and AWK have dynamic typing. Doesn't seem a problem.

But the structure of a Ruby application, at least the structure of the org-ruby gem, feels different to me than the structure of programs I've seen in other languages. Sure Ruby has classes and modules and functions and inheritance as many other languages do. But the design of the org-ruby gem feels a bit unusual to me.

I think that part of the atypicality of the design of the org-ruby gem is that it is a parser that is written in a manner that is not familiar to me. I have a strong background in designing and implementing programming languages, their associated runtimes, IDE's, and such. In my world, there are only a few ways to implement a parser and org-ruby doesn't use any of them.

That said, I don't think the feeling of unfamiliarity stems from org-ruby's parser not being a recursive descent parser or implemented with a parser generator from a formal grammar. I've seem plenty of really terrible parser implementations that left queasy but didn't feel unfamiliar.

Not only do I not think that org-ruby gem is terrible, I think it has a certain elegance to its design.

Having eliminated other possibilities, I'm left to think that the feeling of unfamiliarity must stem from Ruby itself.

For example, C and Pascal have a similar feel when you use them. So too do object oriented languages such as Java and C++. While not OO, C and Pascal are somewhat similar to Java and C++. But a Scheme application doesn't feel anything like a C application (at least not if you understand how to program in Scheme). While I don't have a word for it, I think that Ruby leads to a different way of writing applications than languages that I'm familiar with..

While I'm not sure what that difference is, I hope to figure it out as I move through this project. In the end, I don't want to have a great "Java program" that happens to have been written using Ruby syntax. I want to have a great Ruby program – period.

The Project's Current Progress

"Not much" seems to sum up my current project. While I've been collecting data for months and easing myself into Ruby by working on the org-ruby gem, I just started coding for this project a few hours ago. All I've done so far is write a tiny class to hold a line of data output by the ps(1) program:

 1:  class PSDataLine
 2:    attr_accessor :pid
 3:    attr_accessor :stat
 4:    attr_accessor :time
 5:    attr_accessor :elapsed_time
 6:    attr_accessor :virtual_size
 7:    attr_accessor :rss
 8:    attr_accessor :percent_cpu
 9:    attr_accessor :percent_mem
10:    attr_accessor :command
11:  
12:    # Create the object
13:    def initialize(line_string)
14:      line = line_string.split(' ')
15:      @pid = line[0]
16:      @stat = line[1]
17:      @time = line[2]
18:      @elapsed_time = line[3]
19:      @virtual_size = line[4]
20:      @rss = line[5]
21:      @percent_cpu = line[6]
22:      @percent_mem = line[7]
23:      @command = line[8]
24:    end
25:  end

I know it isn't much but I still learned writing it. For example, when I was tinkering with the org-ruby gem, I just copied and pasted code, looked up some details in the Ruby library, etc… Being that I was modifying existing code, I never learned even the basic syntax.

For example, in order to write my PSDataLine class, I needed to learn how Ruby uses syntax to determine if an identifier is a local variable, an instance variable, a global variable, or a constant. (My thanks to the Rubyist for the good reference.)

I'm pretty busy this week so I don't think I'll make much progress on it. I was only able to get this work done by staying up till midnight on a Friday night and squeezing in time for a proofreading today.

Process Data

While I've only begun writing my program and haven't reached any conclusions about anything yet, I do have lots of data. If you have an interest in seeing what my computer's been up to you can view the process data, read about my sampling mechanism, and copy the data from my process-data Github repository.

Reference: Articles About Firefox's Memory Leaks And Repair Efforts

A blog posting posting by Mozillian Nicholas Nethercote. A second posting of Nicholas about weeks 49 & 50 of the Mozilla effort to reduce the memory footprint.
A different engineer's blog posting discussing the hunt for memory leaks in Firefox.
Another posting from Nicholas regarding the great [[http://bit.ly/LsNjF0][performance improvements in Firefox V7. - A September 26, 2011 article in CRN discussing the lack of progress Mozilla has been making. Even if you ignore any opinions in the article, it has a lot of facts and links.
A Firefox Forum posting expressing user frustration.

Org-Mode + t Makes Tweeting Easy

2012-05-27T16:49:00.000-07:00

Update 2012-06-04 Mon 00:57

You can download the code from its Github repository.

Motivation

After yesterday's ranting about Octopress, it occurred to me that while I now own the content of my blog postings, I don't own the content of my tweets. I'm sure that there's some way to convince Twitter to give me my tweets, but why should I rely on them to store my tweets.

The obvious answer is that my tweets should be created on my computer and then sent to Twitter.

It should be noted that I'm not interested in implementing a full Twitter client. There are more than enough of those. I'm interested in having my tweets, or at least the bulk of my tweets, in a simple and easily accessible format.

The Answer Must Be Cheap

While I want to author and store my tweets locally, it is not a high priority task for me. Whatever solution I come up with must add little or no overhead to my tweeting workflow, must be simple to implement, and cannot require ongoing tinkering to keep working.

Being that my relationship with Org Mode is rather simple: The more I learn about it the more of my life I want to be based on Org Mode. Org Mode just seems to make everything easier. So I decided to use Org Mode for the author, publication, and storage of my tweets. The system I'm describing has these properties:

It took me just a few hours to design, implement, test, and begin using it for tweeting.
Besides not adding overhead to my tweeting, it has made tweeting dramatically simpler.
The entire implementation is a single file of elisp that is 120 lines long. 50 of those lines are standard Emacs header comments that are automatically created. The implementation, along with its documentation, is only 70 lines long. While a small code base isn't one of my goals, not needing ongoing tinkering is. It is hard to imagine that 70 lines of code will require too much tinkering.

Advantages Of Org Mode

Before continuing, you should know a few facts about Org Mode:

At its core, Org Mode is an outliner on steroids. This works great for me as I tend to organize my thoughts as trees. I'm obvious not alone as the outliner page on Wikipedia has a list of many outliner applications along with references to many outliners that are designed to implement specific functionality.
Org Mode augments its outlining functionality with a task management system (details).
Org Mode has a template based data capturing system that allows for the easy capturing of data into a consistent format.
Org Mode is implemented in GNU Emacs. Besides the fact that I spend nearly all of my non-browser computer time in Emacs, Emacs has a rich extension mechanism

The Storage Structure Of Tweets

Each tweet would be an outline entry whose title would be a small description of the tweet plus some other information to be discussed later. The body for each outline entry would be the tweet. In Org Mode this would look like:

* My First Tweet
  This is my first tweet using Org Mode.

While this works fine for one tweet, after 50 or 100 tweets, things are going to get pretty messy. But, as Org Mode is an outliner, it supports the nesting of outline entries via both direct data entry as well as through Org Mode's capturing mechanism. You just need to provide Org Mode with a string that describes the data you want to capture and the outline format for that data.

While, for the moment, sparing you the ugliness of the capture configuration string, I'll tell you that Org Mode supports a date tree outline format.

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday
******* My First Tweet
        This is my first tweet using Org Mode!
***** 2012-05-27 Sunday
******* My Second Tweet
        This is now my 2nd tweet using Org Mode!

Being an outliner, Org Mode supports the folding of the outline. That is, you can collapse the outline to hide parts of it that you are not immediately interested in. For example, let's say that, being today is Monday, I don't want to be bothered looking at my tweets from Saturday Or Sunday. No problem. When you fold the outline headings for Saturday and Sunday, the list of tweets will look like this:

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday ...
***** 2012-05-27 Sunday ...

The "…" is Org Mode's way to let you know that this entry has been folded.

Keep in mind that I get all of this storage mechanism from Org Mode for free. All I needed to do was create a single configuration string for it.

Capturing Tweets

A simple yet highly functional storage mechanism is fine but what I'm looking for is to reduce my tweeting overhead. This means that the workflow is what I care about.

The first part of the workflow is the "capturing" of my tweets. Org Mode's capturing mechanism is rather rich. It allows me to capture data (ie: to tweet) from Emacs with a couple of keystrokes. It also allows me to capture data, including an URL and highlighted text from my browser by clicking on a bookmarklet.

When I initiate a tweet capture, I'm initially prompted for a "title". That is the descriptive text for the tweet. In the above example, "My First Tweet" is a title. After entering my title, I'm given an Emacs buffer, basically a new Emacs window, in which to type my tweet.

When I'm done typing my tweet, a couple of keystrokes tells Org Mode that it should add the tweet to my tweet file and save the file.

Tweeting

Now that I've captured my tweet, I need to actually tweet it. This is where some of the wonders of Emacs come into play. Emacs has lots of hooks that allow you to insert code into the workflow. Org Mode adds many hooks of its own.

Without boring you with the exact details, by using the appropriate hooks I have set up Emacs to take special actions whenever a file is saved that ends in "-update.org". I used the file naming convention of "-update" instead of "-tweet" in case I decide to add other types of status messages at a later date.

Being as I was going for simple, I decided that Emacs would not talk directly to Twitter. Instead, I have Emacs use the wonderfully intuitive, functional, and efficient t application. For my purposes, t's CLI is perfect. Once you've signed into Twitter with t, it will tweet for you with the simple command line of:

t update This is my tweet text.

That seems about as simple as it can get.

Tracking Tweet Status

Being that I'm storing more than one tweet in a file, it is important that I know which tweets have been tweeted and which ones haven't. This is another place where I get to leverage the wonders of Org Mode.

Beyond outlining, Org Mode supports a simple yet highly functional and customizable task management system. The customizability allows you to define task states well beyond the typical "open" and "closed" states of many other task managers.

I used the task management to teach Org Mode about status update tasks. A status update task can be in one of two states: POST and DONE. When it is in the POST state, it needs to be posted. The DONE state means it has already been posted. Org Mode's slogan is

Org: Your Life in Plain Text

and they mean it.

To indicate a tweet's, ie: a task's, state, you simply add the state to the outline header line. For example, using the example from above, if I had already posted my first tweet but not my second, the outline would look like this:

* 2012
*** 2012-05 May
***** 2012-05-26 Saturday
******* =DONE= My First Tweet
        This is my first tweet using Org Mode!
***** 2012-05-27 Sunday
******* =POST= My Second Tweet
        This is now my 2nd tweet using Org Mode!

When I use Org Mode's capturing mechanism to add my new tweet to the file, it automatically adds the state of POST to each new tweet. As mentioned above, when the tweet file is saved it executes my code. While I have 70 lines of code, only two functions consisting of a total of 45 lines is needed to post my tweets. The remaining 35 lines are configuration that, via Emacs and Org Mode's hooks, makes sure that my code is called whenever the tweet file is saved.

The first function, all 9 lines of it, simply asks Org Mode to loop through all tweets (ie: outline headings) in the POST state and call my second function on each of those tweets. The second function calls the CLI t to execute the tweet. This 28 line function that actually does the tweeting is really rather simple. Only 10 lines of it are related to actually tweeting. The remainder of the function is error handling in case, for some reason, my tweet fails.

After successfully tweeting a status update, I tell Org Mode to change the tweet's state from POST to DONE. That's all there is to it.

Some Nice Freebies

Besides the data capturing, the ease of calling an external program, the automatic outline maintenance, the automatic state tracking (ie: POST and DONE), and the automatic execution of my tweeting code whenever the tweet file is saved, the Emacs and Org Mode's richness provide other nice features:

Twitter's 140 character limit per tweet is easily eyeballed while you are typing in Emacs as the editor tells you how long a line is. I have, of course, implemented a character count error check before tweeting, but it is nice to catch the problem when you are typing.
Automated highlighting of tweets.
A common problem with the 140 character limit on tweets is that URLs can be very long. Emacs helps here with its Bitly library (courtesy of me :-). This shortens URLs in Emacs, using Bitly, with two keystrokes. (Bitly happens to be my current URL shortener of choice.)
Org Mode has the ability to automatically record a variety of times associated with each task. I have set up Org Mode to record the time the tweet was entered by me as well as the time that it actually got tweeted.
Org Mode allows you to tag (aka: "label" or "categorize") each outline item. I have leveraged this in anticipation of extending the tweeting system to include status updates to other services than Twitter. Also, in the future I may want to be able to manage multiple Twitter accounts. I'm not sure that I'll ever implement that functionality, but thought I'd throw it in as preparing for it is essentially free. I currently label each status update with two labels. The first label describes the type of status update. In this case it is "TWEET" to designate that I want to update Twitter. The second label is "NS" to indicate that I want to update the status of my @neil_smithline Twitter account.
Emacs is my tweet editor. For those of you who don't know Emacs as well as those of you that know Emacs and hate it, I'm sure you wouldn't want to edit tweets in Emacs. But I have been using Emacs for 25+ years and still think it is pretty slick. Besides the functionality discussed above, Emacs gives me spell checking/correction, the ability to easily change the case of something I've mistyped, functionality based on English grammar (eg: the ability to move, delete, or even transpose words or sentences), etc…

A Sneak Peek

After all my blathering, I thought that a screenshot of my tweets would be nice. This screenshot shows tweets for the past three days. I have folded the tweets from Saturday so that they are easy to ignore. For the sake of this picture, I have unfolded my tweets from Sunday so that it shows what a posted tweet looks like. Finally, I have a tweet for today that is queued up to announce this blog posting.

As you look at this screenshot, remember that almost all of this is automatically generated. For example, for the final tweet I had to enter the title "Tweeting From Emacs" (this can actually be blank but I think the title makes it more useful) and the actual tweet. Everything else, including the Bitly URL shortening, comes for free.

For Technorati: PHTDYXEZKM3Q

Automatefd Gvoice And Gmail To Gcal

2012-03-31T16:54:00.000-07:00

I seem able to hold incredibly large amounts of technical details in my head. In contrast to this, I'm excited when I can remember what year my kids were born in. From my viewpoint, this is just the way I have always been. From the viewpoint of people in my personal life, it can be frustrating.

Being the technophile that I am, I've incorporated many electronic assistive technologies into my life. In 1989 I got my first PDA. It was about the size and weight of a half-brick. But I could carry it wherever I went and it would ring to remind me to do things. What a deal!

My newest electronic assistive technology is a rather complete phone and SMS management system. Its features are:

I have a permanent record of every incoming or outgoing call or SMS.
My mobile phone rings whenever I get an incoming call or SMS.
My home phone rings whenever I get a call.
My mobile rings and gives me a notification whenever I get a call or SMS.
My computer rings and gives me a pop-up whenever I get a call or SMS.

This blog entry is unfinished. It will be updated or followed up with a second blog entry shortly.

Blog In A Box (actually, a repo)

2012-02-15T16:45:00.000-08:00

As I have been developing my blog's technology, I have been surprised at the lack of a pre-packaged blogging solution for Emacs' Org Mode. Org Mode is simply a wonderful tool for organizing thoughts, ideas, text, links, etc… While Org Mode has a rather difficult learning curve, its motto, Your life in plain text., seems to be accurate.

Spurred by ideas from 0xDECAFBAD.com and code from NanoBlogger (NB), I've gone about creating an Org Mode-based blogging system. And, if you are reading this, it is actually working. While NB and Org Mode do all the heavy lifting, I've simply added some glue to hook them together. What I have is far from a perfect solution, but it seems to work pretty well.

There is a bit of a UI problem in that the Org Mode editing has a general look-and-feel like that seems like a typical Org Mode editor with a NanoBlogger posting jammed in. That's not surprising as, to save development time, I've basically just jammed a NanoBlogger posting into an Org Mode buffer.

The other problem that I have with the system is that there is no way to automatically detect the difference between postings that have been published and those that have not published (ie: drafts). Unless you take extra care when publishing your blog, all postings, whether or not they are drafts, will be posted.

Once I've fixed these two problems I will have a simple but reasonably functional blog system. By "simple" I mean that it is based off of NanoBlogger, a minimal blogging platform. That being said, the goals of NB are simple goals compared to most blogging systems. If you want a richer blogging system then NB isn't for you.

The glue that I've written to connect NB with Org Mode is a shell script and some elisp code. Combined they are only a few hundred lines of extensively-commented code. The entire NanoBlogger installation, living up to its name, is only 844K.

Being that NB is so very compact, my shell scripts and elisp code are small, everything is in plain text files, and that Github provides simple and free hosting of both the blogging system and the blog, I'm going to merge my entire blogging system, minus Emacs, with my blog and put it into a single Github repository.

Once that is done, anyone who wants to try my blogging technology should only have to do a few simple steps:
# Fork my blog to their own Github repository.
# Clone their copy of my blog to their computer.
# Delete my blog postings (I'm assuming people will want to create their own postings and not republish mine).
# Run emacs and load the single elisp glue file I've created.
# Start blogging.

I believe that it will take less than 10 minutes to get a blog up-and-running. That is longer than it takes to get a hosted blog on Blogger but much, much faster than most self-hosted mechanisms such as MovableType. The only prerequisites for this blogging system are

Bash and the other common shell script utilities that are required by NanoBlogger such as sed and awk.
A reasonably modern GNU Emacs. It comes with Org Mode pre-installed.
The git command-line program.

Being that this blogging system is only aimed at Emacs users, it is certain that they'll have Emacs already installed on their machine. It is also very likely that they'll have bash and the shell script utilities. Maybe they'll have git or maybe they'll need to install it. It is a simple installation.

There is no need to install a local web server for staging. Provided you have versions from this decade, there is no need to ensure you have the right version of any of this software. Managing versions of installed software can be a major pain for many systems (eg: Python based-apps). You also have full access to your blog's data without having to export anything (everything is already in plain text files) or provide a credit card (Github is free).

My New Blogging Technology

2012-02-14T16:33:00.000-08:00

As I've discussed in previous postings about this blog's technology, this is a static blog written with NanoBlogger. NanoBlogger is a CLI driven blogging system. Alas, I have found the NanoBlogger manual to be difficult to follow and the syntax of the CLI to be non-intuitive. (See the footnote here regarding the existing CLI.) On the upside, NanoBlogger is a CLI system which makes it easy to modify.

I started out with a few small helper scripts, eventually merging them into a single helper script. As I was using that script and learning more about NanoBlogger, I found myself writing smaller helper scripts for my big helper script. Eventually, I was never calling my original, large helper script directly.

At this point it became clear that some refactoring was needed. So I refactored. I came up with a single shell script call nbc.sh that does all the work and a series of shell scripts that simply call nbc.sh.

The key idea behind nbc.sh, or simply nbc, is that the scripts that call it are trivial. By "trivial" I mean that each script is only a few lines long, all the side scripts have identical content, and, if not for the semantics of Git, would simply be multiple symbolic links to the same program.

As you can see below, the code for the side scripts simply stores the command name, $0, of the side script that was called and then calls the nbc.sh script that is in the same directory as the side script.

#!/bin/sh

# Code to load include file.
ORIG_ARG_0="$0"
NBDIR=${0%/*}
. ${NBDIR}/nbc.sh

This means that as I continue to add functionality to nbc, all I need to do is copy one of the side scripts to the filename of the new command I added.

While I won't discuss the exact naming scheme used for these side scripts in detail in this posting, I will say that I have tried to systematic in how the scripts are named. Once you learn a few abbreviations (eg: "e" for "Entry" or "Entries", "u" for "Update") and the rules for combining those abbreviations (standard RPN), you will know the names of all the commands, what each command does, and the type and order of arguments each command takes.

I still need to write richer documentation for nbc and its side scripts, but I can finish this posting off with some concrete examples.

NB supports an "update" command that essentially tells NB to rebuild the blog from scratch. The syntax of this command is:

nb update

My nbc version of this command is:

nbu

All of the commands for nbc start with the letters "nb". The letter "U" is the abbreviation for "Update". So "nbu" tells nb to run the update command. As I said, the code for nbu and the other side scripts are trivial and given above. As far as examples go, nbu is not very exciting. But the nbc's CLI for creating tags is a bit more exciting.

NB's syntax for creating tags, as documented in its manual, is:

nb --title <tag-name> add tag

Gosh! That's both tough to remember and tough to type. In nbc, the command for adding a tag is nbga. All commands start with "nb". After that, the "g" means that we are talking about "taGs". The "a" stands for "Add". The syntax for the command is

nbga <tag list>

Besides being much easier to type and remember, it allows a tag list as compared to a single tag name. That is, you can create multiple tags in a single command rather than needing to call nb once for each new tag.

In NBC, a tag list is a comma-separated list of one or more tag names. So "tag1" is a tag list and "tag1,tag2,tag3" is also a tag list. If you prefer you can space-separate the tag list but you need to ensure that you pass the entire tag list as the first argument to nbga.

As the operations become more complex, NBC's simplifications become greater and more obvious. My next posting about NBC will discuss some of these simplifications.

If you wish to use NBC, look at the source code, or tell me why NBC sucks, you can get the scripts at https://github.com/Neil-Smithline/nbc.

This Blog Is Now Hosted For Free (and it's slashdot proof too)

2012-02-03T16:42:00.000-08:00

While there are many free or nearly free blog hosting sites available, most of them come with a long list of restrictions. Either they restrict you to using their blogging software, throw ads on your pages, give you the "unlimited bandwidth" plan that they'll cut off if your site gets too much traffic, limit you to only the ugly themes, etc.

But www.neilsmithline.com is now hosted on a site that charges me nothing, doesn't put ads on my pages, couldn't care less about the theme I use, and, should I ever be lucky enough, survive slashdotting. The only restriction is that my blog has to be static. Being that I already had a static blog (see previous posting), this was no problem for me at all. I've used lots of dynamic blogs (Drupal, WordPress, Joomla, Blogger, Movable Type, etc.) and just found that "dynamic" is a short way of saying that the blogging system is going to do what it wants and not what I want.

Along with the free hosting I get automatic version control, well-managed collaboration (not that I have any collaborators), data redundancy, eternal archiving (I probably won't need it that long but it's nice to know that nothing will get lost), and the ability to get a copy of my data onto my local machine whenever I want it.

My choice of blogging technology was largely influenced by these two blog postings, 1 and 2, and email conversations I had with their author, Les Orchard. While Les went with Jekyll, I preferred the simpler NanoBlogger. Les uses Github.com for versioning his blog but hosts his blog on his own site as he wants full control of what is published. I simply couldn't agree with him more.

When I moved to my new blogging framework, I quickly got my blog up and running using NanoBlogger for site generation and sftp to publish it to my VPS. Today I investigated replacing sftp with Github and its notification framework. This would essentially mirror Les's blog system except that I am running NanoBlogger locally while he runs Jekyll.

As I was investigating using Github for transportation and archiving of my blog, I decided to take another look at Github Pages. And then, like a lightning bolt, it occurred to me. When I publish my blog by pushing it to Github, if the repository has the magic name of neil-smithline.github.com, then Github would publish my blog at http://neil-smithline.github.com/ automatically. There simply was no benefit to serve my blog from my VPS as Github would do it far better than I could on my VPS.

The only problem I foresee is that I run more than just my blog on www.neilsmithline.com. But the other applications I run are only for my private use so I can just map them to another sub-domain such as whatever.neilsmithline.com. For example, I've been playing with a cloud-based bookmarking system, Asaph, that allows you to run your own cloud bookmark server. But I can access it at and, if you have an interest, can view it at: http://www2.neilsmithline.com/asaph/.

All-in-all, it's like a nerd's dream come true :-D

UPDATED: Sat Feb 11 13:12:02 EST 2012

There seems to be another problem with hosting my blog on Github. While practically all of Github requires SSL, there appears to be no way to get a Github hosted website to use SSL. Thinking about it, the task seems inherently difficult. Github makes all of its sources file public but you must protect your site's SSL private key. Currently, all site information is stored in the Github repository. A private SSL key could only be added if Github updated their configuration to allow the key to be stored out-of-band (ie: not in the repository).

Nastiest Hack I've Seen In Awhile

2012-02-01T16:52:00.000-08:00

I was reading about SL4A, also known as Android Scripting Languages. They are a set of quick-and-dirty ports of programming languages to the Android that interact with the Android OS. I just stumbled upon this "issue page" (it's more like a forum than a bug report).

The developer is trying to run something on his Android, which is basically Linux, that requires superuser permissions. As the responses started to build upon each other, one user suggested starting his shell script with the following lines (the actual posting was in Android's flavor of Python but I've converted it to bash(1) to make it more accessible):

#!/bin/sh
if [ "$USER" != "root" ]; then
    exec sudo $SHELL -c "`cat /proc/$$/cmdline | tr '\000' ' '`" root
fi

I simply couldn't help being entertained at how revolting but equally brilliant that code is. But, I started thinking (always something dangerous) about how much I hate having to deal with stupid commands. For example, this is the output of running apt-get(8) without proper permission:

$ apt-get install foobar
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Being as I often write shell scripts with calls to sudo(8) in the script, I decided that imitation is the best response to being revolted (I know my mom said something like that :-).

So I expanded upon the existing code, making it a full shell script utility. You can look the code or grab a copy at my GitHub page in the rerunasroot repository. I'll be using it in the future.

Seems It Really Works

2011-12-10T16:38:00.000-08:00

Well, it appears this actually does work. I seem to be having a few
issues integrating with NanoBlogger but I'm sure I'll work those out.
The problems that I've seen are:

It seems that adding or even updating a new entry is as costly
as rebuilding the entire blog from scratch. Right now, that's
OK. But what am I going to do when I have 10, or 100 postings?
I hope NanoBlogger can handle this.
NB seems to assume that its input files are hand-written in
some legacy text editor called Vim or something :-). While my
strategy of using GNU Emacs' Org Mode keeps the flavor of NB in
that it is written in a simple markup language, it doesn't seem
to fit the NB model perfectly. There is nothing wrong with NB's
markup, it is just that Org Mode's markup can easily be
exported to virtually every other markup on the planet. So I
just need to learn Org Mode markup, write my content in Org
Mode, and then publish that data into whatever format I choose.
Seems a better solution but NB isn't quite prepared for it.
I would still like better integration with Org Mode and NB. The
steps I've used to produce this entry are:
- Ran a little shell script I wrote to create a new NB
  entry. All I do is give the shell script the title of the
  Org Mode file I want to store the new entry in and it
  does its magic, leaving me with an Org Mode skeleton for
  NB.
- Edit the Org Mode file just created.
- Use Org Mode to publish the entry. This just stores it in
  my local NB folder.
- Run nb update to have my changes put into the local
  copy of the blog.
- Run nb preview to view the changes before publishing.
- Run nb publish to update my blog.

So, while they're are still some problems, I have to say I love it.
The advantages of using Org Mode and the simple NB blogging engine on
both my local and production server is way cool.

Org Mode is actually way cool. I've only started using it in the past
year and will likely spend years learning it as it supports so many
features. But, like all well-designed pieces of software, you don't
need to learn it all at once. I'll yabber about Org Mode another day.
In any case, I'm real happy with my blog and the ease with which I can
add postings to it.

New NanoBlogger Command-Line

2011-12-07T16:27:00.000-08:00

My hope is that I can write this entry in GNU Emacs' Org Mode and have it auto-published into this blog. If you are reading this, then I have succeeded.

This is a static blog. That is, the server side is all HTML. It can be served by virtually any web server. This makes it incredibly convenient for testing and deploying. When I am writing blog entries, I'm just typing in Emacs. I then save the entries and use Org Mode plus some personalized tweaks to publish them to a local directory as something close to HTML.

These HTMLish files are then fed into NanoBlogger. NanoBlogger is the magic tool that turns a few HTMLish files into a fully functional website. NanoBlogger accepts HTML for the text af entries but wants some simply formatted metadata added to the file.

The Metadata is needed to help NanoBlogger out because, as its name implies, is a teeny-tiny blog engine. As a comparison, my installation of NanoBlogger is 768K (yes, that's "K"). I also have installations of Drupal at (14M) and Joomla (107M).

The .txt source for each NanoBlogger posting is a file with the body of the entry in HTML embedded but all the cruft that tends to come before and after the body of a web page is simply not there. The easiest way to get an understanding of it is to view the metadata source for this sample posting.

So you're saying "Big deal." or "Who needs yet another file format?" But it is a big deal. What most younglings have never learned is that WYSIWYG is a distraction from actual content production. Content production is not the same as formatting and editing and, at least in my experience, uses very different parts of the brain. So the less formatting you do while producing content, the better your content will be (at least if you have my brain :-).

While it has largely been lost with the popularization of the Wikipedia, initially, Wiki markup was intended to simplify editing but also served to separate the content production from formatting. While NanoBlogger expects these HTMLish files as input, it is designed to generate them from any number of simpler input formats.

These simpler input formats are quite unlike HTML as they are, well they're simpler. The default input format is the rather famous Markdown though many other formats are supported by NanoBlogger or the community. This article seems to do a good job explaining the benefits of simple languages such as Markdown.

It took me but a few hours to create a converter for GNU Emacs' Org Mode to NanoBlogger.

While not nearly as cool as what I've discussed above NanoBlogger has yet another advantage: it produces a static site. On my local machine (17" MacBook Pro), I can view the entire site without running a web server. I simply go to Firefox (actually Firefox Aurora) and visit the URL, specifically, the file URL of my blog. For me, I visit file://Users/neil/www-nb, and then I can see my entire blog through the magic of a static websites, file URLs, and relative URLs.