Fixed In The Next Release

Teacher Retirement System of Texas Takes it on the Chin

louis@fitnr.com (Louis Marascio) — Sun, 15 Mar 2009 10:37:14 -0500

According to the Austin American Statesman, in the past six months the Teacher Retirement System of Texas has lost $34 billion representing a decline in value of about one third. From the article:

The Teacher Retirement System of Texas trust fund lost more than $34 billion in the past six months - a drop in market value of about one-third, according to an actuarial report released Friday.

The system, one of the largest public pension funds in the world, now has only 68 cents for every dollar that it needs to cover promised benefits to its 1.2 million members over the long term; in August, that number was 90.5 cents.

The financial markets fell 40 percent overall during that same time period.

Sure they like everyone else lost money, big surprise. What sort of perplexes me is that in search of a nominal 8% return (as indicated by the article), they were actually taking on enormous risk. Heck, I wouldn’t even be surprised if the TRS investment committee members were proud of themselves for beating the market—after all, they only lost 33% while the overall market dropped 40%.This isn’t unique to TRS, it’s more of a broad misunderstanding of the risks involved when making what are, essentially, “long the economy” investments.

I first heard that term while listening to a very successful investor, Salem Abraham, talk about his fund. Many really great investors have long been overlooked because they took a different approach than blind faith in the ever increasing equity markets. Usually the argument against investors like Abraham would have been based on the fact that they traded futures. Simply because the futures markets allow one to more easily shoot themselves doesn’t mean that talented folks like Abraham should be dismissed. After all, if you now look at their risk adjusted return compared to the S&P, it’s actually better.

If there is one good thing that I would love to see come out of the recent market turmoil, it would be that the equity markets are just as risky, over the long term, as other “alternative” investments like managed futures. Knowing this, investors should seek out diversification between asset classes. And this means more than just an equity/bond portfolio.

As a final note, you should go watch Elizabeth Cheval’s presentation on correlation and why you should be looking for negative correlation in your diversification. If you watch it, it’ll open your eyes. Or, you might simply dismiss it, and wait for the next market implosion.

Onwards and Upwards: Goodbye Cisco, Hello TBD

louis@fitnr.com (Louis Marascio) — Thu, 01 Jan 2009 10:56:58 -0600

The time has come: yesterday, December 31 2008, was my last day at Cisco. In my mind, my tenure at Cisco was about completing what we started eight years ago with Metreos. Cisco purchased Metreos after we had been operating for 5 ½ years as an independent company and for 2 ½ years I worked to help Cisco execute on the vision that building enterprise telephony applications and integrating them with business processes shouldn’t be painful. I’d like to think that we were successful in that venture.

Now that I have closed the final chapter of the Metreos (epic) novel it’s time to write a new one. I’m an entrepreneur and I’m certain a new idea, product, and company are what’s next for me. Stay tuned to see what happens.

That's a Dilly of a Pickle

louis@fitnr.com (Louis Marascio) — Fri, 26 Sep 2008 14:18:39 -0500

We’ve all had that moment. You know the feeling, your stomach falls out from under you, your legs go a bit wobbly, or your gut climbs into your throat. It’s an “Oh Shit” moment.

Lot’s of times this happens under relatively benign circumstances. Maybe during a test cycle you realize you forgot some fundamental feature or missed a major requirement. No big deal: just adapt and overcome. However, there are times when those moments are particularly scary.

Building software, especially software that has to integrate with other systems, should always breed a healthy dose of skepticism when diagnosing root cause. What I mean is, there is always a feeling that the problem might not be your own, but in reality, you must always assume that your software is creating the problem, even if it isn’t convenient. Doing otherwise is trying to operate outside of your own sphere of influence: you have no control over the software you integrate with, therefore it is typically a sisyphean task to start there in trying to resolve a problem.

In the many years that I have been building the Metreos/Cisco product that I work on today I have found bugs in other people’s software a mere fraction of the time when compared to my own. It is almost always my fault. I’ve also seen, over and over again, other engineers (some young, some old, I don’t think it is related to your experience level) jump to the immediate conclusion that “it must be xyz in abc’s product” thinking to themselves “there’s no way it could be us”. This is a horrible way to approach problem resolution and root cause identification.

Those really nasty problems almost always happen under the umbrella of a customer found problem. So, coupled with the fact that you’re already facing an “Oh Shit” moment you also have the added stress of knowing that the customer is expecting updates, information, and a fix as soon as possible. An engineer who worked for Metreos’ first customer, and now a good friend, always had something funny to say in these situations:

That’s a dilly of a pickle you’ve got there.

That always made me laugh, but I think the thing that I appreciated about the relationship we had was that he always afforded us the space to work the issue in a logical, diligent, and direct manner. Getting yourself out of hairy spots is much easier when the customer is not being difficult.

So, needless to say, I’ve faced my fair share of pickles over the years. I’ve built up a pretty standard method of analyzing and diagnosing the root cause of the problem. What surprises me is how people try to debug these complicated systems without such an approach. Much of the time you see engineers tossing darts in the dark, hoping to find a solution. The funny thing is, and I’ve seen this first hand as well, is that they might toss one of those darts and see the problem fixed. Unfortunately, without understanding the root cause they never really know whether the real problem is fixed or whether they’ve simply treated the symptom.

Here’s how I typically approach things:

Ensure the problem has been reported properly and is being tracked according to your process. If the problem isn’t being tracked you’re doing the customer a disservice. Make sure it’s logged, has a tracking number, and is visible on whatever process dashboard you might have.
Confirm your understanding of the problem and formulate a set of plausible causes, if possible. It’s sort of amazing how many times you’re initial understanding of the problem is adjacent to the actual experience the customer is having. Take the time to reconfirm, in painful detail, what’s happening and what’s expected to happen.
Analyze available data. If no data is available, ask for data that might already be available by the customer. For example, your standard set of logs that might be turned on in production. Data is critical. Without the data it’s impossible to create a sound hypothesis or confirm environmental events. Many times, people don’t get the data needed because they have some aversion to asking the customer to do work in helping diagnose the situation. If you need the customer to gather network traces from a specific port on a specific switch, tell them so they can do it and get you what you need.
Define a decision tree to isolate root cause. You may choose a positive (rule in) or negative (rule out) approach. Typically the approach you choose is based on your confidence level in plausible root causes you’ve identified. Much of the time, this is less formal than it sounds and is driven from gut feel. If X happens then there is no way the problem is A, but if we see Y then it might be B, etc. The set of possible scenarios and ways those scenarios might be confirmed or refuted is critical because it will push you in a specific direction and tell you where you need to instrument your system or what data you might need to gather.
Gather additional data required to walk your decision tree. Again, don’t be afraid of asking your customer to do work here. They will respect you for taking a disciplined approach to solving their problem.
If you have reached a conclusion based on the decision tree and gathered data, then you have isolated a plausible root cause. Otherwise, go to step 3 and repeat. If you believe you have misunderstood the problem or are no longer able to formulate a decision matrix, go to step 2 and ensure you are not in the weeds. You should end up in a situation that the data has shown you where the problem is, and you are able to confirm the fix works by turning it on or turning it off. Not all problems are as cut and dry, but many are.

Customers respect this approach. In fact, you can make customers love you for following a disciplined, data driven problem solving approach like the one above. A couple of words of wisdom (yes, I said wisdom, does that make me full of myself?):

Make sure they know that you’re in control and are on top of the problem.
Make sure they know what the next steps are.
Don’t be afraid of telling them bad news.
Be honest, clear, and direct with the customer.

There is nothing worse than a waffling, weak engineer on the other end of the line when a major customer problem is in progress.

We had no choice but to follow an approach like this at Metreos, and now at Cisco. The system that we have to diagnose is huge, and problems can be caused by everything from a faulty network element to bad software.

Perhaps the nastiest issue that we had at Metreos was one that was caused by a firewall that would behave differently depending on the order in which specific UDP packets were received. We solved this by having the customer take network traces from various points in the network, making sure we had clear good and bad case examples, and doing some good old fashioned detective work. It was stressful, but quite fun and fulfilling once we were able to show root cause. And best of all, in this specific scenario it wasn’t our bug.

The Root of All Meeting Pain: The Battle for Authority Points

louis@fitnr.com (Louis Marascio) — Thu, 25 Sep 2008 09:40:22 -0500

The most perplexing thing to me about working in a large company is this concept of Authority Points. I think it might be the root cause of why people always grumble about big companies, especially when they come from a small company background.

Authority Points

You see, when meetings happen in companies of non-trivial size more people seem to be invited because there are a widely diverse set of interests and motivations within the organization. Meeting organizers tend to error on the side of inviting someone rather than not because they fear not including a stakeholder, even if that stakeholder is only mildly connected to the meeting’s topic or goal.

Now, once in the meeting, all of these folks have an urge. Most of the people invited are climbing the corporate ladder. Most folks motivation is to increase their dominion, grow their influence, and be seen as an important decision maker within the organization. To do this, they must earn Authority Points.

People earn Authority Points in numerous ways: they can do a good job, write a good email, execute well on a good project. But the absolute easiest way to earn Authority Points is to talk during a meeting. I’ll call this “harvesting Authority Points” because I see it as being similar to what most MMORPG players do when they are levelling up: they go around killing all of the easy creatures to get gold and experience points.

Why This Sucks

This is bad because it is one of the primary reasons why meetings go long. It is, in my opinion, the primary reason why most meetings don’t accomplish anything. The people who are there to earn their Authority Points create churn by saying things that have little fundamental value to the goal of the meeting. Much of the time, they create work that is, at best, adjacent to the problem at hand:

Authority Points Harvester: I think we should really talk to
          some-guy-over-in-another-group because they might have some insight.
          
          Other People in the Meeting: (silently) Groan.

It’s unfortunate, really, but inevitable. The only thing you can do is be aware of these people and what they’re really doing when they talk. You can identify them in any number of ways, but one of the best methods is to try to recognize when someone is saying something that is a mere regurgitation of what has already been said in the meeting. They’re doing this because they have nothing of value to add, but they must be heard because they want to be sure to convert the last hour and a half of meeting time into at least one more Authority Point. Sigh.

How Tall is Your Ladder? Premature Optimization or Prudent Foundation?

louis@fitnr.com (Louis Marascio) — Mon, 14 Jul 2008 20:55:31 -0500

Premature optimization: it’s the hater-programmer’s¹ trump card. These days it seems everywhere I look I see these folks yelling, “Stop, you’re optimizing prematurely! Thou shalt perish in programmer hell!” To this I always want to respond, “What exactly is premature optimization, and why do you think I’m doing it?”

It’s funny, really. In the past six months I’ve heard premature optimization thrown about as a reason not to do something at least seven times. So often has it been used, in such dubious circumstances, that I’ve come to realize that the original intent behind this nugget of wisdom has been warped and skewed by hater-programmers with dubious intentions.

According to the ever growing awareness, Wikipedia, Donald Knuth stated the following in the article “Structured Programming with go to Statements” in the ACM Journal Computing Surveys:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

The article on Wikipedia goes on to say:

“Premature optimization” is a phrase used to describe a situation where a programmer lets performance considerations affect the design of a piece of code. This can result in a design that is not as clean as it could have been or code that is incorrect, because the code is complicated by the optimization and the programmer is distracted by optimizing.

This I completely agree with. Premature optimization is, essentially, a way to say that programmers would rather fiddle bits than confront the design of their software critically. I’d even provide an alternate, perhaps more accurate description: premature optimization is optimization without profiling. In other words, you’re optimizing without pinpointing the problem and without the ability to measure the results; however, this doesn’t mean you have to make simplistic choices when faced with a lack of data.

In my opinion² this is one of the most commonly misused tenants of computer science today. So often is “premature optimization” used as an argument against proposed solutions, that I have begun to seriously question the rationale and motives behind most people who propose it as a counter-argument in technical debates. I simply think it is grossly mis-applied. Even more annoying is when people actually suggest that language choice is a premature optimization.

Let me provide a metaphor. Suppose you were asked to fetch a ladder for your friend. Now, you’re into ladders. Ladders are definitely your thing. You have three ladders, each a different size: a step ladder, a fixed ladder, and an extension ladder. Each one has its intended uses, and each one has explicit weaknesses. Which one do you take? You can only pick one.

If you’re a premature-optimization-acolyte you pick the step ladder. Why? Well anything else would simply be overcomplicating the situation without sufficient reason not to. The step ladder is small, easy to transport, and most importantly, sort of cute.

Me? Well, I’m no acolyte. Here is how I see it.

The step ladder is basically useless. It fills a very specific need, but is not adaptable beyond that need. It’s useful in that you can reach items slightly above your head, but its primary utility comes from the fact that the step ladder itself can neatly be tucked away, forgotten, behind some counter or under a bed somewhere. Height, my friend, is not its strength.

So, is it the fixed ladder or the extension ladder? Well, that’s easy depending on the answer to the following question: is the fixed ladder of the over-the-top, uber-Little Giant variety? If yes, then pick the fixed ladder and move on. If no, then the extension ladder is my choice.

Did you hear that? That was several thousand hater-programmers around the world gasping all at once. This is what they’re saying and thinking: “You’ve just prematurely optimized the solution. There was no data to prove that you needed a ladder that could extend to 42 feet.” Bah!

Humans, especially programmers, like to latch on to things we believe we understand. This precept of truth from Dr. Knuth is one of those things. Premature optimization is bad, but inappropriate application of the precept of premature optimization is, IMO, much worse. Why? It suppresses good ideas and creative thinking.

The ladder example provided above is straightforward. A ladder is about height. When asked to choose a ladder without sufficient data as to the height of the area to be reached pick a ladder that provides, in this order, the most flexibility and the most height. If the ladder is a Little Giant then pick it, that thing is amazing, 24 ladders in one! Otherwise, pick the ladder that provides the most height, they can almost always be positioned to reach a lower point, but can very, very rarely (at least safely) be positioned to reach a point above the maximum height of the ladder plus the height of the individual.

How does this apply to software? Well, more often than not, technology, tool, or design choices are changed because the argument against premature optimization is used to argue against what was simply a decision to favor flexibility, or as I call it, a Prudent Foundation.

Prudent foundations are simple. Given everything you know about a problem, pick something that your gut says is flexible and good enough, but nothing more. Be concious of sacrificing flexibility, that’s how we get out of binds later on. The biggest risk we face as engineers is the risk of the unknown. Picking a prudent foundation, or a solution that minimizes the risk of the unknown, is the best we can do given the circumstances we are faced with.

Let’s examine the most egregious misapplication of premature optimization. As I stated earlier I believe this to be language choice. Choosing a programming language is almost never a premature optimization. Language choice is about many other things, but mostly it’s about picking the right tool for the job given the problem’s business context. Optimization, premature or otherwise, plays very little role here. The problem and business context define the criteria under which you should consider your decision.

Picking a prudent foundation is not premature optimization, even if 10,000 hater-programmers believe it be at first glance. Usually, the premature optimization crowd are speaking from a position of comfort. It is premature optimization if it uses anything they aren’t: (a) familiar with, or (b) comfortable with (no, these are not the same thing).

As engineers, one of the most important things we should be doing is discovering and promoting the prudent foundation for the problems we are faced with by making sound technology decisions.

We all know them. These are the people who argue viewpoints based on religion. They are usually arguing against pragmatic solutions and are most commonly found aligned with the forces of evil.↩
That’s right, I get to provide my own, personal, humble, opinions on my very own website. Go figure.↩

Hello C, My Old Friend!

louis@fitnr.com (Louis Marascio) — Wed, 09 Jul 2008 14:18:43 -0500

Recently I needed to parse a large ASCII flat file. The files are generally about 230 MB large with about 1.1 million records each, but there are some that are as large as 1 GB. To do the heavy lifting I decided to use C, something I’ve not done in a very, very long time. Before I go on, let me share with you a little bit of history.

In my recent professional life (say, the past seven years or so) I have mostly been writing C# code. At Metreos and now at Cisco the product that I build is actually constructed of components written in C#, Java, and C++, but the majority is C#. C# is a nice language; it’s very productive and has a very clean syntax, and I enjoy using it to build software. Unfortunately, deep down inside my nerdy heart there is something missing. You see, I started programming by writing C code and it’s always been something I enjoyed doing.

Many moons ago, when I got my first computer and wanted to program, I convinced my Dad to buy me a C book and compiler. Unlike many other people I didn’t start with BASIC or other “beginner” languages. Instead, I went straight into the dark woods of C-land, and I had a blast. Of course, being a first class dork, one of the first things I programmed was a terribly lame adventure game:

> n
          You go north.
          You're standing in a musty, underlit room with only one exit.
          The only thing you see are other nerds.

I eventually migrated to C++, then Java, and finally C#. As I’ve said before, I like just about every programming language as long as my limited brain can grok what’s required to use it. Thus, I have very, very little religion when it comes to programming languages. Some people deride anything that isn’t their favorite. C, C#, Java, C++, Ruby, and Python all have their fair share of bigots. I have better things to do with my energy than to hate other perfectly good languages. Yes, I’m of the “can’t the nerds of the world just get along” camp. Of course, there is only one true editor, all others are simply inferior.

Anyway, I think our progression through programming languages is interesting. Most of the folks that I work with have progressed similarly through languages. Many of them stopped at Java, some went to C#, and others have found nirvana with Ruby or Python. Something that I think is amazing is how many refuse to go back. It’s so common these days to talk to programmers who have found their last resting spot with C# or Ruby, and when asked, simply refuse to consider using a language they’ve already “moved on” from, or at the very least, would only begrudgingly do so.

As software developers, and ultimately, engineers, why do we get religion so fast when we find a solution to a specific problem. All problems are different, and thus should be subjectively evaluated based on their attributes within a specific business context. “Huh?”, you say? Here’s what I mean, please follow along:

All problems are unique across multiple facets. A typical set of facets might be: time to solve, required performance, resources available, and solution criticality. A collection of these attributes make up the problem and it’s container, the business context.
The tools (read: programming language and related tool chain) that we choose to solve given problems should be chosen within a decision making framework constructed from the problem’s facets.

Ultimately you, the engineer and, potentially, your team, are asked to make technical decisions based on a critical review of the problem. Your business leaders and teammates trust you to make these value judgements in an agnostic way. Unfortunately, more times than not, engineers choose the tools they are most comfortable with to solve the problem, without critical examination of the others available to them. Of course, comfort, or more specifically, familiarity, is a valid criteria to consider when making tool chain decisions, but it should be weighed appropriately within the business context at hand.

For example, consider the following two scenarios where you are asked to build a parser for an ASCII flat file:

You must do it by the end of the day. The file needs to be parsed and its records placed into a database where they will be retrieved at will. The program will only be rarely used, and the difference between a 1 hour run time and a 10 minute run time has little impact on the business.
You must do it as fast as you can, without sacrificing quality. The file needs to be parsed and its records placed into a database where they will be retrieved at will. The program will be used daily, and the difference between a 1 hour run time and a 10 minute run time has a measurable business impact.

Ask yourself, what language would you use? Why did you pick that language? Did you even consider programming languages you haven’t used in recent work? I’ve worked with people in the past who will immediately say Java, or Python, or SomeOtherProgrammingLanguage without any discussion as to the alternatives. Heck, I don’t know about you, but I have to force myself to not provide implicit weighting to my personal biases.

So, anyway, like I said, recently I needed to parse an ASCII flat file. The files are about 230 MB large with 1.1 million records each, but there are some that are as large as 1 GB and 5 million records. To do the heavy lifting I chose C. This is why:

This is a hobby, so the business context was simple: learning and fun.
I wanted the program to be as fast and efficient as possible. Ideally, it should take no longer than 15 seconds to completely parse the largest of the available files.
I may have several thousand of these files, so the different in run time between one minute and 15 seconds has a direct impact on my personal hobby productivity.
The entire data set for a single file needs to be stored in memory. This means I need to be careful about how I read the file. My program uses memory mapped files.

I haven’t done a benchmark to know how fast I could have parsed these files using Java or C#, but I do know that I’m generally going as fast as I can reasonably expect to go with my current implementation. My business context didn’t provide time for micro-benchmarking. Had it, I might have written a few benchmarks prior to picking my language.

The examples I provided here are simplistic. The decisions we’re asked to make as software developers in our professional lives are inherently more complex with significantly more ambiguity. It’s our duty as professional programmers to make these decisions with no technology religion to ensure our value judgements satisfy the needs of the business and not simply our desire as programmers to avoid the unknown.

Holy Q Batman!

louis@fitnr.com (Louis Marascio) — Tue, 01 Jul 2008 17:04:25 -0500

One of my hobbies is looking at financial markets. Market microstructure¹ is fascinating, especially when you take a deep look at not only how the markets function (price discovery, information propagation, etc.), but also the various types of market participants and their motivations and expectations. Needless to say, investigating markets and the associated reams of time series data produced by them allows me to exercise particular muscle groups in my brain that don’t get used every day. It’s fun, and just like when you hit the gym after a three month break to work out those deltoids, it usually hurts a lot afterwards!

Quantitative analysis involves the study of time series data and their statistical characteristics. It’s an interesting field with equally interesting tools. Some people never leave Excel while others use Matlab or other commercial and open source tools. Every once in a while, you run across tools and languages that just blow your mind. Enter kdb+ and the Q language from Kx Systems.

Part database and part programming language, kdb+ is one of those dirty little secrets in the financial world that tend to stay hidden from public view. It is used by financial services companies as the basis for some of their important trading and analytical systems. The really mind blowing part, in my opinion, is not the database itself, but the language you use to control, configure, and program it: Q. For example, the definitive book on kdb+ and the Q language is called Q for Mortals². Here, look at this piece of code and you can see why³:

sample:{ 
            t:("DSF"; enlist ",") 0: `:c:/q/data/px.csv;
            tmpx:select mpx:max Price by Date,Sym from t;
            h:hopen `:aerowing:5042;
            rtmpx:h "select mpx:max Price by Date,Sym from tpx";
            hclose h;
            .[`:c:/q/data/tpx.dat; (); ,; rtmpx,tmpx]
          }

Holy Quantitative Analyst, Batman! That is some hardcore code right there. Even more interesting is how the functionality of this code snippet is described in the book (slightly edited):

The [previous] program reads a csv file of time-stamped symbols and prices, places the data into a table and computes the maximum price for each day. It then opens a socket connection to a q process on another machine and retrieves a similar daily aggregate. Finally, it merges the two intermediate tables and appends the result to an existing file.

Wow. Any language that makes use of non-paired back ticks has got to be cool, right? From what I can surmise, engineers who understand kdb+ and Q are in high demand by the financial services industry. I suppose if you understand how to use such a seemingly powerful tool then you can almost set your own price.

Very cool indeed, but I don’t think my brain will allow me to learn such an obscure language. I’ve always been terrible at those obfuscated C contests. The only reason I found kdb+ was that Kx Systems recently released an individual license, allowing mere mortals like myself to download the software, try it out, and eventually realize that we’re not worthy.

Alright, everyone together now: Thaaannnks Kx Systems!

If you’re interested in market microstructure I strongly recommend Professor Larry Harris’ book Trading and Exchanges: Market Microstructure for Practioners.↩
The book is hosted at Kx Systems’ online wiki. To access the site with read only permissions use the user name ‘anonymous’ and password ‘anonymous’, as outlined on the code.kx.com landing page.↩
This code and the description of its functionality is from the Q for Mortals book, 2nd edition. Specifically the Overview section.↩

Why Do I Always Have to be Different?

louis@fitnr.com (Louis Marascio) — Mon, 30 Jun 2008 22:30:00 -0500

I like to use my personal projects as opportunities to learn about new things. Sometimes I explicitly try to reinvent the wheel — after all, this is my time to learn how to make a useful round object. I dedicate a certain amount of time every week to this type of tinkering, and the tool chain used to build this website is a result of such activity.

One day I’ll spend some time explaining why I think tinkering is so important, but to summarize it quickly, tinkering is about becoming and staying a well rounded programmer¹ by gaining different perspectives on technology not encountered in your typical professional life.

My goal for the FITNR website is simple: I want to build a place on the web that I can share some of my thoughts, but I want to do it my way — more on this later. The very reason why I don’t use tools like Wordpress is because launching this website is an opportunity for me to learn — consistent with the philosophy described above.

To further complicate things, I don’t want to rely on any dynamic server side code. I want something that is static on the server — partly driven from my desire not to add to my already too large personal systems administration workload and partly as an experiment. Under this rule all interactivity needs to come from Javascript as executed by the client-side browser. Mashups are all the rave these days, and I’d like to explore that world a little bit.

The manner in which I edit, build, and publish Fixed in the Next Release is partly an experiment in building a publishing system around my favorite tools and partly an exploration of how interesting a website one can build without relying on a server side component. Quite simply, can I make the mashup-potatoes taste good enough — is it possible, without all that lovely server side² butter? This is just one big opportunity to tinker.

To summarize, here are my requirements:

No dynamic, server-side content generation
Must have some way of engaging in public discussion
Must support searching
Must support tagging
Must generate a feed of some type, probably RSS

First, let’s talk about the tool chain. I love programming languages. When I’m tinkering, I like to use languages that don’t always make it into my professional life. For this website, I wanted to use Python for everything and that naturally lead me to Jinja2 for a templating system as the basis for my HTML generation.

Templating

Jinja2 is neat. The template syntax follows that of Django’s lovely syntax, in fact the original hack of this site was written using Django’s template system by itself and I was able to switch to Jinja2 with only one or two minor modifications. Template inheritance, filters, and blocks make putting together something like FITNR a piece of cake.

Editing

When it comes to authoring the content I really only need (or want) one thing: vim. Jinja2 continued to fulfill it’s promise of sweetness by already having a vim syntax definition for their templating.

Launch vim
set ft=htmljinja … sweet colors, nice
Profit?

Of course, the actual articles are written using Markdown and there is vim a syntax definition for it as well. Finally, I integrated SmartyPants to add a bit of flare to my web typography. Luckily, for both of these bits there are nice Python libraries³ that do the heavy lifting.

Revision Control

Revision control was simple: everything lives in Subversion.

> svn co http://svn.fitnr.com/fitnr

It turns out, what I really wanted was to be true to the Unix philosophy, partially shown below (courtesy of Wikipedia):

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams.

This was my primary requirement, and I had to be able to wire it all up with Python easily. No monolithic tools here, thank you very much!

How it Works

It all turned out quite nice, actually. I have a basic site structure as follows:

fitnr/
              assets/
              content/
              pages/
              templates/

The directory structure provides a nice segmentation of the various inputs required to make the site happen:

assets — Javascript, stylesheets, images, etc.
content — articles that will be published, many of which currently sit in an unfinished state
pages — the actual site structure, built with static html and Jinja2 templates, including my feed
templates — common layouts and designs written in Jinja2’s template language

When I want to start working on a new article I create a file in the content directory. This file has a simplistic structure: a series of faux email-style headers that define meta-data, followed by an empty line, followed by the actual content written in Markdown syntax.

The files in the pages directory use the data from the content directory to render the site. Each page can be a Jinja2 template and can generate any content I wish. For example, I have a page that actually generates my feed for me.

I have two simple Python programs. One is called makeit.py that 111 lines of code, including comments, that does the following:

Recurse into the content directory and parse each of the posts.
Recurse into the pages directory and pass each page to Jinja2 for template processing.
Jinja2 processes the template, and loads any required parent templates from the templates directory as they are needed.
Copies the files from the assets folder into a build directory.
Saves each of the rendered pages into the build directory.

The other Python script is a simple web server that uses SimpleHTTPServer from Python’s standard library. It’s all of 21 lines of code and allows me to preview the site before publishing.

So how did I do against my original goals?

No dynamic, server-side content generation — ✔ No problems here, everything is static
Must have some way of engaging in public discussion — ✔ I sort of cheated on this, but only sort of. I created a Google Group called fitnr-discuss that you can join if you’d like to discuss what you see on this site.
Must support searching — Not yet
Must support tagging — Not yet
Must generate a feed of some type, probably RSS — ✔ I generate an RSS v2 feed here

There are quite a few things I’d like to add to the site in the future, but first I need to complete the tagging and searching functionality. For tagging I just need to code up the views that I’d like to expose, and for searching it’s a matter of reading some Google API documentation.

There are some other obvious things that need to be done sooner rather than later, most notable I need to finish the archive functionality so that each post gets a permanent link.

This goes for any profession, really. It’s the desire to learn beyond one’s standard job function that, in my opinion, typically separates the cream of the crop when observing practitioners of any trade or function.↩
You’ll have to forgive me a little bit here. Of course I understand that somewhere on the Internet will be a server side component. I just don’t want to host it.↩
Markdown in Python and smartypants.py ↩

Better Late than Never, I Suppose

louis@fitnr.com (Louis Marascio) — Thu, 22 May 2008 12:00:00 -0500

Well, I suppose it’s time. I’ve avoided jumping online with an official-like presence for long enough. At this point I feel like the guy in the 90’s who was still using a Walkman listening to cassette tapes.

What will you find here? Mostly a discussion on software development, software teams, and other related technical bits and pieces. You’ll find a healthy dose of cynicism and pragmatism in my viewpoints, but more importantly I hope there is a fair bit of humor.