Code / Appnel Solutions

Mea code? Mea culpa!

2008-07-10T20:49:40Z

Just a quick note for anyone who tried to download any of my plugins. A hasty upgrade a week or so ago to appnel.com exposed a bug that made downloads impossible. Thanks to all who reported it. I made a fix the other day and things should be back to normal. Sorry for the inconvenience.

YAPC MTOS and Amazon Web Services Presentations

2008-06-20T20:02:50Z

I’m back after three near sleepless days in Chicago attending and speaking at YAPC:NA (Yet Another Perl Conference North America). I wish I had more time to take in Chicago, but it was great to meet some many of the various Perl coders including the father of Perl, Larry Wall, who’s efforts continue to make the community thrive.

Both of my presentations where geared towards advocacy of MTOS and my latest project Bezos, an effort to create an integrated library for working with the Amazon Web Services cloud and eventually a command line tools.

PDF copies of both including the example code I showed are below.

BTW: For those in attendance or if the video that as was shot during my Amazon presentation makes it into circulation, I reset my security keys. I used the S3 Browser to show how some of the scripts had worked and that app has the annoying trait of displaying all of your account information on startup. I knew that, but did it anyway knowing I could just reset my key.

On the Decentralization, Diversification and Colonization of Perl

2008-06-11T01:48:22Z

I’ve been meaning to write this post for some time, but my relocation, client work and now sweltering heat has kept me busy. As I begin to prepare my presentations for next week’s YAPC::NA conference I feel compelled to finally throw down the gauntlet and write because I believe they are important points that seemed to be missed by most of the Perl community.

I’ve been following Andy Lester’s Perl Buzz blog since it launch and applaud his efforts to bring “fresh blood” to the Perl community. Like Lester I believe the Perl community has a lot to offer and by no means is a dying or irrelevant language. What I believe it does suffer from is what I will call a marketing problem. Some of it is “visibility” as Lester refers to it, but more of it is packaging, perception and presentation.

This thought has stuck with me for years, but it wasn’t until I read Lester’s Perl must decentralize, diversify and colonize post that I that I felt it was time to write about it here.

If you haven’t read the post I’m referring to or follow Perl Buzz I suggest you do.

So let me state upfront again that, for the most part, I agree with most of what Lester wrote.

I said for the most part though. Where I differ is in why other languages, PHP and Ruby (with Rails), are more en vogue. To me these are important points that are getting missed and overlooked by the Perl community that has lead to this condition.

These points are most apparent by comments made in the lead up to his three objects — decentralize, diversify, and colonize.

I’m certain that PHP has become a de facto choice for basic web apps because it’s just How You Do It these days. You see enough PHP in the context of the web, it starts to sink in.

This is generally true, but this view overlooks how PHP came to such a place of prominence in the web applications ecosystem. I don’t think PHP is a better then Perl in any means, in fact I thinks it’s pretty crappy stuff that I’ve never liked working with. Still PHP succeeded because it was just good enough and made running web application easier then anything else for most people.

PHP was free unlike Application Server Pages (ASP) or ColdFusion and was easier for hosting providers to offer and maintain then alternatives such as mod_perl or the resurgent FastCGI. It’s popularity was further enhanced as a default module of Apache making it’s availability through cheap shared hosting nearly ubiquitous.

Of course Perl and CGI for that matter is equally as ubiquitous, but they were and still are perceived as a less attractive option to many because it is harder to get something up and running. FTP a text file with a mix of HTML and PHP script and a .php extension and presto! it runs. Unless severely abused, PHP is more responsive because its engine is typically persistent thereby avoiding the performance tax any language running through CGI must pay. PHP does require a server restart like FastCGI or mod_perl to pickup changes either. Further, numerous vital (and then some) libraries come built-in and are then easily accessed where others languages like Perl leave that to the user assuming they have the rights to do so. The hosting provider could, and indeed some good ones will install a lot more then just the core distribution, but knowledgeable Perl hosts seem to be few and far between these days.

As an active member of the Movable Type community I’ve seen the difficulties Perl/CGI issues have caused users and impacted interest in Perl with a broader less adept user base.

Why is Ruby on Rails so popular? Is it better than Perl on Catalyst? Or is it just that people hear about Rails more? I suspect the latter, because perception is reality. When people perceive Perl as being dead, or not as powerful as other tools, it might as well be.

I whole-heartedly agree with Lester’s observation that perception is reality. Like his observation about PHP I think it misses an important point though.

Rails, in which Ruby would still be nowhere, made it easier for more professional developers to write applications with great user experiences quickly and easily. The genius of Rails is in its marketing.

I remember when I first heard David Heinemeier Hansson present on Rails how intrigued and even excited I was with how productive a developer could be. What really made Rails work so well in this context was that it was “opinionated software” that said “flexibility is overrated.”

This is of course heresy to the Perl community, but I think that perhaps the Perl community’s treasured there-is-more-than-one-way-to-do-it ethos needs to be broadened to include that sometimes one (highly preferred) way of doing things as an acceptable option.

This is why I was disappointed when I read about Catalyst. This is not to say that I think Catalyst is bad or not capable. It’s that it missed the most important advantage of Rails and why it was so successful in attracting a large fanatical community of developers so quickly. Catalyst wasn’t what I wanted.

I think there is a large group of users that don’t care about choice, but getting their job done as quickly and easily as possible. Business managers, entrepreneurs, investors, and the technologists working for them don’t care if a web application is written in Ruby or Perl or PHP or Java. What they do care about is providing a great user experience that will attract and delight users, grow a company, and provide the best return on their investment in time and effort.

Rails made that possible by having an opinion on its conventions (over configuration), JavaScript/AJAX libraries and so on that allowed it to create tool and libraries which helped developers write less software.

This is why so many of the most popular sites and web applications to launch are written using Rails, not just because people hear about it more.

PHP and Ruby on Rails lowered the barriers to entry and have reaped the benefits. Capable and powerful as it is, Perl has not.

I’m not sure precisely what moves the Perl community has to make, but I’m certain that without more (a lot more) attention to perception and marketing of Perl interest will wane. No amount of language theory or lightning talks or state of the onion will stop that.

I do hope that the Perl community does decentralize, diversify and colonize and that a big part of that work includes making it easier for developers to use and deploy Perl applications.

On MT Feed Support

2008-05-09T20:11:43Z

The next version of Movable Type, version 4.15 (aka “Cal”), will remove the RSS feed template from the defaults. This touched off a thread recently was spread across the mtos-dev and ProNet mailing lists discussing the reasoning and potential drawbacks to this decision.

Feeds is a topic near and dear to my heart that I have a long history with. Seems we as a whole have never gotten over all FUD and misinformation of the “syndication wars” that some clarity is needed. Here is the most important takeaway from this post:

There is no longer any practical reason to have multiple feed formats of the same information.

It seems this issue comes up every so often.

Any feed reader or software the consumes feeds worth anything can handle both RSS or Atom feeds. Every modern programming language I know of also have libraries from processing both. Not all take advantage of the more specialized and esoteric features of either — some do, but for most uses either if fine.

Choosing one or the other isn’t choosing sides. It’s a matter of usability and practicality. Having the same feed information in multiple formats forces a user to make a choice that they probably don’t understand and one that is ultimately unnecessary. Dean Allen recently wrote about it and before that Nick Bradbury.

But why did Six Apart choose Atom instead of RSS? MT Product Manager, Byrne Reese posted:

This decision was never about standards politics or what format is better than the other, this release is about performance, and in this day and age, there is no need to publish feeds in multiple formats when they are supported equally[1].

We selected Atom because all things being equal, Atom is an Internet standard and RSS is not, and that is more closely aligned with MTOS’ stated project goals.

[1] Here Byrne asked a post to identify which tools do not support Atom.

It’s also worth mentioning that MT (and other Six Apart tools) are standardizing on the Atom Publishing Protocol (AtomPub) for remote clients to interact with the system. AtomPub uses the same format. RSS does not have an equivalent web services API option. Besides that, there are numerous flaws in the design of RSS that can and often does confuse news readers. The RSS specification is frozen with these flaws; however, there are groups attempting to address them in other ways.

From a software architecture and engineering perspective, to me, it’s quite logical that Atom be used. Popularity, as some as cited, is irrelevant given the circumstances and does not make the tool better.

This decision will only effect new blogs created in MT 4.15 and future versions. It does not stop you from using and creating your own RSS feeds nor will it delete, remove or disable them from your current blog templates.

That said, if you are producing feeds in multiple formats, I recommend you pick one and redirect the traffic to the others to that one. Doing so will help improve performance as Byrne suggests in his reply, means one or more less files to generate when a new post in made.

Have a software product that can’t read Atom files? Post them in the comments.

Appnel @ YAPC:NA 2008

2008-05-01T19:43:55Z

Well I didn’t get a spot to present as OSCON this year, but I was pretty pleased to hear from the organizers of YAPC:NA (Yet Another Perl Conference North America) that not one, but two of my proposals where accepted — in back-to-back sessions nonetheless.

The conference is being held in Chicago June 16-18. The the schedule is here.

I haven’t been to YAPC before so it should be interesting. From what I understand this conference is for the most hardcore of Perl programmers. I have some big plans for what I want to present and will be praying to the gods of “free time” to find some.

Meet The WordPresses

2008-04-18T23:03:46Z

For the past few weeks I’ve been monitoring the blogging questions on LinkedIn out of curiosity. In that time I’ve seen questions that asked for advice on TypePad vs. WordPress that received confused answers. Which WordPress was the asker referring to? There are significant differences.

WordPress.COM and TypePad is a fair comparison because they are both hosted blogging systems. WordPress.ORG is the open source software that you can download and install on your servers. With the source code you can modify it to your heart’s desire and use plugins etc. So if you are referring to WordPress.ORG then Movable Type is a more relevant then TypePad.

While this confusion is easily cleared it is still lost on most people. The WordPresses share the same name, but in many, but not all, respects are different things.

This could have been easily avoided if WordPress.COM would have been named something else. After all, TypePad began as a fork of the MT code and still shares some parts today. Given the popularity of Movable Type then (and still now), Ben and Mena could have chosen to name that service Movable Type. I’m thankful they didn’t.

Go, Blog It! Go!

2008-04-17T18:41:59Z

Six Apart continues to break down the walls around social communities.

Yesterday Six Apart announced the availability of a Facebook application (yes you read that correctly) called Blog It. The TypePad-powered service enables Facebook users to make posts in Facebook out to any number of blogging tools.

Here are a couple of links covering it:

Six Apart CEO Chris Alden wrote up a good summary on his blog here.

Of course, there are those who are less than impressed or dismiss it as irrelevant, but then again Six Apart acknowledged there are shortcomings that they will be addressing. After all, this is a 1.0 release

I think it’s still pretty slick though.

Our Man Arvind

2008-04-16T21:56:44Z

Keeping up to date on the latest MT community news and happenings is easier then ever.

Our man Arvind Satyanarayan has started authoring a weekly column for Blog Herald summarizing the weekly happenings in the MT community. His first two posts are already up:

New posts are to be published every Monday, hence “Movable Type Monday.”

Now if we could only get a feed dedicated to his posts.

BTW: When I say “our” I’m speaking as a member of the MT community. Arvind doesn’t work for Appnel Solutions.

Oh Wow! Persistent Storage for Amazon EC2 Announced

2008-04-14T15:44:09Z

Amazon announces beta functionality that will address the biggest hassle for using EC2 servers for running web applications — a file system that won’t vanish if an instance stops running.

Chief Evangelist Jeff Barr explains the issue the Amazon Web Services Blog:

If you have taken a close look at Amazon EC2, you know that the instances are ephemeral. The instances have anywhere from 160 GB to 1.7 TB of attached storage. The storage is there as long as the instance is running, but of course it disappears as soon as the instance is shut down. Applications with a need for persistent storage could store data in Amazon S3 or in Amazon SimpleDB, but they couldn’t readily access either one as if it was an actual file system.

For my experience and those of others, instances expectedly shutting down has been rare so far, the possibility of catastrophic proportions was real. If an instance were to unexpected shut down it would be the equivalent of a traditional server’s drive array catching on fire — the data would be lost forever unless you had a very good backup system in place.

This shortcoming was acceptable in that EC2 was created for computing power and not general usage hosting, but that’s not what a lot of people wanted to do — many wanted to run web apps with SQL databases and other things.

So the ability to hosts sites or applications or run a SQL database server using EC2 instances was possible the onus has been on everyone using an instance to run file and database backups or even (multiple) replicated database servers even if high availability wasn’t a requirement so data wouldn’t be lost in the event of a failure.

Later Barr continues:

…our forthcoming persistent storage feature will give you the ability to create reliable, persistent storage volumes for use with EC2. Once created, these volumes will be part of your account and will have a lifetime independent of any particular EC2 instance.

In addition to being able to create and then mount these volumes of persistent storage from any EC2 instance, Amazon will be releasing functionality that will take snapshots (backups) of volumes and store it in your S3 account.

I also assume that you can mount these persistent storage volumes from multiple EC2 instances.

This is addition is simply huge because it makes Amazon a viable hosting option for web applications by eliminating the biggest technical hurdles. With this shift from compute to more general purpose server resources, I think a seismic shift is certain for how web applications are developed and deployed and Amazon is poised to win big.

Recognition!

2008-03-27T21:19:00Z

Serdar Yegulalp picked up on my Open Source Is Not Just a License post and highlighted it in a post to his Information Week blog. He writes:

There is indeed a great deal to gain by contributing, and while many people might sniff at the fact that at least some of it is PR (as Tim put it), it’s easy to forget that PR is a crucial ingredient in the glue that holds together a community. A person with a bad reputation as a fair player is less likely to be welcomed into any community; someone who has a track record of playing fairly — or at least attempting to play more fairly — will be welcomed and will be able to reap the benefits all the more enthusiastically. You tend to give more when you know you’ll get more in return.

He goes on:

What Tim means by open source not just being a license is reflected in all of this. Anyone can write and release something under an open source license — yes, even Microsoft — but that doesn’t mean they’ll be used, re-used, built on or well-respected. That takes time and engagement, and a sense that you need to give as good as you get.

There are times when I think I’m talking to myself in what I think. Perhaps I could be mistaken.

Concatenated Indexes for MT

2008-03-19T14:17:08Z

Watching to stream of commits to the MTOS repository I noticed one that intrigued me — concatenated indexes in MT objects. I asked and Brad Choate obliged me with an answer:

Prior to MT 4.1, the MT object schema never supported the declaration of concatenated indexes. Meaning, creating an index on two or more columns. Now that we can declare these, we’re making use of them, since it makes a big difference for the larger MT installations. So yes, it is significant, if you have lots of blogs with lots of entries.

This is a “wonky” sort of thing, but it will be great for performance and a prime example of the good work being done to improve the overall performance of MT in the next official release or two.

Being Competitive is "Dirty"

2008-03-11T20:14:36Z

Anil Dash made a very interesting post noting the pending release of WordPress 2.5 and suggests users consider Movable Type since upgrading means breaking a lot of things. As TechCrunch notes the WordPress crowd (Automattic really) is “pissed” with Matt Mullenweg calling it “desperate and dirty.” This of course is the pot calling the kettle black if it were true, but it’s not. Dash’s arguments are, to my knowledge, factually correct, quite logical and polite. To me it’s just a company being competitive in professional way. It would seem that Automattic is not up to dealing with with that sort of heat yet.

I’m not at all surprised by their reaction though. The WP community lead by the Automattic staff has dished out a lot of crap over the years at Six Apart’s expense without the professionalism that Dash extended them with this one post. I don’t doubt Automattic’s dedication to open source or what they do, but, given past behavior, I’ve always questioned their ability to conduct themselves in a way that professional and enterprise customers demand. This reaction only supports that observation and only makes me rest easier as enterprise consultant specializing in Movable Type systems.

Isolating Performance Issues

2008-03-06T18:17:05Z

Over on the mtos-dev mailing list, David Jacobs notes a post from one of the Tumblr developers about a bug in MySQL’s InnoDB engine severely impacting peformance and goes on to say:

It just goes to show how difficult to isolate performance issues are. In addition to the web server, templates, third party plugins, etc. Sometimes you hit an unforeseen bug in something completely external.

True.

We’ve been discussing performance on the list a lot on mtos-dev since it’s the primary focus of MT 4.2. Web applications of any complexity are exponentially more complex than the desktop software I started writing at the start of my career. Web applications like MT are made of dozens of individual pieces of software with different developers, milestones and objectives. The number of combinations and configurations are virtually infinite and in constant change that testing and debugging issues is quite a challenge. It’s the price we pay for the distributed nature of the Internet where so many small pieces are loosely joined.

The Value of Clean Code

2008-03-06T01:32:56Z

In an O'Reilly LAMP blog post, "Clean Code is Easier to Debug", chromatic writes:

Consistent code requires much less brainpower to decipher its structure, leaving that much more brainpower to find real problems.

I have found this to be quite true and encourage any developer to give it a try. Since I've been paying more attention to how clean my code is I've found that not only can track down issues faster, but that I can refresh my memory of how a piece of code works faster.

"I don't have time for that though Tim," you might say. It doesn't take as much time and effort as you'd think.

The tool that I found most useful and to have the biggest impact on helping keep my code clean and consistent is perltidy. I use it on every piece of Perl code I touch -- and often. I've even gone so far as to use it on other's code when I'm having a hard time reading through it. For instance, there was a time when I'd run all of MT through, its improved a fair bit lately, it so I could figure out how to make us of it. (Documentation should do that, but we're not going there in this post.)

Try it. Here is the perltidy configuration file that I use.

-ce     # cuddle-else
-lp     # line up with parentheses
-syn    # check syntax
-sfs    # space for semicolon
-bar    # open brace always on right
-nbbb   # no blanks before blocks
-pt=2   # Parens tightness (tightest)
-bt=2   # braces tightness (tightest)
-sbt=2  # square brackets tightness (tighest)

You can customize how perltidy formats your code ad naseum. These are just my preferences.

Funny thing I noticed after using perltidy for a bit is that I started to write my code differently so it looked even cleaner once perltidy finished its work.

So while perltidy will make your code consistent and in a sense clean, that's not all there is to it. Clean code also means making code legible and well organized.

For instance, once put through the eye of perltidy I stopped writing long lines of code -- especially nested statements. So, I'd take a line like this:

my $count_iter = $class->count_group_by({
    'asset' eq lc $type ? () :
        (status => MT::Entry::RELEASE()),
    %$terms,
    },{
    group => ['objecttag_tag_id'],
    'join' => MT::ObjectTag->join_on('object_id', { object_datasource => $class->datasource, %$terms }, $args),
    'asset' eq lc $type ? (no_class => 1) : (),
    %$args
});

Note this is one lloooong line of code that takes time to dissect in order to understand everything it's doing. I found that my code was quicker and easier to read, understand and debug if I rewrote the same logic using smaller statements.

my $t = %$terms;
my $a = %args;
if ('asset' eq lc $type) {
    $t->{status}   = MT::Entry::RELEASE();
    $a->{no_class} = 1;
}
$a->{group} = ['objecttag_tag_id'];
my $jt = {object_datasource => $class->datasource, %$terms};
my $join = MT::ObjectTag->join_on('object_id', $jt, $args);
my $count_iter = $class->count_group_by($t, $a);

Another benefit of this approach is that error messages are more accurate in pinpointing where the issue lies. In the first example, any error would be reported as the same line.

Another thing I started doing more was grouping variable assignments and declarations as much as possible. I found that practice made it quicker to find where and what you are assigning to a variable in a large block and its just neater with all those equal signs. (Yes like I once heard David Heinemeier Hansson say, "Call me shallow. I love beautiful code.")

There are a lot of little things you can do to make your code cleaner and more consistent. These are just two ways of many mostly smaller ones I've built in to my style.

If you are interested in learning more ways to write cleaner code I suggest checking out Damian Conway's Perl Best Practices. I don't agree with every practice that he evangelizes and some seem a bit over the top, but overall it's pretty good food for thought.

Incidentally many of the coding best practices from the book have been codified in Perl::Critic, a module that critiques perl source code issues warnings and errors based on the configured options. It has an extensible framework that has been used by many other CPAN developers to create many more policies. There are even modules for testing for Perl::Critic and testing perltidy compliance while running your testing suite.

Though I have every intention, I've yet to give Perl::Critic a try on my code nor have I worked both in to my standard test suite.

There is always room for improvement.

On Testing

2008-02-28T19:37:14Z

A recent thread on the cgi-app mailing list discussed testing that is really worth noting.

The key statement came from Jason Purdy when he said:

My advice would be to factor your code such that the code is separated into a testable module.

I admit to being slow to developing tests for my work once I did I came to realize that this statement above to be very true. Once writing tests where introduce in to my development process it changed how I wrote the code, not only to make writing tests easier and more thorough, but for the betterment of the code.

I’ve yet to break the habit of writing code and then the tests. (Testing advocates suggest you write tests upfront.) This one has been harder for me because often start from existing code, experimenting and capturing ideas that slowly gets molded into working code. On a few occasions I had a written a piece of code before I realized I was doing it.

I’m strange though and this is not an argument or excuse to follow good form.

Bill Costa concurs and expands on Jason’s statement:

This is excellent advice. Try and write your ‘back end’ modules so that a browser is not needed to exercise them. Then for testing you write a simple script that can be executed from the command line to fully test all of the services provided by your module. It then becomes a much simpler matter to create a test script that probes all of the expected operations as well as the boundary conditions and even pathological cases. In other words, don’t assume that service will always be given sane data from the front end. Test for possible inputs that “should never happen”.

The book cited earlier will help you write those command line test scripts in a very efficient manner using well established and vetted test harness modules.

“The book” here is Perl Testing: A Developer’s Notebook by Ian Langworth and chromatic. Like others in the thread, I recommend it if you really want to learn about testing in Perl.

Bill continued:

An added benefit to this approach is, when you later have to make a change to one of these modules, you can run your tester to make sure you didn’t accidentally break something else in the process.

In this way testing the final app itself should be more about usability testing with humans rather than actually trying to test the correct functioning of lower level functions through the app’s web interface.

Agreed. It’s an ideal that all developers should aspire to achieve. It’s easier said then done, but I do believe it pays dividends the closer you get to this ideal.

After starting to write tests for my CPAN modules I decided that I wanted something for my MT plugins — especially the big and involved ones like Feeds.App and Tags.App. I ended up creating my own testing framework for MT that has a lot of room for improvement, but works pretty well and gets the job done. The framework was written for MT3 systems and then quickly hacked to support both MT3 and MT4. It’s available under the Artistic License like Perl. Contributions are always welcome.