Planet Perl

Log::Any released, only two years late

2009-07-13T06:02:15+00:00

Log::Any has finally been released to CPAN. It allows CPAN modules to log messages in a generic way, while letting the application choose (or decline to choose) a logging mechanism such as Log::Dispatch or Log::Log4perl.

This is something I’ve been meaning to implement for nearly two years - talk about procrastination! What pushed me over the edge was that three of my distributions - Mason, CHI, and an upcoming Server::Control - all needed to log various things, and had no good generic way to do it.

As part of this effort, I surveyed the various logging packages on CPAN to see which ones Log::Any should interface with. Log::Dispatch and Log::Log4perl were the heavyweights and obvious first choices, but I was unprepared for the sheer variety of other distributions. Logging, like templating systems, seems to be one of those things that everyone wants to try their hand at! Here’s a partial list of logging packages that all share roughly the same mission - dispatching logs to various outputs - along with the number of distributions that depend on each:

Log::Log4perl	176
Log::Dispatch	40
Log::Agent	18
Log::Report	16
Log::Trace	13
Log::Dispatch::Config	9
Log::Message	5
Log::Channel	3
Log::Dump	3
Log::Handler	2
Log::Info	2
Log::LogLite	1
Log::FileSimple	1
Log::Message::Simple	1
Log::StdLog	1
Log::Simple	-
Log::Simplest	-
Log::Tiny	-
Log::Trivial	-

I particularly like that there are seven packages named with a variation of “Simple”, “Trivial”, etc. Trying to choose between these would be the opposite of simple.

To be fair, Log::Dispatch, presumably the easier of the two main logging packages to configure, requires this just to log normally to a file:

   my $dispatcher = Log::Dispatch->new();
   $dispatcher->add(
       Log::Dispatch::File->new(
           name      => 'foo',
           min_level => 'info',
           filename  => "$dir/test.log",
           mode      => 'append',
           callbacks => sub { my %params = @_; "$params{message}\n" },
       )
   );
   $dispatcher->info('this is a message');

It’s no wonder people tried this and yearned for a simpler alternative. (I’ve gotten permission from Dave to simplify a few of these things with a maintenance release.)

Test::Pod 1.40 now checks for illegal L<> constructs

2009-07-13T04:41:17+00:00

It will be interesting to see what, if anything, barfs because of the new Test::Pod.

This POD construct is illegal according to perlpodspec, and will not be formatted correctly on search.cpan.org, either:

L<perlbuzz|http://perlbuzz.com>

Test::Pod catches it now. My wonder is how many people are using it.

Test::Pod also requires Perl 5.8 now, and I'd like to think that pretty much every module on CPAN could use modern Perl versions at this point.

Is it right to auto-notify authors, if there aren't many?

2009-07-13T04:02:57+00:00

Dear LazyWeb.

The FAIL 100 list has been working reasonably well, but one crucial downside at the moment is the need for people to actively monitor it (both those that are aware that it exists, and those that don't know it exists).

While I certainly agree that it is a bad idea to spam authors in general (I'm certainly for opt-in CPAN Testers summaries and emails) does this still hold true when you are identifying only a small number of authors.

Would it be ok to auto-email the Top 10 positions on the FAIL 100 list, say, each week to let them know their module is considered to be important to fix quickly?

What if it's just the first time it appears, and on subsequent sequential appearances we don't email you.

What if it was the top 5? Or the top 2? Or top 1? Or only if you met a certain threshhold of FAILure score? Or it was at a wider interval, say, monthly?

When is it ok for a whole-CPAN process to judge that an author needs to be nudged?

what i want from a d&d wiki

2009-07-12T20:03:28+00:00

I want a system in which my players can keep logs of the adventures. I want them to be in a clear sequence so you can click "next" and "previous" without having to set the links up yourself. I want it to be possible to have multiple summaries. One unbiased log, a per-player (or per-character) log, and maybe others.

I want some pages to be read-only (like house rules) and some pages to be hidden. For example, I don't want the players to see my notes on the game's secrets. I would like players to be able to create pages that only they can see, but I want to be able to see them all.

I'd like to structure pages so that any page can include content with the above access controls. I'd like really like parameterized transclusion so I can make many kinds of page look similar to one another.

I like tagging, so I want that, and I love the way that TiddlyWiki does it: when you tag a page with X, you are just saying that the page has a relationship with the page X. You can then click from the tag to the page for X to read about what it is or what the tag means.

That might be it.

I really don't need much of anything that's D&D-specific. The fact that some pages are "characters" and some are "locations" is sort of a user-specific layer on top of the generic structured wiki. I think making something D&D-specific is probably more about writing a workflow for using a wiki tool, and that's fine with me. By not building the RPG logic into the software, I can use it in different ways. For example, Obsidian Portal codes in its assumptions about gaming, so you can't have one campaign with two parties of PCs.

If I wanted to take this tool and make it something for many campaigns to use, then I'd also want workspaces. A workspace would just be a namespace with default permissions, and would probably represent a campaign. Nested workspaces would make multiple parties easy, but at that point it's probably overkill. I'm not sure.

I'm going to try installing Wagn, which looks pretty suited to my needs. I'm not certain about one or two parts, but I think that if I get most of what I want, I'll be satisfied enough to just use that. My experience has been that I'm very spoiled by CPAN, and that installing Wagn is going to be a pain. I'm looking forward to being proven wrong.

Learn Perl

2009-07-12T15:03:32+00:00

Recently, Perl golf has become popular on Stack Overflow. This is dumb for more reasons than I care to enumerate, but what really makes me mad is that most people engaging in this dubiously-useful hobby do not know enough Perl to do it right! Here is an example of doing it wrong:

map { ucfirst $_ } split /[.]/, $_;

Do you see the $_? If you knew Perl, you would know that functions usually operate on $_ if you omit it. You might not know which functions do and which functions don't, but you should see the $_ and wonder if you can omit it. Looking at the docs for split and ucfirst reveal that they both do in fact operate on $_ when another expression is not specified. Thus, we can golf this down to:

map{ucfirst}split/[.]/

Much more readable, and much shorter. (Also, if your task was to sentence-case your input, you did it wrong. But hey, at least it's only 22 bytes!)

Seriously though, if you are going to take up Perl golf as a hobby, learning the simplest of simple Perl tricks might be a good way to avoid looking like a reputation-point-whoring moron.

As an aside, Stack Overflow's reputation system is slowly killing the site. In an effort to gain reputation points, people are answering questions that they admit to not knowing the answer to. It seems like every answer in the Perl section starts off with the phrase, "I don't know Perl, but...". If you don't know Perl, and this is a Perl question, how about shutting the fuck up and not making yourself look like an idiot? What a fucking concept...

(This has also infected the Emacs section, which I mostly read for comedic value these days. It's fun to watch how quickly people can produce completely wrong and useless answers, or reimplement core functions in the most inefficient way possible. Need to move backwards one character? Pressing the <- arrow or executing M-x backward-char is too magical, it's much better to get the current position in the buffer, move the point to the beginning of the buffer, and execute a keyboard macro to press the -> arrow key that many times minus one. It's only 25 lines of code! So simple!)

obsidian portal and me

2009-07-12T04:13:57+00:00

A while ago, someone directed me to Obsidian Portal. It's a website where you can collaboratively develop an RPG campaign. A camapign has a wiki, PC and NPC tracking, an adventure (b)log, an item tracker, a map archive, and forums. There might be some other stuff, too.

It's a structured wiki, at least somewhat, so every page has hunks like "DM-only content." Characters have "background" hunks, and so on. It's easy, when writing up an adventure log, to link to a PC's wiki page by the PC's short identifier.

Obsidian Portal is a really fantastic idea. Unfortunately, the implementation is incredibly frustrating to use.

The problems are hard to describe, but it comes down to "the interface gets in the way of getting things done." Creating NPCs is annoying, and there is a largely obnoxious distinction between PCs and NPCs. The "short name" used to link to characters must be universally unique, rather than unique within a campaign. I could not, for example, make it possible to link to [[:valentine]] because someone else in some other campaign had used it.

I don't create a character by clicking "add" on my campaign. I have to go to the global character browser and create one there. A small dropdown allows me to add the character as a PC in a campaign if I want -- but not as an NPC. To do that, I'd need to do a bunch of editing later.

These kind of little irksome problems are absolutely everywhere in Obsidian Portal, making it anything but a joy to use. It has so many great ideas, though. I love the notion that I can have my DM-only hunk of notes on a given adventure (or any other wiki page) or that I can write up pages and reveal them later.

All the "share your campaign with others" stuff is neat, but seems incredibly unimportant compared to the "manage your campaign for your players." Despite this, things seem optimized for other people instead of your players. Maybe I'm wrong or maybe that's what the authors want, but it frustrates me.

Because of this and other ideas, I've been looking a lot at various content repositories, document databases, and structured wikis. So far, Wagn looks, by far, the most likely to be useful. Even it is sort of a half-fit, but it might be good enough to let me get on with running my game. If that falls through, I might be willing to try implementing something, but my time is awfully short to spend writing new structured wikis optimized for Dungeons and Dragons!

Tomorrow I'm hoping to write up what I want for my game and what I'd want to provide if I were writing Onyx Portal (or whatever) for a living.

In the meantime, my players recently got their latest adventure logged onto Obsidian Portal.

Perlbuzz news roundup for 2009-07-11

2009-07-12T03:43:39+00:00

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

Tim Bunce surveys customer relationship management systems in Perl (blog.timbunce.org)
Sending mail through Gmail with Perl (nixtutor.com)
From the DarkPAN: "The Perl community should move forward without worrying overly much about us that have to drive in the slow lane." (use.perl.org)
rgs's blog post about his resignation as bleadperl pumpking (consttype.blogspot.com)
How we should address the DarkPAN problem (perl-toddr.blogspot.com)
I think chromatic's points boil down to TAGNI: They Ain't Gonna Need It, where They is the unknown DarkPAN. I'm inclined to (agree.)
Patrick Michaud is looking for Rakudo release managers (use.perl.org)
perl/vendor/site explained (use.perl.org)
Top five non-technical mistakes made by programmers (makinggoodsoftware.com)
Six modules to help with complexity management (use.perl.org)
Test::Pod is now on github. Bug fixes welcome! (github.com)
CPAN Explorer gets props in Visual Complexity (visualcomplexity.com)
Upgrading Debian packages with newer CPAN releases (use.perl.org)

current project status braindump

2009-07-11T19:40:52+00:00

Lately, I have a lot going on. I think I need to recalibrate my "very busy" alert, because I feel like it's been going off for months, now. Still, things are okay. Here's a bit of a dump on some things I'm working on or should be working on.

Data::GUID

I'm definitely behind schedule on getting Data::GUID refactored to allow different generators and objects, which will allow a pure-perl backend (currently UUID::Generator::PurePerl) when XS isn't available. This is mostly done work, but it needs testing, polish, and some design thinking.

Email::MIME::Kit

I'm really not happy with the way the standard Assembler class works. It does too much and is too hard to extend without reading its source. I need to refactor it, document how it works, and maybe break it into a few pieces. So far, I've avoided this by writing all my extensions as around method modifiers. This has been quite effective, actually.

META.json

I started to do some work on PAUSE to get it to respect META.json, but I need to work on getting some things integrated, and other things coded and committed. Once that's done, the indexer will be able to index based on META.json, which is huge. Next up will be CPAN.pm and CPANPLUS.pm, which I hope will not be to hard to work with. Once those parts of the toolchain can all consume META.json, I can start working on getting my hacks to emit META.json into all the dist building tools -- preferably after making them not be hacks.

At some point in all this, we'll get to "make JSON.pm part of core perl5."

Other Email Stuff

I really need to fix Email::Valid's tests to skip the DNS tests when coping with obnoxious ISPs that provide "helpful" lies to DNS resolvers. will-never-exist.pobox.com does not exist, and if there's an A record for it, then the DNS resolver is lying. (Of course, this also means that anyone using the "domain part must have A or MX record" rule is going to have false positives on validity checks.)

Honestly, I can't think of any more irritating change in basic ISP policies in the last few years.

There are some other bugs with Email::Valid, too, regarding how it uses Mail::Address. I'm pondering writing a new Email::Valid that does all the same stuff, a little better, and with a less unusual interface. It would be nice to be able to easily subclass it to add more checks, like is_address_in_use or contains_forbidden_characters for application-specific use.

Heck, maybe I can use Classifier for this, since the new bounce parser is still on the back burner, four years later.

Pod::Weaver and Friends

Basically, I've decided I don't have the time to think about them this week. This is disappointing, but I'd rather put it off a week than make bad decisions because I'm brain-fried.

OSCON starts next week, and I'm hoping to have plenty of down time to work on ideas and implementations. OSCON is generally, for me, much lower-stress than YAPC. I think I will get a lot of hacking done, and if I don't, I will get a lot of flowcharting done, and for me that's half the battle.

Finally...

I've also been thinking a lot about structured data storage, ultra-easy git helpers for small companies, portable testing, pluggable testing, and Rx. I keep improving our little git scripts to keep track of our internal git repos, and I think at some point I might have something I can describe as a reproduceable ecosystem. Rx really needs me to finish the structured failures branch, which will be phenomenally useful. A lot of that work will be related to the testing stuff I'm thinking about. (I would love to write up a comprehensive appraisal of Ingy's TestML, which I talked to him about quite a bit during his development of it.) The structured wiki stuff is largely about my desire to use something better than Obsidian Portal for my D&D game. I should probably write up both what I want and why I don't care much for Obsidian Portal!

For now, this has just been an exercise in collecting my own thoughts. Maybe tonight or tomorrow I'll actually do a little work on some code. Then again, maybe I'll do some yard work instead.

Help needed from Germans and Solaris users

2009-07-11T16:19:39+00:00

In taking over IPC::Run, my goal was not to make it a better module, refactor it, or otherwise dramatically improve it. I'm just trying to fix the module packaging, and get it passing tests (fixing either specific bugs, or correcting the tests).

I've now managed to flush out all the Windows problems, and all the bugs on platforms I can replicate.

Now I'm stuck, and can't make any further progress. I don't have access to the platforms on which the problems occur.

So I'd like to ask for any volunteers that can help fix the two big failure cases.

The first is a bug specifically related to Solaris. You can see details of the failure here.

http://www.nntp.perl.org/group/perl.cpan.testers/2009/07/msg4535302.html

The second is something related to locale, with all the current failure reports occurring on Linux machines set to the German UTF8 locale.

http://www.nntp.perl.org/group/perl.cpan.testers/2009/07/msg4534455.html.

RT has bugs reported properly for the latter one, but I can't fix it.

What I need for both these bugs is either a patch for the code to fix the bug, or a patch for the tests if you can confirm the test is invalid.

Or even better, I'm happy to hand out commit rights for people to apply the fixes directly.

If I were (pump)king

2009-07-11T15:15:00+00:00

It’s fun to speculate “what if” on all the radical changes I’d want to make, but really, if I suddenly became pumpking, my first thought would probably be “Oh, sh*t… I better not f*ck this up!”

That’s right. All my JFDI inclinations would probably change overnight into DFIU.

What If I released a horribly broken Perl? Do I want to be known as “worst pumpking EVAR!”? Do I want that hanging over my head for all time? Do I want to go to a Perl event and be pointed out to newbies as “the guy who screwed up Perl”? Do I want it on Wikipedia or Yahoo or Google when I apply for jobs? Hell, no!

If I were pumpking, I’d be so conservative I’d make Newt Gingritch look like the mayor of San Francisco.

But I’m not pumpking and so I advocate for change and faster progress. Still, I have a lot of empathy for the reality of the situation that pumpkings are in.¹ I think it’s a situation that is probably unavoidable when one individual signs up to be accountable for the success or failure of a large, public project.

I think the answer — to the extent there is one — is that Perl needs to find ways to spread the potential blame beyond a single figurehead and also lower the public penalty for failure.

That means fragmenting accountabilities in constructive ways that don’t jeopardize overall progress. It means having more explicit sandboxes for experimentation. It means taking smaller steps. It means having better risk management. And, frankly, I think it means having better management as a whole.

That’s hard to contemplate for a consensus-driven community of volunteers. But I think it’s a necessary step to achieve what appear to be some consensus goals. I’m not saying we need a constitutional convention (though it’s an interesting thought exercise), but I think the current model passed a tipping point long ago.

In some future posts, I’m going to come back to these themes from a number of angles and try to put some structure around what I see as the fundamental issues and options. While I don’t think there should be changes until 5.10.1 is released, I think it’s wise to start laying out the framework for a constructive dialog in the future.

Volunteered to be in, no less

Padre standalone installer for Windows - first beta version

2009-07-11T12:21:15+00:00

Curtis Jewell has already announced it on his own blog but as he said his blog is hardly read and is not included in the various Perl blog aggregators I promised to also write about it and link to his post.

so the first beta version of the stand-alone Padre msi installer is ready. It still is only an enlarged version of Strawberry Perl so it install in c:\strawberry but it is a huge step in the direction of making it easily available to the general public of Windows users.

So if you have a windows box, go ahead, download Padre and tell us about it via the standard channels of Padre, the Perl IDE

$me->sleep(604800)

2009-07-10T20:07:01+00:00

I'm off to Crete for one week of doing nothing (besides sleeping, swimming, diving & eating). I have some vague plans on working on the slides for my YAPC talk, but as I'm traveling without laptop, I'm not sure I'll actually do anything...

Empty Classes

2009-07-10T14:34:24+00:00

We have many objects with common attributes like titles, synopses, versions, created/modified, etc. Due to business requirements, every exposed object must have a searchable unique ID called a "pid". Thus, we've been forced to do a rather unfortunate denormalization where every pid is placed in an entity table. This table also lists the object types (brands, series, episodes, etc.) and you can look up any pid and get the object it stands for from the appropriate object type. One benefit of this is that it was (relatively) trivial to strip all titles off of objects, put them in a central "title" table, and allow arbitrary title searches and get back all objects, regardless of type, which have matching titles.

We're now doing this with versions. We'll probably do this with synopses at some point. Because every object links into the entity table, we have standard searches available for many things. If we were to take this to the logical extreme, we could do something rather interesting. We could create an object called a "preview" (we have no plans to do so), which has titles, synopses, promotions, tags and parent (the thing we're previewing) and do this:

package BBC::ResultSource::Preview; use Moose; extends 'BBC::ResultSource::Role::ResultSource'; with qw( BBC::ResultSource::Role::Titles BBC::ResultSource::Role::Synopses BBC::ResultSource::Role::Promotions BBC::ResultSource::Role::Tags BBC::ResultSource::Role::Parent );

And with that, have a new object. Effectively, you could have a completely empty class aside from the roles being listed. The preview table might only have an id and nothing else. How would we build the XML for this? Pseudo-code:

package BBC::Builder; use Moose; with 'BBC::Builder::Role::DoesBuilder; sub build { my $self = shift; my @build_data foreach my $role ($self->resultsource_roles) { # following a predefined sort order next unless $self->meta->does_role($behavior); push $build_data => $self->build_data_for($role); } return \@build_data; }

No longer would we need to explain how separate objects are built as XML and we'd have greater consistency in our XML building. Of course, this is still "blue sky thinking", but it would tremendously simplify a lot of our code.

(Note: I know I can just ask the metaclass directly what roles that instance does, but I thought the does_role() method call would show readers more clearly what would happen under the hood.)

The strictperl experiment

2009-07-10T12:03:10+00:00

The strictperl experiment by chromatic, and the strict-by-default patch by Steffen Mueller, will help me explain what approaches are right or wrong in dealing with Perl 5 evolutions.

strictperl is, as chromatic puts it, unilaterally enabling strict for all code not run through -e. Unsurprisingly, that breaks a lot of code. And I mean a lot, as in "most". Not only on the DarkPAN (I seldom use strict for scripts of four lines), on the CPAN (I have modules that will break under strictperl), but in the core itself. chromatic mentions Exporter and vars as modules that break under it. Well, of course they break. Their very purpose is to manipulate symbols by name, which is exactly the kind of thing that strict prevents. (Incidentally all code that uses Exporter or vars can't run under strict perl, and I think that's most of the Perl code out there actually.) That is why chromatic added a separate makefile target to build strictperl: if the patch was going in perl itself, perl couldn't be even built! That's how broken it is.

It's certainly interesting to dive in Perl's internals, but to experiment with enabling strict on a global level, a simple source filter would have been sufficient. One could have written one in ten minutes, including time to look up the documentation, and it would have been immediately obvious afterwards why it was a bad idea. Except to chromatic, who still thinks it's (quoting) a feature I believe is worth considering.

So now let's look at Steffen Mueller's solution, which is strictperl done right, and which is already in bleadperl.

Currently, with bleadperl, a simple use 5.11.0 will enable strictures in the current lexical scope, like use strict would do, but implicitly. (It will also enable features, but that's unrelated.) Look:

$ bleadperl -e 'use 5.11.0; print $foo'
Global symbol "$foo" requires explicit package name at -e line 1.
Execution of -e aborted due to compilation errors.

Moreover, like in chromatic's strictperl, a simple -e or -E won't enable them, because strictures are not wanted for one-liners.

That way, we don't break code that doesn't want strictures, (actually we don't break anything at all that doesn't require 5.11.0 already, it's completely backwards compatible), but it's still removing some boilerplate for default strictness if you're requesting a perl recent enough.

Steffen's patch itself is not very recent, but I didn't apply it to bleadperl immediately, because I disagreed with the implementation. As you can see in the sources, it uses Perl_load_module to load the file strict.pm and execute its import() method. That's very slow. I applied it when I got around to make it faster with the next commit, which replaces that method call by just flipping three bits on an internal integer variable.

All this goodness is coming to you in Perl 5.12 when someone will be willing to take the pumpking's hat.

Next time, I'll explain why enabling warnings by default is not a good idea.

The Success of Ubuntu

2009-07-09T21:31:09+00:00

In a previous post I wrote about The Ubuntu Business model and Perl. This is the second part of that post trying to look at what made Ubuntu successful, how can that be mapped to Perl and what can the Perl community learn from there.

The success of Ubuntu

We can understand from the previous post that the Perl community cannot copy the business model of Canonical, the company behind Ubuntu so let's look at how did Ubuntu succeed in becoming one of the leading GNU/Linux distributions in such a short period of time. I searched a bit and found a couple of explanations. I am sure some people will say they are not true or that they are not the reasons for the success of Ubuntu and I am sure there are others who will point at other explanations. So let's take this as my subjective list with my subjective explanations.

I'll go over the points and try to relate them to Perl and the Perl community.

On Ubuntu Innovations the author points to the following reasons:

Simple install
Regular release schedule
Live-CD that you can install from
One application for each purpose
Secure by default
Over 20,000 applications can easily be installed
Include non-free hardware drivers
Made the color brown sexy
Get an Ubuntu CD for FREE
The Ubuntu Community

Simple install: Perl comes built in on almost all the Unix like Operating systems. ActivePerl and now Strawberry Perl provide easy installation on Windows as well. What IMHO Perl is missing is a set of distributions for some some of the operating systems based on the idea of Strawberry Perl. Similar to how ActiveState has distributions to several platforms but with the Strawberry philosophy and with a lot more juice.
Regular release schedule: Here Perl clearly has a problem. I hope it will be fixed now that the development moved to Git. It does not have to have exactly time based nor do the releases need to be earth shattering but a minor update every 3-6 months could help improve both the image and the level of real users testing perl.
Live-CD that you can install from: This is mostly irrelevant as Perl does not replace the Operating system. It can be installed on any major Operating system. Actually the Portable Strawberry that can be installed on a disk-on-key might provide a nice demo-ing kit.
One application for each purpose: That's another problematic area of Perl and more specifically of CPAN. In my opinion people like to have choices but don't want to choose. We should improve the situation as people waste a lot of time searching CPAN and I am sure in many cases find a module that is far from the preferred modules of any of the active CPAN authors. There are many partial projects that are trying to address this problem.
Secure by default: I don't know about any issues in this regard.
Over 20,000 applications can easily be installed: There are 17,000 modules on CPAN. Many of them are easily installable but many others have problems. I think this areas is both a success (as CPAN has so many packages) but also needs improving such as perl version and platform aware installing tools and allowing several possibly incompatible trees of the same distribution 1.x, 2.x etc versions)
Include non-free hardware drivers. That's ok, there are modules on CPAN to many proprietary systems. (e.g DBD::Oracle) What might be interesting is to include them in Strawberry Perl or the other future Linux/Unix Perl
Made the color brown sexy Camel ? Onion ? I don't know what to say.
Get an Ubuntu CD for FREE IMHO that's quite irrelevant to Perl and actually that is one of the only places where Canonical invested money. Though I am not sure that was a large chunk of their investment.
The Ubuntu Community - The Perl community is quite awesome though there are places to improve. Let's discuss this a bit further

Perl Community

The Ubuntu project has written directions on how to behave. While many of the entries are obvious it is worth to take a look at both the Code of Conduct and the Leadership Code of Conduct.. The points are:

Be considerate.
Be respectful.
Be collaborative.
When you disagree, consult others.
When you are unsure, ask for help.
Step down considerately.

I'd especially point out the last entry. We have tons of code out there, application and modules on CPAN and in many other places. Some of this code, actually quite a large part of it is more or less abandoned. The original developer or last maintainer has mostly disappeared and there was no real process of handing over the code to others. There is a process in which others can take over a module even when the author is gone but I think it would be much better if the authors took it as their responsibility to transfer the modules they don't want to maintain any more.

I know the way this usually happens is that the person slowly has less and less time and keeps telling himself, that soon he will return to that code... so its not easy. We as the community should work on it.

7 reasons of In another article called The 7 reasons why Ubuntu is so successful I found the following reasons:

A good start (vision ?)
Easy and straightforward installation
ShipIt
Synaptic
Ubuntu forums/Community
User promotion
Fragmented competitors

with the following comments I would highlight:

Promotion of Ubuntu through media and freeCDs
Synaptic was really impressive, and not having to download package information from the net every time i do a search was good

These are mostly the same reasons as we read earlier. Let me point out a few issues:

For those who don't use Ubuntu or don't know what it is, the Synaptic Package Manager is basically a graphical version of CPAN.pm for all the packages distributed by Ubuntu.

It would be probably better if Perl also had a graphical tool to install CPAN packages. (In Padre we are going to have one.)

User promotion. Perl got into many places by enthusiastic people who started to use it to solve problems. We should help these people more.

So what do you think, what does Perl need to be more successful in companies and how can we achieve that?

Padre 0.39 released

2009-07-09T13:01:53+00:00

I am happy to announce the release of v0.39 of Padre, the Perl IDE.

The list of changes as taken from the Changes file.

Some of the refactoring code was moved to PPIx-EditorTools (MGRIMES)
Detection of Moose attributes in Outline view (JQUELIN)
Detection of MooseX::POE events in Outline view (JQUELIN)
Added keywords to META.yml (via Makefile.PL.) (SHLOMIF)
Bumped the required Test::More version to 0.88 - needed for note(). (SHLOMIF)
Open Selection (ctrl-Shift-O) now displays all the files it finds and lets the user select (SZABGAB)
Eliminate crash when pressing F3/F4 while there are no open files (#421) (SZABGAB)
Enable/Disable Window menu options when there are (no) open files. (#417) (SZABGAB)
For Cut/Copy/Paste/Select All, use the focused textctrl instead of the editor textctrl (RSN)
Autoupgrade ascii files to be utf-8 on save if user types in wide characters (#304) (SZABGAB)
Allow the user to use external (xterm) to run the scrips. (SZABGAB)
Add menu option to show selection as hexa or as decimal. (#36) (SZABGAB)
Switch to Locale::Msgfmt and generate the .mo files at install time (RSN)
Add number of lines to GoTo line dialog (#439) (SZABGAB)

Soon it should arrive to a CPAN mirror near you.

Enjoy and thanks to all the people who put effort in Padre!

The CPAN Top 100 website has been upgraded to CPANDB

2009-07-09T06:21:12+00:00

Completing 2 weeks of fairly heavy refactoring, rewriting and debugging of various toolchain elements, I'm happy to report that I have now removed the (faulty) CPANTS dependency graph from the CPAN Top 100 website and replaced it with the highly accurate CPANDB dependency graph.

This is huge relief for me, because all the highly visible bugs that were preventing the concept from progressing to full maturity.

The change to reliable data has meant a few major changes to the results.

1. The falsely inflated scores go away.

CPANTS seemed to have an issue misattributing modules to the correct distribution they came from. For example, it thinks that DBD::SQLite lives in the DBD-SQLite-Aggregation distribution.

Fixing this has caused a big shuffle at the top. "perl" now sits proudly at the top of the table, all the hardcoded exceptions are gone, the counts have zoomed up 10% as better linking was able to be made, and a very surprising #2 position that I originally thought was a bug is confirmed to be correct.

Yes, a text wrapping module is the most depended on thing in the CPAN! :/

2. Author attribution is now correct.

The CPANTS implementation appears not to understand the concept of changing maintainers. As a result, many CPAN modules that had changed hands were showing as being maintained by the original author.

With this corrected, you should now be able to accurately see who you need to annoy to take over a failing module.

3. DBD::mysql is no longer an OMGIBROKECPAN event.

Granted, the PASS vs FAIL counts for DBD::mysql are horrible. But the graph now does NOT consider the mysql driver to be as important as it previously did, which has dropped the FAILure score for it down to a lower one.

4. A new module appears at the top of the FAIL 100

I don't know where it was hiding (presumably as an embedded copy in someone else's distribution), but IO::Zlib has made a surprise jump to the top of the FAIL 100 table.

Please commence the poking and prodding of the author to take it over or fix it :)

What's next?

With the underlying data now correctly in place, I can start to address the extremely simple nature of the website itself. I wanted to delay any more work on creating new metrics until the current ones were no longer broken.

So hopefully I can now begin adding new Top 100 tables, and start to move the website towards something that will work better as a standalone site.

CPANDB Tricks: Your personal version of the FAIL 100

2009-07-09T01:12:08+00:00

You may have seen my FAIL 100 list of modules, the 100 modules who's failure in CPAN Testers cause the most pain to other people.

Now that I have support for CPAN Testers results in the CPANDB schema, it's now easy to replicate the math I was doing on that website in one SQL script.

Here's a quickie report of all the modules that I maintain that have failures causing problems for other people (which is a pretty decent clue to the order in which I need to fix them).

Please excuse the weird indenting, I have no idea how Slashcode managed to achieve that...

use CPANDB; print map { sprintf("%4s %s\n", $_->[1], $_->[0]) } @{ CPANDB->selectall_arrayref( 'select distribution, (fail + unknown) * volatility as FAILure from distribution where author = "ADAMK" and FAILure > 0 order by FAILure desc' ) } 9460 PPI 4389 DBD-SQLite 3237 IPC-Run 1407 File-HomeDir 484 Test-ClassAPI 396 prefork 345 SMS-Send 334 File-Remove 289 Config-Tiny 240 Class-Autouse 217 File-ShareDir 150 LWP-Online 74 Devel-Dumpvar 73 Test-Script 68 Test-Inline 60 Perl-Squish 58 PITA-POE-SupportServer 53 Module-CGI-Install 38 ORLite-Migrate 37 Module-Extract 36 Module-Inspector 28 Chart-Math-Axis 24 Test-File-Cleaner 20 PITA-XML 15 PITA-Scheme 15 Portable-Dist 14 Xtract 10 CSS-Tiny 10 JSAN-Client 10 PPI-HTML 8 CPAN-Inject 8 File-UserConfig 7 PITA 6 PITA-Test-Dummy-Perl5-Build 6 Perl-Metrics 4 Method-Alias 4 Test-NeedsDisplay 3 Algorithm-Dependency-Source-DBI 3 Archive-Builder 3 Data-Package 3 JavaScript-Librarian 2 CGI-Capture 2 PITA-Image 1 PITA-Test-Dummy-Perl5-MI 1 PITA-Test-Dummy-Perl5-Make 1 POE-Declare 1 Parse-CPAN-MirroredBy 1 Perl-Signature

Promises, promises

2009-07-08T19:23:59+00:00

This seems like such a useful little function. Can you guess what it does and why?

sub promise($) { my $subname = shift; my $package = caller; unless ($package->can('AUTOLOAD') ) { # this should be in import() require Carp; Carp::confess("Package ($package) does not implement AUTOLOAD"); } my $fq_name = "$package\::$subname"; my $autoload = "$package\::AUTOLOAD"; no strict 'refs'; *$fq_name = sub { $$autoload = $fq_name; goto &$autoload; }; }

In fact, unless there's something else which does this and there's a fatal flaw, I think I might just upload this to the CPAN.

CPANDB Tricks: Tarballs that fail under Win32

2009-07-08T18:19:46+00:00

OK, this isn't a CPANDB trick as such.

But the build process in CPANDB::Generator now reports tarballs that fail to extract. The list is just short enough that I can past it below.

Many are the sort of random stuff that you'd expect, but a couple stand out for me, especially INGY/YAML-0.39.tar.gz?

[Thu Jul 9 04:13:51 2009] Generating META.yml Data... Failed: AKSTE/Term-Query-2.0.tar.gz Failed: AQUMSIEH/Tk-ToolBar-0.09.tar.gz Failed: ARYEH/Net-AIM-1.22.tar.gz Failed: ASHLEY/Catalyst-Plugin-ReferingKeyword-0.01.tar.gz Failed: BRTEAM/Batch-Batchrun-1.03.tar.gz Failed: BRUMLEVE/autobless-1.0.1.tar.gz Failed: BRUMLEVE/ddb-1.3.1.tar.gz Failed: BRUMLEVE/vm-1.0.1.tar.gz Failed: BRUMLEVE/wildproto-1.0.1.tar.gz Failed: COG/Acme-AsciiArt2HtmlTable-0.01.tar.gz Failed: DCLEE/ponfish-0.03.tar.gz Failed: DIVEC/Lingua-ZH-HanConvert-0.12.tgz Failed: DMEGG/SGMLSpm-1.03ii.tar.gz Failed: DMR/CookBookB-19960430.tar.gz Failed: DOKSNER/MPE-IMAGE-0.98a.tar.gz Failed: DPATES/IPTables-IPv4-0.97b.tar.gz Failed: DPATES/IPTables-IPv4-0.98.tar.gz Failed: DREWF/Cmenu-1.1.tgz Failed: EDLIU/DBD-NET-0.1.tgz Failed: EDMONSON/Games-Rezrov-0.15.tar.gz Failed: EIM/WWW-SMS/WWW-SMS-0.09.tar.gz Failed: ENOOR/TaskForest/TaskForest-1.33.tar.gz Failed: FCO/perltugues/perltugues-0.15.tar.gz Failed: FLUFFY/Archive-Par-1.01.tar.gz Failed: FLUFFY/Getopt-Plus-0.98.tar.gz Failed: FOOF/libvorbis-perl-0.05.tar.gz Failed: FRAJULAC/Fwctl-0.28.tar.gz Failed: GAM/Test-CheckChanges-0.07.tar.gz Failed: GAVINC/Config-Directory-0.04.tar.gz Failed: GAVINC/File-DirCompare-0.5.tar.gz Failed: GAVINC/URI-Query-0.06.tar.gz Failed: GSCHLOSS/Math-Stat-0.1.tar.gz Failed: GSLONDON/Devel-AutoProfiler-1.200.tar.gz Failed: GSLONDON/Hardware-Vhdl-Parser-0.12.tar.gz Failed: GSLONDON/Parse-Nibbler-1.10.tar.gz Failed: GUIMARD/Lemonldap-NG-Portal-0.87.tar.gz Failed: GYEPI/Ovid-0.12.tar.gz Failed: HARDAKER/Makerelease-0.1.tar.gz Failed: HBIERSMA/MQSeries-1.24.tar.gz Failed: HOCHSTEN/XML-Tape-0.22.tar.gz Failed: INGY/YAML-0.39.tar.gz Failed: IZI/pake-0.21.tar.gz Failed: JEFF/CGISession.tar.gz Failed: JHIVER/MKDoc-Auth-0.5.tar.gz Failed: JOHNSCA/Syntax-Highlight-Perl-1.0.tar.gz Failed: JONO/Statistics-Table-F-0.02.tar.gz Failed: JOSTEN/XML-XSLT-0.20.tar.gz Failed: JOSTEN/xslt-parser-0.13.tar.gz Failed: JPRIT/ObjStore-1.53.tar.gz Failed: JPRIT/ObjStore-1.59.tar.gz Failed: JV/mmds-1.902.tar.gz Failed: KNASSAR/Activator-0.10.tar.gz Failed: LBOTTEL/Apache-FormatLog-0.02.tar.gz Failed: LUKKA/FreeWRL-0.14.tar.gz Failed: MACGYVER/SMTP-Server-1.1.tar.gz Failed: MASACCIO/Finance-Bank-Cahoot-1.04.tar.gz Failed: MCRAWFOR/Solstice-1440.tar.gz Failed: MEYERCJM/Harvey-1.02.1.tar.gz Failed: MICHEL/Business-KontoCheck-2.98.tar.gz Failed: MIGO/Games-Checkers-0.1.0.tar.gz Failed: MIKEDLR/Link_Controller-0.037.tar.gz Failed: MLFISHER/pmtools-1.10.tar.gz Failed: MPG/POD-Credentials-0.04.tgz Failed: MRDVT/GPS-PRN-0.05.tar.gz Failed: MSILVA/Language-Tea-0.03.tar.gz Failed: MSIMERSON/Mail-Toaster-5.09.tar.gz Failed: PAULG/Logfile-Radius-1.14.tar.gz Failed: PAYERLE/Net-Hesiod-1.11.tar.gz Failed: PERFSONAR/perfSONAR_PS-Base-0.08.tar.gz Failed: PERFSONAR/perfSONAR_PS-Base-0.092.tar.gz Failed: PERFSONAR/perfSONAR_PS-Client-Echo-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-Client-LS-Remote-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-DB-File-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-DB-RRD-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-Services-PingER-0.08.tar.gz Failed: PERFSONAR/perfSONAR_PS-Services-PingER-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-Status-Common-0.09.tar.gz Failed: PERFSONAR/perfSONAR_PS-Topology-Common-0.09.tar.gz Failed: PHENSON/DFS-Perl-0.50.tar.gz Failed: PLIAM/Apache-SecSess-0.09.tgz Failed: PSIONIC/tic/tic.tar.gz Failed: RFOLEY/File-Content-0.01.tar.gz Failed: RSAVAGE/Module-Metadata-Changes-1.04.tgz Failed: SAMPO/zxid-0.27.tgz Failed: SARTAK/Bot-Applebot-0.01.tar.gz Failed: SIMONW/Tanker-0.021.tar.gz Failed: SJSZ/BSD-arc4random-1.3.tar.gz Failed: STBEY/Bit-ShiftReg-2.0.tar.gz Failed: STBEY/Graph-Kruskal-2.0.tar.gz Failed: STBEY/Math-MatrixBool-5.7.tar.gz Failed: TURNERJW/Tk-JFileDialog-1.4.tar.gz Failed: TURNERJW/Tk-TextHighlight-1.0.4.tar.gz Failed: WARD/WeSQL-0.53.tar.gz Failed: WILL/Array-Sort-0.02.tar.gz Failed: WILL/Math-Systems-0.01.tar.gz Failed: WPMOORE/AFS-Command-1.7.tar.gz Failed: YAK/Time-Compare-0.01b.tar.gz Failed: YEWENBIN/Emacs-PDE-0.2.16.tar.gz Failed: YUMPY/Shell-POSIX-Select-0.05.tar.gz Failed: ZAG/HTML-WebDAO-0.91.tar.gz Failed: ZAG/WebDAO-1.02.tar.gz Failed: ZAG/XML-ExtOn-0.09.tar.gz Failed: ZAG/XML-Handler-ExtOn-0.06.tar.gz

Complexity Management

2009-07-08T16:08:10+00:00

For some reason, I tend to get hired to work on large code bases, even though I've met many people who seem to have those perfect memories and instant analytic abilities which make them a natural for said code. Ultimately, as your code grows, complexity management skills become extremely important. This is why I have a number of modules I've written or contributed to heavily in order to make this easier:

Test::Harness: Complexity mangement starts with tests
Class::Sniff: When your OO hierarchy becomes unwieldy, take a closer look at it.
Class::Trait: Traits (roles) usually provide better OO complexity management than inheritance. (I should deprecate this in favor or Moose::Role)
Devel::Deprecate: You're going to forget to remove that out-of-date feature. This module won't let you forget.
Test::Most and Test::Kit: Stop duplicating test boilerplate in your tests and just get your work done.

Seriously, folks. As you move through your programming career, start paying attention to complexity management. When something is overly complex, figure out how to automate or simplify it. You'll usually gain flexibility and comprehensibility. Complexity management is your real job.

The Corporate CPAN II

2009-07-08T09:54:33+00:00

More than a month ago I wrote about The Corporate CPAN and posted the question also on the CPAN.pm developer list. I got a few responses that are still waiting in my inbox to act on. I don't see that I'll have the tuits for that any time soon so I better share with you the answers as they were.

Ricardo SIGNES also mentioned he is working on some internal system with HDP but did not point to any publicly available code.

Andreas J. Koenig pointed me to the doc/README file of the PAUSE source code on Github. We could look around there but he does not plan to release it to CPAN.

brian d foy pointed me to his work on MyCPAN. See MyCPAN::App::DPAN and MyCPAN::Indexer.

I know it's not much of an analysis on my side but at least now I can file the messages in another folder...

PL_runops

2009-07-08T02:57:31+00:00

I'm going to try to do a series of posts about learning Perl internals. I am still a beginner, I have trouble remembering the many macros, or keeping everything in my head all at once, so hopefully I will be able to make some sense of this stuff in a way that is accessible to other beginners.

While I'm definitely diving right in to the deep end, I think PL_runops is a good place to start as any, there's not a lot you need to learn to see how it works. I don't think Perl has a shallow end.

PL_runops is a variable in the interpreter containing a function pointer, which in most cases will be Perl_runops_standard.

Perl_runops_standard is the function that executes opcodes in a loop. Here's how it's defined:

int
Perl_runops_standard(pTHX)
{
    dVAR;
    while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        PERL_ASYNC_CHECK();
    }

    TAINT_NOT;
    return 0;
}

Perl makes extensive use of macros, which can sometimes be confusing, but in this instance it's not too daunting. This loop will essentially keep calling PL_op->op_ppaddr, assigning the result to PL_op. As long as a valid value is returned it will keep executing code.

Just to get it out of the way, PERL_ASYNC_CHEK is a macro that checks to see if any signals were delivered to the process, and invokes the handlers in %SIG if necessary.

So what's PL_op? It's essentially Perl's instruction pointer, it refers to the opcode currently being executed. When Perl compiles source code it produces an optree. This is probably one of the more complicated structure in Perl internals, but right now we're only concerned with one small part, the op_ppaddr field of a single node in the tree. The op_ppaddr field contains a pointer to the function that implements the op.

PP stands for push/pop, which means that it's a function that operates in the context of the Perl stack, pushing and popping items as necessary to do its work, and returns the next opcode to execute.

pp.h defines a PP macro, which sets up a signature for a function that returns an op pointer. Let's have a look at two simple PP functions. First is pp_const:

PP(pp_const)
{
    dVAR;
    dSP;
    XPUSHs(cSVOP_sv);
    RETURN;
}

This is an implementation of the const op, which pushes the value of a literal constant to the stack. The cSVOP_sv macro is used to get the actual SV (scalar value structure) of the constant's value from the optree. SVOP is an OP structure that contains an SV value. The c stands for "current".

Let's rewrite the body of the macro using some temporary values:

/* the current opcode is op_const, so the op structure is an SVOP
 * Instead of using the long chain of macros we'll put it in a variable */
SVOP *svop = (SVOP *)PL_op; 

/* and then we can simply use the op_sv field, which
 * contains a pointer to a scalar value structure */
SV *sv = svop->op_sv;

This SV is then pushed onto the stack using the XPUSHs macro. The X in XPUSHs means that the stack will be extended if necessary, and the s denotes SV. To read the documentation of these macros, refer to perlapi.

The next thing that executes is the RETURN macro. This macro is defined in pp.h, along with a few others:

#define RETURN          return (PUTBACK, NORMAL)

#define PUTBACK         PL_stack_sp = sp

/* normal means no special control flow */
#define NORMAL          PL_op->op_next

/* we'll use this one later: */
#define RETURNOP(o)     return (PUTBACK, o)

Recall that there are no lists in C. The comma operator in C executes its left side, then its right, and returns that value (unfortunately we have that in Perl, too). The RETURN macro therefore desugars to something like:

PL_stack_sp = sp;
return(PL_op->op_next);

The XPUSHs macro manipulated the local copy of the pointer to the stack, sp as it was adding our SV. This change is not immediately written to the actual stack pointer, PL_stack_sp . The PUTBACK macro sets the "real" stack to the version we've manipulated in the body of our opcode.

Then the opcode simply returns PL_op->op_next. The op_next field in the op contains a pointer to the next op that should be executed. In this case if the code being executed was:

my $x = 42;

then the const op compiled to handle the 42 literal would have pushed an SV containing the integer 42 onto the stack, and the op_next in this case is the assignment operator, which will actually use the value.

So, to recap, when PL_op contains a pointer to this const op, PL_op->op_ppaddr will contain a pointer to pp_const. PL_runops will call that function, which in turn will push ((SVOP *)PL_op)->op_sv onto the stack, update PL_stack_sp, and return PL_op->op_next.

At this point runops_standard will assign that value to PL_op, and then invoke the op_ppaddr of the next opcode (the assignment op).

So far so good?

To spice things up a bit, here's the implementation of logical or (||):

PP(pp_or)
{
    dVAR; dSP;
    if (SvTRUE(TOPs))
        RETURN;
    else {
        if (PL_op->op_type == OP_OR)
            --SP;
        RETURNOP(cLOGOP->op_other);
    }
}

The or op is of a different type than the const op. Instead of SVOP it's a LOGOP, and it doesn't have an op_sv but instead it has an op_other which contains a pointer to a different branch in the optree.

When pp_or is executed it will look at the value at the top of the stack using the TOPs macro, and check if it evaluates to a true value using the SvTRUE macro.

If that's the case it short circuits to op_next using the RETURN macro, but if it's false it needs to evaluate its right argument.

--SP is used to throw away the argument (so that $a || $b doesn't end up returning both $a and $b).

Then the RETURNOP macro is used to call PUTBACK, and to return PL_op's op_other. RETURN is essentialy the same as RETURNOP(NORMAL). op_other contains a pointer to the op implementing the right branch of the ||, whereas op_next is the op that will use the value of the || expression.

This is one of the most basic parts of the Perl 5 virtual machine. It's a stack based machine that roughly follows the threaded code model for its intermediate code structures.

The data types mostly revolve around the SV data structure, and moving pointers to SVs from op to op using the stack.

For me the hardest part to learn in Perl is definitely the rich set of macros, which are almost a language in their own right.

If you search for runops on the CPAN you will find a number of interesting modules that assign to PL_runops at compile time, overriding the way opcodes are dispatched.

For more information on Perl internals, the best place to start is perlguts, and the wonderful perlguts illustrated.

CPANDB Tricks: Smoking the Top 100

2009-07-08T01:03:23+00:00

David Golden writes:

I'd like to be able to easily smoke test at least the volatile 100
list for the 5.10.1 RC's. Is there an easy way I can get a list in
the form:
DAGOLDEN/Sub-Uplevel-0.2002.tar.gz
Just a text file would be fine, or JSON or whatever. Just something
with AUTHOR/DISTNAME-VERSION.SUFFIX.

The Volatile 100 is my replacement for the Phalanx 100. Where the Phalanx 100 represented major functional modules that people might depend on directly the most and was curated by humans, the Volatile 100 in a graph-weighted list of the 100 most depended on distributions and is continuously updated automatically.

Finding information of the kind David needs is exactly why I created the CPANDB module. His query can be boiled down to just a single line of code.

use CPANDB; my @releases = map { $_->release } CPANDB::Distribution->select('order by volatility desc limit 100');

Saudi Arabian Adventures - Days 3-5

2009-07-08T00:00:00+00:00

These events happened during my recent trip to Saudi Arabia.

Saudi Arabian Adventures - Days 3-4
On the third day of teaching my class swapped over to new students at lunchtime. Included in this group was Abdulaziz, who I had spoken to on the phone a few times, and who formally introduced me to the class on the first day. Abdulaziz spoke English with an American accent, and I was later to learn that he gone to college in America.

Lunch on the second last day Abdulaziz took me to the golf course. Saudi Aramco has a huge golf course, with beautifully kept grass, and is in stark contrast to the surrounding desert. Here they serve a variety of meals, cooked to order. Since it was the middle of a business day, nobody was actually playing golf.

There have been a few things that I've noticed over here. Business attire is very much traditional western or traditional Arabic, with both being very common. Teenage girls can be often seen wearing flowing black over-robes that were very Harry Potter-esque. Lawns and garden beds are covered with a huge number of very fine mist sprinklers, and they run during the day. These sorts of sprinklers are practically outlawed in Australia, with most states suffering from severe water shortages.

At the food hall, I learnt an important lesson. If I was asked if I wanted something to "take away", then that actually meant the food serving would feed a football team, and my answer to such questions should always be in the affirmative. The first time I foolishly said no, and found myself wondering what to do with the incredible amount of food in front of me.

Having wireless access in my room is a real blessing. I don't know where I'd get it otherwise.

Saudi Arabian Adventures - Day 5
Today was my second-last day of teaching, and Abdulaziz informed me that I needed to make sure the class finished early (3pm) as a meeting had been arranged between myself and management.

To make my day more interesting, we had a trainer who wanted to install software for her course next week. Her name was Tina, and her opening line was "Hi, you're a software developer, aren't you?" When I asked how she knew, she claimed "It's the look. You're all trendy, with long hair." That's the first time I've ever heard developers get called trendy, so I can only assume that I mis-heard.

At 3pm we finished early so that I could meet with EDMD management. For some reason, whenever anyone said "EDMD" I would hear "AD&D", my mind would fill with images of dungeon-crawling geologists in flowing black robes fighting an onslaught of umber hulks.

The meeting with management was made more interesting because I didn't have my Aramco ID, which meant Abdulaziz had to make a couple of bluff checks at the security checkpoints to get me through.

The meeting itself was very management-focused. What does Perl do, who uses it, what's its future, and so on? I honestly don't know how I fared with these questions, and in hindsight I fear my answers may have been a little too much on the technical side, but everyone seemed happy.

Afterwards I met Omar, who I had spoken to on the phone a number of times before my trip, but who wasn't enrolled in the course. Omar is a dual-classed geo/developer, and already knows Perl. It also rapidly became apparent that he was very open-source friendly. "We have a program that parses $horrible_file_format, and would like to bundle it into a module and release it to the CPAN. Is that something you can help with?"

That night, Fuad, Abdulaziz, Omar and Mohammed took me out to a traditional Arabic restaurant. The trip getting there was an adventure in itself. Saudi Arabia seemed to lack anything resembling road rules, or if they existed, they were ignored by most of the drivers on the road. Lane-changes happened at random, and cars would speed down the road at breakneck velocity. The primary form of communication between drivers was a brief flash of high-beams, which meant "I am going 40km/hr faster than you, and I'm not changing lanes. You should." This was in stark contrast to driving inside the Aramco campus, where everyone was slow, careful, and polite. Somehow, Omar drove through all this without breaking a sweat or losing his cool.

In the car we talked about content management systems, web technologies, AJAX, Catalyst, Jifty, PHP, package management and deployment, virtual worlds, and a variety of other FOSS-oriented technical subjects. This was good, as it distracted me from the traffic around us.

At the restaurant I was treated to Arabic coffee, which apparently is made from the unroasted beans, and is very good. It also rapidly became apparent that this was a restaurant/museum, with many historical photos and items on the upper floors. There was a large water feature in the middle of the building, and a large lantern feature. It was explained to me that these sorts of lanterns were traditionally lit and placed outside houses at the start of the month of Ramadan.

After a tour of the historical section, we were given quite a banquet of food, some items which I have had before (such as the dips), and some which I had not. One of the most memorable (and most tasty) dishes was a thick paste made of a variety of grains. This was eaten with a thin broth, and stuck to one's teeth almost immediately.

After dinner, and tea, and dates, and more talking, we drove back to the camp, with discussions mostly focusing around virtual worlds, 3D web interfaces, and tunnelling procedure calls from Second Life to Perl.

Sleep was an immediate priority when I returned to my room, as it had become quite late, and I was certain I'd need an extra cup of coffee to get me up in the morning.

Posted: 8th July 2009.

Tags: adventure aramco dhaharan food perl saudi travel

Bookmark:

Introducing CPANHQ

2009-07-07T17:22:02+00:00

By Shlomi Fish

Most people who are actively developing using Perl are familiar with sites for searching CPAN (the Comprehensive Perl Archive Network) such as search.cpan.org and kobesearch.cpan.org. These sites provide a simple search bar where users can search for keywords or key phrases, and find a list of results. These sites also feature pages for the various CPAN distributions, the various authors, and the individual pages of documentation inside the modules.

Recently, Andy Lester started the "Rethinking CPAN" mailing list and wrote a Perlbuzz post titled "Rethinking the interface to CPAN", where he gave the case that the current resources may be inadequate for the process of selecting a module on CPAN. For a long time, there was an intermittent discussion and some brain-storming, but very little code (if any) has been written. This has all changed with CPANHQ.

So what is CPANHQ? As documented in the first and only wiki page it has so far (which I wrote a few days ago), "CPANHQ aims to be a community-driven, meta-data-enhanced alternative to such sites as http://search.cpan.org/ and http://kobesearch.cpan.org/. Currently, the functionality is very basic, but we have great ambitions."

I learned about CPANHQ by overhearing a conversation on Freenode's #perl channel. I joined the effort, and made several enhancements to the Catalyst and DBIx-Class-based code. The functionality is still very basic.

Some of the plans and ideas we have for CPANHQ, which may be translated into code in the future are:

The code is and will remain open-source, which makes it easy for people to contribute to.
We plan to have most of the current features of search.cpan.org and kobesearch.cpan.org including pages for authors, distributions, and individual modules and their documentation.
CPANHQ was already enhanced with some jQuery and jQuery UI JavaScript goodness, and we plan to add more of it to make the user experience better.
The site aims to be more integrated than the many separate CPAN-related sites today: search.cpan.org, CPAN Forum, AnnoCPAN, CPAN Ratings, CPANTS, CPAN Dependencies, etc. By putting everything under one roof, we can provide a better user experience than the many sites with very loose integration.
We plan to enhance the site based on meta-data as provided by both the module authors (in the META.yml file and the users of the site.

For example, we have already added support for the META.yml's keywords which allow authors to tag modules, and we plan to support user-tags too.
CPANHQ will be very community-driven. People would be able to open user accounts there, which will allow them to contribute content to the site, such as tags/labels, comments, wiki pages, chatter, etc.

The CPANHQ wiki page contains more ideas and insights. Like I said, the current functionality is incredibly basic, but we hope that with enough help, time and dedication, we can create an attractive and engaging interface to CPAN, to make finding a module on CPAN faster, easier and more fun.

In the meantime, you can look at the screenshots we have of the interface so far, or clone the github repositories, and set up a local web-interface.

Shlomi Fish has been writing using Perl, about Perl and for Perl, since 1996, when he started working for a web-design shop, and discovered Perl and UNIX, which he's been in love with ever since. He initiated and is maintaining many CPAN modules and contributed to many other Perl and non-Perl related projects.

mod_perlite helps Perl gain ground in the PHP arena

2009-07-07T13:57:04+00:00

Michele Beltrame wrote in about the mod_perlite project, which I'd love to see take off. mod_perl has never been easy to install, and ISPs and webhosts are loathe to use it for security reasons. mod_perlite would provide easy access to Perl without the weight of CGI.

mod_perlite is an interesting Apache module currently under development. I'd say it's the equivalent of what mod_php is for PHP: a fast, easy to use, and relatively safe way to use Perl scripts for the development of small and medium web applications.

A project like this brings Perl back into play as a good choice in a field currently dominated by PHP. In particular, PHP is great when you need to run a lot of different small applications on the same web server, and you want all of them to run fast enough. CGI is slow compared to mod_php because both the interpreter and the application need to be loaded for every single request. Solutions like mod_perl and mod_fastcgi are the best solution when you have a few big applications, but they're not a good choice in this scenario, because they keep the application resident: when you've got so many small applications (which, in most cases, are invoked infrequently) you end up with enormous memory footprint.

mod_perlite is the winner here: only the interpreter is kept resident (there's also the idea to keep some modules loaded, which might speed up things further) and applications are loaded at request time. This also helps avoid problems with memory leaks. Using mod_perlite is simple: just configure Apache and all the .pl files will run without modification taking full advantage of mod_perlite and therefore reducing execution time to a fraction.

Unfortunately the module is not yet stable, as there's still work to do. My installation on Gentoo Linux x86_64 went smoothly. mod_perlite works but still suffers some major issues. In fact, at present it's only usable for very simple applications because of a bug in how query parameters passed by Apache are handled.

To make mod_perlite fully usable, the project web site has a three point TODO list:

Fix form POST processing ('read' does not work reliably)
Limit Perl running time.
Find ways to cache code.

The project is looking for help, so if you are interested and know something about mod_perl, writing Apache modules in C, and about overriding system calls in Perl - and if the project seems interesting to you - please shout!

And... if someone argues "we're 10 years late compared to PHP", I'd reply "better late than never". ;-)

Michele Beltrame lives in North-Eastern Italy, where he has been developing web applications in Perl since 1996. He is an active member of the Italian Perl community and of the Italian Perl Workshop organizational committee. In his free time he publishes guidebooks on the mountains surrounding his area.

Perl 6 Roles Question

2009-07-07T10:31:45+00:00

Imagine a PracticalJoke class. It needs a non-lethal explode() from a Spouse role and a timed fuse() from a Bomb role. Unfortunately, each role provides the both methods. With Moose, this is trivial to resolve:

package PracticalJoke; use Moose; with 'Bomb' => { excludes => 'explode' }; 'Spouse' => { excludes => 'fuse' };

How the heck do I do that with Perl 6? No matter how many times I read Synopsis 14, I can't figure out how that would work.

role Bomb { method fuse () { say '3 .. 2 .. 1 ..' } method explode () { say 'Rock falls. Everybody dies!' } } role Spouse { method fuse () { sleep rand(20); say "Now!" } method explode () { say 'You worthless piece of junk! Why I should ...' } } class PracticalJoke does Bomb does Spouse { ??? }

What have I missed?

Italian Perl Workshop 2009 guests

2009-07-07T09:15:00+00:00

The preparations for this year's Italian Perl workshop are hotting up, and thanks to the sponsors, we have invited some great guest speakers for the international tracks (in English): Tim Bunce, Perl DBI interface author. Tim made not only DBI, but also other important CPAN modules (DBD::Oracle, Devel::NYTProf, responsible for the Perl 5.4.x series), he wrote book published by O'Reilly Media, and he has delivered tutorials and sessions for many years at OSCON and other conferences and workshops. Jonathan Worthington, one of the core developers of the Rakudo Perl 6 compiler and the Parrot Virtual Machine. Thomas Fuchs, a Javascript guru, is the author of the script.aculo.us user interface JavaScript library, a member of the Prototype core team and a Ruby on Rails core alumnus. Amy Hoy, is a user interaction and product design expert and regular speaker at OSCON and other workshop and conferences.

Iron Man ideas backlog

2009-07-07T08:55:49+00:00

There are so many topics I want to write about and so little time to write. Since I’m on vacation and supposedly relaxing and enjoying myself, I’m just going to list out some things that I might write about soon. If any are particularly appealing, let me know and I’ll bump it towards the top of the todo stack.

Capture::Tiny, the last capturing module you’ll ever need
Version numbering lessons and pitfalls
Re-imagining PAUSE/CPAN
Perl’s punctuated equilibrium
Disagreeing agreeably
p5p: Wizards, leaders, managers and coaches
How to regression test CPAN modules
Changing Perl 5 development: to what end?

I can see the appeal of microblogging — perhaps I’ll try to compose a 140 character summary of each one and see which are actually worth expanding on to any length.

Open source ‘full text’ bookmarklet and feed filter

2009-07-06T17:01:26+00:00

Last year, I blogged about Full-Text RSS, a utility to convert those useless “partial-text” RSS/Atom feeds into the real, full-story-inline deal.

The only downside is that the author felt it necessary to withhold the source, saying:

Still, I wouldn’t want to offer a feature that middlemen can resell at the expense of bloggers. So while I do want to open this up, I don’t want to make things easy for the unscrupulous.

However, recently Keyvan Minoukadeh from the Five Filters project got in touch to say:

I recently created a similar service (along with a bookmarklet for it). [...] It’s a free software (open source) project so code is also available.

Here it is:

fivefilters.org: Create Full-Text Feeds

I’ve tried it out and it works great, and the source is indeed downloadable under the AGPL.

Five Filters — its overarching project — looks interesting, too:

Edward Herman and Noam Chomsky describe the media as businesses which sell a product (readers) to other businesses (advertisers). In their propaganda model of the media they point to five ‘filters’ which determine what we read in the newspapers and see on the television. These filters produce a very narrow view of the world that is in line with government policy and business interests.

In this project we try to encourage readers to explore the world of non-corporate online news, websites which avoid the five filters of the propaganda model. We also try to make these sources of news more accessible by allowing users to print the stories found on these alternative news sites in the format of a newspaper.

Holy crap, I suck at coding

2009-07-06T16:00:18+00:00

Every so often I look back at old code and think to myself WTF was I on when I was writing it. Just take a look at my Perl module releases in chronological order^[1].

Usually it's small things I find annoying, when just looking at the code it with fresh eyes is enough to unveil small idiocies. It's easy to judge because I don't need to sugar coat any criticism I have when I'm the one under the lens. This stuff is usually the result of refactoring too quickly, and not rounding off all the corners, especially if the code I was refactoring was new code.

More embarrassing is when I find flaws in my assumptions or thought process: things that are more complicated than they should be, or inability to separate an idea from the engineering context it emerged from. This is usually the result of rushing through something (often towards the end of a project) and not thinking things through, being ignorant of a better approach, or just plain old lack of insight.

This process used to make me feel bad, but now it makes me feel good. I used to be concerned with how fast I learned things, eager to understand ideas that would propel me towards finding the ultimate paradigm [singular. ugh!] of the future.

When your goals are as laughable as that failure makes you feel bad.

I think now I'm much more relaxed about everything. I'm not looking for a silver bullet at all anymore. I'm looking for fun ideas to "spice up my routine" as they say. If I remain interested I usually produce dramatically better results than if I need to force myself to work.

Leisure has replaced ambition as the primary purpose of my learning efforts.

However, without ambition to drive me forward, I occasionally feel like I'm not trying hard enough to better myself. Seeing past failures (especially recent ones) is an great way of reassuring myself that my brain has not in fact crawled under a cupboard and died.

Recognition of past failure was always evidence that I am still learning, but in the past I thought it meant that I was not yet any good. Now it just means that I'm still having enough fun to stay curious, and I'm also learning what I should be unlearning.

So now that I am old and wise, I know that when I grow up I want to still be having fun, or at least to realize why this blog post is full of shit.

[1] Unfortunately the backpan directory listing doesn't seem to have the dates right, so the linked listing doesn't contain mistakes that have been superseded with new module uploads, only ones big enough to completely give up on, or that have been fixed by other people ;-)

Resigning

2009-07-06T15:33:05+00:00

So I'm resigning from my role of Perl 5 pumpking. That doesn't mean that I'm not proud of what I did for Perl 5 in the past, or that I don't stand behind my choices anymore.

Let's begin by inspecting some of the core ideas that drove the chromatic rants since the last six months, and examine why they are a sure recipe for failure.

First, the regular snapshots, labeled releases. When you cannot control how much development time you'll have for the next release, nor the list of hot topics that the contributors will volunteer to contribute to, you don't release easily; at the best, you snapshot. Which is fine for alpha-grade software as parrot, as I've already said; but not for production-ready software. Did I point out that Perl was a very complex project with a lot of interdependent parts? Care is essential.

(As a side-effect, it might quite be easy to predict the release dates of parrot for the next century, but not the date where it will be used in production. If that happens at all, that is.)

Secondly, a lack of feature plan for the next versions of Perl. Again, this is not something that you can control in a volunteer-based project. I've said many times that I'd like 5.12 to solve some inconsistencies on the handling of UTF-8 strings with regard to case-insensitivity comparisons and case-changing operators. That would be a backwards incompatible change and would break code -- contrary to the gross mischaracterisations that chromatic presents as truth, P5P is certainly not opposed to backwards incompatibility. But to achieve that, you need a regex + UTF-8 guru, or somebody willing to invest countless hours into becoming one. If someone comes with another itch to scratch, sends patches, implements a neat new feature, then that would warrant a shiny 5.12 without the UTF-8 revamp.

Thirdly, the "untested code is not worth caring about" fallacy. One can try to wave away the DarkPAN into oblivion with a blog post, but if Perl still exists for more important purposes than the amusement of a few computer language geeks, it's because of the DarkPAN. And not all the DarkPAN code is regression-tested, or even testable. Quick glue scripts, crontabbed email reports, network management helpers, post-commit hooks: they cost too much to test, because the environment they run in is orders of magnitude more complex than their internal logic. I've code out there in the DarkPAN that deals with the relative replication delays on two pairs of master and slaves databases. That's not testable. But this code is critical.

Fourthly, the main argument chromatic has to back up his proposals for Perl 5 is "we should change defaults and add syntactic sugar without thinking about the consequences because that will allow me to rip off three lines of boilerplate code." The hints I gave at the time on P5P that maybe adding syntax would be a better idea if there was some semantics behind it were then aggressively relabelled as reactionary. (As a coincidence, modernperlbooks.com started its anti-P5P propaganda shortly afterwards.)

At the contrary, every language extension that went into 5.10 was there because it solved an actual problem, in a more efficient way than the CPAN could provide, and sufficiently thought out (I hope) to avoid being sorry having to maintain it in a couple of versions from now. (But now I'm not so sure about UNIVERSAL::DOES() anymore.)

Fifthly, the way chromatic has arbitrary chosen one particular regression from 5.8 to 5.10 and presented it as if it was as serious as a remote root zero-day vulnerability, willingly ignoring every other regression (or improvement) in the hope of making a point. Note, that regression wasn't even a bug in the language, something that could have made Perl programs misbehave or segfault. It was a performance regression. In a world where most programs are I/O bound anyway. And there were many other more interesting regressions to choose from, I won't hide that. So why this one? Certainly because of its marketing value. It was certainly sexier than a more severe regression on an obscure feature he would have needed to explain to his readers.

At the end, I've had enough of those gratuitous attacks. I've always accepted and even encouraged criticism about my decisions as a pumpking, but only as long as there were based on technical arguments, not on marketing slogans repeated hysterically by someone who remains deaf to any form of discussion. So I'm stepping down from my role of pumpking. I'm burnt out. I don't want to have to justify myself again and again in front of all this.

Moreover, if my resignation can help Perl 5, it will be for the greater good. There are many committers and knoledgeable contributors, and they'll probably start reviewing the patches to apply a bit more: avoiding bottlenecks is good. The release process will be documented and distributed (and thanks to Dave Mitchell for having worked a lot on this) and the whole bus factor of Perl 5 will go up. Don't worry, I'll still be around to ensure that the future of Perl 5 is not handed to the marketroids, and to produce the occasional patch. But I'm withdrawing from the front line. Which will have, I'm sure, positive effects in the end.

Reputations

2009-07-06T14:29:50+00:00

As many of you know, Rafael has stepped down as the P5P pumpking. He's done a great job and I'm sad to see him go, but he couldn't take the mud-flinging any more. I can't say I blame him.

Regarding mud-flinging -- and not just in regards to this particular debacle -- I won't name names (many will guess and many guesses will be wrong), but I have some observations about reputation:

I've been quietly told by several that they're disgusted with one particular individual who should know better than to be rude.
I know of at least one individual who has, unbeknownst to him, not been offered a rather nice consulting gig because of his online behavior.
There is one person I once objected to hiring because of comments he made on a mailing list (the owner saw those comments and rejected the applicant).

I suppose if people aren't concerned about work, they may not be concerned with how they come across online. On the other hand, wouldn't it at least be good to be known as someone who can find ways of getting things done nicely? I struggle with this myself at times and I hate seeing it in others.

Perlbuzz is no longer useful

2009-07-06T08:49:44+00:00

I've been since a long time disappointed at perlbuzz, but here's another blatant proof of their lack of professionalism. This supposed "news roundup" post points to several articles by chromatic and his fanboys, and supplement them by pure FUD, like the retransmission of a twitter message saying "Is getting code into the !perl core a fight against inertia and petty hostility?", linking to an anonymous comment on a random web forum. If this is not calumny, I don't know what it is. It's just unfair to see perlbuzz relaying it without checking.

However, I note that none of the rebuttals written by me or by others elsewhere have deserved a mention.

Someone is actively trying to damage the community image of P5P out there, and perlbuzz is helping him. Willingly?

CPANDB should now be synchronised and up to date

2009-07-06T06:32:16+00:00

Update:

The CPANDB now also contains the data for CPAN Ratings

In my previous announcements for CPANDB, I mentioned that while it was generating the database the correct way, the data was going to be a bit out of sync because data was coming from different places.

This should now be resolved.

The server syncs a minicpan to http://cpan.cpantesters.org/, then a META.yml database is updated from the minicpan, a CPAN::SQLite index is built from the same minicpan, and then both of them are joined together with the ORDB::CPANUploads database to product the completed tables.

The final step is the application of the graph algorithms to produce the weight and volatility scores for each distribution, which are now being added to the database as well.

Now that cpandb is ready for the big time, I'll start to try and merge in the three main outstanding data sets (CPAN Ratings, rt.cpan.org and CPAN Testers) and then rewrite the CPAN Top 100 to turn it into a fully operational death star... I mean website.

You can use the CPANDB module to fetch and ORM-inflate the database, or to download the database directly, use the following URL.

http://svn.ali.as/db/cpandb.gz

The current schema for the database now has the slightly expanded distribution table.

CREATE TABLE distribution ( distribution TEXT NOT NULL PRIMARY KEY, version TEXT NULL, author TEXT NOT NULL, release TEXT NOT NULL, uploaded TEXT NOT NULL, weight INTEGER NOT NULL, volatility INTEGER NOT NULL, FOREIGN KEY ( author ) REFERENCES author ( author ) )

Perlbuzz news roundup 2009-07-05

2009-07-06T02:01:43+00:00

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

OSCON for Postgres geeks (it.toolbox.com)
SpamAssassin looking to drop Perl 5.6 support (taint.org)
chromatic on the value of the pumpking (modernperlbooks.com)
Search your Perl documentation with perldoc-search and/or Perldoc::Search (use.perl.org)
RT @chromatic_x Is getting code into the !perl core a fight against inertia and petty hostility? (ur1.ca)
Saving state with prove (blog.woobling.org)
RT @chromatic_x Why we're still working on Perl 6 after all these years (bit.ly)
Chris Prather's YAPC::NA slides available (chris.prather.org)
RT @chromatic_x Why perl5i matters: (bit.ly)
Gabor Szabo on what he'd like to see in Smolder (szabgab.com)
"Warming up to modular testing" slides from YAPC::NA available (use.perl.org)
"Perl does not belong to DarkPAN. It belongs to us who participate in its wellbeing." (perl-yarg.blogspot.com)
Fitting Wikipedia on an iPhone (radar.oreilly.com)
What is Catalyst, really? (blog.urth.org)
Hard numbers on test-driven development (use.perl.org)
PHP does not make programming simpler. It just pretends that programming is simpler than it is. (use.perl.org)
A WebGUI editing plugin for Padre (blog.patspam.com)
Recap of the Portland.PM code sprint get-together (baconandtech.com)
More notes from the Portland PM code sprint (justatheory.com)
Perl Corehackers wiki now online (corehackers.perl.org)
Josh Ben Jore's first Perl 6 program (use.perl.org)

Links for 2009-07-05

2009-07-05T22:05:03+00:00

Inside Postini’s anti-spam systems : lots of detail
(tags: spam google postini anti-spam)
Get Your API Right : 8 key gotchas when implementing RESTful web APIs. great advice
(tags: apis http web rest patterns architecture webdev web-services)

Let me be clear, again, about the nature of the DarkPAN.

2009-07-05T18:07:10+00:00

The growth in the argument that the DarkPAN is somehow some form of boogieman, some wispy vaporous nothing that is unobservable and thus something able to be ignored astonishes me. And I'm not singling out one person here, I'm hearing it from a number of different people.

It's especially unnerving because a big chunk of it is sitting in plain view.

So let me repeat this again.

Until perhaps very recently, the DarkPAN was generally considered to be more or less all the Perl code that wasn't on the CPAN. Because if it wasn't on the CPAN, it was largly impossible to find. And even if you could find it, you couldn't do anything with it.

With the advent of PPI and Ohloh, the situation has now changed.

Now we know, for example, that just the small part of the DarkPAN that is visible is at least 4 times bigger than the CPAN. Because 4 times more code than CPAN is basically what Ohloh managed to uncover. Last time I looked, it was reporting about 97,000,000 lines of code, and we know that the CPAN is in the general range of 20-25,000,000 lines of code.

I've called this semi-visible part of the DarkPAN the GreyPAN, but that's just a way of categorising it. It's still a part of the DarkPAN. It's still code we can't feed into CPAN Testers or other automated systems to examine how a Perl change will impact it.

We also know that it's mostly not large codebases. Anecdotal reports from the Ohloh people themselves suggest it's made up of lots of different bits and pieces. So in fact, it serves as quite a useful analog for the consistency of the DarkPAN itself.

It's something we can suck in, process, and then run what-if scenarios for. At least at a document-level anyway.

Frankly, I'm sick and tired of this back and forth crap.

I'd like to see both sides of this argument shut up (me included) until we've actually built the thing that can suck in the visible parts of the DarkPAN and process it.

If you'd like to help, I've got most of the "Process it" part working now, but I'd dearly love some help on the "sucking it in" part.

If, after we've finished, we're able to establish that (for example) across all C code there is on average 1 line of utility Perl code for every 100 lines of C code then we've got something to work with.

Because we can take estimations about the total amount of C code in the world and extrapolate from that to make a guess at the total amount of Perl code in the world.

Bootstrapping KiokuDB::Cmd

2009-07-05T18:12:54+00:00

I just closed a ticket in KiokuDB's RT queue that asked for simpler dependencies. The fix was splitting the KiokuDB::Cmd modules into their own distribution, making the core more easily installable on Windows.

The problem is that I depended on Proc::InvokeEditor for the kioku edit command. Unfortunately that module does not install cleanly on Windows, preventing KiokuDB itself from installing, even though it's not required for normal use.

The problem is that KiokuDB's command line tool requires KiokuDB::Cmd to run. What I ended up doing is having the kioku script (which still ships with KiokuDB) allow bootstraping KiokuDB::Cmd by using Module::AutoInstall (if it's available and the user grants permission to do so).

If kioku can't require KiokuDB::Cmd cleanly, then it calls try_auto_install function:

sub try_auto_install {
    if (eval {
        require Module::AutoInstall;
        require ExtUtils::MakeMaker;
        1;
    }) {
        print "\n";
        if (
            ExtUtils::MakeMaker::prompt(
                "Module::AutoInstall is available. " .
                "Should I use it to install KiokuDB::Cmd?",
                "N/y",
            ) =~ /^y/i
        ) {
            my @installed = Module::AutoInstall->install(
                [],
                "KiokuDB::Cmd" => $KiokuDB::REQUIRED_CMD_VERSION,
            );
            return @installed > 0;
        }
    }

    return;
}

If installation seems to have succeeded then we delete $INC{"KiokuDB/Cmd.pm"} and try to load it again.

Otherwise the script exits with an error asking the user to install KiokuDB::Cmd manually.

I think this is a fair tradeoff, especially if you don't need the command line tools in production deployments (you can still make backups by programmatically scanning the database yourself).

Time-based releases in open source

2009-07-05T11:01:25+00:00

Whenever I hear someone saying we should have regular releases, I hear we should release when Venus enters Pisces. Because, when you have so many uncontrolled parameters to deal with, sticking to a predefined calendar boils down to superstition.

Time-based releases actually make sense in one case : when you're selling your software. That way, you can ensure a regular stream of revenue via upgrades. Moreover, if you're a commercial entity, you probably are already paying full-time a couple of developers, so you can predict how much time you'll spend on the next version during the next few months.

All those premises are not true for volunteer-based open source projects, such as Perl.

Actually, time-based releases still make sense if the project is in an alpha stage and is being prototyped. In this case, however, it's more accurate to speak about snapshots than releases, because such a "release" is only a way to publicize the newest changelog. That's the case for parrot, for example: it doesn't aim for backwards compatibility, or even stability, and has no user base. That would be also the case for the development branch of Perl 5 (currently 5.11), although nobody got around to do it since the migration to git -- I agree that would be a nice improvement.

In the specific case of Perl 5, even if you wanted to try to have regular releases, more factors would complicate the task. You can't release bleadperl at any point and call it stable, even when all dual-life modules are perfectly in sync with the CPAN and when all tests pass on all platforms and all configurations. That's because you also need, for a new major release, to have a consistent set of features. For example, it would not have made sense to release 5.10.0 with the lexical $_ variable, but without the (_) prototype.

With the upcoming release of 5.10.1 by Dave Mitchell, we're trying to improve the release process to be less of a pain for the pumpkings. Changes to dual-life modules will no longer be accepted in the core unless they are put on CPAN first. Module::Corelist will refer to tags in the git repository instead of perforce change numbers. The perldelta will hopefully be written a bit more incrementally.

I plan to copy 5.10.1's perldelta in bleadperl when it's done, then to supplement it by blead-only changes. Afterwards I'll have more elements to judge whether we're ready for a 5.12.0 or not -- and that decision will be motivated by the contents of the release, not by the time of the year.

Why "Shut Up and Write Code!" is Unhelpful and Wrong

2009-07-04T21:42:39+00:00

The Perl community needs to discuss the DarkPAN and the present and future of the language and its ecosystem in much more productive ways than accusations, hyperbole, and strawmen. This includes the people who want to destroy the DarkPAN as well as the people who believe that its stability must be preserved.

I write provocatively and deliberately. I'm writing a manifesto, after all. Yet I believe I never devolve to name-calling, strawmen arguments, nor misquoting people. (If I do, please tell me and I'll correct it.)

My motives are simple: I want regular releases of Perl 5. I want the default behavior of Perl 5.12 to encourage modern Perl programming practices. I want missing features added to the language, even in small steps at first if necessary. I want Perl 5 to be a better language for creating interesting new programs than for maintaining clunky old programs.

Here are some selected quotes from Rafaël Garcia-Suarez's The DarkPAN matters:

DarkPAN is just a slang word to express the fact that Perl 5 is now currently used all over the world in production and sometimes critical systems... asking this question, even rhetorically, makes you sound as if you didn't knew that Perl is actually used in production.

When I use the word "DarkPAN", I mean precisely this: code which p5p cannot see, cannot search, does not know the features of, and which may or may not exist. See also "We can't change this misfeature in the core, because it may break the DarkPAN!" -- funny how that sentence always ends there, rather than including "But of course, we may not, and we'll never know, because it's the DarkPAN."

I tend to think that the regular contributors of perl5-porters are a lot more likely to have informed opinions about the Perl 5 core development than people who don't even read it.

Pick on me instead then. If you don't like my opinions, feel free to revert all of my patches over the past eight years.

[Who's] trying to be realistic by attempting to release software with some quality expectations, notably by making the upgrade process seamless and introducing as little bugs as possible?

Yet those quality expectations did not work in the case of Perl 5.10, with well-known defects continuing to spread as more and more people upgrade to Perl 5.10. The existing process does not work. Call it "realistic" if you like, but it did not work for 5.10. (Note also that one of the blockers to 5.10.1 is RT #22977 -- see also RT #50528 -- a bug almost six years old that's been present in eleven stable releases of Perl 5.)

First, we should not consider breaking it. Breakage happens, but our goal is to avoid it. Secondly, breakage is detected after upgrades, usually (or it would have been avoided).

The original point referred to deliberate, backwards-incompatible changes. Note, however, that a regular release policy could revert accidental breakages on a known, fixed schedule. The current Perl 5 release policy does not address this at all, which is part of my objection.

We just don't like to introduce incompatible changes just for the sake of being incompatible.

The only people suggesting making incompatible changes for the sake of incompatibility are people setting up a strawman argument against people like me who write tests, write testing modules, write documentation, write manuals, explore bugs, give talks, teach new users how to write Perl effectively, and occasionally write patches for new features who say "You know, some of the existing features are wrong, and they'd be better off this way."

In other words, this is an attempt to diffuse the real issue. The question is not and has never been "Should existing code randomly stop working?" Instead the question is "Can the defaults change in such a way that existing code will still work with perhaps a one-time, one-line change?" (It may not even require that much change.)

[Code] changes don't happen because someone yells about it on a blog. They happen because someone actually writes code.

I wrote code. Rejected. ("What's the point of a class keywords? You can always use package and @ISA anyway! If you need something more, just use any of several dozen Class::* modules on the CPAN!") Jonathan Leto put together a potential Perl 5.10.1 release that needed only a pull from the appropriate perldelta and a pumpking to release it. Rejected. ("Perl 5.10.1 is just around the corner, really, this time we mean it! Small releases are silly and expensive because of stat calls! A regular release cycle is like believing in astrology!")

But we're not in the land of toy projects here. We can't bless any change (or revert it two releases afterwards) just because it looks shiny at the time. This is not the parrot you're looking for. People do use Perl 5 for serious things, and we must ensure that a certain level of quality is preserved.

I can predict the day and date of Parrot releases into the next century. The number of contributors to Parrot grows over time, as does the frequency of commits. Parrot right now supports features that the Perl 5 VM will never support, and that feature list will only increase over time.

Dismiss all of that because Parrot is neither as widely-used nor as old as Perl 5 if you like, or even because you think I'm a jerk and you hate everything I care about, but there's one overriding metric for software project management: can people use the software? (The corollary is that unreleased software effectively does not exist.)

Quibble about the length of our deprecation policy. Complain that we haven't implemented your pet feature in a way that you like. Beg us to move a milestone forward. Berate us for not fixing a bug you don't report. That's fine.

Yet I believe almost every Parrot committer will agree that our defined support and deprecation policies and our regular release cycle have rescued the project from the very real threats of irrelevance, quality problems, and uselessness.

"Shut up and write some code!" is just a polite (relatively) way of saying "Do it our way or not at all, and above all, do not question the process or its results -- unless you're a core hacker."

The DarkPAN matters

2009-07-04T12:00:00+00:00

There is currently some FUD going about in some parts of the Perl community about why we should break Perl 5 backwards compatibility. A short blog entry, schmarkpan, is a good example of the trend: loud, noisy, but clueless and devoid of any content.

The author writes, DarkPAN was discussed quite a bit at YAPC::NA. But really, what is it? How is it defined? Well, DarkPAN is just a slang word to express the fact that Perl 5 is now currently used all over the world in production and sometimes critical systems: websites with high availibility, financial systems, operating system build processes, and so on. That would also include the so-called GreyPAN, all Perl OSS applications that are not on CPAN. So, asking this question, even rhetorically, makes you sound as if you didn't knew that Perl is actually used in production.

He continues, Who are these people that have a vested interested in Perl and yet do not participate? For a start, I would like to point out that Mr Perez himself is someone with apparently some interest in Perl, but who does not participate in Perl 5 at the slightest. I might be biased, but I tend to think that the regular contributors of perl5-porters are a lot more likely to have informed opinions about the Perl 5 core development than people who don't even read it.

I think attempting to appease a faceless entity with no defined boundaries has fail written all over it. It is fear mongering. Who's the fear monger here? And who's trying to be realistic by attempting to release software with some quality expectations, notably by making the upgrade process seamless and introducing as little bugs as possible? It's a well-known political technique to accuse the opposite party of being guilty of one's own defects. But it doesn't make you right.

If we break it, they can choose not to upgrade. Several fallacies in one single sentence. First, we should not consider breaking it. Breakage happens, but our goal is to avoid it. Secondly, breakage is detected after upgrades, usually (or it would have been avoided). Third, this not only applies to Perl 5, but also to every open source project widely used (in the Perl world, DBI or LWP come to mind -- upgrading them is not a trivial operation either.) This is a matter of common sense.

Then, suddenly and without warning, the discussion shifts from breaking backwards compatibility to simply breaking Perl: What about bug fixes?, I hear you say. That is the joy of open source. There are lots of unemployed hackers out there that would be willing to backport those fixes for you. Oh heavens forbid! Actually paying open source hackers to write code! The sky is falling! Wishful thinking at its finest. Mr Perez might have found a tuit tree in the middle of a magic garden, but I have never seen "lots of unemployed hackers" who are at the same time fluent in the Perl internals, and lots of generous donators that fight over the privilege of paying them to work on Perl. Actually, there is one, currently: Dave Mitchell, who is paid to get 5.10.1 released. Yes, paid: the sky is falling.

Perl does not belong to DarkPAN. It belongs to us who participate in its wellbeing. In other words, "fuck the users". But we're not in the land of toy projects here. We can't bless any change (or revert it two releases afterwards) just because it looks shiny at the time. This is not the parrot you're looking for. People do use Perl 5 for serious things, and we must ensure that a certain level of quality is preserved.

But have you, Mr Perez, even skimmed through a perldelta manpage, to look at the list of incompatible changes? Because there's actually a lot of them, in spite of what you seem to believe. I had code that broke because strict is stricter in 5.10 (as mentioned in the manual that you didn't bother to read.) ghc broke because we removed the magical variable $*. It's not like we fear incompatible change. We just don't like to introduce incompatible changes just for the sake of being incompatible.

I say we make the decisions best for our language of choice. But code changes don't happen because someone yells about it on a blog. They happen because someone actually writes code. If you really want to see something changed in Perl 5, write a patch and come discussing it on P5P. Although you might not have an idea about what you might want to see changed in Perl 5, since your empty, uninformed and slightly inconsistent rant fails to identify any precise problem.

Degradable Gists (update)

2009-07-04T07:03:55+00:00

The good people at GitHub have implemented a JSONP api for Gists, so I have updated my graceful degarding script accordingly (see previous post).

Now the code no longer needs to hijack document.write and has better load time characteristics as well. If you use traditional <script> tags then each gist will block the loading of the page at that point, causing jittery page renderings, and preventing the rest of the content from appearing. This script styles the <pre> tags when the DOM becomes ready, and only fetches the syntax highlighted HTML after the onload event, when all other page elements have finished loading.

The second new bit is the Perl scripts I've added to streamline working with these degradable Gists.

The first script is very simple, given a Gist ID it prints out a <pre> tag suitable for inclusion into your HTML document. For instance make_pre_tag.pl 115368 outputs:

<pre id="fake-gist-115368" class="fake-gist">use Moose;
has fun =&gt; ( isa =&gt; &quot;Constant&quot; );</pre>

The second script, update_gists.pl is much more interesting. It uses XML::LibXML to find all the pre tags with the class fake-gist in a document. If these tags have no ID then App::Nopaste is used to create new Gists. Otherwise AnyEvent::HTTP is used to concurrently fetch the updated versions from http://gist.github.com/.

I start by editing a plain XHTML document in my editor, adding code in the <pre> tags, and then I post the code blocks to github all at once. I use CDATA sections to escape the source code in the XHTML document.

<pre lang="perl" class="fake-gist"><![CDATA[
$object->method(
    foo => $bar, 
);
]]></pre>

The script re-escapes that using HTML entities so that it is valid HTML (which doesn't support CDATA). This way the <pre> tags render fine under quirks mode, allowing jQuery to append HTML strings to the document.

charles proxy is way cool

2009-07-04T02:40:06+00:00

On the recommendation of a friend, I tried out the Charles HTTP proxy for debugging... web stuff. It's nagware, and the nagging is really annoying. Despite that, it was obvious within the first two hours that I was going to buy it. It made it very, very simple to diagnose a number of problems that Firebug and Safari could not sort out. (They both seem to clear their logs fairly aggressively during redirection, for one thing.)

Once I'd sorted out my first bug, I (work) bought a license. A few hours later, I used it to solve another problem. Then two more. Seriously, it's really, really useful, and I've barely scratched the surface of what it can do. If you have to deal with the web as a programmer, check it out. I think it's probably already saved me $50 worth of work time.

My new hackintosh

2009-07-04T00:00:04+00:00

I recently acquired a Dell Mini 9 laptop and turned it into a Hackintosh using these instructions from Gizmodo.

The install was relatively smooth. Not 100% — I had to go through it twice in the end — but not bad. Here is the result: my hackintosh posing with my work laptop, a 15″ Macbook Pro.

I’m currently working on installing all the software I need to feel at home, and getting used to the damn apostrophe key being nowhere near where I expect it to be. The keyboard size, in and of itself, is fine, but the placement of some of the punctuation keys is driving me a little bit mad.

Here are my additional notes:

I used the USB stick install, no DVD drive. Where Gizmodo says Choose “80″ for the primary internal SSD I had to type “81″ instead.

When I upgraded using System Update, it took me to OSX 10.5.7, and then the DellEFI installer didn’t work properly, and I got into an unrecoverable (to me) state of wedgitude: the machine would boot to the grey apple logo, then it would get all these weird video artifacts, and hang. I had to start over on account of this. Second time through, I carefully downloaded the 10.5.6 combo update from Apple and installed that instead. Worked fine.

Wake from sleep wasn’t working. Googling around, stevenf’s hackintosh notes told me I had to disable “Legacy USB support” in the BIOS to make it wake from sleep correctly. I did this and it worked fine. However, I gather that “hibernate” doesn’t really work, so I’m going to have to be careful about not leaving things unsaved when I put the laptop to sleep for a long time.

I’m not very impressed with the battery life, but I hear 10.5.7 improves matters. I’m not going to try it right now — I’m off for a short trip tonight and don’t want to get the laptop wedged again — but I might try next week.

Another thing I’d like to figure out is whether I can manage dual boot with Ubuntu. I might find myself a bit tight on disk space, but I have the 32GB SSD and I’m sure it’ll fit, even if it does cut into my space for music and videos. If I start running out of room, I can always expand with an SD card.

Hackintoshing is, of course, in contravention of Apple’s Software License Agreement. All I have to say to that is: if Apple had a tiny, lightweight netbook available, I would be first in line to buy one. I say this as the owner of a Macbook (my third, counting work laptops), a Mac Mini, an iPhone, and a certain amount of fully licensed Apple software.

DateTime::Duration::W3C?

2009-07-03T13:45:44+00:00

If this is on the CPAN, please let me know. I can't find it. If not, we probably need it at work

The W3C duration standard, derived from the ISO 8601 duration standard, allows one to specify durations in a format like PnYnMnDTnHnMnS (e.g.: "P1Y2MT2H", a duration of one year, two months and two hours). I can't find a proper parser for it and it turns out to have some tricky bits. Basically, I want to do this:

my $duration = DateTime::Duration::W3C->new_from_w3c("P1Y2MT2H");

Which would be more or less equivalent to:

my $duration = DateTime::Duration->new( years => 1, months => 2, hours => 2, );

The only real difference would be (I think) adding string overloading to allow one to print the duration and have it come out in the proper format.

I can write this and try to handle the special cases I find documented (I'll be damned if I'm going to pay 130 Swiss Francs for the damned ISO 8601 standard), but I'd love to know if it's already out there.

How to Count (Parrot Style)

2009-07-03T09:11:54+00:00

Parrot releases are harder to count because of their prolific release cycle.

Brian Wisti, In Which Brian Whinges About The Perl 5 Release Schedule

This is a trivial nitpick (read the rest of the article! It's very good!), but Parrot releases are very, very easy to count. The same goes for Rakudo releases.

The other day someone asked what Parrot might look like in a hundred years. I laughed and thought, "What would its version number be?" Then I realized that I can predict its version number in 100 years. Parrot 101.6 will be out, with Parrot 101.7 on the way.

For all of the lofty talk about "stability" and "maturity" and "predictability" which results in Perl 5 not getting released, the fact that I can predict the release date and version number of a piece of software one hundred years in the future says something about the stability, maturity, predictability, and reliability of a very different kind of development process.

(Oh, and Alias -- I can crash several so-called stable releases of Perl 5 with a one-liner the same way you crashed Parrot in December 2008 with a short program. If you want to make the argument that the 30 stable monthly releases of Parrot in a row don't actually exist because they had bugs, the two so-called stable releases of Perl 5 from the same time period don't exist either. Ontological debates are easy to lose.)

Perl 6 Design Minutes for 27 May 2009

2009-07-02T23:03:51+00:00

The Perl 6 design team met by phone on 27 May 2009. Larry, Allison, Patrick, Jerry, and chromatic attended.

Larry:

changed the time function to return a Rat
thinking about the traits that have been bothering Jonathan and others
have some changes to check into the spec when I'm happy with them
thinking about the primitives we use to define use
breaks down into load and import
thinking of establishing compile-time keywords for both concepts
intended so that I can import from anything acting like a module -- an inlined role, for example
otherwise trying to keep up with the flow of IRC

Allison:

working on the Parrot book
changed its focus to a small, 100-page PIR book from a monolithic Parrot book
the intent is to get something out for YAPC and OSCON
will send out a draft for review
will merge my changes into the repo later this week

Patrick:

released Rakudo #17 last week
was easy again
875 more tests since #16, so we pass 68% of the spectest suite
finished implementing the root_new opcode in Parrot
cleans up a lot of the PMCProxy issues from moving Rakudo to its own HLL
gained half of the speed we lost from the migration
we'll get more back as we update more places that need it
NQP never expected anything like that
I have to rework some it and PCT
haven't quite figured out how to do that
refactoring use and import in Rakudo
the current implementation doesn't work
will hopefully match with what Larry's putting in the spec
it seems like the logical way to do things
updated Rakudo's ROADMAP in docs/ROADMAP
gives us an idea of dependencies and next tasks
may also help people understand what blocks features they want

Jerry:

the bonding period has ended for GSoC
time for students to start coding
everyone on the Perl 6 and Parrot projects is ready

fixed some memory leaks in Parrot and Rakudo
there are still some in Rakudo, but the web examples should be able to live longer
did more profiling
think NFG is important for Parrot in the near term
have some documentation to write
have been editing the Parrot book

Patrick:

how's the command line for Rakudo coming?

Jerry:

I expect to get back to that

Patrick:

the "parens build captures" decision surprised me
what's the rationale?
I really liked "parens mean grouping"
maybe I haven't reconfigured my worldview yet, but it feels messy

Larry:

when used in an argument list, it has the same effect as a capture

Patrick:

it even works when they're used as a term

Larry:

they still mean that you have to look at what you're binding to and decide
am I binding this to a scalar or to an array?
(1, 2, 3) bound to an array...

Patrick:

I'm going to have to think about that
the zip operator in slice context....
is this three or one positional arguments? zip($a,$b,$c)
how many positional arguments are in this case? zip($a,$b,$c;$d)

Larry:

one slice
you wouldn't want to write that

Patrick:

what in the arg list distinguishes the use of the semicolon versus the comma
inside of an argument list we have to recognize a variety of syntactic things
comma, semicolon, colon, array or hash sigil, named parameters
seems like captures need more information than just positional
they need to store metadata about positional arguments
I like the syntactic stuff showing up in the argument list
but I don't want to handle them in three different ways

Larry:

I'll have to think about that

Patrick:

haven't figured out how to deal with slice context either

Larry:

might say that the presence of a semicolon implies the presence of other parens
the comma implies...
that might be more consistent binding for a top-level list

Patrick:

I half expected that answer
I can see the semicolon as just a lower precedence grouping operator

Larry:

otherwise you have a semicolon that's just not there in every other argument list

Patrick:

assuming that, the other commas form an argument list through the infix semicolon
an array in there means Capture of Capture of Capture
we were about to refactor List and Array in Rakudo anyway
the question is "Do we really have a List type now?"
Rakudo assumes that

Larry:

if we can unify args list with List, that's probably healthy

Patrick:

I'd really like that
that makes things a lot cleaner
infix comma and infix semis now just create arglists

Larry:

or Lists
if you define List as "something that has out of band metadata"

Patrick:

any more decisions that you can make about that will help our implementation
I probably won't get around to that this week

Larry:

we make syntactic distinctions
we know that this is an arg list
we treat pairs as named arguments
we don't do that if we know it's not an argument list
it stays positional
that's the only distinction between an arg list and a List
purely syntactic

Patrick:

to summarize
zip($a, $b, $c) has three positional arguments
zip($a, $b, $c; $d) has two, the first of which is itself a list/capture

Need for Speed Part I: DBIx::Class vs DBI

2009-07-02T20:49:15+00:00

Yesterday I had to write some code that goes through ~700.000 datasets (seperated into 6 tables) and denormalise them (see the yet unwritten Part II). As we're using DBIx::Class, I first used it. Even though I avoided some in/deflators and used columns to only get the stuff I needed, the process took ages (~50 items per second or aprox 4 hours for the whole job). Well, 4 hours might be bearable, but this was only the Swiss dataset. The German one, which we have to tackle soon, is at least 10 times as big, and 40 hours is just a way too long runtime.

So I rewrote the core of the programm using raw DBI calls (I had several flashbacks to the 90's :-) The results were quite astounding, as I was now getting 500 items a second. 10 times faster!

And that was before koki told me to set work_mem to a bigger value. I choose 30MB, and got another speedup to ~900 items per second. Sweet!

Of course, it's completely unfair to compare DBI with DBIx::Class performance wise, because DBIx::Class is so much nicer to work with. But if speed is an issue, dropping back to raw DBI is the way to go.

What is Catalyst, Really?

2009-07-02T18:22:46+00:00

A recent thread on the Mason users list reminded me of the problems I had grokking Catalyst when I first looked at it. Raymond Wan wrote "I'm skimming over the MVC part and as my system doesn't use an SQL database, I'm wondering if Catalyst is overkill??"

Those of us who know Catalyst know that this question is based on some wrong assumptions, but it's easy to see why Raymond has those assumptions. Look at the Catalyst tutorial. In the very first chapter, it's already talking about databases and DBIx::Class.

It's easy to look at this and assume that Catalyst is somehow tightly bound to DBIx::Class or SQL databases.

The problem is that the tutorial docs really need to serve two different audiences, though both audiences are Catalyst newbies.

On the one hand, you have people with relatively little web app experience. Presumably, they know some Perl, and they do a web search for "perl web application framework". Eventually, they'll get to Catalyst and start reading about it. For those people, being given a set of standards and a full walk through of Model and View (in addition to Controller) is very valuable. It gives them all the tools they need to get started on simple web apps without having to make too many choices.

The other audience is people who have some real web app development experience with something more primitive than Catalyst. These could be people coming from a Mason-based site without any real controllers, or people who used something like Apache::Template with mod_perl, or maybe they wrote their own controllers "by hand" using the mod_perl API.

Many of those folks will already have some experience with models. To refer back to Raymond, presumably his system already has some sort of model code, it just doesn't use a SQL DBMS. Those people will look at the existing Catalyst tutorial and get confused. Isn't Catalyst flexible? Then why does it look like RoR, with all my tool choices made for me?

It took me a while to realize that Catalyst, at its core, is smaller than you might think, and you can use just the pieces you like.

The very core of Catalyst is its dispatching system. Given a URI, it selects a piece of code to run. Its dispatcher is very powerful (Chained methods are great!), and with plugins like Catalyst::Action::REST, it's even better.

Along with the dispatching system, Catalyst also provides an abstraction over the typical web app request/response cycle. The Request makes it easy to look at incoming query arguments, POST data, file uploads, and headers. The Response lets you set headers and return output to the client, whether that be HTML or a downloadable file.

Catalyst also includes engines (think "environment adapters") for a number of common web application environments, including vanilla CGI, mod_perl, FastCGI, and more. These engines make sure that the Request/Response API works exactly the same in any environment where Catalyst can be deployed.

This is a huge win, since you can write your app without worrying about the deployment environment. If you're writing an app for public distribution, it gives the installing users a choice of how to deploy.

These core pieces are really the only parts of Catalyst you have to use when you use Catalyst. If you don't want dispatch and a request/response API, you don't want Catalyst.

Catalyst (really Catalyst-Devel) also includes a fantastic single-process development server. This server can be started straight from an application's checkout directory with one command. Even better, this dev server can be told to monitor all relevant files and restart itself when any of them change. Note that this is a proper restart, which avoids all the myriad problems that afflict Apache::Reload and its ilk, which attempt to reload modules in the same Perl interpreter.

Just these things - controllers, a request/response abstraction, deployment agnosticism, and a great dev environment - are enough to make Catalyst a great choice. Ignore everything else it does and you'll still have improved your development process and improved your code.

Catalyst also does some other things ...

It has a component system which has allowed people to release a whole host of useful plugins. If you look on CPAN, you'll find things like sessions, powerful authentication, dumb authentication, I18N, and much more. If a plugin does what you need, it'll save you a lot of development time.

Note that the old "Catalyst::Plugin" system is in the process of being deprecated, but the concept of pluggable components is still core to what Catalyst is. All that's changed is the way pluggability works.

Catalyst lets you have multiple views. While many apps will just output HTML via a templating system, this flexibility is great for RESTful apps that may want to output XML, JSON, and still fall back to HTML for browsers (see my REST-ForBrowsers for some help with that).

Catalyst also has configuration file handling built-in. Personally, I avoid it, because it only works within the context of the whole "Catalyst environment". That means it's awkward at best to load the configuration outside of the web environment. I always make sure that application wide setting are available for things like cron jobs. This falls into a category of Catalyst features which are not best practices, but are probably useful for people writing their first web app.

Catalyst gives you hooks for models. Again, this is something I never use, but it's another "useful for web app n00bs" feature.

There's probably many other things I've left out. The point is that Catalyst provides a very powerful set of core web app tools, and that core is actually small.

Relative to a "my way or the highway" type framework (RoR and Jifty, I'm looking at you), it's easy to port an existing application to Catalyst. In fact, using Catalyst::Controller::WrapCGI, you can wrap an existing CGI application with Catalyst, and then convert slowly over to "native" controllers.

And most importantly, you can to Catalyst without touching your model at all! Since many applications have the bulk of their code in the models (at least, they do if you're doing it right), this is a huge win.

Next step is to turn some of this rambling into doc patches. I think a section of the Catalyst tutorial aimed at folks coming from an "old school" web app background would be great, and would really help people like Raymond (and would've helped me a few years back).

The Real Problem With Roles

2009-07-02T13:47:33+00:00

Inheritance, as most of us know, is rather problematic. Specifically, it's been around since 1967 and for the past 42 years, people have been arguing about how to do it right. Some OO languages don't have inheritance (Self, Javascript). Some languages don't have multiple inheritance (C#, Ruby). Some languages have multiple inheritance with many safeguards (Eiffel) or none (Perl). Some languages just do strange things with inheritance (BETA). Even very experienced OO developers focusing on a single language will argue vociferously about whether or not a design has been done correctly (do you really give a damn about strict equivalence in overridden methods?)

The problem is ultimately that classes fulfill conflicting needs. As an agent of responsibility, a class needs to do everything a class needs to do. That sounds like a stupid tautology, but what it means is that as systems grow, classes grow. Thus, classes tend to get larger. Unfortunately, as agents of code reuse -- via inheritance or delegation -- classes should be smaller. How many times have you seen (or written) code which inherits from something because it needs one or two methods but you're pulling in a lot of extra behavior which you don't need?

This responsibility/reuse tension which leads to classes wanting to be both larger and smaller at the same time is much of the reason why inheritance has proven so problematic. Interfaces in Java and C#, along with mixins in Ruby and other languages were an attempt to decouple the responsibility and reuse needs of classes, but they've all had their problems. Roles seem to handle the decoupling of responsilibity and reuse quite nicely, but some of you may have noticed some of the disagreements that I and others have had about the proper use of roles. Why is that?

Ultimately, the disagreement boiled down to the fact that -- like inheritance -- roles serve more than one master. Roles provide behavior and roles provide an interface. Are we faced with another 40 years of arguing because we've again tried to shoehorn too much into one thing? I don't think this need be the case as, unlike classes, the different things the roles provide aren't necessarily antagonistic, but it would be nice if the different parties sat down and tried to come up with a strategy (syntax?) for implementing roles which cleanly addresses these disparate needs. Otherwise, there really will another 40 years of arguing.

Why am I writing Padre?

2009-07-02T09:24:31+00:00

It is probably not that interesting for most of the people but I have my excuses. What is more interesting - at least to me - is why others are involved.

After all there were already more than 40 people who made some contribution to Padre.

So I asked them to start writing down why are they involved. Some of them have already added themselves to the developers page and we started a separate page for translators though that one is new and has no been filled yet.

So why do I write Padre?

There is a short explanation on that page but let me bore you with a longer one here.

I have been teaching Perl 5 for almost 10 years now. Both beginner level and advanced courses. In the beginner courses the majority of the people use Windows with about 20-30 percent using Linux. Most of the Windows users use Notepad++ or a similar editor. Some of them can configure their editor to run the perl script right from the editor, others don't even know how to enable syntax highlighting for Perl 5. Some of the people on Windows don't know what the command line is but even those who know how to open the command prompt are either afraid of it or just dislike it.

Mind you these are not stupid people or anything like that. Some of them have many years of hardware design behind them. They are just used to some kind of IDEs.

Others might use Linux/Unix but many not by choice. In many cases the company they are working for gives them telnet access to an oldish Unix machine and tell them to code in Perl. They hardly know vi, they don't know how to configure syntax highlighting and they don't know how to install any other editor.

I have been using vim for many years and I can testify that it is a superb editor. Emacs is similary strong but I have not used it since university. The problem with both of them is that they are totally different from the editors most people are used to and their learning curve is long and steep. It takes several months or even years to become familiar with them. Most of the people don't want to invest that energy and I certainly don't have the time for that in a 4 days long Perl 5 course.

So I usually tell them to use whatever they are familiar with. I think I never had a student who picked Eclipse or Emacs. There were a few using vi/vim but most of the people used Notepad++ or a similar Windows based editor.

I don't think my students are too different from the average people learning and using Perl. Most of them will never write a full blown web application. Heck most of them won't have a need for object oriented coding as they write only 100-200 line long scripts in Perl.

They never get really familiar with Perl and they will always have to deal with strange code written by others. With all the other tasks they are required to do in their primary language or tool they will keep wondering what is $_ when they see it and will be surprised when they don't.

They will wonder what is the relationship between $_ and $_[0].

So my first objective was to provide a simple Notepad++ like editor with strong support for beginners or people with relatively little experience with Perl.

For example I'd like to make sure that Padre users will be able to highlight $_, press F1 and get an explanation. In case they highlight $_ from a $_[0] expression Padre should be clever enough to point this out and give an explanation on @_ and @_[0].

I knew it is a big task though I did not think it is soo big. In any case I knew I cannot handle this alone, especially as I am not such a good programmer. So from the very beginning I tried to make it interesting, accessible to experienced Perl 5 developers and in turn they made it fun and hackable.

Unfortunatelly most of the experienced Perl developers are also hard-core vim or emacs users and it is nearly impossible to move them away from their editor. Luckily there were a few who got interested by the idea of having a lot of control over their editor and that made the difference between a failing one man project to a project with a lot of potential.

Then at one point came the idea to integrate Parrot and Rakudo and try to provide a platform for Perl 6 development as well. With the awesome work of Ahmad M. Zawawi Padre has a lot to offer to anyone who wants to write Perl 6 code. With some more work we'll be able to integrate Parrot even better and then Padre will be able to handle any language running on top of Parrot.

While there are still lots of problems with Padre and installation can still be difficult it already more or less has the same power as Notepad++ on both Windows and Linux and probably on Mac too and it has some extra features that even vim people started to steal.

The impending death of BZip2

2009-07-02T06:18:57+00:00

adam@svn:~/svn.ali.as/db$ ls -l total 30884 -rw-r--r-- 1 adam adam 9558294 Jul 2 03:45 cpandb.gz -rw-r--r-- 1 adam adam 8538979 Jul 2 03:45 cpandb.bz2 -rw-r--r-- 1 adam adam 5960155 Jul 2 03:45 cpandb.lz -rw-r--r-- 1 adam adam 3014480 Jun 30 06:46 cpanmeta.gz -rw-r--r-- 1 adam adam 2658756 Jun 30 06:46 cpanmeta.bz2 -rw-r--r-- 1 adam adam 1825600 Jun 30 06:46 cpanmeta.lz

Updating minicpan alphas more frequently

2009-07-01T22:03:34+00:00

At the Perl QA hackathon, Andreas Koenig, the maintainer of PAUSE, demonstrated a way to decrease the lag between new distribution uploads to PAUSE and syndication to CPAN mirrors. (See File::Rsync::Mirror::Recent on CPAN for details). Running minicpan against a one of the new, fast updating mirrors results a very fresh minicpan, but, until now, only for regular releases, not for alpha development releases.

Last year, I wrote CPAN::Mini::Devel to include development releases in a minicpan, but it depends on an obscure, large, infrequently updated index.¹ Now, with smaller, fast-updating index files available for rsync, I released CPAN::Mini::Devel::Recent to take advantage of them for minicpan as well.

This is all still experimental, but if you do Perl programming and want bleeding-edge development releases within minutes of their upload to PAUSE, then find a fast mirror, try out CPAN::Mini::Devel::Recent and let me know how it works for you. After installation, you can configure it to be used automatically by adding “class: CPAN::Mini::Devel::Recent” to your .minicpanrc file.

indices/find-ls.gz

CPANDB 0.02 - Now we're starting to get somewhere

2009-07-01T17:17:06+00:00

On the back my my improved and high-coverage CPAN::Mini::Visit and Archive::* fixes, I've finally managed to build a complete-coverage version of CPANDB.

CPANDB is a merged and cleaned up schema that combines the CPAN index, the "CPAN Uploads" database (for PAUSE upload dates), and both class and distribution level dependency information held in META.yml files (replacing the CPANTS dependency graph).

To take a look at it, you can grab a copy of the SQLite database directly from the following URL.

http://svn.ali.as/db/cpandb.gz.

The data sources used to generate it are not perfectly time-synced yet, so I expect to see a few minor flaws for another release or two. But compared to everything else available (from pretty much everybody) this should be a significant improvement.

As well as clearing up the last tiny data quality issues, I'm also yet to merge in the rt.cpan.org database (which is almost ready) and the CPAN Ratings database (which is a text file I really don't want to have to parse).

But don't let this stop you trying it out now (I've appended the schema to the bottom of this post so you can get a clearer idea of what's in there).

As usual, feedback is welcome.

CREATE TABLE author (
author TEXT NOT NULL PRIMARY KEY,
name TEXT NOT NULL
);

CREATE TABLE distribution (
distribution TEXT NOT NULL PRIMARY KEY,
version TEXT NULL,
author TEXT NOT NULL,
release TEXT NOT NULL,
uploaded TEXT NOT NULL,
FOREIGN KEY ( author ) REFERENCES author ( author )
);

CREATE TABLE module (
module TEXT NOT NULL PRIMARY KEY,
version TEXT NULL,
distribution TEXT NOT NULL,
FOREIGN KEY ( distribution ) REFERENCES distribution ( distribution )
);

CREATE TABLE requires (
distribution TEXT NOT NULL,
module TEXT NOT NULL,
version TEXT NULL,
phase TEXT NOT NULL,
PRIMARY KEY ( distribution, module, phase ),
FOREIGN KEY ( distribution ) REFERENCES distribution ( distribution ),
FOREIGN KEY ( module ) REFERENCES module ( module )
);

CREATE TABLE dependency (
distribution TEXT NOT NULL,
dependency TEXT NOT NULL,
phase TEXT NOT NULL,
PRIMARY KEY ( distribution, dependency, phase ),
FOREIGN KEY ( distribution ) REFERENCES distribition ( distribution ),
FOREIGN KEY ( dependency ) REFERENCES distribution ( distribution )
);

Planet Perl

Log::Any released, only two years late

Test::Pod 1.40 now checks for illegal L&lt;> constructs

Is it right to auto-notify authors, if there aren't many?

what i want from a d&d wiki

Learn Perl

obsidian portal and me

Perlbuzz news roundup for 2009-07-11

current project status braindump

Data::GUID

Email::MIME::Kit

META.json

Other Email Stuff

Pod::Weaver and Friends

Finally...

Help needed from Germans and Solaris users

If I were (pump)king

Padre standalone installer for Windows - first beta version

$me-&gt;sleep(604800)

Empty Classes

The strictperl experiment

The Success of Ubuntu

The success of Ubuntu

Perl Community

7 reasons of In another article called The 7 reasons why Ubuntu is so successful I found the following reasons:

Padre 0.39 released

The CPAN Top 100 website has been upgraded to CPANDB

CPANDB Tricks: Your personal version of the FAIL 100

Promises, promises

CPANDB Tricks: Tarballs that fail under Win32

Complexity Management

The Corporate CPAN II

PL_runops

CPANDB Tricks: Smoking the Top 100

Saudi Arabian Adventures - Days 3-5

Introducing CPANHQ

mod_perlite helps Perl gain ground in the PHP arena

Perl 6 Roles Question

Italian Perl Workshop 2009 guests

Iron Man ideas backlog

Open source ‘full text’ bookmarklet and feed filter

Holy crap, I suck at coding

Resigning

Reputations

Perlbuzz is no longer useful

CPANDB should now be synchronised and up to date

Perlbuzz news roundup 2009-07-05

Links for 2009-07-05

Let me be clear, again, about the nature of the DarkPAN.

Bootstrapping KiokuDB::Cmd

Time-based releases in open source

Why "Shut Up and Write Code!" is Unhelpful and Wrong

The DarkPAN matters

Degradable Gists (update)

charles proxy is way cool

My new hackintosh

DateTime::Duration::W3C?

How to Count (Parrot Style)

Perl 6 Design Minutes for 27 May 2009

Need for Speed Part I: DBIx::Class vs DBI

What is Catalyst, Really?

The Real Problem With Roles

Why am I writing Padre?

The impending death of BZip2

Updating minicpan alphas more frequently

CPANDB 0.02 - Now we're starting to get somewhere

Test::Pod 1.40 now checks for illegal L<> constructs

$me->sleep(604800)