I’m looking for a CRM system implemented in Perl. As it turns out, so are the Perl Foundation.

So I thought I’d summarize my interpretation of the comments on that thread, as much for my own benefit as yours, and see if this post flushes out any further information.

We’ll start with the smaller/personal projects and work up from there…

John Cappiello

John mentioned that he was working on something. I sent John an email to ask for an update and he said it had “morphed away from a CRM into something not really overlapping much at all”.

Gábor Szabó

Gábor Szabó mentioned in the thred that he has “a simple CRM I use in-house that I plan to release as open source one day. It is written in Perl. While it is very minimalistic if you are interested I can show it and we can discuss what additional features TPF might need.”

Affinity

Affinity was mentioned in the thread, but I can’t find any references to it being open source.

CustomX

The web pages are in German, so here’s a translation to English of their web site, which gets visually mangled in the process, and a much more readable translation of their 2006 whitepaper.

It’s available under the Perl Artistic License. The base language is German. I don’t know what level of internationalization/locaization they support.

They currently use mod_perl v1 and say it’s “not tested” for v2, which seems to suggest a lack of maintenance. Databases supported include Postgres, Oracle, MySQL, and Informix. It’s extensible via plugins.

I couldn’t find any indication of a development community.

Wice CRM Groupware

Another German CRM system, so here’s a translation of their home page. It’s in use by a range of companies, and they even have a glossy magazine (though it looks like a once-off produced in 2005).

“Wice is a LAMP system with Linux as operating system, Apache as web and application server, MySQL as the database and the system is developed as an Apache module in mode_perl.”

“We have Wice a plugin architecture, which we almost arbitrary adjustments, enhancements and interfaces can be realized [...] There are also numerous standard plugins, such as a web, an integrated CMS, or a Customer Self Service Center.”

The source code is not ‘free’, however. You get the source code only when you licence the software.

Donor.com

“Our software has been serving non-profits for 30+ years. We have spent the past 5 years wrapping all of our C-based business logic with Perl to expose everything as Web Services (50 modules and 300+ methods so far). Our presentation layers are the WebGUI CMS (also Perl) and a cross-client GUI (Win/OSX/Linux) written in wxPerl. Our systems integrate everything from CRM, inventory management, event tracking, credit card processing, bulk email, direct mail, magazine subscriptions, sponsorships, etc. Almost everything in our system is configurable. We have not yet open-sourced all of our code, but we did just transition our ownership to a new 501(c)3, the DonorWare Foundation, to help facilitate that in the future.”

Jim Brandt replied to comment in the thread saying “I got another response from the folks who run donor.com. Turns out they are heavy perl users, so we’re looking at their system right now to see if it will meet our needs. I’ll post with more details once we know.” No news yet.

Josh Berkus expressed some pros and cons of using donor.com.

Other Perl CRM Systems?

What have I missed? Are there any other open source CRM systems out there implemented in Perl?

For comparison, here are some details (mostly skimmed from Wikipedia) of open source CRM systems not implemented in perl…

CiviCRM

Designed to meet the needs of advocacy, non-profit and non-governmental groups. Implemented in PHP. The license for the current version is the GNU AGPL 3.

CiviCRM may be deployed either standalone or alongside Drupal and Joomla! content management systems. Both the Drupal and Joomla! professional associations use CiviCRM. The standalone version is intended to work alongside other CMSs.

CiviCRM is used by many large NGOs including Amnesty International, Creative Commons and the Wikimedia Foundation for their fundraising.

They’re currently considering a new architecture:

In short, we propose using PHP Doctrine for the ORM layer [...] we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.

Link to the CiviCRM project page at Ohloh.net.

vtiger CRM

An open source fork of SugerCRM. Implemented in PHP, initially released in 2003. Seems to be supported by AdventNet, who make a non-open source Java based CRM called Zoho. Licences: vtiger Public License 1.1 and SugarCRM Public License 1.1.2.

Hosted at sourceforge. Link to vtiger project page at Ohloh.net.

epesi BIM

Implemented in PHP. Links to the developers wiki, forums, and Ohloh.net project page.

A search on Ohloh.net for tag:crm yields 649 projects with the CRM tag!

It turns out that Ohloh is a great way to look for projects. The Analysis Summary on each project page gives a useful overview of some key metrics: “Large, active development team”, “Few source code comments” etc.

Sorting the list in various ways and taking the projects (other than those above) that appear on the first page of each, gives this list, in no particular order:

OpenERP

“Python. Formerly known as TinyERP. OpenERP is an Open Source enterprise management software. It covers and integrates most enterprise needs and processes: accounting, hr, sales, crm, purchase, stock, production, services management, project management, marketing campaign, management by affairs, …

Once in production, it automates and help you to control all activities: sales triggers manufacturing orders, accounting entries are updated by stock operations, incoming mails are tracked in the system, the integrated document management system helps your team to collaborate, …”

Open For Business Project (Apache OFBiz)

Java. “The Open For Business Project is a set of tools and enterprise applications including ERP, CRM, e-commerce, SCM, MRP, and CMMS/EAM. It uses a service oriented and events driven architecture and tools to automate all aspects of application development and maintenance.”

Omni ERP

Python. “Omni ERP is an innovative business application platform; it is completely based in open source technologies and brings a whole new level of modularity and flexibility to the business solutions environment.
It uses a new approach combining new software engineering techniques like plugin based architecture, inversion of control and aspect oriented programming with a refreshing new RIA UI to bring a new level of experience to the SME market”

ADempiere

Java. “ADempiere Business Suite ERP/CRM/MFG/SCM/POS done the Bazaar way in an open and unabated fashion. Focus is on the Community that includes Subject Matter Specialists, Implementors and End-Users. We are a community fork of Compiere”

ERP5

Python. “ERP5 is a full featured high end Open Source / Libre Software solution published under GPL license and used for mission critial ERP / CRM / MRP / SCM / PDM applications by industrial organisations and government agencies.

It is distributed to linux community via packages for numerous distributions (Mandriva, Debian, Ubuntu,…) and a dedicated Live CD.”

JFire

Java. “JFire is an ERP, CRM, eBusiness, and SCM/SRM solution for business enterprises. It uses JavaEE, JDO, and Eclipse RCP, and is designed to be highly customizable. It is a complete and extensible solution that fulfills business needs like user management, online trade with business partners, points of sale, various distribution channels forming a distribution network, store management, etc.”

CK-ERP

PHP. “CK-ERP is an open source accounting / MRP / ERP / CRM system that runs on top of multiple middlewares. It provides accounting and back office functionalities to SMEs and utilizes the underlying middleware to administer accounts/groups.

It comprises 22 modules – Administration, Multilingual Facility, Contact Management, Customer Relationship, Customer Self Service, Vendor Relationship, Material Requirement Planning, Warehouse, Inventory, Service, Accounting Ledger, Bank Reconciliation, Accounts Payable, Accounts Receivable, Purchase Order, Sales Order, Quotation, POS for Cashier, POS for Manager, Human Resources, Staff Self Service and Payroll.

Operating platform can either be LAMP or LAPP. Backend database engine can be anyone of MySQL, PostgreSQL and SQLite”. Oddly, their home page says “minimal documentation will be made available to users of CK-ERP” and their roadmap is old.

Blue ERP

PHP. “Blue ERP is an open source, web based ERP application. Its goal is to provide a flexible and user friendly interface that can work out of the box and be modified to suit specific needs easily.

The main goals of the projects are:* provide a feature full ERP application
* be open in licence and in spirit – in blue ERP everything is open, especially the development
* be user friendly by providing adequate documentation and assistance to users to encourage widespread adoption”

OTRS::ITSM

After all the PHP systems, it’s nice to find one implemented in Perl! Though the description suggests it’s geared towards the needs of an IT Service organization.

“ITIL compliant Open Source IT Service Management Solution (ITSM). Requires the OTRS (Open Ticket Request System) framework.

Besides all OTRS features, new ones include Incident, Problem and Configuration Management, integrated Configuration Management Data Base; process spanning Communication management: within the IT Service organization, towards Customers/Users/Management and Suppliers/Providers; powerfull statistical features for (Trend-)Analysis, key figured Reporting, ITSM-Planning/-Controlling; flexible configuration, customizing and expandability regarding your individual requirements.”

[That's quite enough "CRM" systems! Ed.]

I recall other bloggers complaining of unattributed redistribution of their work. Now a site called rapid-dev.net has started redistributing Plant Perl posts, including mine, with an advert at the top.

I wouldn’t mind if the page had clear attribution, but it doesn’t. In fact, at the bottom it says “Author: hoanatwho”.

That doesn’t feel right. Especially as many of my posts, and probably many others from Planet Perl, use the first-person pronoun “I”.

Why does this matter? A couple of months ago Merlin Mann wrote a long but excellent piece that explains why far better than I could.

Nobody but me is allowed to decide why I make things. And — if and when I choose to give away the things that I make — nobody but me is allowed to define how or where I’ll do it. I am independent.

Merlin discusses, with his typical style, the motivations of those who make their work available for free, and the perils of presuming to understand their motives. Although written mostly about bloggers it seems very applicable to authors of Open Source software. For me it echoes how I feel about coding and, to an extent, the freedom that Perl give me to express my thoughts.

If you have a blog I recommend you at least make the licence for reuse clear. My blog has a “Terms of Use” link in the sidebar that refers to Creative Commons “Attribution-Noncommercial-Share Alike 3.0″ license.

Looking at the Planet Perl page I see it has no licence. Perhaps that should be fixed — even if only to say that the license of the feeds being aggregated must be respected.

At OSCON this year¹ I’m giving a “State-of-the-art Profiling with Devel::NYTProf” talk. It’ll be an update of the one I gave last year, including coverage of new features added since then (including, hopefully, two significant new features that are in development).

This year I’d like to spend some time talking about how interpret the raw information and using it to guide code changes. Approaches like common sub-expression elimination and moving invariant code out of loops are straight-forward. They’re ‘low hanging fruit’ with no API changes involved. Good for a first-pass through the code.

Moving loops down into lower-level code is an example of a deeper change I’ve found useful. There are many more. I’d like to collect them to add to the talk and the NYTProf documentation.

So here’s a question for you: after looking at the NYTProf report, how did you identify what you needed to do to fix the problems?

I’m interested in your experiences. How you used NYTProf, how you interpreted the raw information NYTProf presented, and then, critically, how you decided what code changes to make to improve performance. What worked, what didn’t. The practice, not the theory.

Could you to take a moment to think back over the times you’ve used NYTProf, the testing strategy you’ve used, and the code changes you’ve made as a result? Ideally go back and review the diffs and commit comments.

Then send me an email — tell me your story!

The more detail the better! Ideally with actual code (or pseudo-code) snippets².

1. OSCON is in San Jose this year, July 20-24th. You can use the code ‘os09fos’ to get a 20% discount.
2. Annotated diff’s would be greatly appreciated. I’ll give credit for any examples used, naturally, and I’ll happily anonymize any code snippets that aren’t open source.

Ever copied and pasted a chunk from perldoc output and found you were getting mysterious errors from perl? I have.

I’ve learnt to rewrite the ‘-’ characters because although they look like ‘-’ characters they’re really a unicode HYPHEN: U+2010. Some other chars get mangled too, but that’s the most frequent problem for me.

So I was delighted to see a blog post by marcus ramberg called Fixing the POD synopsis in OSX wherein he fingers nroff as being the problem and gives a simple solution:

alias perldoc='perldoc -t'

Trouble is using perldoc -t means you loose the nice bold text that nroff gives you. So I went digging…

It seems the problem only affects people using UTF-8 and that nroff has a -T option that lets you specify an output encoding to use. So if perldoc ran ‘nroff -Tascii’ instead of plain ‘nroff’ that would avoid the hypen problem and let us keep the bold text.

It turns out that perldoc has an option to specify the nroff command to use, so the solution is simple:

alias perldoc="perldoc -n 'nroff -Tascii'"

This is, of course, still a hack. My main worry is that pod docs using non-ascii characters may get mangled. A much better fix would be to arrange for the ascii characters to not get mapped to unicode at all. So I went digging again…

nroff calls groff -m tty-char ... and tty-char refers to a tty-char.tmac file that defines the character mappings. The groff man pages point me to groff_tmac man pages which tell me I can get groff to look for .tmac files elsewhere by passing a -Mdir option or setting the GROFF_TMAC_PATH environment variable.

I looked at the default file, /usr/share/groff/1.19.2/tmac/tty-char.tmac on my Mac, and… decided it was time to go to sleep! The formatting is probably simple enough but I’m out of tuits.

So, what’s needed is for someone to determine what change is needed to the tty-char.tmac file, or the files it refers to, to avoid unwanted conversions to unicode. Then put a modified file into a directory and either add a -Mdir option to the nroff alias above, or set the GROFF_TMAC_PATH environment variable. Setting the env var has the benefit of ‘fixing’ all the man pages.

So, anyone want to dig deeper? (For all I know the solution can already be found on google…)

It is sad, but inevitable, that the TIOBE index of programming language “popularity” (sic) would be gamed.

Once you start measuring something, and advertising the results, people with an interest in particular outcomes naturally start to look for ways to influence those results. (It’s the Observer Effect writ large.)

The fact that TIOBE’s methodology, which I’ve discussed previously here and here, is simplistic makes it particularly open to gaming. Anyone, or any community, with access to many web pages can simply add the magic phrase “foo programming”, where foo is their language of choice, to get counted.

And it seems that’s exactly what the Delphi community did at the end of 2008¹. They made a concerted effort and it seems to have paid off. (I’d be very interested in hearing about similar behaviour in other language communities.)

Is that behaviour gaming? The author of the post who exhorted is readers to “Update your Delphi related blog or site to say Delphi programming on every page in visible text (update the template). Stand up and be counted. You can make a difference!” doesn’t seem to think so, as he also said “I am not suggesting we game the system, just that we help TCPI get an accurate count.“

An accurate count of what, exactly? That’s always been the fundamental question with TIOBE. It should be obvious that most web pages that talk about “delphi programming” wouldn’t actually contain the phrase “delphi programming”. The same applies to every other language. That’s the paradox at the heart of the TIOBE Index. And yet, somehow, TIOBE seem to think that counting pages containing the phrase “delphi programming” lets them claim that:

The ratings are based on the number of skilled engineers world-wide, courses and third party vendors.

Eh? How can they possibly defend that claim? Certainly their documented definition doesn’t support it, or even mention it.

I presume they’re thinking that CV’s, job postings, and adverts are most likely to contain the magic phrase. It should be obvious, again, that the number of CV’s, job postings, and adverts referring to a given programming language would naturally only be a small fraction of the total web pages referring to the language. (And only distantly related to the “popularity” of a language.) Yet that “small fraction” is what TIOBE measure and make bold claims about.

The fact that TIOBE is making a comparison based on a small fraction makes it even more troubling that TIOBE CEO Paul Jansen appears to support language communities changing their pages to include the magic “foo programming” phrase. In an email quoted on delphi.org he says:

For your information, I think your action has already some effect. Tonight’s run shows that Delphi is #8 at this moment. There is a realistic chance that Delphi will become “TIOBE’s Language of the Year 2008″

He’s endorsing the artificial insertion of the magic phrase. Clearly this distorts the TIOBE index in favour of language communities that infect as many pages as possible with the magic phrase.

That sure seems like an invitation to game the system! It’s likely to lead to other language communities doing the same, and so to further devaluation of the TIOBE Index.

(For alternatives to TIOBE you could look at sites like http://www.langpop.com/, James Robson’s Language Usage Indicators, or my popular comparison of job trends blog post with ‘live’ graphs.)

I have, on a couple of occasions, used the phrase “perl programming” in blog posts for my own amusement, and linked it to my original TIOBE or not TIOBE – “Lies, damned lies, and statistics” post. I haven’t suggested that others do the same. TIOBE’s endorsement of artificial insertion changes that. Now it seems like we’re going to get a dumb “race to the bottom” to see which language community controls the most web pages.

If, as a result, the TIOBE Index is affected significantly, then I simply hope they’ll drop their pretentious claims and state clearly exactly what they’re counting, how they’re doing it, and what it means: not much.

1. Many thanks to Barry Walsh for his blog post that alerted me to this.

I’ve been thinking that I haven’t blogged much lately. Assorted half-baked ideas would cross my mind and then evaporate before I’d find the time, or motivation, to actually start writing.

The folks at the Enlightened Perl Organisation have solved the motivation problem by announcing the Iron Man Blogging Challenge: in short, “maintain a rolling frequency of 4 posts every 32 days, with no more than 10 days between posts“.

So about one post a week. I can aim for that!

Can you? “The rules are very simple: you blog, about Perl. Any aspect of Perl you like. Long, short, funny, serious, advanced, basic: it doesn’t matter. It doesn’t have to be in English, either, if that’s not your native language.” Why not try? Help yourself and help perl at the same time.

I’ll try to capture the half-baked ideas for perl blog posts as they cross my mind, then build on them as time and mood allow. Hopefully about one a week will mature into an actual blog post.

Meanwhile…

I remembered that almost exactly a year ago schwern blogged that he was “horrified at the junk which shows up when you search for perl blog on Google“. It seems the situation hasn’t improved much since. The planet.perl.org site is top, but the rest are a bit of a mishmash.

The problem is partly that “perl blog” isn’t a great search term. Google naturally gives preference to words that appear in urls and titles (all else being equal), but blogs rarely explicitly call themselves blogs on their pages or urls. I suspect many on the first page of results are there because ‘blog’ appears in the url. (To help out I’ve included “perl blog” in the title of this post :)

Searching for “perl blogs” (plural) works better because it finds pages talking about perl blogs, which is useful when searching for perl blogs.

One entry in the “perl blogs” results was the Perl Foundation’s wiki page listing perl blogs. That was new to me. This blog wasn’t on it so I’ve added it. Got a perl blog, or know someone who has, that’s not listed on that page? Go and add it, now! It’ll only take a moment.

Along similar lines, I’ve added the phrases “perl blog” and “perl programming” to the sidebar of my blog pages. The first is to help people searching for “perl blog”. The second is mostly for my own amusement.

I recently came across this article on slimming down an iPhoto library by removing the ‘Original’ photo where a ‘Modified’ one existed.

(That’s one part of what the free iPhoto Diet app does, but it seems that’s not being maintained and doesn’t support recent versions of iPhoto.)

Inspired by the basic three-line shell script in the article I worked up this somewhat more advanced version:

#!/bin/sh -ex
cd ~/Pictures/
du -sh 'iPhoto Library'
dest=$HOME/Pictures/iPhoto-Redundant-Originals-`date "+%Y-%m-%d-%H%M"`
# for all files in iPhoto Library/Modified/...,
# move the corresponding iPhoto Library/Originals/...
# files into a zip archive file
find 'iPhoto Library/Modified' -type f -print \
    | perl -pe 's{iPhoto Library/Modified/}{iPhoto Library/Originals/}' \
    | zip -9 -T -m $dest -@ 2>&1 \
    | grep -v 'zip warning: name not matched'
du -sh 'iPhoto Library'

It has some advantages over the original: the photos are moved into a unique zip file, rather than the trash, and the file hierarchy is preserved, so files can be restored easily.

It could be modified to only operate on a subset of files, such as those older than a certain age.

Extra notes:

1. After running this you’ll find that the photo will appear black in the ‘edit view’. That’s not a problem. The ‘edit view’ (which you enter by double clicking on a photo, for example) reconstructs the final image by taking the original and reapplying the edits-so-far. Since the original file has been removed you’ll just see a black image. Don’t worry. In all the other views, and for printing etc., your final modified picture will appear perfectly.
2. iPhoto handles automatic rotation of images from cameras that record their orientation by performing a rotation ‘edit’ for you when you import the image. That rotation creates a modified copy, and that’s a common cause of bloat in your iPhoto library. It’s also why you may be surprised to see some originals being archived even though you haven’t editied them. It would be nice if iPhoto had an option to handle rotations destructively.

Usual caveats: this worked for me, your mileage may vary, cross your fingers, read the referenced article (and follow the links in contains), read the comments on them, quit iPhoto, take a backup, wear a tinfoil hat.

In the spirit of re-tweeting, this is a short post to highlight some great examples of “modern perl”. (I’m using the term modern perl very loosely, not referring specifically to any one book, website, or module or organization.)

Firstly I’d like to highlight a couple of recent posts by Jonathan Rockway:

* Unshortening URLs with Modern Perl. An interesting example application built with modern perl modules like MooseX::Declare, MooseX::Getopt, HTTP::Engine, AnyEvent::HTTP, TryCatch, and KiokuDB.

* Multimethods. Another great example from Jonathan highlighting the combined power of MooseX::Types, MooseX::Declare, and MooseX:: MultiMethods.

Then, from his work at the BBC, Curtis “Ovid” Poe has given us a great series of thoughtful posts on the benefits of replacing multiple inheritance with roles in a complex production code-base. The slides of his Refactoring With Roles talk is a good place to start. Then dive in to the blog posts back here and work your way forward.

I ♥ modern perl!

Seeing this video of treemap for perlcritic memory usage reminded me of something…

I’d really like to be able to use treemaps in NYTProf reports to visualize the time spent in different parts, and depths, of the package namespace hierarchy. Currently that information is reported in a series of tables.

A much better interface could be provided by treemaps. Ideally allowing the user to drill-down into deeper levels of the package namespace hierarchy. (It needn’t be this flashy, just 2D would be fine :-)

In case you’re not familiar with them, treemaps are a great way to visualise hierarchical data. Here’s an example treemap of the disk space used by the files in a directory tree (from schoschie)

Perl already has a Treemap module, which can generate treemap images via the Imager module. Treemap is designed to support more output formats by sub-classing.

I guess it wouldn’t be hard to write a sub-class to generate HTML client-side image map data along with the image, so clicks on the image could be used to drill-down into treemaps that have more detail of the specific area that was clicked on.

More interesting, and more flexible, would be a sub-class to generate the treemap as a Scalable Vector Graphics diagram using the SVG module (and others).

I’m not going to be able to work on either of those ideas anytime soon.

Any volunteers?

I needed to install a perl extension from a third-party. It’s an interface to a shared library, a .so file, that they also supply.

Normally I’d add the directory containing the shared library to the LD_LIBRARY_PATH environment variable. Then when the extension is loaded the system dynamic loader can find it.

For various reasons I didn’t want to do that in this case.

An alternative approach is to pre-load the shared library with a flag to make it’s symbols globally available for later linking. (The flag is called RTLD_GLOBAL on Linux, Solaris and other systems that use dlopen(). This hack may not work on other systems.)

But how to pre-load the shared library, only when needed, and without changing any existing perl code? This is where the pesky little .bs files that get installed with perl extensions come in handy.

They’re known as ‘bootstrap’ files. If the .bs file for an extension not empty then DynaLoader (and XSLoader) will execute the contents just before loading the shared object for the extension.

So I put code like this into the .bs file for the extension:

    use DynaLoader;
    DynaLoader::dl_load_file("$ENV{...}/...so", 1)
	    or die DynaLoader::dl_error();

Not a recommended approach, but neat and handy for me in this case.

I’ve uploaded Devel::NYTProf 2.09 to CPAN.

If you’re using VMS the big news is that Peter (Stig) Edwards has contributed patches that enable NYTProf to work on VMS. Yeah! Thanks Peter.

For the rest of us there’s only one significant new feature in this release: NYTProf now includes a heuristic (that’s geek for “it’ll be wrong sometimes”) to handle modules using AutoLoader. The most common of which are Storable and POSIX. You may have encountered a warning like this when running nytprofhtml:

Unable to open '/../../lib/Storable.pm' for reading: No such file or directory

It’s a symptom of a deeper problem caused by AutoSplit, the companion to AutoLoader. The details of the cause, effect, and fix aren’t worth going into now. If you’re interested you can read my summary to the mailing list.

The upshot is that NYTProf now reports times for autoloaded subs as if the sub was part of the parent module. Just what you want. Time spent in the AUTOLOAD’er sub is also reported, naturally.

There is a small chance that the heuristic will pick the wrong ‘parent’ module file for the autoloaded subroutine file. That can happen if there are other modules that use the same name for the last portion of the package name (e.g., Bar::Foo and Baz::Foo). If it ever happens to you, please let me know. Ideally with a small test case.

The list of changes can be found here (or here if you’re a detail fanatic).

One other particularly notable item is that the savesrc option wasn’t working reliably. Thanks to Andy Grundman providing a good test case, that’s now been fixed.

Enjoy!

I usually get a cold around January. Nothing major, gone in a week or so. This year I excelled myself by succumbing to the ’flu and following it up with assorted complications. January was mostly a write-off, February was better but still a struggle (what with jet lag and unexpected surprises). Fortunately my Functional Biochemistry practitioner has pinpointed the underlying cause for the slow recovery and I’m fast returning to my usual good health.

My wife suggested I take some time-out for myself. After the rough few weeks I needed a boost, so I did…

I’m just back from a wonderful two day retreat organized by Timothy and Eva from the Kalyana Centre for Mindfulness in Co. Kerry.

The retreat was held in silence, practising Kum Nye, a Tibetan Buddhist movement practice. It was lead by Matthias Steurich — a rare opportunity to practice with him in Ireland. It was wonderful. Just what I needed.

The location was beautiful. A large house overlooking Inch Strand in County Kerry in the West of Ireland. I took this photo on my iPhone on one of my walks along the beach:

I’ve uploaded the screencast of my NYTProf talk at the London Perl Workshop in November 2008.

It’s based on a not-quite-2.08 version and includes some coverage of an early draft of the ‘timings per rolled-up package name’ feature I discussed previously.

It also shows how and why anonymous subroutines, defined at the same line of different executions of the same string eval, get ‘merged’.

The demos use perlcritic and Moose code. It also includes a nice demonstration showing NYTProf highlighting a performance problem with File::Spec::Unix when called using Path::Class::Dir objects.

It’s 36 minutes long, including a good Q&A session at the end (wherein a market rate for performance improvements is established). Enjoy.

I’ve just released NYTProf 2.08 to CPAN, some three and a half months after 2.07.

If you’ve been profiling large applications then the first difference you’ll notice is that the new version generates reports much faster.

The next thing you may notice is that statement timings are now nicely formatted with units. Gisle Aas contributed the formatting code for 2.07 but I had to do some refactoring to get it working for the statement timings.

Another nice refinement is that hovering over a time will show a tool-tip with the time expressed as a percentage of the overall runtime.

Almost all the tables are now sortable. I used jQuery and the tablesorter plugin for that. I’ve not added any fancy buttons, just click on a table heading to sort by that column. You’ll see a little black arrow to show the column is sorted. (You can hold the shift key down to add second and third columns to the sort order.)

A profiler isn’t much use if it’s not accurate. NYTProf now has tests for correct handling of times for string evals within string evals. In fact the handling of string evals got a big overhaul for this version as part of ongoing improvements in the underlying data model. I’m working towards being able to show annotated performance reports for the contents of string evals. It’s not there yet, but definitely getting closer.

A related feature is the new savesrc=1 option. When enabled, with a recent version of perl, the source code for each source file is written into the profile data file. That makes the profile self-contained and, significantly, means that accurate reports can be generated even after the original source files have been modified.

Another new option is optimize=0. You can use it to disable the perl optimizer. That can be worth doing if the statement timings, or counts, for some chunk of code seem odd and you suspect that the perl optimizer has rewritten it.

The final new feature noted in the NYTProf 2.08 Changes file is that it’s now possible to generate multiple profile data files from a single application. Since v2.0 you could call DB::disable_profile() and DB::enable_profile() to control profiling at runtime. Now you can pass an optional filename to enable_profile to make it close the previous profile and open a new one. I imagine this would be most useful in long running applications where you’d leave profiling disabled (using the start=none option) and then call enable_profile and disable_profile around some specific code in specific situations – like certain requests to a mod_perl app.

There’s one more new feature that I’ve just realised I’d forgotten to add to the Changes file before the release: Timings per rolled-up package name. What’s that? Well, it’s probably easiest to show you…

These images are taken from a profile of perlcritic. Each shows the time spent exclusively in subroutines belonging to a certain package and any packages below it. Hovering over a time gives the percentage, so I can see that the 57.3s spent in the 36 PPI packages accounted for 42% of the runtime.

This gives you a quick overview for large (wide) codebases that would be hard to get in any other way.

Tables are generated for upto five levels of package name hierarchy, so you can drill-down to finer levels of detail.

I can visualize a much better UI for this data than the series of tables nytprofhtml currently produces, but my limited free time and jQuery skills prevent me doing more. Patches welcome, naturally.

Enjoy!

p.s. I’ve a screencast from my NYTProf talk at the London Perl Workshop in November I hope to (finally) upload soon. It includes a demo of the package roll-up timing tables.

My first Open Source spin-off from TigerLead.com: UltraDNS a Perl client API for UltraDNS.

I’ve a new release of NYTProf ready to upload but I’m stuck.

The CPAN Testers service is reporting a failure on a number of systems but I can’t reproduce it locally or work out the cause.

Can you reproduce the failure with Devel::NYTProf 2.07_94? If so, could you give me remote access via ssh? (Or spare some time to investigate yourself – I’ll happily clue you in if you can reproduce the problem.)

Update: No one could reproduce it. It seems that the failures was not what it appeared to be. A clue was that only one tester was affected. Devel-NYTProf-2.07_94.tar.gz unpacked itself into a directory called Devel-NYTProf-2.07. It seems that when using CPANPLUS if the user had already got an old Devel-NYTProf-2.07 directory its contents got merged and tests would fail. I’m not convinced that’s the whole story, but Devel-NYTProf-2.07_95.tar.gz unpacked into a Devel-NYTProf-2.07_95 directory and didn’t run into the problem.

Update: More usefully, Andreas made my wish come true by pointing out the –solve parameter to the ctgetreports utility in his CPAN::Testers::ParseReports distribution. It “tries to identify the best contenders for a blame using Statistics::Regression. [...] The function prints the [...] top 3 candidates according to R^2 with their regression analysis.” Cool.

After almost exactly two years working for Shopzilla I’ve moved on.

I’ve enjoyed my time with Shopzilla, met some great people, and worked on some challenging problems. I’m especially grateful to them for giving me the opportunity to develop and release some very useful Open Source software: Gofer, DashProfiler, and especially NYTProf v2.

When I started with ValueClick back in 1999 it was a small company. Over the years I was with them it grew rapidly, both from within and via mergers. When I left in 2006 it had become a very large company. For me Shopzilla was always a very large company.

I really like to make a significant positive impact where I work. I believe I did that at ValueClick and Shopzilla, but it’s always easier at a smaller company. There’s far less inertia to overcome.

So I’m delighted to be working now at a small company with a great team, product, technology, approach, and growth: TigerLead.com

I’m not keen on job titles, but I needed to put something on my LinkedIn profile. After exploring some options I settled on “Entropy Minimizer”!

I’m also delighted to be working with the PostgreSQL database, at last, as I’ve not had a chance to work with it before. (Although I used to be good at tuning Ingres, the distant forerunner of PostgreSQL, back in the days when RDBMS were novel and we thought a 50MHz cpu was fast.)

When arranging trips I used to manually enter the main details (dates, times, confirmation numbers, hotel addresses and phone numbers etc.) into iCal and forward confirmation emails to my wife so she’d have a copy if she needed them.

Then I discovered Tripit. Now I simply forward confirmation emails as they arrive to plan@tripit.com – something I could easily automate. My iCal automatically sync’s with a feed from tripit that has much more information than I’d ever enter manually, including links back to tripit.com for the full details. That then syncs to my iPhone so the information is always at hand. Perfect, and perfectly simple.

As I’ve found the service very useful over many months and many trips, I recently sent tripit invites to a bunch of people, mainly Perl folk. Many joined. A few pointed out that they already used dopplr and wondered if tripit was the same. I didn’t know, so I took a look.

I signed up with dopplr and was happy to see that, like tripit, I could forward confirmation emails to dopplr. So I forwarded the flight and hotel emails for my upcoming London Perl Workshop trip. Just like tripit, dopplr parsed the emails and worked out I was making a flight from Shannon to London and staying in a hotel in London.

That was all it did, though. It hadn’t noted my confirmation codes, flight times, hotel address. It had remembered nothing that would be useful to me.

All it had done was noted that I’d be in the same city as a few other people I know who use dopplr. tripit does that, but also let’s me control who sees which trips and who can collaborate with me on trips.

I did a quick search and found another review that reached a similar conclusion. It seems that dopplr are better at publicity, but tripit has the features I find genuinely useful.

To give it tripit a whirl, all you need to do is forward some recent confirmation emails to plan@tripit.com – you don’t even need to signup first.

NYTProf development rolls on, with 2.07 being another significant release.

Here’s a summary of highlights from the many improvements.

Windows, at last

Windows wizard Jan Dubois has contributed patches that enable NYTProf to run on Windows. Many thanks Jan!

Corrected Subroutine Total Time

A significant error in time reported as spent in a subroutine has been fixed. It was showing the sum of the inclusive and exclusive time instead of just the inclusive time.

String Evals

One of NYTProf’s innovative features is tracking of subroutine calls by calling location. In earlier versions you couldn’t see calls made from within string evals. Now you can. Even for evals within evals:

That’s a big step forward for profiling code that makes heavy use of evals.

(The use of the µ micro symbol is also new – part of some helpful interface polishing contributed by Gisle Aas. They’re not used on the statement timings yet, but will be at some point.)

Recursion

Previously the ‘inclusive time’ reported for a subroutine wasn’t useful for subroutines called recursively. The overall inclusive time included the inclusive time of nested calls, typically inflating it beyond usefulness.

Now the inclusive time only measures the inclusive time of the outermost calls. NYTProf now also tracks and reports the maximum call depth for each calling location.

Disabling the Statement Profiler

NYTProf started life as a statement profiler. I added a subroutine profiler later and was delighted with how well that worked out. The subroutine profiler is novel in a few ways and also extremely robust and efficient. Part of the efficiency comes from not streaming data to disk while the program is running. It accumulates the counts and timings in memory.

When profiling large and/or performance sensitive applications the costs of NYTProf’s statement profiler, in performance impact and disk space, can be prohibitive. So now I’ve added an option to disable the statement profiler. Setting the stmts=0 option turns NYTProf into a very lightweight subroutine profiler.

Caller Information for XSubs

NYTProf reports time spent in subroutines broken down by calling location. It’s one of its key features. The caller details are embedded as comment annotation into the report for each perl source code file.

That has worked very well for perl subs, but there’s no natural home for XS subroutines because there was no source code file available to annotate.

There’s a partial solution included in NYTProf 2.07: XS subs that are in the same package as profiled perl subs, get annotated subroutine ’stubs’ appended to the report for the corresponding perl source file.

It works quite well, in particular the common UNIVERSAL methods get annotated:

Also, now these subs have a place in the reports, any references to them are hyper links, naturally.

Sadly it doesn’t work so well for packages that don’t contain any perl subs, like DBI::st. I plan to add per-package report pages for them in a future release.

Clock Docs

I’ve greatly expanded the description of the various clocks available for profiling: POSIX real-time clocks, gettimeofday(), and Time::HiRes.

I’ve also added a caveats section that discusses issues with SMP systems, processor affinity, and virtual machines.

Feedback Wanted

How has NYTProf helped you optimize your applications? Let me know.

Perhaps as blog posts of your own, linking back here, or comments here, or CPAN ratings, or email to the mailing list or just to me. I’d love to hear from you.

Thanks!

Hot on the heels of NYTProf 2.04, I’ve just released NYTProf 2.05. It has a few small but significant bug fixes. From the Changes file:

  Fixed alteration of $! during profiling, thanks to Slaven Rezic.
  Fixed clock_gettime() clock selection to fallback to CLOCK_REALTIME
    if CLOCK_MONOTONIC is not available, e.g, on linux 2.4.
  Fixed error when application ends after DB::disable_profile().
  Added some docs to Devel::NYTProf::Apache
  Added clock=N option to enable user to select the clock.

All the bugs were long-standing ones that happened to come to light and get fixes after 2.04 was released.

The first was the cause of a couple of odd bugs were applications would behave differently when profiled. The problem was that in certain situations the profiler would alter the value of $! between statements, so code that tested for $! would think something had gone wrong.

The second caused weird timing values on some platforms. The statement times would be zero and the subroutine times would be garbage. The problem was that some systems, such as Linux 2.4, define the CLOCK_MONOTONIC macro but don’t implement that clock in the kernel. NYTProf wasn’t testing that the clock worked. Now it does. If the clock_gettime() call fails with the selected clock it falls back to CLOCK_REALTIME (and if that fails NYTProf aborts).

As a bonus I’ve added the ability to select what clock you want to use by setting clock=N in the NYTPROF environment variable, where N is the integer id of the clock you want to use (usually you can find the CLOCK_* macros in the /usr/include/time.h file). This only works if your system supports clock_gettime(), which most modern unix-like systems do, with the notable exception of OS X.

Why would you want to use a different clock?

Well the default, CLOCK_MONOTONIC, is relatively expensive to call, and NYTProf calls it twice per statement. So it’s the dominant cost of the profiler.

Some systems, like Linux 2.6, offer other clocks, like CLOCK_MONOTONIC_FAST that returns the most recently cached value of the clock. That’s typically just 1/10th the cost of CLOCK_MONOTONIC, so your code runs much faster under the profiler.

Naturally there’s a trade-off: the accuracy of the timings drops significantly. Still far better than Devel::DProf, but far worse than normal. You’re dependent on how often the cached time value gets updated by your kernel. I saw a figure of “600 times per second” quoted for one system, but I can’t find the link now. My advice would be to steer clear of this feature unless you feel comfortable with the issues like these.

Enjoy!