The Making of MarkMail

Welcome community plumbers (aka Drupal)

2010-01-15T19:36:00.001-08:00

Towards the end of last year we loaded up the Drupal archives and subscribed to their active lists. Drupal is a pretty successful project with lots of real-world use and we figured they might be reasonably chatty online.

As preparation for this post, I checked out the Drupal front page. I love a good analogy and their HTML <title> but a good smile on my face:

    <title>drupal.org | Community plumbing</title>

It made me think we might re-consider the MarkMail title, given what we do. MarkMail helps organize development communities and so my first thought was:

    <title>markmail.org | Community organization</title>

More specifically, we help with project histories so I changed it to:

    <title>markmail.org | Community histories</title>

And then, perhaps best of all, I landed on:

    <title>markmail.org | Community libraries</title>

And as the chart above shows, these folks are pretty chatty and have been so since about 2004.

So here's a hearty welcome from us at MarkMail

to the busy folks over at Drupal

The wide world of Ubuntu: 1.9 million messages

2010-01-13T10:48:00.000-08:00

A few months ago, we loaded up the publicly available Ubuntu archives and subscribed to their active lists. MarkMail now searches 313 Ubuntu lists and 1,917,153 messages. The first Ubuntu list started in July 2004 and there are currently 200 active lists, recently accumulating 4,061 messages per day.
Ubuntu is very much a world-wide project. Below is a recent snapshot of the message counts for Ubuntu lists associated with specific counties:

Agentina	26,923
Austria	5,597
Brazil	67,940
Columbia	20,758
Germany	19,438
Spain	41,489
Italy	43,618
Nicaragua	11,761
Russia	19,321
United Kingdom	21,995

Magic Eraser Policy

2009-11-08T19:35:00.000-08:00

If you've been around email long enough, you know there's a reason that gmail has that magic undo send feature. (Well, ok, once you see what it is, it's not that magic).

Because of the nature of digital information, it's too easy to have that email end up somewhere you'd rather not see it. Once you hit send, you don't have much control over the copying process. That email immediately becomes part of your permanent record. And, well, you've probably been there yourself at one time or another, wishing you hadn't sent it.

Even though there are a ton of sources on email etiquette. it's not too surprising that folks send email they wish they hadn't. We get weekly evidence of that. Like search engines and other popular archives, we get our share of requests to remove content. Sometimes folks ask for individual emails to be removed. Other times, we get more open-ended requests like, "please remove my name from your site".

The funny thing is that mailing list tools aren't really set up for redaction. They don't make it easy to remove individual emails, let alone parts of emails, from their archives. So, what does MarkMail do with this issue?

MarkLogic Server to the rescue!

One of the cool things for us, is that MarkMail actually has a magic eraser. With little pain, thanks to the real-time index provided by our MarkLogic server, we can remove emails from our index at a moment's notice. With a single line of XQuery code, we can move the document representing that email into a hidden collection and in nanoseconds, it's gone from all further queries to MarkMail. Yes, that is neato.

So we have the technology.

Still, we don't take message removal lightly. As an authoritative record of public history, removals are treated as the exceptional case. MarkMail provides content that it receives from publicly available sources. Everything we serve, we received another source. List administrators control their own lists and set policies on archives and we respect that. By posting on their list, you follow their rules, and we do too. So, if you want something removed from MarkMail, you'll usually have to get it removed from the original source first.

We've recently added our removal policy to the site, which has full details on how we deal with removal requests. The mechanism starts at our feedback page. In a nutshell, we will remove emails under two specific cases:

Content in Violation of the MarkMail Content Policy (e.g. spam, porn, virus, fraud, illegal activities, copyright violations)
Content Removed from Official Archive

When content clearly violates our content policy, we may simply remove it. No surprise there. That's bad stuff. Any other material needs to be removed from the original source before we'll act. In all cases, we prefer requests from the archive owner and/or requests that come with evidence that the content has been removed from the originating source.

This policy attempts to balance the rights of many parties, including those who have posted content to public lists and forums, those who own and administer the public lists and forums, and those who link to and reference the MarkMail archives as an information source and accurate record of history. We hope you find it reasonable.

Easy Change

2009-11-04T09:32:00.000-08:00

As we look to make MarkMail pay for itself, it's pretty darn obvious that the traffic the site gets is sufficient to generate some revenue from advertising. A good number of well-known developer-centric sites display ads (e.g., sourceforge.net, www.xml.com, vim.org, linux.org, many others) with varying degrees of success. It even looks like msdn.com displays ads (albeit for other MS properties). And there are also sites like Expert Exchange that charge users a premium to search their archives, while also displaying ads.

The plan is to make minor adjustments to the site layout and provide relevant ads that are meaningful to a software developer. Over time, we'll be looking to bring in ad content that is

targeted to the list archive the developer is reading
suitable for a library (quiet, text-only, unobtrusive)
designed for and targeted to a developer audience

We'll be enabling Google AdSense text-based ads soon, as they appear to be best-of-breed, simple to implement. But we'll also be looking to sell advertising space ourselves.

In the meantime, before we're fully set up, if you're reading this and you have something worth communicating to some of the millions of folks that end up at MarkMail, give us a holler.

PS. If you sign up for a MarkMail account, you'll have access to a switch that will enable you to opt out of the AdSense ads.

The New Guy

2009-10-08T16:17:00.000-07:00

Hi Folks,

I'm the new guy and this is my inaugural post.

I first ran into MarkMail a few years ago, during my tenure at Clearwell Systems. Back then, both Clearwell and MarkMail were building "search engines for email". If you didn't look deeply, you'd have thought we were competitors. But we weren't. And we aren't, still.

At Clearwell, we knew we were breaking ground. Like many start-ups, we really didn't preconceive which applications our efforts would enable and which markets would find our solutions valuable. We just believed, with all our hearts and minds, that there was something there and we could get it done. I've since learned the history isn't too different at MarkMail. To make a long story short, Clearwell went one way, focusing on enterprise email environments, like Microsoft Exchange, PST Files, and Lotus Notes, eventually cracking the nut on common Electronic Discovery use cases. We built a really sweet e-discovery product that continues to get rave reviews and save customers $$$.

And, MarkMail went another, supporting public, open-source communities and mailing lists (MailMan, Ezmlm, Google Groups, and others). Through these efforts, MarkMail has become a large scale, highly respected, and high-traffic service for software developers. Ironically enough, as a software developer, it was not uncommon for me at Clearwell to ultimately use MarkMail. So I've been a fan for quite sometime.

And now, thanks to the hard work of good folks at Mark Logic (and Jason Hunter in particular), I'm here to help further the MarkMail mission. I bring to MarkMail, deep experience in software development practice, high-performance computing, and user interface engineering (OpenLaszlo, LaszloMail). For a good percentage of this time, I've been in and around all sorts of email, communication, and collaboration tools, especially those used by developers. And in my most recent gigs, I've been focused on making sure my engineering efforts are part of a broader business success. And I plan to do the same for MarkMail.

So... to take a line from Bette Middler, "Enough about me." What am I going to do for MarkMail? Well, a lot I hope. In the immediate term, I've got a few obvious directives, like keeping the site going and growing. I’ll also be focusing on the changes needed to make the site responsible (aka pay) for its own operations, while keeping a focus on the general communities that it serves. To that end, you can expect another post (or two) about upcoming changes.

Sounds like fun, eh? Well, you'll get to hear it all, assuming I continue to be able to make time for blog posts like this one. Thanks for listening and please holler at me with advice and comments. And, of course, if you have feedback or ideas related to how MarkMail might help you, please holler.

Oh... and one more thing. If you're in the Bay Area and you want to see me in person, you can also find me occasionally playing out with the Tribal Blues Band (I'm the one in the blue shirt).

MarkMail at the first MarkLogic User Group

2009-06-23T14:09:00.001-07:00

Last week I spoke at the inaugural Mark Logic User Group meeting in Reston, VA (near where a lot of our government customers are based). The topic was MarkMail: where the idea came from, how we built it on the cheap, how Mark Logic began using it internally, and some lessons we learned as we scaled out the public high-traffic site. It's a similar talk to the one I gave at the Mark Logic User Conference in San Francisco last month.

For those interested, the slides are available as a downloadable MOV file. Click to advance.

The slides are fairly simple. Most of the fun of the talk (well, at least for me) is in the stories I tell, usually relating to the quotes in italics at the bottom of slides. I suppose you'll just have to use your imagination.

MarkMail at the Mark Logic User Conference

2009-05-05T17:53:00.000-07:00

The Mark Logic User Conference is coming up next week. If you're coming to the show, I encourage you to attend the talk on MarkMail I'll be giving on Wednesday. I'll tell the story of MarkMail as it progressed from my first idea to a night project built with Ryan Grimm to the robust web site you see now at markmail.org (and even to the other web sites you don't see, because they're running behind people's firewalls). It's in the conference's technical track so there'll be a lot of focus on the core tech.

If you're not coming to the show, why the heck not? :) It's not too late to register.

MarkMail at One Year: Looking Back

2008-12-09T16:27:00.001-08:00

It's now been a little over a year since we launched MarkMail. We've sure come a long way!

We're now seeing well over a million unique visitors every month and more than 5 million page views.

The Googlebot crawler (whose activity isn't included in the above statistics) has also been active. It now crawls between 1.0 and 1.3 million pages every day to keep its index fresh. That's about 15 page hits every second -- or 15 Hertz, enough to make a nice low background rumble noise. It's really enjoyable to get so much Google attention because it wasn't that long ago when we were just trying to get Google to index more than a million of our pages, nevermind crawl that many in a day.

Our content size has grown also, from 4 million messages at launch, covering just the Apache Software Foundation archives, to 34 million messages today, spanning all sorts of communities. For us to grow so big so fast has been possible only because of the community support we've received. There's a long list of various community members who have worked with us to accumulate and load their list archives. We'd like to thank all those folks, as well as the people who placed a MarkMail search box or other MarkMail link on their site or helped spread the word in blogs and emails and tweets.

Looking forward, where do we go from here? We have some big plans. I'll get into details with a later post.

Google Code Adds Gadgets: MarkMail Helps

2008-10-09T00:14:00.001-07:00

Google today announced new support for embeddable "gadgets" on Google Code project pages. Particularly exciting to us, they introduced MarkMail as the recommended gadget for viewing and searching Google Code project list archives.

For those who haven't encountered one in the wild, a Google Gadget is an embeddable web object that puts a bit of third-party dynamic content into the middle of a web page. Gadgets are the things you place on your iGoogle home page or your Google Desktop, but you can also add them to your own web page with one line of JavaScript, or anyone else's page if it supports the OpenSocial APIs.

We've coordinated with the Google Code team over the last several months to load about 500 GoogleGroups lists (3.8 million emails) and build a new MarkMail Gadget (launching today!) to let Google Code developers search and analyze their lists using MarkMail.

The new MarkMail gadget lets you view messages, threads, attachments, and senders, and a traffic chart (wouldn't be MarkMail without it!) for any set of messages you want to follow. The messages you choose to track with the gadget can be those from a single list, set of lists, a person, containing a term or phrase, or any combination. In fact, anything you can use in a search on MarkMail can be used as input to the gadget view. The new gadget offers two features not yet available on MarkMail.org: a daily traffic chart (MarkMail.org only does monthly traffic charts) and a view that coalesces threads.

So what does this mean for you? If you're a project leader (either on Google Code or somewhere else) it's now easier than ever to embed a MarkMail traffic chart and recent message list inside any of your project pages. If you're just a lurker, you can personalize your view on MarkMail traffic and embed that view into iGoogle or Google Desktop, or any other page.

To help you set up the right links, we created a Gadget Embedding Wizard that guides you through the process of embedding. You can also find our gadget in the Google Directory where they have additional embedding instructions.

Tim O'Reilly in describing Web 2.0 says, A platform beats an application every time. We agree. We think you should be able to access mailing list archives whenever and wherever you want, be it at MarkMail.org or on another page that's been MarkMail-enabled via a gadget. So have fun, and let us know how this works for you!

1.4% of Emails Mention Google

2008-10-02T18:07:00.000-07:00

As Google celebrates its 10 year anniversary we thought it'd be fun to use our archive of 30 million mailing list messages to see how Google's popularity has grown over time across the list-o-sphere. Boy has it grown!

In 2008 (so far) the word "Google" appears in 1.4% of emails in our archive, up from 1.15% last year and 0.75% five years ago.

While shockingly high, that 1.4% number is actually calculated with some conservative restrictions. We're excluding all mentions that occur inside quote blocks (where someone replies to another who said the word). It'd be 2% if we didn't have that rule. We're also excluding from our calculations all the Google Groups lists we follow, where Google is often the topic of discussion. With those lists added in? It's 13%.

You can explore this yourself with our public interface. You'll want to query for "google", use the "opt:noquote" flag, and set "-list:googlegroups" to exclude those lists. Then you can add date constraints either by typing "date:2008" in the search or dragging on the chart. Track the numbers as a result of your searches, do a little division, and you get your percentages.

You'll see that so far in 2008 there were 50,826 emails saying "google" across 3,607,973 emails total. That's 1.4%. For 2003 it's 21,165 emails out of 2,770,480 total, or 0.75%.

Ruby on Rails on MarkMail: 200,000 Emails

2008-09-22T15:46:00.000-07:00

Interested in Ruby on Rails? If so, you'll be happy to learn we've loaded the full RoR mailing list archive. It holds about 200,000 emails and includes both the original Mailman lists from 2004-2006 and the GoogleGroups lists from 2006 onward.

Fun facts:

Frederick Cheung is the #1 most frequent poster
DHH is #22
The traffic never fully recovered after it transitioned from rubyonrails.org to GoogleGroups. You can compare the two charts (keep an eye on the y-axis).
Maybe it's because DHH didn't make the move to GG?

Don't forget, we have the regular Ruby lists too.

FreeBSD, the Unknown Giant

2008-09-11T16:17:00.000-07:00

My last entry about NetBeans and OpenOffice.org and their million messages reminded me that I've never announced here our load of the FreeBSD archives, an even larger and older community. They have more than 2.5 million messages, stretching back to 1994.

FreeBSD doesn't get as much attention at Linux but is a great operating system. Here's a description from an IBM developerWorks article:

"The FreeBSD operating system is the unknown giant among free operating systems. Starting out from the 386BSD project, it is an extremely fast UNIX®-like operating system mostly for the Intel® chip and its clones. In many ways, FreeBSD has always been the operating system that GNU/Linux®-based operating systems should have been. It runs on out-of-date Intel machines and 64-bit AMD chips, and it serves terabytes of files a day on some of the largest file servers on earth."

Here's the historic traffic chart (excluding automated bug and check-in messages):

Looks like it's a giant in traffic as well. The freebsd-questions list alone gets a couple thousand emails a month, half a million in its history. Got a FreeBSD problem? I bet the answer's in there.

Announcing NetBeans and OpenOffice.org

2008-09-11T15:50:00.000-07:00

Last week we finished adding the NetBeans and OpenOffice.org mailing lists to the MarkMail archive. Both communities host more than a million messages each. Here's the NetBeans activity graph (with automated bugs and check-in messages removed):

Looks like they've seen a resurgence in activity going up for the last 4 years. They have more list activity than Eclipse, too. (Eclipse directs user questions to web forums that aren't included in our stats.)

Here's the OpenOffice.org traffic (same automated message removals):

The folks at CollabNet worked with us to transfer the massive archives, and yesterday we issued a joint press release announcing the new list availability. We also boasted passing 27.5 million emails in total. That was yesterday. Today we're passing 28 million. Chugga, chugga!

A Tale of Two Search Engines, Revisited

2008-09-02T13:40:00.001-07:00

As Jason announced previously, last Wednesday night I delivered
A Tale of Two Search Engines — a presentation for the Software Architecture and Modeling SIG of SDForum — about building and running the Krugle and MarkMail vertical search engines for code and email, respectively.

Here are my tidied-up slides.

Note carefully that my presentation style is a very visual, story-telling approach for live, interactive audiences -- i.e., the slide deck is quite large and not geared towards a reading-at-home audience. Heck, I only broke down and used bullet points on 4 slides right at the end. :-)

That said, I'll start blogging some of the stories, go deeper on various technical details, and/or get into any of the "fun topics" that people are interested in. Feel free to leave comments here about any that you particularly want to hear about.

Special thanks to Ron Lichty for dragging me into giving this presentation and the wonderful SAMSIG audience for making it so much fun.

Enjoy,

John

Loaded Red Hat: A Thousand Emails a Day

2008-08-25T15:14:00.000-07:00

How did we celebrate the success of our memory swap ballet last week? We loaded 1.7 million Red Hat emails. It's geeky, but so are the lists! We've now got the complete set archived.

The first Red Hat messages start back in May 1996. At that time then there were just a few hundred emails each month. The chatter has grown a lot since, with recent numbers topping 30,000 messages a month. That's 1,000 per day.

It's interesting to see the #1 most common file attachment is of type patch. That makes sense as these are mostly developer lists.

But can anyone explain why on a Linux list the #2 most common attachment is the Outlook-generated winmail.dat!? Is that a good sign or bad sign?

How to Shutdown All Your Machines Without Anyone Noticing

2008-08-19T17:34:00.001-07:00

Last week we discovered we had to replace some bad memory chips in 2 of the 3 machines we use to run the MarkMail service. This blog post tells the story of how we managed to replace these memory chips without (almost) any of our visitors noticing.

Architecture

First, a word about our architecture. The three machines I'm talking about here all run MarkLogic Server. We have some other machines in the overall MarkMail system that do things like queue and handle incoming mail, but they're not directly involved in the web site experience. I'm talking here about the three MarkLogic machines that work together in a cluster and that you interact with when you hit http://markmail.org.

The MarkLogic machines have specialized roles. One machine (I like to picture it up front) listens for client connections. It's responsible for running XQuery script code, gathering answers from the other two machines, and formatting responses. The other two machines manage the stored content, about half on each. They support the front machine by actually executing queries and returning content.

I'll refer to the front machine as E1, which stands for evaluator #1. We don't have an E2 yet but we're planning for that when user load requires. The back-end machines are D1 and D2, which stands for data manager #1 and #2.

The bad memory was on E1 and D1.

We'll Fix E1 First

We decided to fix E1 first because it's easiest. We gathered the MarkMail team and started at 5pm. That's the time period with our lowest traffic. It's a little counter-intuitive but since we're a global site we're as busy at 2am (Pacific) as we are at 2pm. The time around 5pm Pacific still sees a lot of traffic, but relatively less. Why? We theorize it's because we get the most traffic during the visitor's local business hours, and the 5pm to 8pm Pacific time slot puts the local business hours out in the middle of the Pacific.

The E1 server is important because it catches all requests. Our plan was to place a new host, essentially E2, into the cluster and route all traffic through it instead of E1. There's no state held by the front-end machines, so this is an easy change. We borrowed a machine, added it to the MarkLogic cluster, told it to join the "group" that would make it act like E1, and has our reverse proxy start routing traffic to it instead. We did all this with the MarkLogic web-based administration. It was far too easy, frankly.

We immediately saw the E1 access logs go silent and we knew our patient was, in effect, on a heart-lung bypass machine. We told our sysadmin in the colo to proceed.

That's when he told us that on more careful inspection the memory problems were on D1 and D2. The E1 server was just fine. Hmm...

We decided to call the maneuver good practice for later and put things back like we found them.

OK, We'll Fix D1 First

Performing maintenance on a machine like D1 requires more consideration because it's managing content. If we were to just unplug it, the content on the site would appear to be reduced by half. It'd be like winding the clock back to April, with our home page saying we just passed the 10 million message mark.

All email messages go into MarkLogic data structures called "forests". (Get it? Forests are collections of trees, each document being an XML tree). Our D1 server manages forests MarkMail 1 and MarkMail 2, the oldest two. They're now effectively read-only because we're loading into higher numbered forests now on D2.

Turns out that's a highly convenient fact. It means we could back up the content from D1 and put it on our spare box, now acting like a D3. Then with a single transactional call to MarkLogic we could enable the two backup forests on D3 and disable the two original forests on D1. No one on the outside would see a difference. Zero downtime.

It worked great! It took a few hours to copy things because it's hundreds of gigs of messages, but like a chef on TV we knew what we were going to need for showtime and prepared things in advance.

With the new memory chip placed in D1 we did a transactional switch-back, put the two original forests back into service and had the spare box unused again, ready to help with D2.

We Need an Alternate Approach for D2

Had we planned in advance to work on D2 we probably would have followed the same "use a backup forest" approach we used to work on D1 because it allowed for zero downtime. It would have required pushing ingestion activities to another machine like D1 so the forests could settle down and be read-only, but that's done easily enough. We didn't do this, however, because we were too impatient to wait for the data to copy between machines. Instead we decided to leave the data in place and do a SAN mount switch.

We host all our forest content on a big SAN (a storage area network, basically a bunch of drives working together to act like a big fast disk). All the data managing machines (D1, D2, and the spare acting as D3) have access to it. Usually we partition things into individual mount points so they can't step on each other's toes and corrupt things. You never want two databases to operate against the same data! Here we decided to remove the isolation. We'd have D2 "detach" the MarkMail 3 and MarkMail 4 forests and have our spare machine (acting like D3) quickly "attach" them. We would essentially transfer a few hundred gigs in seconds.

This system change couldn't be made transactionally, so we had a decision to make: Is it better to turn off the MarkMail site for a short time or let the world see a MarkMail with only half its content? We decided to just turn off the site. Our total downtime for the switch was 43 seconds going over, just over a minute coming back after the memory change.

We think we could do it faster next time with some optimizations in the MarkLogic configuration -- turning off things like index compatibility checks, which we know we don't need. Maybe 20 seconds, or even 15.

The Moral

Looking back, we're happy that we could cycle through disabling every machine in our MarkLogic cluster yet not have any substantial downtime. Looking forward, we expect operations like this will get easier. If and when we add a permanent E2 machine to the cluster it means we won't have to do anything special to take one of them out of commission. Our load balancer will just automatically route around any unresponsive front-end servers. We were also happy to see that our configuration for SAN-based manual failover works. We proved that as long as another machine can access the SAN, we'll be able to bring the content back online should a back-end machine fail.

Everyone on the MarkMail team works at Mark Logic, the company that makes the core technology that powers our site. In fact, in years past some of us have been directly involved in building the technology. But despite our familiarity, we were still delighted to take the production MarkLogic cluster out for a walk and get it to do tricks. It did the right thing time after time with every disconnect and reconnect and reconfiguration, and we couldn't help but feel a point of pride. This is some fun software! If you're a Mark Logic customer, we trust you know what we mean.

A non-techie friend once asked why managing a high-uptime web site was hard. I said, "It's like we're driving from California to New York and we're not allowed to stop the car. We have to fill the gas tank, change the tires, wash the windows, and tune the engine but never reduce our speed. And really, because we're trying to add new features and load new content as we go, we need to leave California driving a Mini Cooper S and arrive in New York with a Mercedes ML320."

So far so good! Here's to the long roads ahead...

Pillow Talk

2008-08-17T17:06:00.000-07:00

We have a bit of a tradition at MarkMail where we give away T-shirts at the conferences we attend. Printed on the front of the T-shirts we put the mailing list traffic chart generated by the community whose conference we're attending. Last year we did it at ApacheCon in November, then again at XML 2007 in December. We did it at JavaOne too. They're fun because they're personalized, and to the recipients the long bars often bring back memories of fast growth, new product releases, and raging flamewars.

One of the recipients of a T-shirt at XML 2007 was B Tommie Usdin. Tommie doesn't like to wear T-shirts. No, she likes to make T-pillows out of them instead. Recently she emailed us a picture of her handiwork:

We just had to share.

MarkMail has Procmail

2008-08-16T08:20:00.000-07:00

This last week we loaded the Procmail list archives into MarkMail and I wanted to pause and mention it here because it's the kind of thing that readers of our blog would probably appreciate.

Procmail, for those who don't know, is a tool for filtering email. It lets you define complex rules for email processing. You can file messages into folders, quarantine spam, block viruses, and more. First released back in 1990, it's an oldie but goodie for people who want to do advanced things with email and aren't afraid to do some rule file hacking.

Of course not all is great with Procmail. It's arcane and fickle, with a rule syntax that confuses new users. It hasn't had a new release in a long while, nor have its official docs been kept up to date. Answers to common questions aren't on the web site. As a result, every time I've wanted to do something non-trivial with Procmail, I've had to spend a fair amount of time Googling for answers and hunting for samples.

I think that can change. With this list load MarkMail lets you search 25,000 emails spanning the last 8 years where people have been doing Q&A for each other. I expect those emails will give me some good A's for my Q's. Hopefully they'll do the same for you.

P.S. Of course there's almost as many emails about Procmail outside the Procmail list as there are emails inside it.

Interview with KDE

2008-08-07T06:33:00.000-07:00

Earlier this week KDE News published an interview with us about our recent loading of the KDE community list archives.

Our interviewer, Jos Poortvliet, asked some interesting questions on topics we haven't spoken much about before: how we select which lists to load, and what technical challenges you hit in gathering and loading 2.7 million emails. If you're curious about how things work at MarkMail on the loading side, check it out.

Blogger Names us a "Blog of Note"

2008-08-06T15:11:00.000-07:00

Earlier today we received a comment on one of our blog posts that said, "Congratulations on being named a Blog of Note this week!". It seemed like a perfect comment-spam ploy: Say something nice so the blog owner won't delete your comment. Yet something about the post smelled non-fishy. The comment didn't have any sketchy links like most spams do. I thought maybe it was real. To my surprise and happiness, it was!

We were listed by the Blogger Team as a "Blog of Note" for August 4th:

Thanks, Blogger!

Here Comes the Sun

2008-07-22T11:38:00.000-07:00

Last week, in collaboration with Sun and CollabNet, we loaded the mail archive histories for java.net, Sun's open source developer playground for Java projects and home to projects like GlassFish, jMaki, AppFuse, Grizzly, Hudson and WebWork.

The load includes more than 1,000 mailing lists and roughly 1,000,000 messages. Their growth curve is fantastic (the last month is partial):

Just about half the java.net mails are auto-generated as a result of checkins or bugs. If we remove those, the curve is still beautiful. Looks like people are writing more than 15,000 human-to-human emails every month on java.net.

With such a large community, it's fun to look at community-wide analytics. It's a little-known feature that you can go to our browse page and add an arbitrary query to the URL and it'll show you list-by-list numbers for all messages matching that query. For example, you can view the total number of messages per list throughout time, or the counts for just last week.

You can browse the lists where people from "sun.com" have written the most. If you want to see the top lists, do it as a regular search.

There's been a lot of coverage about this, from Marla Parker, Eduardo Pelegri-Llopart, Clark Richey, and javaHispano. Plus we issued our own press release!

A Tale of Two Search Engines

2008-07-18T00:44:00.000-07:00

If you're local to the Bay Area, you may be interested in attending an upcoming talk from the SDForum Software Architecture & Modeling SIG on August 27th. It's titled A Tale of Two Search Engines and will be given by our own John Mitchell, one of the developers on MarkMail. Here's his abstract:

Betwixt the rigid structure of relational databases and the unbridled chaos of random content lies the world of search engines. Search engines shine in the middle ground where the messy complexity of reality makes everything harder than we imagine.

While the soap operas of general-purpose search engines dominate the news, specialized search engines are coming to dominate their vertical niches. Special-purpose search engines can aggressively leverage domain-specific intelligence to return highly relevant results.

This talk will present the architecture, implementation, and stories behind the creation of two specialized search engines for code and email: Krugle and MarkMail.

If you're interested in MarkMail I think you'll enjoy it. (And no, John doesn't really use words like "betwixt" in daily conversations.)

The Perl Review: Now with Video

2008-07-02T18:15:00.000-07:00

The folks at The Perl Review recently enhanced the interview I mentioned here Monday with a new screencast video showing MarkMail in action. The intro is terrific. There's a guy hitting his Mac with a hammer!

It's a strange (happy) feeling to have others produce advertising videos for you. Thanks, brian d foy!

Interview with The Perl Review

2008-06-30T12:20:00.000-07:00

The Perl Review, a quarterly newsletter about all things Perl, recently published an interview with us where we discuss several topics relating to MarkMail:

How we load mail
Our choice between Java and Perl
Our model of permalinking
Comparative community sizes
What's in store for the future
How this is different than Google

It's a more technical interview than some of the ones we've done previously with Apache, The Content Wrangler, and InfoQ.

Diacritics, or should I say dịẫçritícs

2008-06-10T13:59:00.000-07:00

We changed our indexing this week regarding how we handle diacritics -- those accent marks you see on vowels and some consonants in many languages.

Previously we resolved all queries in a diacritic insensitive manner. That meant that a search for "francois" would match both "francois" and "françois", and a search for "françois" would do the same. Basically we specified in our MarkLogic Server configuration that the c versus ç difference be ignored.

Now we've changed the configuration so the diacritic sensitivity choice depends on the search term. A term containing diacritics will trigger a diacritic sensitive match, while a term without diacritics will remain diacritic insensitive. That means a search for "francois" will match with and without diacritics (the same as before), but a search for "françois" will respect the ç character constraint and won't match "francois" anymore.
To summarize: If you care enough to type a diacritic, we'll care enough to match it for you!
This is a particularly helpful change as we've expanded from English-only content into lists written in Japanese, Vietnamese, Spanish, German, Italian, Dutch, Portuguese, Slovak, Polish, and Farsi. We even have one mail in Frisian. Who knew!

On a per-message basis, we get more traffic from these lists than our English lists. Perhaps they're underserved by other email archive systems? Maybe the other systems have issues hosting messages with the non-ASCII characters. We've definitely had trouble finding "clean" historical archive records for non-English lists, ones where the diacritics were reliably preserved. Luckily for us, being built on XML, we have native support for all Unicode characters.

We hope you find the new indexing logic helpful.