Papertrail

Better log colors thanks to Solarized

Wed, 15 May 2013 14:00:00 -0700

Fellow Seattleite Ethan Schoonover created a color scheme called Solarized, and as of today, Papertrail uses its palette when colorizing log messages. Solarized logs look like this:

Solarized is “a sixteen color palette designed for use with terminal and GUI applications.” Solarized is uniquely well suited to Papertrail’s needs because its readability comes mostly from differences in hue (rather than from excessive use of contrast) and because the palette works well on different backgrounds.

As you can see, the result is more readable color text and less eye fatigue. That it’s also better looking is a nice bonus.

This is live now and is automatically used when viewing colorized logs in Papertrail. Enjoy!

More colorful logging with ANSI color codes

Wed, 01 May 2013 17:11:10 -0700

Papertrail’s event viewer now honors ANSI color codes in log messages. Here’s an example:

The light blue and purple colors were chosen by the application which generated these logs (Ruby on Rails) and are embedded in the log messages. The messages contain ANSI color codes. These escape characters can set the foreground color, background color, and treatment (such as bold) for sections of the message.

This is live now. To use it, simply change your framework or app’s the logging configuration in your Web framework or custom app logs to generate colorized logs. Colors will be shown in Papertrail automatically.

Papertrail already automatically colorizes the orange sender name and blue program name, since they are links to show related context (see Event Viewer Intro).

Finally, a word of caution: it’s easy to go overboard here. A tiny, tiny bit of color goes a long way.

Please send any comments (or screenshots!) and enjoy.

Send Ruby AWS SDK logs to remote syslog and Papertrail

Tue, 02 Apr 2013 17:44:00 -0700

In the AWS Ruby blog, Trevor Rowe covers Logging HTTP Wire Traces with the AWS SDK for Ruby (aws-sdk gem) and overriding the default Logger class.

By default, aws-sdk logs to $stdout for Ruby apps or Rails.logger for Rails apps. Trevor’s example passes in an alternative Logger instance that is backed by $stderr:

AWS.config(:logger => Logger.new($stderr))

Let’s send these logs to a syslog destination like Papertrail, using Papertrail’s open-source remote_syslog_logger gem.

remote_syslog_logger

remote_syslog_logger provides a Logger-compatible class that’s backed by a UDP remote syslog destination (rather than an instance of IO, such as $stderr). To use it, add remote_syslog_logger to your Gemfile (or require it), then instantiate RemoteSyslogLogger with the syslog destination hostname and optionally, port. For example:

AWS.config(:logger => RemoteSyslogLogger.new('syslog.domain.com', 1234)

See Papertrail’s Add Systems page for these settings.

This works without any other system-wide config changes. A RemoteSyslogLogger instance can be passed to any Ruby class which expects a Logger instance.

Customize

By default, RemoteSyslogLogger will use its running program as the program name in the syslog message. Often this is just rails. Let’s set the program to something more descriptive, like myapp-development. Let’s also enable the AWS Ruby SDK http_wire_trace option to output HTTP headers (which one would typically only do in development). Here it is:

AWS.config(:logger => RemoteSyslogLogger.new('syslog.domain.com', 1234, :program => "myapp-#{RAILS_ENV}"), :http_wire_trace => true)

The README has more.

Extend

To use two Loggers, one could wrap RemoteSyslogLogger within a container class which simply invoked each of them.

Other ways to accomplish a similar result include writing logs to local syslog and configuring it to transmit them, or write to a text log file and use a daemon to watch and send it (see rsyslog, syslog-ng, or remote_syslog).

Happy debugging!

Change groups without leaving the event viewer

Fri, 22 Mar 2013 12:25:00 -0700

Papertrail’s log viewer now supports switching groups. We’re always looking for small tweaks that make a big impact on our own, and hopefully, your workflow. Thanks to Ben Fyvie for suggesting this.

Here’s how it works. In the event viewer, the upper left corner has changed to show the current group or sender:

Click it to see all groups:

Choose the one you want to change to. The active search and seeked-to time are retained.

Here’s how this tiny change helped me this morning. I was troubleshooting a problem by looking at Papertrail’s own Web servers group. I’d already seeked to a time and then searched to find the HTTP logs I was interested in. They led me squarely to a problem with logs from another part of our stack.

Instead of reloading the event viewer on another group and going through that again, I just changed to the group I wanted. Papertrail retained my search and time, so the log messages that I cared about were already highlighted. I was able to continue troubleshooting without losing my train of thought.

This is common in my troubleshooting process, whether it’s changing from production logs to staging ones (to see whether the same error occurred there), or from the server logs of an API request to the client’s logs. Now context stays with me.

Also, when I need to see both side-by-side, the dropdown lets me open a different group in a new browser tab. Finally, my saved searches are more useful now that they’re easy to use with any group.

Thanks to Ben for this idea. Is there a small tweak that would improve your workflow? We want to know. Enjoy!

Search Alert: StatHat graphs

Wed, 13 Mar 2013 15:35:51 -0700

We’ve just added support to graph the number of log messages matching a search in StatHat.

StatHat provides a simple way to graph important metrics and easily measure how it is performing compared to the past. It provides ways to receive alerts based on changes in data or absolute thresholds and automatically lets you know about important changes to the metrics you send.

To create a search alert, select StatHat from the list of search alerts and fill out the details:

Once you’ve done that, just sit back and wait for matching messages to arrive and a graph to be created for you. Soon you’ll have a graph that looks like this:

Special thanks to ejholmes on GitHub for the initial work on this search alert integration and for StatHat for adding a JSON content type for their API to make it faster and more efficient for us to send the stats.

To learn how we implemented this alert, check out the source code on GitHub or ask me directly in our chat room.

— Eric

Papertrail is now brought to you by Papertrail Inc.

Tue, 19 Feb 2013 17:02:15 -0800

When we started Papertrail, it was within a company called Seven Scale (which we’d used for prior ventures). As Papertrail has grown, it no longer made sense to use the Seven Scale name.

Over the next few weeks, references to “Seven Scale” will change to “Papertrail Inc.” Our customers will see zero impact from this change.

Thank you for your support and happy logging.

Announcing the Dashboard Finder: a better jumping-off point

Thu, 14 Feb 2013 13:31:00 -0800

We’re proud to announce the Dashboard Finder, which is now live on Papertrail’s Dashboard. Try it:

How it came to be

The Dashboard is where you’ll land after logging in to Papertrail. It shows groups, saved searches, and a subset of log senders. Until today it had two weaknesses:

Navigating to resources not shown on the Dashboard took at least a second step. This was common with more than a few senders, when only a few were directly reachable from the Dashboard:
As more groups and saved searches were created, finding the best one required knowing what you want and where it is. For example, two groups and five searches could relate to the “Acme Anvil” product, and I may have only created one of them myself:

Until today, the Dashboard needed a quick way to get anywhere and an easy way to see everything.

Announcing the Dashboard Finder

Over the last few months, we’ve been trying different ways to fill these gaps, using them ourselves, and iterating. Today we’re proud to unveil the Dashboard Finder:

Small in screen real estate, the Finder hides its power. Have an idea where you’re going? When you arrive at the Dashboard, hit the “s” key (or click in the text box) and type the name or a fragment of a saved search, group, or sender.

That’s it. The Dashboard Finder stays out of the way until you need it, then makes it easy to load any view in the event viewer.

The arrow keys work naturally. I can see and reach everything related to “Production” with a few keystrokes:

My fingers never leave the keyboard.

We hope the Dashboard Finder saves as much time for you as has for us. Like everything at Papertrail, we’re learning as we go, and we want your feedback. If you use the Finder in a way that it’s not perfect for, tell us. And if it works beautifully for you, tell us that too.

Enjoy!

Announcing search alert minimum thresholds

Mon, 11 Feb 2013 14:26:46 -0800

As users know, Papertrail has offered painless log search alerts for some time. Notify a Campfire chat room, email a team, or kick off a PagerDuty escalation process when something important happens. Get periodic summaries about less-critical issues.

Notifications are inherently binary, though: either an alert fires or it doesn’t. You want to know or you don’t. There’s often a gray area in monitoring, though, where two of something is fine, 20 is not.

Papertrail now lets you decide that gray area. Papertrail search alerts can now include a minimum number of events that must occur (during the alert interval) for the alert to be invoked. The default and existing behavior is a minimum of 1.

What’s possible with a minimum threshold?

Allow for endemic problems. With some issues, a constant undercurrent occurs all the time. Internet-facing problems like 404s, brute-force attacks, and site scrapers are the obvious examples. I don’t care that they happen, but I do care when they cross above what I consider noise.

Allow for known issues. If you’re short on RAM, the fact that Linux’s “oom_killer” killed a process once to save RAM probably isn’t severe. If it kills 5, it is. This is particularly handy with code exceptions (known bugs) and services where log output can’t easily be modified.

Detect when less-critical issues become trends. For example, I’d like to know when more than 10 slow queries occur across all of my database servers. That’s not a slow query, that’s a trend.

Combine these. Take velocity into account when deciding what the alert should do. When 50 of something happen in a 10 minute period, it’s urgent; tell me in HipChat. Separately, email a summary to my team every day, regardless of volume.

How to use it

Use this in combination with log filtering and “Show related logs” links to ignore the events that don’t matter, react to those that appear to be important, and link directly from your admin dashboard to those that you know are relevant.

Perhaps the best part of Papertrail’s search alert threshold is that it comes without a ton more time or effort on your behalf. There’s not a zillion controls because there doesn’t need to be; there’s the one that matters.

As product designers, we always try find the knobs that solve the most problems - expose the most power - with the least, and least visible, complexity. There’s not many better examples of it than this.

A fresher home page

Mon, 11 Feb 2013 09:13:00 -0800

Last week we quietly released a new papertrailapp.com public home page.

Our goal was simple: incorporate what we’ve learned from operating Papertrail. We’ve actually seen how people interact with the site, which wasn’t the case when the original site was designed.

While the text hasn’t changed dramatically, the new site’s layout highlights the elements that we’ve seen firsthand are important to visitors. It also does a much better job of demonstrating that we care about stylized design. It feels more modern, which is a more accurate representation of Papertrail’s product and our view of the world.

The result speaks for itself. Feel free to send comments.

Help Papertrail make engineers and entrepreneurs happy - we're hiring

Wed, 06 Feb 2013 19:04:00 -0800

Help Papertrail make engineers and entrepreneurs happy - we're hiring:

We’ve grown to need part-time technical customer support. A few things make this a unique position: work from anywhere, any time, and choose how much. Float between a few hours and 20 hours/week (if not more). Also, do as much beyond tech support as you’re interested in.

Learn more: https://gist.github.com/troy/5066341. (Expired StackOverflow Careers listing.)

More signal, less noise with simple, free log filtering

Fri, 01 Feb 2013 15:14:00 -0800

Papertrail can now filter log messages on behalf of your apps and systems.

Constantly logging a noisy system error that you’re well aware of? Don’t care about requests from monitoring probes? Give us a regular expression and we’ll ignore them. No need to touch your app or system config.

Keep reading or dive right in the docs.

Why is this important?

Usefulness is in the eye of the beholder. It’s not our job to tell you what log messages might be useful, and we’d prefer that it isn’t left to your application or system’s default configuration, either. The logs that the author chose to generate may not have any relationship to the ones you care about.

Our job is to expose intelligently-chosen control and configurability. More succinctly: the knobs that matter. This is one of them.

Logging configs are harder to change than they could be, especially with multiple senders and deployment environments. Papertrail’s log filtering transfers that control from the daemon or framework author to you.

That’s true for managed services too, where a hosting provider may not be able to modify their logging policies to suit a single customer.

Also, Papertrail is in a unique position to provide infrastructure-wide logging control. In addition to message content, Papertrail can filter by sender, program, or log destination, which means there’s a single place to silence almost any unnecessary logs.

We hope this saves time which might otherwise be spent fighting with app configs.

Am I charged for filtered messages?

No. Log filtering is included with all Papertrail accounts and filtered messages don’t consume log data transfer.

Let’s see it in action

A standard Ruby on Rails app generates these messages for every page fragment (“partial”) used in a page:

Rendered layouts/_head.html.erb

Although we have easy ways to make Rails less verbose, assume I’m mostly happy with the stock logging except these messages. Knowing the list of page elements that were rendered usually serves no purpose in production, so they’re basically noise. And changing the log output either means omitting other messages (by decreasing verbosity) or monkey-patching Rails.

So, let’s have Papertrail ignore them. Use the filter:

Rendered

In your log destination configuration (Account), it looks like this:

Users of Papertrail via hosting services may see a slightly different presentation for clarity.

Next, let’s just ignore these messages from senders that begin with “www” (like www1 and www2), not all senders. Also, we don’t care about static asset requests (like images and CSS files), either, so let’s also ignore messages from those senders that contain “/assets”. Use the filter:

www\d+ \W: (Rendered|\/assets)

See it in Rubular. The filter is a regular expression. This regex matches any sender name with “www” followed by one or more digits, then any app name from those senders, then either of the messages we want to filter.

Use it

Use Papertrail’s log filtering to customize where an app should be verbose and where it should be quiet. No need to be limited by the sending service’s default settings or the implementor’s preferences. Just decide what you don’t want and create a regular expression to match it.

Head over to Log Filtering and read more, then configure it in Account settings.

A touch more elegance in Heroku add-on interface

Fri, 25 Jan 2013 16:29:00 -0800

Papertrail provides a Heroku add-on for aggregating and managing logs from apps, including dyno/app output, router requests, and platform events. This calls for a consistent interface between the Heroku Dashboard and Papertrail (when accessed as an add-on), so Papertrail displays part of the heroku.com site header.

Until recently, the header was, well, really tall:

We’re all about making the most of every pixel on any Papertrail interface. These pixels were not well spent. To that end, we’ve been exchanging ideas with Heroku about how to make the header shorter.

On Wednesday, Heroku’s Zeke Sikelianos released boomerang, which is “a JavaScript widget that gives users a way back to Heroku from Addon provider sites.” The header now looks like this:

Zeke shrunk the header from 148 to 30 pixels. That 118 pixel savings is about 10% of the screen real estate on many displays, so it made a dramatic difference.

He took it further and made the header more useful by adding links to the app settings that a visitor might need. Now Web-centric add-ons feel more like an integrated experience within Heroku’s Dashboard.

We released this on Thursday and have been basking in its glory for the last day. Enjoy!

Ordering saved searches alphabetically

Fri, 05 Oct 2012 18:25:00 -0700

Aside from the new group-wide log context link, we made one much smaller change: saved searches are now alphabetized in Papertrail’s Dashboard.

Back Story

Until today, saved searches were shown in the order that they had been created. This made sense with a relatively small number of searches (and when a relatively small set of people was creating them); at that point, when it was created was meaningful.

As more saved searches are created (and in many Papertrail groups), name starts to matter more than chronology. Also, saved searches can act as an informal institutional knowledge transfer: a database administrator might create a search that a developer finds useful, and vice versa. In that case, the search’s name - like “503 errors” or “User-facing slow queries” - is all that matters, and that’s how coworkers will discover and use it.

Enjoy, and send any comments our way or to support@.

Changing behavior of links to surrounding context

Fri, 05 Oct 2012 14:59:33 -0700

As you’ve probably seen, Papertrail’s log viewer shows links to display a specific message in context. Click the name of a log sender or a program to jump to that message within the related messages generated right before and right after it:

What Changed?

We made a subtle but important change to the program name link, shown in blue (like “access_log” and “sshd”).

Until today, clicking the name of a program which generated a message would show that message surrounded by logs from that program on only that log sender. For example, only messages from “sshd” running on the system named “taco.”

As of today, clicking a program shows that message surrounded by all messages from that program, from all log senders that you’re looking at (typically a group). In the past, it only included the specific sender which generated the message.

The log sender link (shown in orange, like “taco”) has not changed. It still shows a message surrounded by all messages from that log sender.

Back Story

This came about based on how we saw Papertrail used by customers and from using it ourselves. In our own case, we often wanted to see a message in context with related messages from across a multi-server fleet.

We found that it was more useful than honing in on a specific system+app (which the orange link accomplishes well). One of Papertrail’s goals is to be users first, so we’re always looking for tiny improvements that improve workflow.

What It’s For

If you haven’t tried contextual links, here’s couple examples how they’re useful to us:

History: after seeking to a HTTP request that served a user-facing error, click the program name (“access_log”) to show the preceding and following requests. With the change above, the context includes requests made to other systems in the group, like on a load-balanced Web server cluster.
Long events: when a search matches one line in a multi-line traceback or exception, clicking the name of the log sender (“taco”) will show the entire traceback.
Impact: spot a kernel problem in a syslog message? Click the log sender to show that message amidst messages from just that system and see the impact on apps.

Enjoy, and please send comments or suggestions.

Announcing read-only account permissions

Mon, 13 Aug 2012 00:50:59 -0700

I’m pleased to announce that Papertrail now supports a read-only account access level. Need to share short-term log visibility with a consultant? Want to spread ops visibility throughout your company, without the risk of someone accidentally changing a setting? This is for you.

Read-only users have permission to tail and search logs, view groups and saved searches, and download archives. They can’t edit, change, or delete account assets (like systems, searches, or log messages). Also, they can’t view or change payment information and are not emailed about plan limits.

Account administrators can set privileges for all other users on the Members page:

Give it a whirl and send us any suggestions.

If I could think of anything funny about time zones, it would be here

Sat, 16 Jun 2012 11:14:40 -0700

Almost everything we touch at Papertrail has a timestamp, which means it also has an expressed or implied time zone. Here’s more about why and how we work with time zones, then an introduction to new time zone-aware search alerts.

Who Cares?

Modern time zones came about in 1884, when a Canadian railroad engineer proposed a “universal day” based on a world meridian. At the International Meridian Conference, the Observatory of Greenwich was chosen as the meridian and zones were created every 15 degrees of longitude (proceedings).

Papertrail encounters a few issues that 19th century railroads did:

Zones can change. The items we care about - log messages, users, servers - can all be mobile. Like trains and passengers, they can start, traverse, and end in different places.
Reasonable people expect different things. Sysadmin Sally might have servers in 6 countries, set them to GMT, and convert to local time in her head. She probably expects something different than Developer Dan, who logs in once a week to troubleshoot a problem (and shouldn’t need to care about that conversion). Sally and Dan may be coworkers.
Humans aren’t the only audience. Aside from operations, Papertrail needs to present the right zone and level of zone awareness to webhooks, permanent archives, and other targets.
The status quo isn’t great. Log aggregation often transfers the burden on to the user, either by “punting” and treating timestamps as strings rather than times (so everyone has to do what Sally does) or by converting them to a single zone.

Our goal is to deliver a service that doesn’t require knowing any of that.

OK, How?

As much as possible, our starting point is the principle of least astonishment: Papertrail should just work, and then be tweakable where needed. We’ve spent effort to:

Let servers and apps live anywhere and configured however the operator wants. Multiple data centers, cloud presences, and PaaS deployments are the norm. They often don’t - and should’t need to - use the same time zone.
Deliver a sequential view of events across all infrastructure, without requiring reliable time sync. Troubleshooting interactions between 2 servers is very difficult when the events are out of order, and if Papertrail blindly trusted local timestamps, they often would be.
Support distributed teams. Each engineer should be able to view messages in whatever zone is most logical to them. Same for timestamps provided by staff, like when seeking to a time.
Present a consistent view of inconsistent logs. Some log messages contain explicit timestamps (ISO 8601), others have implicit zones (known only to the server), and some senders may generate logs in multiple zones concurrently (for example, OS, Web server, and app events). Manually tracking and normalizing these is a particularly demeaning use of humans.
Make it easier for log consumers (search alerts, API clients) to ignore zones. Transfer the burden of localization on to Papertrail. More below.

Time Zone-Aware Log Alerts

As of March 29, 2012, search alerts can now receive event timestamps in any timestamp. Hit the Dashboard, find the saved search you have an alert for, mouseover it, and click “Edit.” Under “Manage Alerts,” choose the alert.

You’ll see a time zone dropdown. Choose the zone that Papertrail should convert the timestamps of matching messages to.

The Making Of Zone-Aware Alerts

Here’s the back story. We tried a few incarnations and found that even teams which are spread across time zones usually have a zone that the team thinks of as standard (what server clocks are set to, a particular office, UTC, etc.).

We also found that for emails, automatically changing timestamps to each recipient’s Papertrail account zone sounds ideal, but it’s actually not very intuitive because other than email, most alert methods inherently present a single time zone to many recipients. For example, a single chat room alert is seen by many viewers.

(There’s also an unwritten expectation with team-wide alerts that the whole team received the same thing. When a coworker pastes a log message in IM, converting it from their time zone to yours isn’t even on your radar.)

Also, treating email uniquely would mean that for the same issue, the same individual could receive a Campfire alert in one zone and an email in another. Besides violating the principle of least astonishment, that’s just dumb. There’s a better way: render each event’s timestamp as a link directly to that message in the Web viewer (where the time is localized to you). So Papertrail does that.

That’s why alert timestamps are set once for the alert. It helps that we’ve actually tried, and lived with, these decisions.

I hope you enjoyed this introduction to time zones in Papertrail. As always, if something isn’t working well, doesn’t seem obvious, or simply can be better, tell us.

Search Alerts: Integration with Boundary

Wed, 13 Jun 2012 17:29:18 -0700

We’ve added a new Papertrail search alert to create a Boundary event based on your application or system log messages.

Boundary provides network-centric application visibility and they value realtime data as much as we do. They provide a way to create events, which show up as red indicators at the bottom of a graph. Events can include a short description and links.

For example, a system shutdown could show up as DB rebooted (tagged with the name of the system), or a puppet deploy run could add a Code deploy marker. Events include a [papertrail] link right to first related message.

To create a search alert, just select Boundary from the list of search alerts and fill in the details:

Once you’ve done that, any new log messages that match the alert will show up as Boundary events in their Dashboard.

The Boundary event is tagged with any tags provided in the Search Alert as well as the hostnames of related log messages.

To learn how we implemented this alert, check out the source code on GitHub or ask me directly in our chat room.

— Eric

Improving PagerDuty Alerts

Wed, 13 Jun 2012 13:33:00 -0700

Our PagerDuty search alert was one of our first user-contributed alerts. @darron dropped by our chat room and pointed places where small improvements to the alert could give a lot more information when jumping on a problem.

Over the past week, I talked with a few customers about how they were using PagerDuty alerts. They suggested a few more small improvements and noted that the parameters could be more clearly explained.

First, we’ve added the list of systems that generated messages, so you’ll immediately know which systems are affected (right from the email or text message). It’s in the search alert at the end of the description.

An important aspect of PagerDuty’s triggers is the ability to tie multiple alerts to the same incident, so they can be acted on as one problem. PagerDuty does this with the incident_key option, which their documentation says it:

…identifies the incident to which this trigger event should be applied. If there’s no open (i.e. unresolved) incident with this key, a new one will be created. If there’s already an open incident with a matching key, this event will be appended to that incident’s log. The event key provides an easy way to “de-dup” problem reports. If this field isn’t provided, PagerDuty will automatically open a new incident with a unique key.

From reading that and our customers’ feedback, there’s no right answer about whether an incident key should be unique to a server or unique to a saved search. To deal with this, I introduced a token called %HOST% that can be used in the incident key.

If present, %HOST% will be replaced with the hostname of the system that triggered the alert. This means if you have three hosts that were matched by a saved search, you can receive three PagerDuty alerts (or just one).

For instance, imagine a search alert that triggers when the Linux OOM Killer is invoked. It’s likely that each occurrence is a distinct server-specific problem, and a single Papertrail search alert can create separate host-specific PagerDuty incident:

Contrast that with a search alert for a Web app error that could occur on dozens of Web servers. For that, the error is probably not host-specific and multiple occurrences probably are the same issue. The Papertrail search alert can update a single alert-specific PagerDuty incident.

To see how we implemented this alert, check out the source code on GitHub.

As always, drop us a line or say hi in our chat room if you have any more suggestions or just want to talk shop.

— Eric

Multiple accounts made easy

Tue, 08 May 2012 13:00:00 -0700

Need to manage logs for multiple organizations, like for a consultancy’s customers or to keep work and personal logs separate? This is now easy.

If you already have a Papertrail account and are granted access to another organization’s logs, you’ll see a new menu item in the Dashboard header. It shows the entities you have access to (Acme Anvils, Western Widgets, etc.). Just click to switch contexts:

An email notification is also sent with each grant. To see current members or to grant access to a new email address, head over to Members.

Let us know if you have any questions or feedback.

Archive file timestamp changes

Mon, 07 May 2012 20:14:00 -0700

Update Monday, May 21: These changes are now active.

Along with realtime search and tail, Papertrail generates daily or hourly log archive files. They downloadable from Papertrail’s site and optionally, uploaded to your own S3 bucket.

The current archive format has 2 flaws that we want to correct:

Zone: The timestamps are in PST8PDT. They should be in UTC.
Format: Although the current timestamps use a common format, ISO 8601 is at least as common and is more widely documented (RFC 3339). Nearly all languages include libraries to parse ISO 8601 timestamps.

For comparison, a current timestamp (PST): 2011-02-10 00:19:36 -0800

And the same local time in ISO 8601 (UTC): 2011-02-10T00:19:36Z

This change will affect archives created on or after Monday, May 21, 2012. The archive file for Sunday, May 20 may contain 16 hours of logs and Monday, May 21 may contain 32 hours.

Please open a support request if we can answer any questions or help modify a post-processing script.