Loggly by Hoover

Getting Your Product Sticky

2012-01-16T23:22:31-08:00

A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators. When it gets dark, we forget everything you've typed into us."

That comment wasn't far off the mark. Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session. Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point. It made it extremely difficult to get back to something meaningful the next time you logged in.

We shouldn't be here if we aren't meaningful. We should deliver users a 'punch in the gut' feature that makes a lasting impact. One they don't want to avoid.

Saving Time with Sticky Features

Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority. Unfortunately us solving scale issues aren't readily obvious to users. Users always expect things on the web to be fast. They could care less how hard a problem it was to solve. They don't think to themselves, "Wow, that's fucking fast!". No. Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y? This thing is wasting my time!".

And there it is laid bare: Don't waste your user's time. It's the most valuable resource they have. Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.

I say give them sticky features!

Saved Search and More

And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search. Saved search provides users a way to write a search query and then preserve the search to run again later. Saved searches can generate facet graphs or they can simply run a regular search across a given time range.

The shell has been reworked to provide context changes to use with the saved search feature. You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.

Here's a quick screencast running through some of our new sticky features:

Coming Up Soon

We're continuing to add features that increase stickiness to the product. Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt. We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.

All these featuers are leading up to a major revamp of the way we provide value for our user's events. Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon! If you have a feature you'd like to see us implement, please do drop us a line. We're keen on not wasting your time!

Loggly's Outage for December 19th

2011-12-21T16:44:56-08:00

Sometimes there's just no other way to say "we're down" than just admitting you screwed up and are down. We're coming back up now, and in theory by the time this is read, we'll be serving the app again normally. There will be a good amount of time until we can rebuild the indexes for historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.

So What Happened?

Sometime yesterday afternoon ALL of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff. Originally we stated we had not received reboot notices from Amazon, but the truth is that (4) of the staff here, myself included received two separate vague notices, one from about 10 days ago, and another from 3 days ago, which stated 'some or all' of our instances were scheduled to be rebooted. These notices were found in our spam folders on Gmail, placed there with a very large red notice reading: "Warning: This message may not be from whom it claims to be. Beware of following any links in it or of providing the sender with any personal information." Meh.

Loggly uses a variety of monitoring mechanisms to ensure our services are healthy. These include, but are not limited to, extensive monitoring with Nagios, external monitors like Zerigo, and using a slew of our own API calls for monitoring for errors in our logs. When the mass reboot occurred we failed to alert because a) our monitoring server was rebooted and failed to complete the boot cycle, b) the external monitors were only set to test for pings and established connections to syslog and http (more about that in a moment), and c) the custom API calls using us were no longer running because we were down.

Combined, these failures effectively prevented us from noticing we were down. This in of itself is was the cause of at least half our down time, and to me, the most unacceptable part of this whole situation.

The Human Element

The other cause to our failures is what some of you on Twitter are calling "a failure to architect for the cloud". I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes". A reboot of all boxes has never been tested at Loggly before. It's a test we've failed completely as of today. We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.

While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough. The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.

In this case we didn't ever build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing. With other challenges facing us, we decided to live with the risk. Now we're dealing with the fallout of our decision.

So, How Do We Make This Right?

Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes. We've been OK with those limitations in the past simple because we normally archive data to S3 when we catch it, and we are capable of rebuilding indexes on the fly if we lose one or more indexers.

The first step in addressing this is to start sharding our customers across multiple Loggly deployments. This will prevent further outages to the entire customer base. The second step is to start doing Loggly deployments on dedicated hardware. Because we keep large amounts of data on our boxes (the indexes) this is pretty much a requirement for fast recovery times when a deployment goes down. While S3 is AWESOME for backups, it sucks big-time for rebuilding a large amount of search index data.

The second step is to ensure more robust external monitoring. With multiple deployments, this issue becomes less of an issue, but clearly we need more reliable checks than what we rely on with Zerigo or other services. Sorry, but simple HTTP checks, pings and established connections to a box do not guarantee it's up!

Finally, we accept full responsibility for the impact to our customers. We will be in touch with our paid customers sometime over the next week to address compensation for this outage.

We welcome feedback below, and encourage useful criticism of our architectural choices. All I would ask is that you consider Loggly's infrastructure isn't the same as yours, and I've greatly simplified the reasons for not being more redundant in our deployments. We can, and will, endeavor to do better in the future.

Kord Campbell, CEO

Enabling CORS in Django Piston

2011-12-05T15:28:32-08:00

Here at Loggly, one of our goals is to make our API accessible and easy to integrate. By enabling CORS (Cross Origin Resource Sharing) on our API endpoints, we hope more Javascript developers can take advantage of what our product has to offer.

CORS is an addition to the browser security model that allows XHR requests to be made from one domain to another. CORS allows Javascript applications to access resources on domains other than the original document's domain, working around the same-origin policy. While Javascript application developers have crafted techniques like JSONP, Flash proxies, XHR receivers, and server-side proxies to circumvent the same-origin policy, CORS makes these hacks unnecessary.

To take advantage of CORS both the server and the browser need to support the standard. The browser needs to initiate a negotiation with the server and the server must signal to the browser which domains are allowed to make cross-domain requests. Our current API is implemented in Django Piston, an open-source project that enabled us to quickly build a RESTful API on top of Django. Piston does not support CORS out-of-the-box, but it wasn't hard to write some code to enable it and we'd like to show how it was done.

A full explanation of CORS is beyond the scope of this post, but the central idea behind CORS is a negotiation between the browser and server of allowed and disallowed actions. This negotiation is done via HTTP headers. The essential headers are the following:

Origin: Sent by the browser signifying the originating domain.
Access-Control-Allowed-Origin: Sent by the server, listing the origin domains allowed to make requests to the server's domain. Can be a comma-separated list of domains or "*" to allow requests from all domains.
Access-Control-Allow-Methods: Sent by the server, listing the HTTP methods the browser is allowed to use in requests to the server.
Access-Control-Allow-Headers: Sent by the server, listing the HTTP methods the server is willing to accept from the browser.

Essentially, to enable CORS we need to have Django Piston respond to an OPTIONS request with the server-sent headers and send the requisite headers along with responses.

The Resource class is the heart of a Django Piston-built API. The code that injects the headers into responses lives in a subclass of the base Resource class. We've called this class CORSResource:

The CORSResource performs two simple tasks. First, it intercepts any OPTIONS method requests to handle the pre-flight negotiation between the browser and the server. Since OPTIONS requests do not have a response body, an empty HTTPResponse() is returned along with the requisite headers. Second, CORSResource intercepts responses from the Django Piston handlers (where responses are generated) and decorates them with the CORS headers.

To use CORSResource, we simply instantiated our endpoints with the CORSResource sub-class instead of the base Resource class. The change to our API's urls.py file look like this:

We hope this post helps other Django Piston API implementors enable CORS in their own APIs. We're planning to release this implementation in the coming weeks and we're looking forward to see what Javascript developers are going to do with direct access to our API.

Happy hacking!

(image from http://blogs.bournemouth.ac.uk/research/2011/09/01/sharing-your-research-data/)

Java Code Coverage With Cobertura and Jenkins

2011-11-29T14:44:13-08:00

Introduction

Writing software is tough, tools and processes that reduce the difficulty are always welcome. Code Coverage is an interesting developer tool because it tells you how much you don't know. Not having a test for a particular piece of code doesn't mean that it doesn't work, it just means you aren't as sure as you could be that it works. In this article, I will be covering getting the open source, Java Code Coverage tool, Cobertura, working with Ant, Jenkins and Github.

Cobertura

Since getting continuous integration working in a particular language can be complicated, it is a best practice to break the problem down into discreet chunks. Fortunately, Cobertura makes this easy, because the source code comes with an example ant project. You can download a binary distribution here: http://cobertura.sourceforge.net/download.html. Inside of the download will be a relative path ..examples/basic. If you cd into that directory, you can generate a code coverage report on a sample Java project by typing:

ant

If you don't have junit installed, you will get output like the below, on my OS X Lion laptop:

If you run ant clean it will remove the reports directory, which contains, old reports. One tricky bit with running the example, is that the build.xml file is set to look for cobertura, and the cobertura lib directory, in a relative path two directories above.

This can be tricky if you check in only the examples directory into your git repository and don't have cobertura installed properly. What will happen then, is that the examples won't build properly and it will be tricky to figure out. If you find yourself in this situation, one easy hack is to simply place the cobertura jar files, including the dependencies in the lib, inside of your ant home. On Ubuntu linux this is /usr/share/ant/lib.

Jenkins

With the basic Cobertura configuration out of the way, the next thing to do is to create a throwaway git repository, on github or your own server. Next you will want to check in the whole Cobertura binary distribution that you download from their site: http://cobertura.sourceforge.net/, or only check in the examples root, and install Cobertura properly on your build server. Inside of jenkins you will need to do the following things:

1. Point Jenkins at your git repository.

2. Install the Cobertura jenkins plugin: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin

3. Configure the reports directory correctly. Because I only checked in the examples directory, my reports configuration looked like this:

**/reports/cobertura-xml/coverage.xml

4. build it.

Here is a screenshot of what it looks like when Jenkins builds the example project with code coverage:

Conclusion

If you were able to follow along at home, and have Jenkins automatically building the example project with code coverage, then the next step is to convert this knowledge to your own code base. One problem I had when getting Cobertura working with my code base was configuring the junit stanza properly to fork. Finally, an even simpler way to get code coverage cooking for your Java code base is to install the Eclipse plugin eCobertura. If you are really stuck getting things to work, this is a nice easy win. I hope you enjoyed the article, and if you have any Cobertura tricks, I would love to hear about them.

Ant: http://ant.apache.org/

Cobertura Plugin Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin

Jenkins: http://jenkins-ci.org/

Github: https://github.com/

Cobertura: http://cobertura.sourceforge.net/

Measuring Code Coverage With Cobertura: http://www.ibm.com/developerworks/java/library/j-cobertura/

eCobertura: http://ecobertura.johoop.de/

Automated Integration

2011-11-18T16:55:15-08:00

One of the fundamental challenges of distributed coding is deciding what/when to integrate. Sure, that patch your colleague just sent you looks good, but is it actually ready to go into master? At Loggly, we've been feeling our way towards a disciplined integration process. A year ago, our frontend developers were all making commits directly to trunk in a single SVN repo. Once every few weeks, we'd run `svn up` on our servers, and hope for the best. Today our code goes through peer review, unit testing, and static analysis before it even touches our master branch.

Like most projects these days, the process starts on github. Fork. Push a feature branch to your repo. Open a pull request. Go through a couple rounds of discussion and revision. Merge. Every change to our code goes through this process. At first we thought it would slow us down, that we'd want pull requests for the nontrivial code and to just push to master for the easy stuff. After just a few days, we found the pull requests were slowing us down not at all, and that we all enjoyed the greater transparency into our colleagues' work.

Once we merge, the automation kicks in -- our default integration branch is 'proposed', so clicking merge doesn't actually get the code into the master branch. Jenkins polls our 'proposed' branch once a minute, then runs a simple preflight script on the code.

Rather than keep that preflight in a jenkins configuration page, we have it checked into the codebase so that any developer can run it too; this way there's no excuse for breaking the build -- you should have seen it break locally =P

Here's our preflight script. Let's go through it line by line.

DIR="$( cd "$( dirname "$0" )" && pwd )"

APP=$DIR/..

First we figure out where we're running, so that we can find the other scripts distributed with the app.

$DIR/purge_pyc

Next we purge pyc files. This is done because if a user recently switched from a branch which contained files which don't exist in our branch, the pyc files may still be around, and may be found by the interpreter.

$DIR/syncenv

Next we run a script to sync our python virtual environment, and ensure all requirements are present.

$DIR/runtests

Here, of course, we run our unit tests. Each run prints a coverage report, so that as we recover from our testing debt, we can measure our progress.

&& $DIR/lint...

Next, and this is important, we run pylint over the parts of our app that we expect to pass with no warnings. As we clean up our app, we continue to add modules to this list. Pylint does a few useful things for us. It looks for trivial name errors of the kind that could quickly cause code to stacktrace -- using a module without importing it, etc. It also enforces certain kinds of coding discipline. Our functions and modules can't exceed a certain length. The cyclomatic complexity of our functions is limited.

If all of this passes successfully, Jenkins automatically pushes the checked-out commit to master, which is where we base our development. Thus, we're always basing our development on known-vetted code.

If any of it fails, Jenkins still has a couple more tricks to pull. Here's our on-failure script:

This comes in two parts. The first runs a standard-issue git-bisect between origin/proposed and origin/master. Since origin/master has already been vetted by jenkins (that's how it became master), we know there'll be a regression somewhere between the commits. This goes into the session output, and is e-mailed to the relevant committers. Next, we roll the proposed branch back to the already-vetted master branch. Whatever pull request broke the build will have to be re-made from scratch.

I'm Presenting at MongoSV!

2011-11-10T14:36:50-08:00

In May I had the opportunity to speak at the MongoDB conference in SF. I ran through how Loggly uses MongoDB to store statistical data on the data we receive and how customers can use our API calls to render visualizations of their logging data. I also hacked a visualization example of events per device using the smashing protovis.js library, which you can view here.

I’m thorougly excited to announce I will be speaking again at the MongoSV conference in Santa Clara in December. I'm stepping up the presentation this time, and will be showing more details about how we've integrated MongoDB into our infrastructure, and how we use MongoDB to generate our roll up reports. I'll run through how we use these roll up stats to drive our own metrics dashboard and how exposing the data to end users via our REST APIs provides a whole host of cool options for visualizing data.

If you’re in town or live within a reasonable distance, defined as the entire Contentintal United States, be sure to come check out the conference! The Mongo/10gen crew always puts on great events filled with lots of technical talks and a rocking after-party.

See you there! Again.

Ajax This!

2011-11-08T14:46:46-08:00

Today’s blog post is more of a tool than a toy. Lately I’ve been working on a bookmarklet that utilizes Pusher (if you want to learn a little about Pusher, check out my previous blogpost). The bookmarklet I’m working on is really silly, but these technologies have the potential to be used for some really cool apps.

I needed to figure out a way for javascript to manipulate objects on web pages remotely through ajax calls. You can already do this with click events and Pusher by sending the class or ID of any element you click on, but what if there is no class or ID? I also want this to work on any webpage. As a result, I needed to figure out how to build a selector that was as unique as possible. One way to do this is to select using all attributes the object has to offer.

Enter, the custom attribute selector syntax:

This selector will only select objects that have a class value of “whatever” AND a myattribute value of “customattribute”.

Some stack overflow and basic google searching told me I can tease attributes out using a regex(gross). I knew there had to be a better way so I started fiddling with javascript objects I captured on click and tried to figure out how to uniqify my selector string. What I discovered is that every object I clicked on contained a ton of info. The parts that I use to build my unique-ish selector are:

localName

The name of the html tag. in the gist above, this value is customhtml.

attribute

All attribute(s) associated with the object (if any). In the gist above, these attributes are class=”whatever” and myattribute=”customattribute”

parentNode

This is the object that contains the one clicked on. Technically my gist has no parent, but I use the parent node to map out a path from the object I clicked on all the way up to the body of the page. If the object you click on doesn’t have any attributes ( a <p> object for example) you can still select it based on the specific path to that object.

textContent

This is any text that’s in the object. For example <p>writing stuff</p> will have a textContent value of “writing stuff”.

Apart these elements aren’t bad at selecting the object you want, but together they get pretty close to behaving like a this object.

Here is how I stitch these elements together to create my this-ish selector:

There are a few tricks here:

I found that if the object you click on has no attributes the path doesn’t help too much, so if the object I’m building a selector for doesn’t have any attributes I just give it a class with an empty value, and it works really well.
In an attempt to keep my selector as unique as possible I’m also using :contains() to select on specific strings as well as the path. This helps ensure that you’re manipulating the exact object you want.
If you’re clicking on an image tag, I found that contains won’t allow you to select that image, but if we take it out it works, so I added a filter to keep image tag selectors from using :contains(). (although it currently only works in the developer tools javascript console, I haven’t yet figured out why it’s not working for my bookmarklet).

Generally, the resulting selector ends up looking something like this:

While this method doesn’t currently work on every single element of every single webpage, it has worked on most of the pages I have tried out, including the google search page.

Anyone who has every tried to explain, on the phone (so 1991), to a grandparent or non-tech savvy friend/family member how to do certain things on the web should be able to see the power of this. With these selectors it’s now possible to build a bookmarklet that allows two or more people to see what elements are being clicking on. Now describing webpage elements to your grandfather isn’t needed. Hook him up with your Pusher powered bookmarklet and all he has to do is go to a URL, click a bookmark button, and let’s say... click on the text you just highlighted in yellow. On the flip side, you’ll know (almost)exactly what elements on the page he’s clicking on because his clicks will make highlights on the page as well. Super neat!

Obviously this method isn’t perfected, but if you’re interested helping me make it more precise, please fork my project on github.

https://github.com/brainTrain/ajaxThis

PagerDuty, Loggly, and Alert Birds

2011-10-28T16:48:52-07:00

Well, I'm back, and this time I'm here to talk about an awesome product that we use all the time, PagerDuty. We use it internally for our own alerting (as do a number of Fortune 500 companies along with a million other startups), but we've also integrated it into Alert Birds, which is our alerting tool. With Alert Birds, you can configure saved searches that run against Loggly, and you'll run those searches over a period of time that you've selected, and Alert Birds will escalate alerts in PagerDuty. Before you can do any of those things, however, you need to set up the PagerDuty endpoint:

After you've done that, the next thing you'll need to do is to configure a saved search, and then configure the alert that you want to run. The search itself is pretty straightforward, it has a name, a search string e.g.

(this is why it's cool to send us JSON!), and a list of inputs and devices that you choose - you may want to run a particular search on only your web servers, for instance. The interesting bit is the alert itself, which runs a search that you choose, but has a number of options as to what conditions consitute an alert, and what the message should be:

This is where PagerDuty comes in. Although you can send a GET or POST request to an endpoint of your choosing with the alert data, triggering an alert in PagerDuty is far more useful, as they can SMS/email/phone you, and they also handle escalations and reporting. So, in the example above, if my web servers are spewing 500 exceptions, I want my ops folks to get notified, provided there are more than 10 - I don't want to wake anyone up over a little blip! I'm a nice IT manager like that. Anyhow, once an alert is in a critical state, it will run your search every minute until you're below the threshold, and once that happens, Alert Birds will automatically resolve your alert in PagerDuty.

That's pretty much all there is to it! You can find the docs on Alert Birds here, please do drop me a line at support@loggly.com if you need a hand, and until next time, happy alerting!

Alert Birds, OAuth, and Loggly

2011-10-10T17:15:15-07:00

In today's edition of the Loggly Blog, we're going to talk about integrating with Loggly using OAuth. You can authenticate against Loggly in a couple different ways, but we're going to focus on OAuth today, because that's how Alert Birds is able to run saved searches on a user's behalf. OAuth can be a little tricky to set up at first, which is a big reason why we're open-sourced Alert Birds, and the code can be found at http://github.com/loggly/alertbirds-community-edition.

To build an app, the very first thing you'll need to do is to register an app in your Loggly instance, at /account/applications/. Apps are called 'Consumers' in OAuth parlance, and after your application gets approved, you can get started working with our API using OAuth.

In basic terms, the OAuth flow looks something like this:

1. Get a request token, which is good for a single request

2. Hit the authorization URL, and pass the request token

3. Log into Loggly when redirected

4. Authorize the application with your consumer key to access Loggly on your behalf

5. Get redirected back to your app, and with the OAuth verifier and the request token, get an access token, which is what you'll need to make future requests

This sounds pretty simple, but everything has to be just right for the flow to work. Query string parameters have to be encoded in the right order, you need to send the correct HTTP method that Loggly is expecting, and you can't lose the OAuth verifier or the request token, because if the access token request fails, you'll have to ask the user to re-authenticate your app - no good!

The OAuth wrapper you'll probably want to use is here: https://github.com/loggly/alertbirds-community-edition/blob/master/lib/oauth.py, and a working client code implementation is here: http://github.com/loggly/alertbirds-community-edition/blob/master/controllers/main.py. The wrapper class simply wraps the OAuth 2 Python library, and makes it a little easier to interact with Loggly, and the client code is where you'll find the interesting stuff in terms of hanging onto the request token, handling the return from Loggly, etc.

A simplified walkthough of what's happening in main.py is this:

That's pretty much all there is to it. Using the lib/oauth.py wrapper from Alert Birds should make it considerably easier to get going. Happy coding!

Get Real with Pusher! Getting Those Alert Birds Squawking...

2011-10-04T21:43:50-07:00

Pusher rocks! It gets real with the web in a major way. Websockets! I'm the new kid on the Loggly block and I've been working on Loggly's first alerting app, Alert Birds. Alert Birds uses Pusher because it's fast (real time is kinda hard to beat) and it's easy to use.

I'm giving you guys some code to play around with. Using Pusher and Sound Manager, we're going to set up a site that makes Loggly's Alert Birds hop and sing on a page whenever anyone clicks on them. That means you see when your buddies click birds, and vise verse. Just think of it as a bird chatroom. Trust me, it's awesome.

If you want to follow along with the code, clone the github repo I created here:

http://github.com/brainTrain/PusherBirds

All you need to do in order to get pusher working is sign up for an account, grab some credentials and get coding.

Pusher has two main components:

Listening for events.
Triggering events.

Let's start with listening:

To listen for a pusher event you only need the app key, channel name and event name. With these compoenets listening to a Pusher channel is pretty straight forward and looks something like this:

Triggering is just as straight forward and requires the same elements as listening, plus the app secret and a backend to handle and send data to Pusher.

First you'll need some data to send, let's go nuts with ajax!

This brings us to the backend component of triggering events, so let's look at the trigger.php file.

The first thing you need to do is head over to pusher's Publisher Libraries and choose the backend language you heart the most.

You can find Pusher's list of Publisher Libraries here: http://pusher.com/docs/rest_libraries

Once you do that simply follow the Library authors instructions and go nuts! In my case I chose the generic php library to demonstrate this simple example. It's all fairly straight forward but I did run into a namespace issue with my host. To fix it I simply commented out the namespace in Pusher.php.

In trigger.php I add Pusher.php using php's require function, put my keys, secrets etc into variables and use Pusher.php's Pusher() class to validate. The last line triggers the event and grabs data from the button variable I sent to the URL using my ajax call.

I should point out that Pusher is somewhat flash dependent as well. For browsers that don't support websockets, they have a solution. Flash. Which means it's probably a good idea to handle flashblockers for any pusher app you create.

Here's Pusher's list of supported browsers: http://pusher.com/docs/browser_compatibility

That's it!

At this point you should have everything you need to communicate with our Alert Birds. Go nuts!

Creating Heroku Addons: Getting Started

2011-09-27T16:46:08-07:00

Recently the Loggly add-on went live for all Heroku users. Creating this add-on involved implementing a few API endpoints in our service for Heroku to call. This is a quick overview of what the development process looks like.

Heroku's add-on API specs have a number of calls that fall into the following three categories:

Provisioning/deprovisioning: These are calls that ask the add-on service to create or remove resources on behalf of the Heroku user. One thing to remember here is that Heroku will ask the add-on service to create resources per Heroku *app*, not per Heroku user account.
In most cases, when Heroku makes a provisioning call to the add-on service, a new account is created for a Heroku app.
Single-sign-on: Heroku users do not need to provide separate account credentials to access services the add-on provides. As long as they are logged into Heroku, they are able to use or configure the add-on. This is accomplished by forwarding the user to the add-on's URL. A token and a timestamp are forwarded along with this request for authentication.
Plan change: Heroku allows add-on providers to change users based on tiers of service. A Heroku user is free to upgrade or downgrade add-on tiers at any time.

Each of these required actions corresponds to a REST endpoint that the add-on service

must implement:

POST and DELETE to http://<addon_service>/heroku/resources respectively provision and deprovision accounts.
GET to http://<addon_service>/heroku/resources/<id>?token=<token>&timestamp=<timestamp> does authentication and any necessary forwarding for SSO.
UPDATE to http://<addon_service>/heroku/resources/<id> will update the account's service tier.

Heroku provides an awesome tool called kensa that simulates API calls from Heroku on these endpoints to test the add-on service.

To install kensa open up a terminal and make sure you have Ruby and gems installed and type:

sudo gem install kensa

Kensa is used to generate the boilerplate manifest that describes properties of the add-on:


Brasero:blogpost ivan$ kensa init

Initialized new addon manifest in addon-manifest.json

Brasero:blogpost ivan$ cat addon-manifest.json 
{
  "id": "myaddon",
  "api": {
    "config_vars": [ "MYADDON_URL" ],
    "password": "QQMmzfm3pkzzE3Fn",
    "sso_salt": "eqs5xOh7kGqen5hd",
    "production": "https://yourapp.com/",
    "test": "http://localhost:4567/"
  }
}

Edit the id to name the add-on. Edit the test and production urls to reflect the hostnames of your respective test and development hosts.

Heroku doesn't call it as such, but kensa is also a tool that cleverly puts test-driven development into your add-on development process. When you run the tool in test mode, it makes REST calls to the host defined in the test directive of the addon-manifest.json file:

Brasero:workspace ivan$ kensa test provision

Testing manifest id key
  Check if exists [PASS]
  Check is a string [PASS]
  Check is not blank [PASS]

Testing manifest api key
  Check if exists [PASS]
  Check is a hash [PASS]
  Check contains password [PASS]
  Check contains test url [PASS]
  Check contains production url [PASS]
  Check production url uses SSL [PASS]
  Check contains config_vars array [PASS]
  Check containst at least one config var [PASS]
  Check all config vars are uppercase strings [PASS]

  Check all config vars are prefixed with the addon id [PASS]
  Check deprecated fields [PASS]
done.


Testing POST /heroku/resources
  Check response [FAIL]
    ! expected 200, got 401

The output above is typical from early runs of kensa, but by running the kensa tool early and often the test failures will guide development.

After all the provision tests pass, invoke the next set of tests for deprovisioning by running:

Brasero:workspace ivan$ kensa test deprovision <some account id>

Follow through with the planchange, and sso tests. When all tests pass, the integration is complete!

What remains is to upload addon-manifest.json to your Heroku provider account and to document the add-on for the Heroku Addon Catalog. More information on how to do that is available here.

Other tips:

Until the add-on leaves Beta, the only plan available to test is the test plan. Be sure your add-on has a test plan during development.
Don't go into Beta until the add-on is feature complete. An add-on cannot be taken out of Beta back into Alpha without help from Heroku.

Java logging, and our awesome community

2011-10-20T10:57:48-07:00

Since I'm on a roll with the blog posts, I thought I'd quickly cover some of the ways you can log out of your Java code to us. I've been buried in *our* java code for nearly two years now, and we've been talking about improving the quality of the logging that we're doing there recently. As it turns out, there are some pretty interesting projects out there that look like they may be just what we need...

Before talking about them, though, I thought I'd give you a quick run-down on how we're doing things now. We use a very slightly tweaked version of the log4j SyslogAppender (version 1.2.16) - the tweak is that we upped the message size from 1k to 32k. Yep, we're completely ignoring the syslog RFC, but its been working perfectly fine for us for quite a while, so I'm ok with that. We configure log4j as described on our wiki so that we log to a local syslog-ng (using a different facility for each app) which forwards to the appropriate ports on logs.loggly.com. Its a very simple approach, but has been very reliable for us.

So why would we want to change it?

The main reason is that we set this up before we could send JSON data, and we log a lot of performance data, which is a perfect fit for JSON. We're going to be moving all of our java logging to JSON over the next few months, because it will let us dive deeper into our logs, without all the noise.

A couple of weeks ago, Patrick Lightbody from neustar emailed us to tell us about a java class he'd written to send data into Loggly using http. Its an extension of java.util.logging.Handler and is only a couple of hundred lines of nice clean code . He shared it on github and described how to use it in his email.

If you're using java.util.logging, give it a whirl!

This got me thinking that we've been a little, um, remiss in communicating just how many libraries people have written for us, so I did a search on github for java projects with loggly in the repo name and found some other projects that also look pretty nice...

Mårten Gustafson has implemented a logback/SLF4J appender
Spidertracks has implemented a log4j appender that uses an embedded HSQL db to guarantee delivery of your log events
Raoul Schmidigerhas is working on a log4j appender
Thomas Engelschmidt is working on a clojure logger

All of these projects use our HTTP interface, rather than TCP or UDP, which makes sense since we don't currently support JSON except over HTTP. We're planning on fixing that :-)

Expanding the search a little, there are a bunch of projects in github, on top of the ones we've created, that should make it easy to log out of javascript, python, ruby, and C#, as well as the Java stuff I talked about above.

We're obviously pretty happy that so many people are working on making it easier to get data to us, and I want to say thank you to everyone who has been contributing to all of those projects. I'd like to encourage all of you reading this to jump in and help out if you can. Everyone who is contributing to these projects should drop us a line, and we'll send you one of our X-Ray Beaver tee's as a thank you :-)

Alerting with Alert Birds

2011-09-14T11:54:18-07:00

Back during my ISP heydays in the late 90s, I stumbled across a project called Peep The Network Auralizer. Peep basically takes things like netflow events from remote clients and sends them to a audio enabled server. Network traffic sounds like running water. Pings sound like birds chirping. SSH logins like frog croaks. The result is something that sounds like one of those sleep machines from the 80s.

What was really interesting about Peep, besides the fact it could lull you to sleep, was it could notify you of events that were important in aggregate but not necessaraly by themselves. A ton of failed SSH logins became a chorus of servers crying for help. High network load became a rushing river.

Now I look back on it, my use of Peep was my first pass at alerting and monitoring complex events in my infrastructure. Today it's also a good excuse to build an app for Loggly which makes a bunch of stupid bird noises.

Alerting Rules

Alerting is the #1 Loggly customer requested feature, so much so in fact that I broke down a few months ago and wrote a post on doing alerting with Loggly and Pagerduty.

After I wrote the post one of our customers commented that it sucked he had to run his own server to execute the cron jobs which ran the searches on Loggly. He's right. Requiring your users to run yet more infrastructure just to use a core bit of your service isn't the best way to win favors. And, it certainly doesn't scale well.

On that note, I'm pleased as punch to announce Loggly is now shipping alerting via the Alert Birds service. You can follow the Alert Birds on Twitter too!

Using Alert Birds with Loggly

Using Alert Birds with your Loggly account is simple. Navigate to the Alert Birds website and then log into your Google Account. Once you are logged in, you'll be given a form that allows you to enter your Loggly's account 'subdomain'. Once you submit your subdomain, you'll be taken to Loggly to login to that account and grant access to the Alert Birds' app. You'll be taken back to the Alert Birds' site when you are done authing the app to Loggly.

Alertbirds allows you to create an unlimited number of alerts, each of which are comprised of saved searches run on Loggly and endpoints that get called when searches return above or below a given threshold. Alerts are run on a schedule you define, and the endpoints currently (only) support pushing events to PagerDuty's APIs for notification. Alert Birds also allows you to trigger sound events in a browser viewing the Alert Birds' website.

Here's a short video on using the Alert Birds' application (sorry for saying um so much):

For more detailed setup info, you can reference our wiki page on Alert Birds. If you'd like to trigger an alert for Kord on his blog, click here.

Apps for a Logging Platform

We've Open Sourced the Alert Birds project to help illustrate how easy it is to build a logging application on Loggly. Alert Birds was writtien to run on Google's AppEngine service, but we'll be launching other apps soon that run on other frameworks and services.

Keep an eye out for more applications and app support coming out of Loggly in the near future. We're working on apps like a 3D globe for visualizing website traffic, web analytics, application reporting, and more!

IBM developerWorks podcast with Andy Glover

2011-09-07T15:50:50-07:00

Andy Glover (from App47) hosts a podcast on IBM developerWorks, and asked me to chat about some of the things we're doing here at loggly. I never realized how many ... pauses I take while I'm ... speaking. I'm going to pretend that this is because I'm ... carefully considering what I'm saying, rather than that I took a few too many knocks to the head back when I was playing Rugby.

Anyway, here's the link: http://www.ibm.com/developerworks/java/library/j-gloverpodcast3/index.html

Beware the strong gravitational pull of developerWorks. Its way too easy to lose hours and hours browsing through all of the awesome content there.

Thanks to Andy for letting me ramble on about the wonders of search :-)

DreamForce presentation

2011-09-07T15:47:20-07:00

Last week at DreamForce, I did a talk about how magical Loggly is, and a few people have asked to get copies of the slides. So, to make it a bit easier for everyone, here they are...

http://loggly.com/assets/4e67ec60dabe9d63dc002074/dreamworks_2011.pdf

Enjoy!

I was able to demo at DreamForce, so most of the exmaple slides ("Simple search" through "uniq") never saw the light of day there. They're included here, just for the sake of completeness.

Big thanks to Heroku & SalesForce for inviting us to DreamForce to celebrate us becoming an Add-on for Heroku - we had a lot of fun, and everyone seemed to love Hoover. Then again, how could you not?