Loggly by Hoover

Logs. This time it's personal

2013-05-04T13:42:05-07:00

Here at Loggly, we know logs are special. Why are they special? Because we know the specifics of the data that makes them special to you.

Logs are different. Logs are personal. Almost all monitoring tools and platforms reach into your systems and pull out impersonal data. Data such as CPU load, free disk space, and transaction times, are often critical but is not yours. But logs allow your system to reach out in a way only your system can.

I know, as a developer, the first time I saw the code I wrote, and the systems I built, send its logs to Loggly, it was like that system had extended its reach and was now a little bit resident on the Loggly platform too.

Many developers, when they first see their logs messages up in Loggly, feel a sense of excitement and pride. After all, they wrote each one of those messages. It's like watching the initial pulse of the system, as it comes to life.

So sign up with Loggly and send your logs to us. Then enjoy seeing your system reach out all the while letting us do all the hard work.

At Loggly we bring life to your system... by bringing your system to life.

At Loggly, we're not your father's IT company

2013-04-01T14:52:12-07:00

A few days ago an editor from CIO.com asked Loggly to provide two tips on hiring/retaining engineering talent for a future article. “Sure -- happy to help”, I replied. I then mentally outlined how we build and keep our highly sought after development team happy (out of ~~24 employees~~ 25 employees, another just signed, 22 of our hires write code). The standard Silicon Valley startup checklist popped up:

Competitive salary, equity packages, etc
Open vacation, flex hours, etc
Catered lunches, snack bars, and top-shelf beverages
Latest-greatest technology and development environments

But to really attract, build and keep top talent you need more. You need the ability to:

Work on a product that thrives on big data for a formidable challenge ($100b market, per IDC)
Solve a true market created big problem, that matters... and one that people pay for
Make a difference, and be recognized for it.

The last line was the true angle to build off and two examples quickly came to mind: how we target our hires and welcome them, and how we keep engineers aligned and motivated on the problem their code will solve, essentially the direct connection to our customer's log management challenges.

I’ll skip the tip I sent over on our on-boarding process, maybe I shouldn’t ever share it ;) but the second tip was about sending our developers to work a half-day in at the Loggly booth at a tradeshow. After reviewing our article suggestions, I heard back from the CIO.com editor:

"Thanks for the tips... Unfortunately, they weren't quite right for the article.
(Every IT guy I have ever known has DREADED being asked to man the booth or attend trade shows and conferences)."

I was shocked... and then I wasn’t. It only confirmed what I’ve already known, we are not our father's IT company. We aren’t:

Building an old-school IT product (legacy deployment cycles aimed at non-cloud savvy companies run by stiff CIOs)
Selling an old-school IT product (hundreds of end-users who are also our buyers sign up WEEKLY)
Selling into the CIO and expecting our technology to be forced down (vs adopted, loved and shared)
Looking, hiring or retaining old school ‘Dilbertized IT guy' engineering talent who hides at the mere thought of customer interaction

So why would I expect CIO.com to immediately see the value of the tips? I shouldn’t. At Loggly, we are different by design. We've developed a 100% cloud-based service loved and used by over 2,500 of the industries leading cloud-centric brands (AirBnb, Adroll, Sony, EA, GrubHub, Heroku) and we have a fun mascot named Hoover people love and want to interact with (here's Loggly's Hoover vs. the @Spotify shark at #Pycon).

This translates directly into my role driving Marketing and Revenue where I look for innovative developers and SysOps people (guys and gals) who are looking to simplify log management (we can help you!! Free 30 day trial of Loggly), and it flows directly into who we hire, how we retain talent and how we motivate our teams.

Heading into PyCon, I had the envious Marketing challenge of selecting booth staff after getting numerous emails, Skypes and SMS from engineers asking if they could attend and work the booth, a great problem to have. As it turned out, our CTO and two lead developers attended for a half-day and it served to energize them about:

The billion dollar opportunity we are solving in log management (the booth was busy!)
The ability to hear from customers and prospects exactly what they want in a product and how they are using it (the good and the bad)
The way they can make a difference with their efforts to improve and continually optimize the product (our customers enjoyed the face-to-face conversations)

Our engineers arrived at back at work on Monday really stoked to share their booth experience, their conversations, and on how they can make the next killer version of cloud-based log management to solve the challenges of our next 2,500+ customers.

So while the tip wasn’t good enough for CIO.com, I’ll take that as badge of honor. We are doing something different, things are going great and people are excited to play a role.

We are still growing fast and hiring, and NEED more smart, fun loving people to join us. We have current openings for:

If you aren't the luddite "IT-guy" and a person looking for a change, let's chat! Or please send a peer our way.

Cheers!

Use the source, Loggly

2013-04-15T14:23:46-07:00

Here at Loggly we make great use of open-source software to avoid remanufacturing wheels. It allows us to employ the latest technologies to build exciting, interesting and large-scale systems quickly -- all without worrying that the building blocks will forever remain a black-box. But working with open-source at such an accelerated pace requires its own set of skills. Here are five key lessons learned from our Engineering team -- that might help you.

1. Assume at your own risk 

It's easy to assume that the project developers implemented a certain feature in a certain way, and before you know it that assumption becomes taken for granted. Without realizing it, one is building a system on that assumption. This is insidious and one needs to deliberately check the source and test the system before making any major decisions. Time is short at a small company like ours, but it'll be even shorter if you have to re-implement due to an invalid assumption.  Also, never make customer adoption assumptions in a vacuum, the Steve Jobs effect is awesome but rare. Leverage your customer community for responsive feedback.

2. Community support 

Look for tools that can help debug, deploy, and manage the technology. If the tools ecosystem is healthy, then the open-source technology is probably vibrant too. Your sysadmins will thank you too, if they don't have to build everything from scratch. And take the pulse of the community. Join the mailing lists. Do the same questions come up over and over again? And do they get answered?

3. The project's strength of vision 

We at Loggly always feel better when the direction of the project aligns closely with the problems we're trying to solve. For example, ZeroMQ is a technology with a clear purpose. Problems arise when this clear direction is missing, because the items high on your list may not be high on the project maintainers' list. 

4. Gambling on the "Killer Feature" 

Sometimes you just have to take a gamble because of a "killer feature". ZeroMQ is a good example, a technology we chose during its early days. Data-driven experiments showed that it was an order of magnitude faster and less resource intensive than the alternatives, but it was a very young project at the time we chose it. But we chose it because of its performance, even though it it was immature -- but it did have a strong vision. 

5. When there are multiple contenders, choose the one that is weak on the things you want to be strong on, and strong on the things you'd rather be weak on. 

This is my favorite insight from the Loggly Engineering team -- it's comparitive advantage applied to software design. Engineering is about trade-offs. Imagine you need to choose from 2 competing technologies. One has great algorithm support, but is weaker when it comes to IO. The other technology is not as strong when it comes to algorithms, but has superb IO support. So if you're planning to design and implement novel algorithms, you're going to become strong in algorithms -- in fact you want to be strong in that area. So choose the system with the strong IO support, and focus on the algorithms yourself.

  Hopefully these pointers help you when you next come to evaluating an open-source technology. We believe it shows in the adoption of our world's most popular cloud-based log management service.

Even better, why not join us and help choose the technology for Loggly's next generation systems?

If you love your logs, set them free

2013-03-12T12:33:15-07:00

Imagine this. It's the night before Thanksgiving, you're having dinner with your wife and in-laws at a nice restaurant. Your cellphone rings. It's the Senior Director of Engineering, your boss, responsible for the new Cloud Portal you and your team brought up a few weeks back. This can't be good. And it happened to me, at a big well-respected engineering focused company. #truestory

It turns out a key customer had decided on this night, of all nights, to place a large order for licenses, but our Portal is returning HTTP 500. "What could be happening?", he pleads. "Hmmm, can we get the logs?" I reply. "We can't", he tells me, "Operations never set up systems to take them off the box." Sigh... Sound Familar?

This happens all the time within IT organizations. It turns out that one of the worst places logs can be stored is on the machine generating those logs. What if that machine contains Insider Information, and access to it is restricted? What if the network link to it is down? What if all you can do is think really, really hard to work out why that key customer can't place that order? But it doesn't have to be this way.

If you love your logs, set them free. Get them off the host machines. You, your boss, and your customers will thank you when you can diagnose issues from anywhere, and in real-time. Get them into Loggly, and let us do the hard work for you. Best part, you can do it all without on-premise software, local agents, or disrupting production machines.

Anyway, my boss and I solved the issue, and the order was placed, but it took until the taxi ride home. Life Lesson? Enjoy Thanksgiving and other holidays stress-free... learn from others so this doesn't happen to you.

The Truth is in the Logs

2013-02-14T22:31:36-08:00

Last Friday, The New York Times reporter John Broder published a less than rosy picture of highway trip between Washington D.C. and Boston, cruising in Tesla’s Model S luxury sedan. The purpose of the trip was to range test the car between two new supercharging stations with a "speedy road trip". Broder wrote about his anxiety-ridden stretches between charging stations, when energy consumption was outpacing mileage. Among other complaints, he was unhappy about the need to power off the heat on a cold Northeastern day and to drive slowly, in an effort to conserve battery power. The trip ended not at the charging station, but on the back of a tow truck having run out of electricity―a result he visually documented with a feature photo of the car being towed. Ouch.

This wasn’t the PR outcome that Tesla Motors chief executive Elon Musk expected. Avoiding a factless he-said-she-said on Twitter, he gave an interview saying that he was planning to publish the log files of the reporter's vehicle, since they were at odds with details in the story. As reported in Venture Beat: “Musk claims Broder failed to mention how much he was punching the accelerator early in the ride, a move that Tesla warns its customers will drain the battery faster. Also, Musk says Broder took a detour through Manhattan. And he didn’t fully charge the car before departing.”

NYTimes article about Tesla range in cold is fake. Vehicle logs tell true story that he didn't actually charge to max & took a long detour.
— Elon Musk (@elonmusk) February 11, 2013

While the logs have yet to be published, if they are and if they do support Musk’s claims, it will be advantageous for Tesla -- and the “logging” community at large; bringing light to the next big category of business intelligence. "Your honor, I'd like to call the #LogFile as my next witness." Amusing, but true and will be happening more and more often inside and outside of business.

As this story demonstrates, software is behind everything these days. Massive machine data is being generated every minute not only from our computers and cell phones, but from the cars we drive, the appliances we use, and virtually anything with a chip, motor or battery run with an operating system. The golden nuggets of truth exist within the log files, holding the detailed data about an event that happened, one that cannot be disputed.

Such data, when mined and organized properly, provides a wealth of indicators to solve problems, such as why a web transaction timed out, the difference between a server being “on” and actually performing as intended, or to defend the veracity of a product’s claims. The tremendous value for IT departments (sysops, techops, devops and product developers) in having real-time access to log data for feedback on product performance is limitless. Log files can show where users struggled or took too long to accomplish tasks, or where your applications or hardware let them down. In trial-by-pubic cases like these log file data shines in a new light, far outside of the walls of technology but in the vernacular of the general public.

Logs don’t lie, but the truth can’t come out unless there are affordable and easy ways to release insights from these enormous log databases within the window of time in which they matter. Moving the conversation forward, suppose Tesla was able to collect and analyze log files on all of its vehicles and then receive alerts on issues to determine if they need addressing or if they are simply isolated events caused by user error, right back to the driver before the situation arose. That's the end game: taking aggregate user data to find nuggets of wisdom that can then be fed back to the end user with guidance -- seamlessly.

That's where we come in. Loggly’s cloud-based log management service gives companies fast, centralized access to all of their log data, so they can solve issues, identify problems and make customers happy again understanding and answering "is this needle in the haystack or tip of the iceberg". In the court of public opinion and social everything or in the case of 100% cloud driven buisness... anything less than real-time is becoming really-late.

When consumer product CEOs start talking about logs, that's a pretty good sign that log files and log analytics are not just a stream of text and data to throw on the backup server every night―if you didn’t run out of storage for them already. Smart log file mining helps companies keep customers happy and their bottom line bigger. If you run a data-driven business, the more your company can act on that data to improve application/service/product performance and experience, the better off your customers will bef.

If Elon Musk is driving the future of Tesla, an automobile brand off his log files, shouldn't your cloud application-driven business also being harnessing the full power of log intelligence?

New Relic + Loggly: Working Together

2013-03-13T21:54:15-07:00

We're really pleased to announce that New Relic and Loggly are partnering to join forces and bring an integrated, multi-faceted view on application and system behavior to stream line root cause analysis. We've long been fans of New Relic for monitoring the performance of our systems; it's pointed us towards a number of areas where we've been able to make huge performance gains. When our operations team started to complain that they could see when there was a problem in New Relic, but that it was tedious to correlate the same time range within Loggly, we knew many of our 2,500+ users probably had the same challenge and would love a better solution. So why not build a free one?

Introducing the 'New Relic - Loggly Integration' Chrome Extension
Deployed with a few clicks from the Chrome Store, our operations team is now able to pinpoint system performance issues in one click, and identify the source the problem in seconds... aka solve issues faster and get work done. Using New Relic, they get plenty of server stats that tell them when a problem occurred and now they can click through to Loggly and see the log events that directly correlate to the graph that they just looked at within New Relic. What a timesaver, and it is free!

How can you get this goodness?
If you already use New Relic + Loggly, you're just a minute away from taking advantage of the integration. Download the free extension directly from the Chrome store. If you want to experience the power of New Relic for yourself, we have a special offer with our partnership to offer Loggly users New Relic Standard for Free. Visit newrelic.com/loggly and sign up for free!

Note, because Loggly has an open API, it'll be very easy to continue improvements to the New Relic + Loggly extension. Let us know if you'd like to see more enhancements or integrations with other products via email or in the comment section.

Happy logging and now with even easier root cause analysis.

Graphing Numeric Data

2012-09-06T14:14:04-07:00

Here's another reason to start logging with JSON log data. It really is the best & easiest way to get more insight out of your log files. If you're still logging plain text, think about all of the numerical data that's going to waste because it's a pain to analyze. If you're sending us numerical data in JSON format, we can graph it for you. Analysis is now simplified!

Whether you want to see the day's CPU load or the month's memory usage, the graphs are easy to set up. Run the "graphjson" command directly from the Loggly Shell. Provide us with the key names of the values you want to plot and a search query. The x-axis uses the Loggly time stamp and the y-axis uses the numeric value of the specified JSON field.

Check out the video & wiki page for help.

Uniq-uely Different

2013-03-09T06:52:31-08:00

We've extolled the merits of sending in logs that are formatted in JSON for a while now, but I've been holding off on this post because I wanted to have something really exciting to tell you about.

To bring the non-JSON users up to speed: you can format your log output so that it will send something like this:

   { 
     "Name": "Hoover Beaver",
     "Age": 30,
     "Occupation": "Tree Feller",
     "Address": "30 Hoover Dam Road"
   }

Instead of the typical unstructured log events:

"Hoover Beaver" 30 "Tree Feller" "30 Hoover Dam Road"

This means that you can search for "json.age:30" and know that you're only searching the Age field rather than the entire event. Don't bother learning how to awk/sed, just tell us what field you want.

Now that everyone's on the same page, I can tell you about one of my favorite analytics features. We've made some enhancements to our command "uniq". "Uniq" works by collapsing duplicates and returning a word count. For example, if I want to find out the distribution of twenty-somethings across all Tree Fellers, I can run the following command which will give me a count for every unique age between 20 and 29 across all Tree Fellers:

 > uniq json.age json.occupation:"tree feller" AND json.age:[20 TO 29]

count       json.age
__________________________
455             23
358             21
276             24
121             28

This also works well with strings. If you're interested in seeing the distribution of error messages since your last big release, try something like this:

 > uniq json.error json.version:2.1

count       json.error
__________________________
146             ERROR 55: Trees are falling in the forest
102             WARNING 21: Too many beavers in the stream
89               ERROR 29: Can't find the right river
73               ERROR 98: The dam has broken

In the past, we used tokenized strings to report back uniq values, which made it quite difficult to filter out the useful information. Today, we're storing fields < 100 characters as both tokenized & untokenized strings. This analytics feature is available to only JSON users right now. If you need advice on how to change your log format & make the switch, email our support team.

Try it out and let us know how it goes!

edit: If you want uniq values through our API, try using Facets. Try something like this:

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD?q=inputname:INPUT%20json.FIELD:VALUE

Currently uniq search terms are limited to one within the UI. If you'd like to use the uniq command for more than one term please use the API:

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD/?q=json.FIELD:TERM1%20json.FIELD:TERM2%20json.FIELD:TERM3&from=NOW-7DAYS&until=NOW

Building Great Software...5 Cool Tips For Success!

2012-03-15T16:08:03-07:00

Writing software is hard. Writing it well is even harder. I have been rolling some ideas around in my head on how to make software that is better for the people creating it and the people using it. Software has some distinct (and not-so-distinct) phases that it goes through as you work on it. The first of those phases for developers is the design phase. Here are some ideas on making better software during the design process.

If you start with "you know what would be cool?", kill the idea.

OK, so maybe that's a little harsh, but that's my knee jerk reaction when I hear a "you know what would be cool" idea. Let me rephrase my suggestion; stop and think. You may be wondering what I have against doing cool things. Nothing. There's nothing wrong with doing cool things. The problem is that we developers gravitate toward using a cool piece of technology for the sake of using it rather than finding the best fit for our problem. We often attempt to solve a problem either using a cool new piece of technology in a way that it wasn't intended for, or new language features when an alternative would have been better suited. Redis, mongodb, rabbitmq, Python, Ruby, Clojure, Scala, Node.js are all great technologies that meet various needs in their own ways. Sometimes they may be the best fit for your problem, but not always. Just think twice when you think you're about to do something "cool". Developing software is less about being cool and more about building something that works well.

Aim for simplicity.

Software developers have a knack for building and stacking and inheriting and composing until the original simple solution is unrecognizable in a mound of clever engineering. When you're thinking about the next piece of code that you're going to write, start simple. Don't build something in your mind that is more complex than it has to be. You should always use the simplest thing that works, except for when you shouldn't. Next point.

The simplest solution isn't always best.

Sometimes the simplest thing just won't cut it. Sometimes you need that really weird class hierarchy or a really tricky algorithm or something that really hurts your brain to even think about. Usually this is going to happen on the second iteration of a problem. You should still always start with the simplest thing. If that doesn't seem to hold up, go on to something more complicated, but keep the complexity to a minimum.

Don't build more than you need.

It's often tempting to try to build a solution that handles every contingency or meets the needs a user doesn't even know what they have yet. When you are tempted to design everything into your application, think twice. Don't design what you don't need. If you can come up with a good reason for designing it in, then go for it. Otherwise, don't bother. Build just what you need and no more.

Think long-term.

I've seen so many developers (including myself) focus on getting something working now and neglect foreseeable issues with maintenance and deployment (which are the two major areas that this impacts). When you are thinking about the next piece of code that you are going to write, take the time to make sure you are focused on the long term solution. This may sound like it contradicts my previous idea of doing the simplest thing possible, but I think these ideas can co-exist. Start simple. Get something working. But don't plan on getting it "justworking"; plan on getting it working so that the next generation of developers that work on it will be happy you made the decisions you did rather than having them curse your name.

These aren't new thoughts. I've heard them stated by other experienced developers in various ways. And they're not immutable laws of nature. I think they're good guidelines, but the world won't come crashing down on your ears if you do something different. But I do believe that your experience of designing software will benefit from it. And as a result, your software will be better for you and your customer.

Introducing the Loggly Wood Shop

2012-03-02T09:40:14-08:00

We've been talking about doing a Loggly swag store for a while, and I'm excited to announce it's finally here! Now everybody everywhere has access to our snazzy swag. We have some rocking t-shirts, stickers and beer glasses available for purchase. I'm not just saying this because I work here, when I wear Loggly t-shirts out on weekends people compliment me on them.

We’ve always made a point to produce swag that has lasting appeal. All of us have gone to conferences and collected a bunch of stuff, only to take it back to our room, look at it, and leave it there. We are proud to say that people actually wear our swag out and about, track us down at events specifically to get their hands on it, and tweet at us to ask about how they can get it. We've even been offered money for the shirts off our backs, no joke. We are incredibly lucky that the allure of our brand continues beyond our product. Even folks who’ve never used our product, and never will, love the branding and culture we’ve created and want to be a part of it. The Loggly store was born from this enthusiasm and support, and we’re extremely grateful for it.

We’d like to give a special shout out to our fantastic illustrator Andre Jolicoeur who has done an outstanding job bringing Hoover the beaver and the Loggly brand to life with his creations. We also owe a great deal of praise to Shopify, whose service made it a breeze for us to get our store up and running. The Shopify Textmate bundle made development simple and faster than I had anticipated, and the entire launch process was demonstrated clearly through the Shopify interface.

To celebrate our launch we’re offering 20% off all merchandise this week, redeemable by using the discount code beaverlicious at checkout. Happy shopping!

Partnering with Scalr Gets You a Margarita

2012-02-23T16:12:41-08:00

We’ve all dreamed of doing our job while sipping a margarita on the beach, Loggly and Scalr can get you there...well almost. We love things that scale and automate and all things cloud and that is why we partnered with Scalr. Scalr is an open source cloud management tool that brings automation to web applications. The cloud is all about scalability, growth, and making things easier to manage. Scarl gives you the ability to manage as many servers as you need on different cloud computing services and adjusts load capacity as you have spikes and valleys in your traffic. It’s like a system’s admin machine that never sleeps, doesn’t need energy drinks and coffee, so now your system’s admin can actually can have a life. Loggly comes in and takes care of all the logs, we store them, make them searchable, as well as providing features such as monitoring, troubleshooting and user analytics so nothing is lost and you can figure out what your users are doing. Sit back, relax and see what Scalr can do for you.

DNS Management: automatically created and updated for you
Fault tolerance: servers crash, at the worst times, now the problem is detected and automatically resolved for you
Multi-cloud deployments: no need to commit to just one, infrastructure can be spread out across multiple providers
Integrated SSH and key management

We're super excited to be partnered with such a cool company that's helping web apps scale and making your life easier.

Big Data Uncovered?

2012-02-08T15:14:58-08:00

I recently came across an eWeek article titled 2012: A Cloudy Year for Big Data by Frank Ohlhorst. You could easily say I have a few opinions on the matter of big data! :)

First, I agree with Frank’s first notion that big data is neither big or new. The fact is, I've been saying things like "Dude, that's a ton of data!" since I started notching out the opposite sides of floppies back in the 80s. Remember these?

Ohlhprst quickly follows up his vague handwaving that ‘big data’ a new term with, “For most of its existence, big data has been out of the reach of small and midsize businesses (SMBs) because the storage and processing power needed to make this technology work is too expensive.”

Companies have been doing for years what they need to do better business, regardless of whether or not it’s expensive. In manufacturing, the costs of a small company optimizing on how to efficiently making tons of a cheap product can actually be quite a bit more expensive than a larger company making a few units of a complex product. In the same vein, smaller business may have more complex business optimization processes than larger ones, and require relatively larger amounts of data are required to solve those problems than with larger companies.

I agree with Frank that small business typically don't always have the resources necessary to solve massive scale problems, but again the problems are relative. For example, small software startups don't have project managers where larger ones do, not because they can’t afford them, but because they really don’t need them in a full-time capacity. I think this may be part of why SaaS services have been a huge hit and the term cloud has taken off because of it. SaaS allows companies to tackle a wide variety of problems across the entire business, all the while providing cost effective high tech solutions to solve problems in a way you could never have done before. For the first time in history the quality of business processes is experiencing sustained growth.

“These new cloud-based capabilities are on a growth path and are creating more opportunities for even the smallest of businesses to leverage big data without the traditional expenses of compute farms and massive storage arrays.”

Yes. However, compute farms and storage aren't the main thing that these companies need. They need access to the raw data that contains the data about their business, and the tools to extract the data in which they can take action. Figuring out your company’s problems requires brain power, understanding, data and tools. CPU and disks don't solve complex problems. People do.

It's All About Application Analytics

Ohlhprst also describes big data analytics as being comprised of three primary elements: volumes of unstructured data, processing power and algorithms. However, big data doesn't always imply unstructured data. Log files, or what Loggly calls Big Time Data™ typcially contain a large amount of structure. Dealing with structured data isn't always easy, and if you write software that 'expects' a certain structured format, your analysis can sometimes be broken or flawed if it encounters data that doesn't fit the structure you coded for. One way around this problem is to apply extra meta data to the data set. One technique to solving to this problem is adding a search index to the data, which is the approach Google pioniered and what Splunk and Loggly do for log files or time series event data. By being able to do text search data, and interact with it in realtime, or near real time, the user can optimize on solving the problem

Ohlhorst continues, “For it to be true big data, there has to be lots of it, and most SMBs don’t generate that volume of data internally, which leads them to seek out alternative data sources. Here, the cloud delivers.” Not true. Big data should be defined as an amount of data that a human can not reasonably digest. Generating large amounts of meaningful data is actually a bigger problem. Again, it's understanding the problem you have before you can solve it.

Yup. Ohlhorst explains that throughout 2012, data sets and others can be expected to grow exponentially - “The amount of data being generated globally increases by 40 percent a year", according to the McKinsey Global Institute, a data analytics research firm. True. The access of this data, mostly through the web, generates vast amounts of data as well. Ohlhorst continues that information needs to be organized, sorted and processed- and that takes computing power. Frankly nowadays, CPU is cheap enough that most of these problems can be solved on your laptop. Fast CPUs for crunching 'big data' aren't the problem any more than a search engine's main problem is crawling for data. The real bottleneck is adding meaning to the data that a customer can digest and make actionable.

PaaS/IaaS Accessibility Is a Problem

I’m glad that Ohlhprst recognized that Amazon isn’t the only one in the game in offering private cloud-based big data analytics platforms. He believes that since this technology is designed as a complete platform and not as a service, these platforms are still out of the reach of the SMB market.

Ohlhorst is right that these platforms are out of reach - but not just because they are designed as a complete platform and not as a service. I think it's because SMBs don't know they need it, don't have the data to put on it, and don't have the resources to manage it. There are plenty of hosted solutions out there (see SalesForce and their app marketplace) that provide some serious horsepower to the most important task - managing a company's contacts.

And of course, there had to be a Splunk mention in his article. Splunk sells expensive enterprise software. Their software is often times the most expensive piece of software a company has ever bought. Sounds like Oracle, eh? They aren't converting big data analytics into cloud services; they are simply taking their product and making a slimmed down version into a cloud offering they can generate leads with. Any serious big data customer they land will have to buy that very expensive solution and install it on a bank of computers and then pay people to manage it. SaaS is not what Splunk is taking to the market when they go IPO. It's their hellaciously expensive software licenses.

Big Time Data™. It's in the future of your small business.

Getting Your Product Sticky

2012-01-16T23:22:31-08:00

A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators. When it gets dark, we forget everything you've typed into us."

That comment wasn't far off the mark. Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session. Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point. It made it extremely difficult to get back to something meaningful the next time you logged in.

We shouldn't be here if we aren't meaningful. We should deliver users a 'punch in the gut' feature that makes a lasting impact. One they don't want to avoid.

Saving Time with Sticky Features

Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority. Unfortunately us solving scale issues aren't readily obvious to users. Users always expect things on the web to be fast. They could care less how hard a problem it was to solve. They don't think to themselves, "Wow, that's fucking fast!". No. Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y? This thing is wasting my time!".

And there it is laid bare: Don't waste your user's time. It's the most valuable resource they have. Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.

I say give them sticky features!

Saved Search and More

And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search. Saved search provides users a way to write a search query and then preserve the search to run again later. Saved searches can generate facet graphs or they can simply run a regular search across a given time range.

The shell has been reworked to provide context changes to use with the saved search feature. You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.

Here's a quick screencast running through some of our new sticky features:

Coming Up Soon

We're continuing to add features that increase stickiness to the product. Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt. We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.

All these featuers are leading up to a major revamp of the way we provide value for our user's events. Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon! If you have a feature you'd like to see us implement, please do drop us a line. We're keen on not wasting your time!

Loggly's Outage for December 19th

2012-04-30T17:28:38-07:00

Sometimes there's just no other way to say "we're down" than just admitting you screwed up and are down. We're in the process of rebuilding the indexes of historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.

So What Happened?

Yesterday afternoon all of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff for maintenance purposes.

The cause of our failure is what some of you on Twitter are calling "a failure to architect for the cloud". I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes". We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.

While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough. The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.

In this case we didn't build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing. With other challenges facing us, we decided to live with the risk.

So, How Do We Make This Right?

Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes. We've been OK with those limitations in the past simple because we systematically archive data to S3 when it arrives and we are capable of rebuilding indexes on the fly if we lose one or more indexers.

Our primary method to address this will be to start sharding our customers across multiple Loggly deployments. This will prevent further outages to the entire customer base. We've already been investigating other data centers on both dedicated hardware and other cloud-based services.

Finally, we accept full responsibility for the impact to our customers. We will be in touch with our paid customers sometime over the next week to address compensation for this outage.

We welcome feedback below.

Kord Campbell, CEO

Enabling CORS in Django Piston

2013-03-09T06:55:38-08:00

Here at Loggly, one of our goals is to make our API accessible and easy to integrate. By enabling CORS (Cross Origin Resource Sharing) on our API endpoints, we hope more Javascript developers can take advantage of what our product has to offer.

CORS is an addition to the browser security model that allows XHR requests to be made from one domain to another. CORS allows Javascript applications to access resources on domains other than the original document's domain, working around the same-origin policy. While Javascript application developers have crafted techniques like JSONP, Flash proxies, XHR receivers, and server-side proxies to circumvent the same-origin policy, CORS makes these hacks unnecessary.

To take advantage of CORS both the server and the browser need to support the standard. The browser needs to initiate a negotiation with the server and the server must signal to the browser which domains are allowed to make cross-domain requests. Our current API is implemented in Django Piston, an open-source project that enabled us to quickly build a RESTful API on top of Django. Piston does not support CORS out-of-the-box, but it wasn't hard to write some code to enable it and we'd like to show how it was done.

A full explanation of CORS is beyond the scope of this post, but the central idea behind CORS is a negotiation between the browser and server of allowed and disallowed actions. This negotiation is done via HTTP headers. The essential headers are the following:

Origin: Sent by the browser signifying the originating domain.
Access-Control-Allowed-Origin: Sent by the server, listing the origin domains allowed to make requests to the server's domain. Can be a comma-separated list of domains or "*" to allow requests from all domains.
Access-Control-Allow-Methods: Sent by the server, listing the HTTP methods the browser is allowed to use in requests to the server.
Access-Control-Allow-Headers: Sent by the server, listing the HTTP methods the server is willing to accept from the browser.

Essentially, to enable CORS we need to have Django Piston respond to an OPTIONS request with the server-sent headers and send the requisite headers along with responses.

The Resource class is the heart of a Django Piston-built API. The code that injects the headers into responses lives in a subclass of the base Resource class. We've called this class CORSResource:

The CORSResource performs two simple tasks. First, it intercepts any OPTIONS method requests to handle the pre-flight negotiation between the browser and the server. Since OPTIONS requests do not have a response body, an empty HTTPResponse() is returned along with the requisite headers. Second, CORSResource intercepts responses from the Django Piston handlers (where responses are generated) and decorates them with the CORS headers.

To use CORSResource, we simply instantiated our endpoints with the CORSResource sub-class instead of the base Resource class. The change to our API's urls.py file look like this:

We hope this post helps other Django Piston API implementors enable CORS in their own APIs. We're planning to release this implementation in the coming weeks and we're looking forward to see what Javascript developers are going to do with direct access to our API.

Happy hacking!

(image from http://blogs.bournemouth.ac.uk/research/2011/09/01/sharing-your-research-data/)