Zenoss Blog: No Node Left Behind

The Sound of One Server Pinging - Announcing Zenoss 4.2.3

community@zenoss.org — Thu, 10 Jan 2013 01:24:21 GMT

I sat zazen alone in an abandoned monestary atop a clouded mountain, the smoke of burning oud drifting down from the incense holder. And then I heard it upon the wind, "what is the sound of one server pinging?" The question resounded, only a whisper, but a whisper from everywhere, all directions, all at once. The whipser contained within it everything and nothing. My mind was filled then with chaos and confusion as I returned, though at my core disquieted, to the routine of my meditation. Day in and day out, one server pinging, echoing insistently in my mind. A great despondance settled upon me, was this task too much for us? Could we bear the burden of what it meant we would do? I consulted then the great master, begging for another path, a lighter load to carry. The master said nothing, and shook his head, consigning us to our fate... It is time to release Zenoss 4.2.3.
The importance of the sound of one server pinging though, I understood it now even as the master shook his head. It is frantic power, on call admins sleeping peacefully through the night, clouds bringing productivity to the masses anywhere they found themselves, and IT managers living a life without stress. With this great task we struggled, the weeks drew into long months and the weather grew cold. Still, though, we heard it, loudly now, clearly now, the sound of one server pinging. Coming down from the mountain, the great coders and engineers of Zenoss give their great accomplishment to the world, patching the bugs, unifying the RPM installer, and providing security out of the box. We at Zenoss have released 4.2.3.

The State of Open Source Cloud Adoption [Results & Winner]

community@zenoss.org — Tue, 30 Oct 2012 14:47:26 GMT

Last month, we reached out to you for your help with a survey on The State of Open Source Cloud Adoption. While doing surveys at Zenoss are fairly common, this survey was the first of it's kind. To our knowledge, no one until now has made an effort to get a snapshot of open source cloud adoption. That being said, the results produced recieved a lot of attention from a variety of media outlets and gave the world a better look into the actual deployment of open source clouds. Feel free to read the full report here and check out the infographic below.

Building on the successful reception of the survey, we took the results a step further. Last week, we hosted a roundtable discussion with representatives from CloudStack, OpenStack, and Ecualyptus. We wanted to get a candid perspective from the three most popular platforms and really discuss how they see the future of open source clouds. You can check out the recording of the roundtable discussion here.

To those of you who participated, thank you for your valuable insight and time. Also, if you opted-in and left your email address at the end of the survey, you were eligible to win a Google Nexus 7. We did the drawing last week and shipped off the Nexus 7 to Russ Wenner. Congrats Russ and thanks for taking the survey!

Big Data Monitoring Meets a Little Android

community@zenoss.org — Mon, 17 Sep 2012 18:19:33 GMT

Zenoss Blog Post

This is an original post written for the Community Blog by Gareth Llewellyn.

The DataSift platform consumes, processes, filters, stores and streams terabytes upon terabytes of social data to businesses around the world and we in the Operations team needed to know of any problems then acknowledge and investigate them as quickly and easily as possible.

Zenoss had long been the mainstay of our monitoring for general server and application health but we were planning on extending our monitoring by coupling it with metrics that were to be emitted by every piece of software and hardware in the platform.

Utilising a fork of Etsy’s statsd, a fork of Graphite and coupled with our own interfaces into each; we started gathering thousands of metrics a second. These metrics are displayed on screens in the Operations area and are also evaluated for erroneous behaviour by another piece of inhouse software^[1].

(The strobe light that can be seen in the picture is also hooked up to Zenoss; http://www.youtube.com/watch?v=OqkILJCdCA0 )

When erroneous events are detected they are sent to Zenoss (having been deployed and configured by Chef) which automatically knows about the sever or class to which the metric relates to and can set the severity accordingly (unless we have sent an override as part of the JSON API request).

This got us halfway to our goal, Zenoss was now monitoring the platform using a variety of Zenpacks as well as being informed of platform metrics and what they mean in regards to health. However we weren’t yet in control of the alerting and acknowledgement if we were on the move, in a meeting, asleep in bed or otherwise not near a desk or easy reach of a laptop.

Being open-source with native JSON support, HTTP libraries, background services, notifications and everything else we’d need to create an app that would alert us within moments of Zenoss detecting an issue, allow us to acknowledge the event, delve into the details and all from a phone or tablet, Android was an easy choice. The only problem was that someone had to write it.

Meet Rhybudd

Rhybudd is Welsh for ‘warning’ which whilst being difficult to pronounce fits perfectly with the idea of a piece of software that responds to events and warns you.

As Rhybudd uses the Zenoss JSON API it is quite easy to perform the initial authentication:

GIST Embed: https://gist.github.com/3407638

Once the httpclient object had the cookie returned by this request one can call other API functions quite simply:

GIST Embed of https://gist.github.com/3407692

With Android versions of each API endpoint complete the app needed to utilise them on a regular basis to poll Zenoss and then process the alerts:

GIST Embed of https://gist.github.com/3408033

If Rhybudd detects an event that matches your configured filters it will then raise a notification with sound and vibration (if enabled) containing a brief overview of what caused the alert.

In older versions of the application clicking on the notification would then launch another poll which depending on your connection speed and other factors could cause an unacceptable level of delay before you could start acknowledging events. One of the key tenets of the Android Design Guide is to make an app feel responsive so Rhybudd will now only force a refresh on launch if the data in the local cache is determined to be stale. This means the latest events will be ready to view almost instantly each time you launch the app (with background polling enabled).

Performing a single tap on an Event will either display a popup on a phone allowing you to acknowledge or view details of the event whereas a tablet will shift itself around to instantly display information about the event in all the extra screen space available and provide an acknowledge button.

You don’t however have to tap on every event to acknowledge it, you can long press on several events to acknowledge all selected events at once or tap the acknowledge all button which would as its namesake suggests acknowledge all currently unacknowledged events.

Rhybudd has other features and functionality that can be read about in more detail on the Zenoss wiki but at it’s core it is an app that can poll your Zenoss instance from as often as every 30 seconds up to every hour and then can alert you via Android notifications with an alert sound of your choice (it can even be set to repeat the notification sound indefinitely until you acknowledge the alert if you’re a heavy sleeper!). Once alerted you can acknowledge the alert, delve a little deeper and even add log messages or escalate to colleagues.

Problems and Mistakes

Not all UI elements need to be used. Originally Rhybudd offered the user a seek bar slider to specify the time between polling intervals, UI wise it made sense to me (no doubt in the early hours of a morning) but many people found that trying to move a tiny circle across 3” of seek bar with an accuracy of 1pixel / second incredibly infuriating. It’s now a drop down menu.

Backwards and forward compatibility with the remote platform is a requirement. When Rhybudd was first released it had only been tested against Zenoss 3.2 but people were already using the alpha version of 4. Unfortunately neither the Beta status of Rhybudd nor the Alpha status of Zenoss 4 prevented people from leaving negative feedback if they hit an issue.

People want to be notified about their Events not spammed about problems. Until the latest version the app would spawn a notification for every new event that was received during the poll cycle. In a cascade failure scenario that could result in hundreds of individual notifications crushing the Android notification queue, status bar, sound subsystem and everything else in between. As seen above the new version creates a single notification with extracts from the most important alerts and a count to indicate how many other events there are.

Looking Forward

Rhybudd has met its initial goal of providing myself and the rest of our Operations team with a way of pulling alerts from Zenoss on our schedule and empowering us to acknowledge, respond and escalate to them as required but the challenge now is to ensure that the other people who have the app installed can do what they want to be able to do with it too.

There are still a few rough edges on the app that I plan to smooth out but what it really needs is some direction from the community as to which features you would find most beneficial.

Feel free to email me Gareth@NetworksAreMadeOfString.co.uk or send me a tweet; @NetworkString to let me know any feedback you may have.

The app is free, doesn’t have ads, is open source and available on Google Play now.

[1] A basic proof of concept version of which can be found here; https://github.com/NetworksAreMadeOfString/Graphite-to-Zenoss

The State of Open Source Cloud Adoption (Survey)

community@zenoss.org — Wed, 12 Sep 2012 21:26:35 GMT

Since 2006, we’ve been conducting an annual survey on the State of Open Source Source Cloud Adoption.

Recently, we sent out an email to the community about taking this short survey. In case you missed it, I would like to invite you to take the survey on the State of Open Source Cloud Adoption in your enterprise. This survey is, of course, completely anonymous, and shouldn’t take more than 5 minutes.

At the end of the survey, you’ll have the chance to opt-in to receive the results after the survey is complete and will be entered to win a Google Nexus 7.

Thanks in advance for your participation!

Click Here to Take the Survey!

Image Credit: NetworkWorld

Zenoss Core 4 Service Pack 1

community@zenoss.org — Tue, 28 Aug 2012 16:43:56 GMT

Hi Everyone,

One of the benefits of having Open Source software is that we can be much more responsive to the needs of our users. And that's what we're doing by making Zenoss Core 4 Service Pack 1 available.

This Service Pack has been a combined effort of our Open Source community and Zenoss developers. A big thanks to Ryan Matte, Egor Puzanov and Zenoss Developers Tim Sanders, Evan Powell, Joseph Hanson and Jason Peacock for their efforts in making Zenoss faster and more robust. The Service Pack process itself is community-run and maintained, and open to your participation.

In a few days, I will be rolling Service Pack 1 into the auto-deploy script so that deploying a fully-patched version of Zenoss will become a completely automatic process. We want everyone to have the best possible experience with Zenoss, and this is a great way to make that happen.

We will continue to use Service Packs to roll out timely fixes to notable bugs found by our Open Source community.

We know that many of you use Zenoss Core for your own business and consultancies, and we see this as a way of partnering with our community to deliver the timely updates you need to make your Zenoss Core 4 deployments a success. In addition, we are planning an additional official maintenance release before the end of the year, which will result in freshly-built RPMs with many more improvements included.

Happy monitoring, and please be sure to check out Zenoss Core 4 Service Pack 1 and post your feedback below.

As this is a community-run effort, we are open to addressing issues you encounter in your deployments of Zenoss Core 4, so be sure to file bugs on jira.zenoss.com for issues you encounter, and chat with us here (or on #zenoss on irc.freenode.net) to let us know what you need from us to make your Core 4 deployments a success!

Best Regards,

Daniel Robbins

And the winners are...

community@zenoss.org — Fri, 24 Aug 2012 18:58:22 GMT

With Zenoss Core 4 now released, the Zenoss BugHunt has been completed, and the results have been tallied! The winners of our BugHunt are:

Andrea Consadori, with 247 points, including 23 P1 bugs
Yuliyan Lesev, 160 points (10 P1 bugs)
Jane Curry, 93 points (5 P1 bugs)

A big thanks to Andrea, Yuliyan and Jane, and all our participants for helping us make the BugHunt and Core 4 a success! Complete results can be viewed here. We will have follow-up posts in the next week or so on how BugHunt participants can claim their Core 4 t-shirt or sticker set

(Official contest legal rules)

CORE 4 + WIKI

community@zenoss.org — Tue, 07 Aug 2012 03:04:22 GMT

Well, it's here, folks -- Zenoss Core 4, technically 4.2.0, is now available for download!

Thanks to everyone who contributed to our Bug Hunt and helped to make not only the Bug Hunt a success, but Zenoss Core 4 significantly more robust and refined than would have been possible otherwise.

The Core 4 release marks a milestone not just in product development, but in our community efforts, as our new Mediawiki-powered wiki is now live, and is the new central location for all community online documentation resources. Everything Zenoss - from installation, configuration, operation, troubleshooting, scaling out, ZenPack development -- can be documented here, and you are encouraged to add content. The wiki is a place where the community and Zenoss staff and work together to build a world-class technical resource, all out in the open.

We have made a number of improvements to ease of installation. First, the community and Zenoss-developed auto-deploy script is now available for Core 4. In addition, the install process in our Installation Guide has been significantly improved, thanks to your Bug Hunt bugs. All this means a much improved initial Zenoss experience for new and experienced Zenoss users alike.

Thanks again for making the Zenoss Core 4.2 release a success, and stay tuned for Bug Hunt results and other developments related to Core 4

Core 4: Release Candidate Available

community@zenoss.org — Thu, 26 Jul 2012 22:07:43 GMT

Hi everyone, just wanted to post a quick note that Zenoss Core 4 is now, as of build 4.2.0-1583, a release candidate. I've tweaked my Core 4 beta auto-deploy script to now install the release candidate. Please test - and continue contributing to the Bug Hunt -- we're going until the release

Core 4: Release Candidate BugHunt Round-Up

community@zenoss.org — Fri, 13 Jul 2012 17:28:30 GMT

Within Zenoss, we are currently working on getting a Zenoss Core 4 release candidate ready for you. We're close.

As for the BugHunt, we're going to run the Core 4 BugHunt until the official Zenoss Core 4 GA (General Availability) release is available. So we're in the final stretches, but there is still time to make a difference!

Now, let's take a look at all of the surprising changes to our Leaderboard. There has been a tremendous amount of activity, with several new contenders, and Yuliyan Lesev moving from 4th place into 2nd place, and is now in contention for winning one of our top prizes!

Andrea Consadori, 220 points (21 P1 bugs)
Yuliyan Lesev, 112 points (7 P1 bugs)
Ricardo Sff, 86 points (7 P1 bugs)
Jane Curry, 81 points (4 P1 bugs)
Jo Rhett, 72 points (8 P1 bugs)
David Petzel, 36 points (4 P1 bugs)
Jamie Shardlow, 30 points

In addition, we had significant bug reports from Bryan Irvine, Lerry Weidig, Andrew Kirch, Nils Valentin, Robert Martin, Peter Moore and James E. Flemer. A lot of people really focused on reviewing our documentation, which resulted in a lot of improvements, as well as a lot of BugHunt points being awarded. (Note - we have an updated v07 of the install guide available for review.)

Amazing work, everyone. As you can see, the Leaderboard is still very fluid and we will see what happens as we enter the final stretches. It's great to see everyone getting involved in testing, reporting bugs on JIRA, and working hard to make Zenoss Core 4 the best it can be.

Be sure to stay tuned for the release candidate -- there are still more BugHunt points to be awarded by finding release candidate bugs!

(Official contest legal rules)

Core 4: Operation Documentation

community@zenoss.org — Wed, 11 Jul 2012 16:16:34 GMT

As we get closer to having an official Zenoss Core 4 release candidate, it's time to get our installation documentation in order. We've incorporated many of your suggestions and improvements into our new install documentation. There have been fairly major changes, including the use of yum for all packaging operations, and other significant tweaks. Please give it a read and test it out. If you find bugs or areas of improvement, file a bug at jira.zenoss.com -- you'll need to create an account first -- and mark it as a P1 or P2 bug so it gets reviewed quickly.

Remember that documentation bugs will receive credit in our BugHunt -- so be sure to tag your bugs with the "bughunt" tag. See my previous post if you want more info on our official Core 4 BugHunt. We take our documentation seriously and this also means that you get lots of BugHunt points for any documentation bugs

We also realize that the install process is fairly complex, and we plan to keep the Zenoss Core 4 auto-deploy script around as a community-supported option for getting Zenoss Core up and running with a minimum of fuss. This script does everything for you -- just run it on RHEL/CentOS 6 64-bit and it will download everything you need. Then just head to http://127.0.0.1:8080 to complete setup.

Bug Hunt - The Semi-Finals!

community@zenoss.org — Tue, 19 Jun 2012 16:57:20 GMT

Giving away three MacBook Pro 15" systems with Retina displays... Oh boy... what were we thinking?!

The Core 4 Bug Hunt is getting down to the wire -- The official Core 4 release will be here in a matter of weeks!

We are giving away amazing prizes to the top three bug finders. The top three bug hunters can choose from:

MacBook Air (11" or 13")
MacBook Pro (13" or 15") - And lucky you, Apple just released their 15" MacBook Pro with Retina Display
IMac (21" or 27")
all-expenses-paid trip to Austin, Texas for a Zenoss Mastery Class

And EVERYONE who submits a qualified bug will receive a limited edition t-shirt.

If you want to try out Zenoss Core 4 Beta, the fastest way is to use my new auto-deploy script (this script has been updated since my last blog post.) Download and run this script on a CentOS or RHEL 6 64-bit installation, and it will download and install everything for you, and then you simply go to localhost:8080 to complete Zenoss setup.

Then file bugs at jira.zenoss.com. For any relatively serious bug, give it a priority of 2 or above. This will ensure that it will get reviewed by our defect review in a timely manner. Our developers will review and re-prioritize the bug if necessary and then add it to our backlog, which is our bug queue. When this is done, it is eligible for points. 8 points for a P1, 4 for a P2, 2 for a P3 and 1 for a P4. I will enter a bug comment stating that points have been awarded when I do my next tally. Please see this blog post for detailed instructions on how to use JIRA and submit a bug.

And now, our current Leaderboard:

Andrea Consadori, 190 points (19 P1 bugs)
Ricardo Sff, 86 points (7 P1 bugs)
Jane Curry, 81 points (4 P1 bugs)
Yuliyan Lesev, 29 points (1 P1 bug)

The big news for the semi-finals is that Yuliyan Lesev has moved into striking distance, and can potentially move to a top 3 position before Core 4 is released. Yes, there is still time for you to do some serious bug hunting and move in to the top three position.

Here are my tips for those who want to win: first, when you file a bug, be sure to add the tag "bughunt" to it. Also, do not under-prioritize your bug. Keep it at priority of 2 or higher so it gets reviewed by defect review.

Next, be sure not to just rely on my auto-deploy script. Try installing it manually (download from here) on RHEL or CentOS 5 and 6, 64-bit, and look for issues in our PDF documentation. Any install documentation problems are considered serious bugs and will in turn likely get you serious points. After you get the beta up and running, cosmetic issues you find will get you minor points, but actual problems with the functionality or reliability of the software will get you more points, because the issue will get a higher priority.

Happy Hunting!

(Official Contest Legal Rules)

SouthEast LinuxFest 2012

community@zenoss.org — Mon, 04 Jun 2012 20:22:22 GMT

Hi All, just a quick note letting you know that we will be at SouthEast LinuxFest 2012, in Charlotte, NC, happening at the end of this week, Friday June 8th through Sunday the 10th.

Look for me (Daniel Robbins) and Rob Booth at the Zenoss booth, and we can chat about Zenoss and Open Source, give you a demo of what we're working on, and chat about community efforts at Zenoss. Hope to see you there!

Technical Deep Dives: Events!

community@zenoss.org — Tue, 29 May 2012 22:03:25 GMT

A couple of weeks ago, we started a new live webcast series called 'Zenoss Technical Deep Dives' to help you take advantage of all the power inherent in Zenoss.

Our first Deep Dive was hosted by Chet Luther, and explained the inner workings of event transforms and impact policies. If you have not checked out this webcast, I strongly encourage you to do so -- it is excellent technical material.

And our next Deep Dive is coming up tomorrow, covering automation of events via advanced triggers and notification -- You can sign up for the live webcast (or if you missed it, you can use the same link to access an archived video of the event.) Enjoy!

Attention All Zenoss Bloggers...

community@zenoss.org — Fri, 25 May 2012 16:14:13 GMT

Due to blog spam issues, I needed to remove "create blog" privileges for all registered users.

This means that I will need to re-approve Zenoss community bloggers on an individual basis.

If you have a blog on community.zenoss.com that you maintain, or if you would like to start a new personal blog here, just send me a private message and I will re-grant/grant you permissions to get back to blogging.

This new approach should prevent new spam blogs from popping up here, which I hope everyone will enjoy

Best Regards,

Daniel Robbins

Zenoss Core 4 Beta Released

community@zenoss.org — Fri, 18 May 2012 01:07:07 GMT

Everyone, the first Zenoss Core 4 Beta is now officially available!

Let's go over what you need to know about this new release.

We will continue generating new builds of the Core 4 Beta every 3-4 days until Core goes GA. The beta is feature complete, so each subsequent build will see a rapid decrease in the number of bugs over the next several weeks as we are in the hardening phase of the release cycle.

Now is the ideal time to help us find and smash bugs -- and we are giving away some amazing prizes (MacBook Pro, anyone? Not just one, but *3* people will get the top prize) for those who participate in our bughunt! There is still plenty of time (and bugs!) for people to start participating in the bughunt and win!

OK, now for the brand new stuff:

New jobs framework (using "Celery") with associated UI improvements. You will now see job status information in the bottom-right of the UI when things like modeling a device have been scheduled, or completed. This lets you know what Zenoss is up to and is a major usability improvement. This also allows us to dramatically improve the performance of Zenoss (and UI responsiveness in particular) over time, since we can move more compute-intensive operations into the new jobs framework rather than synchronously waiting for them to complete.

ExtJS 4.1 - Uniform grid controls, enhanced browser support and better performance across the board. For now, a handful of ZenPacks may not display some custom UI elements (like custom columns) correctly. We are currently in the process of fixing these ZenPacks. I will explain how to do this for any of your own ZenPacks that have custom JavaScript in a future blog post so you'll have all the info you need to get them ready for the official Core 4.2 release.

Commercial Windows ZenPacks have now been open sourced! This means Core users will be able to leverage enhanced modeling and monitoring functionality covering the following MSFT services and applications: Active Directory, MSMQ, Exchange, MS SQL and IIS.

In addition to these changes, the beta includes all of the major updates from Core 4 Alpha: RabbitMQ based on the open AMQP messaging framework supporting a new event management system that can handle over 100 million events per day, improved JSON API support, RelStorage which improves scale out and performance of ZODB access, zenperfsnmp scalability improvements, REST eventing API, SNMPv3 trap support, IPv6 monitoring support, collector daemon improvements, and enhanced zenping event suppression.

To install, first grab the installation guide from the download link, and then grab the most recent beta build of the VMWare virtual machine or grab RPMs for Red Hat Enterprise Linux/CentOS 5 or 6 (yes, we are supporting RHEL 6 now,) 64-bit. If using the RPMs, you'll want to grab the zenoss RPM as well as the zenoss-core-zenpacks RPM. Note that we are placing builds in their own "builds" subdirectory now, so you can easily find the docs in the main zenoss-beta directory.

I'll be posting info on an updated auto-deploy script in the next few days to help you to quickly deploy the latest Beta build and give it a spin. Enjoy!