For the rebooting box (DR node), it was running ak-2009.09.01.3.0 when we started working on it, and during the rebooting issue ak-2009.09.01.4.0 was released, and we upgraded to it. It didn’t coincide with the reboots stopping.

The prod boxes were running ak-2009.09.01.3.0, and still are – we’re basically not touching them until we get somewhere on the cases.

Thanks!

I am sympathetic to your plight, but you are suffering from a bad case of hindsight bias: there is no reason to believe that a disk failure would induce system failure, and trust me that there is nothing in the logs that they were missing. The missing piece of data — and I feel I have said this several times over now — was the console log that you didn’t collect (not your fault, of course — you weren’t told to collect it). Yes, “modern computers” need to have their console logged; how else is one supposed to debug failures in which no operating system dump is taken? (And yes, such failures exist — viz. yours.) The best guess at the moment (and it’s always going to remain a guess for the moment) is that the drive failure was inducing an HBA logic failure (that is, a failure in the HBA firmware itself), and that the HBA failure was in turn inducing both an operating system panic (which is — or was — our defined failure mode in such a case) and then (we hypothesize) a dump abort. (On the 7210 — unlike the 7310 and 7410 — we rely on the same HBA for both system disks and data disks.) We’re guessing here, but your drive is making its way back to our team to see if we can reproduce this in house. One final note: because these HBA logic failures have been a real difficulty for us, we have added logic in our Q1 release (due out in the next few weeks) that will reset the IOC on failure. So when we get your drive, we may well see the system recover now in a way it wouldn’t previously; we’ll let you know.

First, sorry to hear that your experience has been rocky. And yes, all of the teams on the project — support, development, test — have been enormously overtaxed. That, I’m afraid, is the curse of hypergrowth, which is exactly what we have experienced over the past year. And we have experienced that hypergrowth because we were largely right about one critical point in the enterprise storage industry: customers are sick of paying high rents to a couple of companies. What we attempted to do to address this was very ambitious: build enterprise-grade storage out of commodity parts. Fourteen months later, I think we’re finally approaching that goal — but as you have seen, it’s been a very rocky path (much rockier than we naively thought it would be). All of which isn’t meant to excuse any problems you’ve had with the product, but rather to give you an honest assessment of how we got here.

Now, that said: I am quite troubled by the corruption that you believe that you are seeing. Can you please send me a case number (my first name dot my last name at sun dot com) so I can unravel what happened here? (Jim can attest that my offer is in earnest.)

Finally, there actually is a mechanism to provide holistic feedback about the product, but you could be forgiven for not noticing it: in the BUI, there is a feedback link at the bottom of every page. Mail sent via that link goes straight to the development team, and no one else — and we have responded to everyone who has gotten in touch with us that way. (And we’ve gotten some very useful feedback that way.) We provided that exactly so people like you could unload: we’re technical and we’ll listen to you — so please don’t hesitate to send us that A4 worth of feedback, with our thanks in advance for doing us the service.

7901 Tue Jan 19 20:29:56 2010 Audit Log minor
root : Open Session : object = /session/type : value = shell : success
7900 Tue Jan 19 15:32:52 2010 System Log critical
upgrade to version unknown failed
7899 Tue Jan 19 15:32:46 2010 IPMI Log critical
ID = 15c : pre-init timestamp : System ACPI Power State : sys.acpi : S5/ G2: soft-off
7898 Tue Jan 19 15:32:41 2010 Audit Log minor
KCS Command : Set ACPI Power State : system power state = no change : device power state = no change : success
7897 Tue Jan 19 15:32:41 2010 Audit Log minor
KCS Command : Chassis Control : action = power down : success
7896 Tue Jan 19 15:04:52 2010 Audit Log minor
KCS Command : Set SEL Time : time value = 0×4B55CA14 : success

The item at 15:04 was the last “normal” message (seems those occur about once per hour?); the next few messages almost make it sound like the system believes it was powered down for an update which failed? The shell session is when I logged in to boot it back up.

Other machine — I asked if it’s possible that it’s a bad HBA or a midplane issue (as support had previously suspected but never replaced), but the senior support reps on the phones indicated that it was definitively a drive, and not in another part of the system. No console on the system – not used to needing console logging on modern computers. ;)

One thing worth noting – I had no idea that the ILOM logs wouldn’t be bundled with the support pack (I’d assume you have a way to retrieve them from within the OS), so I’ve never really looked at them beyond a glancing view, and only on the one that was rebooting itself daily. One interesting thing now that I am looking, is that the system that was rebooting has -far- more events in the SP log than the other two. It’s logging things similar to the following on a seemingly regular basis – appears to be 15 minutes after the hour, ever hour on a quick lookover:

27537 Thu Feb 25 17:15:38 2010 Audit Log minor
KCS Command : OEM Set LED Mode : device address = 0×18 : LED = 0×3 : controller’s address = 0×20 : HW info = 0×3 : mode = 0×0 : force = false : role = 0×41 : success
27536 Thu Feb 25 17:15:38 2010 Audit Log minor
KCS Command : OEM Set LED Mode : device address = 0×18 : LED = 0×5 : controller’s address = 0×20 : HW info = 0×5 : mode = 0×0 : force = false : role = 0×41 : success
27535 Thu Feb 25 17:15:38 2010 Audit Log minor
KCS Command : OEM Set LED Mode : device address = 0×2C : LED = 0×2 : controller’s address = 0×20 : HW info = 0×2 : mode = 0×0 : force = false : role = 0×41 : success

I see events like this on the other two systems, but extremely infrequently (ie – days between them.)

Regarding my comment on log analysis – that question came up because it was taking the engineers so long to get through the support packs.. I asked if the support reps had any tools to strip out extraneous information from the logs for them, and just leave the “unusual” stuff – the answer from the support manager was no, which is what surprised me. There’s a /ton/ of data there, and having to strip out the important parts by hand is time consuming! As I mentioned, the drive errors were in each log that was sent, but were missed by the engineers reviewing them – a proper log analysis program would certainly help with that.

Oh, you won’t bore me at all. ;) I’m sorry to hear you are having issues too – I was really hoping we were the exception to the rule. I’m also hoping that some people who have had great success will post.

I’m also finding that it seems the support techs are over-busy — seems like “squeaky wheel gets the grease” in my case as I’m now receiving quite prompt attention. I’m bummed to hear that EMEA doesn’t send you directly to US-based techs when needed.. it makes sense to have local high-level support, but send things back to the people who are most familiar in extreme cases.

I’m also feeling like the product is still “beta”, but it’s not being sold that way. ;( As far as not getting influence on the dev team, have you posted your issues online somewhere? It seemed to help me. ;)

ZFS corruption – that is extremely frightening. I’d love to hear details on what corruption you’re seeing and exactly how you trigger it. As far as the logging aspect goes, that is *really* frightening – and unfortunately on par with what I’m finding.

Failover – the issue of not failing over because of service-level issues is something I’ve found to be very common on active/passive configurations like this.. it’s a hard thing to get right, but I’d hope that would be something heavily tested and properly engineered before public release.

My title is meant to be tongue-in-cheek.. obviously the intentions were not to build a product that is disappointing, but that sure does seem to be the way that it’s ended up.

Sun support has been absolutely abysmal because 1) the first two levels of support know nothing about the product (not necessarily their fault) but consequently they give incorrect diagnosis, bad advice, and send out engineers to fix things that aren’t broken because they don’t understand what has gone wrong; 2) there’s only 3 (three) people in back line support for the whole of Europe, Middle-East and Africa (yes, just three!). They are so busy that there’s no chance they will get time to work on your problem. And they are being unfairly given work that would be better done by the development team. Something that the developers could fix in an hour will take two weeks to investigate by someone not familiar with the source code, and as you can see they simply don’t have that time to spend on each problem.

The product itself has been almost as abysmal, and I am sad to say this because I’ve been a fan of Sun technology. Not that the underlying components are bad: Solaris is great and ZFS is getting there. But the appliance software has so many bugs, even 14 months after release, that it’s still only beta quality. Yet there’s been no provision within Sun for customers to feed back a holistic view of their system and the support structure is totally inadequate for a new class of product such as the 7000 series. We’ve been so frustrated it’s unbelievable, if only we could talk to someone technical that was willing to listen to us and had the ability to influence the development team. Our list of problems is so large that just the titles fill an A4 page; most of which are serious or very serious. People like Bryan prove that the engineers do care, but …

For example we’ve had assurances that Sun will debug the ZFS corruption that occurs when a disk is replaced during a resilver but, 10 months later, the problem still exists and is easily repeatable. Would you trust any data to a system like that, or to a company which on the surface doesn’t appear to take it seriously?

Not only that but when you suffer corruption the system almost completely fails to inform you. If you are looking hard you will find a one-line message in the BUI but there’s no alert, no SNMP, no email, nothing, not even a permanent record in a log file of which what’s been corrupted. And it gets worse, the system then forgets that any corruption has happened at all, leaving you with no message in the BUI, just a bunch of files which appear to be fine but when you access them you get corrupt data. In Sun terminology that’s both data “corruption” and data “loss”.

Another example of a problem: the Q3 update provided the missing functionality to backup and restore the appliance config, yet when you perform a restore it breaks the box so badly you have to boot into a previous firmware revision.

Yet another example: services can fail (including the appliance kit daemon itself) and the head won’t fail-over to the other head (if you have a clustered pair of heads).

I could go on for hours about other serious problems but don’t want to bore you ;-)

I was going to be charitable and say that your title is misleading (“designed to disappoint”), because on the whole the collection of technology inside the appliance is really good. Yet there are definitely aspects where the design simply hasn’t been thought through properly (like the clustering and the reporting of corruption) so you’re right.

As for the other machine: you may indeed have a bad HBA — which would necessitate replacing the HBA, not the machine. But we would need a console log to know what happened, and given that we don’t seem to have that, it’s going to be hard to know definitively. I’m not sure what you mean about “no system in place to automatically identify anomalies”; I went through your logs and there’s very little anomalous about them — that’s part of what makes this mysterious. The log that we need — that of the SP console at the time of machine reset — is not something that the appliance can gather for itself…

Regarding the system you mentioned – the escalation team also said that it appeared to be a SP-level problem, but not that it was an explicit poweroff command – thanks for that info! They mentioned that they’d be looking at SP logs, but didn’t ask me to provide them (I thought they already had them as part of the system dump?), but I’d be happy to grab those logs for you and anyone else that is interested in looking at it. Want me to fire them to the email address you’ve posted from?

It’s not clear (at all) that you need new hardware here — we need to figure out why the machine is resetting, not throw hardware at it. As for why the two units failed simultaneously, the only thing I can tell you is that I don’t know. You said that one of the nodes “turned itself off”; assuming that I’m looking at the right data (MD5 hash of the hostname is 3df2a064148a0dc90237d936ec52c8b7) on the right day and time (1/19, 15:32 GMT), it appears that this was the result of an explicit power off. (Or rather, it is indistinguishable from an explicit power off.) So at this point, we would need to see the entire audit log for the SP. Unfortunately, there is not a way to get that via IPMI, so you would need to log into the SP, “cd /SP/logs/event/list” and then type “show”. This should at least tell us what was going on from the SP’s perspective, and why/how it decided to power off the appliance.

And while resource forks have been factored out of our main workflow, knowing that they can be handled well if we need them is very comforting.

So while it is a poor excuse, to some degree, it’s momentum. We’re in a bit of a holding pattern right now trying to decide what we want to do, depending on what Sun says on this issue.

We’d also feel a lot more comfortable if we could just know precisely what is causing this. While I suspect it’s this interaction, and heck, there’s a chance it’s not Sun’s fault at all (perhaps the problem is in the Snow Leopard client), this is the kind of thing you really want to know for sure if you are going to trust it with your data, ya know?

If your clients are OS X, any reason you can’t use NFS? We have a team using an OpenSolaris box as a bulk storage server for their HD video editing needs on OS X via NFS, and it’s worked out quite well.

I have no reason to think that our problems are related to ZFS. I realize now that my comments probably implied that, and I apologize.

Our current issue seems likely to revolve around the CIFS server, although we aren’t 100% its not an interaction between the client implementation (OS X 10.6) and the Sun CIFS server implementation. We aren’t seeing it from any other OS, but we also aren’t seeing this when using a 10.6 client to connect to a Windows based or Linux based server.

We found this because we deal with a lot of moving around of filesets that range from ~750MB to 5GB, so we do a lot of automated checksums, and started to see repeatable failures.

Brian, I’ll email you about this. The very fact that you offered is much appreciated.