c0t0d0s0.org - Technology

A really long rant ...

nospam@example.com (Joerg Moellenkamp) — Thu, 02 Sep 2010 21:18:00 +0200

I sat a on my fingers for all the weekend but now i can't take all this comments in the blogs and the mailing lists any longer. Ten years ago, i would have gone ballistic long ago... but the fuse got longer of the past few years. However i'm still able to detonate, however despite the undirected explosion 10 years ago, it's more directed today.

And this article is such a directed explosion. I worked a while on this article and this was one of the reasons why it was relatively silent on this blog the last few days. At first i didn't wanted to publish it, but some events of today led me to think otherwise. I don't know if this is a wise move, because the torch of this rant will burn down some beards. On the other side, i think the article is worth the publication, but you have to judge about that.

However i try to prevent me from exploding thus i used this weekend to play with my new toy (i purchased an iPad last week and it's really fscking cool, i just thought "Like Enterprise ... i just wouldn't gave them away like they do in TNG" ) and worked a lot on the planing on the modifications on my newly purchased building.

However the mail by Garrett d'Amore asking about the removal of SVM was a drop. It was one of those drops, that led to a severe spill-over. So this blog article will be a long rant. If you don't like rants, just skip this.

And keep in mind, it's a rant ... it's not meant to be fair or objective ... think of it as a way to write down my frustration about a lot of events in the last few weeks and especially in the last few days.
Continue reading "A really long rant ..."

Statistics

nospam@example.com (Joerg Moellenkamp) — Fri, 27 Aug 2010 09:17:44 +0200

At the moment you read a lot about this X-Force report and Sun is said to keep more vulnerabilities unpatched. But More interesting than the number of unpatched number of patches is the number of "Percentage of Critical and High 2010 H1 Disclosures with no patch" on page 20 in this report.

1. Google: 33%
2. IBM: 29% (the owner of X-Force)
3. Oracle: 22%
4. Linux: 20%
5. Microsoft: 11%
6. Novell: 10%
7. Sun: 8%

Old code set free - again

nospam@example.com (Joerg Moellenkamp) — Fri, 27 Aug 2010 06:20:06 +0200

Some code is so old that it predates the "invention" ( ) of free software by the FSF. Such an example is the RPC code provided by Sun to the world under a permissive license in 1984 and it was use in many implementations like the one used in Linux to provide NFS services. This was possible due to the licensing Sun choose at that time:

Sun RPC is a product of Sun Microsystems, Inc. and is provided for
unrestricted use provided that this legend is included on all tape
media and as a part of the software program in whole or part. Users
may copy or modify Sun RPC without charge, but are not authorized
to license or distribute it to anyone else except as part of a product or
program developed by the user.

However it isn't technically free software, as it was freeded before free software was formally defined ... and out of some strange reasons, people found the license now free enough. In some Linux distributions this situation was considered as a serious bug (I would file a bug against something different in regard of this, but that's a different story ) However due to whatever reasons, this particular issue wasn't resolved for years ...

As Tom Callaway wrote in a recent blog entry, this situation has been resolved now:

So, we restarted the effort with Oracle, and on August 18, 2010, Wim Coekaerts, on behalf of Oracle America, gave permission for the remaining files that we knew about under the Sun RPC license (netkit-rusers, krb5, and glibc) to be relicensed under the 3 clause BSD license.

Meine Präsentation auf der Froscon 2010

nospam@example.com (Joerg Moellenkamp) — Mon, 23 Aug 2010 08:09:57 +0200

Digging into 795 rperf numbers

nospam@example.com (Joerg Moellenkamp) — Wed, 18 Aug 2010 21:49:44 +0200

The new "IBM Power Systems Performance Report POWER7, POWER6 and POWER5 results" holds an interesting piece of information. A reoccuring question be colleagues and befriended admins is the impact of LPARs to the performance. It looks like IBM needs the LPARS to get some speed out of their larger systems.

Just a few examples: When using a 795 with 4.25 GHz and 64 cores a configuration with 4 LPARS a 16 cores yield a relative performance of 926.28. The same system with just 1 LPAR with 64 processors yield a relative performance of 777.09. So leaving the scaling to the OS instead of dividing it into 4 small systems gives you just 83.89% of the performance. When using a 795 with 4.25 GHz and 64 cores a configuration with 8 LPARs with 16 cores each yields a rperf value of 1852,56. With 2 LPAR with 64 cores each you get 1554,18. Interestingly is 83,89% again.

At first i thought "16 cores are easily fitting on a processor book (with 4 procs each). A 64 cores LPAR has to use two processor books. So when you use a configuration larger than a processor book you will leave 16,11% on the way". But doing the same calculation with some other data showed otherwise. But the move from 32 to 64 core lpars just reduce the performance by 5,7 percent respectively 5,4 %. 32 cores fit on a processor book, too. Thus the difference should be similar to the 64 to 16 cores situation.
So my interpretation is a little bit different: The scalability of AIX seems to have sweet spot between 16 and 32 cores. I thought a moment about an intra-book bottleneck, but the CPUs on the book are fully meshed (1 hop from each CPU to every other), so i don't think it's a problem.

When you look into this chart (please click into the image for a larger version), you may find some interesting points. The light blue line is a hypothetical perfectly scaling 4.0 GHz Power7 in a 795. The data is based on rPerf number of 103.41 for an 8 core system(source: Page 20 of the document). Please look at the right side of the chart at 256 cores. You end up at 81,9% of the hypothetical performance when you use 64 core LPARs and at 86% of the hypothetical performance when using 32 core LPARS (the difference is interestingly pretty much the same as computed before for 64 cores instead). At 64 cores your load is distributed at 8 processors, thus just 2 processor board. Still there is an serious impact of almost 20%. Will be interesting to further dig into this topic.

However it's important to know, that the operating system is limited to 64 core SMP no matter how many cores are in the system by the LPARs configuration. So this numbers doesn't factor in scalability challenges of AIX above 64 cores as the os has not to scale above this point while generating this rperf numbers. The numbers for the large core number configurations are not single OS image numbers. Further penalties for the OS scaling comes on top. Furthermore this benchmark is a pure CPU/memory benchmark. As IBM explained in their own description of the benchmark, there is no I/O and no networking involved.

That said, a number of really interesting data points are missing in the pdf:

The rperf number of a fully blown unpartitioned system
The rperf number on a fully blown system with just one LPAR in the size of the complete system
Somehow i have the impression, IBM is hiding something. When you look into the mentioned pdf there is SPEC number for an unpartitioned system, but no rperf for it. The existence of the SPEC numbers hints to the point, that they had indeed an OS that was able to scale to 256 cores. On the other side, there is no SPEC number for the partitioned systems, but the rperf numbers. Just call it a presentiment ...

Arms race

nospam@example.com (Joerg Moellenkamp) — Wed, 18 Aug 2010 10:16:49 +0200

Congratulations to IBM for the new lead in TPC-C. IBM has once again the lead in a benchmark that got meaningless a few years ago. Just before you ask, i wrote the same when Oracle was first. But somehow it looks like IBM wasn't able just to call it a day So they did a TPC-C benchmark run again and at the end they were able to yield 10,366,254 tpmC.

The result is indeed impressive. However there is are some key difference. The response times are vastly longer in the IBM result.

Response Times (in seconds)	90th %	Average	Maximum
New Order	2.1	1.137	24.041
Payment	2.1	1.138	21.293
Order-Status	2.06	1.095	20.169
Delivery (interactive)	1.64	0.749	17.953
Delivery (deferred)	0.95	0.42	2.48
Stock-Level	2.08	1.113	21.547
Menu	1.64	0.77	23.037

Now look at the response times at the Oracle result.

Response Time in Seconds	90 th %	Avg.	Max.
New-Order	0.170	0.168	5.885
Payment	0.160	0.156	5.758
Order-Status	0.150	0.150	5.433
Delivery (Interactive)	0.120	0.134	3.869
Delivery (Deferred)	0.040	0.021	2.839
Stock	0.210	0.182	4.796
Menu	0.120	0.136	4.474

With similar response times, the number of transaction is more in the range of 8,9 million as stated by the diagrams in the full disclosures.

This is the diagram from the IBM result:

Somewhere between 80% and 100% of the final result the reaction time explodes.

This is the matching result of the TPC-C result of Oracle:

There is no equivalent "explosion" in this result....

Furthermore i want to hint you on certain points in the configuration as stated by the full disclosure.

This configuration has 48*380 GB SAS cards with battery backed write cache, thus 17,8 GiB battery protected write cache in total.
The configuration used 3*224 SSDs summing up to 672 SSD. I would suspect that they use SSD with Sandforce 1500 (the same 177 GB like they used in the TPC-C that is documented at several places to be generated with Sandforce based SSD). This controller has an interesting capability. It's capable to do compress and some benchmarks have suggested, that the performance of this drive is quite different with compressible and less compressible data. It's would be an interesting point to research how compressible TPC-C data is.
The SSD use MLC. The Sandforce Controller have some special mode of operation to enable the use of MLC to reach better durability, but this mode is based on compression, too. Interesting questions are: Are the SSD really capable to hold 3 years of TPC-C load due to the usage of MLC and what would be the impact on durability of less compressible data.
The database is completely on the SSD. Just the database log is on disk. That's similar to the Oracle config
The configuration of the database on the system is ... well ... interesting. They use a partitioned database and all this partition are bound to certain resource sets. So essentially they splitted the system in several ones, to be exact ... into 32 partitions per systems bound to a resource set each. With this amount it's ensured that all requests are CPU local factoring out the interconnect.
The configuration doesn't provide any availability protection, but that's okay, because TPC-C doesn't mandate such. Just keep this in mind with the pricing and with the configuration. There are no mechanisms ensure availability and you can't transform it into an available configuration because the storage is direct attached to a single node (the storage is in the i/o drawers). The Oracle configuration is highly available by accident, as it uses shared storage and RAC

Conclusion

I think TPC-C is now really a corporate ego thing. We are in an arms race here. I'm really interested, how Oracle strikes back

A third kind of SSD drives

nospam@example.com (Joerg Moellenkamp) — Fri, 13 Aug 2010 21:41:14 +0200

Just found this article about a new SSD from Micron. This article reminded me of my thought that there should be a third category in SSD. In the harddisk market there are consumer drives, enterprise drives and near-line drives. A consumer drive is that stuff you include in your desktop that is designed to run for a limited number of hours a day and mostly sits idle in the computer casing. An enterprise drive is designed to to work 24/7 with being busy most of the time (80% for example). Nearline drives are the one that can run for 24/7 but they are just designed for 20% busy time. Of course you you could use any disk for each task, but it's a simple equation: You swap busy time for the lifetime of the drive. You can use your SATA home user drive for your database but expect to see your storage array very often.

That said, i think we need a similar distinction in the SSD area. At the moment there are just enthusiasts drives and enterprise drives. Out of a strange reason there are enthusiasts drives that got the enterprise moniker, but thats a different story. But there is something similar important to the busy time for SSD, even when they aren't susceptible to mechanic wear. It's something i would like to call "write time". It's specified as the amount of time the drive is writing data to the media. That's unimportant for rotating rust, but for Flash based solid state drive it's completely different.

Let's do some calculation: Microns new drive is said to deliver 275 Megabyte/s. 275 Megabyte 60 seconds 60 minutes 24 hours 365 days * 5 years equates to 40,3840095 Petabyte. However the endurance of the drive is specified to be 4.5 Petabyte in 5 years. Thus you could only write in roughly 1/10 of the time data onto the disk. It has an allowable write time of roughly 10%.

So that may not a problem for usage if normal filesystems are used, where read and write load occurs on the disk, but now think of the ZFS separated log ... this is a write-only device for almost all practical purposes. You never read from it in normal operation,so there are no reads to relief the drive from the write load.

So you should not size the amount of SSD disks not only for the expected write bandwidth, you should also take into consideration how this influences the lifetime of your drive and how long you expect the server to stay in operation. At 275 Megabytes per second you would brick a drive designed for 4.5 PB within five months. As well as you choose the right disk type by thinking about the busy time, you have to factor in the write time for your SSD.

Where should someone set the differences between the types? Enthusiast drives are easy to spot ... MLC drives and all drives without capacitors in case they have caches. Near-line and enterprise SSD? I would draw the line somewhere around 50%. When you can't get through the warranty time at max speed with 50% writes, it's a nearline drive. Above that it's an enterprise drive. But that's only my personal opinion and a first idea.

What's so special about the Parallel Sysplex?

nospam@example.com (Joerg Moellenkamp) — Mon, 26 Jul 2010 20:04:26 +0200

With this article i want to learn a thing or two and i want to start a discussion.

Whenever you express, that a mainframe hasn't no advantages over a high-end Unix system, the first comment or reply to a comment is "Oh, but IBM has Parallel Sysplex and delivers availability xyz can only dream of". Often the discussion ends there, because few people know, what this "Parallel Sysplex" is. I had to read into a lot of documentation to learn more about the Parallel Sysplex.

I've learned a few things in that: A sysplex doesn't seem to be much different from something like we simply call a cluster in Unix. A parallel sysplex is an extension of this and enables the system to act as one large system. However you can't consider it as a tool that is able to distribute arbitrary tasks on sub-process level onto several systems.

As far as i understand the concept of the Parallel Sysplex, it's not the way that any and every software is enabled to be a part of a parallel sysplex as it has to use certain facilities that make a parallel sysplex out of a sysplex. Your application have to use a thing called the coupling facility. And parallel sysplex exactly contributes this facility to the applications running on the mainframe.

Parallel Sysplex doesn't help you to make your zLinux more available. You have to work with the Linux cluster-frameworks to do so, albeit there seems to be some workarounds

I understand that the parallel sysplex makes DB2 on z/OS more scalable and available, but where is the difference to Oracle RAC on Solaris on a 2 big node cluster? I understand that a parallel sysplex makes CICS more available and scaleable, but where is that different for example from using Oracle Tuxedo in a highly available configuration with clustering support? I understand that that parallel sysplex can make storage more available, but how is that different from the Availability Suite with it's Remote mirror facility, from SamFS/QFS, from the global filesystem in Sun Cluster.

I know, all these things working differently, but that's not the point. I want to know, why some people think "Oracle can just dream of". What's so special about that ... the parallel sysplex is an ... well ... interesting architecture, however i fail to see any reason why it's special that everyone talking about mainframes is just pointing to it.

Perhaps it's just the way that mainframe people doesn't understand the open systems world to the same extend as open systems people understand the mainframe world: Barely to an extend at all. Additionally i have the impression that many services on a mainframe are so ubiquious in that world that they consider properties of this software components as properties of the mainframes. Another hypothesis is, that most mainframe guys doesn't know what's possible on Unix (real Unix, not the pengiun variant) and thus consider their technologies as superior. I talked to a mainframe guy on a conference some time ago and i had to explain that SunCluster has gone great lengths since Sun Cluster 2.

We talk about synchronous storage replication in Solaris for quite a time now (2000), service restarting isn't a problem since SMF. Instruction Retry on hardware basis is available since the introduction of the M-Series, the same for memory mirroring. Hot swap of I/O, memory and CPUS is possible since last century.

May be i just don't get get, but may you are able to help me: I want to postulate "There is nothing in a Parallel Sysplex that a equivalent configuation on the basis of an M9000 isn't capable to do". But i'm a open systems guy from the core, so i ask the mainframe aware people to explaon me, why Parallel Sysplex is considered as superior to Unix-HA clustering when used with applications developed for high availability. Please don't answer with "You have no idea, ParSys is so superior" ... i want some real reasons.

ZFS v15 in FreeBSD

nospam@example.com (Joerg Moellenkamp) — Wed, 14 Jul 2010 11:08:46 +0200

The ZFS in FreeBSD is now at version 15. In version 15 user/group space accounting (aka Quotas) was introduced into Solaris. However there are still missing features:

jmoekamp@hivemind:~# zpool upgrade -v This system is currently running ZFS pool version 22. The following versions are supported: VER DESCRIPTION --- -------------------------------------------------------- [...] 16 stmf property support 17 Triple-parity RAID-Z 18 Snapshot user holds 19 Log device removal 20 Compression using zle (zero-length encoding) 21 Deduplication 22 Received properties 23 slim ZIL

Pressure?

nospam@example.com (Joerg Moellenkamp) — Wed, 07 Jul 2010 10:17:43 +0200

According to the Coraid website, NetApp has send a "nice" letter to Coraid, a vendor of an ZFS based storage appliance. NetApp should stop to bully around small vendors, get back to innovation and sort that thing out with someone in the same or a larger weight class, for example with Oracle. I can just explain this behaviour by increasing pressure in the market by ZFS.

PS: At the end the lawsuit is still valid, as far as i know most of the relevant patents has been invalidated ....

Cable lock-(in)

nospam@example.com (Joerg Moellenkamp) — Fri, 02 Jul 2010 22:23:02 +0200

Okay ... i really thought that enforcing a certain brand of cables was a thing of the past. Apparently not as reported by @randybias on his Twitter timeline:

This is really a common problem as shown here or here. Especially the later one is showing the pain you could experience when you use those 10GbE direct attach cables . You have to think about this DAC like two hardwired transceivers). As both ends contain the transceiver logic it's easy to implement an ID rom in that cable and check it before bringing up the link.

I should note, that there is a standard for the interface that connects those DACs to the switches/NICs. It's called SFF 8431. But now lets look into reality: The Intel card of the user in the second links doesn't support the DAC cables from HP. HP seems to just support HP cables, not allowing anyone to use other cables by software. Switches from HP are equally rigid, they just support HP branded DACs and nothing else. Cisco Nexus switches seems to have at least a mechanism to allow unsupported DACs to work to get HP cables running . Randy wasn't able to use Arista DACs with HP Flex 10.

Welcome in the world of cable-locks, where the intersection of supported with one systems and supported with the other system may be empty I thought, standards were made for a reason.

The challenges of many cores

nospam@example.com (Joerg Moellenkamp) — Thu, 01 Jul 2010 12:07:12 +0200

There is an interesting article inIEEE Spectrum:"The Trouble With Multicore". David Patterson writes in this piece about the problem multitudes of cores are introducing to programming of such systems. Worth a read!

Trailer for "Java 4-ever"

nospam@example.com (Joerg Moellenkamp) — Sat, 26 Jun 2010 14:23:16 +0200

(click here to view)

Update: The same video at Youtube.

Kris erklaert die Welt - heute: Relationale Algebra

nospam@example.com (Joerg Moellenkamp) — Wed, 28 Apr 2010 12:25:28 +0200

Kris hat einen tollen Artikel über relationale Algebra geschrieben. Hochinteressant und eine eindeutige Leseempfehlung: Was bedeutet eigentlich 'Relationale Algebra'? Insbesondere gefällt mir:

Das ist genug Mathematik. Wir wollen Informatik machen.

Das bedeutet als erstes einmal: Weg mit den doofen Unendlichkeiten. Komplexe, reelle, rationale, ja sogar natürliche Zahlen - das ist alles unendliches, nicht anfaßbares Gekasper von irgendwelchen Mathespinnern. Wir wollen Werte, die man anfassen und mit dem Debugger im Speicher sehen kann. Informatik macht Diskrete Mathematik, alles schön endlich. Da kann man eine Funktion dann auch schon mal als Wertetabelle statt als Rechenvorschrift definieren.

Und so landen wir bei den Tabellen.

TPC-H@3TB on the M9000-32

nospam@example.com (Joerg Moellenkamp) — Mon, 12 Apr 2010 21:34:53 +0200

A while ago i assumed (and some Sun colleagues with me) that the M9000 could give the IBM p595 quite a run in TPC-H. At that time I couldn't substantiate it as it isn't allowed to compare TPC-H benchmarks at different scaling factors (e.g. TPC-H@1TB and TPC-H@3TB). Sun hadn't a 3 TB result, and IBM hadn't a 1 TB result. That changed: Today the TPC.org published the TPC-H@3TB result for the M9000-32.

On 04/10/10 a Sun M9000-32 was reported to yield 188,229 QphH@3000GB with 32 processors, 128 cores, resulting in $23.99/QphH@3000GB. Same amount of memory in both systems. Just as a comparision: The p595 fully blown processor-wise with 32 processors and 64 cores was able to yield 156,537 QphH@3000GB end of last year.

Some comments:

The Sun system is able to yield 20.246 percent more performance than the IBM system.
The result is the best non-clustered TPC-H@3TB result.
This benchmark uses the current top end machine of IBM in a fully blown config. The Sun machine has still some growth potential, as you you could still opt for the M9000-64 with 64 sockets.
While using the same number of sockets, the p595 needs less cores for the same job. So the core-based pricing with Oracle would take a larger dent out of the budget with the Sun system than with the IBM system, when using the same db. However the p6 core has a multiplier of 1, whereas the SPARC64 core has a multiplier of 0.75. So it's 64 oracle cores with IBM versus 96 Oracle cores with Sun. This somewhat kills the $/perf metric, despite the point that the hardware is significantly cheaper. Would like to see a socket based licensing in the future
The IBM result was done with Sybase IQ Single Application Server Edition v.15.1 ESD #1.2, a somewhat specialized database engine for reporting, analytics, and data warehousing. That's interesting as TPC-H is the decision support benchmark in the TPC benchmark. By the way: The limits of Sybase IQ SASE may explain why the data base license is really cheap compared to Oracle. The Sun result used Oracle Database 11g Release 2 Enterprise Edt. I would like to see a IBM result with a general purpose database, too.

Marriage is a graph

nospam@example.com (Joerg Moellenkamp) — Tue, 16 Mar 2010 20:34:26 +0100

Storing a marriage in a database can be really mind-bogling when you want to store every possible way to marry a significant other in a database: Gay marriage: the database engineering perspective. Excellent read.

AIX processor limit

nospam@example.com (Joerg Moellenkamp) — Thu, 25 Feb 2010 10:53:09 +0100

A colleague found an interesting piece of information on page 23 in the "Release Notes of AIX 6.1":

AIX Version 6.1 with Technology Level 6100-04 supports a maximum of 64 processor cores and a maximum of 128 logical processors. The logical processors correspond to the total number of Simultaneous Multithreading (SMT) threads. This means that the maximum supported configuration is 64-core POWER5 or POWER6 in SMT2 mode (2 threads per core).

Perhaps this is the reason why there are just a few published benchmark for a larger machine. Of course you could use LPARS, but then the SAP-SD benchmark would be a 3-tier result and not a 2-tier one fore example. And as far as others and i've read through all the service pack docs, this limit is still in AIX 6.1, albeit the newest AIX 6.1 TL4 SP3 is said to be able to capable to use up to 256 threads, but didn't found this info in the docs so far, the info is just in an IBM preso. The newest available release note at the IBM site is still stating the limit, but it's from November. To use more than that, the support doesn't go into 5.3 or 6.1, you will need AIX 7.1 for that ... will be interesting to see the ISV reaction on that. However, this looks like they have to do more in-depth work to scale up to 1024 procs to me ...

PS: The processor of a plain-standard Solaris 10? Just look here.

To dedup or not to dedup - that results in a lot of questions

nospam@example.com (Joerg Moellenkamp) — Mon, 08 Feb 2010 20:42:41 +0100

In some discussions in mail and in some community forums linking to the articles about deduplication and hashing there was a slight misunderstanding. I should explain some things.
Continue reading "To dedup or not to dedup - that results in a lot of questions"

Selfmade SSDs - or: the tale of thinking too complex ....

nospam@example.com (Joerg Moellenkamp) — Fri, 01 Jan 2010 12:29:43 +0100

Frequent readers of my blog know that i'm using an Notebook via iSCSI to emulate a SSD. At the moment my SSD is an old Acer Aspire notebook. It works reasonably well, it's much faster than a hard disk and i can test the behavior of applications with SSD without buying one.
Continue reading "Selfmade SSDs - or: the tale of thinking too complex ...."

Presentation at the Apache Hadoop Get Together Berlin

nospam@example.com (Joerg Moellenkamp) — Fri, 06 Nov 2009 22:09:09 +0100

On 16th December i will talk about Hadoop on Sun at the Apache Hadoop Get Together in Berlin. The talk will be divided in three parts:

Advantages of using Hadoop on CMT systems like the T5120
Using Hadoop in conjunction with Solaris Features like Container and Resource Management in non-dedicated architectures
Hadoop and Grid Engine

There will be short live demonstrations in the second and third part of the presentation.

My presentation will not be the only one. Richard Hutton from nugg.ad talks about data processing with hadoop in his presentation "Moving from five days to one hour." Nikolaus Pohle from Nurago talks about data analysis with Hadoop for online market research.

About that data loss at Microsoft/Danger

nospam@example.com (Joerg Moellenkamp) — Tue, 20 Oct 2009 15:48:00 +0200

Yesterday the usual news outlets speculated about the rumours circulating about the reasons for the data loss at Danger. For the uninitiated: Danger is the the company behind the Sidekick.

TGDdaily started by reporting the information provided by a reader with the article: "Oracle, Linux, Sun to blame for Sidekick Danger data loss":

TG Daily reader Tommy T informed us earlier today that: Microsoft was running on Sun Solaris/Linux/Oracle-based Danger servers at the Verizon Business Center in San Jose as part of a contractual obligation to T-Mobile.

That isn't really a surprise. Despite all the press fuzz, many many sites still deploy and use Solaris for their mission critical stuff, while using Linux for stuff that scales well on clusters.

Of course Microsoft was asked for the reasons and just reported

Sidekick runs on Dangers proprietary service that Microsoft inherited when it acquired Danger in 2008. The Danger service is built on a mix of Danger created technologies and 3rd party technologies.

I think that's a good answer, we don't blame anyone but we tell the world, the application wasn't developed at Microsoft and the infrastructure wasn't Microsoft as well. Especially as Microsoft doesn't need additional doubts about the feasibility of their operating and infrastructure environment after Microsoft loosed the .net implementation at the London Stock Exchange in favor of of a Solaris/Linux implementation. I don't expect Microsoft to shed more light into the situation.

And it isn't that way, that they are the only shop on the world using RAC on Sun. The Inquirer expresses this situation in a nice way - "Oracle and Sun fingered for Sidekick fiasco"

That would all be well and good but for the fact that the Sun-Oracle combo is not exactly rocket science and is popular among those who manage very large, mission critical corporate databases.

I think, the truth about this outage is much more complex, as the various media outlets state. IT systems are called systems for a reason. They are large complex interrelationships of components. There is a database, there is server, there is storage, there is a SAN, there is a network. And there are many devils of the detail in any system. Whoever says something different, simply lies.

But at the end it's pretty irrelevant. It's like with aircraft crashes. There isn't a single reason why an aircraft crash happens. When there is a major CNN moment at a service, there are several resons why it happened. It may start with a small hick-up and other effects leads to a disaster.

At the end it's a large heap of rumors surrounding this situation. Daniel Eran Dilger writes in "Microsofts Pink/Danger backup problem blamed on Roz Ho":

According to the source, the real problem was that a Microsoft manager directed the technicians performing scheduled maintenance to work without a safety net in order to save time and money. The insider reported:
In preparation for this [SAN] upgrade, they were performing a backup, but it was 2 days into a 6 day backup procedure (its a lot of data). Someone from Microsoft (Roz Ho) told them to stop the backup procedure and proceed with the upgrade after assurances from Hitachi that a backup wasnt necessary. This was done against the objections of Danger engineers.

As far as I'm understanding the rumor, they made a backup, deleted they only other backup in the course of this procedure, stopped the backup and did a firmware update. Sorry ... i don't even do an update of of notebook without starting a backup before doing so and my blog replicates it data every six hours to a system in my office at home.

A backup is a safety net. You don't do any task that could put your data at risk without having an backup. And you should better test the restore. Of course this takes time, at my blog a few minutes ... at Danger a few days. But you are sure that you don't appear at CNN. (okay, i would appear at CNN, but i would get some question in our coffee kitchen at work). Sh.t happens, customers understand this, but they don't accept that you don't value their data. Perhaps this is the real story of the this mishap.

In preparation for this [SAN] upgrade, they were performing a backup, but it was 2 days into a 6 day backup procedure (its a lot of data). Someone from Microsoft (Roz Ho) told them to stop the backup procedure and proceed with the upgrade after assurances from Hitachi that a backup wasnt necessary. This was done against the objections of Danger engineers.

I really can't imagine a single reason why a HDS/HP/EMC/Sun storage support engineer says "Hey. You don't need a backup. We will do the firmware upgrade without it". I just can imagine that the the engineer said: "Well ... i did several upgrades so far and there wasn't a problem that sent me to the tapes"

By the way: There is one thing, that keeps me really puzzled. It's a piece of information at TheRegister:

It involves 20 or so CentOS Linux servers [...]

Sorry, CentOS is a nice variant of Linux, but i don't really think that's a good idea to use a distribution of Linux with at least questionable commercial support (The "Commercial Support" menu item at centos.org delivers a page stating that there is no content) in a mission critical environment. I know that an enterprise Red Hat or Suse distribution/support agreement costs you some bucks, but in such an environment i would opt for the professionally supported variant. But again: This is a example of an attitude. And this attitude is not one of valuing the data of the users.

A hickup in a service has most often their reason in the realm of technology, but real disaster has more reasons, it has deeper reasons. Just 10% are technological reasons. Or to say it more precisely: A disaster can be root caused by a technical reason, but the step from a hickup to a disaster needs always reasons outside the realm of technology. And given the rumors surrounding the Danger outage are true, this outage is one of the best examples to show it.

Sun Storage F5100 officially announced

nospam@example.com (Joerg Moellenkamp) — Mon, 12 Oct 2009 08:48:30 +0200

Sun has now announced the F5100 officially. The specs are really impressive: The largest version with 80 FMods yields 1.6 Million read IOPS and 1.2 million write IOPS with 4k blocks. 12.8 GB/sec read and 9.7 GB/sec write. 1,920 GB capacity. It uses 386 watts while active. More specs are available at the Sun specs page for the Sun Storage F5100 Flash Array. Furthermore the documentation of this devices should reappear soon at docs.sun.com. Until that moment you will find some additional information about the device at Sun's website. This device isn't that expensive as you may think. The device starts at $45,995 (with 20 Flash Modules giving you 480 GB, 397 K IOPS read and 304 K IOPS write).

When you look in such a device you will see 80 densly packed FMods.

In the back of the device you will find the SAS connectors for the device. There are four domains with 4 SAS connectors each. For a maximum performance configuration you configure 5 FMods and a SAS connector into a zone.

Most important: The FMods doesn't have a battery on each module to protect the caches. This is done centrally by super capacitors in so called Energy Storage Modules. You plug them into the front of these devices.

Just a warning: Those ESM are quite potent super caps. Read the manual with care before handling those modules. You need a lot of power to protect all the caches and those ESM are capable to deliver this power.

Observations on memory reliability

nospam@example.com (Joerg Moellenkamp) — Sun, 11 Oct 2009 20:48:56 +0200

Robin Harris points to an interesting study about DRAM failures in his blog storagemojo (BTW: Robins blog is really a great read). He points to the paper "DRAM Errors in the Wild: A Large-Scale Field Study"( written by Bianca Schroeder (University of Toronto), Eduardo Pinheiro and Wolf-Dietrich Weber (both from Google)) in his article.

Some of the numbers are really terrifying: 4.15% unrecoverable errors for of of the platforms are much more then i had thought and i'm somewhat conservative in my thinking how far i trust hardware. Furthermore hard errors (as in "bit permanently flipped and put it to the trashbin") are vastly more common reasons for errors as most people think.

As a sidenote: After the discussion about DRAM prices in the M3000 i've got some flak because of memory prices at different quality and got many comments "there never failed a dimm at my home pc". But given the point that Google is said to use cheaper hardware and the amount of errors (especially the unrecoverable ones) there may be a point behind the fixation of Sun in regard of memory quality

Somewhat stable Solid State

nospam@example.com (Joerg Moellenkamp) — Fri, 09 Oct 2009 12:33:00 +0200

A reader think it makes no sense to use the STEC SSD, and we should switch to the Intel X25E drives. That sounds reasonable at first as the X25E is much cheaper as the STECs. But as usual the devil is in the detail. So why do many people still use some of the more expensive STECs? Do they have too much money? Are they morons? I will tell you a dirty little secret. No, not at all. When you take it really seriously the X25E aren't enterprise ready. At least in their default setting.
Continue reading "Somewhat stable Solid State "

Different CPU, same challenges

nospam@example.com (Joerg Moellenkamp) — Tue, 06 Oct 2009 11:57:54 +0200

Nice to see, that i'm not the only one who thinks, that IBM will run in the same challenges like Sun in regard of massive multicore processors. Great to see, when this position is somewhat confirmed by someone who isn't known as friendly to Sun and is said to be firmly on the blue side.

BlueToTheBone writes in his column "The Four Hundred" about Moore's Law and the Performance Wall:

Well, with the Power7 chips coming next year, IBM has to get the multithreading fixed in DB2 for i or get a whole lot of excuses ready for why customers buy more cores and threads, running at lower clock speeds, and don't see performance go up.

But he points to an even more interesting point, that isn't really a known territory to an open systems guy like me. Besides of this "bytecode-compiled is slower than iron-compiled" stuff (which isn't true since the invention of Just-in-time-compilers) he has a very valid point. Much of the software is really old, and it wasn't written for a environments with hundreds of cores. We learned the hard way, that there were a vast amount of pitfalls in the Open System world which thinks multithreaded/multiprocess for quite a time now. Now software developed in RPG and COBOL (and many code lines originate from a time when many of us weren't even a glint in the eyes of our parents) hits with Power7 on an environment that doesn't fulfill on the promise of ever-increasing single-thread performance. BlueToTheBone comes to a similar conclusion than the one i've made a while ago. Perhaps many applications stay on Power6 while newer developments can move to Power7:

. It might even mean putting off a move to Power7 iron and sticking with Power6 or Power6+ boxes as you dig through your code and see how parallelization can and cannot be used to make your applications run faster as well as offer more capacity to support more work.

Me too!

nospam@example.com (Joerg Moellenkamp) — Mon, 05 Oct 2009 15:19:17 +0200

Looks like IBM is somewhat concerned about the Exabyte V2 announcement and that announcement that is like to happen on October 14th. At least there is a loud "We have it, too - kind of" from Armonk. They tout their product IBM DB2 Pure Scale. Power machines with AIX, some infiniband gearand special edition of DB2.

Some educated guesses

Mr. BlueToTheBone is right. Details are sparse at the moment. But just to write down some points, you remember for having a deeper look at after an official announcement. Those points are purely speculative given that the information in the ElReg article are really sparse:

It will be interesting how this solution with a centralized lock management will compare with the distributed lock management in RAC. Perhaps this will be the scaleability stopper for the IBM solution. I think we will see a solution with fewer, but bigger nodes. Given that the job of lock management is consolidated on one node, i would say that the front end node will be a really big machine.
But have the same opinion like Chris Mellor (hell freezes over, i have the same opinion): IBM has a storage problem. There was the 1 Million IOPS Quicksilver marketing stunt. But that system were just 40 Fusion-IO cards in a lot of servers. So that doesn't look like something playing in the same ballpark like the Sun F5100
The QuickSilver flash was composed of one or two 250GB Fusion-io ioDrives with PCIe interfaces in 29 host servers connected by Fibre Channel to the 13 SVCs. There were 41 ioDrives in total.
Just for the uninitated: The SVC (Storage SAN Volume Controller) is a server as well. A full rack of hardware for 1 Million IOPS. But perhaps they just plug the Fusion-IO in the database servers and don't use SVC at all... who knows
The p6 pSeries systems doesn't have PCI Express 2.0. With the PCIe 1.0 variant you have 240 MBytes/s. Thus you have somewhat around 2 GByte/s on the x8 PCIe of a p520 for example. That isn't enough for QDR. Some systems like the 520 provide GX busses for better connectivity, but nevertheless there doesn't seem to be QDR cards supported in the systems. But those 520 are 4 RU systems, don't know if they use them for such a cluster. Thus i would expect to see DDR infiniband in those systems. I assume they will have half the bandwidth in the fabric
There isn't a system in the pSeries lineup that allows to put 12 disks on 2 rack units. That's the other side of the storage problem, albeit Chis Mellor at El Reg doesn't mention it. That's the point why the X4275 is such an interesting system. You can have 12 Disks and 2 sockets on 2 rack units. I assume the database servers have seperate storage arrays.

There are some other points, but those are much deeper in the realm of speculation, so i just wait for the announcement for further comments.

Frankenbase

Some people say, that Exabyte V2 in its current incarnation is just rushed together in the light of the upcoming merger. But the stuff that IBM told exclusively in the ears of Bluetothebone sounds like kludged together just to counter an competitors announcement. I'm sure that IBM will show of a strange configuration, calling it DB2 Pure Scale. But Frankenstein creation will look as a beauty beside this solution.

Guide to AES

nospam@example.com (Joerg Moellenkamp) — Sat, 26 Sep 2009 10:40:53 +0200

The cryptographic algorithm AES is everywhere. You find it in your router, you find in your OS. Your WiFi network use it. And probably the three-letter agency in your country,too . But how does it works? Jeff Moser wrote (well, it's more a comic) the "Stick Figure Guide to the Advanced Encryption Standard (AES)". A really great explanation of AES.

Fun with VoIP

nospam@example.com (Joerg Moellenkamp) — Wed, 23 Sep 2009 07:38:34 +0200

Asterisk is a cool piece of software. You can use it to do almost everything in the realm of telephony except of doodling on your notepad while listening to a boring telco. But i hadn't the idea to implement something like that: The Annoyatron Telemarketer Torture. I want this on my cellphone when someone calls me to sell me a private health insurance. Well ... as i switch into "maximum unfriendliness", they don't call me often ... looks like i have some negative mark in their data warehouse

The new TPC-H benchmark from IBM

nospam@example.com (Joerg Moellenkamp) — Sun, 20 Sep 2009 21:09:58 +0200

As my colleague Ingo said a while ago: "It doesn't matter how fast you stir your meal while cooking when you wait for your sweetheart to search the wine. You can't eat a minute earlier". And i found a another example for this: The newly published TPC-H 3 TB benchmark for the IBM p595, the biggest system in the Power series.

But i will go into the past before: In 2007 (in IT thats halfway to infinity) Sun benchmarked the E25K with 72 sockets at 1.8 GHz with two cores per socket and 256 GBytes of memory. This benchmark yielded 114,713.7 QphH.

2 years, 224 GBytes main memory and 3.2 GHz later the biggest machine available from IBM is just 34 percent faster (to be exact: 154,115.8 QphH) than the biggest machine available from Sun from two years ago ... based on a design from the beginning of this century.

By the way: Just in case you wonder about the memory configuration - 512 GB is the maximum configuration that can be used with 667 MHz. More downclocks the memory. Using 4 TB of memory would downclock it to 400 MHz.

Perhaps the challenge gets a little more clear, when we stay in the same shop: Almost 4 years ago IBM published a result of the p5-595. This system yielded with 1.9 Ghz Power5 CPU 100,512.30 QphH. 4 years of development, 256 GByte more memory and 2.6 times the frequency gives you just roundabout 50% more performance. Nice, but not that impressive, especially given the effort put into this CPU in regard of cycles.

An thats perhaps the most important takeaway from this point: Ingo was right. Just increasing the frequency isn't sufficient. It gets clearer with every published benchmark. You have to do more or different. Perhaps the future of large SMP systems is in a more efficient communication between the components, more real bandwidth (not all bandwidths added into a number for marketing purposes) and less bandwidth between the CPUs and between CPU and bandwith.

But i have a request to our benchmarking team: I would like to see an TPC-H result for the M9000 at 3TB. Giving the existing results for the M9000, putting some knowledge into the equation and doing some extrapolations i don't think that that we have to fear this competition. But thats just an educated guess.

BlueToTheBone, Sun, SGI and the SPECjbb2005 benchmark

nospam@example.com (Joerg Moellenkamp) — Fri, 18 Sep 2009 23:03:00 +0200

There is a nice example for my point, that BlueToTheBone tries to downplay everything coming Sun, even when doing so makes a complete fool out of him.

Continue reading "BlueToTheBone, Sun, SGI and the SPECjbb2005 benchmark"