It's a UNIX system!

illumos distros update

noreply@blogger.com (Henkis) — Sun, 01 Sep 2013 20:47:00 +0000

OpenIndiana has finally been updated (to oi_151a8) and with it a few new features and of course many bug fixes. All this should also be true for OmniOS r151006 which has been out for a few month already (and probably SmartOS). Here are a few new features I noticed more than others while scanning the illumos commits since the last update.

3035 LZ4 compression support in ZFS and GRUB
3236 zio nop-write
3137 L2ARC compression
3122 zfs destroy filesystem should prefetch blocks
3805 arc shouldn't cache freed blocks
3561 arc_meta_limit should be exposed via kstats
3537 want pool io stats
3507 Tunable to allow block allocation even on degraded vdevs
749 /usr/bin/kstat" should be rewritten in C

Bug fixes
3422 zpool create/syseventd race yield non-importable pool
3606 zpool status -x shouldn't warn about old on-disk format

Hardware support
3815 AHCI: Support for Marvell 88SE9128
3797 AHCI: Support for ASMedia ASM106x
3500 Support LSI SAS2008 (Falcon) Skinny FW for mr_sas(7D)
3178 Support for LSI 2208 chipset in mr_sas
3408 detect socket type of newer AMD CPUs
3701 Chelsio Terminator 4 NIC driver for illumos

More in depth information on the two larger enhancements to ZFS can be found here:
LZ4 Compression
L2ARC Compression

ZFS Analytics

noreply@blogger.com (Henkis) — Tue, 23 Apr 2013 08:26:00 +0000

While woking with ZFS performance I created a dashboard to get a good overview with lots of different statistics. It's powered by Dtrace, python and graphite. There is a high level of detail but still easy to correlate different statistics.

It feels almost like fishworks analytics lite but without advanced features such as drill-down and heat maps. An example from a box running OpenIndiana:

You get a good view of how the layers interact, the latency for reads in ZFS compared to reads in from the physical disks, average latency, maximum latency, average read size and see how much more data ZFS reads form prefetch including hit rate etc.

I based this on the iomon dtrace script with some glue to send it into graphite, I also added ARC statistics and CPU/Network statistics. ( There is a iomon-graphite effort available on the web but that did not give me correct statistics and did not include things like CPU and network utilization ).

iomon.d

SPARC T5 and M5 systems released

noreply@blogger.com (Henkis) — Wed, 27 Mar 2013 07:38:00 +0000

Oracle have announced new systems based on the new T5 and M5 processors. The T5 has doubled the number of S3 cores from the T4 and increased the clock frequency to 3.6GHz.

The M5 processor is also based on the S3 core (rebranded M4) clocked at 3.6Ghz but is has 6 cores and 48MB L3$. The M systems supports up to 32 M5 processor so a fully configured systems will have 192 cores and 1536 strands (hardware threads). The M5-32 have 32TB of memory in a single system.

Old and existing processors for reference:
T3 16 cores @ 1.65GHz 6Mb L2$ 1-4 socket systems PCIe 2.0
T4 8 cores @ 3.0GHz 4MBL3$ 1-4 socket systems PCIe 2.0

New processors:
T5 16 cores @ 3.6GHz 8MB L3$1-8 socket systems PCIe 3.0
M5 6 cores @ 3.6Ghz 48MB L3$ 32 socket system PCIe 3.0

Oracle claims 2.3x performance gain compared to the T4 with increased single-thread performance.

All new systems both T5 and M5 supports LDOM (Oracle VM for SPARC).

Did everyone read that the M5-32 Supports 32TB of memory? No wounder they had to rewrite the virtual memory subsystem in Solaris 11.1

SPARC Servers

T5,M4 and Athena support in S10U11

noreply@blogger.com (Henkis) — Sat, 23 Feb 2013 23:13:00 +0000

The kernel patch for Solaris 10 update 11 show support for several new platforms: SPARC T5, SPARC M4 and the Fujitsu Athena (SPARC64-X).

7086173 Solaris support for SPARC M4 platforms
7086179 Solaris support for SPARC T5 platforms

7124696 Solaris support for Athena processor

7142242 Solaris support for Fujitsu's SPARC64-X Athena platforms

The Oracle SPARC T5 and M4 processors are unannounced when I write this but the T5 are expected early 2013.

getupdates.oracle.com/readme/147147-26

LDOM 3.0, T5 and M10

noreply@blogger.com (Henkis) — Wed, 13 Feb 2013 10:02:00 +0000

The release notes for LDOM 3.0 confirms what we already knew, logical domains will be supported not only on upcoming SPARC T5 systems but also on the new generation for Fujitsu SPARC Server, M10.

"To enable all the OracleVM Server for SPARC 3.0 features, you must run the required

system firmware versions on the following platforms:

UltraSPARC T2 server Run at least version 7.4.2 of the system firmware.
UltraSPARC T2 Plus server Run at least version 7.4.2 of the system firmware.
SPARC T3 server Run at least version 8.2.1.b of the system firmware.
SPARC T4 server Run at least version 8.2.1.b of the system firmware.
SPARC T5 server Use any installed version of the system firmware.

This firmware is preinstalled on the SPARC T4 and SPARC T5 servers. The required firmware
for Fujitsu M10 systems is preinstalled on your system. For information about the required
Oracle Solaris OS version, see “Required and Recommended Oracle Solaris OS Versions” on
page 12."

The new SPARC M10 systems can now be seen on Fujitsus site (English translation).
From the Register: Fujitsu launches 'Athena' Sparc64-X servers in Japan

Solaris 10 1/13 released

noreply@blogger.com (Henkis) — Sun, 10 Feb 2013 23:58:00 +0000

The last update of Solaris 10 has now been released, update 11. The final name of the release is Solaris 10 1/13 and not 8/12 as plans earlier indicated. This is probably due to the upcoming SPARC T5 systems, to have the releases close in time. Supports the latest and greatest X86 systems, SPARC M-series, SPARC T4/T5 while still working fine on my 15 year old Ultra 2.

Live Upgrade and Zone Preflight checkers
Install and boot from iSCSI
pkgdep command
SSH, SCP, and SFTP Speed Improvements
USB 3 Support
Support for new hardware
Enhanced FMA (AMD,Intel)
Samba 3.6.8 with SMB2 support

Solaris 10 has full support for another five years and extended support until 2021.

Oracle Solaris 10 Downloads
Oracle Solaris 10 1/13 What's New

LDOM 3.0

noreply@blogger.com (Henkis) — Wed, 28 Nov 2012 09:22:00 +0000

Oracle have release Oracle VM for SPARC 3.0 (LDOM).

Enhances the resource affinity capability to address memory affinity.
Retrieves a service processor (SP) configuration from the SP into the bootset, which is on the control domain.
Enhances the domain shutdown process. This feature enables you to specify whether to perform a full shutdown (default), a quick kernel stop, or force a shutdown.
Adds Oracle Solaris 11 support for Oracle VM Server for SPARC Management Information Base (MIB).
Enables a migration process to be initiated without specifying a password on the target system.
Enables the live migration feature while the source machine, target machine, or both, have the power management (PM) elastic policy in effect.
Enables the dynamic resource management (DRM) feature while the host machine has the PM elastic policy in effect.

I hope this release removes the restriction that prevented dynamic reconfiguration of resources after a live migration.
This release also seems to have been tested on upcoming SPARC processors: "7159011 M4/T5: Migration fails initialization on Logical Domains Manager startup" including Fujitsu Athena: "RFE: Cross CPU Migration support between Athena server and T series".

Oracle VM Server for SPARC Documentation

Solaris 11.1 available for download

noreply@blogger.com (Henkis) — Fri, 26 Oct 2012 18:44:00 +0000

Solaris 11.1 is now available for download from Oracle.

The usual SPARC/X86 [text|live|usb|AI] images are available as well as repository images. There are also pre-upgrade repository images available for those of you who are upgrading from 11/11 and have not upgraded to a recent SR or do not have a support contact.

Oracle Solaris 11 Download

Solaris 11.1 announced

noreply@blogger.com (Henkis) — Wed, 10 Oct 2012 22:20:00 +0000

Solaris 11.1 have been announced and will be released later this month. It is the first update of Solaris 11 since it release november last year. It contains a few interesting features, I've only list a few, over 200 projects have integrated into this release.

General

New Virtual memory subsystem (VM2.0 or parts of it)
Scales beyond 100TB, predicts and adapts to page demand, higher performance
RSyslog
USB3 support
Install on UEFI/4K
Interactive install on iSCSI
FedFS

Zones

Parallel zone update
Must faster LOFI
FS statistics per zone
Physical to virtual Solaris 10 migration
Better support for shared storage
Remote Administration Daemon
Secure zone administration with C, Java, Python API
VNIC config in zone XML
Faster install and attach

Security

ASLR
OpenSCAP Security Compliance Checking and reporting tool
Audit remote server

Networking

Data Center Bridging (DCB) IEEE902.1Qaz
Link aggregation span across switches
VNIC migration
Edge Virtual Bridning (EVB) support
High performance SSH

X86

GRUB2
UEFI support
Improved hardware support

I asked the Solaris panel on Oracle Openworld about memory set for zones, it is not part of this release but might be implemented now that the initial part of VM2.0 is implemented.

Solaris 11.1 Whats's new

SPARC T5, M4 and SPARC64-X

noreply@blogger.com (Henkis) — Tue, 09 Oct 2012 11:26:00 +0000

Short summary of SPARC processor information that was disclosed at Oracle world, in the near future Oracle will release two different SPARC processors and Fujitsu will release a new SPARC64 processor with support for LDOM.

SPARC T5 (early next year)

16 Cores 128 threads
28nm
25% increased thread performance to T4
2.5x throughput compared to T4
Scales from 1 to 8 processors
PCIe Gen 3
8MB L3 cache
LDOM virtualization (as with all previous T-series)
Solaris 10 update 11 or Solaris 11.1

SPARC M4 (next year)

6 cores 48 threads
Scales to 32+ sockets
48MB L2 cache
28nm
3.6GHz
5-6x performance per socket compared to M-series
LDOM virtualization
32TB+ memory configurations
Solaris 11 only (but S10 support in LDOM)

SPARC64-X

LDOM virtualization
16 cores 32 threads
24MB L2 Cache
3 GHz
On Chip DB floating point
Crypto acceleration
Runs both S10 and S11 in lab

Oaktable world and OpenWorld 2012

noreply@blogger.com (Henkis) — Wed, 03 Oct 2012 18:44:00 +0000

I am attending both Oracle OpenWorld an ZFS Day/Oaktable world and will post updates as soon as I get some spare time or at the latest early next week.

It was great to talk and listen to the Joyent/Nexenta/illumos guys at Oaktable world.

Oaktable world
Oracle OpenWorld

Oracle forces wesunsolve to close

noreply@blogger.com (Henkis) — Thu, 13 Sep 2012 08:07:00 +0000

In another blow against the community and the people who work with or/and have interest in their products Oracle now has forced wesunsolve.net to close.

In the last two years it has been of tremendous value for administrators who find their support site hard to navigate and the a good overview of patches and updates.

I see nothing Oracle can gain by doing this, no patches where available only metadata with links to their own support site for downloads ( if you have an account with access to the patches ).

Since they already done a much, much worse thing by closing the OpenSolaris source this comes as no surprise. The way companies treat innovation and community efforts are important when you chose a operating system or database engine. Tell your sales representative what you think about these things.

A huge thanks to the people who put time and effort into making wesunsolve.net, it was very useful and will be missed.

wesunsolve.net

Solaris 10 10/12 and T5-8

noreply@blogger.com (Henkis) — Sun, 19 Aug 2012 22:30:00 +0000

Oracle keeps quiet about upcoming Solaris release and their features. I've however figured out the name (and planned release month) for the possibly last Solaris 10 update. It seems like s10u11 will be named Solaris 10 10/12 indicating a October release.

Not much is known about this release more than that it will probably feature a ZFS tech refresh, fully integrate OCM and live upgrade enhancements. It could possibly also support the new T5 processors since they might be quite similar to the current T4 processors except for the doubling of cores.

I have also seen fragments of information indicating that Oracle was running Solaris 11 Update 1 on SPARC T5-8 machines as early as March.

Upcoming SPARC CPUs

noreply@blogger.com (Henkis) — Fri, 17 Aug 2012 23:32:00 +0000

The upcoming Hot Chips symposiums "Big iron" session will feature two future SPARC processors:

"SPARC64 X; Fujitsu’s new generation 16 core processor for the next generation UNIX servers

16-core SPARC T5 CMT Processor with glueless 1-hop scaling to 8-sockets"

The SPARC T5 is expected to be built using 28nm technology and double the number of cores compared to the current T4 processor. The Sun Oracle server line should also include a 8 processor version, T5-8 which will then be have four times the number of cores (128) compared to the current T4-4 (32).

This session will be held August 29, hopefully more information will surface afterwards. Otherwise it would be a safe bet to say that we will know more about the SPARC T5 after Oracle OpenWorld in October.

The Register has an article about both the T5 and the M4: Drilling into Oracle's performance boasts.

Joyent presentations @ FISL 13

noreply@blogger.com (Henkis) — Sun, 29 Jul 2012 00:25:00 +0000

Joyent had a several speakers at the FISL 13 conference and presentations/slides are now available online.

Bryan Cantrill, Corporate Open Source Anti-Patterns: Doing It Wrong
video, slides
Bryan speaks his mind about corporate open source patterns with insights from Sun, the OpenSolaris project and Joyent. He does a bad job hiding what he thinks of Oracle ;)

Brendan Gregg, Performance analysis, the USE method
slides, video
Brendan on performance analysis using the USE method with good examples.

Update: Added Brendans video.

Lots of packages for SmartOS, soon for OpenIndiana

noreply@blogger.com (Henkis) — Fri, 20 Jul 2012 20:54:00 +0000

There is now a huge package repository available for illumos-based distributions, initially a dependency prevents it from running on OpenIndiana but that is being fixed:

9000 packages available for SmartOS and illumos

The packages contains a current PostgresSQL (9.1.3), MySQL, Apache, Ruby 1.9.3, Python 3.2.3 both with lots of modules plus many other useful packages.

All should work on SmartOS and when fixed for OpenIndiana this slightly modified procedure (without sudo and install gtar first) should work, as root:

# pkg install gnu-tar
# curl http://pkgsrc.smartos.org/packages/SmartOS/2012Q2/bootstrap.tar.gz | (cd /; gtar -zxpf - )
# pkgin -y update
# pkgin avail | wc -l
    9401
# pkgin search ...
# sudo pkgin -y install

I'll update this entry as soon as it works for OpenIndiana.

Good summary of enhancements in illumos ZFS

noreply@blogger.com (Henkis) — Fri, 20 Jul 2012 16:05:00 +0000

I found a good summary of enhancements to the free ZFS implementation in illumos:New features in open source ZFS

Also well worth a read is is Matt Ahrens post about the performance of the new async destroy: Performance of zfs destroy.

OpenIndiana updated (oi_151a5)

noreply@blogger.com (Henkis) — Thu, 05 Jul 2012 21:34:00 +0000

A new pre-stable release of OpenIndiana was released a few days ago (oi_151a5), the fifth since the initial illumos-based oi_151a development release in September. Besides bugfixes and minor enhancements the this new release also includes a refresh of the illumos code base which includes a few new noticeable features:

ZFS feature flags
ASynchronous destruction of ZFS file systems
ZFS send progress output

There have also been quite a few userland updates, all is documented in the release notes including a list of CVE-fixes:

OI_151a_prestable5 Release Notes

Update or download images here.

LDOM 2.2 released

noreply@blogger.com (Henkis) — Mon, 28 May 2012 10:57:00 +0000

A new relase of LDOM, currently known as Oracle VM for SPARC has been released. One of the major features of the new release is the ability to do live migration between SPARC T2,T3 and T4 processors. Enabling this features does however have some performance impact:

From ldm(1M):
"cpu-arch=generic|native

Specifies one of the following values:

generic uses common CPU hardware features to enable a guest domain to perform a CPU-type-independent migration.

native uses CPU-specific hardware features to enable a guest domain to migrate only between platforms that have the same CPU type. native is the default value.

Using the generic value might result in reduced performance compared to the native value. This occurs because the guest domain does not use some features that are only present in newer CPU types. By not using these features, the generic setting enables the flexibility of migrating the domain between systems that use newer and older CPU types."

Another major feature is SR-IOV support, which can enable bare metal I/O performance for logical domains, read more here: SR-IOV feature in OVM Server for SPARC 2.2

There a new set of patches available to update the system firmware to 8.2.0 which is needed for the new features.

Announcing Oracle VM Server for SPARC 2.2 Release

illumian available for download

noreply@blogger.com (Henkis) — Mon, 28 May 2012 06:58:00 +0000

A new illumos based distribution is available for download, illumian. illumian is the based on the APT packaging system and is a successor to Nexenta Core Platform (NCP) which was built by nexenta using the OpenSolaris source and APT packages.

The next version of NexentaStore (4.0) should also be built upon illumian, previous versions was built on NCP.

There is currently one image available for server text install on X86 hosts.

illumian.org

ZFS feaure flags and async destroy

noreply@blogger.com (Henkis) — Thu, 24 May 2012 12:34:00 +0000

The first features unique to the open ZFS implementation have been integrated into illumos. As discussed earlier it is feature flags and async destroy of datasets.

illumos gate
ZFS feature flags update

Solaris 11 / SPARC News

noreply@blogger.com (Henkis) — Tue, 22 May 2012 20:50:00 +0000

Here is a good summary of a recent online forum, "Solaris 11: What's new since launch?"

Solaris 11 Update 1 (late this year)

Updated Virtual memory subsystem
Faster Solaris 11 updates with improved python performance
Already running on the upcoming T5/M4 SPARC(R) chips
VNIC configuration switch hosts with their zones

There are also hints on what future Solaris 11/hardware updates might bring

Hotpatching similar to KSplice (Remember DUKS in Solaris 8?)
Offloading of compression and Oracle arithmetics to CPU besides crypto
Schedulers for DB or JVM workloads

Summary: What's new with Solaris 11 since the launch?

ZFS feature flags update

noreply@blogger.com (Henkis) — Wed, 02 May 2012 11:38:00 +0000

ZFS feature flags have been mentioned earlier and now the code is now available from Delphix so that it can be integrated into illumos. With this in place new ZFS features can be implemented in a clean and compatible way, first out seems to be async destroy of datasets (feature flag com.delphix:async_destroy).

Hopefully we will see other new feature soon after this is in in place.

ZFS Feature Flags Presentation (PDF)
Feature flags webrev

OmniOS

noreply@blogger.com (Henkis) — Mon, 16 Apr 2012 06:44:00 +0000

OmniOS is a new illumos-based server distribution with commercial support available was announced at the DTrace conference.

It contains the features you expect like Crossbow, ZFS, DTrace, IPS and Comstar but also includes KVM and updates in userland (Python, GCC, Perl, OpenSSL etc.)

"OmniOS is our vision of what OpenSolaris could have been had it remained in the open. It runs better, faster and has more innovations,” continued Schlossnagle. “OmniTI did not want to lose the benefits that OpenSolaris technologies brought to customers, so we decided to pursue the continuation of the OS on our own. We've been running OmniOS in our data centers for six months and have seen tremendous results. We’re excited to announce our news at the DTrace conference because of its importance and relevance to this community."
- Theo Schlossnagle, CEO of OmniTI

More information, install images and source repositories are available here: omnios.omniti.com

I have only installed the image into VirtualBox witch was painless and quick, I might post an update when I've had time for some exploring.

OmniTI Debuts OmniOS, an Open Source Operating System for the Solaris Community

S11 and S10 inside LDOM 2.1 on T4

noreply@blogger.com (Henkis) — Tue, 21 Feb 2012 16:04:00 +0000

I've finally managed to get some time to play with live migration on a pair of SPARC T4-2. This post is not really adding any new information but is a walk-trough and initial reflections. I am going to continue to write LDOM instead of Oracle VM for SPARC Domains or something like that, even Oracle people still say LDOM and everyone else knows what is.

An interesting note is that I've used Solaris 10 as I/O and Control domain for the T4 servers while the LDOM is installed with Solaris 11 11/11. The disks for the LDOM are on LUNs over FC and MPxIO is used for multipathing from the I/O domain:

t42-01# dskinfo list-long
disk                                   size lun   use           p   spd type lb
c0t5000CBA015B85D98d0                  279G -     rpool         -   -   disk  y
c0t5000CBA015B93B90d0                  279G -     -             -   -   disk  y
c0t50002870000254901593534030832420d0  33G  0x0   -             4   4Gb fc    y
c0t50002870000254901593534030832420d0  33G  0x1   -             4   4Gb fc    y

Examples of migrating and reconfiguring the LDOM while running:

t42-01# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 6h 37m
ldms11-01        active     -n----  5000    16    8G       0.0%  24m

t42-02# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 1h 26m

ldms11-01:~$ uptime 
  5:11pm  up 19 min(s),  1 user,  load average: 0.00, 0.00, 0.01
henrikj@ldms11-01:~$ prtconf -v |grep Mem
Memory size: 8192 Megabytes
henrikj@ldms11-01:~$ psrinfo | wc -l
      16

t42-02# ldm set-vcpu 96 ldms11-01
t42-02# ldm set-memory 200G ldms11-01

t42-02# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  UART    16    16G      0.1%  12d 6h 50m
ldms11-01        active     -n----  5000    96    200G     0.1%  24m

ldms11-01:~$ prtconf -v |grep Mem
Memory size: 204800 Megabytes
ldms11-01:~$ psrinfo | wc -l
      96

When performing a live migration between the two hosts, running processes and open network connections are as expected intact, there is only a small delay in the network traffic visible. For my initial tests the delay was about 10 ms.

The live migration seems to work very well and the T4 seems to perform several times faster than the T2/T3 for general workloads. The only thing missing is that LDOM 2.1 is unable to dynamically reconfigure memory and CPU resources for a domain after migration. A reboot is then required, hopefully this will be fixed in the 3.0 release, which people at Oracle Open World said would be focused on removing current limitations (including migration between different types of sun4v processors).