c0t0d0s0.org - Solaris

Translations

nospam@example.com (Joerg Moellenkamp) — Sat, 19 May 2012 11:32:49 +0200

I really like to read our internal aliases. Because sometimes you are stumbling about a piece of information that reminds you that it could be a article for the blog. In this case it was about the topic of the virtual memory translation. I will simplyfing things a bit, so dear experts, i know that the situation is more complex For people interested in more details the chapters 9-12 and chapter 3 Section 3.11.7 of "Solaris Internals: Solaris 10 and OpenSolaris Kernel Architecture (2nd Edition): Solaris 10 and Open Solaris Kernel Architecture" is a good start.

Whenever you use some virtual memory, there has to be some mapping from the virtual addresses to the real addresses. However to prevent the CPU to look again and again in - from CPU cycles perpective - distant memory areas, there are little caches in modern processors called translation lookaside buffer or short TLB. This TLBs are rather small, a T4 core has 128 entries. While sounding small, the TLB with such a number of entries has an astonishing high hit rate.
Continue reading "Translations"

Upcoming event - Oracle Solaris 11: Whats New Since the Launch

nospam@example.com (Joerg Moellenkamp) — Sun, 15 Apr 2012 20:03:06 +0200

On April 25th an webbased event about Solaris 11 takes place: It's named Oracle Solaris 11: Whats New Since the Launch.

Agenda
9:00 a.m. PDT	Keynote: Oracle Solaris - Strategy and Update Markus Flierl, Vice President, Oracle Solaris Engineering
9:40 a.m. PDT	Oracle Solaris 11: Extreme Engineering - A Technical Update Dan Price, Senior Principal Product Engineer, Oracle Solaris Engineering Bart Smaalders, Senior Principal Product Engineer, Oracle Solaris Engineering
10:20 a.m. PDT	Customers and Partners: Why We Moved to Oracle Solaris 11 A discussion of the reasons why businesses and commercial software developers have adopted Oracle Solaris 11, from the people responsible for these decisions
11:00 a.m. PDT	Oracle Solaris: Core to the Oracle Systems Strategy John Fowler, Executive Vice President of Systems, Oracle

9:00 am PDT is 18:00 in Berlin, 17:00 in London and i assume much to late in Tokyo with 01:00 am the next day ...

Using the Support Repository for Solaris 11

nospam@example.com (Joerg Moellenkamp) — Sun, 15 Apr 2012 11:23:18 +0200

"How to Update Oracle Solaris 11 Systems Using Support Repository Updates" is a great article written by Glynn Foster in order to explain the use of the Support Repository for Solaris 11. A must read.

Veranstaltungshinweis: 2. Oracle Breakfast

nospam@example.com (Joerg Moellenkamp) — Fri, 13 Apr 2012 20:56:50 +0200

Am 26. April findet das zweite Oracle Oracle Breakfast in Hamburg in der Geschäftsstelle (Kühnehöfe 5) statt: Also Futtern mit technischem Content. Auch diesmal gibt es zwei Vorträge.

Agenda
9:30	Willkommen zum Frühstück
10:00	Solaris 11 im Detail - Einbindung in heterogene Netze (CIFS-Dienst etc.) Joerg Moellenkamp
11:30	Kaffeepause
12:00	ZFSSA praktisch Einbindung einer ZFS SA in heterogene Netze, aber wie? Vortrag & Livedemo unter VirtualBox Dirk Nitschke
13:30	geplantes Ende

Anmelden könnt Ihr euch mit einer formlosen Mail an oraclebreakfast_ham@c0t0d0s0.org. Das ist ein Forwarder an die Addresse der Kollegin, die das intern bei uns organisiert, deren Mailaddresse ich nicht unbedingt für Spammer verteilen möchte ...

Solaris 11 LKSF

nospam@example.com (Joerg Moellenkamp) — Fri, 13 Apr 2012 20:37:30 +0200

After having some discussions i now made my mind about it: In the next weeks you will see many republications of old articles in the blog as i will republish all articles in the LKSF, however checked and updated for Solaris 11 (some Opensolaris based stuff in the lksf is working slightly different, and if it's just for different package names). However this will take time, as i will do this on weekends and evenings. At the end i will just recollect them and create a Solaris LKSF pdf again.

Performance impact of Zones.

nospam@example.com (Joerg Moellenkamp) — Thu, 05 Apr 2012 18:46:34 +0200

I was really astonished when i saw this question. Because this question was a old acquaintance from years ago, that i didn't heard for a long time. However there was it again. The question: "What's the overhead of Zones?". Sun was and Oracle is not saying "zero". We saying saying minimal. However during all the performance analysis gigs on customer systems i made since the introduction of Zones i failed to measure any overhead caused by zones. When there was overhead in that situation it was in the error of measurement.

What i saw however, was additional load intoduced by processes that wouldn't be there when you would use only one zone Like additional monitoring daemons, like additional daemons having a controlling or supervising job for the application that resulted in slighly longer runtimes of processes, because such additional daemons wanted some cycles on the CPU as well. Like additional sshd daemons, like additional init processes. So i ask when someone wants to tell me that he measured a slight slowdown, if he or she has really measured the impact of the virtualization layer or of a side effect described above.

It seems to be a little bit hard to believe, that a virtualisation technology has no overhead, however keep in mind that there is no hypervisor and just one kernel running that looks and behaves like many operating system instances to apps and users. While this imposes some limits to the technology (because there is just one kernel running you can't have zones with different kernels versions running ... obvious even to the cursory observer), but that is key to it's lightweightness and thus to the minimal overhead.
Continue reading "Performance impact of Zones."

Solaris 11 features: nscfg

nospam@example.com (Joerg Moellenkamp) — Mon, 02 Apr 2012 09:43:00 +0200

As you may have noticed many configuration tasks around name services have moved into the SMF in Solaris 11. However you don't have to use the svccfg command in order to configure them, you could still use the old files. However you can't just edit them, you have to import the data into the SMF repository. There are many reasons for this need but the ultimate one is in the start method. I will explain that later. In this article i want to explain, how nscfg can help you with with the naming service configuration of your system.

Continue reading "Solaris 11 features: nscfg"

Future of Solaris

nospam@example.com (Joerg Moellenkamp) — Sun, 01 Apr 2012 20:01:43 +0200

Alan Coopersmith gives some interesting and important insights into the future of Solaris.

Yet another cheatsheet update ...

nospam@example.com (Joerg Moellenkamp) — Sat, 31 Mar 2012 19:18:47 +0200

Yet another update of the Solaris 11 cheatsheet. Some additions about disk devices, some bug fixes (there is one additional bug "Configuring system locale:" must be "Configuring time zone" ... will be fixed in the next version)

SPARC Power Management Article at OTN

nospam@example.com (Joerg Moellenkamp) — Sun, 25 Mar 2012 13:50:06 +0200

My colleague Karoly Vegh pointed in a tweet to a really interesting article about the usage of Power Management of SPARC T-series systems. The article explains how to use the power management, how it works, what it's able to do and how to use it in a dynamic fashion according to anticipated load patterns. You find the article "How to Use the Power Management Controls on SPARC Servers" written by by Bruce Evans, Julia Harper, and Terry Whatley on OTN.

Oracle Solaris 11 Developer Webinar Series

nospam@example.com (Joerg Moellenkamp) — Sat, 24 Mar 2012 09:24:42 +0100

Starting March 27th a number of events called "Oracle Solaris 11 Developer Webinar Series" (that said, i think they are not only interesting for developers, but for admins as well that support admins in their job) will take place by WebEx/Phone:

All events start at 9am Pacific Daylight Time (Should be 18:00 in Germany, 17:00 in UK, 01:00 in Tokyo). Of course the events are for free (before someone asks) and available for anyone who signs up. The chat is staffed with Solaris 11 Engineering Experts sitting in to answer questions. And most important: Mostly technical content and the events will be recorded.

Solaris 11 cheatsheet updated

nospam@example.com (Joerg Moellenkamp) — Fri, 23 Mar 2012 06:46:43 +0100

There is an updated version of the Solaris 11 cheatsheet.

Tutorial for Automated Installer

nospam@example.com (Joerg Moellenkamp) — Mon, 19 Mar 2012 11:31:52 +0100

Stefan Hinker wrote a nice tutorial helping you at the setup of a local install server: "Setting up a local AI server - easy with Solaris 11".

Performance impact of the new mtmalloc memory allocator

nospam@example.com (Joerg Moellenkamp) — Fri, 16 Mar 2012 08:50:44 +0100

I wrote at a number of occasions (here or here), that it could be really beneficial to use a different memory allocator for highly-threaded workloads, as the standard allocator is well ... the standard, however not very effective as soon as many threads comes into play.

I didn't wrote about this as it was in my phase of silence but there was some change in the allocator area, Solaris 10 got a revamped mtmalloc allocator in version Solaris 10 08/11 (as described in "libmtmalloc Improvements"). The new memory allocator was introduced to Solaris development by the PSARC case 2010/212.

But what's the effect of this new allocator and how does it works? Rickey C. Weisner wrote a nice article with "How Memory Allocation Affects Performance in Multithreaded Programs" explaining the inner mechanism of various allocators but he also publishes test results comparing Hoard, mtmalloc, umem, new mtmalloc and the libc malloc. Really interesting read and a must for people running applications on servers with a high number of threads.

Oracle Solaris 11 Cheatsheet

nospam@example.com (Joerg Moellenkamp) — Thu, 15 Mar 2012 08:06:47 +0100

In the last few days i created a cheat sheet for Solaris 11 ... while it's still a work in progress (it will be surely longer in the future). it's already available on OTN as part of the "Evaluating Oracle Solaris 11"-Collection . You can download the "Oracle Solaris 11 Cheat Sheet General Administration" here.

Cheatsheet for configuring a new nodename in Solaris 11

nospam@example.com (Joerg Moellenkamp) — Sat, 03 Mar 2012 20:39:55 +0100

In Solaris 11 the file /etc/nodename is gone. In order to change you nodename you have alter the configuration of an smf service:

root@fuchikoma:/home/jmoekamp# hostname
tachikoma
root@fuchikoma:/home/jmoekamp# svccfg -s svc:/system/identity:node setprop config/nodename = "fuchikoma"
root@fuchikoma:/home/jmoekamp# svcadm refresh svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# svcadm restart svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# hostname
fuchikoma

You don't have to edit config/loopback

root@fuchikoma:/home/jmoekamp# svcprop  svc:/system/identity:node | egrep "^config"        
config/nodename astring fuchikoma
config/loopback astring fuchikoma

Cheatsheet for configuring the networking in Solaris 11

nospam@example.com (Joerg Moellenkamp) — Sat, 03 Mar 2012 13:49:55 +0100

There are quite a number of changes in the procedures to configure some of the networking parameters. Many things have changed, that were just editing of a file in the past, have now command-line based tools in order to change their parameters. Before you ask: The reason for this steps are quite simple.

At first ist much easier to script the execution of a command, instead of the editing a file. I wrote a lot of those scripts in the past and from my experiences people have an infinite wisdom in creating ways and means to edit a file in a way that it breaks your automatic edting scripts. Furthermore putting all this configuration statements into SMF has an interesting consequence. In the past, it wasn't always clear to most people when a change of the configuration got active. At the time of the save, at the restart of the daemon? By using SMF it's clear. As soon as you type in svcadm refresh. Furthermore ifconfig came totally overloaded with parameters with the time. So dladm and ipadm were a very good move from my perspective.

However i know that this is maybe a strange for an long-time user, and to be honest, i was spelling many curses in front of the display, albeit i see the advantages. So i'm putting up my cheatsheet online. My colleague Detlef is planing something similar. As soon his is ready, i will linking to his as well, as i'm sure it's more complete.

With automatisms

In the datacenter it's not that useful, however when you have an Solaris VM for example on your notebook you are using for administrative purposes, it's really nice.

At first we create the network configuration profiles for the datacenter and the office.

root@tachikoma:/home/jmoekamp# netcfg create ncp datacenter
root@tachikoma:/home/jmoekamp# netcfg create ncp office

Now we have to fill both with the configuration data. At first the one for the datacenter. I think the values are speaking for themself.

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp datacenter
netcfg:ncp:datacenter> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> static
ipv4-addr> 192.168.1.27
ipv4-default-route> 192.168.1.1
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> exit
root@tachikoma:/home/jmoekamp#

Now we do the same for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp office
netcfg:ncp:office> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> dhcp
ipv4-default-route>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> exit

Now we can just switch the networking configuration by enabling one of the both profiles:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201000843 mtu 1500 index 6
        inet 192.168.1.27 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Let's switch to the office profile:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201004843 mtu 1500 index 7
        inet 192.168.1.185 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Of course the networking configuration is not just the interface configuration but a lot other stuff as well, link the configuration of the DNS. That options are in the location profiles. However i don't want to activate the profile as well as the location both manually. Thus i configure an automatic activation of such configuration. In my example the datacenter profile configures an IP address of 192.168.1.27, the office profile uses DHCP which results in an IP-address out of the 192.168.1.100-199 range. Thus i'm using this for my automatic location switching. I can configure an activation mode and by choosing conditional-any this location profile will be activated automatically when any of the configured conditions are true. It's pretty simple in my case: If the IP address is 192.168.1.27, i'm in the datacenter, if not i'm in the office.

So, i'm configuring the location profile for the datacenter first. I think this is speaking for itself again, when you take the information into consideration, i gave you in the last paragraph:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc datacenter
Created loc 'datacenter'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is 192.168.1.27
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> datacenter.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:datacenter>
netcfg:loc:datacenter> exit
Committed changes

Now i will configure the one for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc office
Created loc 'office'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is-not 192.168.1.28
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> office.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:office>
netcfg:loc:office> exit
Committed changes
root@tachikoma:/home/jmoekamp# netcfg

Okay, let's try this out. At first we activate the datacenter network configuration profile.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'

In theory, the resolv.conf should now change to nameserver 192.168.1.1. Let's check this.

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  datacenter.c0t0d0s0.org
nameserver      192.168.1.1

Okay, that was successful. Now we will switch to the network configuration profile office.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'

We have a look into the resolv.conf again. The nameserver should have switched to 192.168.1.43:

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  office.c0t0d0s0.org
nameserver      192.168.1.43

Nice.

Without that automatic stuff ....

Configuring all this stuff manually isn't hard as well. Perhaps you have chosen "Automatic" initially at install time in the CUI. So at first we have to get rid of the automatisms.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp defaultfixed
Enabling ncp 'DefaultFixed'

Okay, now we configure an ip interface called net0 and assign an ip address

root@tachikoma:/home/jmoekamp# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             unknown    0      unknown   pcn0
root@tachikoma:/home/jmoekamp# ipadm create-ip net0
root@tachikoma:/home/jmoekamp# ipadm create-addr -T static -a  192.168.1.28/24
net0/v4

Of course we need a defaultrouter. With this command we configure such a persistent default route.

root@tachikoma:/home/jmoekamp# route -p add default 192.168.1.1
add net default: gateway 192.168.1.1
add persistent net default: gateway 192.168.1.1

Configuring the dns server is a little bit more complex:

root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop "config/nameserver = net_address: ( 192.168.1.1 )"
root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop 'config/domain = astring: ("c0t0d0s0.org")'
root@tachikoma:/home/jmoekamp# svccfg -s "name-service/switch" setprop 'config/host = astring: "files dns"'
root@tachikoma:/home/jmoekamp# svcadm refresh name-service/switch
root@tachikoma:/home/jmoekamp# svcadm enable dns/client

Okay, let's check the result:

root@tachikoma:/home/jmoekamp# grep "192.168.1.43" /etc/resolv.conf
nameserver      192.168.1.43
root@tachikoma:/home/jmoekamp# grep "host" /etc/nsswitch.conf
hosts:  files dns

Everything works as designed.

Being relaxed in regard of retention

nospam@example.com (Joerg Moellenkamp) — Mon, 27 Feb 2012 21:16:15 +0100

Robin Harris of StorageMojo pointed to an interesting research paper: "Optimizing NAND Flash-Based SSDs via Retention Relaxation". The idea of the method described in that paper is basically: The physics behind the process of writing to NAND-flash allows you to write you faster, when you shorten the retention time. The shorter the retention time, the faster the write. Retention time? That the span of time you have to be capable to read the data from a device.

Obviously I mapped that automatically into the ZFS world: For an separated ZIL device write latency is a really, really important metric.However: The normal live time of the data writen to the sZIL? Just until the transaction group has been committed to the pool. A few seconds. However the need retention time is not that short, because you have address failures. You write something like a ZIL for a reason. In the case of a system failure the data has to survive at least as long until the same or a different system is able to commit the data - that wasn't commited before the failure - that is still on the sZIL to the pool devices. However how long is this? A day perhaps a week. But surely not the standard of 1year or 10 years. However the standard mechanism writes so it could be read for 1 or 10 years as mandated by some standard bodies. And this is exactly the margin the idea in the paper uses. The model in the paper suggest a write speedup of 2.5 in response time for a TPC-C-similar load (figure 13) when you target for a retention time of two weeks instead of a year.

When you combine sZIL and flash storage capable of such a relaxed retention you could basically use such an optimization for pretty much all data storage needs, as the data is still stored in the pool. Further more you can forget about the refreshment of the data as suggested by the paper for data with longer retention time needs, because by it's nature all data is really short-lived and other processes take care of making the data persistent for the next 10 years. And as a weird idea at the end: You could even use a retention-relaxed SSD as an sZIL device in front of a pool of SSD without relaxed retention.

Using encrypted lofi devices for backup

nospam@example.com (Joerg Moellenkamp) — Mon, 27 Feb 2012 16:26:14 +0100

My colleague Constantin wrote a nice article about using encrypted loopback file based pools for backup purposes for example. And don't forget you can do deduplication or snapshots as well for your backuped data. And in case you don't know how migrate your data into this files ... use shadow migration

More than this

nospam@example.com (Joerg Moellenkamp) — Sun, 26 Feb 2012 08:36:12 +0100

I had many interesting discussions in the last few days. However some of them gave me the impression that i should explain one thing. I want to use one example for it: A customer asked me while having some coffee at the Tech Days: "Zones? Isn't that just jails like in FreeBSD?". It's that old question, i get since the introduction of Zones. Just to make it clear: It's not a text in regard of FreeBSD. It's a text about the tendency of people just to pick a single feature and to say "Isn't that feature like ". But it isn't that easy.

I don't want to discuss about that question, if a feature is really like another. However i want to introduce a different tought into this:"A feature is not only a feature when you have an overarching architecture and an overarching idea where the architecture is heading to.It's an enabler. And a feature is often just the next question".

A feature is just the next question? Yes, that right. Because every feature you introduce is just the starting point for the next feature. When we introduced Zones years ago, we had a lot open questions not much later: How do you patch those zones? Especially when you have dozens of them? How to delegate administration? How do you install zones in a fast manner? How to implement bootenvironments, for the OS as well as the Zones? How do you reduce hard-to-find problems of an architecture that shares a kernel, but has several copied userlands? Questions, that have perhaps no technical background, but resulting in technical changes because of operational requirements. Perhaps, at the beginning Jails and Zones may have been similar concepts. But when you look today into the construct, Zones is a lot more. Zones is a large interdepending web of features inside of zones and outsides of zones to enable customers to work with them as efficient as possible.

However: Some of the challenges are just solvable when you have an overarching architecture and the power to decide on the architecture. And this is what i want to say with "a feature is not only a feature". Sometimes it's an enabler for a different feature. For example you have to be capable to say "ZFS is the only filesystem for booting and keeping zone roots" then you have a foundation you can use to implement other features. You can take all the mechanisms of ZFS for example for granted to base other features on it.

An example from the automobile sector? Ever asked why the automatic parking for a VW Golf is that cheap? Well, it just reuses an electro motor that it's there for power steering. You just need some software and a computer to give directly orders to this already existing power steering motor? When you allow all engineering teams to use it's own power steering implementing automatic parking in all vehicles is much more difficult.

Or in Solaris: Fast zone cloning? You need snapshots for it. So go to the ZFS people. Exclusive IP-Stack for a zone? You need a revamped IP stack for it. Ask the Crossbow people. Many things are severely depending on each other? Boot environments like in Solaris 11? Just feasible with a filesystem capable to do snapshots. IPS was invented to a part to have a packaging format that is much more aware of a concept like zones than just post/pre install scripts where zones was just an afterthought. Bandwidth Management? The Crossbow people again. Resource Management for your zones? You could use the foundation already laid out by the people with the Solaris Resource Manager years ago.

And there i'm at the point where i'm saying it doesn't suffice to have a feature, for example like Jails. At the introduction of a feature, a new journey just begins to solve all implication of a new feature. And you have to go all the way to make it really good.

Solaris 11 in Common Criteria evaluation

nospam@example.com (Joerg Moellenkamp) — Tue, 21 Feb 2012 08:12:31 +0100

As i got the question on the Solaris 11 Techdays: As reported by the Oracle Tech Network Solaris 11 is now in evaluation under the Operating System Protection Profile using the extended packages Advanced Management, Extended Identification and Authentication, Label Security, and Virtualization at level EAL4+.

Less known Solaris 11 features: Shadow Migration

nospam@example.com (Joerg Moellenkamp) — Sun, 12 Feb 2012 08:08:29 +0100

In the ZFS Storage Appliance we have little nice feature enabling you to do migrations of data in the background. It's called Shadow Migration. It's a really useful feature. Imagine you have a RAIDZ. After a time you recognize that RAIDZ wasn't a good decision for your workload and RAID10 would be much better choice. But how to transform it into a RAID10 and how to do it with minimal interruption? You can do this with the Shadow Migration feature. With the Shadow Migration feature, you can migrate the data from one local or remote filesystem to another, while you are already accessing the new one to get the data on the old ZFS filesystem. This feature is available in Solaris 11 as well.

For this demonstration we will use two zfs pools consisting out of files. So we have to create the files first:

root@test:/test/brainslug# mkfile 128m source1
root@test:/test/brainslug# mkfile 128m source2
root@test:/test/brainslug# mkfile 128m source3
root@test:/test/brainslug# mkfile 128m source4
root@test:/test/brainslug# mkfile 128m target1 
root@test:/test/brainslug# mkfile 128m target2
root@test:/test/brainslug# mkfile 128m target3
root@test:/test/brainslug# mkfile 128m target4
root@test:/test/brainslug# mkfile 128m target5
root@test:/test/brainslug# mkfile 128m target6

Now the pools are created. At first our RAIDZ pool consisting out of 4 files. It's named source

root@test:/# zpool create source raidz \
 /test/brainslug/source1 \
 /test/brainslug/source2 \
 /test/brainslug/source3 \
 /test/brainslug/source4

The second one is the future target of the shadow migration. It consists out of six "disks"

root@test:/# zpool create target
  mirror /test/brainslug/target1 /test/brainslug/target2 \
  mirror /test/brainslug/target3  /test/brainslug/target4 \
  mirror /test/brainslug/target5 /test/brainslug/target6

When you did a basic install, the tools and daemons needed for shadow-migration are not included. You have to install them and enable the shadowd afterwards:

root@test:/test/brainslug# pkg install shadow-migration
root@test:/test/brainslug# svcadm enable shadowd

Now you should see the shadowd daemon running.

root@test:/test/brainslug# ps -ef | grep "shadow"
    root  3292     1   0 14:32:33 ?           0:03 /usr/lib/fs/shadowd

Okay to test the shadow migration we create a filesystem in the source pool:

root@test:/test/brainslug# zfs create source/somestuff

Now we have to fill this file with a some data. Let's create some play data.

root@test:/test/brainslug# dd if=/dev/urandom of=myfile bs=1024 count=300000                        
300000+0 records in
300000+0 records out
root@test:/test/brainslug# mkdir demodata
root@test:/test/brainslug# cd demodata
root@test:/test/brainslug/demodata# split -b 128k -a 5  ../myfile

This should yield a significant number of 128k files. Now we copy them to the newly created filesystem source/somestuffWe will copy the files into the zfs filesystem posing as our old filesystem:

root@test:/test/brainslug/demodata# cp * /source/somestuff/
root@test:/test/brainslug/demodata# cd /
root@test:/# zfs list source
NAME    USED  AVAIL  REFER  MOUNTPOINT
source  294M  42,1M  46,4K  /source

Just to have something to compare, you could simply count the files and calculate the md5 checksum of a file.

root@test:/# ls -l /source/somestuff | wc -l
    2345
root@test:/# md5sum /source/somestuff/xaadmd
3fb4a6be2f93c3d93998db52061244aa  /source/somestuff/xaadmd

Shadow migration will only works, when the source filesystem read-only. So we have to put the source filesystem into such a state:

root@test:/# zfs set readonly=on source/somestuff

Okay, now let's configure the shadow migration:

root@test:/# zfs create -o shadow=file:///source/somestuff \
target/newlocationforsomestuff

That's all. The command may take some moments to get back. The migration of data starts right in the moment you create the new filesystem. It runs in the background and starts to copy all data to the new filesystem. Important to know: You can do shadow migration via NFS as well and it can be an UFS filesystem as well. you just have to declare the source of the shadow migration like nfs://fileserver/directory

Okay. With shadowstat we can check the process of migration.

root@test:/# shadowstat    
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  25,5M   -       -       00:01:10

The cool think about shadow migration is: You can already use the new filesystem. Despite the fact that the migration is still running, you will already see all files and when you access one file it will be migrated in the moment you access the file on the new filesystem. You don't have to wait with the access, until the block would be migrated by the normal background migration. When you try to access data, that isn't already migrated, it's migrated in the moment you access it in the new filesystem.

root@test:/# md5sum /target/newlocationforsomestuff/xaadmd               
3fb4a6be2f93c3d93998db52061244aa  /target/newlocationforsomestuff/xaadmd
root@test:/# ls -l /target/newlocationforsomestuff | wc -l                       
    2345

Afterwards it proceeds with the further migration of all data in the pool. You can observe that with the shadowstat command.

root@test:/# shadowstat 
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  97,8M   -       -       00:01:50
target/newlocationforsomestuff  128M    -       -       00:02:00
target/newlocationforsomestuff  147M    -       -       00:02:10
target/newlocationforsomestuff  165M    -       -       00:02:20
target/newlocationforsomestuff  186M    -       -       00:02:30
target/newlocationforsomestuff  202M    -       -       00:02:40
target/newlocationforsomestuff  211M    -       -       00:02:50
target/newlocationforsomestuff  224M    -       -       00:03:00
target/newlocationforsomestuff  236M    -       -       00:03:10
target/newlocationforsomestuff  243M    -       -       00:03:20
target/newlocationforsomestuff  249M    -       -       00:03:30
target/newlocationforsomestuff  256M    -       -       00:03:40
target/newlocationforsomestuff  260M    -       -       00:03:50
target/newlocationforsomestuff  266M    -       -       00:04:00
target/newlocationforsomestuff  272M    -       -       00:04:10
target/newlocationforsomestuff  278M    -       -       00:04:20
target/newlocationforsomestuff  286M    -       -       00:04:30
No migrations in progress

root@test:/#

Successfuly migrated.

Do you want to learn more?

Docs
docs.oracle.com: Migrating ZFS File Systems"
docs.oracle.com: Migrating File System Data to ZFS File Systems

Blogs
blogs.oracle.com: What is Shadow Migration
blogs.oracle.com: Shadow Migration Internals

Oracle Solaris 11 Techday Düsseldorf geschafft ...

nospam@example.com (Joerg Moellenkamp) — Tue, 07 Feb 2012 21:52:31 +0100

... bleiben noch sechs Veranstaltungen (man kann sich noch anmelden). Der Event in Düsseldorf war schon mal ein guter Auftakt. Viele Leute da, viele Fragen gehabt in den Kaffeepausen. Morgen nun Stuttgart ...

Fand meinen Vortrag ein wenig holprig, zuviel Reuse, bin irgendwie nicht in den Flow gekommen. Habe meinen Vortrag aber für die nächsten Veranstaltungen noch ein wenig umgebaut. Und bevor jemand fragt: Ja ... die Slideanzahl ist noch zweistellig Hab doch nur 30 Minuten ....

PS: Mag mal jemand den Flughafenbetreibern in DUS sagen, das ein Hinweisschild ganz nett wäre, das es da einen Starbucks oben bei der Besucherterasse gibt? Ich bin stumpf dran vorbeigelaufen.

Oracle Solaris 11 Tech Days 2012

nospam@example.com (Joerg Moellenkamp) — Mon, 23 Jan 2012 15:18:11 +0100

Im Februar läuft eine Reihe von Events in Deutschland und der Schweiz zum Thema Solaris 11. Die Events versprechen technisch sehr interessant werden, da die Sprecher jeweils sehr tief in der Materie sind. Über Detlef Drewanz - der bei allen Events dabei ist - muss ich seit dem Containerleitfaden genauso wie über Uli Gräf (der an einigen, aber nicht allen Orten spricht) wohl nichts mehr sagen. Christian Christian Ritzka und Elke Freymann sind ausgewiesene Experten zum Thema OpsCenter. Und ja .. ich halte auch einen Vortrag über Datamanagement in Solaris 11. Und da ich schon zweimal die Frage gesehen habe: Die Veranstaltung ist kostenfrei


08:30 - 09:00	Registrierung
09:00 - 09:15	Begrüßung
09:15 - 10:00	Was ist neu in Oracle Solaris 11 Viele Features, die im Rahmen der Solaris 10 Entwicklung in Solaris 11 Express eingeflossen sind, finden sich auch in Solaris 11 wieder. Diese Präsentation gibt einen Überblick über die neusten Features.
10:00 - 11:00	Oracle Solaris 11 Installation Die wohl herausragendste Eigenschaft von Oracle Solaris 11 ist das neue Package System IPS und der Autoinstaller, die die Installation und das Management von Oracle Solaris 11 vereinfachen. Lernen Sie die neuen Techniken kennen und lassen Sie sich zeigen, wie einfach das Patchen unter Oracle Solaris 11 ist.
11:00 - 11:30	Pause
11:30 - 12:30	Oracle Virtualisierung In Oracle Solaris 11 sind umfangreiche Virtualisierungstechniken integriert. Lernen Sie alles über die neue Netzwerk Virtualisierung in Oracle Solaris 11 und wie sie komplette multi-tier HW Infrastrukturen in einer einzelnen Maschine zusammen mit dem Oracle Virtual Machine framework und Solaris Zonen realisiert werden kann.
12:30 - 13:30	Mittagessen
13:30 - 14:15	Management von IT Infrastrukturen Virtualisierung heist nicht nur "Hypervisor". In diesem Vortrag zeigen wir, wie sich virtualisierte Oracle Solaris 11 Umgebungen zentral verwalten lassen.
14:15 - 14:45	Das Solaris Schulungsprogramm Oracle University stellt zusammen mit unseren Schulungspartnern ein umfassendes Programm zur Vertiefung von Solaris Wissen zur Verfügung. In diesem Vortrag werden die Ausbildungpfade, Kurse und Zertifizierungen für Solaris 11 beleuchtet und verfügbare Lernformen vorgestellt.
14:45 - 15:15	Pause
15:15 - 15:45	Oracle Solaris 11 Datamanagement Oracle Solaris 11 hat umfassende Datamanagement Funktionen integriert. Lernen Sie die neusten ZFS features wie Data Encryption und Deduplikation kennen und wie Sie diese Funktionen über die CIFS-Integration im Kernel auch anderen Plattformen zur Verfügung stellen können.
15:45 - 16:15	Panel, Q&A
16:15 - 16:45	Erfrischungen, Zeit zur Diskussion mit den Experten

Die genaue Agenda mit den Sprechern in den einzelnen Orten und eine Möglichkeit zur Anmeldung findet ihr auf den Eventseiten:

Um zahlreiches Erscheinen wird gebeten!

Simulating the cloud - a practical example.

nospam@example.com (Joerg Moellenkamp) — Wed, 28 Dec 2011 21:37:00 +0100

Work in Progress - this entry will change often in the next days and weeks

A few days^H^H^H^Hweeks ago, i wrote about simulating the cloud that is most often tagged with the name "network" or "intranet" and sometimes "internet" . This would not be c0t0d0s0.org without an article to explain how you can configure this. This article will explain how you simulate a complete network on a single host with routers, switches, dynamic routing protocols and so on

Scope

At first i want to set the expectations right. I don't want to simulate a cloud in the sense of cloud computing here. I'm thinking about something more complex:

I'm talking about the simulation of this cloud, that often hides a lot of complexities and traps in architectural diagrams.
Continue reading "Simulating the cloud - a practical example."

Simulating that clould

nospam@example.com (Joerg Moellenkamp) — Sat, 10 Dec 2011 09:45:38 +0100

In the past i wrote quite often about a thing that i call systemic features, when features start to fit together seamlessly in order to create possibilities more than the sum of the features. One of the systemic features is the simulation of the cloud. I don't talk about that thing that most people connect in mind with the word cloud (the grid with a credit card checkout ), but the cloud-like icon in many architectural diagrams called "Network" or "Internet" that sits between the client and the application that often resembles the "a wonder happens here" box in many architectures.

It's not new: I talked about this mid November at the DOAG conference in Nuremberg. And i've playing around with this at customers an privately for a while now.

Many customers have networks as large and as complex as the internet part of a smaller country perhaps 15 years ago. The interesting question is: How can you test your application for it's resiliency against failures in this cloud shaped icon. How does your application react, when your network is doing its high availability magic.

And interestingly Solaris 11 can help you here. The thoughts behind this are pretty simple.

A router is a computer that runs an operating environment that is tailormade to do network stuff, but at the end it's a computer with a OS (yeah, i know, hardware offloading makes this a little bit more complex, but at the end it's that way)
A zone is a virtual operating environment.
Each zone can have it's own set of routes.
Each zone can have it's own set of firewall rules.
Each zone can have it's own set of processes.
Routing protocols are not more than processes collecting information from the network and configuring the routing table.
You can install a vast array of dynamic routing protocols on a zone.
I can have up to 8192 zones (given enough memory)
In Solaris 11 i can emulate switches (etherstubs)
I can limit bandwidths in Solaris 11 out-of-the-box with crossbow

When i'm combining all this features i can set up a vast array of zones doing nothing else taking each incoming packet on a interface, routing it on a multitude of ways between each other, and send it out on a outgoing interface. Even when the system in your environment are placed in many separate networks of your network you can still use a system with many networking cards or something called server-on-a-stick (single high-bandwidth connection to a vlan-trunking capable switch and using the switch ports as a fan-out).

So in order to emulate a complex corporate network, all i have to do is configuring a lot of etherstubs, configuring many vnics, replicate the physical bandwidths with the maxbw setting on the vnics, set up a lot of zones, perhaps translate the ACL of the routers into firewall rules for firewall functionality of Solaris, installting the routing daemons and configure it similar to the configuration of the routers (in regard of timeouts and so on).

Now i can test, how my applications react, when the network starts to converge against a new topology because of the failures of some lines. I can test, to which topology my network will converge after an line outage (which is nothing more than a deny-all firewall rule). I can test the impact, when the network converges that way, that my traffic flows over a 2 MBit/s line instead of a 155 MBit/s line. For even more complex failure modes i can even use the htbx driver to introduce additional latencies, packet drop or packet reorderung as shown in this article. In essence you can emulate your complete internal network in a single box and with Zones and Crossbow in Solaris 11 it's so low overhead (at the end it is still just one kernel) that you can really emulate the reality and not a simplyfied view, as you don't have emulate via separate hardware or many independent operating system instances in virtual machines.

At the end you could simply use a single Solaris system, put it between all your test systems and use this solaris system as a emulation device for your corporate network. It's simulating the cloud-shaped icon in your architectural diagrams.

10 years of ZFS

nospam@example.com (Joerg Moellenkamp) — Wed, 02 Nov 2011 08:51:13 +0100

ZFS celebrated its 10th birthday on October 31st. So whatever you plan as a filesystem to kill ZFS ... may take a while

It's facepalm time ...

nospam@example.com (Joerg Moellenkamp) — Thu, 27 Oct 2011 14:04:07 +0200

Surely youve recognized that my blog was down for a few days and with it all services on the system. The problem that led to this situation was a really dumb one. Perhaps this article is more a story about not thinking about a failure mode just because its not a problem under your preferred operating system (Or to be exact: It was a problem before Solaris 10, but afterwards it was solved). And and most it's a story about being totally problem-blind in the first moment.

Perhaps I should explain first that c0t0d0s0.org isnt run with Solaris, it uses this-other-unixoid-operating-system in a well-known non-commercial variant. Thats the dirty secret of c0t0d0s0.org. No technical reason for it, but webserving and mail could be done by any operating system and thus I used that operating system with ubiquitous availability at almost all providers of dedicated servers. Im able to migrate the server from one dedicated server provider to another within 2 hours including moving the data and did this three times in the past (from 1&1 to Hetzner, and two times within Hetzner). This saved quite significant money until now and thats the basic reason why I dont want donations and when you do donations I would donate this money to kiva.org)

Hetzner has reasonably priced dedicated servers and I had no problems in the past, however they have one important shortfall: No serial console in the standard product. When you need a console, you have to make a support call and they connect one. As you need it really seldom, its okay. As I found out later: I With this serial console I would have recognized the problem within a minute, and fixed in a second. However: The console was exactly thing that I didnt had to my disposal at this moment.

So it was a lot harder to find out whats happened. However i wanted my server back as soon as possible (out of personal reasons I was just able to start the recovery in the evening and as I have job to do I could only do the further stuff in the evenings as well) and thus I just reimaged the server after keeping a copy of the logfiles. I have a quite extensive backup regimen with very regular rsyncs and database replication on my server at home thus I knew I would perhaps just lose an hour of minutes of data and that was okay for me, additionally I was able to mount the disks of the non-working installation and to copy the delta of mails between the last backup and the last mail in the queue to my backup.

What had happened: At 10:something my server provider had a large power outage. The UPS didnt take over as planed and thus a lot of servers rebooted. One of them was mine. Damned but thats the basic reason why Im a fan of proper enterprise architecture and not of some singular availability features, no matter what marketing tells you. Real availability is hard work and often expensive. But: When you really bet your business on IT, you need an architecture that is even capable to cover an UPS that proofs to be not so uninterruptible. The availability feature UPS may fail (and did fail my case) but a proper enterprise architecture keeps your service up and running. Even more important: With a proper enterprise architecture you dont need the feature UPS for availability reasons at all because your service can survive the outage of some parts. Perhaps you want the UPS out of other reasons like dont want the hassle of bringing up all the systems again.. But you dont need it with such an architecture to keep your business running. By having a proper planed enterprise architecture with servers on two seperate sides with different power grids you may forget about the UPS because a UPS wont help you with a prolonged power outage for example because of region-wide blackout. An outage that maybe will take out the connectivity as well as its not that unprobable that your local carrier has the same power problem However my business is being an architect, not blogging and thus I didnt have such an architecture in place for c0t0d0s0.org But thats a different story.

Okay: After a while my system worked again and thus I had time to find out what had happened. I knew that the system was still reacting on pings, thus I knew the kernel of this-other-unixoid-operating-system in a well-known non-commercial variant was working. Looking into the logfiles I saw complete bootup of the kernel and some of the daemons were starting up .... like acpid for example. However I couldnt log into SSH. No signs in the logfiles of a ssh daemon startup. The apache was in a half-reacting state. Port 80 was open but it didnt reacted to HTTP commands.

Out of this I concluded: The kernel and the boot configuration is okay. The bring up of the services is at least working partially, because otherwise it would start services at all. And as it reacted on the networking there must have been at least a working boot of services mandated by rcS.d, as otherwise there would be no networking. The problem must be in apache that is frozen halfway. And out of other reasons ssh isnt started at all. There must have been a major fsckup in the startup of the services As I had no console as explained before I needed to conclude from the leftovers what had happened.

And now was a little bit puzzled. 5-6 years ago I would have recognized this problem in an instance (because sometimes i've produced ... well ... suboptimal startup scripts) but now today it took a while, because I didnt felt prey for 5-6 years to such a problem. It took me a lot of more thoughts what might had happened. When you do one operating system for a living and one for hobby, you tend to project your mindset of one to the others and you dont do justice to this other OS.

As you all may know, Solaris ditched init.d with Solaris 10 in order to introduce SMF (not to forget the equally important features like the contract filesystem and the Fault Management Architecture). One of the nice advantages of SMF is that services that arent interdependent will be started in parallel without waiting for another. This has two advantages: At first the system can start up much faster, at second a service not able to start up cant block the startup of the rest (short of services needed by all others).

The init.d concept is a different. All services are started in sequence. The sequence is numerical and then alphabetical. That is of course slower but more important depending on the way you write your script a script or binary hanging or waiting for user interaction can block the startup. The variant of this-other-unixoid-operating-system is using init.d

And its quite easy to block a service. For example by integrating a new SSL key and certificate. My key had a password and apache was asking for this key in order to startup. This exacly happened. Acpi started up because it was started before Apache (guess what: ACpi is before APache in an alphabetical order, and way before Ssh). This is the basic reason why you strip of the password from your key. Guess what I did last week: I put a new key and certificate on my server and I forgot to strip the password from it.

And that exactly happened: In my version of this-other-unixoid-operating-system the ssh daemon is started after the apache daemon. When Apache waits for something you won't get SSH. Damned it's facepalm time. Basically I felt prey to a beginners error because Im working with an operating system that reacts totally differently on such situations. On Solaris such a situation just dont matter at all you get at least your ssh login and the system non-availability is just a service-non-availability you can fix within a second. However given the init.d system of this-other-unixoid-operating-system the outcome was somewhat more problematic. However: What a dumb error on my side ...

However: The reinstallation wasn't that bad ... the system could used a reinstallation because of some tests and experiments. So it was worth the work in the evenings. And on the other side: Who had the glorious idea to start apache before ssh?

That said, this-other-unixoid-operating-system in newer variants have a different startup mechanism up upstart all the services. However: My heating control is running on a beagleboard-XM at the moment using a really current variant of this-other-unixoid-operating-system just released a few days ago. It uses this new startup mechanism. And its justs my unimportant personal preference, that doesnt matter: But I dont like it. And I have a lot of reasons for it. It looks by far too much designed for desktop needs. However my dislike would require an article I wont write in this blog nowadays. But as I wrote: Thats my personal opinion that doesnt matter.

However its really important that this-other-unixoid-operating-system gets away from the old init.d mechanism to something more current. I think in 2011 every operating system deserves something more functional, something better than init.d init.d is simple and well understood, however it creates classes of problems unnecessary today. Especially: In order to keep die-hard Solaris admins to fall prey to such a beginners error because such problems were parts of their distant past. And now i will start to cut holes for my eyes in the brown paper bag for my head.

Nice example for the power of boot environments

nospam@example.com (Joerg Moellenkamp) — Tue, 04 Oct 2011 20:06:04 +0200

There is a nice example of the power of boot environment. Boot environments are something like snapshots of your operating system installation made writeable. As you may already assume, they are based on ZFS snapshots and the clone functionality. This is possible due to the usage of ZFS as the root filesystem.

So: Please don't try this at home. Whey you try it, don't try it on any Solaris 11 Express installation of any value. But don't try it. I don't want to hear any story. that you've deleted your ERP system by accident because you used the wrong terminal window. Leave that to trained professional stunt admins with the right equipment (Solaris 11 Express)

Assume you have a system, configured with all your application, everything is running fine. So you think it would be nice to have something like a freezed state of this situation. No problem. This command will do the trick.

# beadm create rescuenet
# init 6

When you reboot your system you will see it as a new entry in the grub menu.

Okay, but boot into the old environment starting "Oracle Solaris ..." first by selecting it in the grub menu (it should be already selected, or you used beadm activate already. Now i will drop the atomic bomb on your installation.

# rm --no-preserve-root -rf /

Essentially we've just nuked the installation. After a moment the system should just freeze. Reset the system and boot again via grub into the boot environment starting with "Oracle Solaris ...":

Okay ... on a normal system this would send you to the tapes. With Solaris 11: Reset the system. Boot into the boot environment "rescuenet" via selecting it in grub.

Tada! Just creating a boot environment with a single command after a config change may safe your butt later .... and btw ... this even works in zones ... they know the concepts of boot environment,too.

How to activate IPoIB Connected mode in Solaris 10 Update 9

nospam@example.com (Joerg Moellenkamp) — Mon, 03 Oct 2011 23:03:36 +0200

Just a short hint: The What's new document of Solaris 10 Update 9 states, that the support for IPoIB Connected Mode has been added in the release. However you have to search a bit in order for some information how to activate it. The necessary step is documented in the manpage for the ibd driver. Let's assume you have to instances of the ibd driver running (ibd0 and ibd1). In this case you have to change one line at the end of /kernel/drv/ibd.conf file to enable_rc=1,1; and reload the ibd driver respectively reboot the system. After that you ibd devices should show an mtu size of 65520 bytes instead of 2044.

PS: The process for Solaris 11 is better, as you just use dladm for it. However connected mode is the default there anyway. In Solaris 10 unreliable datagram was kept as the default, as one of the rules in Solaris is that you have to opt-in to such changes between updates.