JET Stream

Solaris Virtualization Epiphany

2009-05-16T02:31:34-07:00

I was giving my standard pitch covering Sun Virtualisation Technologies which essentially entails me talking at length about the following slide. I kind of make the point that it is a means not an end, and that really what you're trying to do is run more applications on the same piece of hardware, and the trick is to try and do it most efficiently.

I've done this talk countless times with a pretty wide variety of customers, and I tend to start talking from the right, and move to the left, explaining how the "layer of additional non app resource utilisation" grows from right to left. (not true for Dynamic System Domains on the far left though.)

It finally occurred to me yesterday that the Unix and Windows world have an utterly different view of Virtualisation, and the need for Virtual Machines.

In the Unix world, application co-existence is second nature to most sys-admins. Virtualisation is all about application isolation. i.e. You START with all the apps on a single O/S instance, and you apply more and more constraints: resource management --> containers/zones --> virtual machines --> dynamic system domains.

In the Windows world, application co-existence is so rare that it isn't even considered. Virtualisation is all about application consolidation. (and the primary way to do it is to create lots of virtual machines, one per app).

The problem is simply viewed from completely different angles.

It's not just an application co-existence issue either. There's a scalability problem. A lot of "other" O/S's can't scale to fit the larger more powerful Intel/AMD multi-core servers available today, and the ONLY way to make use of them is to carve them up into smaller machines that Windows or Linux can digest. Solaris, on the other hand can scale up to hundreds of CPUs, and hundreds of Gigabytes of memory.

From the slide above, Sun/Solaris provides a wide choice of technologies to allow multiple applications to be consolidated onto a single server. Don't let the limitations of "other" O/S's blinker you into thinking that a Virtual Machine is the only answer.

(I'm not saying that Virtual Machines is the WRONG answer btw, it's ONE of the answers, but it shouldn't be the only one that is considered.)

tags: virtualization containers ldom zones vmware xen

JET 4.7 Now available

2009-05-08T01:04:22-07:00

So, I've been working on JET 4.7 for a while now. A couple of weeks ago I released it internally to Sun, and no-one's complained about any showstopping bugs yet, so I've released a fresh bundle to the sun.com downloads site. The best way to go straight to it is to follow the link at the top of the wiki site.

JET 4.7 Changes:

The MAJOR JET change is that I've kind of merged JetNAS functionality into the main SUNWjet module. This isn't an exact replication of the JetNAS code, and it does things a little differently, but in short you can choose to place the NFS functionality somewhere different from your JET server.

Other JET changes are covered by bugids: 6835016, 685027, 685023, 685019, 685030.

In summary:

6835016: JET needs to support alternative servers for NFS media. The JET User Guide has already been updated to cover this functionality. Additionally, I've written a short article about it on the JET wiki site.

685027: JET jetjump.xml script does not stop graphical login from starting. Tidied it up. graphical login no longer starts until J/S is complete.

685023: JET should not create /dhcp directory. We used to do this to shorten the dhcp macro. This is no longer necessary, so we've removed the complication of having it.

685019: JET ksh scripts do not work when using OpenSolaris. There were a couple of constructs in our ksh code that ksh93 choked on. We've updated them so that they are compatible with ksh and ksh93. This allows all you OpenSolaris people to install JET on your OpenSolaris based Jumpstart server. (Note, AI is required to "jumpstart" OpenSolaris.)

685030: JET error message when suitable server is not found could be more meaningful. Well, this is just an attempt to reduce the e-mail traffic. Probably the first error, people new to JET see. We've updated the error message to tell you what to do to fix it.

JET 4.6 - 4.7 Change Summary:

For all you "external" people, the last externally available version was 4.6. Here are the changes from 4.6 - 4.7:

4.6.1: Ability to compress the rpool, and specify /var for zfs boot builds.

4.6.2: 6798648: JET needs to workaround Bug: 6528699: This is to stop Jumpstart from labelling ALL your disks and blowing away all your data. Also added a tweak to the ssh scripts to generate the keys properly.

4.6.3: Added an additional bootadm update-archive to the end of the finish.common script. We were getting some situations where the first reboot had a stale boot-archive, possibly due to a race condition in SPARC S10 Update 6 and above builds.

JET bundled Modules:

We've updated some/most of the bundled products. ALL of them (if required) have been updated to make them ksh93 compatible. Some of them have had more substantial changes:

JetLDOM

This module has had the most substantial rewrite. We've gone against ALL JET precedent and completely changed the structure of the ldom.conf file, so it's not backward compatible (for once). We needed to do that so that we could provide a much easier mechanism to make use of the ldm 1.3 functionality. Sorry.... but you'll like the new one better.

JetFLASH

I've updated JetFLASH so that is actually checks the flar (if it can), to see that it is the right architecture, and the right boot filesystem for the build. (Yes, zfs based flash is coming, but it is a different format to the ufs based one). Bruce added some code to clean up service tags if required.

JetZFS

Quite a few changes in this module. You can now update your rpool properties, I've provided more verbose progress messages (so you can see exactly what it is doing), added code to deal with converting EFI disks back to SMI (it does it for you if required), and finally added a strange hack so that you can optionally use ::: instead of = in your variables to get around a xVM OC gui restriction.

JetZONES

Fixed a bug where make_template was in the wrong place, and added some code to boot the zone before trying to add stuff to the rootpath/var dir. (because it might not be there unless the zone is booted). (Thanks to Mike Hendon for pointing it out). (plus a bunch of new functionality like support for branded zones)

Summary

So just a reminder, the following modules have NEW template variables. Old templates (other than the ldom based ones) will continue to work, but if you want to make use of the new functionality (with the new comments that describe them), you'll need to refresh your templates. (Have a scan through sample.template in /opt/SUNWjet/Templates to see what's changed.)

tags: Solaris JET Jumpstart

JET/Jumpstart without NFS

2009-04-15T06:52:40-07:00

I've been working on JET 4.7 (current released JET version is 4.6.3).

The main change is that I've now provided support to allow the Solaris Media and the /opt/SUNWjet directories to be hosted by a server (or servers) other than the JET server. This effectively removes the requirement for the JET server to provide NFS services.

Along the way I was able to tidy up and consolidate a lot of code, particularly the client_allocation_* methods. I've added a few functions, and used the variables more consistently throughout. For those of you doing DHCP/PXE based builds, I've removed the /dhcp hack that was used to overcome the 255 char limitiation of the DHCP macros.

Full details on the wiki site

I'm hoping to release it by the end of April.

tags: Solaris JET Jumpstart

FOLLOWUP: Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server

2009-01-06T02:57:55-08:00

A few months ago I wrote Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server, which detailed some of the hoops that I needed to jump through to get it to work. The good news is that the new features in VirtualBox 2.0 (and now 2.1) have gotten rid of 2 of those hoops:

I no longer need to play with vnics and that crazy script as Host Interface networking simply works out of the box
I don't need to install the 32bit internal only SC package, as VirtualBox now supports 64bit guests.

I'm in the process of rebuilding my laptop cluster making use of the above new features.

Hurrah!!!

tags: JET Jumpstart SunCluster Solaris VirtualBox xVM

JET 4.6 externally available

2009-01-05T02:03:52-08:00

As a follow-on to my previous post, JET 4.6 is now available externally. Check the JET wiki which has a link to the download.

This is mainly a bug fix release, and has been given a major number increase because it's the JET version that will be in the next release of xVM OpsCenter. (and we like to release the OpsCenter releases at major version numbers).

In terms of changes, I've fixed a couple of bugs with turning ntp on, and multiple disks in zfs pools for ZFS disks. This version also has tweaks in place to workaround some DHCP issues when installing S10 U6 and Solaris Nevada.

tags: Solaris JET Jumpstart

JET 4.4.7 available externally

2008-11-04T16:17:25-08:00

As a follow-on to my previous post, JET 4.4.7 is now available externally. Check the JET wiki which has a link to the download.

Also have a look at the JET User Guide which has now been posted on the wiki. I expect this to grow over time, but its a conversion from the original user guide with updates for correctness.

tags: Solaris JET Jumpstart

JET 4.4.7 released

2008-10-21T04:14:54-07:00

I've just released JET 4.4.7 internally. Bruce is in the process of putting together a new bundle including 4.4.7 for the external download, so it'll arrive in couple of weeks. If you need it NOW just ask nicely.

Changes

The latest version currently available externally is 4.4, and we've added a bunch of features since then. Here's a brief overview of what's new/changed/fixed:

4.4.1

6660631: Disabling UFS logging no longer works.
6673213: Selecting dhcp should automatically select grub if required.
6675467: JET needs to workaround Nevada installation bug 6633017.
6315828: JET "shutup_sendmail" variable does not work on Solaris 10.
6526561: Typo in make_client, missing r on JS_error line 130.

4.4.2

6656549: Add ability to do fdisk commands in the profile.

4.4.3

6694299: SUNWjet fails to install on system with super long group.
6690411: base_config_profile_fdisk needs to be ingored for sparc.
6698860: "rarp" service should not be enable by JET (on dhcp builds)

4.4.4

6706581: JET needs to support ZFS root installs. This is a pretty big one, with the inclusion of 5 new base_config variables. Might require a separate blog article to explain, but as usual, the comments in the template should get you on the way.

4.4.5

6735314: JET incorrectly sets NFSMAPID_DOMAIN to "dynamic".
6735318: JET should support noatime mount options.
6740625: JET client_environment should never output a warning message.

4.4.6

Never released.

4.4.7

6675467: (again, as we need to apply workaround for Sol10 U6 as well)
6761787: JET custom_files should allow the overwritten file to be backed up. (inspiration from Christopher Hubbell)

Summary of Changes

A lot of the changes are bug fixes, but a number of them begin to make use of some of the new Solaris features, or leverage new ways of doing things. The most important of these is the ability for JET to install ZFS boot Solaris. This applies to Nevada build 90 and above, as well as the imminently to be released Solaris 10 Update 6. Other things that are kinda cool is the ability for JET to fdisk your disk for you when doing an install, as well as set noatime mountpoint options (which more and more people seem tobe doing by default now.

tags: Solaris JET Jumpstart ZFS

belenix, opensuse and ubuntu on Nevada with VirtualBox

2008-09-24T10:43:54-07:00

I was in the process of playing with some stuff using VirtualBox, and it occurred to me that this was a pretty cool screenshot, so I did, and here it is.

tags: Solaris belenix ubuntu opensuse virtualbox

Using Sun Secure Global Desktop

2008-09-17T06:50:29-07:00

Over the past few weeks/months, I've been involved (sometimes peripherally) with setting up our demo kit in the UK Customer Briefing Centre. One of my design goals was to ensure seamless access to our demos from anywhere in the world or more realistically anywhere in the UK, and more specifically, at any customer site.

Within the CBC network (which is not connected to our internal Sun network) we set up a Sunray Server also running SGD. We initially had some hiccups getting the firewall traversal to work, and had to resort to using VNC which was adequate, but painful as we were tunnelling through ssh, so there were a bunch of steps required to ssh in, start vncserver, then re-ssh in to tunnel the right port, then start vncviewer and after all of that the screen refresh rate wasn't stellar.

However, once we got SGD working it was absolutely awesome.

If you want to have a play with how it works, simply go to Sun's SDG demo site . Unfortunately MyDesktop doesn't work (for obvious reasons) for anonymous users, but it will let you play with the webtop functionality.

For our purposes, we used the MyDesktop functionality. Running a desktop session inside the CBC from anywhere in the world was as easy as pointing a browser at the right place, logging in and clicking on the Desktop icon. Within seconds, as if by magic, a full screen desktop takes over your current screen (you can switch back to local using Alt-TAB). I'd show you a picture of it, but it just looks like a normal desktop and feels remarkably snappy even if it is remote.

The reason we were doing all of this is that we (The UK Systems Practice) have been running some "Discovery Days" around LDOMS and ZFS which have live demos of the technology. The cool part is that we get to showcase the SGD technology at the same time!

tags: Solaris SGD ZFS LDoms

Helford River

2008-09-17T04:29:01-07:00

So I got the boat down, and managed to go sailing in and around the Helford river in Cornwall. It was absolutely awesome. On the day the picture was taken, the wind was pretty light and so I decided to brave going out single handed. Even got the asymmetric out on the downwind leg.

Really Nice JET Tutorial

2008-08-21T05:26:42-07:00

Joerg Moellenkamp has written a pretty cool 16 part tutorial on Jumpstart using JET here. It's already linked at the JET wiki page.

tags: Solaris Jumpstart JET

No, I haven't stopped sailing

2008-08-17T14:05:31-07:00

I've not blogged about sailing in some time. Its not that I've not been Sailing. Its more due to me not blogging enough. I'm going to try and do a weekly recreational blog. (Watch me fail).

Since my last post, I've been helping out at my local Sailing Club (Burghfield Sailing Club)
with their Oppie Club. My oldest daughter got quite into it, and I got her an old wooden Oppie called Major Tom. Here's me sailing it.

As you can see, it's for kids, not adults. (And yes, I did eventually pick up the kid swimming behind the boat).

I've gotten a lot better sailing my Laser 2000, and have been trying to take part in the Wednesday Evening Pursuit races that take place during the Summer at the club. (On Wednesday evenings, funnily enough). I actually came first in one of them this year, and pretty happy with my progress.

Last week we were doing pretty well in quite windy conditions (I think we were 4th or 5th with 15 mins to go), when my main halyard broke, resulting in me having to DNF. The bigger problem was that we're taking the boat down to Cornwall on holiday next week, so I had to get it fixed pretty quickly. This weekend, I spent Saturday morning down at the Sailing club taking the mast down, and feeding a piece of wire up the inside of the mast, and rerunning the old main halyard. (This is harder than it sounds) The Chandlery at the Sailing Club was closed (the owners had a day at the races at Ascot or something) so I couldn't get a new halyard. Felt I might as well run the old rope, and once it's in, its easy enough to replace without having to take the mast down. On Sunday, today, they were open, so I was able to replace the old rope with a nice shiny new halyard.

It's also the first time I'll be towing the boat anywhere, so I also needed to get a couple of towing essentials for the trip. (Fortunately the family car has a towbar). I've decided to play it safe and get a new Jib Halyard as well, as it is also about 8 years old. A trip to the LaserDirect and £100 pounds later, I've got a bunch of stuff coming, hopefully by Wednesday this week, which will give me enough time to fit before setting off on the weekend.

It's also the first time that we'll be taking the L2K onto the open water. The place we've rented is in a sheltered cove, so we don't HAVE to go to sea, but it might be fun to go out for a laugh.

If this is my last ever blog post, you can guess what happened.

More on JET and iso images

2008-08-08T08:42:07-07:00

I've made some minor modifications to share_isos. I realised that add_install_client creates entries in /etc/dfs/dfstab for the "boot" filesystem if not already mounted. Unfortunately, it only checks the dfstab, rather than checking the actual shares. The result is that I need to populate the /etc/dfs/dfstab file.

The complication is that now that it's in dfs, the stubs get shared when the server boots, so I need to unshare them before doing the lofi mounts.

I've updated the share_isos script to do that. I've decided I'm simply going to make it an add_on JET module that can be installed if that feature is required/desired. (Look out for a JetISO module any day soon).

Finally, while testing all of this I decided to so something pretty obscure. I'm in the process of building a JET.vdi and I'm trying to make the .vdi as small as possible. (A virtual disk image of the smallest Solaris I can use, with JET and its modules installed so that it is nice and easy to distribute (so far its about 800k). I obviously don't want to prepopulate with any Solaris media as that will simply make my vdi too large. The cool thing about iso images is that I can NFS mount a directory containing them, and then I can lofiadm and mount them locally.

So I ended up with a JET server running in a virtualbox, NFS mounting a Sol 10 0508 iso from my home server, lofimounted and shared with share_isos, and I used that to jumpstart another Solaris virtualbox machine on the same PC (which was running XP). The really good news was that it worked. The not so good news is that it took over a day to install. I think there are a few too many levels of indirection and virtualisation between the iso and the virtual disk.

I should really do a bit more testing and figure out the bottleneck. Ideally I'd prefer to keep the isos on the host and use shared folders to allow the virtual JET server to access them, but that won't be possible if the performance is as bad as my original test.

tags: Solaris Jumpstart JET virtualbox

JET server using Nevada and iso images

2008-08-07T07:58:05-07:00

Things have been slightly problematic on the JET server running on Nevada front mainly due to 2 issues.

There is/was a bug in cpio (See BugID 6686818) which meant that setup_install_server didn't copy the media correctly.
setup_install_server on earlier Solaris releases didn't understand zfs, therefore it was difficult to copy onto a zfs filesystem.

Why not just use iso images?

Why not indeed? I'm in the habit fo simply downloading and keeping the .iso images for all releases, and I used to then "expand" it by lofimounting and then running setup_install_server. I realised I really didn't need to do that, I could just use the lofi mounted image, and get away from having to ever run setup_install_server, and not need to use up double the space per O/S image I wanted to keep.

So.. what was missing?

I needed a way to lofiadm, mount and nfs share the Solaris media. Also because x86 builds also lofi mount the boot directory and mount it into the /tftpboot directory, I'd also need to lofimount a lofimounted filesystem as well.

Firstly I did everything manually to ensure that everything works fine, then I wrote a script, which reads a config file (which lives in /opt/SUNWjet/etc).

Inline here:

#!/usr/bin/ksh -p
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
################################################################################

# sanitize PATH, and ensure required components are in front
PATH=/sbin:/usr/sbin:/bin:/usr/bin:${PATH}
export PATH

BINDIR="`cd ${0%/*}; pwd`"
ETCDIR="`cd ${BINDIR}/../etc; pwd`"

PATH=${BINDIR}:${PATH}
export PATH

MEDIALOC=${ETCDIR}/solaris_iso_locations

if [ "$#" != "0" ]; then
	echo "Usage: $0"
	echo " "
	exit 1
fi

#
# Check to see if the media locations file exists.
#

for iso in `cat ${MEDIALOC} | grep -v "^#" | awk '{print $1}'`
do
	lofiname="`lofiadm | awk '$2 == iso {print $1;}' iso=$iso`"
	if [ -z  "${lofiname}" ]; then
		lofiname="`lofiadm -a $iso`"
		echo "Added $iso as lofi filesystem: $lofiname"
	fi
	mp="`cat ${MEDIALOC} | awk '$1 == iso {print $2;}' iso=$iso`"
	mounted="`df -k | awk '$1 == lofiname && $6 == mp {print $6;}' lofiname=
$lofiname mp=$mp`"
	if [ -z "${mounted}" ]; then
		mount -F hsfs $lofiname $mp
		mounted=$mp
		echo "Mounted $lofiname at $mp"
	fi
 	shared="`share | awk '$2 == mp {print $2;}' mp=$mp`"
	if [ -z "${shared}" ]; then
		share -o ro,anon=0 $mounted
		echo "Shared $mounted"
	fi
	# Deal with lofi mount in x86
	mountboot="`cat /etc/vfstab | awk '$1 == mpboot {print $3;}' mpboot=\"$m
p/boot\"`"
	if [ -n "${mountboot}" ]; then
	bootmounted="`df -k | awk '$1 == mpboot {print $6;}' mpboot=\"$mp/boot\"
`"
		if [ -z "${bootmounted}" ]; then
			mount $mountboot
			echo "Remounted $mountboot"
		fi
	fi
done

It parses the solaris_iso_locations file which looks like this (I've attached my filled out one)

# This file is used to specify the mountpoints for iso images. You need # to create the mountpoint, update this file, and then run /opt/SUNWjet/bin/ # share_isos. This will lofimount all isos listed here. You will then # need to run add_solaris_location for each image as normal. # You should run the share_isos command everytime the JET server is # rebooted. /backup/Solimages/s10sparc_0508.iso /export/install/iso/Solaris_10_0508s /backup/Solimages/s10x86_0508.iso /export/install/iso/Solaris_10_0508x /backup/Solimages/solarisdvdb90x.iso /export/install/iso/Solaris_11_b90x /backup/Solimages/solarisdvdb90s.iso /export/install/iso/Solaris_11_b90s

What it does.

In short, it reads the config file and mounts (if not already mounted) the iso images on the path specified. It will then NFS share the mountpoint, and parse the /etc/vfstab and do the /tftpboot lofi mounts if necessary. (NOTE: add_install_client creates those entries in /etc/vfstab the first time it is run for that O/S. You MUST edit the /etc/vfstab and change the mount_at_boot flag to "no", otherwise your booting your JET server will fail.)

Running share_isos produces the following:

# /opt/SUNWjet/bin/share_isos
Mounted /dev/lofi/2 at /export/install/iso/Solaris_10_0508s
Shared /export/install/iso/Solaris_10_0508s
Mounted /dev/lofi/3 at /export/install/iso/Solaris_10_0508x
Shared /export/install/iso/Solaris_10_0508x
Remounted /tftpboot/I86PC.Solaris_10-1
Mounted /dev/lofi/4 at /export/install/iso/Solaris_11_b90x
Shared /export/install/iso/Solaris_11_b90x
Remounted /tftpboot/I86PC.Solaris_11-6
Mounted /dev/lofi/5 at /export/install/iso/Solaris_11_b90s
Shared /export/install/iso/Solaris_11_b90s
# /opt/SUNWjet/bin/share_isos
#

Note: if you run share_isos again, it does nothing as everything is already mounted and shared, so its easy to re-run it whenever you want just to confirm it's all good. (I considered having JET call it automatically, but I'm not completely sure how I want to do that yet. Well thought out suggestions welcomed.)

The bottom line is that I can now easily use my laptop running Nevada and a ZFS filesystem and use it as a JET server for any version of Solaris without ever needing to run "setup_install_server". And the bonus is I've now got about 12GB of space back!!

tags: Solaris Jumpstart JET

Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server

2008-08-05T05:15:04-07:00

I was intrigued by Gordon Johnston's blog article about creating a 2 node Cluster Express cluster using VirtualBox, and I decided I had to try it.

Here's the final result:

(You can click on the image for a fullsized picture.)

It shows a VNC connection to my laptop which has 4 windows up: The 2 VirtualBox instances showing cxnode1 and cxnode2, a terminal window on the laptop itself, showing the quorum configuration and the iSCSI share, and in the background, the VirtualBox console showing the network configuration of cxnode1. Finally, in the main screen, I'm running Sun Cluster Manager and connected to the cluster, showing the topology and the Resource Group configuration of apache.

Rather than simply duplicating what Gordon did, I decided to go with Solaris Cluster 3.2 U1 as my cluster software. This meant that I had to use an internally available package of 32 bit kernel drivers, as SC 3.2 is shipped as a 64bit only product. (Cluster Express has both 32 and 64 bit variants.) The reason for going with Solaris Cluster is that there are instances where I want to duplicate a scenario that a customer is seeing, and in most cases, that's on Solaris Cluster.

I'm hesitant to duplicate Gordon's article, as I pretty much followed his guide, so I urge you to read his article as it is quite detailed, I'll only highlight where I changed things.

I only really wanted a working cluster, went for the simpler set up of HA Apache
Cluster node O/Ss would be Solaris 10 05/08
Host O/S is Solaris Nevada Build 93 (as that was what was already running on my laptop)
I wanted to Jumpstart the nodes using JET.

Preparing your physical host

I already had VirtualBox 1.6.2 installed, so there wasn't much to do there. The main trick was configuring my laptop to allow Host Interface networking. This bit is Solaris specific. You basically need to do 2 things.

Firstly, you need to set up a bunch of virtual nics, and assign MAC addresses to them. As this needs to be redone on every reboot, I followed Dave Tong's advice and set up an rc3.d script. (and no, I didn't bother to use SMF either, you can bite me too if you want.)

#!/sbin/sh
state="$1"
case "${state}" in
'start')
    echo 'Plumbing VNICs'
    for MAC in 10 20 30 40
    do
        VNIC=`/usr/lib/vna nge0 c0:ff:ee:0:0:$MAC`
        ifconfig $VNIC plumb
    done
    ;;
'stop')
    ;;
*)
   echo "Usage: $0 { start | stop }"
    exit 1
    ;;
esac

(I did use Gordon's cute fake MAC address c0:ff:ee 'cos its easy to remember)

Secondly, you REALLY REALLY need to make sure that you give the VBoxSVC privileges to configure the interface, otherwis your virtual machines will fail when starting. Whenever you start VirtualBox, you need to get the PID of the VBoxSVC, and run: ppriv -s AEI+net_rawaccess <pid>

Creating your Virtual hosts

Basically I followed Gordon's instructions for this, with TWO exceptions. I needed to select Network in the Boot Order menu in Settings-->General-->Advanced as PXE is not enabled by default. I also found that I couldn't do a PXE boot if I used the Intel driver, so I used the PC-net FAST III driver throughout which worked fine.

Installing Solaris on the Virtual hosts

I've already got JET installed on my laptop, so preparing to install Solaris was an amazingly simple process:

/opt/SUNWjet/bin/make_template cxnode1 base_config
Edit the template
/opt/SUNWjet/bin/make_client cxnode1
Start the cxnode1 Virtual Machine, Press F12, Select Lan Boot, and watch it install.
/opt/SUNWjet/bin/make_template -T cxnode1 cxnode2
Edit the cxnode2 template and change 3 variables.
/opt/SUNWjet/bin/make_client cxnode2
Start the cxnode2 Virtual Machine. Press F12, Select Lan Boot and watch it install.

Of course, the devil is in the detail, I'd already configured the Solaris media, and these are the changes I made to a default template for the build to work:

base_config_ClientArch=i86pc
*base_config_ClientEther=c0:ff:ee:0:0:10
base_config_ClientOS=10_0508x
base_config_client_allocation="dhcp"
*base_config_sysidcfg_ip_address=192.168.1.201
base_config_sysidcfg_netmask=255.255.255.0
base_config_profile_fdisk="alldisk"
base_config_profile_cluster=SUNWCXall
base_config_profile_root=free 
base_config_profile_swap=2048
base_config_profile_s6_mtpt="/globaldevices"
base_config_profile_s6_
base_config_ipmp_networkifs="pcn0_pcn1"
*base_config_ipmp_networkif_pcn0_pcn1="prod sp 192.168.1.211 192.168.1.212 24 cxnode1 192.168.1.201"

(the ones with * are the variables I had to update when I created the cxnode2 template)

Installing Solaris Cluster on the nodes

As the nodes were running in 32bit mode, (VirtualBox 1.7 should have 64bit support, which means this part won't be necessary in the future) I needed to use an internally available set of 32 bit kernel drivers in a packages called SUNWscka. Basically, install sc3.2 as usual using the installer, but don't configure anything, and don't configure a quorum device. Then pkgadd SUNWscka on both nodes, and reboot them both.

The rest

After that, I just followed Gordon's instructions for setting up the Quorum server, setting up the iSCSI devices on the hosts, mounting them on the cluster nodes, and creating a zfs pool using them.

Sadly, this now means I can play with and configure all sorts of things in a Sun Cluster environment from the comfort of my laptop!

tags: Solaris Jumpstart JET xvm virtualbox

Solaris Bootdisk Layout discussion

2007-09-25T09:12:21-07:00

There's been a vibrant discussion on the sysadmin-discuss(at)opensolaris.org alias about boot disk layouts, and it was suggested that we try to come to some form of consensus and document it.

Sun has a bunch of installation standards called EIS (Enterprise Installation Standards), and one of those standards is the EIS Bootdisk Standard. As one of the contributors to that document, I found it quite interesting to see that the sysadmin discussion was coming to similar conclusions about how a boot disk should be laid out as we had come to internally.

In the interests of sharing and getting to a common standard, I've now posted the relevant bits of the EIS standard on the BigAdmin wiki, and encouraged the members of the sysadmin-discuss alias to comment and contribute.

If you're interested, you should have a look at it, and join the alias to participate in the discussion.

tags: Solaris Bigadmin

CEC 2007 Here we come (again)

2007-09-03T03:34:22-07:00

(I've reposted so that technorati picks up the tag we'll be using for this event)

(Hint, tag using suncec2007 and we'll be able to agregate all content using said tag)

Its that time of year again. CEC2007 will be in Las Vegas this year. I've never been, and I'm half looking forward to it, but as I think I have an addictive character, I'm a little scared of of Vegas.

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables.

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

The theme this year is "Shift Our Universe, Our World, Your Move", sub themes being: Shift to red: (all the groovy bleeding edge stuff that GregP talks about, Shift to green: (all the nice eco-friendly stuff we've been doing), and Shift to grow: (which focusses on all the things we need to do to enable our growth (and our customer's as well))

We're also going to be giving out a bunch of prizes for participants and attendees of the conference. They'll be mainly based on level and quality of participation in the sessions. Stay tuned for more details.

As per last year, I'll be blogging my perspectives of the conference again.

tags: Sun suncec2007

Coolstack: Not just for Coolthreads

2007-08-10T08:34:32-07:00

As per a previous article, I upgraded by home server to Nevada build 69, because I was about to make some changes, and wanted to be fully up to date before I did so.

I wanted to play with Confluence which is the wiki that Sun has chosen to run their wiki site. Confluence can run in a number of ways and connect to a number of different databases. Since I hadn't yet played with Postgres, I decided I would use that, and I also decided to run it using Tomcat instead of as a standalone app.

Given that I'm a bit of a neophyte when it comes to talking native SQL, I decided that I needed to run a tool of some sort to configure Postgres. I've used phpMyAdmin for MySQL for a while now, and figured there MUST be an equivalent for Postgres. Using my amazing powers of deduction, I googled phppgadmin, and lo and behold there it was: phpPgAdmin. All I needed was a php compatible webserver. Unfortunately the supplied apache2 with Nevada isn't php ready, so I had to then decide.. do I compile myself, or do I find something easier. I vaguely remember hearing something about coolstack, which is "a collection of some of the most commonly used open source applications optimized for the Sun Solaris OS platform."

CoolStack 1.1 is available for both SPARC and x64, and contains pretty much all I needed, including tomcat. So a few downloads and pkgadds later, and I've got apache running, which means I can get phpPgAdmin running, which means I can perform my initial Postgres configuration, which means I can finally install Confluence under Tomcat, and everything's perfect. (Well it wasn't perfectly perfect, because I needed to read a couple of READMEs along the way, and a bit of on-line documentation, but it was fine.)

So, I've got tomcat listening on port 8081 as per normal and confluence sitting inside there as an applet talking happily to Postgres, and so I decide that I actually want to do the apache proxy thing so that I can have my confluence wiki appear to sit on my normal webserver port. This is quite easy to configure, you just need a few lines like this:

LoadModule proxy_module libexec/mod_proxy.so
LoadModule proxy_connect_module libexec/mod_proxy_connect.so
LoadModule proxy_ftp_module libexec/mod_proxy_ftp.so
LoadModule proxy_http_module libexec/mod_proxy_http.so
LoadModule proxy_ajp_module libexec/mod_proxy_ajp.so
LoadModule proxy_balancer_module libexec/mod_proxy_balancer.so
ProxyPass               /wiki           http://localhost:8081/wiki
ProxyPassReverse        /wiki           http://localhost:8081/wiki

(I don't actually know how many of the LoadModules I really need btw, I suspect I don't need the last 2)

Here comes the bad news. The apache2 that comes with CoolStack doesn't come with the mod_proxy.so modules, which means that I can't use it for proxying. I tried copying the original mod_proxy.so modules to the coolstack one and loading it up, but when I tried to access the wiki, my system pretty much froze and the disks churned like mad. Had to kill the webserver to get things back to normal.

How did I fix? I reverted to the original Nevada supplied webserver, and configured the CoolStack based one to run on a different port which I can use when I want to use phpPgAdmin. At some point I'm going to have to solve the problem permanently, probably some further investigation, and trying to figure out exactly what's going wrong. But for now, its doing what I need, I just hope I don't want to run a php based app anytime soon.

tags: Sun Solaris CoolStack AMP

CEC2007: Here we come

2007-08-10T06:34:16-07:00

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables.

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

As per last year, I'll be blogging my perspectives of the conference again.

tags: Sun suncec2007

New JET Resource

2007-08-10T06:15:47-07:00

Sun has just launched http://wikis.sun.com, which is great because it allows me to use it for putting all the JET info there rather than here.

I have created a space there called JET, which will become the primary Sun location for all information relating to the Jumpstart Enterprise Toolkit.

Have a look: Jet wiki

tags: Solaris Jumpstart JET

Home Server tweaking

2007-08-01T03:47:23-07:00

I've been working on my home server again (remember.. the Ultra40 I won at CEC last year) . I've been lazy for the past few months, and been stuck at Nevada Build 63 for a while. While I've been live-upgrading my laptop with every Nevada release, I kept on finding really good excuses to NOT upgrade my home server.

The main issue is that it actually does real work. It's my home DHCP, SMTP, Samba, Web, Jumpstart, (and JET development) and print server. The household tends to notice when it's down. (especially since all the home directories are mounted off it, which means that everything stops when its down. We get real tears in our house when people can't do their homework!)

The fact that I use Live Upgrade means that downtime is usually just a reboot. Chris Gerhard, who has a similar home set up upgraded before me and found a samba problem, which caused me to delay upgrading until I knew it had been fixed, then we had a minor problem with gnome-terminal character corruption, finally, gaim was replaced by pidgin, and in doing so, support for one of the IM protocols was kind of accidentally broken.

I don't have a problem being on the bleeding edge on my laptop, cos I can work around most of the issues, especially since I do most of my work on my home server.

Anyway, the long and short of it was that I was just about to do some other work on my Ultra40, namely try and get apache with php working (among other things), and decided it would be prudent to upgrade to Nevada Build 69 before I did anything.... just to be sure.

Happy to report that it all went swimmingly (as usual), and downtime was in fact reduced to a single reboot, which them indoors didn't even notice.

One interesting point. On rebooting and starting gnome, it automatically detected my USB plugged in HP printer, and gave me a pretty little "Add Printer Queue" window which allowed me to add my printer. Unfortunately it didn't detect that I had already added it manually many months previously. I suppose one can expect that to happen as new features begin to overlap manual workarounds. I'm pretty sure that printing should just work straight out of the box on a fresh Build 69 install without any manual intervention.. but I'm not going to bother to try.

I then proceeded to get my AMP working... topic for my next article.

tags: Solaris SUN X86 LiveUpgrade AMP

Do you Tiddlywikle?

2007-05-31T02:47:14-07:00

Bruce Porter just sent me a link to something called TiddlyWiki.

It's author JeremyRuston, describes it as: a reusable non-linear personal web notebook.

In short its a wiki on an html page. You don't need ay server-side logic, just a web browser.

And the way it works is pretty cool too!

Printing on Nevada b56

2007-01-29T14:16:12-08:00

I got asked in a comment in my previous article to describe EXACTLY how I got printing working.

I've got an HP5150 USB printer. I plugged it into a convenient USB port on my Solaris server, and then used the printmgr gui to set it up. I selected Use PPD, and in the Printer-->New Attached Printer form, filled in the following:

Printer Name: unix
Printer Port: /dev/printers/0
Printer Make: HP
Printer Model: HP DeskJet 5150
Printer Driver: Foomatic/hpijs (recommended)

This allowed standard Unix printing to work. (In short, it pretends to be a standard postscript printer, and you can send pretty much anything to it)

Solaris - Solaris

I also turned on the ipp-listener service using svcadm. On a remote Solaris client (my laptop, also running Solaris Nevada b56), I used printmgr again, and did Printer-->Add Access to Printer. This uses the ipp service by default if available. lpstat -v shows:

# lpstat -v
system for unix: zaphod (as ipp://zaphod/printers/unix)

The printer name is "unix" and my server is called "zaphod". With that, I could print from my laptop to my printer attached to my home server.

Samba Printing

The Samba printing was a little more troublesome. In short, all the Windows machines, and any machine for that matter trying to print natively to my "unix" queue had difficulties because it was expecting either Postscript or plain text. I could have set all my printers up as generic Postscript printers, but decided I'd rather print natively. To do this I could have probably played with the filters so that it passed through everything, but that seemed too much like hard work. Instead a set up a different queue (to the same printer) which would act as a "raw" queue.

#!/bin/sh
lpadmin -p hp5150 -v /dev/printers/0 -T unknown -I any
accept hp5150
enable hp5150
lpadmin -p hp5150 -o nobanner

I think my smb.conf was already pretty much properly configured for printing, but I'll repeat the relevent bits just in case:

# If you want to automatically load your printer list rather
# than setting them up individually then you'll need this
   load printers = yes

# NOTE: If you have a BSD-style print system there is no need to 
# specifically define each individual printer
[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
# Set public = yes to allow user 'guest account' to print
   guest ok = no
   writable = no
   printable = yes

To set up the printer on the local side, I had to manually install the HP5150 drivers, and then add the printer using the Add Printer wizard in Windows. The key to getting this to work properly, is to set it up as a "Local Printer", then select a Local Port, and specify the Local port as (in my case) as: \\zaphod\hp5150

Summary

I actually configured all of this when both my home server and my laptop were running Nevada B54, so it is possible that it is no longer necessary to set up 2 queues, but I've LiveUpgraded and the settings were retained, so I've had no need to change. A lot of things seem to have changed in the Solaris printing model in Solaris Nevada, but I've found it is now a lot easier than it ever was. (And it is the expectation that it is going to be really difficult that makes you do more than you need to). If you trust that it just works, and use the printmgr tool, I think it is difficult to go wrong.

tags: Solaris SUN X86 samba printing

It just works!: Zones + ZFS + BrandZ + LiveUpgrade

2007-01-26T11:06:33-08:00

I've successfully LiveUpgraded my home server twice so far. (I've been slightly lazy and decided to do alternate builds, so I've gone from Nevada b52 to b54 to b56. Apart from the minor zfs issues which require me to:

Create an exclude file for the lucreate so that it doesn't try to copy my zfs partitions.
Manually empty mountpoints where some of my zfs filesystems mount.

it all works really well.

I'd shied away from creating zones because LiveUpgrade didn't support zones in the earlier builds, but I've now been assured that it works fine now.

Zones + ZFS. WOW

It just works. When you create a zone, if it detects that your zonepath is on a zfs filesystem, it'll automatically create a new zfs dataset for you for the zones you are creating. You don't need to do anything special, it just does the "Right Thing(tm)".

Furthermore, if you decide to clone a zone, and it's on a zfs filesystem, it simply creates a snapshot of the cloned zone for you. You don't need to do anything special, it just does the "Right Thing(tm)".

This means that once you've created your first zone, all your additional zones can be created and booted in a matter of seconds. Talk about rapid provisioning!

The icing on the cake: BrandZ

I've simply followed the instructions here and I've now got CentOS running in a zone, running on top of Solaris. Got Skype working!!

I love it when a plan comes together. We've now got some really clever integration of some individual cool Solaris technologies. The combined benefit is compelling.

Just waiting for ZFS boot to become available (well, more available than it is), and my home server aspirations will be complete.

tags: Solaris SUN X86 ZFS BrandZ

Addendum: Building/Installing/Configuring....

2006-11-22T06:55:12-08:00

I've been checking my spamassassin scores, and wasn't particularly happy with it's ability to rate SPAM. Given it's reputation I was pretty sure that it was due to a configuration error on my part.

I had a sneeky suspicion that not all the "tests" I was running were actually happening. I ran spamd in debug mode, and started seeing a bunch of interesting errors. In short, I was missing Net::DNS, so anything that required any sort of DNS lookup was failing.

Simply did the perl -MCPAN -e 'install Net::DNS' thing and that got DNS lookups working.

Then started to debug the other errors in the log. Apparently the bayes plugin was having problems talking to my bayes files because I didn't have DB_File. Tried the CPAN trick again, which failed because it couldn't find db.h. Turns out I didn't have Berkeley DB installed. Short trip to sunfreeware.com later:

db-4.2.52.NC-sol10-intel-local

It installs in /usr/localBerkeleyDB.4.2 . Given that CPAN was looking in /usr/local/BerkeleyDB, I created the link, and hey presto perl -MCPAN -e 'install DB_File' worked fine.

At that point I ran sa-learn against my Junk folder to generate some content, and checked the permissions of the bayes files. (Note: bayes_path in the local.cf for spamassassin needs to contain the prefix of the bayes files. e.g.:

use_bayes 1
bayes_path /export/home/spamd/bayes/bayes
bayes_file_mode 0666

# ls /export/home/spamd/bayes/
bayes_journal  bayes_seen     bayes_toks     bayes.mutex

Now that its all sorted, I'm getting much higher SPAM scores, and it seems much more accurate.

Some would say I should have just used Blastwave, but this was more fun, and a good learning experience.

tags: Solaris SUN X86 spamassassin

BTConnect and Dynamic IP SMTP server problems.

2006-11-21T09:54:59-08:00

So, I've got Exim working really nicely... I think.

I've got a "dynamic" IP address from my BTConnect business broadband connection. This means that I have trouble connecting directly to SMTP servers from my server because their reverse lookup of my name doesn't match... i.e. it resolves to a dynamic BT address, and some SMTP servers are rightfully suspicious of that and punt me away.

So I've configured Exim to relay to mail.btconnect.com. This works because I've registered my 2 domain names with them, and their mail relay will accept stuff coming from those domains.

However, I also forward e-mail for my sister, and have a couple of little mailing lists configured, and these get bounced because I retain the name of the original sender on my outgoing forwards.

Hmmm. This should be solvable by authenticating myself on BT's mail server.

Trawl of the BT Broadband site tells me that:

1: BT Connect, (the business one) doesn't allow SMTP authentication:

 # telnet mail.btconnect.com 25
Trying 194.73.73.217...
Connected to mail.btconnect.com.
Escape character is '^]'.
220 C2bthomr06.btconnect.com ESMTP Mirapoint 3.7.4b-GA; Tue, 21 Nov 2006 17:50:20 GMT
ehlo <mydomain>
250-C2bthomr06.btconnect.com Hello host<nnn>.btcentralplus.com [xx.yy.zz.aa], pleased to meet you
250-8BITMIME
250-SIZE 52428800
250-DSN
250-ETRN
250 HELP

2: BTInternet, (the consumer one) does:

# telnet mail.btinternet.com 25
Trying 217.146.188.192...
Connected to pop-smtp1-f.bt.mail.vip.ird.yahoo.com.
Escape character is '^]'.
220 smtp808.mail.ird.yahoo.com ESMTP
ehlo <mydomain>
250-smtp808.mail.ird.yahoo.com
250-AUTH LOGIN PLAIN XYMCOOKIE
250-PIPELINING
250 8BITMIME

Phone call to the BTConnect helpline. They understood my problem. Were slightly sympathetic, but confirmed that the only way to send none BT domain e-mail via their BTConnect server was by pre-registering EVERY domain with them. Clearly impractical. (Heck, I don't know who's going to send my sister an e-mail)

Phone call to the BTInternet helpline. I used to have an account. It was suspended due to lack of use. Got it re-instated. But they said I had to dial up at least once every 6 months to keep it going. I don't have a dialer at home, I'm bound to forget, and I'm not crazy about plain text authentication. But I reconfigured Exim to use mail.btinternet.com as my smart relay, and started authenticating. Thankfully, my sister's e-mail stopped bouncing at that point.

I wasn't particularly happy with the solution. I had a couple of long term fixes in mind:

Convert to Static IP from BT, which means I could register my name in the rev-arpa tables. Costs money. (Though they should really do it for free, my "dynamic" IP address has not changed in about 3 months.)
Sign up for BTInternet's premium e-mail solution @ 1.50 a month, which means I could continue to use the mail.btinternet.com mailserver. Costs money, and not happy with plaintext auth.
Something Else.

I ended up doing something else. I had a rant about this to one of my friends who happens to be my secondary MX (for when I break my machine), and he chuckled and reminded me that when he set the MX up, including my account on his server he also enabled encrypted SMTP authentication for me. All I needed to to is point at his server in the first place. (Which I am now doing)

The only other thing I want to do is set Exim up so that outgoing mail from my domain sent to mail.btconnect.com, otherwise to use my friend's mailserver. I know it's possible, I just need to write the rules. Here they are:

# Use this one for mail originating from my domain so I don't overload smart_route2
smart_route1:
  driver = manualroute
  domains = !+local_domains
  senders = *@<mydomain>
  transport = remote_smtp
  route_list = * mail.btconnect.com

# Use this one for mail NOT originating from my domain. (i.e. mailing list/alias expansion)
smart_route2:
  driver = manualroute
  domains = !+local_domains
  transport = remote_smtp
  route_list = * <my friend's smtp server>

(obviously <mydomain> is my actual domain)
Exim is COOL.

tags: Solaris SUN X86 SMTP Exim

VNCviewer trick

2006-11-17T08:04:00-08:00

I've been using "VNC Viewer Free Edition 4.1.1 for X" so that I can remote control other machines in the house, but if the machine I try to connect to is down/unavailable/switched off etc, the vncviewer command kind of hangs and I can't do anything on my desktop (except wiggle my mouse) until it times out after a few minutes. Needless to say, this was annoying me. Wrote a simple script called vncsafe:

#!/bin/sh
ping $1 >/dev/null 2>&1
if [ $? = 0 ]; then
        vncviewer $1 &
else
        zenity --error --text="$1 is not responding."
fi

Works a treat

tags: Solaris SUN X86 VNC

I ate all the SPAM

2006-11-16T14:04:44-08:00

I was chatting to Chris about my spamassassin setup, and he was saying that he's configured his with .forward files for all his users (family), and explained the reason for doing so.

Basically, he wanted to set a low SPAM threshold, and have all the SPAM delivered to himself so he could check it. Given that I have 4 daughters under the age of 12, I thought that would be a GOOD THING(tm) as well.

I also wanted to configure it so that I could vary the level of SPAM detection based on the user.

I created a .forward file in each user's account that looks like this:

# Exim filter

if
        $header_X-Spam-Score: does not begin -
then
        if $header_X-Spam-Score: is above 10
        then
                deliver <to me>
        endif
endif

I had to change the exim configuration to write the X-Spam-Score calculated by spamassassin in the acl_check_data filter to $spam_score_int because the user filter stuff in exim doesn't like decimals.

Finally, I wrote a filter in Thunderbird which puts all mail that's in my inbox that's NOT addressed to me in a folder called SPAM. That means I can check for false positives, forward the e-mails to my offspring if required, and add a rule to the whitelist to allow in the future.

The .forward file allows me to specify varying limits of SPAM checking to different people. e.g. the 7 year old twins have a setting of 5 (i.e. .5), while my wife has a setting of 1.

Need to let this run for a while so I can tweak between too much or too little as required.

tags: Solaris SUN X86 spamassassin

Building/Installing/Configuring apps on Solaris Home server

2006-11-16T04:26:14-08:00

I've diverted slightly from Chris Gerhard's home server implementation by deciding to NOT use the blastwave packages, and instead either use the sunfreeware stuff, or compile it myself.

I've got a few reasons for doing it this way:

Many of these packages have compile time options which allow me to include/exclude things. I kind of like the ability to be able to do that, rather than having to stick with the "defaults" compiled by someone else.
Using blastwave (which is really great) means that you need to install a bunch of "dependent" libraries. It just kind of irks me that a lot of the libraries/applications dependencies that I install are already sitting in /usr/sfw. I then start getting really confused, and run into complications about LD_LIBRARY_PATHs and making sure things are seeing the right library etc. I think it's difficult to use blastwave without sort of subscribing to it wholesale. It's a bit like buying a "kit car" fully assembled.
The sunfreeware stuff tries to make use of existing Solaris libraries as much as possible, so should reduce the number of additional packages I had to download.

Additional Software Installed on my home server:

From Blastwave:

xineui (and whatever it needed to get that working.. about 81MB)

From Sunfreeware:

imap-2004g-sol10-x86-local (had manually update /etc/inetd.conf and run inetconv, then update /etc/services. Also need to generate a certificate as this is the SSL version, so wouldn't let me do plain text authentication.)
openssl-0.9.8d-sol10-x86-local
libiconv-1.9.2-sol10-x86-local

From CPAN:

spamassissin

    (# perl -MCPAN -e 'install Mail::SpamAssassin' , download the dependents listed as well.)

From Source:

Exim 4.63
Clamav-0.88.6

Building Exim

Building Exim was pretty straightforward. Grab the tar.bz2, unzip, untar, copy src/EDITME to Local/Makefile. Edit the Makefile. It is well commented and lets you turn on/off options, specify where you want the spool, bin directory and config files to go. I created zfs filesystems for /usr/exim and /var/spool/exim. Also elected to make sure I had all the content scanning and TLS/SSL options turned ON. I've got Sun Studio 11 installed, and it used this complier when I ran the make. Then ran make install to stick everything in the right place.

Before actually running exim, I had to make sure that clamav and spamassassin were all up and running, so I got those to the point where the clamd and spamd daemons were running, then simply editted the /usr/exim/configure file to set things up.

Building Clamav

Clamav is cool because it has the "configure" script. So I ran it, it found everything it wanted, and then I ran make and it just compiled. Followed by "make install" to put it all in /usr/local (which is also a zfs filesystem).

Plugging it all in

I kind of cheated and hacked the /lib/svc/method/smtp-sendmail file. I did this because there are dependencies on sendmail which I need to maintain. I cheated a little bit more by having this same file star clamd, freshclam and spamd. It gives me less granularity of control, but the reality is that I usually manage all of these as a group anyway.

The other tiny problem I had was that spamd didn't honour the LD_LIBRARY_PATH variable because bits of it are setuid. (Its a Solaris security thing). Anyway, this meant that I had to use crle to make sure that all the apps had /usr/sfw/lib and /usr/local/lib in their paths.

In my next blog

I'll cover in some more detail the settings I used for exim, and the fun I had with outgoing SMTP e-mail.

tags: Solaris SUN X86

More home server stuff

2006-11-16T03:24:35-08:00

Almost there...

My "services" configurations are almost done now. I'm currently learning about the interactions among: Live Upgrade (which is REALLY cool), SVM mirrored root disks, GRUB menus, and ZFS. (oh, it'll be so much easier when zfs boot is available, but until then.....). The problem kind of starts with the fact that the boot failsafe doesn't recognise /dev/md devices, which means I need to break my mirrors if I need to fix a booting problem.... unless I've got Live Upgrade working which means I'll ALWAYS have a bootable environment.

The man page of lucreate suggests that I use lucreate to do my mirroring.

So, I've currently got the following:

2 boot disks partitioned as follows:

slice0=7GB
slice1=2GB
slice3=7GB
slice4=free
slice7=32MB

Solaris nv Build 52 installed on c1d0s0, with an intent to mirror onto c2d0s0. My ABE for lu is going to be slice3 mirrored.

The lucreate man page says I simply need to type:

# lucreate -c slice0 -m /:/dev/md/dsk/d30:ufs,mirror \
-m /:/dev/dsk/c1d0s3,d31:attach \
-m /:/dev/dsk/c2d0s3,d32:attach -n slice3

which doesn't work due to a couple of "issues" with the interaction between LU and SVM. They are almost all fixed now, and the syntax of the command required is slightly different. I adopted the quick and easy workaround... simply create the metadevices manually first.

# metadb -a -f -c 3 c1d0s7 c2d0s7
# metainit d31 1 1 /dev/dsk/c1d0s3
# metainit d32 1 1 /dev/dsk/c2d0s3
# metainit d30 -m d31
# metattach d30 d32
# lucreate -c slice0 -m /:/dev/md/dsk/d30:ufs -n slice3

This fails after about 30 mins. My filesystem is full. No room to write the GRUB config stuff. Hmm, my slice0 is only 54% full. Slice3 is 100%. It's gone and copied all my data that's mounted via zfs. Referring to the Manual again and I learn how lucreate decides what to put into the BE or not. It looks in vfstab. Zfs filesystems aren't listed there, so it figures that my /opt/SUNWspro (which is a zfs filesystem) is on the / partition, and dutifully copies it. Hmm. Need to exclude all my zfs filesystems. (There are a couple of bugs logged against this as well, as one would think that lu would be clever enough to do this by itself... and it does kind of.)

# zfs list | awk '{print $5}' | grep -v \- > /luexcludefile (edit it so that I only list top level directories)

Try the lucreate command again with the -f /luexcludefile . This fails because of package dependencies. (i.e. I'm excluding filesystems that are in the sadm/package files.). Need to use the -I flag. I need to clean up the configuration before I start again, ludelete doesn't work properly because the process failed. Manually delete by editting /etc/lutab file, and simply getting rid of the partially created BE. (ludelete is preferred, only do this if you think you know what you are doing) My final, successful lucreate command was:

# lucreate -I -l /data/downloads/luerror -m /:/dev/md/dsk/d30:ufs -n slice3 -f /data/downloads/luexclude

So, I luactive that and reboot. df -k shows I've forgotten to mirror the swap disk. oops.

# mkfile 1024m /var/tmp/swapfile
# swap -a /var/tmp/swapfile
# swap -d /dev/dsk/c1t0s1
# metainit d21 1 1 /dev/dsk/c1t0s1
# metainit d22 1 1 /dev/dsk/c2t0s1
# metainit d20 -m d21
# metattach d20 d22
# swap -a /dev/md/dsk/d20
# swap -d /var/tmp/swapfile
# rm /var/tmp/swapfile
# (vi /etc/vfstab and change swap location to the md device.)

I'm now running off my mirrored BE: slice3. I ludelete the single sliced slice0, and repeat the process I've just done, rebuilding the slice0 BU as a mirrored slice.

I feel a lot safer now. I've got 2 mirrored bootable partitions on my system. I'd need to to a LOT of stuff wrong for me to get into an unbootable position (again).

Did I mention that the motivation for getting this all to work was that I managed to break my boot environment? (Don't ask, I'm too embarrassed to say how). And that the only way for me to recover was to re-jumpstart the system from my laptop? The whole system ended up being down for about 3 hours, as it took that long to get all the services reconfigured and ready to go again.


 tags: Solaris SUN X86