JET Stream

New OVM Server for SPARC WP

2013-03-15T09:40:02+00:00

I've been working mainly on Engineered Systems, more specifically SPARC SuperCluster for the past year, and it occurred to me that the way that we use layered virtualisation within the SSC is not very heavily publicised, and although you could derive these configurations from the OVM Server for SPARC documentation, it wasn't clear that this was considered a valid way of configuring a T-Series based server.

One of the ways that the SSC achieves such high levels of performance and efficiency, is that all the domains are built as root domains, which means that there is zero virtualisation overhead within these domains. Implementing Root Domains with Oracle VM Server for SPARC co-written by me, Mikel Manitius and Jeff Savit explains how this is achieved, and how to use these techniques on your own T4 based servers, to get the same kind of benefits.

Happy Reading!

JET now available on OTN

2012-06-06T21:27:37+00:00

I know some of you have been waiting patiently, so I'm pleased to announce that the JET bundle is now available for download on the Oracle Technology Network.

I've migrated most of the content from the old Sun wiki site, and got the download in a single handy location on OTN.

Download JET now.

The version available is the current latest, which is 4.9.4. This version contains a number of updates, the most significant of which is the ability to specify slot locations instead of the traditional cXtYdZsN nomenclature. This is pretty useful when trying to Jumpstart multiple servers with SAS2.0 based HBAs, as they will have the WWN embedded in the cXtYdZsN name, and it's pretty difficult to guess what that will be until you've booted the server.

The JetSDS and JetZFS modules have also been updated to use the slot terminology.

Happy JETing,

JET 4.8 Now available

2010-10-25T07:13:36+00:00

It has been a long while. We've had about 6 internal releases of JET 4.7.1, .2, .3 etc, but we've recently gone to 4.8

The main highlight is that JET 4.8 has the new sysidcfg variables required for the unattended installation of Solaris 10 09/10. It also knows about the "new" Solaris 10 Recommended patch structure.

The best way to get it is to click the link at the top of the wiki site.

JET 4.8 Changes:

As mentioned above the main JET change is the addition of a new base_config_sysidcfg variable to allow the auto-registration stuff to be configured.

Other changes:

Ksh93

All the modules have been updated to be ksh93 compliant, for those of you who are using OpenSolaris/Nevada/Solaris 11 Express Jumpstart servers.

Recommended Patches

The Solaris 10 Recommended patches (since around this time last year) have had a different directory structure, a better installation script, and a "password" to ensure that you've read the "README". JET understands this new structure and uss the installer script if present. A new base_config variable: base_config_recpatch_passcode has been added to supply the password.

Assorted Bug fixes

I've fixed an issue with the hosts file with DHCP Sparc hosts, check_client for custom checks f and m scripts, nfsmapid is started in the miniroot so nfsv4 shared files have the right perms, search items are now first in resolv.conf, use of correct awk in get_matrix_byname, added clearer fdisk comments, fixed an update_template bug, stopped n-1 custom from forcing another boot, made setupdumpdevice work only in a global zone, further refined run_sshkeygen, updated pkginfo to install only in "THISZONE".

Menu.lst enhancements

There's now a bit more flexibility in dealing with the menu.lst file. JET has always had the capability of simply appending a file to the menu.lst file using base_config_menulst_append. I've added 3 more variables to allow even more control:

1: base_config_grub_append: lets you add arguments to the boot line in the menu.lst file.

2: base_config_grub_timeout: change the default timeout from 2 seconds to whatever you set here.

3: base_config_grub_singleuser: add a boot option in menu.lst to boot net single user.

Custom Files enhancements:

You can now optionally specify owner:group:mode for each files that custom_files copies. Under normal circumstances JET simply duplicates the perms of the file on the JET server. In implementing this, I discovered an interesting bug with NFS permissions in the net-booted Solaris, which meant I had to work around that by manually starting nfsmapid in the Jumpstart phase to ensure correct and consistent permisssions.

Auto-Reg

Implemented the auto_reg functionality. Probably easier to simply paste the comments in the Template here:

#
# From Solaris 10 Update 9 onwards, a new sysidcfg keyword exists: auto_reg
# This allows automatic Solaris registration to take place.
# auto_reg can be all, noproxy, anon, none or disable
#
base_config_sysidcfg_auto_reg=disable
#
# "all" requires ALL additional fields to be filled in
# "noproxy" requires only oracle_user and oracle_pw fields to be filled in
# "anon" requires all the http_proxy fields to be filled in
# "none" or "disable" do not need any additional fields.

base_config_sysidcfg_auto_reg_oracle_user=""
base_config_sysidcfg_auto_reg_oracle_pw=""
base_config_sysidcfg_auto_reg_http_proxy_host=""
base_config_sysidcfg_auto_reg_http_proxy_port=""
base_config_sysidcfg_auto_reg_http_proxy_user=""
base_config_sysidcfg_auto_reg_http_proxy_pw=""

Other Modules

Some minor changes/updates have been made to some of the modules JetSDS, JetZFS, JetZONES. Mainly bugfixes and error reporting enhancements, so no huge functionality changes, but you might want to scan through the comments in the templates as I have rewritten some of them to make some of the existing functionality clearer.

Summary

As usual, we maintain backwards compatibility, so your existing templates will continue to work. To make use of the new functionality, you'll need to refresh your templates. All new clients (created with make_template) will automatically have the new variables. (Have a scan through sample.template in /opt/SUNWjet/Templates to see what's changed.)

In case anyone asks. Upgrading JET is as simple as pkgrming your existing JET modules, and then pkgadding the new ones. JET won't remove any of your existing configuration, Template or Client files.

tags: Solaris JET Jumpstart

The Cloud Holy Grail

2010-06-18T01:53:17+00:00

I went to the Oracle Enterprise Architects Club yesterday which was titled Enterprise Cloud: Hype or Reality, with talks from 3 perspectives: Customer (I have a private Cloud), Partner (I build and run Public/Private Clouds for customers), and Oracle (We have a full stack to deliver PAAS).

It helped me clear up in my own mind that the Cloud is really only a cloud to the consumer. To the cloud provider, it is not cloudy at all, but a very specific set of scoped services delivered by a H/W and S/W stack. i.e. The consumer does not need to know how or what sits inside the cloud, simply that it delivers a reliable service using self-service provisioning with predictable and transparent pricing.

It is the job of the Cloud provider to build an infrastructure that supports the user expectation of a Cloud service. For some reason Cloud architectures seem to immediately become a VMWare sales guy's dream. The definition of Cloud as a set of systemic qualities by its nature has no specification for HOW that could be delivered, and there are countless ways of building a technology stack that delivers Cloud services. I think I could make a strong case for a Cloud Infrastructure being composed of a pair of clustered M9000-64s with thousands of Solaris Containers connected to an array of S7410s. It would exhibit the same systemic qualities as any of the other cloud architectures out there.

The point is, the technology stack bit is easy, and your architectural choice is about where you want the orchestration of multi-tenancy to live. In the middleware layer with a Weblogic grid? In the DB layer with an Oracle RAC grid? In the O/S layer with Solaris Containers? Or in the Server layer with OVM? My answer? All of the above. Doing ONLY the last is hugely inefficient. Having an agile and efficient infrastructure is key to minimising your running costs.

The 2 hard parts are self-service and billing. In the IAAS and SAAS worlds, the delivered service is sufficiently constrained to make this solvable.

The self-service delivery of PAAS requires a much more rich and complex set of interfaces for the user to be able to articulate and deploy the required "service". This could be simplified by providing pre-configured models and simply allowing the customer to select the ones they want.

The HARDEST part of delivering PAAS is the cost model. This is simple economics: The cloud provider has an investment in infrastructure that needs to be paid for over its lifetime, and has running costs that tend to be relatively stable whether customers are using the service or not. The cloud provider has to provide a commercial proposition to the customer that delivers the service cheaper than if the customer did it himself, while still being able to cover his costs and make a profit. Obviously in the IAAS world I can simply charge per allocated resource, and in the SAAS world I can simply charge per transaction or per user. What is the PAAS model? Transaction? Resource? User? A combination of them all? Do I need to have a reservation charge as well as a usage charge? How does/can the customer compare the cloud cost against his own?

The solution to the PAAS billing problem? Make it a Private Cloud and you can skip the billing piece.

In summary, the Holy Grail of PAAS is to come up with a commercial model that allows customers to use the service at a lower cost while still allowing the Cloud provider to make a profit. When someone comes up with an answer, please let me know.

tags:solaris containers virtualisation oracle cloud

Jumpstarting over multiple subnets

2009-12-17T01:57:33+00:00

The question of Jumpstart and subnets is always coming up. I've finally documented the issue and the ways of "solving" it on the JET wiki site. As with any problem, there is no single easy answer, but a number of different solutions that can be combined in a number of ways.

tags:jet jumpstart solaris

Solaris Virtualization Epiphany

2009-05-16T02:31:34+00:00

I was giving my standard pitch covering Sun Virtualisation Technologies which essentially entails me talking at length about the following slide. I kind of make the point that it is a means not an end, and that really what you're trying to do is run more applications on the same piece of hardware, and the trick is to try and do it most efficiently.

I've done this talk countless times with a pretty wide variety of customers, and I tend to start talking from the right, and move to the left, explaining how the "layer of additional non app resource utilisation" grows from right to left. (not true for Dynamic System Domains on the far left though.)

It finally occurred to me yesterday that the Unix and Windows world have an utterly different view of Virtualisation, and the need for Virtual Machines.

In the Unix world, application co-existence is second nature to most sys-admins. Virtualisation is all about application isolation. i.e. You START with all the apps on a single O/S instance, and you apply more and more constraints: resource management --> containers/zones --> virtual machines --> dynamic system domains.

In the Windows world, application co-existence is so rare that it isn't even considered. Virtualisation is all about application consolidation. (and the primary way to do it is to create lots of virtual machines, one per app).

The problem is simply viewed from completely different angles.

It's not just an application co-existence issue either. There's a scalability problem. A lot of "other" O/S's can't scale to fit the larger more powerful Intel/AMD multi-core servers available today, and the ONLY way to make use of them is to carve them up into smaller machines that Windows or Linux can digest. Solaris, on the other hand can scale up to hundreds of CPUs, and hundreds of Gigabytes of memory.

From the slide above, Sun/Solaris provides a wide choice of technologies to allow multiple applications to be consolidated onto a single server. Don't let the limitations of "other" O/S's blinker you into thinking that a Virtual Machine is the only answer.

(I'm not saying that Virtual Machines is the WRONG answer btw, it's ONE of the answers, but it shouldn't be the only one that is considered.)

tags: virtualization containers ldom zones vmware xen

JET 4.7 Now available

2009-05-08T01:04:22+00:00

So, I've been working on JET 4.7 for a while now. A couple of weeks ago I released it internally to Sun, and no-one's complained about any showstopping bugs yet, so I've released a fresh bundle to the sun.com downloads site. The best way to go straight to it is to follow the link at the top of the wiki site.

JET 4.7 Changes:

The MAJOR JET change is that I've kind of merged JetNAS functionality into the main SUNWjet module. This isn't an exact replication of the JetNAS code, and it does things a little differently, but in short you can choose to place the NFS functionality somewhere different from your JET server.

Other JET changes are covered by bugids: 6835016, 685027, 685023, 685019, 685030.

In summary:

6835016: JET needs to support alternative servers for NFS media. The JET User Guide has already been updated to cover this functionality. Additionally, I've written a short article about it on the JET wiki site.

685027: JET jetjump.xml script does not stop graphical login from starting. Tidied it up. graphical login no longer starts until J/S is complete.

685023: JET should not create /dhcp directory. We used to do this to shorten the dhcp macro. This is no longer necessary, so we've removed the complication of having it.

685019: JET ksh scripts do not work when using OpenSolaris. There were a couple of constructs in our ksh code that ksh93 choked on. We've updated them so that they are compatible with ksh and ksh93. This allows all you OpenSolaris people to install JET on your OpenSolaris based Jumpstart server. (Note, AI is required to "jumpstart" OpenSolaris.)

685030: JET error message when suitable server is not found could be more meaningful. Well, this is just an attempt to reduce the e-mail traffic. Probably the first error, people new to JET see. We've updated the error message to tell you what to do to fix it.

JET 4.6 - 4.7 Change Summary:

For all you "external" people, the last externally available version was 4.6. Here are the changes from 4.6 - 4.7:

4.6.1: Ability to compress the rpool, and specify /var for zfs boot builds.

4.6.2: 6798648: JET needs to workaround Bug: 6528699: This is to stop Jumpstart from labelling ALL your disks and blowing away all your data. Also added a tweak to the ssh scripts to generate the keys properly.

4.6.3: Added an additional bootadm update-archive to the end of the finish.common script. We were getting some situations where the first reboot had a stale boot-archive, possibly due to a race condition in SPARC S10 Update 6 and above builds.

JET bundled Modules:

We've updated some/most of the bundled products. ALL of them (if required) have been updated to make them ksh93 compatible. Some of them have had more substantial changes:

JetLDOM

This module has had the most substantial rewrite. We've gone against ALL JET precedent and completely changed the structure of the ldom.conf file, so it's not backward compatible (for once). We needed to do that so that we could provide a much easier mechanism to make use of the ldm 1.3 functionality. Sorry.... but you'll like the new one better.

JetFLASH

I've updated JetFLASH so that is actually checks the flar (if it can), to see that it is the right architecture, and the right boot filesystem for the build. (Yes, zfs based flash is coming, but it is a different format to the ufs based one). Bruce added some code to clean up service tags if required.

JetZFS

Quite a few changes in this module. You can now update your rpool properties, I've provided more verbose progress messages (so you can see exactly what it is doing), added code to deal with converting EFI disks back to SMI (it does it for you if required), and finally added a strange hack so that you can optionally use ::: instead of = in your variables to get around a xVM OC gui restriction.

JetZONES

Fixed a bug where make_template was in the wrong place, and added some code to boot the zone before trying to add stuff to the rootpath/var dir. (because it might not be there unless the zone is booted). (Thanks to Mike Hendon for pointing it out). (plus a bunch of new functionality like support for branded zones)

Summary

So just a reminder, the following modules have NEW template variables. Old templates (other than the ldom based ones) will continue to work, but if you want to make use of the new functionality (with the new comments that describe them), you'll need to refresh your templates. (Have a scan through sample.template in /opt/SUNWjet/Templates to see what's changed.)

tags: Solaris JET Jumpstart

JET/Jumpstart without NFS

2009-04-15T06:52:40+00:00

I've been working on JET 4.7 (current released JET version is 4.6.3).

The main change is that I've now provided support to allow the Solaris Media and the /opt/SUNWjet directories to be hosted by a server (or servers) other than the JET server. This effectively removes the requirement for the JET server to provide NFS services.

Along the way I was able to tidy up and consolidate a lot of code, particularly the client_allocation_\* methods. I've added a few functions, and used the variables more consistently throughout. For those of you doing DHCP/PXE based builds, I've removed the /dhcp hack that was used to overcome the 255 char limitiation of the DHCP macros.

Full details on the wiki site

I'm hoping to release it by the end of April.

tags: Solaris JET Jumpstart

FOLLOWUP: Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server

2009-01-06T02:57:55+00:00

A few months ago I wrote Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server, which detailed some of the hoops that I needed to jump through to get it to work. The good news is that the new features in VirtualBox 2.0 (and now 2.1) have gotten rid of 2 of those hoops:

I no longer need to play with vnics and that crazy script as Host Interface networking simply works out of the box
I don't need to install the 32bit internal only SC package, as VirtualBox now supports 64bit guests.

I'm in the process of rebuilding my laptop cluster making use of the above new features.

Hurrah!!!

tags: JET Jumpstart SunCluster Solaris VirtualBox xVM

JET 4.6 externally available

2009-01-05T02:03:52+00:00

As a follow-on to my previous post, JET 4.6 is now available externally. Check the JET wiki which has a link to the download.

This is mainly a bug fix release, and has been given a major number increase because it's the JET version that will be in the next release of xVM OpsCenter. (and we like to release the OpsCenter releases at major version numbers).

In terms of changes, I've fixed a couple of bugs with turning ntp on, and multiple disks in zfs pools for ZFS disks. This version also has tweaks in place to workaround some DHCP issues when installing S10 U6 and Solaris Nevada.

tags: Solaris JET Jumpstart

JET 4.4.7 available externally

2008-11-04T16:17:25+00:00

As a follow-on to my previous post, JET 4.4.7 is now available externally. Check the JET wiki which has a link to the download.

Also have a look at the JET User Guide which has now been posted on the wiki. I expect this to grow over time, but its a conversion from the original user guide with updates for correctness.

tags: Solaris JET Jumpstart

JET 4.4.7 released

2008-10-21T04:14:54+00:00

I've just released JET 4.4.7 internally. Bruce is in the process of putting together a new bundle including 4.4.7 for the external download, so it'll arrive in couple of weeks. If you need it NOW just ask nicely.

Changes

The latest version currently available externally is 4.4, and we've added a bunch of features since then. Here's a brief overview of what's new/changed/fixed:

4.4.1

6660631: Disabling UFS logging no longer works.
6673213: Selecting dhcp should automatically select grub if required.
6675467: JET needs to workaround Nevada installation bug 6633017.
6315828: JET "shutup_sendmail" variable does not work on Solaris 10.
6526561: Typo in make_client, missing r on JS_error line 130.

4.4.2

6656549: Add ability to do fdisk commands in the profile.

4.4.3

6694299: SUNWjet fails to install on system with super long group.
6690411: base_config_profile_fdisk needs to be ingored for sparc.
6698860: "rarp" service should not be enable by JET (on dhcp builds)

4.4.4

6706581: JET needs to support ZFS root installs. This is a pretty big one, with the inclusion of 5 new base_config variables. Might require a separate blog article to explain, but as usual, the comments in the template should get you on the way.

4.4.5

6735314: JET incorrectly sets NFSMAPID_DOMAIN to "dynamic".
6735318: JET should support noatime mount options.
6740625: JET client_environment should never output a warning message.

4.4.6

Never released.

4.4.7

6675467: (again, as we need to apply workaround for Sol10 U6 as well)
6761787: JET custom_files should allow the overwritten file to be backed up. (inspiration from Christopher Hubbell)

Summary of Changes

A lot of the changes are bug fixes, but a number of them begin to make use of some of the new Solaris features, or leverage new ways of doing things. The most important of these is the ability for JET to install ZFS boot Solaris. This applies to Nevada build 90 and above, as well as the imminently to be released Solaris 10 Update 6. Other things that are kinda cool is the ability for JET to fdisk your disk for you when doing an install, as well as set noatime mountpoint options (which more and more people seem tobe doing by default now.

tags: Solaris JET Jumpstart ZFS

belenix, opensuse and ubuntu on Nevada with VirtualBox

2008-09-24T10:43:54+00:00

I was in the process of playing with some stuff using VirtualBox, and it occurred to me that this was a pretty cool screenshot, so I did, and here it is.

tags: Solaris belenix ubuntu opensuse virtualbox

Using Sun Secure Global Desktop

2008-09-17T06:50:29+00:00

Over the past few weeks/months, I've been involved (sometimes peripherally) with setting up our demo kit in the UK Customer Briefing Centre. One of my design goals was to ensure seamless access to our demos from anywhere in the world or more realistically anywhere in the UK, and more specifically, at any customer site.

Within the CBC network (which is not connected to our internal Sun network) we set up a Sunray Server also running SGD. We initially had some hiccups getting the firewall traversal to work, and had to resort to using VNC which was adequate, but painful as we were tunnelling through ssh, so there were a bunch of steps required to ssh in, start vncserver, then re-ssh in to tunnel the right port, then start vncviewer and after all of that the screen refresh rate wasn't stellar.

However, once we got SGD working it was absolutely awesome.

If you want to have a play with how it works, simply go to Sun's SDG demo site . Unfortunately MyDesktop doesn't work (for obvious reasons) for anonymous users, but it will let you play with the webtop functionality.

For our purposes, we used the MyDesktop functionality. Running a desktop session inside the CBC from anywhere in the world was as easy as pointing a browser at the right place, logging in and clicking on the Desktop icon. Within seconds, as if by magic, a full screen desktop takes over your current screen (you can switch back to local using Alt-TAB). I'd show you a picture of it, but it just looks like a normal desktop and feels remarkably snappy even if it is remote.

The reason we were doing all of this is that we (The UK Systems Practice) have been running some "Discovery Days" around LDOMS and ZFS which have live demos of the technology. The cool part is that we get to showcase the SGD technology at the same time!

tags: Solaris SGD ZFS LDoms

Helford River

2008-09-17T04:29:01+00:00

So I got the boat down, and managed to go sailing in and around the Helford river in Cornwall. It was absolutely awesome. On the day the picture was taken, the wind was pretty light and so I decided to brave going out single handed. Even got the asymmetric out on the downwind leg.

Really Nice JET Tutorial

2008-08-21T05:26:42+00:00

Joerg Moellenkamp has written a pretty cool 16 part tutorial on Jumpstart using JET here. It's already linked at the JET wiki page.

tags: Solaris Jumpstart JET

No, I haven't stopped sailing

2008-08-17T14:05:31+00:00

I've not blogged about sailing in some time. Its not that I've not been Sailing. Its more due to me not blogging enough. I'm going to try and do a weekly recreational blog. (Watch me fail).

Since my last post, I've been helping out at my local Sailing Club (Burghfield Sailing Club)
with their Oppie Club. My oldest daughter got quite into it, and I got her an old wooden Oppie called Major Tom. Here's me sailing it.

As you can see, it's for kids, not adults. (And yes, I did eventually pick up the kid swimming behind the boat).

I've gotten a lot better sailing my Laser 2000, and have been trying to take part in the Wednesday Evening Pursuit races that take place during the Summer at the club. (On Wednesday evenings, funnily enough). I actually came first in one of them this year, and pretty happy with my progress.

Last week we were doing pretty well in quite windy conditions (I think we were 4th or 5th with 15 mins to go), when my main halyard broke, resulting in me having to DNF. The bigger problem was that we're taking the boat down to Cornwall on holiday next week, so I had to get it fixed pretty quickly. This weekend, I spent Saturday morning down at the Sailing club taking the mast down, and feeding a piece of wire up the inside of the mast, and rerunning the old main halyard. (This is harder than it sounds) The Chandlery at the Sailing Club was closed (the owners had a day at the races at Ascot or something) so I couldn't get a new halyard. Felt I might as well run the old rope, and once it's in, its easy enough to replace without having to take the mast down. On Sunday, today, they were open, so I was able to replace the old rope with a nice shiny new halyard.

It's also the first time I'll be towing the boat anywhere, so I also needed to get a couple of towing essentials for the trip. (Fortunately the family car has a towbar). I've decided to play it safe and get a new Jib Halyard as well, as it is also about 8 years old. A trip to the LaserDirect and £100 pounds later, I've got a bunch of stuff coming, hopefully by Wednesday this week, which will give me enough time to fit before setting off on the weekend.

It's also the first time that we'll be taking the L2K onto the open water. The place we've rented is in a sheltered cove, so we don't HAVE to go to sea, but it might be fun to go out for a laugh.

If this is my last ever blog post, you can guess what happened.

More on JET and iso images

2008-08-08T08:42:07+00:00

I've made some minor modifications to share_isos. I realised that add_install_client creates entries in /etc/dfs/dfstab for the "boot" filesystem if not already mounted. Unfortunately, it only checks the dfstab, rather than checking the actual shares. The result is that I need to populate the /etc/dfs/dfstab file.

The complication is that now that it's in dfs, the stubs get shared when the server boots, so I need to unshare them before doing the lofi mounts.

I've updated the share_isos script to do that. I've decided I'm simply going to make it an add_on JET module that can be installed if that feature is required/desired. (Look out for a JetISO module any day soon).

Finally, while testing all of this I decided to so something pretty obscure. I'm in the process of building a JET.vdi and I'm trying to make the .vdi as small as possible. (A virtual disk image of the smallest Solaris I can use, with JET and its modules installed so that it is nice and easy to distribute (so far its about 800k). I obviously don't want to prepopulate with any Solaris media as that will simply make my vdi too large. The cool thing about iso images is that I can NFS mount a directory containing them, and then I can lofiadm and mount them locally.

So I ended up with a JET server running in a virtualbox, NFS mounting a Sol 10 0508 iso from my home server, lofimounted and shared with share_isos, and I used that to jumpstart another Solaris virtualbox machine on the same PC (which was running XP). The really good news was that it worked. The not so good news is that it took over a day to install. I think there are a few too many levels of indirection and virtualisation between the iso and the virtual disk.

I should really do a bit more testing and figure out the bottleneck. Ideally I'd prefer to keep the isos on the host and use shared folders to allow the virtual JET server to access them, but that won't be possible if the performance is as bad as my original test.

tags: Solaris Jumpstart JET virtualbox

JET server using Nevada and iso images

2008-08-07T07:58:05+00:00

Things have been slightly problematic on the JET server running on Nevada front mainly due to 2 issues.

There is/was a bug in cpio (See BugID 6686818) which meant that setup_install_server didn't copy the media correctly.
setup_install_server on earlier Solaris releases didn't understand zfs, therefore it was difficult to copy onto a zfs filesystem.

Why not just use iso images?

Why not indeed? I'm in the habit fo simply downloading and keeping the .iso images for all releases, and I used to then "expand" it by lofimounting and then running setup_install_server. I realised I really didn't need to do that, I could just use the lofi mounted image, and get away from having to ever run setup_install_server, and not need to use up double the space per O/S image I wanted to keep.

So.. what was missing?

I needed a way to lofiadm, mount and nfs share the Solaris media. Also because x86 builds also lofi mount the boot directory and mount it into the /tftpboot directory, I'd also need to lofimount a lofimounted filesystem as well.

Firstly I did everything manually to ensure that everything works fine, then I wrote a script, which reads a config file (which lives in /opt/SUNWjet/etc).

Inline here:

#!/usr/bin/ksh -p
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
################################################################################

# sanitize PATH, and ensure required components are in front
PATH=/sbin:/usr/sbin:/bin:/usr/bin:${PATH}
export PATH

BINDIR="`cd ${0%/\*}; pwd`"
ETCDIR="`cd ${BINDIR}/../etc; pwd`"

PATH=${BINDIR}:${PATH}
export PATH

MEDIALOC=${ETCDIR}/solaris_iso_locations

if [ "$#" != "0" ]; then
	echo "Usage: $0"
	echo " "
	exit 1
fi

#
# Check to see if the media locations file exists.
#

for iso in `cat ${MEDIALOC} | grep -v "\^#" | awk '{print $1}'`
do
	lofiname="`lofiadm | awk '$2 == iso {print $1;}' iso=$iso`"
	if [ -z  "${lofiname}" ]; then
		lofiname="`lofiadm -a $iso`"
		echo "Added $iso as lofi filesystem: $lofiname"
	fi
	mp="`cat ${MEDIALOC} | awk '$1 == iso {print $2;}' iso=$iso`"
	mounted="`df -k | awk '$1 == lofiname && $6 == mp {print $6;}' lofiname=
$lofiname mp=$mp`"
	if [ -z "${mounted}" ]; then
		mount -F hsfs $lofiname $mp
		mounted=$mp
		echo "Mounted $lofiname at $mp"
	fi
 	shared="`share | awk '$2 == mp {print $2;}' mp=$mp`"
	if [ -z "${shared}" ]; then
		share -o ro,anon=0 $mounted
		echo "Shared $mounted"
	fi
	# Deal with lofi mount in x86
	mountboot="`cat /etc/vfstab | awk '$1 == mpboot {print $3;}' mpboot=\\"$m
p/boot\\"`"
	if [ -n "${mountboot}" ]; then
	bootmounted="`df -k | awk '$1 == mpboot {print $6;}' mpboot=\\"$mp/boot\\"
`"
		if [ -z "${bootmounted}" ]; then
			mount $mountboot
			echo "Remounted $mountboot"
		fi
	fi
done

It parses the solaris_iso_locations file which looks like this (I've attached my filled out one)

# This file is used to specify the mountpoints for iso images. You need # to create the mountpoint, update this file, and then run /opt/SUNWjet/bin/ # share_isos. This will lofimount all isos listed here. You will then # need to run add_solaris_location for each image as normal. # You should run the share_isos command everytime the JET server is # rebooted. /backup/Solimages/s10sparc_0508.iso /export/install/iso/Solaris_10_0508s /backup/Solimages/s10x86_0508.iso /export/install/iso/Solaris_10_0508x /backup/Solimages/solarisdvdb90x.iso /export/install/iso/Solaris_11_b90x /backup/Solimages/solarisdvdb90s.iso /export/install/iso/Solaris_11_b90s

What it does.

In short, it reads the config file and mounts (if not already mounted) the iso images on the path specified. It will then NFS share the mountpoint, and parse the /etc/vfstab and do the /tftpboot lofi mounts if necessary. (NOTE: add_install_client creates those entries in /etc/vfstab the first time it is run for that O/S. You MUST edit the /etc/vfstab and change the mount_at_boot flag to "no", otherwise your booting your JET server will fail.)

Running share_isos produces the following:

# /opt/SUNWjet/bin/share_isos
Mounted /dev/lofi/2 at /export/install/iso/Solaris_10_0508s
Shared /export/install/iso/Solaris_10_0508s
Mounted /dev/lofi/3 at /export/install/iso/Solaris_10_0508x
Shared /export/install/iso/Solaris_10_0508x
Remounted /tftpboot/I86PC.Solaris_10-1
Mounted /dev/lofi/4 at /export/install/iso/Solaris_11_b90x
Shared /export/install/iso/Solaris_11_b90x
Remounted /tftpboot/I86PC.Solaris_11-6
Mounted /dev/lofi/5 at /export/install/iso/Solaris_11_b90s
Shared /export/install/iso/Solaris_11_b90s
# /opt/SUNWjet/bin/share_isos
#

Note: if you run share_isos again, it does nothing as everything is already mounted and shared, so its easy to re-run it whenever you want just to confirm it's all good. (I considered having JET call it automatically, but I'm not completely sure how I want to do that yet. Well thought out suggestions welcomed.)

The bottom line is that I can now easily use my laptop running Nevada and a ZFS filesystem and use it as a JET server for any version of Solaris without ever needing to run "setup_install_server". And the bonus is I've now got about 12GB of space back!!

tags: Solaris Jumpstart JET

Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server

2008-08-05T05:15:04+00:00

I was intrigued by Gordon Johnston's blog article about creating a 2 node Cluster Express cluster using VirtualBox, and I decided I had to try it.

Here's the final result:

(You can click on the image for a fullsized picture.)

It shows a VNC connection to my laptop which has 4 windows up: The 2 VirtualBox instances showing cxnode1 and cxnode2, a terminal window on the laptop itself, showing the quorum configuration and the iSCSI share, and in the background, the VirtualBox console showing the network configuration of cxnode1. Finally, in the main screen, I'm running Sun Cluster Manager and connected to the cluster, showing the topology and the Resource Group configuration of apache.

Rather than simply duplicating what Gordon did, I decided to go with Solaris Cluster 3.2 U1 as my cluster software. This meant that I had to use an internally available package of 32 bit kernel drivers, as SC 3.2 is shipped as a 64bit only product. (Cluster Express has both 32 and 64 bit variants.) The reason for going with Solaris Cluster is that there are instances where I want to duplicate a scenario that a customer is seeing, and in most cases, that's on Solaris Cluster.

I'm hesitant to duplicate Gordon's article, as I pretty much followed his guide, so I urge you to read his article as it is quite detailed, I'll only highlight where I changed things.

I only really wanted a working cluster, went for the simpler set up of HA Apache
Cluster node O/Ss would be Solaris 10 05/08
Host O/S is Solaris Nevada Build 93 (as that was what was already running on my laptop)
I wanted to Jumpstart the nodes using JET.

Preparing your physical host

I already had VirtualBox 1.6.2 installed, so there wasn't much to do there. The main trick was configuring my laptop to allow Host Interface networking. This bit is Solaris specific. You basically need to do 2 things.

Firstly, you need to set up a bunch of virtual nics, and assign MAC addresses to them. As this needs to be redone on every reboot, I followed Dave Tong's advice and set up an rc3.d script. (and no, I didn't bother to use SMF either, you can bite me too if you want.)

#!/sbin/sh
state="$1"
case "${state}" in
'start')
    echo 'Plumbing VNICs'
    for MAC in 10 20 30 40
    do
        VNIC=`/usr/lib/vna nge0 c0:ff:ee:0:0:$MAC`
        ifconfig $VNIC plumb
    done
    ;;
'stop')
    ;;
\*)
   echo "Usage: $0 { start | stop }"
    exit 1
    ;;
esac

(I did use Gordon's cute fake MAC address c0:ff:ee 'cos its easy to remember)

Secondly, you REALLY REALLY need to make sure that you give the VBoxSVC privileges to configure the interface, otherwis your virtual machines will fail when starting. Whenever you start VirtualBox, you need to get the PID of the VBoxSVC, and run: ppriv -s AEI+net_rawaccess <pid>

Creating your Virtual hosts

Basically I followed Gordon's instructions for this, with TWO exceptions. I needed to select Network in the Boot Order menu in Settings-->General-->Advanced as PXE is not enabled by default. I also found that I couldn't do a PXE boot if I used the Intel driver, so I used the PC-net FAST III driver throughout which worked fine.

Installing Solaris on the Virtual hosts

I've already got JET installed on my laptop, so preparing to install Solaris was an amazingly simple process:

/opt/SUNWjet/bin/make_template cxnode1 base_config
Edit the template
/opt/SUNWjet/bin/make_client cxnode1
Start the cxnode1 Virtual Machine, Press F12, Select Lan Boot, and watch it install.
/opt/SUNWjet/bin/make_template -T cxnode1 cxnode2
Edit the cxnode2 template and change 3 variables.
/opt/SUNWjet/bin/make_client cxnode2
Start the cxnode2 Virtual Machine. Press F12, Select Lan Boot and watch it install.

Of course, the devil is in the detail, I'd already configured the Solaris media, and these are the changes I made to a default template for the build to work:

base_config_ClientArch=i86pc
\*base_config_ClientEther=c0:ff:ee:0:0:10
base_config_ClientOS=10_0508x
base_config_client_allocation="dhcp"
\*base_config_sysidcfg_ip_address=192.168.1.201
base_config_sysidcfg_netmask=255.255.255.0
base_config_profile_fdisk="alldisk"
base_config_profile_cluster=SUNWCXall
base_config_profile_root=free 
base_config_profile_swap=2048
base_config_profile_s6_mtpt="/globaldevices"
base_config_profile_s6_
base_config_ipmp_networkifs="pcn0_pcn1"
\*base_config_ipmp_networkif_pcn0_pcn1="prod sp 192.168.1.211 192.168.1.212 24 cxnode1 192.168.1.201"

(the ones with \* are the variables I had to update when I created the cxnode2 template)

Installing Solaris Cluster on the nodes

As the nodes were running in 32bit mode, (VirtualBox 1.7 should have 64bit support, which means this part won't be necessary in the future) I needed to use an internally available set of 32 bit kernel drivers in a packages called SUNWscka. Basically, install sc3.2 as usual using the installer, but don't configure anything, and don't configure a quorum device. Then pkgadd SUNWscka on both nodes, and reboot them both.

The rest

After that, I just followed Gordon's instructions for setting up the Quorum server, setting up the iSCSI devices on the hosts, mounting them on the cluster nodes, and creating a zfs pool using them.

Sadly, this now means I can play with and configure all sorts of things in a Sun Cluster environment from the comfort of my laptop!

tags: Solaris Jumpstart JET xvm virtualbox

Solaris Bootdisk Layout discussion

2007-09-25T09:12:21+00:00

There's been a vibrant discussion on the sysadmin-discuss(at)opensolaris.org alias about boot disk layouts, and it was suggested that we try to come to some form of consensus and document it.

Sun has a bunch of installation standards called EIS (Enterprise Installation Standards), and one of those standards is the EIS Bootdisk Standard. As one of the contributors to that document, I found it quite interesting to see that the sysadmin discussion was coming to similar conclusions about how a boot disk should be laid out as we had come to internally.

In the interests of sharing and getting to a common standard, I've now posted the relevant bits of the EIS standard on the BigAdmin wiki, and encouraged the members of the sysadmin-discuss alias to comment and contribute.

If you're interested, you should have a look at it, and join the alias to participate in the discussion.

tags: Solaris Bigadmin

CEC 2007 Here we come (again)

2007-09-03T03:34:22+00:00

(I've reposted so that technorati picks up the tag we'll be using for this event)

(Hint, tag using suncec2007 and we'll be able to agregate all content using said tag)

Its that time of year again. CEC2007 will be in Las Vegas this year. I've never been, and I'm half looking forward to it, but as I think I have an addictive character, I'm a little scared of of Vegas.

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables.

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

The theme this year is "Shift Our Universe, Our World, Your Move", sub themes being: Shift to red: (all the groovy bleeding edge stuff that GregP talks about, Shift to green: (all the nice eco-friendly stuff we've been doing), and Shift to grow: (which focusses on all the things we need to do to enable our growth (and our customer's as well))

We're also going to be giving out a bunch of prizes for participants and attendees of the conference. They'll be mainly based on level and quality of participation in the sessions. Stay tuned for more details.

As per last year, I'll be blogging my perspectives of the conference again.

tags: Sun suncec2007

Coolstack: Not just for Coolthreads

2007-08-10T08:34:32+00:00

As per a previous article, I upgraded by home server to Nevada build 69, because I was about to make some changes, and wanted to be fully up to date before I did so.

I wanted to play with Confluence which is the wiki that Sun has chosen to run their wiki site. Confluence can run in a number of ways and connect to a number of different databases. Since I hadn't yet played with Postgres, I decided I would use that, and I also decided to run it using Tomcat instead of as a standalone app.

Given that I'm a bit of a neophyte when it comes to talking native SQL, I decided that I needed to run a tool of some sort to configure Postgres. I've used phpMyAdmin for MySQL for a while now, and figured there MUST be an equivalent for Postgres. Using my amazing powers of deduction, I googled phppgadmin, and lo and behold there it was: phpPgAdmin. All I needed was a php compatible webserver. Unfortunately the supplied apache2 with Nevada isn't php ready, so I had to then decide.. do I compile myself, or do I find something easier. I vaguely remember hearing something about coolstack, which is "a collection of some of the most commonly used open source applications optimized for the Sun Solaris OS platform."

CoolStack 1.1 is available for both SPARC and x64, and contains pretty much all I needed, including tomcat. So a few downloads and pkgadds later, and I've got apache running, which means I can get phpPgAdmin running, which means I can perform my initial Postgres configuration, which means I can finally install Confluence under Tomcat, and everything's perfect. (Well it wasn't perfectly perfect, because I needed to read a couple of READMEs along the way, and a bit of on-line documentation, but it was fine.)

So, I've got tomcat listening on port 8081 as per normal and confluence sitting inside there as an applet talking happily to Postgres, and so I decide that I actually want to do the apache proxy thing so that I can have my confluence wiki appear to sit on my normal webserver port. This is quite easy to configure, you just need a few lines like this:

LoadModule proxy_module libexec/mod_proxy.so
LoadModule proxy_connect_module libexec/mod_proxy_connect.so
LoadModule proxy_ftp_module libexec/mod_proxy_ftp.so
LoadModule proxy_http_module libexec/mod_proxy_http.so
LoadModule proxy_ajp_module libexec/mod_proxy_ajp.so
LoadModule proxy_balancer_module libexec/mod_proxy_balancer.so
ProxyPass               /wiki           http://localhost:8081/wiki
ProxyPassReverse        /wiki           http://localhost:8081/wiki

(I don't actually know how many of the LoadModules I really need btw, I suspect I don't need the last 2)

Here comes the bad news. The apache2 that comes with CoolStack doesn't come with the mod_proxy.so modules, which means that I can't use it for proxying. I tried copying the original mod_proxy.so modules to the coolstack one and loading it up, but when I tried to access the wiki, my system pretty much froze and the disks churned like mad. Had to kill the webserver to get things back to normal.

How did I fix? I reverted to the original Nevada supplied webserver, and configured the CoolStack based one to run on a different port which I can use when I want to use phpPgAdmin. At some point I'm going to have to solve the problem permanently, probably some further investigation, and trying to figure out exactly what's going wrong. But for now, its doing what I need, I just hope I don't want to run a php based app anytime soon.

tags: Sun Solaris CoolStack AMP

CEC2007: Here we come

2007-08-10T06:34:16+00:00

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables.

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

As per last year, I'll be blogging my perspectives of the conference again.

tags: Sun suncec2007

New JET Resource

2007-08-10T06:15:47+00:00

Sun has just launched http://wikis.sun.com, which is great because it allows me to use it for putting all the JET info there rather than here.

I have created a space there called JET, which will become the primary Sun location for all information relating to the Jumpstart Enterprise Toolkit.

Have a look: Jet wiki

tags: Solaris Jumpstart JET

Home Server tweaking

2007-08-01T03:47:23+00:00

I've been working on my home server again (remember.. the Ultra40 I won at CEC last year) . I've been lazy for the past few months, and been stuck at Nevada Build 63 for a while. While I've been live-upgrading my laptop with every Nevada release, I kept on finding really good excuses to NOT upgrade my home server.

The main issue is that it actually does real work. It's my home DHCP, SMTP, Samba, Web, Jumpstart, (and JET development) and print server. The household tends to notice when it's down. (especially since all the home directories are mounted off it, which means that everything stops when its down. We get real tears in our house when people can't do their homework!)

The fact that I use Live Upgrade means that downtime is usually just a reboot. Chris Gerhard, who has a similar home set up upgraded before me and found a samba problem, which caused me to delay upgrading until I knew it had been fixed, then we had a minor problem with gnome-terminal character corruption, finally, gaim was replaced by pidgin, and in doing so, support for one of the IM protocols was kind of accidentally broken.

I don't have a problem being on the bleeding edge on my laptop, cos I can work around most of the issues, especially since I do most of my work on my home server.

Anyway, the long and short of it was that I was just about to do some other work on my Ultra40, namely try and get apache with php working (among other things), and decided it would be prudent to upgrade to Nevada Build 69 before I did anything.... just to be sure.

Happy to report that it all went swimmingly (as usual), and downtime was in fact reduced to a single reboot, which them indoors didn't even notice.

One interesting point. On rebooting and starting gnome, it automatically detected my USB plugged in HP printer, and gave me a pretty little "Add Printer Queue" window which allowed me to add my printer. Unfortunately it didn't detect that I had already added it manually many months previously. I suppose one can expect that to happen as new features begin to overlap manual workarounds. I'm pretty sure that printing should just work straight out of the box on a fresh Build 69 install without any manual intervention.. but I'm not going to bother to try.

I then proceeded to get my AMP working... topic for my next article.

tags: Solaris SUN X86 LiveUpgrade AMP

Do you Tiddlywikle?

2007-05-31T02:47:14+00:00

Bruce Porter just sent me a link to something called TiddlyWiki.

It's author JeremyRuston, describes it as: a reusable non-linear personal web notebook.

In short its a wiki on an html page. You don't need ay server-side logic, just a web browser.

And the way it works is pretty cool too!

Printing on Nevada b56

2007-01-29T14:16:12+00:00

I got asked in a comment in my previous article to describe EXACTLY how I got printing working.

I've got an HP5150 USB printer. I plugged it into a convenient USB port on my Solaris server, and then used the printmgr gui to set it up. I selected Use PPD, and in the Printer-->New Attached Printer form, filled in the following:

Printer Name: unix
Printer Port: /dev/printers/0
Printer Make: HP
Printer Model: HP DeskJet 5150
Printer Driver: Foomatic/hpijs (recommended)

This allowed standard Unix printing to work. (In short, it pretends to be a standard postscript printer, and you can send pretty much anything to it)

Solaris - Solaris

I also turned on the ipp-listener service using svcadm. On a remote Solaris client (my laptop, also running Solaris Nevada b56), I used printmgr again, and did Printer-->Add Access to Printer. This uses the ipp service by default if available. lpstat -v shows:

# lpstat -v
system for unix: zaphod (as ipp://zaphod/printers/unix)

The printer name is "unix" and my server is called "zaphod". With that, I could print from my laptop to my printer attached to my home server.

Samba Printing

The Samba printing was a little more troublesome. In short, all the Windows machines, and any machine for that matter trying to print natively to my "unix" queue had difficulties because it was expecting either Postscript or plain text. I could have set all my printers up as generic Postscript printers, but decided I'd rather print natively. To do this I could have probably played with the filters so that it passed through everything, but that seemed too much like hard work. Instead a set up a different queue (to the same printer) which would act as a "raw" queue.

#!/bin/sh
lpadmin -p hp5150 -v /dev/printers/0 -T unknown -I any
accept hp5150
enable hp5150
lpadmin -p hp5150 -o nobanner

I think my smb.conf was already pretty much properly configured for printing, but I'll repeat the relevent bits just in case:

# If you want to automatically load your printer list rather
# than setting them up individually then you'll need this
   load printers = yes

# NOTE: If you have a BSD-style print system there is no need to 
# specifically define each individual printer
[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
# Set public = yes to allow user 'guest account' to print
   guest ok = no
   writable = no
   printable = yes

To set up the printer on the local side, I had to manually install the HP5150 drivers, and then add the printer using the Add Printer wizard in Windows. The key to getting this to work properly, is to set it up as a "Local Printer", then select a Local Port, and specify the Local port as (in my case) as: \\\\zaphod\\hp5150

Summary

I actually configured all of this when both my home server and my laptop were running Nevada B54, so it is possible that it is no longer necessary to set up 2 queues, but I've LiveUpgraded and the settings were retained, so I've had no need to change. A lot of things seem to have changed in the Solaris printing model in Solaris Nevada, but I've found it is now a lot easier than it ever was. (And it is the expectation that it is going to be really difficult that makes you do more than you need to). If you trust that it just works, and use the printmgr tool, I think it is difficult to go wrong.

tags: Solaris SUN X86 samba printing

It just works!: Zones + ZFS + BrandZ + LiveUpgrade

2007-01-26T11:06:33+00:00

I've successfully LiveUpgraded my home server twice so far. (I've been slightly lazy and decided to do alternate builds, so I've gone from Nevada b52 to b54 to b56. Apart from the minor zfs issues which require me to:

Create an exclude file for the lucreate so that it doesn't try to copy my zfs partitions.
Manually empty mountpoints where some of my zfs filesystems mount.

it all works really well.

I'd shied away from creating zones because LiveUpgrade didn't support zones in the earlier builds, but I've now been assured that it works fine now.

Zones + ZFS. WOW

It just works. When you create a zone, if it detects that your zonepath is on a zfs filesystem, it'll automatically create a new zfs dataset for you for the zones you are creating. You don't need to do anything special, it just does the "Right Thing(tm)".

Furthermore, if you decide to clone a zone, and it's on a zfs filesystem, it simply creates a snapshot of the cloned zone for you. You don't need to do anything special, it just does the "Right Thing(tm)".

This means that once you've created your first zone, all your additional zones can be created and booted in a matter of seconds. Talk about rapid provisioning!

The icing on the cake: BrandZ

I've simply followed the instructions here and I've now got CentOS running in a zone, running on top of Solaris. Got Skype working!!

I love it when a plan comes together. We've now got some really clever integration of some individual cool Solaris technologies. The combined benefit is compelling.

Just waiting for ZFS boot to become available (well, more available than it is), and my home server aspirations will be complete.

tags: Solaris SUN X86 ZFS BrandZ

Addendum: Building/Installing/Configuring....

2006-11-22T06:55:12+00:00

I've been checking my spamassassin scores, and wasn't particularly happy with it's ability to rate SPAM. Given it's reputation I was pretty sure that it was due to a configuration error on my part.

I had a sneeky suspicion that not all the "tests" I was running were actually happening. I ran spamd in debug mode, and started seeing a bunch of interesting errors. In short, I was missing Net::DNS, so anything that required any sort of DNS lookup was failing.

Simply did the perl -MCPAN -e 'install Net::DNS' thing and that got DNS lookups working.

Then started to debug the other errors in the log. Apparently the bayes plugin was having problems talking to my bayes files because I didn't have DB_File. Tried the CPAN trick again, which failed because it couldn't find db.h. Turns out I didn't have Berkeley DB installed. Short trip to sunfreeware.com later:

db-4.2.52.NC-sol10-intel-local

It installs in /usr/localBerkeleyDB.4.2 . Given that CPAN was looking in /usr/local/BerkeleyDB, I created the link, and hey presto perl -MCPAN -e 'install DB_File' worked fine.

At that point I ran sa-learn against my Junk folder to generate some content, and checked the permissions of the bayes files. (Note: bayes_path in the local.cf for spamassassin needs to contain the prefix of the bayes files. e.g.:

use_bayes 1
bayes_path /export/home/spamd/bayes/bayes
bayes_file_mode 0666

# ls /export/home/spamd/bayes/
bayes_journal  bayes_seen     bayes_toks     bayes.mutex

Now that its all sorted, I'm getting much higher SPAM scores, and it seems much more accurate.

Some would say I should have just used Blastwave, but this was more fun, and a good learning experience.

tags: Solaris SUN X86 spamassassin