Technical Musings

LSI 2008 SAS Chipset (mps0) problems on recent FreeBSD versions

2018-01-08T19:18:00.000-04:00

I've encountered a problem with never versions of FreeBSD and my LSI SAS 2008.

I have a number of LSI 2008 based HBAs in service in my FreeBSD storage devices. Most are re-flashed Dell PERC, some are IBM, some are branded LSI.

This particular card is flashed to MPT2BIOS 7.11.10.00 (2011.06.02) in IT mode.

Under FreeBSD 10.2 (and many earlier versions) I have no problems with it.

Attempting to install FreeBSD 11.1, I couldn't get FreeBSD to detect the HBA. I could see my two SES devices (SAS Expanders), but no mps0, no drives.

I did a bit of troubleshooting, and found this across the FreeBSD versions:

10.2 - No Problems
10.3 - Not Tested
10.4 - mps0 shows up, all drives show up, but I have screens and screens of SCSI SENSE errors, can't import zpool's, etc.
11.0 - mps0 dosen't show up, no drives show up, only ses0 ses1
11.1 - same as 11.0

I unfortunately wasted too much time troubleshooting my inactive server. Once I knew what I needed to do to get it online again, I couldn't spend more time to file a bug report and help track this down.

I did encounter this thread, sounds very similar to my problem:

https://forums.freebsd.org/threads/63800/

For now, I'm back to 10.2 on this server until I have the time to fire up a parts machine that I have in storage.

Let me know if you're running into something similar, or have any suggestions.

FreeBSD 11.x Guest Keyboard Stops Working in VMware

2017-12-31T16:36:00.000-04:00

If you're firing up a 11.x instance of FreeBSD in VMware, you may find that the keyboard works until you shift focus to a different window. Shifting focus back to the FreeBSD window will not give you keyboard input again.

For me the fix was this:

In the .vmx file (or in the GUI under Edit Settings -> Options -> General, then Configuration Parameters...) add this line:

Let me know if you needed to do something different to make it work.

keyboard.typematicMinDelay = "2000000"

Let me know if you needed to do something different to make it work.

Installing vCenter 6.0.0 gives: Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details.

2015-04-01T14:06:00.000-03:00

I'm rolling out a new vCenter 6.0.0 install from scratch to test our environment on it, and quickly ran into a show stopper:

Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details.

I was installing on a fresh new Windows 2012R2 server with all the updates, but I hadn't installed the .NET 3.5 feature.

The release notes from VMware mention this exact error, but in the context of an ipv4 address issue, so that wasn't much help, as all of my ipv4 addressing was correct.

After a bit of poking around, I noticed that a 1603 error in a MSI is often due to .NET being missing.

After installing .NET 3.5 and updates, the install worked properly.

There is no mention of .NET being required in the Release Notes, the Windows Requirements, etc.

Hopefully that saves someone a bit of time. :-)

Now let's see what that new Web Client performs like.

Quick iperf test on FreeBSD 10.1 running as a VM in ESXi 5.5

2015-03-01T19:05:00.003-04:00

Take this with a grain of salt, as I really haven't done enough testing to state anything conclusive.

However, under ESXi 5.1u3, I was only able to get my iperf scores up to 2/3Gb/s, with the occasional 5Gb/s bust on my 10Gb/s network.

With ESXi 5.5u2 (patches as of today's date) I am able to get scores like this fairly consistently:

[ 3] 0.0- 5.0 sec 5.70 GBytes 9.79 Gbits/sec

[ 3] 5.0-10.0 sec 5.69 GBytes 9.77 Gbits/sec

[ 3] 10.0-15.0 sec 5.69 GBytes 9.77 Gbits/sec

[ 3] 15.0-20.0 sec 5.70 GBytes 9.80 Gbits/sec

[ 3] 0.0-20.0 sec 22.8 GBytes 9.78 Gbits/sec

Not bad, not bad.

Even if the scores are not accurate, it's at least showing a dramatic improvement over my 5.1 configuration.

This is from a FreeBSD 10.1 VM under ESXi 5.5 to a bare-metal FreeBSD 10.1. VM-to-VM performance is a bit lower, but still quite good:

[ ID] Interval Transfer Bandwidth

[ 3] 0.0- 5.0 sec 4.89 GBytes 8.41 Gbits/sec

[ 3] 5.0-10.0 sec 5.46 GBytes 9.37 Gbits/sec

[ 3] 10.0-15.0 sec 4.32 GBytes 7.42 Gbits/sec

[ 3] 15.0-20.0 sec 4.87 GBytes 8.37 Gbits/sec

[ 3] 0.0-20.0 sec 19.5 GBytes 8.39 Gbits/sec

The FreeBSD VM's are still vmx-08, and I'm using the built-in vmx driver in FreeBSD 10.1

This is also with the default configuration for sysctl values - I was planning on building a small script to check different tcp settings, but with speed like this, there are more pressing things to worry about.

ESXI 5.5u2 Hangs during upgrade from 5.1

2015-03-01T18:52:00.001-04:00

Quick post:

If you're upgrading from ESXi 5.1 to 5.5 Update 2, and you don't have all of the network ports hooked up on your host, you may hang the upgrade process in an infinite loop.

After upgrading two other hosts from 5.1 to 5.5U2 within ~30 minutes, I was concerned that the third server was still at it after 2 hours.

Watching the console carefully from KVM showed it complain for about 5 seconds about a missing MAC address. Looking at the server's physical connections, it did indeed have one network port not connected. Most of my servers are Dell Blades, so all network ports are always hooked up, however, I don't completely trust putting all of my computing in one box, so I still retain a fairly beefy external server in case my blade enclosure ever bites it.

This particular server had been running for years with that port unplugged, but it only now posed a problem.

Connecting the port solved the issue.

Now if the vCenter Web Client didn't suck so badly, I'd be having a good night. :-)

Alternate Method for Converting VMware vCenter Server from Physical to Virtual (VM)

2014-04-17T10:02:00.000-03:00

At some point you are going to figure out that the advantages of having your vCenter Server as a VM outweigh the risks, and you're going to want to convert your existing physical vCenter into a virtual one.

There are a few ways of going about this, including just migrating vCenter to a fresh VM.

However, if you're stuck using only the VMware Converter standalone, you may find it frustratingly hard to make the conversion.

If you throw caution to the wind and just convert the vCenter Server on the fly while it's running (using the sync command to try an clean up after), you may get lucky, but more than likely you will have database corruption and other fun.

If you follow VMware's procedures for converting the machine and shut down all the services, the SQL databases, etc. then the VMware Converter can't connect to your vCenter to add the machine to the cluster.

One way around that would be converting to a seperate ESXi host that isn't in your cluster, and later join the host to the cluster, bringing vCenter with it.

However, that's a bit difficult when you are remote to a network (like I was) so there is no oppourtunity to provision physical hardware.

The answer? Double-Conversion.

Shut down all of the necessary services on your vCenter server, and run the Standalone VMware Converter.

Once there, simply convert your vCenter server to a VMware Workstation/Fusion .vmdk image.

Once that is complete, reboot vCenter so all the services are functional again, and convert that .vmdk into a VM on your cluster with the Standalone VMware Converter. vCenter treats this like any other vanilla VM, nothing special is needed.

Yes, you have a period of time where the old vCenter is running and controlling the network, and you're going to be 'going back in time' so to speak to the time of the conversion, but it's no worse than restoring vCenter from a backup.

I connected my new vCenter VM to a dummy network, and did all the final cleanup on it (installing vmxnet3, pvscsi, etc).

Make sure vCenter is running. If you have 5.1 with the Single Sign On (SSO) service, it will probably keep you from running vCenter.

It's not an error in the conversion process. When enough hardware changes, it breaks the SSO service, which still starts normally. However, vCenter won't start. The full article from VMware about this condition is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036170

The set of commands I had to use was:

set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m MASTER_PASSWORD

(Where MASTER_PASSWORD is your master password when you installed vCenter/SSO)

When you have everything on your vCenter VM running smoothly, shut down your physical vCenter server, and connect your vSphere Client to the raw ESXi host holding your vCenter VM. Once you start it, you'll be able to connect to vCenter as per normal.

Make sure you've read the various forums on how to protect yourself from a bad situation with a virtual vCenter server.

Oh, and if you can keep the old physical server around for a bit, you have a quick emergency backup in case you run into a problem cold-starting your datacenter.

FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

2014-03-13T19:22:00.001-03:00

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay.

Using MessagePack as Transport Beween a Python Tornado Server, and a JavaScript Browser Client

2014-01-15T14:41:00.000-04:00

First of all: Good Luck. :-)

MessagePack is a great binary serialization format, allowing you to transport globs of variables across whatever transport you like. It's very similar to JSON, but leaning towards a tighter packet size , as shown in the graphic I took from the msgpack.org site above.

MessagePack also has a great variety of languages supported. If you code it, you can probably find a msgpack.X file on their site.

Now the problems:

1) MessagePack has evolved over the years, and updated the specifications for the protocol. This isn't well documented, nor is there a "Version 2.1" or similar tag on the specification documents. If you look around, you can see that some links refer to this as v5, but how do I know that we're not at v8 and that linking site is stale? This makes it less obvious about what version a msgpack library supports, which means it's more difficult to choose compatible msgpack libraries when you're using different languages.

2) No CHANGELOG, because there are no revisions/versions. Good luck seeing what msgpack was 3 years ago without detective work.

3) Because the specification has changed, some of the MessagePack libraries are up-to-date, others are years behind. The pure Python u-msgpack.py is up-to-date as of Dec 2013, but the JavaScript msgpack.js (v1.05) is not.

Through trial and error, I was able to get the JavaScript version working with the u-msgpack.py version, but not the standard msgpack.py (which is a wrapper for the C library).

However, since u-msgpack.py is following the new specification, it can create msgpacks that the JavaScript version can't decode, particularly when Python is sending a long string.

I've patched msgpack.js, and have contacted the author (uupaa) to see if he has an updated version. If that dosen't pan out, I'll fork and update the code so others can have access to it. I'm also interested in patching it to work with JavaScript Blobs, so I can send binary data to it.

There is one small error in the u-msgpack.py file:

The problem is in _pack_string. It's calculating the length of the string before it's encoded to UTF-8.

I think you must encode the string before you find the length of it, as some characters need to encode as double-byte or longer.

An example would be the French name Allagbe, or the French word precedent , where the 'e' is with Acute (http://www.fileformat.info/info/unicode/char/e9/index.htm)

Python's encoder makes this b'Allagb\xc3\xa9', which is one byte longer than than the original string.

u-msgpack encodes this as b'\xa7Allagb\xc3\xa9' - notice how it's only 7 bytes long - it's trimming the \xa9 char from the msgpack.

When you feed this trimmed string through a .decode('utf-8') method, you'll crash with a python error : 'utf-8' codec can't decode byte 0xc3 in position 6: unexpected end of data

The solution is to encode to UTF-8 before calculating the string length, as detailed below:

def _pack_string(x):

x = x.encode('utf-8')

if len(x) <= 31:

return struct.pack("B", 0xa0 | len(x)) + x

elif len(x) <= 2**8-1:

return b"\xd9" + struct.pack("B", len(x)) + x

elif len(x) <= 2**16-1:

return b"\xda" + struct.pack(">H", len(x)) + x

elif len(x) <= 2**32-1:

return b"\xdb" + struct.pack(">I", len(x)) + x

else:

raise UnsupportedTypeException("huge string")

With this patch in place, I'm able to pass plenty of French and Italian names through u-msgpack.py to msgpack.js

Now.. if you're trying to make all of this work across Tornado's WebSockets, keep in mind that what you get from Tornado will probably not fit into u-msgpack.py's .unpackb() function properly, but I'm out of time to detail that fun today.

In the end: I'm happy with MessagePack, but I wish it was a little easier to get into.

Python - The One True Language

2013-12-20T18:46:00.002-04:00

I find it interesting when people bad-mouth Python - To the point where I felt the need to write this little blog article.

First, I need to state the obvious - Python is the 'One True Language'. :-)

I say that as someone who boot-strapped himself from Commodore BASIC (PET, not that fancy C64 thing), to MS-Basic, to Turbo C. Many an hour was logged in front of this little window..


Sexy IDE for 1987

C was clearly my first true love (well perhaps Angeline who sat in front of me in Gr. 7, but she never responded to me like an 16550 UART call in C did). Compared to the other languages at the time, it gave you incredible power, without the difficulty of ASM.

I've had the (mis?)fortune to have programmed in just about every popular language at one time of another in my 30+ year career, and I must say Python is the first language that usurped C's place in my heart.

My only regret is that I didn't try Python earlier.

A few years ago I had a choice to make between Perl and Python as my next scripting language for Unix tasks that were too complex for shells or PHP, and I made the choice to go for Perl, as evidenced by the 2006 receipt still in my O'Riley Programming Perl book on my shelf.

At the time it was a close call; Perl had more popularity, and looked to be continuing to grow. This was an important point when you want to invest time in a new language (anyone remember the D Language?). I also think I liked the camel on the Perl book cover more than the python as well... who was into snakes? (I did not realize the Monty Python reference at the time). These things do matter when you're down to brass tacks.

So the books were purchased and well read, and a few years were spent in Perl.

I enjoyed the C-like syntax, and relaxed feel. However, it wasn't everything that I wanted.. and I'm not sure exactly how I stumbled on Python again, nor what the light bulb moment was - I worked a lot back then, so memories are fuzzy. Something was lacking, something kept me driving forward into new languages.

Whatever made me stop and reconsider, I must say I'm very happy to be in Python now - and nearly 100% of what I write today is Python.

I need a bit of Javascript for DOM work, and a bit of C for some very heavy transactional stuff, but even some of my stress tests and message passing routines are now Python.

Why do I bring this up?

I feel a number of programmers looking at Perl/Puby/Python may fall into the same situation, even today when the Python community continues to gather strength. No one is really out there bashing Perl or Ruby, but Python has it's haters.

Some people love to badmouth Python.. I don't really understand them - I think it must be a machismo thing. The fact that it's an interpreter, and that it doesn't thread well (you shouldn't use threads boys.. that' 1990's tech) seem to enrage some.

If you're considering investing time into Python or XYZ, then my suggestion is to take the time to try Python first. I'm not going to get into a pointed discussion on the pros and cons of the various languages - That's covered in detail across the web.

Instead I'll state: They are all good languages, but if you take the time to learn Python and really 'drink the kool-aid', I think you will be rewarded.

Python brings a joy back into programming that I haven't felt since I used to sit in front of that little Turbo C IDE back in 1988. I've hit a productivity level that is a career milestone thanks to Python.

Next: PyCharm - The Turbo C IDE for Python. :-)

PS - I predict an OS entirely in Python soon - Obviously with a special interpreter, but it will be here.

FreeBSD 9.1 to 9.2 Upgrade with freebsd-update

2013-12-11T07:59:00.000-04:00

FreeBSD 9.2 brings a very nice update to ZFS : LZ4 compression. Upgrading from 9.1 to 9.2 is quite easy if you have a standard install (no customization of the code).

The basic process, which should work for other versions as well is this:

1) Get your system up-to-date with the latest patches for your version of FreeBSD. This isn't necessary, but I think may avoid some problems.

freebsd-update fetch
freebsd-update install
reboot

2) Upgrade the kernel

freebsd-update upgrade -r 9.2-RELEASE
freebsd-update install
reboot

3) Update userland (finishes off the 2 part install)

freebsd-update install
reboot

4) Update pkg if needed

pkg update

5) Enjoy

With my 9.1 -> 9.2 update, I didn't have any messy mergemaster conf files to pour through, nor did my packages notify me that I needed to upgrade them - One of the more painless upgrades I've made.

pkg - FreeBSD's new Package Manager is Finally Working

2013-12-11T07:49:00.001-04:00

I was about to write a small entry about how to hook up FreeBSD's next-generation package manager (pkg) to PC-BSD's repository, as FreeBSD wasn't supplying package updates due to the security incident that occurred in Nov 2012.

However, pkg is now able to pull properly from FreeBSD sources, and it's a nice little program.

For all those still using the old pkg_* type tools, just type pkg and follow the prompts.

So far I am finding it easier to upgrade and maintain my packages with pkg, which is always a good thing.

Boot from ZFS root with FreeBSD 9.2

2013-10-07T20:26:00.000-03:00

(Update: The newly released FreeBSD 10 makes a ZFS setup in the menu system easy.. give it a try)

Booting from a ZFS root has always been a dream of mine, but I've never managed to make it work.

Since I'm deploying new FreeBSD 9.2 images for testing in my lab, I thought this would be a good time to give it a shot again.

I based this off the FreeBSD install page, with some simplification and a few small additions.

I took a freshly installed FreeBSD 9.2 system to act as my workstation for this, and plugged in a 8 gig USB Flash drive to be my ZFS root.

In my setup, da0 is the booted system, da1 is the blank USB Flash Drive.

I often do not use swap partitions on USB Flash, as it can prematurely wear the drive. I make my USB Flash drives as disposable as possible, so keeping crash dumps isn't a concern.

NOTE: Solaris always called it's root ZFS pool 'rpool' and I recommend you do the same, as it's a existing standard. My use of 'zroot' here is because I was only fooling around... and seem to be too lazy to change the text below without testing for typos.

Here is what I have executed to make it work:

gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot -l boot0 da1
gpart add -t freebsd-zfs -l root0 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

#This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the #disk. If this happens for you, run:
#
# sysctl kern.geom.debugflags=0x10

kldload /boot/kernel/opensolaris.ko
kldload /boot/kernel/zfs.ko

zpool create zroot da1p2
# DO NOT FORGET TO MAKE IT P2! Specify da1, and zfs will kill your gpt tabes
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

#Make sure your FreeBSD 9.2 CD is in your drive

mount -t cd9660 /dev/cd0 /mnt

cd /mnt/usr/freebsd-dist

cat base.txz | tar --unlink -xpJf - -C /zroot
cat kernel.txz | tar --unlink -xpJf - -C /zroot
cat doc.txz | tar --unlink -xpJf - -C /zroot
cat lib32.txz | tar --unlink -xpJf - -C /zroot
cat ports.txz | tar --unlink -xpJf - -C /zroot
cat src.txz | tar --unlink -xpJf - -C /zroot

chroot /zroot

echo 'zfs_enable="YES"' > /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'hostname="freebsd92zfs"' >> /etc/rc.conf

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

tzsetup

cd /etc/mail
make aliases

exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache
zpool export zroot

I then pull the USB Flash Drive, pop it into a spare machine, and presto, it's booting.

I'll fill in more about this as I play with ZFS on root for my USB Flash Drive systems.

(FIXED) Network mbuf Leak / Exhaustion in FreeBSD 9.0 / 9.1

2013-03-12T00:32:00.000-03:00

UPDATE: Patch included below

There is a rather annoying bug floating around in FreeBSD 9.0 / 9.1 regarding network mubf leaking to the point of mbuf exhaustion.

These is a problem report (PR) filed about this from last year, but it looks to be abandoned.

http://www.freebsd.org/cgi/query-pr.cgi?pr=165903&cat=

I am experiencing the same mbuf leak on fresh 9.1-RELEASE machines (AMD64). Most of my machines are ESXi 5.1 VM's running the e1000 (em0) NIC. This VM is stock, just one freebsd-update done, nothing custom.

I have also experienced this condition on an older 9.0-STABLE from Jul 1st 2012. I did not notice it much before that date, but I can't tell for sure. I have a few machines on that build that I still use, so confirmation was easy.

I do not experience the error if I load up vmware tools and use the vmx3f0 adapter, it's just with em0.

I have set the mbufs to very high numbers via sysctl kern.ipc.nmbclusters=322144 to buy more time between lockups/crashes. Most often the systems stay functional, they just need a reboot or more mbufs if I add them. Some times the servers lock up or crash as I ifconfig down/up the adapter or attempt to add more mbufs via sysctl.

Is anyone else able to reproduce this?

I have attempted to update the PR or post to the list, but the freebsd.org server and my mail server no longer seem to get along. I'll have to troubleshoot that later this week.

UPDATE Apr 19th 2013:

Gleb Smirnoff was kind enough to quickly forward me a patch that fixed the problem for me. You will need to apply this to usr/src/sys/netinet/if_ether.c

I've now run for 2 days, and my mbufs have not increased at all. Thanks for the quick response from the FreeBSD-Stable list.

Index: if_ether.c
===================================================================
--- if_ether.c    (revision 249327)
+++ if_ether.c    (working copy)
@@ -558,13 +558,13 @@ in_arpinput(struct mbuf *m)
     if (ah->ar_pln != sizeof(struct in_addr)) {
         log(LOG_NOTICE, "in_arp: requested protocol length != %zu\n",
             sizeof(struct in_addr));
-        return;
+        goto drop;
     }

     if (allow_multicast == 0 && ETHER_IS_MULTICAST(ar_sha(ah))) {
         log(LOG_NOTICE, "arp: %*D is multicast\n",
             ifp->if_addrlen, (u_char *)ar_sha(ah), ":");
-        return;
+        goto drop;
     }

     op = ntohs(ah->ar_op);

Passthrough PCIe devices from ESXi to FreeBSD - Leave MSI Enabled to Avoid Interrupt Storms With More than One vCPU

2012-12-18T19:42:00.000-04:00

If you want to pass a PCIe device (such as a LSI SAS controller) through ESXi to a FreeBSD VM, you're probably aware that you may need to disable MSI/MSI-X interrupts to make it work.

However, I'm finding that while this setting will make the system boot, if you add a second vCPU, you will quickly stat to see Interrupt Storm messages on your console, and one of your CPU's will be nearly 100% used on interrupt tasks.

The easy fix for me? Only disable MSIX, not MSI.

Here are the relevant lines from my /boot/loader.conf:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

All of my interrupt storm issues are now gone.

VMWare Enhanced vMotion Compatibility (EVC) Not Needed between AMD Bulldozer 6234 and AMD Piledriver 6348 on VMware ESXi 5.1

2012-12-12T17:20:00.000-04:00

I recently upgraded one of our datacenter servers from an AMD Bulldozer 6234 to a AMD Piledriver 6348.

I did find the new chip was faster for memory/CPU operations in our VM's, but I'll post about that in a different article.

What surprised me was that I didn't need to turn on EVC to allow VM's to migrate between hosts running the different CPU's.

Either VMware 5.1 doesn't recognize the new instruction set, or they don't matter in terms of migration.

Either way, I can migrate to the newer chips without worry, and since the 63xx are socket compatible with the 62xx and 61xx series, I have a fairly easy upgrade path for our servers if I wish.

ESXi 5.1 running FreeBSD 9.0 / 9.1 and VMware Tools

2012-11-22T19:43:00.000-04:00

Note - I have another article for ESXi 5.0. These instructions won't work for a 5.0 box.

ESXi 5.1 includes support for the official VMware Tools on FreeBSD 9.0 (yay!).

I'm somewhere between 9.0 and 9.1 RC3, but I feel these instructions will work for either.

I'll be testing the throughput of the vmxnet3 driver under FreeBSD shorty. For now, I can just confirm that bothe vmxnet2 and vmxnet3 drivers show in FreeBSD, and can ping.

Here's a quick script to install it for you.

You will need to start a VMware Tools install from the guest before running this.

echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

ESXi 5.0 running FreeBSD 9.0 / 9.1 and VMware Tools

2012-11-22T19:32:00.000-04:00

Note - I have another article for ESXi 5.1. If you are running 5.1, these instructions won't work for you.

Installing the proper VMware Tools for FreeBSD 9.0 / 9.1 can be a pain.

The hard work to figure this out has already been done by others (http://ogris.de/vmware/), so here is a quick script that will install the official VMware Tools using Dru's patch.

Make sure to start a vmtools install from your vsphere console so ESXi will make the CD available to you.

If you run into problems with a missing library when you try and execute /usr/local/bin/vmare-toolbox-cmd, try looking here (http://lists.freebsd.org/pipermail/freebsd-questions/2010-June/217718.html) it solved the problem for me.

(Cut-n-Paste into a terminal window with root access)

echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
cd lib/modules/source
tar xvf vmblock.tar
tar xvf vmmemctl.tar
tar xvf vmxnet.tar
tar xvf vmxnet3.tar

fetch http://ogris.de/vmware/vmxnet.diff
fetch http://ogris.de/vmware/vmxnet3.diff

echo #
echo # If it's thinking you have a previously applied patch (-R) say yes.
echo #

patch -p1 < vmxnet.diff
patch -p0 < vmxnet3.diff

cd vmblock-only
make && make install
cd ..

cd vmmemctl-only
make && make install
cd ..

cd vmxnet-only
make && make install
cd ..

cd vmxnet3-only
make && make install
cd ..

cd ~/vmware-tools-distrib

./vmware-install.pl

echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

Blogger Code Syntax Highlighting

2012-11-04T15:05:00.001-04:00

I really like Wordpress.org's Syntax Highlighter.

http://wordpress.org/extend/plugins/syntaxhighlighter/screenshots/

Particularly the button to copy the raw code without the line numbers.

Unfortunately, this is not easy to implement on Blogger.

The closest I have managed is from this blog:

http://www.cyberack.com/2007/07/adding-syntax-highlighter-to-blogger.html

Which does work, but we're missing the important buttons. Here's a demo of it in action on my saturate.c code - Note Blogger still completely mangles my code after a few lines, and we're missing the important buttons that Wordpress has.

Time to switch to Wordpress, or does anyone have any bright ideas?

(Update - Pastebin seems to work well for me, but it's an external link, not quite the same thing. http://pastebin.com/4SexdvLq )

Saturate.c



#define _FILE_OFFSET_BITS 64  /* enable large file support  */
#include 
#include 
#include 
#include 
#include 

float nCM=0; /* count of 100's of megs we've written */


void finish(const char *where) {
        printf("%s: wrote %.1f gig file\n",where,0.1*nCM);
        exit(0);
}

void hitError(const char *where) { perror(where); finish(where); }

void writer(int id)
{
        char c=0; /* The byte we're writing */
        int cm=100*1024*1024;
        FILE *f[50];

        int     r;
        int     block[262114];
        int     i;
        char    fName[40];
        int     numFiles=4;

        printf("This is the %i writer, going to create %i files.\nOpened : ", id, numFiles);

        for(i=0; i
(end Code)

Fixing "Open File - Security Warning" prompts in Windows 7 / 2008

2012-06-26T11:39:00.001-03:00

If you enable Folder Redirection from a GPO, or even if you are trying to run a program from a non-local source, you'll run into this dialogue box:

For me, it was particularly annoying as it would happen when accessing any item from the start menu on our remote desktop server, as we were redirecting the AppData folder as well.

The fix is through GPO. I assume you know how to make a new GPO, and so to save time, here is the location.

User Configuration - Administative Templates - Windows Components - Internet Explorer - Internet Control Panel - Security Page ... then it's Site to Zone Assignment List

I set both values for my file server here. You could also use a wildcard like *.yourdomain.com if you liked.

2 is the value for Trusted Sites. Check the help in the GPO Management box for other options.

The only downside to this is that now the user can't add their own trusted domains, as once it's specified in GPO, it can't be altered by the user.

Running Windows 2012 Server RC on ESXi 5.0 U1

2012-06-06T10:44:00.004-03:00

If you're wanting to play with the upcomming Windows 2012 RC trial under ESXi 5.0 U1, you may find that it hangs if you're running your Virtual Machine as "Windows 8 (64 Bit)".

You need to set your Virtual Machine to "Windows 2008 R2 (64 bit) not "Windows 8 (64 Bit)". Once you make this change, it boots, and looks to run smoothly.

Vmware tools looks to install correctly, I can power up and down, and I can use the VMXNET3 network adapter in Windows. I also have the PVSCSI adapter enabled, and it looks to be running well.

I'm curious to test out it's built in iSCSI Target / clone abilities for running diskless (iSCSI Boot) Windows 7 Pro workstations.

Oh, and if you want to be able to use 2012 / Windows 8, check this site for where to find simpe things like shutdown and control panel. I'm not sure it's wise to change such a staple of Windows useage, but it does make for a cleaner interface.

So far, I like the new interface.. we'll see how useful it is after extended use.

ZFS raidz - Make Sure To Select the Correct Number of Disks

2012-05-08T15:33:00.000-03:00

I've read that selecting the correct number of drives when building a raidz ZFS array is important if you are trying to maximize performance.

http://mail.opensolaris.org/pipermail/zfs-discuss/2010-September/044701.html

Of course the best way to maximize performance is to use a straight stripe or a mirror instead of a raidz, but that isn't always an option when you have price, power, or physical space restrictions.

Since I'm perpetually curious, I set up tests to compare the write speed of a 3, 4, 5, 6, and 7 drive raidz array.

I used my saturate.c program to put the arrays under heavy write load, repeated 6 times, and took avg and std deviation.

My results are not clean enough to post, but they would suggest that the number of drives is important - Follow the rules below;

RAIDZ1 vdevs should have 3, 5, or 9 devices in each vdev
RAIDZ2 vdevs should have 4, 6, or 10 devices in each vdev
RAIDZ3 vdevs should have 5, 7, or 11 devices in each vdev

With some luck, I hope to have the time to revisit the tests and obtain results that are postable.

Supermicro USB Boot Issues, H8SGL

2012-05-04T13:53:00.000-03:00

Thought I'd pass on a small tidbit;

A SuperMicro H8SGL(-F) won't boot from USB properly when there are more than 12 bootable items in the server.

In my case, filling my enclosure with drives surpassed this limit. Quite the surprise, as it was working great with 8 drives in, but at 16 drives, a routine maintenance reboot left it unable to start again.

The symptom is : It will just act like there isn't a boot sector on the USB drive.

I'm unsure at this stage if it's just USB, or if it's any bootable device.

I'm guessing that the MB's BIOS keeps a very short table of bootable items, and as it discovers more, the first to be discovered scroll off and are no longer available, even when they still list on a F11 Boot Selection.

I fixed it by turning off the BIOS boot option inside my SAS adapter's BIOS, and everything was fine.

It goes to show that you _ALWAYS_ need to reboot a server after any hardware/software changes, just to make sure. If I had learned about this problem when we were in some sort of urgent situation where I needed to quickly power cycle this server, well, that would have been a bad thing.

ZFS Testing - Saturate.c

2012-05-03T13:53:00.000-03:00

Benchmarking ZFS is hard.

ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.

I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.

As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.

The throttling-effect of using a single SAS card was one of the items I needed to check .

Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times. I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.

This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.

Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.

Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.

I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.

In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.

It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.

You can simply execute the file using time to check how long it takes like this;

time ./saturate

I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us. I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.

Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).

Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes. Not bad for a Free Operating System.

I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.

Code on PasteBin: http://pastebin.com/4SexdvLq

PowerShell Script to Set Public Folder Replication Schedule Recursively in Exchange 2007

2012-04-19T17:44:00.001-03:00

I've recently been tasked to replicate a lot of Exchange 2007 Public Folder data to another server. I'll blog the steps here shortly, but first, I had to write a quick power-shell script to do it, as there were over 100 public folders in the root public folder, and they were all set to not replicate.

I had already used the MS script AddReplicaToPFrecursive.ps1, but it doesn't set the replication schedule if it wasn't set before.

This script will walk through all of your public folders and set the replication schedule to "always"

It logs to C:\set_pf_log.txt and you can see two work files it also makes in C:\ - modify as you wish.

It's my first PowerShell script, so be gentle. :-) I don't know how to handle wrapping long folder names from the Get-PublicFolders command, so it won't process a long PF name.

# Set Public Folder Replication Schedules Recursively
#
# NOTE: If you have very long public folder names, they may wrap, and will not be handled correctly here.
#
# Define Varables
#

$filename_log = "C:\set_pf_log.txt"
$filename_pf_long = "C:\set_pf_long.txt"
$filename_pf_short = "C:\set_pf_short.txt"

#
# First, get a list of all the public folders and write them to $filename_pf_ong
#
write-host "Fetching Public Folders and writing to $filename_pf_long"
write-host "Fetching Public Folders and writing to $filename_pf_long" | out-file $filename_log -append

get-publicfolder "\" -recurse | fl identity > $filename_pf_long

#
# Now we need to trim the file, as it also outputs 'Identity : ' which we don't want
#

write-host "Trimming $filename_pf_long into $filename_pf_short"
write-host "Trimming $filename_pf_long into $filename_pf_short" | out-file $filename_log -append

get-content $filename_pf_long | foreach-object {$_ -replace 'Identity : ',''} > $filename_pf_short

# Now loop through the file, executing our action for each public folder
#
# NOTE - The action is commented out, using write-host to output to the console instead. If you want to make this actually run, remove the # in front of set-publicfolder
#

foreach($line in get-content $filename_pf_short)
{
    if ($line -ne "")
        {
        write-output "Processing $line"
        write-output "Processing $line" | out-file $filename_log -append
#         set-publicfolder $line -ReplicationSchedule Always
        }
}

write-host "Complete!"

A Call to Revive the Uniform Driver Interface (UDI)

2012-04-18T12:40:00.000-03:00

(UDI’s Home - http://www.projectudi.org/ )

Over the years, I have wished for some sort of universal driver, so that we didn’t need one driver for FreeBSD, one for Linux, one for Windows Xp, one for Windows 7, etc.

I started searching the internet to see if anything was in development on such a framework, and was surprised to see that it did indeed exist, and was already pretty much dead before I even knew it existed.

The Uniform Driver Interface (UDI) was, created by a cooperation of some big industry players (Intel, SCO, Adaptec, Sun, IBM, DEC, Compaq/HP, etc).

However, I notice Microsoft wasn’t part of it, which is one of the reasons it is currently languishing.

It’s specification was released in 2001 under a BSD-style license, so it’s very Open-Source.

During my 'autopsy-research', I found some muttering about it being as complex and bloated as CORBA, but I can’t find any proof of that.

I think the real damage to it was being blasted by Richard Stallman http://www.linuxtoday.com/developer/1998100500205OP

I understand where Richard Stallman’s stance on UDI is coming from, but I do not agree. I’ll get into why shortly.

First, let me state this;

The _BEST_ think for Free Software would be UDI, because the largest thing holding back Free Software Operating Systems is drivers.

Disagree?

- Try running quality accelerated X-Windows on FreeBSD, and a recent ATI card.

- Try booting/using OpenIndiana on a cheap desktop quality board with a generic Marvell or other SATA controller and Network Card?

- How’s your Intel GMA500 video working under Ubuntu?

This list is endless. Linux is in the best position for hardware drivers as it has the larger installed base, and so hardware vendors do have decent support. Solaris/(x)BSD’s seem to be next, and it quick drops off the face of a cliff after that.

If you wanted to revive the interesting Plan-9 OS, or if you wanted to branch from an existing Free OS and make some large changes to the kernel/scheduler, or you wanted to try and design your own OS, you’re now going to need to write or port drivers.

This is a huge amount of time wasted on trivial matters that shouldn’t be. I believe in re-useable code, and not doing the same thing twice. We need to understand that skilled programmers are a resource like anything else, and we shouldn’t be squandering their time across repeating the same work for different OSes.

Sure, it’s a rite of passage to code your own drivers – But when you have to code 27 drivers just to obtain basic functionality for your new OS, so you can work on what is really driving you to create that new OS, you’re wasting time.

Some Open-Source OS drivers come from the hardware vendor directly. A large percentage of the drivers in Open-Source OSes are reverse engineered. While some are very stable, fast, and feature-rich, many are not.

Why should the Free-OS crowd have to make do with lower quality drivers than those who run Windows?

Why should the programmers on the freebsd-scsi list have to deal with bug-stomping on reverse engineered drivers for a product that they didn’t make, when they could be working on the CAM structure (a product they are making)?

At the risk of over-simplifying Richard Stallman’s position, this is how I see his stance against it;

There is a chance that a UDI driver made by/for an Open-Source OS team would be used in a commercial product and possibly distributed by the hardware vendor.

And you know what? Yup, there is a chance in the very early stages of a switch-over to UDI drivers that something like that may happen. If you’re concerned, slap a GPL v3 license on the driver you code and be done with it.

However, no hardware manufacturer will ever take drivers from the Open-Source community and use them as their own for any considerable period of time– They will develop their own drivers, so they can properly support and control the quality and performance of their hardware.

If the hardware vendors want to keep them proprietary, so be it. This is another key point (I think) in Richard Stallman’s stance against UDI

- If the big hardware vendors could release closed source UDI drivers, then the pressure on the hardware vendors to release open-source drivers would cease.

That’s very true, and possibly a concern. However, I look at this as a game of chess with big-picture goals, and short-term compromises.

Yes, you may be taking a step backwards in the Open-Source drivers category, but you’d be _vaulting_ Open-Source OS’s miles ahead.

Just imagine a copy of FreeBSD, Ubuntu Linux, OpenBSD, or OpenIndiana with the same driver support as Windows.

As a user, think of what you could do with that system.

As a programmer, think of what you could concentrate on instead of tracing down bugs in reverse-engineered driver that so many people use with your software.

You do realize that with technology like WINE, and Thinapp (which works under WINE nicely), a good Firefox cross-OS implementation, we're closer than ever to properly challenging Microsoft's dominance on the PC market? If an Open-Source OS could run on anything that Windows runs on, what options would that give you?

UDI is something that the hardware vendors want (it’s why they all joined the working group). It’s something that any OS designer would want (less work for me? Great!), and it’s something that the end user will want (you mean all my hardware now works with FreeBSD?)

How do we get UDI started again? It’s a bit of a chicken-and-egg scenario:

1 - We need pressure on the Hardware Vendors to start releasing real UDI drivers for current hardware that people want.

2- We need pressure on the Open-Source OS developers to take the ProjectUDI sample implementations of UDI for their OS and integrate it into the recent releases of each OS.

With drivers available, and OSes to use said drivers, UDI will start gaining some ground.

If there are bloat or performance problems with UDI, they will be exposed, and worked around or repaired. Are there other hurdles to UDI acceptance? Lets expose and conquer them.

I’m very interested in any comments or suggestions about UDI, how it died, and anything we can do to get this project started again.