I have a client-facing server running CentOS 5.7 and Plesk 10.3. When clients need web space, I put them on my Plesk server so they have shiny buttons to click when managing their own web space. Recently I had a series of unfortunate events cause an outage on one client.

It starts with my craving to have things standardized. All client account domain directories are in lower case. All, that is, except for one: AmazingClient. Their main domain’s vhost directory is /var/www/vhosts/AmazingClient which, in Plesk-land means that any reference to that client’s domain is always in that case. It bugs me. More than it should. When I created the client account several months ago, for some inexplicable reason, I used CamelCase in their name. One recent evening I decided to change the capitalization for their account’s main domain. Simple, right?

I did say that I’m using Plesk, did I not?

Before I go any further, I know what you might be thinking. “Domains aren’t case sensitive! What nonsense are you on about?!” They’re not case sensitive when approaching domains from a DNS perspective. However, I’m looking at this from a filesystem and Plesk user account perspective.

To change something as simple as the case of a domain’s vhost directory, one cannot merely rename it. There are many configuration files to consider as well as Plesk-specific tasks that rely on the domain’s directory not being glibly swapped out from underneath it. To change a domain’s name in Plesk, one has to go into the client’s control panel, and click on the Websites & Domains tab.

From there you will find the domain that you want to change the case of (remember, this isn’t about “domain” in the DNS sense, but rather the representation of that domain within Plesk and on the filesystem) and click on its link. From there you will come to the Host Settings page for that domain. Once on the Host Settings page, you’ll have the option to change the domain name. Here comes the trouble: you can’t change the name merely based on case. Even though Plesk sees the client domain differently in the backend based on case, in this Host Settings interface case is not taken into account. Plesk will complain that the domain already exists. You need to change the domain name to something different, then change it back to the original domain name, minus the capitalization. (Plesk FAIL #1)

In my case, I wanted to swing it from AwesomeClient.com, to awesomeclienttemp.com, and then back to awesomeclient.com (sans the capital “A” and “C”).

Tipping Over the Edge of Doom

When trying to move from AwesomeClient.com to awesomeclienttemp.com I received this error:

Internal error: [domain path] is out of webspace
Message is out of webspace
File Webspace.php
Line 334
Type PleskFatalException

After that error, the Websites & Domains tab is no longer accessible to that client account. Trying to use it receives the same “Internal Error: [domain path] is out of webspace” error.

You see, it appears that Plesk, upon requesting a domain rename, copies the domain’s existing files and then deletes the old ones. It does not perform a mere rename action (Plesk FAIL #2). This client uses quite a bit of space and it apparently maxed out their quota. I say “apparently” because, by a strict accounting for free space and quotas on the server, it should have been allowed – but just barely. Perhaps there’s more space that Plesk needs than a simple doubling of existing files. (Plesk FAIL #3?) Plesk certainly didn’t perform any kind of filesystem or account limitation checking prior to attempting the move. (Plesk FAIL #4)

The client site was still responsive; there didn’t appear to be any negative effects. I needed to investigate further, but as the night wore on I decided to postpone a thorough examination until another day.

Ask Not For Whom Your Cell Phone Tolls

Bright and early the next morning, I got a call. It was from the client.

“Our website seems to be down, so… uhh… if you could look into that…”

Super.

Nothing was being served up in response to any page requests for this domain. Apache’s error logs were showing requests for this client’s files as hitting in the default vhost root, not their own. Then, it hit me.

Plesk does not use the standard Apache configuration files. I mean, it does, but not really. It auto-generates Apache configuration files based on the information that is stored in its own customer database within MySQL. That’s why the domain was just fine the evening before, but didn’t fail until the wee hours of the morning. The configuration files had been latently generated based on the failed attempt at changing the domain account name.

Silly me… I expected there to be rollback statements in any of the SQL DML statements made to the database. I expected that a fatal error would be caught and changes rolled back. They weren’t. (Plesk FAIL #5) Silly, silly me.

Of course, I wasn’t going to be able to change the domain information because the Websites & Domains tab bombed out permanently with an internal error. I couldn’t access the officially sanctioned means of modifying the domain account. This called for some database mangling.

Let Pry Through the Portage of the Database

I logged into mysql and dumped the psa database. From there, I used grep to scour the .sql file for any mention of awesomeclienttemp. Sure enough, the bad change was recorded in the database. There were dozens of records in several tables that pointed to the bad domain. That was causing Apache configuration files to be written with bad data, among other applications. There was also mention of the original, unsullied domain. I guess not all of the SQL statements that are part and parcel of a domain change were able to be executed before the error condition was achieved. (Side note: ROLLBACK!! ROLLBACK!! ROLLBACK!!)

Solving the problem was a simple as searching for and replacing the string awesomeclienttemp with AwesomeClient. I used mysql to perform that, but it could have been done on the dump file and then imported. For those interested, I used the replace() function and performed a select statement first just to make sure that I was changing the data that I expected to. Once satisfied with the results I performed an update statement also using the replace() function. Here’s an example of changing some values in the dns_recs table of Plesk’s psa database:

mysql&gt> SELECT REPLACE(displayVal,'clienttemp', 'Client') FROM dns_recs WHERE displayVal LIKE '%clienttemp%';
+--------------------------------------------+
| REPLACE(displayVal,'clienttemp', 'Client') |
+--------------------------------------------+
| mail.AwesomeClient.com.                    |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
+--------------------------------------------+
mysql> UPDATE dns_recs SET displayVal=REPLACE(displayVal,'clienttemp', 'Client') WHERE displayVal LIKE '%clienttemp%';

With the database in a better state, there is still one more thing left to do. Plesk doesn’t dynamically look to the database for configuration information. It looks to regular files that have been dynamically generated from the database’s information. That generation happens on a schedule, but can be expedited using the httpdmng command. Specifically, I used:

/usr/local/psa/admin/bin/httpdmng --reconfigure-domain AwesomeClient.com

You could also use the –reconfigure-all option to perform a regeneration of all domain configuration files. After running httpdmng the domain was up and running.

Apache Test Page or Blank Page Problems

I glossed over some of the troubleshooting techniques I used while tracing the problem to its root. If you’re having trouble with seeing the Apache test page, then search through your httpd.conf file and make sure that your DirectoryIndex directive is set to look for all of the variants of an index.html page that you use. For example, index.html, index.htm. index.php, etc.

Furthermore, just to reiterate, check all of your vhost conf files, such as yourdomain/conf/vhost.conf (or any conf files that reside in that directory) for the DocumentRoot directive and make sure that it’s pointed to what you want it to be pointed at. Do not edit the files that are named similar to 13279881860.14852200_httpd.include. Those are auto-generated by Plesk and at worst you could cause destruction of files in your domain; at best you will have to re-edit those files every time a new one is generated.

Of course, do a dummy check to make sure that the domain you are trying to access is really resolving to the IP address of your web server. Just… do it. It takes 5 seconds and you have the outside chance of being pleasantly surprised.

The Takeaway

Plesk is rickety. If anyone has used a better control panel for client-facing servers, let me know. I’ve worked with cPanel and Plesk, but never with any of the others that I’ve listed in this giant list of web based server control panels. Most people will shout “Just don’t use a control panel!” but that’s not a terribly client friendly option. I’m not categorically against control panels when used in the correct situations. I am, however, against misbehaving control panels.

Let me know your experiences in the comments below.

I spend a lot of time doing text based data processing. A *lot* of time. During an analysis, I often want to do things like look at ‘Top Talkers’, ‘Most Frequent Visitors’, or really anything that comprises a list of unique identifiers sorted by count. As a result, I’ve translated two actions into a series of pipes:

1. What’s the count of events per thingy: “ | sort | uniq -c | sort -n“
2. Who has been doing whatever: “ | sort -u“

This tends to work pretty well in most cases. Today, however, was not one of those cases. While attempting get a list of unique MACs I started out with a source (i.e. non-uniqued) 16GB text file with one MAC per line. This is where things got annoying. Muscle memory kicked in and since this matched Action #2, I ran the following command: cat macs_all.txt | sort -u >; macs_unique.txt

I expected it to take a few minutes, so I went back to the other things I was doing and let it go. I checked back 15 minutes later, and it was still running. Waited 5 minutes…still running. When the command had been running for 45 minutes, I got fed up and decided that I could do better. Perl, being my go to tool, came to the rescue in the form of hashes. I won’t go into gritty detail, but a Perl hash is a data structure that consists of a list of key/value pairs. Whenever you assign a value to a key it will add an entry for the key if it doesn’t exist, or update the value if it does. Since a key cannot be in the same hash multiple times, it makes for a pretty good hack to generate a unique list. This is what I ended up doing:

#!/usr/bin/perl -w
 
use strict;
 
my %unique;
 
while( my $line =  )
{
  next unless $line;
  chomp $line;
  $unique{$line} = '';
}
 
for my $key ( keys %unique )
{
  print "$key\n";
}

This worked significantly better for me. The output was not sorted, but that’s fine, I didn’t need it sorted, only unique. The timing information looked a lot better too.

packs@ node1:~> time cat macs_all.txt | sort -u > macs_unique.txt
 
real    181m12.417s
user    176m13.926s
sys     1m42.335s
packs@ node1:~> time cat macs_all.txt | ./fast_uniq.pl > macs_fast_uniqed.txt
 
real    8m9.074s
user    7m28.176s
sys     0m46.271s

The times can’t really be directly compared, since output from fast_uniq.pl isn’t actually sorted. Given the pretty substantial difference I think we can reasonably accept the fact that fast_uniq.pl is better in this use case. After seeing this, I’m tempted to add some functionality so I stop using both sort and uniq entirely.

I’m interested to hear if anyone else has done something similar or explain to me how much my code sucks.

The mechanism that I used to do this was with the authorized_keys file. For a thorough explanation of that file, take a peak at the man page for sshd. To explain it very simply, the authorized_keys file holds the public keys of other users/systems that are allowed to connect to that machine. For example, I place my main user account’s public RSA key into the authorized_keys file on the Linux servers that I manage. When I connect to the remote servers using SSH, it checks to see if I’m who I say I am by challenging me with the public key that it has stored. The user account on my laptop uses the private key to validate itself (yes, the private key is password protected) and I am then allowed to haxor on the servers to my heart’s content.

Here’s an example of a public key:

ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQDclBxY7lOaolHGaogdcc9GaTQLWMcn2PK4hnQfWlJgeeGqgS66jL4XJyiR9HcgaebBW88Z2sevUxd7g25WhuuRAazfOcElEaE+h6MMPZ94gHY+x+iVAdlNKxLT/bTvCUXLEft/yZFpnknnv7jX4ChfSiII9OiAiCzuSdyHt1/1LgEHgvDIwKMzvTgImm5X/3IhtOitjJY3Q6yhKQ6LdenQtG/v+ANqKe6opDuUKc3k9hRmj7aHROxL52paQTEgEMoVLbIoZY4/yGUzmrZQU45jNqMrbXdAxG4XexZxb7bpTLu91s0DJQGx43JNXwhJVinPgxHLmfyoCSqR1WPqn8E3 testuser@testserver

The public key, when placed in a system’s authorized_keys file, can have some extra tidbits added to it that sshd honors. An SSH protocol 2 public key follows this format:

options, keytype, base64-encoded key, comment

In the above public key, you see the keytype as ‘ssh-rsa’ followed by a space, then the key itself followed by a space and finally a comment, which in this case is a username and hostname combination. That’s a helpful hint to know who this key supposedly belongs to. Notice that there are no options included in the above key, which would come before the keytype.

Some of the options that are available to be parsed by sshd include:

• environment= Changes an environmental variable for the user that is on the receiving end of the connection.
• from= Only allows connections that use this public key to be initiated from certain hosts. Helpful for the extremely paranoid or the very security conscious (the only difference between the two being pay grade).
• no-X11-forwarding Because we don’t need users installing xorg and then browsing the web on a remote instance of Chrome.

There are plenty of other options, however the final one that I’ll mention is the most crucial to this topic: command="command"

With the command= option, you can cause a command to be run immediately upon a successful connection to a remote host. Once the command is run, the connection is closed. Notice how that works. The command is immediately run and then once the command finishes, the connection is closed. This is not something that you’d want to do to a key that is intended to be used interactively by a human.

What could this be good for? In my specific scenario, I am using a backup tool that moves all of the data to stdout which is then piped to ssh for a secure transfer to remote storage. The remote connection would normally look like this: ssh remoteuser@remoteserver ” cat > backupfile.zip” However, if I edit the authorized keys file, I can restrict the incoming ssh connection to only be allowed to use that specific command.

It’s just another layer of security to keep people from doing things that they shouldn’t be doing. Have different ways of achieving a similar goal? Any caveats you know about? Let me know in the comments.

The main problem surrounded Apple’s custom EFI. Apple hardware does not use a BIOS, but instead uses EFI (note: not, specifically speaking, UEFI). Or rather, it uses an ancient, bastardized version of EFI 1.1. There is a BIOS compatibility layer that allows OSs that can only communicate with a BIOS to operate on the hardware. Most notably Windows. Apple’s OS also runs on a hard disk that has been partitioned using the GPT partitioning scheme, which isn’t itself a huge deal, but you might be surprised at the anemic support for GPT boot disks in even modern operating systems.

To use the Mac Mini to boot an OS that needs BIOS compatibility and a MBR disk should be relatively easy. Right? Right!

Unless Apple is involved.

There are several things that Apple has mutated away from the EFI standard, one of them being not using the EFI system partition for anything except firmware updates. Their custom EFI implementation has the boot process (as well as some extra filesystem drivers) baked in. The whole EFI experience just never worked like I expected it to. The other trouble is that Boot Camp has been changed in OS X Lion. If you wanted to be hand held through the partitioning process and the creation of a hybrid GPT/MBR disk, you’re invited to use Boot Camp. However the latest alterations only allow media with Windows images to be accepted. You can no longer (from my ability to understand) use Boot Camp to install non-Windows OSs. Of course, it was always unsupported, but at least it was doable.

During the whole process, I used the EFI boot manager rEFIt which apparently only recently works with OS X Lion. I read more about the GPT partitioning scheme than I ever have previously. I learned more about EFI than I ever wanted to know (although all of that information will come in very handy in the near future). I hand-rolled bootable USB thumbdrives. I tweaked partition tables. I did very nearly everything I could think of except rolling my own EFI boot partition. After the hours had steadily ticked away I decided it was no longer worth it.

After countless errors concerning boot media, partition problems, and blinking cursors, I concede that the latest Mac Mini has defeated me. It has been shipped back to Amazon and I can go back to my Apple-less existence. Speaking of Amazon, I believe that they deserve some praise in this.

Amazon made the returns process easier than any return I have ever made. Anywhere. I stated that the reason I returned it was because software I had intended to use with it was not compatible. As a result of the return not being their fault, I had to pay return shipping. Within just a few clicks, Amazon created a return label. I printed it out, boxed the mini up, taped the label to the box and handed it over to the man behind the UPS Store counter. Within 15 seconds I was walking out of the store. I have the fortune of living just a few hundred miles from an Amazon return center located in the Las Vegas area so the return was processed and money credited back within two days. Thank you, Amazon. You were the only bright spot in this debacle.

I am now investigating other pieces of hardware for this project based on the recommendations of several colleagues. If you have a recommendation, share it with me and the rest of my readers in the comments below. I’ll certainly write about my second attempt at this project as it happens.

In the end, I’m not mad. The Apple wasn’t designed to do what I was asking it to do. It was my fault. My only lingering frustration is that the Mac seems to take any standard technology that it uses and twists it in new and different ways so that your familiairty with a standard becomes more of a liability than an asset. Sound like another familiar company that SysAdmins like to pick on? Then again, Apple isn’t intended to be in the business market. Let us pause and mourn the passing of the Xserve (I handed my G5 Xserve over to Best Buy for free recycling last year. So, so sad…).

Any similar experiences with an Apple product? Have you managed to wedge an alternate OS on 2012 Apple hardware? Let me know in the comments below.

A SOCKS proxy is easy to set up. It’s nothing more than an SSH server and an SSH client that speak the SOCKS protocol. In my case, I use OpenSSH. If you use a different SSH server or some other form of making a SOCKS proxy, this little post will be of little use to you. However, stick around because there’s a note down below concerning the false sense of security many people have when using a SOCKS proxy.

There’s a handy little option in the OpenSSH client that allows for the creation of a a local port binding that immediately forwards traffic to that port to another machine: -D. After that part of the command, simply include the username and host for the OpenSSH server that you want all local traffic bound for that local port to be relayed through. It makes it all the sweeter if you have RSA keypairs set up between hosts.

In my case, I usually use this set of options:

ssh -fCND localhost:8080 user@myserver.thenubbyadmin.com

Let me peel back those other three options that I use:

• -f sends ssh to the background just before the command is executed.
• -N refuses to execute remote commands. This way I know nothing is going to be run via the SSH connection on the remote machine. I’m paranoid.
• -C compresses the TCP traffic. This might not be ideal if you have a good connection as it is stated in the man pages for the OpenSSH client that -C slows down your throughput on fast connections.

Application Support

The applications that you want to use with the SOCKS proxy need to have explicit options to support it. It’s not something that can be done underneath the application without its knowledge. For example, most web browsers have an option to use a SOCKS proxy within their advanced options section.

You will want to go to the options page of your application and search for SOCKS support. From there, tell the application to use localhost:port# as the proxy. In my case, I made port 8080 to be the local port that listens for traffic and then forwards it to my remote server.

If you need a secure connection that can be put in place without an application’s knowledge, you’ll need to implement a VPN.

You’re Not as Anonymous as You Think You Are

If you’re using the SOCKS proxy for the purposes of secure browsing, know that your DNS requests are an entirely different application layer traffic. Unless your DNS client is also set up to use the SOCKS proxy, your DNS requests will be plainly visible on the network that you are trying to remain anonymous / protected on. This can cause problems if you’re on an untrusted network. Owning the DNS servers that a machine is using is one of the most sure ways of wreaking havoc.

Have any other SOCKS tips? Do you use a different client or server? Let me know in the comments.

At this point, go out and grab the CentOS ISO that interests you. Have it on your filesystem because we’ll be mounting it and copying some files from it. Once you’ve got the ISO you can move on to partitioning the drive.

Partitioning

First, you’ll want to partition the USB drive. We’ll be using plain ol’ MBR style partition tables and two primary partitions. I’m not going to hand-hold you through this part of the process. Use whatever partitioning tool you want and follow the guidelines below. GParted is fine if you use Gnome, parted is great if you want to use a shell, and fdisk works on both Windows and *NIX environments.

The partition layout will be thus:

1. A primary partition that uses the FAT16 filesystem and is at least as big as your ISO plus about 50MB. You need to give it the boot flag.
2. A primary partition that uses ext2 and is at least as big as your ISO. Preferably you’ll just use up the rest of your USB drive’s free space for this partition.

Once your partitions are set up, we’ve got some file moving to do.

Setting the Filesystems Up

You’ll want to mount your two partitions so that you can access them. In my case, the first partition (the FAT16 boot partition) is /dev/sdc1 and the data partition (the one formatted in ext2) is /dev/sdc2. I’ve mounted sdc1 as /mnt/usbboot and sdc2 as /mnt/usbdata. I will be using that nomenclature throughout the rest of this post.

You’ll also want to mount your CentOS ISO as a filesystem because we need to copy some files off of it. In my case, I ran mount -o loop /path/to/iso/file.iso /mnt/centosiso and will be using /mnt/centosiso in my examples below. Now that we’ve got all of our filesystems mounted, we’ll start the procedures.

First, go to the mounted CentOS iso and copy the /isolinux directory to the boot partition of the USB drive.

cp -r /mnt/centosiso/isolinux /mnt/usbboot

Rename the isolinux folder on the USB drive to syslinux

mv /mnt/usbboot/isolinux /mnt/usbboot/syslinux

Rename the isolinux.cfg file to syslinux.cfg

mv /mnt/usbboot/syslinux/isolinux.cfg /mnt/usbboot/syslinux/syslinux.cfg

Now we need to copy the contents of the /mnt/centosiso/images folder to the USB boot partition. Notice that I emphasis that this is a copy of the contents within the ISO’s images folder. A little later on we’ll be copying over the entire ISO as a file.

cp -r /mnt/centosiso/images /mnt/usbboot

Finally, we copy the .iso file itself to the data partition (not the boot partition that we were just working with!):

cp /path/to/iso/file.iso /mnt/usbdata

Once all that is done, we have to install a bootloader. I’ll use the simple syslinux loader. We want to use our smaller volume (the one that we set the boot flag on up in the partitioning section) as the target for the syslinux command.

syslinux -i /dev/sdc1

Now, we dismount our USB drive and test it out by booting from it on another system!

 

---

Finoto!

You should now have a bootable CentOS 6 USB drive. CentOS 6 is somewhat unique as a result of the bug that requires the images directory to be included on the boot partition, but other than that it’s relatively straight forward.

macmeta:~ user$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.7.3
BuildVersion: 11D50d

You can retrieve the specific information that you want using the following options: -productName | -productVersion | -buildVersion

macmeta:~ aoi$ sw_vers -productVersion
10.7.3

You can cat out /System/Library/CoreServices/SystemVersion.plist and eyeball the XML that comes back. In my case:

macmeta:~ user$ cat /System/Library/CoreServices/SystemVersion.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>ProductBuildVersion</key>
	<string>11D50d</string>
	<key>ProductCopyright</key>
	<string>1983-2012 Apple Inc.</string>
	<key>ProductName</key>
	<string>Mac OS X</string>
	<key>ProductUserVisibleVersion</key>
	<string>10.7.3</string>
	<key>ProductVersion</key>
	<string>10.7.3</string>
</dict>
</plist>

Or you could merely add grep -C 2 ProductVersion.

macmeta:~ user$ cat /System/Library/CoreServices/SystemVersion.plist | grep -C 2 ProductVersion
	<key>ProductUserVisibleVersion</key>
	<string>10.7.3</string>
	<key>ProductVersion</key>
	<string>10.7.3</string>

Using system_profiler SPSoftwareDataType gives you lots of information including kernel version:

macmeta:~ user$ system_profiler SPSoftwareDataType
Software:
 
    System Software Overview:
 
      System Version: Mac OS X 10.7.3 (11D50d)
      Kernel Version: Darwin 11.3.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: macmeta
      User Name: USER (user)
      Secure Virtual Memory: Enabled
      64-bit Kernel and Extensions: Yes
      Time since boot: 56 minutes

Those accustomed to using ye olden *NIX uname -a will find that it only gives the Darwin kernel information.

macmeta:~ user$ uname -a
Darwin macmeta.tc.ph.cox.net 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

And I saved the ugliest for last! osascript is a tool that can be used to run AppleScript. with the -e option, it will run a single line script that you enter. Let’s try osascript -e 'system info':

macmeta:~ user$ osascript -e 'system info'
AppleScript version:2.2.1, AppleScript Studio version:1.5.2, system version:10.7.3, short user name:aoi, long user name:AOI, user ID:501, user locale:en_US, home directory:alias Macintosh HD:Users:aoi:, boot volume:Macintosh HD, computer name:macmeta, host name:macmeta.tc.ph.cox.net, IPv4 address:192.168.11.119, primary Ethernet address:40:6c:8f:0d:48:a6, CPU type:Intel 80486, CPU speed:2300, physical memory:2048

As you can see, it brings back a ton of info, including the system version.

Know of any other ways to find the system version and build numbers of OS X? Let me know in the comments.

As of OS X 10.7.3, to find out information concerning what version of Apple’s EFI firmware you are running, perform the following command at a terminal:

ioreg -p IODeviceTree -b -n efi | grep efi -C 4

In my case, the output is as follows:

+-o Root  <class IORegistryEntry, id 0x100000100, retain 10>
  +-o /  <class IOPlatformExpertDevice, id 0x100000110, registered, matched, active, busy 0 (15221 ms), retain 37>
    +-o chosen  <class IOService, id 0x100000101, !registered, !matched, active, busy 0, retain 5>
    | +-o memory-map  <class IOService, id 0x100000102, !registered, !matched, active, busy 0, retain 6>
    +-o efi  <class IOService, id 0x100000103, !registered, !matched, active, busy 0, retain 8>
    | | {
    | |   "firmware-revision" = <0a000100>
    | |   "device-properties" = <fe04000001000000040000000e0200000500000002010c00d041030a000000000101060000027fff04001e00000073006100760065006400$
    | |   "firmware-abi" = <"EFI64">
    | |   "name" = <"efi">
    | |   "firmware-vendor" = <4100700070006c0065000000>
    | | }
    | |
    | +-o kernel-compatibility  <class IOService, id 0x100000104, !registered, !matched, active, busy 0, retain 4>

And the text that I’m most interested in is “firmware-abi” = <”EFI64″>

To explain the above, ioreg is used to query the Apple I/O registry. We then need to select the Device Tree plane and do so with -p IODeviceTree. -b is just a nicety to put the object name in bold letters. -n scopes the query down to only those things that have a certain name in them. In our case, we want the name of “efi”.

We then pipe the whole mess to grep where we search for the word “efi” and then, with -C, give four lines of context above and below each mention of the word “efi”.

Simple!

I have a new Mac Mini running OS X Lion that I need remote access to. I’ve enabled “Screen Sharing” in Sharing Preferences, created a VNC password and ensured that Screen Sharing is allowed through the firewall.

Using various VNC clients, I receive different forms of bizarre errors and refused connections. In Remmina on Fedora 14, I cannot connect to the Apple VNC server. Either a connection is made and I see a very brief flicker of a remote connection before the window is closed or the connection perpetually hangs at the “Connecting…” stage.

Using UltraVNC on Windows 7 I receive the error:

Server closed connection

- the server running as application

My Solution

Edit your bit-depth settings in your VNC client (not on the receiving Apple computer). Change the bit depth to something other than 256 colors or 32-bit. For example, I can connect to the Mac’s VNC server using 15, 16 or 24-bit color depths. As soon as I choose 256 colors or 32-bit I am unable to make a connection.

I suppose Apple wants their GUI to be enjoyed in its full glory and 256 colors is just too shabby. 32-bit? Well, that’s not minimalistic enough.

However, I’ve been a bad IT person. I haven’t proactively monitored their IT assets. I can make any number of excuses for myself, but none really mollifies me (nor would it satisfy anyone else with even just a hint of a desire to do a job right). An interesting fact is that some of my very first attempts at blogging (back in 2007 or 2008) were as a result of my attempts at making a monitoring system for this organization. That blog, a shared Drupal CMS between me and a friend, is long gone. However the nagging need for a thorough “Meta Server” has haunted me ever since.

The organization is suddenly expanding into a new field for them. They have high hopes and are tackling this new project head on. Their exact growth potential is unknown at the moment, but I want to put the long needed information reporting infrastructure in place now before much more moves forward. A few new websites will be made for the organization’s endeavors and both will be a large part of the success (or failure) of this new phase of growth. If I don’t know about their website / server / PC infrastructure health before I get panic-stricken phone calls, then I can consider myself a complete failure as a SysAdmin.

“What is a Meta Server?!”

A “Meta Server” is what I like to call any node whose sole purpose is to collect and display information that is largely of no interest to a standard user. You’re unlikely to see the term anywhere else because I just made it up (or if it is in use elsewhere, please, whoever came up with it, don’t sue me).

It’s the kind of thing that is where notes are stored, wikis are stood up, NMSs sweep, trends are graphed and bitknobs are virtually twiddled. In larger places, you might have a “Meta Rack” but I’ve never worked in an environment so large as to need stacks of meta servers. Oh if only…

My intention for this server is not to be just an NMS, in spite of the title of this blog post. It’s much more than that. Let’s take a look at what my plans are for it.

The Software

My plans for the meta server are many. I want monitoring, alerting, trending, help desk, asset/inventory management, log collection, imaging, perhaps a wiki… lots of stuff. Since it’s a small office with little need for a multitude of servers I can’t separate these roles out onto difference pieces of hardware, nor do I really need to unless there’s some glaring incompatibility between packages. Even if there was some kind of package incompatibility between tools, I’d prefer to just use Linux-VServer or something similar to stand up a virtual instance.

To expand on the list of topics just above, I’m very interested in the health of the network itself as well as the nodes on the network. I want to scan the network for devices and get alerts when new things show up. I want to poll each device, PC, laptop, printer, server, WAP, modem, switch… you name it, for vital statistics. That can be through SNMP, netflow, sflow, or an agent installed onto the operating system (in the case of a PC with a full OS on it). I want speed and latency statistics for our ISP connection too. This bundle of requirements necessitates probably three, maybe four separate tools.

I haven’t settled 100% on the applications that I’ll be using, but I have a pretty good idea that either OpenNMS or Pandora FMS will be the main monitoring and alerting system. For prettier graphs to look at, Observium is high on my list. I might use ntop for netflow analysis, rTPL for scheduled throughput tests and smokeping for latency monitoring. Munin may play a part as well; I haven’t decided yet.

For log collection, I’m interested in Splunk‘s community edition, but graylog2 is appealing as well. I’d probably use the Snare Agent for Windows to collect logs from my Windows hosts and send it all to graylog2 – if that’s the direction I go. However, Alien Vault’s OSSIM is also in the running.

For a help desk, I’m almost 100% sold on RT. I’m currently using SpiceWorks‘s help desk on a Windows server, but that’s a bit heavy for my needs. I don’t use most of its other management tools. It’s asset management and network monitoring is… okay. It’s a bit rigid for my tastes, however.

I’m also interested in having a simple PXE boot imaging tool on the network. I have long been a fan of the FOG project. This goal of mine isn’t to start creating an extensive image library. Instead, I just want to take the occasional quick image of a PC before a major change and also to be able to boot a PC over the network onto an anti-virus image to perform an offline virus scan. I’ll need some decent storage space to keep a few images around depending on which user’s PC may need to be quickly backed up. A few hundred GBs would be nice.

I’m considering the use of Monit for some automated response, but since I don’t have many *nix devices to contend with, that might be wasted effort. Then again, automating things is never wasted effort!

There might be a documentation wiki thrown in there for good measure. Currently I use the hosted wiki service of Zoho, but I am considering moving it in-house.

It has occurred to me that such a small device may be a prime target for theft so I’m considering volume encryption to protect the data. If someone wants the hardware bad enough, they can have it. However, I don’t want them to have any valuable data to play with. I’m sure it’s very unlikely that a smash-and-grab thief would have the interest or skills to do much with the data, but… still.

The Hardware

First, let’s clear something up. As often as I refer to this needed device as a “server” it is not, in fact, a “server” by any enterprise understanding of that word. It is just a device that serves, but is not intended to be made up of components that are traditionally thought of as “server grade.”

Since it’s a small office, it doesn’t have a proper server rack. It has more of a closet than anything. The closet is better than the “Just set that server down next to my desk; it’ll be fine, my door locks!” situation I found myself in years ago. The device needs to be small.

I’ve been collecting a large list of small form factor PCs for quite some time now. I like small things, especially when the alternative is cramming a 6 year old workstation next to the building’s demarc point to act as a caching proxy (I bet you’ve done that). I’ve got to narrow the market of small PCs down to a manageable pool using some base requirements.

My goals for the hardware are the following:

1. I don’t want a development board. Things like the RaspberryPi or the Hawkboard won’t cut it. I want a production piece of equipment that is manufactured in decent quantities and is intended for a broader consumer base than hardware hackers and developers. The BeagleBoard is on the edge of that grouping because of its wide acceptance, but it’s still iffy. I’m sure some will say “Oh but the Hawkboard…” or “Hey, the BeagleBoard does…” and that’s fine. It’s just that in my research, they don’t seem to be dead ringers for well supported, hard-working devices.
2. At least 250GB of internal storage. I don’t want to deal with CF cards like many micro-ITX boxes use. They don’t have enough storage and regardless of how advanced the wear leveling algorithms are, I don’t want to worry about block wear on a device that already has precious little space. NAND flash is also prone to soft read errors that makes you reliant on the ECC of the card itself. That’s too many variables for my comfort at this moment. Also, I want the storage to be internal as a matter of preference. I’m trying to get away from a “Just attach another USB hard drive!” mentality. That and I just don’t like USB buses as a rule.
3. At least 2GB of RAM. I’m going to have a lot of daemons running and I’d prefer to take a RAM ceiling out of the equation.
4. Price: I want to keep it around the $500 mark. If it can happen for less that’s great.

Over the years of keeping this project on the backburner and also considering building other similar meta servers, I’ve looked at NetGate cases, the Fit-PC, BeagleBoard-based systems, SlimPRO, Pearl D series, and various models of thin clients and plug PCs. I do have a soft spot for plug PCs, but none of them meet the above criterion.

The Final Choice

After searching far and wide, the one PC that kept coming to the top of the pack was the Fit-PC. I’ve had my eye on it for years and watched as its design have iterated past version 1, through version 2 and now on to version 3. It’s a handy little thing (form-factor pun intended) that has some decent resources. The latest version can include a 250 GB platter hard drive with 2GB of RAM and a 1GHz APU G-T40N processor. The price is a tad steep at $480 plus VAT and shipping.

I mulled the option. After shipping and tax, it would be over $500. Could I really justify that much money for what I was getting? Certainly the value of what I was going to do with it was worth it. I just… I wasn’t sure.

Then it hit me. A Mac Mini. The cheapest brand-new Mac Mini one can find (legitimately) is about $569 USD. And what does one of those shiny suckers have in it? 4GB of RAM, a 500GB hard drive and an Intel i5 processor. Furthermore, I can get it on Amazon.com straight from Apple’s store with no shipping fees and no sales tax. It has twice the RAM (2GB would probably suffice, but more is always better!), twice the hard drive space (very much useful considering the imaging server portion of the project) and an Intel i5 (could come in handy for encryption and report generating). Furthermore, love or hate Apple, their hardware is rock solid (iPhone recpetion issues notwithstanding).

For just a handful of dollars more, I get double the resources (more than double if you consider the CPU). I had been approved for a $500 purchase, but with just one five minute phone call later a $569 Mac Mini had been approved.

As for the OS, I am highly unlikely to be using OS X (highly). I’ll almost certainly put CentOS 6 on it and be on my merry way.

Retrospective

In talking with colleagues, I’ve taken a tiny bit of flak for making such an expensive NMS. Certainly, I think I could perhaps build a similar box for slightly cheaper, but without the i5. I’d likely need to use an Atom processor to keep the price down. However, the time for me to build the thing still costs my client money. Perhaps other solutions exist off-the-shelf with similar specs – but I wasn’t able to find them. Once again, research time costs.

In the end, I’m sitting here, a brand new Mac Mini still in its box next to me. I don’t have any regrets… yet. I’m eager to get this project going and hope to blog more about its progress, starting with the installation of CentOS on Apple hardware.

So what do you think? Did I blow it? Did I have other compelling options that I missed? Would you be happy with a Mac Mini “Meta Server?” Let me know in the comment below.