This script examines the state of each volume and the space used/free for each volume with a quota, and reports back any errors. The layout is copied from the check_netapp plugin, and is fairly simple to follow along. Of note, be sure to set the appropriate @INC path for your installation, and the location of your public key if using PKI with NexentaStor. Unfortunately some of the API is not documented, and the functionality is incomplete – I had to play around with the $folder->execute() method to find the correct command to get the space used and quota allocated. It seems like this should be a built in, documented method in the API, so here’s to hoping it makes it in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#!/usr/bin/perl
 
BEGIN { unshift (@INC, "/opt/build/NexentaStor-SDK-Linux/examples/perl");}
 
use NZA::Common;
use strict;
 
$ENV{'NEXENTA_ID_RSA'} = '/home/nagios/.ssh/id_rsa.pub';
 
#Configure space % threshold here for warnings
my $space_threshold = .9;
 
my $IPADDR = $ARGV[0];
if (!$IPADDR) {
        print "Usage: checkhealth.pl <ip -ADDR>\n";
        exit 1;
}
my $volume = nms_remote_obj($IPADDR, '/Root/Volume');
my $folder = nms_remote_obj($IPADDR, '/Root/Folder');
my @errors;
 
my $volnames = $volume->get_names('');
for my $vol (@$volnames) {
        #errors: No known data errors
        #state: ONLINE
        my $volstatus = $volume->get_status($vol);
        if (${$volstatus}{'state'}[0] ne 'ONLINE') {
                push (@errors, "Bad volume status for $vol: ", ${$volstatus}{'state'}[0] );
        }
        if ( ! ${$volstatus}{'errors'}[0] =~ /No known data errors/ ) {
                push (@errors, "Errors for $vol: ", ${$volstatus}{'errors'}[0] );
        }
 
        my $folder_names = $folder->get_names('');
        for my $fol (@$folder_names) {
                $folder->reconstruct($fol);
                my $quota;
                my $used;
                my $lines = $folder->execute($fol, "get -H used $fol");
                for my $line (@$lines) { $used .= $line };
                my $lines = $folder->execute($fol, "get -H quota $fol");
                for my $line (@$lines) { $quota .= $line };
                my @data = split (/\s+/, $used);
                $used = $data[2];
                my @data = split (/\s+/, $quota);
                $quota = $data[2];
                if ($quota =~ /none/i) {
                        next;
                }
 
                $used = convert_space_units($used);
                $quota = convert_space_units($quota);
                #print "$fol - used: $used, quota: $quota\n";
                if ($quota > 0) {
                        if ( ($used/$quota) > .9 ) {
                                push (@errors, "Volume almost full: $fol");
                        }
                }
        }
}
 
if (scalar(@errors) > 0 ) {
        print "Nexentastor healthcheck failed! ";
        for my $error (@errors) {
                print "  $error.";
        }
        print "\n";
        exit(1);
}
else {
        print "Nexentastor healthcheck OK\n";
        exit(0);
}
 
 
sub convert_space_units($) {
        my $string = shift;
        # K, M, G, T
        my %factor = (
                K => 1000,
                M => 1000000,
                G => 1000000000,
                T => 1000000000000,
        );
        if ($string =~ /([.\d]+)(\w+)/) {
                if ($factor{$2}) {
                        return $1 * $factor{$2};
                }
        }
        return "";
}
</ip>

cd /root/NexentaStor-SDK-Linux/examples/perl/
./test.pl 192.168.1.100


To setup authentication using ssh keys, do the following:
On the client machine, generate a rsa key if one doesn’t already exist:
ssh-keygen -N "" -q -f ~/.ssh/id_rsa
And on the nexentastor appliance, add that key. The Key ID should be a name you recognize to keep track of the key, and the key value should start with ssh-rsa AAAAzxvdsa….. and end with XXX== – make sure you leave off the name at the end of the key, this is confusing:

nmc@nexentastor:/$ setup appliance authentication keys add
Key ID : myclient
Key Value : ssh-rsa AAAAB3Nza
......
.....
.... 1I89Q==

Searching for this issue on the net, I found that this is usually caused by a corrupt row in the database. The solution is to find the corrupt row, and delete it (and/or update it with proper, non-corrupt values).

Unfortunately, there is no easy way to find the row that is giving you grief (or even the table if you don’t know what is causing this error). You basically have to narrow down your search by looking at parts of the database. If you have some sort of date or timestamp column in your table, you can narrow it down a lot more easily, if you know approximately when the error started:


mydb=# select id from mytable where date_created > timestamp '2009-02-27 00:00:00' and date_created < timestamp '2009-03-01 23:59:00';
id
--------------------------------------
3ab226cf-a972-463d-b5a1-148fe39672b5
10daca73-b2b3-470c-a258-5c92d21cfbb6
(2 rows)


Luckily it's only two rows, and the first one is the culprit:

mydb=# select * from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';
ERROR: missing chunk number 0 for toast value 87246446


Now, on to delete it:
mydb=# delete from mytable where id='3ab226cf-a972-463d-b5a1-148fe39672b5';

If you don’t have a date/timestamp column in your table, you could use LIMIT and OFFSET to narrow it down. I.e. a series of statements like (increasing the offset each time):

select * from mytable order by id limit 1000 offset 5000;


The migration itself was pretty easy, but I wanted to gain some of bash’s features, specifically the emac-like keystrokes for moving forward/back a word in the command line, and also to the beginning and end of the line. The keystrokes, among others, are:

ctrl-a Move cursor to beginning of line
ctrl-e Move cursor to end of line
meta-b Move cursor back one word
meta-f Move cursor forward one word
ctrl-w Cut the last word
ctrl-u Cut everything before the cursor
ctrl-k Cut everything after the cursor
ctrl-y Paste the last thing to be cut
ctrl-_ Undo


I’m a big fan of iTerm – I use it as my primary terminal and have been for the last few years, as it matured in terms of performance and has a lot of great features. The most compelling feature it has over the built in OSX Terminal.app, for me, is that you can specify which characters to include when selecting words. A common operation in a terminal is to select a path, i.e. /home/myuser/testfile. In Terminal.app, if you double click on home, it will just select home – so you have to go back, double click on the first “/” character, and mouse right to get the whole word. With iTerm, you can set a preference with characters to include when connecting, i.e. “/”, so when I double click on home in iTerm, it selects the full path. Specifically, in iTerm -> Preferences -> Mouse, i have “Characters considered part of word:” set to /-_.

One thing I couldn’t figure out was how to get the “meta” key in iterm set to the alt/option key on the mac keyboard. Turns out it’s buried in a non-intuitive place, but it can be done. In iTerm, go to Bookmarks->Manage Profiles, and under Keyboard Profiles select Global. On the right side, you need to set Option Key as “+Esc” for this to work – yes, its strange that it has to be Esc instead of Meta, but hey, at least it all works.

To set up OracleVM 2.2 Server, with a NFS repository for VM’s:

• Download the ISO, and run through the install process
• Once it is up and you are logged in:
• Download and install the updated ovs-agent rpm from oracle here: http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
  • wget http://oss.oracle.com/oraclevm/server/2.2/RPMS/ovs-agent-2.3-31.noarch.rpm
  • rpm -Uvh ovs-agent-2.3-31.noarch.rpm
• Add your repository with the repos.py tool, make it the root repository, and initialize it:
  
# /opt/ovs-agent-2.3/utils/repos.py --new mynfsserver.company.com:/volumes/datavol/ovs
# /opt/ovs-agent-2.3/utils/repos.py -l
# /opt/ovs-agent-2.3/utils/repos.py -r 7c885f7e-7dfe-4108-bae4-4dc329e2f017
# /opt/ovs-agent-2.3/utils/repos.py -i


At this point, you are ready to go, and can continue setting things up (including your Oracle VM Manager VM).

Over the past year or two, we have transitioned all our servers and hosting to Xen – specifically, we use the Oracle VM management tools on top of xen, and most of our VM’s are Centos 5 x86_64. Since we use NFS NAS’ as storage across our infrastructure, it would be very convenient if we could use a NFS volume as the root drive for VM’s. With the NFS root, we gain things like easy use of filer snapshots, and on-the-fly volume resizing – if we’re ever running short on space in a given VM, its a single command (or a click on a web page) to expand the root drive.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

The steps below are the result of a lot of work – the RHEL5 kernel has code in it for an NFS root, but I was never able to get it work correctly, at least under xen. In the end, after lots of experimenting, I was able to build a new kernel from kernel source, that is compatible with Xen and a NFS-Root. Repeat: you cannot build a custom RHEL5 kernel that is capable of booting from NFS under xen.

There are shortcomings/tradeoffs with this approach – you are not able to do any NFS exports from the nfs-root vm, and there is a bit more performance overhead with an NFS-root vm. If you are using a VM to host a high-transaction DB, for instance, I wouldn’t recommend a NFS-root, but for most purposes, it works and performs just fine.

First, you need to build a kernel that supports both xen and an NFS-root – you’ll need a linux machine with a complete build environment, i.e. gcc, make, etc. I have been through this process with 2.6.31, but I would guess that the latest stable kernel version available at http://kernel.org will work just fine.
- Download the latest kernel source
- unzip, cd into source directory
- Copy in attached .config as starting point ( copy as …./linux-(version)/.config )
- make menuconfig

Important config options are listed here:

http://www.virtuatopia.com/index.php/CentOS_5_-_Build_a_Custom_Xen_Kernel_with_NFS_Root_Filesystem_(CONFIG_ROOT_NFS)_Support

Most importantly, nfs client options and nfs_root need to be built into kernel (not as modules).  Also need to make sure to build Xen modules, and select most of the iptables filters (state is an important one). 

- make
- make modules_install

Once the make and install is complete, you will have a kernel capable of nfs-booting under xen. To collect all the pieces needed:
- copy over the file vmlinux in the base build directory (this one is ~85MB), this is the kernel (can’t use bzImage with xen nfs boot).
- Tar up /lib/modules/kernelVer to distribute to nfs client vm.

See the next article for creating a proper vm.cfg under xen / Oracle VM.

My previous solution was simple – I considered one computer my “master”, and simply sync’ed all the files from it to my other computers. Since I do most of my itunes listening (and rating) at work, I used my iMac as the master and my macbook and home desktop were the slaves. This has its downfalls though, as anything I added or rated on my other systems never made its way back to the master system, and I had to make those changes manually.
I decided to search again a few days ago, and found two applications that do exactly what I want:

myTuneSync from SocketHead studios
Syncopation from Sonzea

I checked them both out, and they both work as advertised (both offer free time-limited trials). You install the application, point it at the other itunes library to keep in sync with, and set them up for automatic updates. Syncopation is the clear winner for me, with its cleaner interface, faster performance, and ability to synchronize file deletions as well. Syncopation works only on macs, so if you have a mix of mac and Windows (or only Windows), myTuneSync is for you.

Interestingly, I found these programs two days before the new iTunes 9 was released, and looking at the keynote, I wondered if I just wasted money on something that was now a standard feature of iTunes. (Unfortunately) It turns out that the syncing included with iTunes is limited to purchases from the iTunes store, and doesn’t include anything added by you. It also seems that the metadata does not get transferred over, only the purchased song files.

Since PostgreSQL is our database of choice, I started researching the process, thinking it would be a couple mysql-esqe GRANT statements and that would be it. Turns out that it is a huge PITA in postgres – even my solution I’m documenting here has its shortcomings, but as far as I know is the best / only way to accomplish this task. I ran across quite a few sites that helped with pieces of this, but none that actually tied the process together for production usage. I am by no means a Postgres expert, but do have a good bit of experience mucking around with back-end settings and figuring how to script common tasks.

There are two main things to watch out for when trying to create a read-only user in PostgreSQL, especially if you come from a DB like MySQL:

For this example, we’ll use database name “mydb”, database user/owner “mydbuser”, and we’ll create a read-only user named “mydbuser_ro”. This assumes that you did not define a schema for your database and are using the default “public” schema.

Once complete, you can verify the settings with a quick sql query. You should see something like this, with the user=UC (for Usage/Create), and user_ro=U (for Usage):
mydb=> select * from pg_namespace where nspname='public';
nspname | nspowner | nspacl
---------+----------+------------------------------------------------------------------
public | 10 |
{postgres=UC/postgres,mydbuser=UC/postgres,mydbuser_ro=U/postgres}