David Latham

Automatically mount NVME volumes in AWS EC2 on Windows with Cloudformation and Powershell Userdata

2021-05-14T16:13:00.006+12:00

Introduction

This post explains how I go about with AWS Cloudformation and Powershell userdata scripts.

attaching EBS volumes to Windows EC2 instances
identifying them in the OS
initializing them
formatting them
and assigning labels and drive letters.

Prerequisites

You should know something about Cloudformation and Powershell.

The AWS Image you use for your EC2 instance should be one provided by AWS and should have the following utility already deployed to C:\ProgramData\Amazon\Tools\ebsnvme-id.exe

AWS Cloudformation Step

I typically do not declare EBS Volumes together with the EC2 Instance declaration in Cloudformation but prefer to declare them as separate resources along with their own ebs attachment resources. Here is an example EC2 instance with three additional volume declarations on top of the root volume which is automatically mounted on C:\

 WindowsServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
            VolumeSize: 65
            Encrypted: true
            VolumeType: gp3
      ImageId: !Ref WindowsImageId
      KeyName: !Ref WindowsKeyPairName
      InstanceType: !WindowsServerInstanceType
      CreditSpecification:
        CPUCredits: standard
      IamInstanceProfile: !Ref WindowsServerInstanceProfile
      SecurityGroupIds:
        - !Ref ManagementSecurityGroup
        - !Ref ApplicationSecurityGroup
      SubnetId: !Ref WindowsServerSubnetId
      Tags:
        - Key: Name
          Value: WindowsServerInstance
      UserData:
        Fn::Base64: |
          <powershell>
            mkdir C:\setup
            Read-S3Object -BucketName example-bucket -Key /userdata/windows_bootstrap.ps1 -File C:\setup\bootstrap.ps1
            . C:\setup\bootstrap.ps1
          </powershell>

  DataVolume:
    Type: AWS::EC2::Volume
    Properties:
      Size: 100
      VolumeType: gp3
      Encrypted: true
      AvailabilityZone: !GetAtt WindowsServerInstance.AvailabilityZone
      Tags:
        - Key: Name
          Value: DataVolume
  DataVolumeAttachment:
    Type: AWS::EC2::VolumeAttachment
    Properties:
      InstanceId: !Ref WindowsServerInstance
      VolumeId: !Ref DataVolume
      Device: /dev/sdb

  BackupVolume:
    Type: AWS::EC2::Volume
    Properties:
      Size: 100
      VolumeType: gp3
      Encrypted: true
      AvailabilityZone: !GetAtt WindowsServerInstance.AvailabilityZone
      Tags:
        - Key: Name
          Value: BackupVolume
  BackupVolumeAttachment:
    Type: AWS::EC2::VolumeAttachment
    Properties:
      InstanceId: !Ref WindowsServerInstance
      VolumeId: !Ref BackupVolume
      Device: /dev/sdc

  ScriptsVolume:
    Type: AWS::EC2::Volume
    Properties:
      Size: 5
      VolumeType: gp3
      Encrypted: true
      AvailabilityZone: !GetAtt WindowsServerInstance.AvailabilityZone
      Tags:
        - Key: Name
          Value: ScriptsVolume
  ScriptsVolumeAttachment:
    Condition: DeployB2BResources
    Type: AWS::EC2::VolumeAttachment
    Properties:
      InstanceId: !Ref WindowsServerInstance
      VolumeId: !Ref ScriptsVolume
      Device: /dev/sdd

When this template builds, a windows instance will be available with 3 additional and un-initialized volumes attached.

The userdata property is configured to pull down a powershell script from S3 as permitted by the WindowsServerInstanceProfile. I've not bothered to explain how all that works. I assume if you are reading this, you already know how to allow an EC2 Instance access to objects in S3 via Instance Profiles and Policies.

Userdata Script

The userdata script below, declares a function and calls the function once for each of the volume attachments.

### Function to format a disk based on the disk number, label and drive letter.
function setup_disk {
	param (
		$DiskNumber,
		$Label,
		$DriveLetter
	)
	Initialize-Disk "$DiskNumber" -PartitionStyle MBR
	New-Partition -DiskNumber "$DiskNumber" -DriveLetter "$DriveLetter" -UseMaximumSize
	Format-Volume -DriveLetter "$DriveLetter" -FileSystem NTFS -NewFileSystemLabel "$Label" -Confirm:$false
}

# Create a hashtable of disk numbers and block device mappings
$hash = $null
$hash = @{}

get-disk | where partitionstyle -eq 'raw' | where-object IsSystem -eq $False| select Number | foreach-Object {
  $num = $_.Number
  $blk = (c:\ProgramData\Amazon\Tools\ebsnvme-id.exe $num | Select-String -Pattern 'Device Name: .*$'| ConvertFrom-String -Delimiter ':' | Select P2).P2
  $blk = $blk -replace '^......'

  $hash.add($blk, $num)
}

# Set up each of the volume attachments
setup_disk $hash.sdb "Data" "D"
setup_disk $hash.sdc "Backup" "E"
setup_disk $hash.sdd "Scripts" "I"

Explanation

The get-disk command returns a list of all the disks attached to the system. We can use the get-disk command to iterate through each attached disk that is not a system disk (exclude C:\) and then call the AWS Provided ebsnvme-id.exe tool to find out which device name it has and add it to a hashmap called $hash. This just maps Disk Number to Volume Attachment Device Name. Unfortunately Windows does not always enumerate disks in the same order so we have to build this map correctly. As a general rule it is never a good idea to rely on the order of things in computers.

Seeing as though our Cloudformation VolumeAttachment resources declare these device names, we have the mapping we need to proceed.

PS C:\setup> $hash

Name                           Value
----                           -----
sdb                            2
sdc                            1
sdd                            3

Finally, our little script, calls the setup_disk function with the value of the hash for each volume we want and the parameters for Label and Drive Letter.

ASUS USB-N10 Nano on OpenSuse 13.1 Linux

2013-12-26T19:14:00.000+13:00

Today I bought myself a wifi adapter for my Opensuse desktop PC. I was tired of running a network cable across the carpet to my modem.

The model I bought was an ASUS USB-N10 Nano from PB Technologies in Henderson. It was less than $20 so I thought a good deal. When I bought it, I completely ignored all the printed garbage about support for Windows and Mac and figured, that it will be supported under linux.

It is supported under Linux but it was not too simple to set up on my Opensuse.

In the end and after much google-ing, I found this repository:
https://github.com/pvaret/rtl8192cu-fixes

The driver that ships with the kernel is not so hot. The network keeps dropping even though it appears to remain connected. With this version, things are much more stable.

Also it compiles nicely under the latest kernel. The proprietry driver from Realtek did not compile under the kernel shipped with Opensuse 13.1.

So here is what I ended up doing:

~ git clone https://github.com/pvaret/rtl8192cu-fixes.git
~ su - (become root)
# zypper in dkms kernel-devel kernel-headers gcc make
# cd /path/to/git/repo... (you get the picture)
# cd ../ (go to directory above tree) on my system I put this source tree under /usr/local/src/
# dkms add rtl8192cu-fixes
# dkms install 8192cu/1.8
# depmod -a
# cp blacklist-native-rtl8192.conf /etc/modprobe.d
# echo "options 8192cu rtw_power_mgnt=0 rtw_enusbss=0" > /etc/modprobe.d/50-8192cu.conf
# reboot

And that's it. Now if the kernel is ever updated during normal patching, dkms will recompile our driver.

EDIT:
I have made an RPM package for this driver. It's hosted on the openSUSE build service. Currently I have successfully managed to create rpm's for Centos, Redhat and Fedora (x86_64) only. The openSUSE one failed because the build environment does not have DKMS installed and so the install test fails at the very end.
Anyway - find the packages here: https://build.opensuse.org/package/show/home:linuxplayground/8192cu-1.8

so the openSUSE version is here: 8192cu-1.8-1.0.x86_64.rpm

These RPM packages will:

save the source in /usr/src/8192cu-1.8
Add the source tree to DKMS
Build the kernel module using dkms build ...
Install the kenrel module using dkms install ...

To uninstall the module and all the dkms source, builds and modules then just execute: #rpm -e 8192cu
:-)

Restrict Commands executed via pre-shared SSH keys

2013-12-20T11:17:00.000+13:00

Introduction

Using pre-shared SSH keys is a great way to make logging into remote hosts quick and easy. No pesky passwords to remember. The downside is that if your organization relies on passwords for access control, the pre-shared key will negate any password control you have. For example: If your organization rolls a password for a system account, and the new password is not shared with all the original people, then some people who should not have access any more will continue to have access via their pre-shared key.
Also, anyone with access to a user's account on a client host, could access the system account via the pre-shared key on the remote host. This is not ideal.

Sometimes, it is neccessary to allow certain commands only to be executed over SSH from specified client hosts without a password. Especially when thinking about automated tasks.

In my examples, I will demonstrate how we can create a simple remote procedure call type scenario using pre-shared keys and some built in linux security tools that lock things down.

These examples will use the "oracle" user which has been set up on both the client and remote hosts identically.

[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

The Steps

Create the public private key on the private host

Use ssh-keygen to create a public/private key pair without a password.

[oracle@centos-dev-1 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa): [ENTER]
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase): [ENTER]
Enter same passphrase again: [ENTER]
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
64:cf:72:de:f7:85:af:fe:36:87:ce:8c:6e:31:8d:ed oracle@centos-dev-1.vbox.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|        o        |
|       o o       |
|        S +  +   |
|         + .+ o. |
|          . .+o..|
|            .=E++|
|           ooo*+*|
+-----------------+
[oracle@centos-dev-1 ~]$ ls -l .ssh/
total 8
-rw-------. 1 oracle oinstall 1675 Dec 20 10:33 id_rsa
-rw-r--r--. 1 oracle oinstall  412 Dec 20 10:33 id_rsa.pub

Share the public key to the remote host

You can use the ssh-copy-id tool or you could manually do it. I will use the ssh-copy-id tool to share the key.

[oracle@centos-dev-1 ~]$ ssh-copy-id -i /home/oracle/.ssh/id_rsa.pub oracle@centos-dev-2
The authenticity of host 'centos-dev-2 (192.168.56.102)' can't be established.
RSA key fingerprint is df:87:48:48:0a:15:fa:ca:e4:a7:41:8e:9e:af:19:b5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'centos-dev-2,192.168.56.102' (RSA) to the list of known hosts.
oracle@centos-dev-2's password: 
Now try logging into the machine, with "ssh 'oracle@centos-dev-2'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[oracle@centos-dev-1 ~]$ ssh oracle@centos-dev-2
Last login: Thu Dec 19 13:38:03 2013 from centos-dev-1.vbox.local
[oracle@centos-dev-2 ~]$ hostname
centos-dev-2.vbox.local
[oracle@centos-dev-2 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-2 ~]$ exit
logout
Connection to centos-dev-2 closed.

At this point, we have successfully enabled transparent ssh logins using a pre-shared public key from centos-dev-1 to centos-dev-2 as the "oracle" user.

The result was a file created on the remote host called /home/oracle/.ssh/authorized_keys which looks like this:

[oracle@centos-dev-2 ~]$ hostname
centos-dev-2.vbox.local
[oracle@centos-dev-2 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-2 ~]$ cd /home/oracle/.ssh/
[oracle@centos-dev-2 .ssh]$ ls -l
total 4
-rw-------. 1 oracle oinstall 412 Dec 20 10:35 authorized_keys
[oracle@centos-dev-2 .ssh]$ cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqdZI3PbacEoZ4uIk1hIc9Tl2pmLZtv0Y1/ZowqNOLRRQV84ziiNXoUXa8y+x9DxMzIPxKHdyYy0k0HFl06oIjfP4l3P3GY9L0MM/oYxYNr0UaJz46hDy7p3cKVjmeY/Xn3PsepNY3kAdp4zdNb1Hfs9G/PiB70LTfseJZjqwBJ6edYAt7YztWuO9OgrmDoW7QAdsPTlQGF9iW0gUqH/1UA5hXtGePRq2My9ahXmXOu1hy3H1wFygCBZaxP7A3qohgvVIT76DQ1mxhgLDiQCVkTzxVhHgMmVi4A7C0vpr2G5WxpLblVHWpjnXn/eAdQBQLnFtj/pza1U5K3Dy/GDxMQ== oracle@centos-dev-1.vbox.local

Restrict the public key on the remote host

Here we edit the /home/oracle/.ssh/authorized_keys file that was created in the previous step to include some restrictions for the key. We prepend the following to the key:

from="centos-dev-1.vbox.local",command="/usr/local/bin/remote-commands.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

The key now looks like this:

from="centos-dev-1.vbox.local",command="/usr/local/bin/remote-commands.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqdZI3PbacEoZ4uIk1hIc9Tl2pmLZtv0Y1/ZowqNOLRRQV84ziiNXoUXa8y+x9DxMzIPxKHdyYy0k0HFl06oIjfP4l3P3GY9L0MM/oYxYNr0UaJz46hDy7p3cKVjmeY/Xn3PsepNY3kAdp4zdNb1Hfs9G/PiB70LTfseJZjqwBJ6edYAt7YztWuO9OgrmDoW7QAdsPTlQGF9iW0gUqH/1UA5hXtGePRq2My9ahXmXOu1hy3H1wFygCBZaxP7A3qohgvVIT76DQ1mxhgLDiQCVkTzxVhHgMmVi4A7C0vpr2G5WxpLblVHWpjnXn/eAdQBQLnFtj/pza1U5K3Dy/GDxMQ== oracle@centos-dev-1.vbox.local

The extra options are broken down as follows:

from="centos-dev-1" Only allow this key to be used when connections are made from centos-dev-1 host.
command="/usr/local/bin/remote-commands.sh" This is a script that will be executed by the owner of this key (oracle) whenever the key is used to establish an ssh connection. We will write the script soon.
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty These options add extra security and prevent the key being used for anything other than running the command specified in command="".

None of this is very useful if the oracle user can simply log into the host using his password and edit this file. Use the chattr command to lock the file for editing as follows: (NOTE: Only the root user can do this)

[root@centos-dev-2 ~]# hostname
centos-dev-2.vbox.local
[root@centos-dev-2 ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@centos-dev-2 ~]# lsattr /home/oracle/.ssh/authorized_keys 
-------------e- /home/oracle/.ssh/authorized_keys
[root@centos-dev-2 ~]# chattr +i /home/oracle/.ssh/authorized_keys 
[root@centos-dev-2 ~]# lsattr /home/oracle/.ssh/authorized_keys 
----i--------e- /home/oracle/.ssh/authorized_keys
[root@centos-dev-2 ~]# rm /home/oracle/.ssh/authorized_keys 
rm: remove regular file `/home/oracle/.ssh/authorized_keys'? y
rm: cannot remove `/home/oracle/.ssh/authorized_keys': Operation not permitted
[root@centos-dev-2 ~]# ls -l /home/oracle/.ssh/authorized_keys 
-rw-------. 1 oracle oinstall 551 Dec 20 10:41 /home/oracle/.ssh/authorized_keys
[root@centos-dev-2 ~]# su - oracle
[oracle@centos-dev-2 ~]$ cat .ssh/authorized_keys 
from="centos-dev-1.vbox.local",command="/usr/local/bin/remote-commands.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAqdZI3PbacEoZ4uIk1hIc9Tl2pmLZtv0Y1/ZowqNOLRRQV84ziiNXoUXa8y+x9DxMzIPxKHdyYy0k0HFl06oIjfP4l3P3GY9L0MM/oYxYNr0UaJz46hDy7p3cKVjmeY/Xn3PsepNY3kAdp4zdNb1Hfs9G/PiB70LTfseJZjqwBJ6edYAt7YztWuO9OgrmDoW7QAdsPTlQGF9iW0gUqH/1UA5hXtGePRq2My9ahXmXOu1hy3H1wFygCBZaxP7A3qohgvVIT76DQ1mxhgLDiQCVkTzxVhHgMmVi4A7C0vpr2G5WxpLblVHWpjnXn/eAdQBQLnFtj/pza1U5K3Dy/GDxMQ== oracle@centos-dev-1.vbox.local

As you can see, once this attribute is set on the file, not even the root user can edit or delete it. The root user will first need to remove the "immutable" attribute with "chattr -i /home/oracle/.ssh/authorized_keys".

The oracle user can still read the file which is a requirement of the SSH daemon.

Write a command parser on the remote host

This script will parse the command sent by the ssh client and execute based on some simple tests.

[oracle@centos-dev-2 ~]$ ls -l /usr/local/bin/remote-commands.sh 
-rwxr-xr-x. 1 root root 315 Dec 19 13:30 /usr/local/bin/remote-commands.sh
[oracle@centos-dev-2 ~]$ cat /usr/local/bin/remote-commands.sh 
#!/bin/sh
#
if [ -z "${SSH_ORIGINAL_COMMAND}" ]; then
        echo "This script can only be executed via SSH remote calls"
        exit
fi

case "$SSH_ORIGINAL_COMMAND" in
  "get hosts")
    cat /etc/hosts
    ;;
  "get resolv.conf")
    cat /etc/resolv.conf
    ;;
  "get env")
        env
        ;;
  *)
   echo "Bad Command" 
    ;;
esac
exit

The script is owned by root:root but all users have read and execute permissions on the script.

That's it! We are ready to begin our tests now.

Test

The script we write allows clients to execute 3 commands:

get hosts
get resolv.conf
get env

Any other command will be rejected.

Using the key to establish an ssh connection without passing a command will be rejected. So logins are now disabled.
Executing the script from the local host will be rejected too. This script should only be called by remote hosts.

Test "get hosts"

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh centos-dev-2 get hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Test "get resolv.conf"

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh centos-dev-2  resolv.conf
; generated by /sbin/dhclient-script
search vbox.local
nameserver 192.168.56.1

Test "get env"

My fake oracle environment is nothing like what a real one would look like. This example is given just to prove that the remote script does execute with all of the expected environment variables.

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh centos-dev-2 get env
SELINUX_ROLE_REQUESTED=
SHELL=/bin/bash
SSH_CLIENT=192.168.56.101 46697 22
SELINUX_USE_CURRENT_RANGE=
USER=oracle
PATH=/usr/local/bin:/bin:/usr/bin
MAIL=/var/mail/oracle
PWD=/home/oracle
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
HOME=/home/oracle
SHLVL=2
SSH_ORIGINAL_COMMAND=get env
LOGNAME=oracle
CVS_RSH=ssh
SSH_CONNECTION=192.168.56.101 46697 192.168.56.102 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1
_=/bin/env

Test some bad commands

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh centos-dev-2 userdel oracle
Bad Command
[oracle@centos-dev-1 ~]$ ssh centos-dev-2 "su -c \"userdel oracle\""
Bad Command
[oracle@centos-dev-1 ~]$ ssh centos-dev-2 rm -fr /*
Bad Command

Test logging in without a password

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh centos-dev-2
PTY allocation request failed on channel 0
This script can only be executed via SSH remote calls
                                                     Connection to centos-dev-2 closed.

Using SSH to force the requirement for a password

Because a shared key exists, ssh will automatically try to use it for authentication. We now need to explicitly declare that no public private key should be used for authentication.

[oracle@centos-dev-1 ~]$ hostname
centos-dev-1.vbox.local
[oracle@centos-dev-1 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-1 ~]$ ssh -o PubkeyAuthentication=no centos-dev-2
oracle@centos-dev-2's password: 
Last login: Fri Dec 20 10:35:22 2013 from centos-dev-1.vbox.local
[oracle@centos-dev-2 ~]$ hostname
centos-dev-2.vbox.local
[oracle@centos-dev-2 ~]$ id
uid=500(oracle) gid=501(oinstall) groups=501(oinstall) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[oracle@centos-dev-2 ~]$

Python suds (SOAP Library) does not support SSLv3

2013-12-12T12:07:00.000+13:00

The Oracle Virtual Machine (OVM) manager exposes a SOAP (wsdl) web-service. At the time of writing this can be found here: (for version 3.2.x)

https://<hostname>:7002/ovm/core/wsapi/soap?wsdl

You can access this url in your web browser and will be prompted to accept the self signed cert etc. (There is a way to add a real certificate but that's not what this article is about.)

In order to consume this webservice in python, I thought to try out the (now quite old) "suds" library. Opensuse still ship this library and you can install it with:

zypper in python-suds

EDIT: I have added the forked library to github here: https://github.com/linuxplayground/suds-sslv3-fork
The problem I found was that when trying to connect to the service, I was receiving an exception error around SSL.

Here is how it looked...
The example code:

#!/usr/bin/python
from suds.client import Client
client = Client(url='https://ovm:7002/ovm/core/wsapi/soap?wsdl')
print client

Output:

david.latham@davidpc:~/bin/ovm-webservices> python broken.py 
Traceback (most recent call last):
  File "broken.py", line 3, in 
    client = Client(url='https://ovm:7002/ovm/core/wsapi/soap?wsdl')
  File "/usr/lib/python2.7/site-packages/suds/client.py", line 112, in __init__
    self.wsdl = reader.open(url)
  File "/usr/lib/python2.7/site-packages/suds/reader.py", line 152, in open
    d = self.fn(url, self.options)
  File "/usr/lib/python2.7/site-packages/suds/wsdl.py", line 136, in __init__
    d = reader.open(url)
  File "/usr/lib/python2.7/site-packages/suds/reader.py", line 79, in open
    d = self.download(url)
  File "/usr/lib/python2.7/site-packages/suds/reader.py", line 95, in download
    fp = self.options.transport.open(Request(url))
  File "/usr/lib/python2.7/site-packages/suds/transport/https.py", line 60, in open
    return  HttpTransport.open(self, request)
  File "/usr/lib/python2.7/site-packages/suds/transport/http.py", line 62, in open
    return self.u2open(u2request)
  File "/usr/lib/python2.7/site-packages/suds/transport/http.py", line 118, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/lib64/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError:

That last line, "SSL23_GET_SERVER_HELLO:unknown protocol" had me stuck for ages. Eventually I found a blog post that suggested this can be fixed by adding a new HTTPHandler to the URLLIB2 Opener.
Rather than mess with the urllib2 library, I decided to download the source for the suds library, and fix the library.
Download suds from here:

svn co http://svn.fedorahosted.org/svn/suds/trunk

In the source tree add the following httpsconnectionv3.py to /path/to/downloaded/source/suds/transport/httpsconnectionv3.py

#!/usr/bin/python
import httplib, ssl, urllib2, socket

class HTTPSConnectionV3(httplib.HTTPSConnection):
    def __init__(self, *args, **kwargs):
        httplib.HTTPSConnection.__init__(self, *args, **kwargs)
        
    def connect(self):
        sock = socket.create_connection((self.host, self.port), self.timeout)
        if self._tunnel_host:
            self.sock = sock
            self._tunnel()
        try:
            self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_SSLv3)
        except ssl.SSLError, e:
            print("Trying SSLv3.")
            self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_SSLv23)
            
class HTTPSHandlerV3(urllib2.HTTPSHandler):
    def https_open(self, req):
        return self.do_open(HTTPSConnectionV3, req)

Then update /path/to/downloaded/source/suds/transport/http.py with the following additions:

include the new library in the imports
- from httpsconnectionv3 import *
Add the following line to the function "u2handlers" after the line "handlers.append(u2.ProxyHandler(self.proxy))"
- handlers.append(HTTPSHandlerV3())

Now you are ready to test using this library instead of the default suds library that ships with your OS.
The example code but this time you place your example code in the downloaded source tree so that python uses the local library. OR You can make the above modifications to your installed library which should be somewhere in /usr/lib/python-2.7/site-packages/suds.
Up to you.
Example Code:

david.latham@davidpc:~/bin/ovm-webservices/ovm_wsapi> cat getVMList.py
#!/usr/bin/python
from suds.client import Client
client = Client(url='https://ovm:7002/ovm/core/wsapi/soap?wsdl')
username='admin'
password='xxxxxxxx'
loginResponse = client.service.login(username,password)
print "Logging in .... <%s>" % loginResponse
if loginResponse == True:
        vmlist = client.service.vmGetAll()
        for vm in vmlist:
                if vm.vmRunState == 'RUNNING' or vm.vmRunState == 'STOPPED':
                        print "%s | %s" % (vm.name, vm.vmRunState)
        logoutResponse = client.service.logout()
        print "Logging out ... <%s>" % logoutResponse
else:
        print "login failure"

Output:

david.latham@davidpc:~/bin/ovm-webservices/ovm_wsapi> ./getVMList.py 
Logging in .... 
puppetmaster | RUNNING
ksplice-test | RUNNING
zimbra-test-el6 | STOPPED
ksplice-client | RUNNING
DB2_Server_RHEL58 | STOPPED
ora133 | STOPPED
secret_host_1 | STOPPED
zimbra-test | RUNNING
sqlsandpit | RUNNING
secret_host_2 | STOPPED
repo_server | RUNNING
Logging out ...

So that fixes suds for sslv3

Simple symmetric / Shared key encryption tutorial

2013-07-02T20:58:00.002+12:00

I was looking for a way to explain cryptography to my 8 year old son this evening and we devised a simple share key encryption / decryption method. The key is symmetrical because both sides have the same key and the same key is used for both encrypting and decrypting.

We took a simple ceasar cipher and extended it slightly by creating a stronger key. Something like this:

Plain Text = HELLO WORLD

Key = [2, -5, 3, 7]

to encrypt we take the positional value for each letter and apply the replacement for the next key bit in turn. For example:

Start at keybit = 1 [2]

H = 8, H + 2 = 10, 10 = J

keybit = 2 [-5]

E = 5, E - 5 = 0 (26), 26 = Z

keybit = 3 [3]

L = 12, 12 + 3 = 15, 15 = O

keybit = 4 [7]

L = 12, 12 + 7 = 19, 19 = S

Now start at key bit =1

O = 15, 15 + 2 = 17, 17 = Q

and so on.

So long as the same key is used for decryption the thing works a treat.

So I explained that simple key can be made and shared between friends then private messages can be sent to and fro and only those with the key will know what the messages mean.

To demonstrate I made a program that he can use to make the process a bit quicker...

#!/usr/bin/python

import sys

alphabet = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z',' ',',','.','!','?','\'','#','@','$','%','^','&','(',')','-','_','=','+',']','[','{','}','\\','|','/','"',':',';','<','>','`','~','1','2','3','4','5','6','7','8','9','0']

def encrypt(key, message):
    ciphertext = []
    pos = 0
    for c in message:
        i = alphabet.index(c)
        j = (i + key[pos]) % len(alphabet)
        ciphertext.append(alphabet[j])
        pos += 1
        if pos > len(key)-1: pos = 0
    return "".join(ciphertext)


if __name__ == '__main__':
    key = [3,-6,2,-7,23435363, -3243543, 324]
    for line in sys.stdin:
        encrypted = encrypt(key, line.strip())
        print encrypted

# vim:expandtab:ts=4:softtabstop=4:autoindent

And test it as follows

dave@davidpc:~/bin/simpleenc> echo "Hello World" | ./encrypt.py 
KYneJ9!dave@davidpc:~/bin/simpleenc> echo "Hello World" | ./encrypt.py 
KYneJ9!rlnWrlnW

The Decrypt program is almost identical (and really shouldn't be a separate programme at all)

#!/usr/bin/python

import sys

alphabet = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z',' ',',','.','!','?','\'','#','@','$','%','^','&','(',')','-','_','=','+',']','[','{','}','\\','|','/','"',':',';','<','>','`','~','1','2','3','4','5','6','7','8','9','0']

def decrypt(key, message):
    plaintext = []
    pos = 0
    for c in message:
        i = alphabet.index(c)
        j = (i - key[pos]) % len(alphabet)
        plaintext.append(alphabet[j])
        pos += 1
        if pos > len(key)-1: pos = 0
    return "".join(plaintext)
        

if __name__ == '__main__':
    key = [3,-6,2,-7,23435363, -3243543, 324]
    for line in sys.stdin:
        decrypted = decrypt(key, line.strip())
        print decrypted

# vim:expandtab:ts=4:softtabstop=4:autoindent

And the test...

dave@davidpc:~/bin/simpleenc> cat << EOF | ./decrypt.py 
> KYneJ9!rlnW
> EOF
Hello World

Samba4 Windows AND Linux authentication

2013-05-16T17:05:00.000+12:00

Introduction

This post describes all the millions of configuration files you need to set up on a CLIENT LINUX server / machine in order to ssh to it with your domain credentials. I have this working in a LAB environment with the following caveats:

SELINUX disabled (work on this required)
Firewall Disabled (not hard to fix this if required)
DNS Setup not working completely for Forwarding to external Nameserver by Samba4. Don't know why. (more investigation required)
You will need to have a user called binduser which has permissions in AD to look up other users. I am sure there is documentation on this around somewhere :)

Install packages

yum -y install samba-winbind pam_ldap pam_krb5 nss-pam-ldapd oddjob-mkhomedir bind-utils

Configure openldap (/etc/openldap/ldap.conf)

[root@linuxclient ~]# cat /etc/openldap/ldap.conf 
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

#BASE   dc=example,dc=com
#URI    ldap://ldap.example.com ldap://ldap-master.example.com:666

#SIZELIMIT      12
#TIMELIMIT      15
#DEREF          never

TLS_CACERTDIR /etc/openldap/cacerts
URI ldap://pdc.vbox.local/
BASE dc=vbox,dc=local

Configure PAM_LDAP (/etc/pam_ldap.conf)

[root@linuxclient ~]# cat /etc/pam_ldap.conf 
uri ldap://pdc.vbox.local/
base dc=vbox,dc=local
binddn cn=binduser,cn=Users,dc=vbox,dc=local
bindpw SECRET
scope sub
nss_base_passwd cn=Users,dc=vbox,dc=local
nss_base_shadow cn=Users,dc=vbox,dc=local
nss_base_group cn=Users,dc=vbox,dc=local
nss_map_objectclass posixAccount user
nss_map objectclass shadowAccount user
nss_map objectclass posixGroup group
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute uniqueMember member
nss_map_attribute shadowLastChange pwdLastSet
pam_login_attribute uid
pam_filter objectClass=user
pam_password md5
pam_member_attribute member
ssl no
tls_cacertdir /etc/openldap/cacerts
am_sasl_mech DIGEST-MD5

Configure KRB5 (/etc/krb5.conf)

[root@linuxclient ~]# cat /etc/krb5.conf 
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/admind.log

[libdefaults]
 default_realm = VBOX.LOCAL
 dns_lookup_realm = true
 dns_lookup_kdc = true

[domain_realm]
.vbox.local = VBOX.LOCAL
vbox.local = VBOX.LOCAL

[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
validate = false
}

[realms]
VBOX.LOCAL = {
}

VBOX.LOCAL = {
}

Configure the NSLC Daemon (/etc/nslcd.conf)

[root@linuxclient ~]# cat /etc/nslcd.conf                                                                                                                                                       
binddn cn=binduser,cn=Users,dc=vbox,dc=local                                                                                                                                                  
bindpw SECRET                                                                                                                                                                                
scope sub                                                                                                                                                                                       
base dc=vbox,dc=local                                                                                                                                                                           
scope  group  sub                                                                                                                                                                               
scope  hosts  sub                                                                                                                                                                               
pagesize 1000                                                                                                                                                                                   
referrals off
filter passwd (&(objectClass=user)(!(objectClass=computer)))
map    passwd uid              sAMAccountName
map    passwd homeDirectory    unixHomeDirectory
map    passwd gecos            displayName

filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
map    shadow shadowLastChange pwdLastSet
map    shadow uid              sAMAccountName

filter group (&(objectClass=group)(gidNumber=*))
map group uniqueMember member

uid nslcd
gid ldap
uri ldap://pdc.vbox.local/
ssl no
tls_cacertdir /etc/openldap/cacerts

Configure Authconfig for SSH (/etc/sysconfig/authconfig)

set: USEKERBEROS=yes

[root@linuxclient ~]# cat /etc/sysconfig/authconfig 
IPADOMAINJOINED=no
USEMKHOMEDIR=yes
USEPAMACCESS=no
CACHECREDENTIALS=yes
USESSSDAUTH=no
USESHADOW=yes
USEWINBIND=no
USESSSD=no
USEDB=no
FORCELEGACY=no
USEFPRINTD=no
USEHESIOD=no
FORCESMARTCARD=no
USELDAPAUTH=no
IPAV2NONTP=no
USELDAP=yes
USECRACKLIB=yes
USEIPAV2=no
USEWINBINDAUTH=yes
USESMARTCARD=no
USELOCAUTHORIZE=yes
USENIS=no
USEKERBEROS=yes
USESYSNETAUTH=no
PASSWDALGORITHM=sha512
USEPASSWDQC=no

Update authconfig

authconfig --updateall

Check PAM Configuration

[root@linuxclient ~]# cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        sufficient    pam_krb5.so use_first_pass
auth        sufficient    pam_winbind.so use_first_pass
auth        required      pam_deny.so

account     required      pam_unix.so broken_shadow
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
account     [default=bad success=ok user_unknown=ignore] pam_winbind.so
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3 type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    sufficient    pam_krb5.so use_authtok
password    sufficient    pam_winbind.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     optional      pam_oddjob_mkhomedir.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_krb5.so

Configure DNS Settings

You need various AD names to resolve. Especially for kerberos... Note: There is a problem here. The Domain controller must FORWARD dns traffic it knows nothing about.

[root@linuxclient ~]# cat /etc/resolv.conf 
#nameserver must be the IP address of the domain controller.
domain vbox.local
nameserver 192.168.56.101

Reboot

A reboot is required to make sure everything works.

Disable the touch pad while typing on Linux

2013-05-02T09:27:00.001+12:00

The Problem:

If you have a Synaptics touch-pad like the one I have on my HP ProBook 6560b, you might want to learn how to disable the touch-pad while typing. This particular laptop has a touch-pad that is positioned poorly so it is common for your palm to be in contact with the pad while you type. The solution is to disable it for 2 seconds while typing. It is a minor inconvenience to have to wait two seconds to use it again but at least your typing is uninterrupted which could be a major inconvenience.

The solution:

Use the syndaemon tool that ships with the synaptics packages. In my case they shipped by default. You could always try the gui...

david.latham@davepc:~> cat bin/touchpad.sh
#!/bin/bash
syndaemon -k -i 2 -d

Now just configure your desktop manager to run this script on start up. It is a trivial thing to do in KDE under system settings -> Sartup and Shutdown, Add Script...

Firewall Script for openSuSE and others

2013-03-29T23:01:00.001+13:00

openSuSE and, I guess, Fedora have introduced their own firewall (iptables) configuration guis and services that try to make the job of configuring iptables easy. While that might be useful in a standard desktop environment, I found that it didn't do much for my Virtualbox nat.

I couldn't create the masqerade rules very easily and I didn't want all the extensive rules governing types of ICMP traffic and logging. As the SuSEfirewall2 service GUI didn't help and the configuration files were too complicated for this old school blogger, I decided to roll my own script.

On my system, I simply disable the SuSEfirewall2 service and then call this script on boot.

You can find this script and any changes I make to it on my github. Of course, I won't be putting every rule I have in place online, that would be a crazy security risk. I just wanted to show the basics that can be easily extended by adding more rules into the filter section.

#!/bin/bash
IPT=/usr/sbin/iptables

# Stop the system firewall
systemctl stop SuSEfirewall2.service

# Reset all
$IPT -F
$IPT -X
$IPT -t nat -F
$IPT -t nat -X
$IPT -t mangle -F
$IPT -t mangle -X

# set default policies
$IPT -P INPUT DROP
$IPT -P OUTPUT ACCEPT
$IPT -P FORWARD DROP 

$IPT -A INPUT -i lo -j ACCEPT
$IPT -A OUTPUT -o lo -j ACCEPT

# filter all all related and established (reply traffic)
$IPT -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# filter allow all traffic from vboxnet0 including new traffic.
$IPT -A INPUT -i vboxnet0 -j ACCEPT

# filter custom rules
$IPT -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW,RELATED,ESTABLISHED -j ACCEPT

# forward traffic to and from vboxnet0
$IPT -A FORWARD -i vboxnet0 -j ACCEPT
$IPT -A FORWARD -o vboxnet0 -j ACCEPT

# postrouting so traffic from vboxnet appears to be from these devices.
$IPT -t nat -A POSTROUTING -s 192.168.56.0/24 -j MASQUERADE

# final rule to block inbound traffic
$IPT -A INPUT -j DROP
$IPT -A FORWARD -j DROP

# turn on ip_forward
sysctl net.ipv4.ip_forward=1

# turn on dnsmasq
systemctl start dnsmasq

Reset the Oracle XE database password for OVM 3.1.1.

2013-02-09T11:22:00.000+13:00

You have installed Oracle OVM with the bundled XE database and after a time, you find that you can no longer log into the OVM Admin console.

You get an error similar to: Unexpected Error logging in. Consult logs...

What has probably happened is that the ovs database password has expired in XE. Now using the XE environment is unsupported and for demo purposes only. In fact the new version of OVM ships with a bundled MySQL database and no longer the XE. Even better, the bundled MySQL database is supported in production environments. So this post is obsolete even before I finish writing it.

To be sure you are facing this problem, grep for OVMDS in /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log

####<Jan 28, 2013 12:19:51 PM NZDT> <Warning> <JDBC> <ovm.e-it.co.nz> <AdminServer> <Odof Tcp Client Thread: /127.0.0.1:54321/128430> <<anonymous>> <> <0000JcuiwIkCslYFLrfP8A1GSLRc000002> <1359328791693> <WL-001129> <Received exception while creating connection for pool "OVMDS": ORA-28001: the password has expired

OK so the XE Database password needs resetting.

First some environment variables:

[root@ovm logs]# echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/xe
[root@ovm logs]# echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/u01/app/oracle/product/11.2.0/xe:/u01/app/oracle/product/11.2.0/xe/bin
[root@ovm logs]# echo $ORACLE_SID
XE

Now use sqlplus as follows: (OldPassWord is the one you used when you installed OVMM. The OVMM installer either prompted for this or you selected the option to use the same password for everything during install... Remember?)

[root@ovm bin]# sqlplus ovs/OldPassWord

SQL*Plus: Release 11.2.0.2.0 Production on Sat Feb 9 10:38:48 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

ERROR:
ORA-28001: the password has expired

Changing password for ovs
New password: NewPassw0rd
Retype new password: NewPassw0rd

Password changed

Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

SQL> exit

Now you need to update the password stored in Weblogic that is used to connect to the database. Happily there is an admin tool for this:
It's found in: /u01/app/oracle/ovm-manager-3/bin

[root@ovm bin]# ./ovm_admin --modifyds XE localhost 1521

Oracle VM Manager Release 3.1.1 Admin tool

Please enter the Oracle VM Manager database schema : ovs

Please enter the schema password for ovs : NewPassw0rd
Please re-enter the password : NewPassw0rd

Please enter the password for weblogic : <password you used when you installed ovmm>

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to WebLogic server ...

Connected ...
Modifying Oracle VM Manager Data Store 'OVMDS'...
Modified Data Store successfully ...
Exiting...
Restart of Oracle VM Manager is required for Data Store change to take effect ...

Now restart OVMM. I like to allow a minute between stop and start. This takes a while on my test environment...

[root@ovm bin]# /etc/init.d/ovmm stop && sleep 60 && /etc/init.d/ovmm start

Finally you should be able to log into ovm console as you used to. Same username and password you always used.

This process updates the backend XE database password only. Thankfully this should all be old news as the new version ships with MySQL and password resets in MySQL are much simpler. The big question and one which I have not yet figured out the answer to is: Will the new OVMM support an external MySQL database... In Production...

Edit...
I had to edit the error message found in the AdminServer log on this blog because of the xml being interpreted instead of displayed... Fixed now thanks to: http://entities.online-toolz.com/tools/text-html-entities-convertor.php

Extending the AD Schema on Samba4 - Part 2

2012-12-21T16:39:00.001+13:00

Importing LDIF files into Samba4 and Active Directory

This is part 2 of the Extending AD Schema on Samba4 series. The examples below are tested using the Samba4 LAB I created. If you want more information on how that works then please read http://david-latham.blogspot.co.nz/2012/12/samba4-ga-release-virtualbox-lab.html

For part one, please read http://david-latham.blogspot.co.nz/2012/12/extending-ad-schema-on-samba4.html

Unfortunately the format of an ldif file for creating new attributes and classes in the Schema Configuration are differ between Samba4 and Microsoft.

The tools are slightly different too. So this article will attempt to make it all clear.

Find all the latest versions of code on this post at https://github.com/linuxplayground/yubikey-ldap/tree/master/microsoft-schema

Samba4 - ldbadd & ldbmodify

As far as I can tell the only way to create a new class with a custom attribute in Samba4 (on the Linux command line) is first add the attribute with ldbadd and then add the class with a seperate ldbadd followed by an ldbmodify command to update the User schema to include the new auxiliary class.

I wrote a README.md file for when this eventually ends up on GIT. Rather than retype it all, I shall just copy it as is:

YubiKey LDIF Implimentation Into Samba4 Active Directory
=======================================================

This is an implimentation of the Openldap implimentation by Michal Ludvig <http://logix.cz/michal/devel/yubikey-ldap/> applied to Samba4 Active Directory.

CAUTION
-------
This process will permanently modify your schema. If it breaks you will not be able to recover unless from a backup. Please backup your schema files before starting. On a default install they can be found at /usr/local/samba/private/sam.ldb and all the files in /usr/local/samba/private/sam.ldb.d/

yubikeyid.ldif
--------------
    dn: CN=yubiKeyId,CN=Schema,CN=Configuration,dc=samba4,dc=internal
    changetype: add
    objectClass: top
    objectClass: attributeSchema
    attributeID: 1.3.6.1.4.1.40789.2012.11.1.2.1.1
    cn: yubiKeyId
    name: yubiKeyId
    lDAPDisplayName: yubiKeyId
    description: Yubico YubiKey ID
    attributeSyntax: 2.5.5.5
    oMSyntax: 22
    isSingleValued: FALSE

Add the yubiKeyId attribute into the Schema Configuration first with:
    ldbadd -H /usr/local/samba/private/sam.ldb \
      yubikeyid.lidf \
      --option="dsdb:schema update allowed"=true

yubikeyuser.ldif
----------------
    dn: CN=yubiKeyUser,CN=Schema,CN=Configuration,dc=samba4,dc=internal
    changetype: add
    objectClass: top
    objectClass: classSchema
    governsID: 1.3.6.1.4.1.40789.2012.11.1.2.2.1
    cn: yubiKeyUser
    name: yubiKeyUser
    lDAPDisplayName: yubiKeyUser
    description: Yubico YubiKey User
    subClassOf: top
    objectClassCategory: 3
    mayContain: yubiKeyId

Next add the yubiKeyUser class into the Schema Configuration with:
    ldbadd -H /usr/local/samba/private/sam.ldb \
      yubikeyuser.lidf \
      --option="dsdb:schema update allowed"=true

updateUserClass.ldif
--------------------
    dn: CN=User,CN=Schema,CN=Configuration,DC=samba4,DC=internal
    changetype: modify
    add: auxiliaryClass
    auxiliaryClass: yubiKeyUser

Apply the User class update with:
    ldbmodify -H /usr/local/samba/private/sam.ldb \
      updateUserClass.ldif \
      --option="dsdb:schema update allowed"=true

Add YubiKeys to Users
---------------------
An example ldif:
    dn: CN=David Latham,CN=Users,DC=samba4,DC=internal
    changetype: modify
    add: objectClass
    objectClass: yubiKeyUser
    -
    add: yubiKeyId
    yubiKeyId: abcdefgh1234
    yubiKeyId: xyzxyz123456

Apply it with:
    ldapmodify -h samba -f addKeyToUser.ldif

Test it with:
    ldapsearch -h samba -b "CN=David Latham,CN=Users,DC=samba4,DC=internal" yubiKeyId

    SASL/GSSAPI authentication started
    SASL username: administrator@SAMBA4.INTERNAL
    SASL SSF: 56
    SASL data security layer installed.
    # extended LDIF
    #
    # LDAPv3
    # base with scope subtree
    # filter: (objectclass=*)
    # requesting: yubiKeyId
    #

    # David Latham, Users, samba4.internal
    dn: CN=David Latham,CN=Users,DC=samba4,DC=internal
    yubiKeyId: abcdefgh1234
    yubiKeyId: xyzxyz123456

    # search result
    search: 5
    result: 0 Success

    # numResponses: 2
    # numEntries: 1

Acknowledgments
===============
Michal Ludvig for defining the schema.
Microsoft Documentation for information on attributeSyntax, oMSyntax and objecClassCategory

Active Directory & the LDIFDE tool

Use the following ldif file and the ldifde tool as described in the README.md below:
YubiKey Implimentation in Microsft Active Directory
===================================================
This is an implimentation of the Openldap implimentation by Michal Ludvig <http://logix.cz/michal/devel/yubikey-ldap/> applied to Microsft Active Directory.

Notes
-----
In order to manage the Schema from a Windows client, please add the following line to your smb.conf under the [globals] section and restart samba4:

    dsdb:schema update allowed = true

You can use tools like ADSI Edit to manage the keys for users.
There are also tutorials on the internet explaining how to crate a dialogue box / context menu tool for updating custom attributes in the Active Directory Server Admin tool. (dsa.msc)

For a complete tutorial on all of this look at:
    <http://www.informit.com/articles/article.aspx?p=169630&seqNum=1>

Implimentation
--------------

Log into a Windows Server 2003 as a domain administrator and start a
command prompt.

Then execute:
    ldifde -i -f path\to\yubikey-ads.ldif -j .

You should see something like:
    6 entries modified successfully

    The command has completed successfully

To test if this is all working you could add some kuys using the ADSI Edit
snap-in.
* Browse to your Domain -> CN=Users
* Right mouse click the username you want to edit
* Select Properties
* Scroll down to and select YubiKeyId
* Click Edit
* Add values until you are done
* Click OK until you are finished.

The LDIF File

#
# YubiKey LDAP schema for Microsoft Active Directory Server
#
# Install with ldifde -i -f path\to\yubikey-ads.ldif -j .
# on a Windows Command prompt
#
#
# Author: Michal Ludvig
# Consider a small PayPal donation:
# http://logix.cz/michal/devel/yubikey-ldap/
#
# Converted to Microsoft Active Directory Server format by
# David Latham
#
dn: CN=yubiKeyId,CN=Schema,CN=Configuration,DC=samba4,DC=internal
changetype: add
objectClass: top
objectClass: attributeSchema
cn: yubiKeyId
description: Yubico YubiKey ID
attributeID: 1.3.6.1.4.1.40789.2012.11.1.2.1.1
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
oMSyntax: 22
lDAPDisplayName: yubiKeyId
name: yubiKeyId

dn:
changetype: modify
add: schemaUpdateNow
schemaUpdateNow: 1
-

dn: CN=yubiKeyUser,CN=Schema,CN=Configuration,DC=samba4,DC=internal
changetype: add
objectClass: top
objectClass: classSchema
cn: yubiKeyUser
description: Yubico YubiKey User
subClassOf: top
governsID: 1.3.6.1.4.1.40789.2012.11.1.2.2.1
mayContain: yubiKeyId
rDNAttID: cn
objectClassCategory: 3
lDAPDisplayName: yubiKeyUser
name: yubiKeyUser

dn:
changetype: modify
add: schemaUpdateNow
schemaUpdateNow: 1
-

dn: CN=User,CN=Schema,CN=Configuration,DC=samba4,DC=internal
changetype: modify
add: auxiliaryClass
auxiliaryClass: yubiKeyUser
-

dn:
changetype: modify
add: schemaUpdateNow
schemaUpdateNow: 1
-

Extending the AD Schema on Samba4 - Part 1

2012-12-21T16:20:00.000+13:00

My last post on Samba4 showed how easy it is to install and configure an AD Service on Linux. If you've not read it then please have a look. (http://david-latham.blogspot.co.nz/2012/12/samba4-ga-release-virtualbox-lab.html)

This post show's how to extend the Samba4 Active Directory Schema. Specifically for YubiKey integration.

YubiKey's can be purchased for a relativlely low price from Yubico. Please visit their website (www.yubico.com) for more information.

LDAP Integration is very well covered by Michal Ludvig on his website and github. (http://www.logix.cz/michal/devel/yubikey-ldap/) In fact we are planning to leverage his implementation at our work and are considering donating towards what's obviously a very good cause.

Now seeing as though LDAP and AD are so similar and exhibit many of the same APIs, I began to wonder how this might fit in with Samba4. Eventually we might end up using Samba4 for our domain and so I needed to figure out if I could, somehow, "port" the LDAP Schemas for Yubikey to AD. I think I have managed it with extensive help from another blog. Thanks to Kurt Hudson and his wonderful article on www.informit.com titled, "Making AD Work Harder | Extending the Active Directory Schema to Track Custom Info"

I should also acknowledge, once again, the Samba Wiki for a crucial piece of the puzzle in which we allow extensions to the Schema. (https://wiki.samba.org/index.php/Samba4/Schema_extenstions)

Step 1: Backup Your Master Server

In my case this was fairly simple. I just took a snapshot of my Virtualbox image. You need to do this though. Messing with an AD schema can be fairly dangerous, especially if it's a Samba4 AD. Samba's Wiki makes this quite clear.

Step 2: Enable Schema Extensions

This is fairly straight forward and well explained on the Samba Wiki. First edit your smb.conf and add the following line into the [globals] section.

dsdb:schema update allowed = true

Then restart the samba service.. (btw: If you are interested in an init script for Samba4 on Centos 6 then find it at the end of this article.)

Step 3: Set up Active Directory Schema Console (on Windows)

From here on, we will be working in Windows space. The Active Directory Schema console is not registered by default. So register it and create yourself a console.

In a windows command prompt type:

regsvr32 schmmgmt.dll

Then use MMC -> Add Remove Snap-ins -> Active Directory Schema snap-in. (follow the instructions on Kurt Hudson's page.

Step 4: Add Yubikey Class and Attribute

You need an attribute before you can create an object class.
The details for the Yubikey schema are available on Michal Ludvig's github site: https://raw.github.com/mludvig/yubikey-ldap/master/ldap-schema/yubikey-SunDS.ldif
The SunDS describes most closely the AD Version used in this article:
Here it is:

dn: cn=schema
objectclass: top
#
# YubiKey LDAP schema for Sun Directory Server and OpenDJ
#
# Copy this file to /config/schema/05-yubikey.ldif
#
# Author: Michal Ludvig 
# Consider a small PayPal donation:
#         http://logix.cz/michal/devel/yubikey-ldap/
#
# Converted for OpenDJ and Sun Directory Server by
#         Ludovic Poitou 
#
attributeTypes: ( 1.3.6.1.4.1.40789.2012.11.1.2.1.1
  NAME 'yubiKeyId'
  DESC 'Yubico YubiKey ID'
  EQUALITY caseIgnoreIA5Match
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.26{128} )
objectClasses: ( 1.3.6.1.4.1.40789.2012.11.1.2.2.1
  NAME 'yubiKeyUser'
  DESC 'Yubico YubiKey User'
  SUP top
  AUXILIARY
  MAY ( yubiKeyId ) )

Step 4a: Attribute

In the Active Directory Schema Snap-in Console do the following:

Right click the Attributes Node and select Create Attribute on the context menu. A message about the permanence of your operation will show up. Well, jump right in. You backed up in step 1.

Fill in the details as follows:

Common Name = yubiKeyId
LDAP Display Name = yubiKeyId
Unique X500 Object ID: = 1.3.6.1.4.1.40789.2012.11.1.2.1.1
Description = Yubico YubiKey User
Syntax = Case Insensitive String
Minimum = 0
Maximum = 128
Multi Valued = Ticked

Click OK to add the attribute to the Attributes list. If you expand the list, and scroll to the bottom, you will find yubiKeyId and you can right click, select properties and see something like this:

Note, I ticked "multi valued" but in the part where I create a dialogue box to enter the key id's I do not allow for multiple values. That's on my TO DO List.

Step 4b: Add Yubikey Schema Object Class

Begin by right mouse clicking Classes and select Create Class in the context menu.
Fill in the details as follows:

Common Name: yubiKeyUser
LDAP Display Name: yubiKeyUser
Unique X500 Object ID: 1.3.6.1.4.1.40789.2012.11.1.2.2.1
Description: Yubico YubiKey User
Parent Class: top
Class Type: Select Auxilary

Click Next

Click Add next to Opional:
Scroll all the way to the bottom and select yubiKeyId
Click OK
Click Finish

Expand the Classes node, scroll to the bottom and right click yubiKeyUser, select Properties to see the following:

So that's the GUI method.

My next post will show how to import ldif files to make this process easier.

Read part 2 here: http://david-latham.blogspot.co.nz/2012/12/extending-ad-schema-on-samba4-part-2.html

Samba4 GA Release - Virtualbox LAB

2012-12-17T17:42:00.001+13:00

Introduction

Configuring Samba has always been a pain in the you-know-what. There is always some kind of permission here or there missing or misconfiguration resulting in endless hours of log trawling and frustrated users.
My profile photo on this blog was taken during just such a time...

So I decided to build a LAB for Samba 4. My LAB is for a fresh install. I have not yet tried an upgrade. Maybe that will come in a later post.

The Samba4 how-to on their WIKI is very good and formed the basis of most of the work. (http://wiki.samba.org/index.php/Samba4/HOWTO)

Lab Environment

LAB Built on Virtual Box using Ubuntu 12.04 LTS
Samba Server

CentOS 6.3 x86_64 - Text based minimal install
512mb RAM
4GB OS (default partition layout from installer)
8GB Secondary disk
1xNIC on Host Only Adapter (Use dnsmasq on your host machine as per: http://david-latham.blogspot.co.nz/2012/05/create-nat-for-virtualbox-host-only.html
1xNIC on Internal Only Adapter (You will be serving the domain on a subnet assigned to this nic)

Client

Windows2003 R2 Standard Edition

Samba OS Build

Hard Disk Configuration

The Samba HOW-TO wiki states that some specific mount attributes are applied to your disks. Here is my fstab.
[root@samba ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Thu Dec 13 13:55:41 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root / ext4 user_xattr,acl,barrier=1 1 1
ther "system" lines not shown here>
# SAMBA
/dev/mapper/vgsamba-lvwinhome /samba/home ext4 user_xattr,acl,barrier=1 1 1

Network Interface Configuration

/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=samba.example.com

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="dhcp"
DHCP_HOSTNAME=samba.vbox.local

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE="eth1"
NM_CONTROLLED="no"
ONBOOT="yes"
BOOTPROTO="none"
IPADDR="10.1.1.1"
NETMASK="255.255.255.0"

Iptables

For Iptables, I went with an open configuration. You should have a good and proper IPTABLES configuration in your production environment. This LAB is about SAMBA4 not IPTABLES.

/etc/sysconfig/iptables

The IPTABLES script below impliments a SNAT on all traffic to the IP Address of eth0. Your IP address might be different. This all depends on how you configured your host only network in Virtualbox.

# Generated by iptables-save v1.4.7 on Thu Dec 13 14:26:37 2012
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -o eth0 -j SNAT --to-source 192.168.56.76
COMMIT
# Completed on Thu Dec 13 14:26:37 2012
# Generated by iptables-save v1.4.7 on Thu Dec 13 14:26:37 2012
*filter
:INPUT ACCEPT [12:1634]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Thu Dec 13 14:26:37 2012

Enable IPv4 Packet Forwarding

Once you do this, routing will take care of itself.

Set net.ipv4.ip_forward = 1 in /etc/sysctl.conf
execute

sysctl -p to read the new value.

Software pre-requisites and services

Software packages

First establish the following dependancies. (Note I used the groupinstall "Development tools" for convenience.

Read: http://wiki.samba.org/index.php/Samba_4/OS_Requirements

yum -y install libacl-devel libblkid-devel gnutls-devel readline-devel python-devel gdb pkgconfig krb5-workstation zlib-devel setroubleshoot-server setroubleshoot-plugins policycoreutils-python libsemanage-python setools-libs-python setools-libs popt-devel libpcap-devel sqlite-devel libidn-devel libxml2-devel libacl-devel libsepol-devel libattr-devel keyutils-libs-devel cyrus-sasl-devel openssh-clients bind-utils dhcpd tcpdump man manpages wget
yum -y groupinstall 'Development tools'

NTP

Download and install the latest NTP.

DO NOT USE THE Distro Version as it does not support "signed NTP"
Use the "more complex ntp.conf" (2c)
Instructions here: http://wiki.samba.org/index.php/Samba4/HOWTO#Step_10:_Configure_NTP_.28Optional.29
Note: I used ./configure --prefix=/ to ensure that NTP would be installed in my path. This version did not ship with any useful init script so I just added the following to my rc.local:

# Start NTP Daemon
/sbin/ntpd -c /etc/ntp.conf

DHCP Server

Configure dhcpd to provide ip addresses on the internal network.

dhcpd should also provide the netbios and wins ip addresses.

subnet 10.1.1.0 netmask 255.255.255.0 {
        option routers          10.1.1.1;
        option subnet-mask      255.255.255.0;
        option domain-name      "internal.local";
        option domain-name-servers 10.1.1.1;
        option netbios-name-servers 10.1.1.1;
        range 10.1.1.10 10.1.1.100;
}

Edit /etc/sysconfig/dhcpd and set DHCPDARGS=eth1
service start dhcpd
chkconfig dhcpd on

DNS Server

I did not configure bind or dnsmasq for the Domain. Samba4 Ships with an internal DNS server which is adequate for the purposes of demonstrating Samba4 awesomness. The excellent BIND Nameserver is supported by Samba4 along with dns replication and all that jazz. I have not tried it. If you have, let us know in the comments below how it went.

Once Samba is installed and operational there is a step required to make Samba forward DNS queries to your host's network as well as make the OS resolv.conf use the correct DNS. This is because dhclient likes to automagially configure your resolv.conf every time you boot. My work-around is just to overwrite it in /etc/rc.local.

# fix resolv.conf
mv /etc/resolv.conf /etc/resolv.conf.old
cat >> /etc/resolv.conf << EOF
# Updated by rc.local script
domain samba4.internal
nameserver 10.1.1.1
EOF

Install Samba4

Installing Samba4 is fairly straightforward once you have all the pre-requisites in place. It's well documented on the samba4 wiki so I won't get into it in any great detail here.

I highly recommend completing the file system and kerberos tests outlined in the wiki how-to. (http://wiki.samba.org/index.php/Samba4/HOWTO#Step_6:_Testing_Samba4)

smb.conf

This is my lab smb.conf.

workgroups, realm, netbios and server role all define the function of this samba instance.
dns forwarder tells samba where to send dns requests for hosts it knows nothing about.
The netlogon share is for the logon script. I have a log on script located in it's path and I set the default logon script in windows for each user to point to this script.
sysvol is (unused at this stage)
home is where all the user's WINDOWS home directories live

There are no unix home directories that map to the windows home directories

share is where all standard shares will live.

[root@samba ~]# cd /usr/local/samba/etc
[root@samba etc]# cat smb.conf
# Global parameters
[global]
        workgroup = SAMBA4
        realm = SAMBA4.INTERNAL
        netbios name = SAMBA
        server role = active directory domain controller
        dns forwarder = 192.168.56.1

[netlogon]
        path = /usr/local/samba/var/locks/sysvol/samba4.internal/scripts
        read only = No

[sysvol]
        path = /usr/local/samba/var/locks/sysvol
        read only = No
[home]
        comment = Home Directories
        path = /samba/home
        read only = No
[share]
        comment = Office Shares
        path = /samba/home/shares
        read only = No
[root@samba etc]#

Configuring File Permissions and Shares

Once these shares were configured in this basic way using smb.conf I was able to browse to the folders in my windows client using as follows:

\\samba\home
\\samba\share
\\samba\netlogon\logon.bat
\\samba\sysvol

This was easily the most difficult part of the process for me to work out. Once worked out though, it's a piece of cake. Critically your file system must be mounted with the acl mount option.

I have figured out that the most straight forward way of managing file shares is via Windows. ie: Let samba take care of the acl entries in Linux.

User accounts and home directories

Home directory ACLs

Before creating user accounts its critical to make sure that the windows ACLs are configured correctly for \\samba\home. Mine are as follows:

First remove inheritance. IE: The permissions for \\samba\home must not inherit from \\samba. When removing inheritance hit the remove button and remove any other permissions groups / users listed.

Then add new ones as follows:

Administrators = Full Control
Everyone = No Control (all boxes unticked)
Domain Users = Special Permissions as follows

Traverse Folder and Execute File = TICKED
All other special permissions = UNTICKED

Creating new users

Run the dsa.msc snap in. Installation instructions for the dsa.msc are covered in the samba4 wiki howto.

Create a new user using the "New -> User" task in the DSA Snapin. The following screenshots show what to do:

Add user Test User with username test.user

Set a password

Confirm details


Bring up the Profile tab under properties for this new user.

Here, on the profile tab you should set the logon script for your user and map the home drive on the connect button. When you apply this, the test.user folder will be created on the share for you. The permissions will also be configured appropriately as per the next screenshot.

Confirm folder security permissions are correct.

Use a similar approach to managing shared folders for group access.

So far I have not encountered any gotchas with this approach.

Oracle Cloud Control

2012-10-13T15:55:00.000+13:00

I had the occasion recently to deploy an OVM stack at a customer site. Initially it was my intention to only deploy Enterprise Manager 12c and take advantage of the cloud control features within. As it turned out, this provides merely a "remote control" function of an existing OVM Manager. With this in mind, I could not find many advantages to using EM12c for Oracle virtual machine management.

The next hurdle was with the hardware. Of course, the physical servers we deployed the OVS 3.1.1. hyper-visor on were not "Oracle Certified." Specifically DELL PowerEdge M620 Blade Servers. (http://www.dell.com/us/enterprise/p/poweredge-m620/pd?~ck=anav)

The customer had selected the Broadcom® 57810S-k Dual Port 10Gb KR blade NDC on board Network Adapter as well as the Mezzanine Broadcom 5719 Serdes Quad Port 1Gb providing a total of 6 ethernet ports, two of them 10GB.

The storage array is Equalogic iSCSI SAN. (Sorry the exact make / model of the SAN escapes me right now.)

The OVM 3.1.1. Server installation went perfectly well. We selected the 3rd port for management (1st port on the 1g mezzanine card) knowing that ports 1 and 2 were to be used for iSCSI and possibly Live Migration.

The blades were able to be discovered without difficulty into OVM manager. Apon inspection, however, eth0 was showing as DOWN in all three blade servers. The iDRAC console also showed that the link was UP but the OS State was reported as DOWN in each case. I was able to successfully configure eth1 and create a single port bond with the appropriate IP address for the SAN and discover the storage. Just without redundancy at this stage. Of course this had to be resolved before production.

I did some research and discovered that the OEM 3.1.1. hyper-visor is based on RHEL / OL 5.7 and that the broadcom driver (bnx2x) needed an option to force all ports back to legacy state.

From: ftp://ftp.supermicro.com/CDR-X9_1.11_for_Intel_X9_platform/Broadcom/Server/Linux/Driver/README.bnx2x.TXT

Driver Parameters
=================

Several optional parameters can be supplied as a command line argument
to the insmod or modprobe command. These parameters can also be set in
modprobe.conf. See the man page for more information.

The optional parameter "int_mode" is used to force using an interrupt mode
other than MSI-X. By default, the driver will try to enable MSI-X if it is
supported by the kernel. In case MSI-X is not attainable, the driver will try
to enable MSI if it is supported by the kernel. In case MSI is not attainable,
the driver will use legacy INTx mode. In some old kernels, it's impossible to
use MSI if device has used MSI-X before and impossible to use MSI-X if device
has used MSI before, in these cases system reboot in between is required.

Set the "int_mode" parameter to 1 as shown below to force using the legacy
INTx mode on all NetXtreme II NICs in the system.

   insmod bnx2x.ko int_mode=1

or

   modprobe bnx2x int_mode=1

The way I achieved this was to add a file into /etc/modprobe.d as follows:

/etc/modprobe.d/bxn2x.conf
options bxn2x int_mode=1

Reboot the blades and viola, eth0 shows status UP, ethtool eth0 indicates 10gb speed on the port and the iDRAC console shows all good on the port.
I then added eth0 to the bond I created earlier and left it in Active / Backup configuration.

I was unable to make multipathing work through the GUI. I suspect that OVM manager only discovers a single path to storage and does not attempt anything further. I am confident that the Active Backup bond configuration will provide sufficient storage redundancy for the storage pool, server pool repository and PVM Guest LUNS.

I hope this helps anyone who is also struggling with this issue.

Oracle Virtual Machine (OVM) LAB on VirtualBox

2012-05-14T13:41:00.002+12:00

Oracle Virtual Machine (OVM) 3.1.1. was released on the 8th May 2012 and finally it's supported on VirtualBox. This is great news for anyone who wants to give installing an OVM Lab a go.

I gave it a go.

I created 3 VirtualBox guests:

Openfiler for iSCSI with a 40GB virtual disk for chopping up into LUNs and iSCSI.
Oracle Linux 6.0 on which I installed OVM 3.1.1. in Demo mode. This guest has 4GB of RAM assigned to it and a 25GB HD. Probably a bit big, but OVM is a large application with an Oracle XE database and Oracle Weblogic services installed, I thought better safe than sorry.
Oracle Virtual Server 3.1.1. This guest had just a 4GB HD and 1536MB of RAM which I figured would be just enough to get 1 virtual machine up and running on it.

My VirtualBox is configured with a couple of host only networks:

vboxnet0 => 192.168.56.0/24 => Management network. I also have dnsmasq configured on this network to serve IP addresses via DHCP to clients on this network. Dnsmasq is installed on my laptop for this specific purpose. See previous blog post...
vboxnet1 => 192.168.57.0/24 => Used for the storage network.
vboxnet2 => 192.168.58.0/24 => Production network for virtual machines.

So I set up openfiler to offer 2x8GB LUNS, 1x2GB LUN for the cluster storage network on my server pool and 10GB NFS Share for the repository.

The installation of OVS and OVM on Oracle Linux all went by without a hitch. I made sure that each VirtualBox guest set their own hostnames against dnsmasq using the DHCP_HOSTNAME parameter on /etc/sysconfig/network. This meant that all guests resolve nicely on DNS which is a prerequisite for a successful OVM LAB.

The new OVM3.1.1. GUI is actually quite nice. It's more polished and intuitive than 3.0.3. and more of the Right Mouse Button menus are enabled.

I also made sure that SELINUX and IPTABLES was disabled in both OVS and the OVM guests.

Server discovery went fine. No problems at all. Just enter the agent password and hostname of the OVS server.

Storage discovery was a little tricky. The NFS Share (Filesystem share) was no problem. OVM was able to find and mount it wouthout any difficulty. The only trouble I had there was making sure that Openfiler was configured correctly. But that's not relevant to the OVM LAB really.

The iSCSI discovery had me stumped for quite some time. I could discover the iSCSI target presented by Openfiler without issue but could not get OVM to show me any LUNS. (Physical Disks) After much trial and error I finally figured out that my OVS server did not have the multipathd service running. Once I enabled and started multipathd (no change to default configuration), I was able to refresh the iSCSI storage in OVM and the physical disks were presented. To recap: First perform iSCSI storage discovery using the OVS guest as the Storage Manager. Then start multipathd on the OVS guest, then refresh the iSCSI storage on OVM and your LUNs will appear.

I was able to create a server pool, assign the repository, assign the 2GB iSCSI LUN to the pool for it's cluster configuration data and actually configure a virtual machine. I configured an Ubuntu VM with just 512MB of ram and one iSCSI LUN.

Finally after so much success, my LAB failed me at the most crucial stage. Starting the virtual machine. - It would not start. It turns out that VirtualBox does not present hardware acceleration to it's guests. So my OVS guest was running without any hardware acceleration in the virtual CPU. OVM was not able to start the VM. What a shame but there you go.

The good news is that many of the minor irritations in OVM3.0.3 seem to be cleared up and folks are able to now use VirtualBox for their LABs but they can't start OVM Virtual Machines if their OVS is actually a VirtualBox guest.

Next step... Commandeer an old server somewhere, install OVS, bind it to the right LAN and carry on.

Create a NAT for VirtualBox Host Only Network

2012-05-09T18:07:00.000+12:00

VirtualBox does a fairly good job with the built in NAT feature on the virtual network device. But there are some drawbacks:

You can't easily monitor the network on the built in NAT. So if your VM is misbehaving, you couldn't use tcpdump to troubleshoot.
The built in NAT reaches directly to the Internet in a transparent manner. If you wanted to control access to the Internet you would have to switch to a bridge device or go with host-only.
Most importantly for me, it does not support GRE packets very well. This means that if you want to access a Microsoft PPTP VPN from your VM, it won't work with the VirtualBox built in NAT. (Well it didn't for me anyway)

EDIT: An updated script for the firewall component is described in a new post found here: http://david-latham.blogspot.co.nz/2013/03/firewall-script-for-opensuse-and-others.html

So with some simple tweaks to your host, you can NAT the VirtualBox host only network and enjoy some extra features.

In VirtualBox create a default host-only network. This is done in File -> Preferences -> Network. Click the green icon with a PLUS sign in it. A new network is created called vboxnet0. Then click the little screwdriver icon to edit the vboxnet0 settings. Select the DHCP Server tab and make sure "Enable Server" is unticked. It will be ticked by default.

Make sure that the DHCP server is not enabled. This is important because you will configure DHCP with dnsmasq on the next step.

Then install dnsmasq. You need this for DNS. I like that with dnsmasq, and dhcp client id's specified on the VM's, I can do things like "ssh vm_hostname" from the host pc.

sudo apt-get install dnsmasq

Accept the defaults.

dnsmasq uses a config.d approach for configuration so create a file called: /etc/dnsmasq.d/virtualbox as follows:

# cat /etc/dnsmasq.d/virtualbox
interface=vboxnet0
dhcp-range=192.168.56.2,192.168.56.150

Restart dnsmasq:

sudo /etc/init.d/dnsmasq restart

Now configure IPTABLES. I took a lazy approach and added these commands to /etc/rc.local
I have not bothered to learn how to do this with ubunut's default UFW becasue I don't much like UFW and so don't use it.

sudo ufw disable (if you're interested ...)

Add the following lines to /etc/rc.local (before the last line which is "exit 0")

# Flush IPTABLES
iptables -F
iptables -t nat -F
# Loopack
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# Allow all from virtualbox
iptables -A INPUT -s 192.168.56.0/24 -j ACCEPT

# Accept SSH  If you have openssh installed that is.
iptables -A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

# Accept DNS
#iptables -A INPUT -p tcp --dport 53 -j ACCEPT
#iptables -A INPUT -p udp --dport 53 -j ACCEPT

# Accept DHCP
#iptables -A INPUT -p tcp --dport 67 -j ACCEPT
#iptables -A INPUT -p udp --dport 67 -j ACCEPT

# Masquerade virtualbox network
iptables -t nat -A POSTROUTING -s 192.168.56.0/24 -j MASQUERADE

#turn on ip forwarding.  Can be done in /etc/syctl.conf - As I said - lazy.

sysctl -w net.ipv4.ip_forward=1

Then reboot your computer.

sudo reboot

Now change the nic on your VirtualBox vm to a Host Only network and start the VM. If you did everything right, your VM will start, obtain a lease from dnsmasq and have access to the internet.

The firewall rules above allow everything from VirtualBox to go out on the internet. If you wanted to lock things down a bit, you could add more rules to do so and remove the line that accepts everything from the 192.168.56.0/24 subnet.

Oh and your Microsoft PPTP VPN will work now. Even if you use wireless...

Have fun!

Parsing the Multipath

2012-04-29T20:37:00.000+12:00

Having worked a little with XIV I have come to like the "xiv_devlist" tool. "xiv_devlist" lists all the attached disks on the SAN and the number of paths that are currently active. This is very useful when you want to ensure that all the paths to storage are active.

Find latest up-to-date code for this post on my Github profile: https://github.com/linuxplayground/mpath_devlist

It surprised me to find that there are no useful tools to parse the output from Multipath.

Here is an example:

multipath -ll

mpath2 (3600507680191014a3800000000000100) dm-7 IBM,2145                       
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 0:0:1:8  sdaa 65:160 [active][ready]                                       
 \_ 1:0:1:8  sdbk 67:224 [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:0:8  sdas 66:192 [active][ready]                                       
 \_ 0:0:0:8  sdi  8:128  [active][ready]                                       
mpath0 (3600507680191014a380000000000010a) dm-17 IBM,2145                      
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 1:0:1:0  sdbc 67:96  [active][ready]                                       
 \_ 0:0:1:0  sds  65:32  [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:0:0  sdak 66:64  [active][ready]                                       
 \_ 0:0:0:0  sda  8:0    [active][ready]                                       
mpath18 (3600507680191014a38000000000000ff) dm-24 IBM,2145                     
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]                                            
 \_ 1:0:0:7  sdar 66:176 [active][ready]                                       
 \_ 0:0:0:7  sdh  8:112  [active][ready]                                       
\_ round-robin 0 [prio=20][enabled]                                            
 \_ 1:0:1:7  sdbj 67:208 [active][ready]                                       
 \_ 0:0:1:7  sdz  65:144 [active][ready]                                       
mpath17 (3600507680191014a38000000000000fe) dm-23 IBM,2145                     
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]                       
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:6  sdaq 66:160 [active][ready]
 \_ 0:0:0:6  sdg  8:96   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:6  sdbi 67:192 [active][ready]
 \_ 0:0:1:6  sdy  65:128 [active][ready]
mpath16 (3600507680191014a3800000000000109) dm-22 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:5  sdap 66:144 [active][ready]
 \_ 0:0:0:5  sdf  8:80   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:5  sdbh 67:176 [active][ready]
 \_ 0:0:1:5  sdx  65:112 [active][ready]
mpath9 (3600507680191014a380000000000010c) dm-14 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:15 sdah 66:16  [active][ready]
 \_ 1:0:1:15 sdbr 68:80  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:15 sdaz 67:48  [active][ready]
 \_ 0:0:0:15 sdp  8:240  [active][ready]
mpath15 (3600507680191014a3800000000000108) dm-21 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:1:4  sdbg 67:160 [active][ready]
 \_ 0:0:1:4  sdw  65:96  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:4  sdao 66:128 [active][ready]
 \_ 0:0:0:4  sde  8:64   [active][ready]
mpath8 (3600507680191014a380000000000010b) dm-13 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:14 sday 67:32  [active][ready]
 \_ 0:0:0:14 sdo  8:224  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:14 sdag 66:0   [active][ready]
 \_ 1:0:1:14 sdbq 68:64  [active][ready]
mpath14 (3600507680191014a3800000000000107) dm-20 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:3  sdan 66:112 [active][ready]
 \_ 0:0:0:3  sdd  8:48   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:3  sdbf 67:144 [active][ready]
 \_ 0:0:1:3  sdv  65:80  [active][ready]
mpath7 (3600507680191014a3800000000000105) dm-12 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:13 sdax 67:16  [active][ready]
 \_ 0:0:0:13 sdn  8:208  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:13 sdaf 65:240 [active][ready]
 \_ 1:0:1:13 sdbp 68:48  [active][ready]
mpath13 (3600507680191014a3800000000000106) dm-19 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:1:2  sdbe 67:128 [active][ready]
 \_ 0:0:1:2  sdu  65:64  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:2  sdam 66:96  [active][ready]
 \_ 0:0:0:2  sdc  8:32   [active][ready]
mpath6 (3600507680191014a3800000000000104) dm-11 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:12 sdae 65:224 [active][ready]
 \_ 1:0:1:12 sdbo 68:32  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:12 sdaw 67:0   [active][ready]
 \_ 0:0:0:12 sdm  8:192  [active][ready]
mpath12 (3600507680191014a38000000000000fd) dm-18 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:1  sdal 66:80  [active][ready]
 \_ 0:0:0:1  sdb  8:16   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:1:1  sdbd 67:112 [active][ready]
 \_ 0:0:1:1  sdt  65:48  [active][ready]
mpath5 (3600507680191014a3800000000000103) dm-10 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:11 sdav 66:240 [active][ready]
 \_ 0:0:0:11 sdl  8:176  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:11 sdad 65:208 [active][ready]
 \_ 1:0:1:11 sdbn 68:16  [active][ready]
mpath11 (3600507680191014a380000000000010e) dm-16 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:17 sdaj 66:48  [active][ready]
 \_ 1:0:1:17 sdbt 68:112 [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:17 sdbb 67:80  [active][ready]
 \_ 0:0:0:17 sdr  65:16  [active][ready]
mpath4 (3600507680191014a3800000000000102) dm-9 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 0:0:1:10 sdac 65:192 [active][ready]
 \_ 1:0:1:10 sdbm 68:0   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 1:0:0:10 sdau 66:224 [active][ready]
 \_ 0:0:0:10 sdk  8:160  [active][ready]
mpath10 (3600507680191014a380000000000010d) dm-15 IBM,2145
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:16 sdba 67:64  [active][ready]
 \_ 0:0:0:16 sdq  65:0   [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:16 sdai 66:32  [active][ready]
 \_ 1:0:1:16 sdbs 68:96  [active][ready]
mpath3 (3600507680191014a3800000000000101) dm-8 IBM,2145
[size=40G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:9  sdat 66:208 [active][ready]
 \_ 0:0:0:9  sdj  8:144  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 0:0:1:9  sdab 65:176 [active][ready]
 \_ 1:0:1:9  sdbl 67:240 [active][ready]

So if you wanted to check that there are 4 active / ready paths to each "mpath" disk, you would have to visually scan through the whole output above.

A simple perl script that takes the output from multipath -ll as its input, can make this task much simpler.

dave@davidpc:~/devel/paths$ cat paths.txt | ./mpath_list.pl 
00      PATH    UID                                     DM-ID   ACTIVE  RESULT
2       mpath2  3600507680191014a3800000000000100       dm-7    4 of 4  OK
0       mpath0  3600507680191014a380000000000010a       dm-17   4 of 4  OK
18      mpath18 3600507680191014a38000000000000ff       dm-24   4 of 4  OK
17      mpath17 3600507680191014a38000000000000fe       dm-23   4 of 4  OK
16      mpath16 3600507680191014a3800000000000109       dm-22   4 of 4  OK
9       mpath9  3600507680191014a380000000000010c       dm-14   4 of 4  OK
15      mpath15 3600507680191014a3800000000000108       dm-21   4 of 4  OK
8       mpath8  3600507680191014a380000000000010b       dm-13   4 of 4  OK
14      mpath14 3600507680191014a3800000000000107       dm-20   4 of 4  OK
7       mpath7  3600507680191014a3800000000000105       dm-12   4 of 4  OK
13      mpath13 3600507680191014a3800000000000106       dm-19   4 of 4  OK
6       mpath6  3600507680191014a3800000000000104       dm-11   4 of 4  OK
12      mpath12 3600507680191014a38000000000000fd       dm-18   4 of 4  OK
5       mpath5  3600507680191014a3800000000000103       dm-10   4 of 4  OK
11      mpath11 3600507680191014a380000000000010e       dm-16   4 of 4  OK
4       mpath4  3600507680191014a3800000000000102       dm-9    4 of 4  OK
10      mpath10 3600507680191014a380000000000010d       dm-15   4 of 4  OK
3       mpath3  3600507680191014a3800000000000101       dm-8    4 of 4  OK

And the perl script that creates this output:
EDIT: Revised script (2012-04-29) Added size, Removed flag variable from script.

#!/usr/bin/env perl
use Term::ANSIColor;

$count = 0;
print "00\tPATH\tUID\t\t\t\t\tDM-ID\tSIZE\t\tACTIVE\tRESULT\n";
while(<STDIN>) {
        my ($line) = $_;
        chomp($line);
        if( $line =~ /^mpath(\d+)/ ) {
                my( $id ) = $1;
                /^(\w*) \((\w*)\)\s*(dm-\d*)/;
                if( $count > 0 ) { print $count, " of 4\t";
                        if( $count == 4) {
                                print  color 'bold green'; print "OK\n"; print color 'reset';
                        } else {
                                print color 'bold red'; print "PROBLEM\n"; print color 'reset';
                        }
                        $count = 0;
                }
                my ($pname) = $1;
                my ($puid)  = $2;
                my ($pdmid) = $3;
                print $id,"\t",$pname,"\t",$puid,"\t",$pdmid,"\t";
        }
        if ( $line =~ m/^\[(.*?)\]/ ) {
                print $1,"\t";
        }
        if ( $line =~ /^\s\\_.*active\]\[ready/ ) {
                $count += 1;
        }
}
if( $count > 0 ) { print $count, " of 4\t";
        if( $count == 4) {
                print  color 'bold green'; print "OK\n"; print color 'reset';
        } else {
                print color 'bold red'; print "PROBLEM\n"; print color 'reset';
        }

        $count = 0;
}
print color 'reset';
exit 0;

TO DO:

Call "multipath -ll" from inside the perl script thus remove the requirement to pipe the results of multipath -ll into the script.
Make sure multipath -ll is a valid system call.
Add a summary of the results, IE: 18 paths found, ALL GOOD.
Add a check to ensure user is root at the beginning of the script and exit with message if not.
Sort output results in order of mpath number.

Policing my network

2012-04-15T00:26:00.000+12:00

Policing my network - well not actually policing it, just locking it down.

I have decided it's time to worry about what my kids might find on the Internet - my oldest is approaching 7 and beginning to spend more time on the 'puter.

So here is the plan. Not completely implemented just yet but getting there.

Enable mac address filtering on the router. Only devices I know about are allowed to connect to the wireless router and a password is still required to gain access and an IP address on my LAN. - DONE! It works great and a quick visit to my router admin page shows me who is currently using it. Look this is just good sense. If you are not doing mac address filtering book-mark this blog right now, and go and configure your router!
Have a conversation with the child about the dangers and pitfalls of the Internet. NOT DONE YET! - I am kind of dreading this bit. How do I explain to a child that it's bad to do something in such a way that they don't immediately go and do it. (It's what I would have done if my parents warned me about porn on the Internet and then gave me access to the Internet.) Kind of like God telling Adam and Eve not to eat the forbidden fruit then showing them where it is...
Create an account with openDNS (http://www.opendns.com/) - start off with the free one. - DONE! Testing at the moment on my own PC. Tried visiting a couple of porn sites I remember from my bachelor days and yes - openDNS is blocking them. ---- I Should note, however, that it did nothing to prevent google from showing me all sorts of scary (sic) stuff when running a search on google images. I guess this bit will just have to be covered by GOOD PARENTING! (Discussed in Step 2 - easier said than done...)
Impliment openDNS or similar on border router (ADSL router). Now nothing in the network will have access to said scary stuff except google image searches that is.
Build small Linux desktop for child. -- Here it is. The bit about Linux. I intend to revive an old P100 or something slow like that I have lying around with a stable 12.04 (LTS) version of Ubuntu but with LXDE desktop to keep things sane. Then I will configure PAM to restrict login times for the boy's user account, and then show some pop up warning (called by cron) so the boy has a chance to log out gracefully. Because we don't want him on the thing all the time. I have heard that kids will be up all night (just like me - it's now 00:25 in the morning) using their 'puters.

So here are some links that might be useful.

www.opendns.com
http://www.informit.com/articles/article.aspx?p=165226

So as I progress through all this, I will post here about how it all goes.

Of course, if I was LOADED,$$$, I would buy the boy a MAC. Apparently they have this sort of functionality built in.

OGG to MP3 Conversion with ID3 tags

2012-02-18T16:03:00.001+13:00

I spend ages every time I need to do this to remember the correct format of the command:

here it is at last:

/usr/bin/find -type f -name "*.ogg" | while read file; do echo "working on $file"; /usr/bin/gst-launch filesrc location="${file}" ! oggdemux ! vorbisdec ! audioconvert ! lame quality=4 ! id3v2mux ! filesink location="`/usr/bin/dirname "$file"`/`/usr/bin/basename "$file" .ogg`.mp3" 2>&1 && /bin/rm "$file"; done | /usr/bin/tee -a $HOME/ogg2mp3results.txt

Find all the files
Convert them into mp3 with quality = 4 which is something like 44000khz and br=127kbit
Save them with an mp3 extension next to the original ogg
Remove the ogg file
Tee the output into a log.

HA!

Configuring MySQL on Redhat Linux while utilising network (iSCSI) storage.

2011-08-31T22:12:00.006+12:00

Challenges:
MySQL only allows a single location to be specified as the default data store for all databases.

Consider the following typical scenario:

A database server running one instance of MySQL has its datastore located at /mysql/databases.
This server will contain multiple databases for multiple applications.
The database tables are a mixture of InnoDB and MYISAM storage engines.
The storage is provided by a SAN and separate storage volumes (iSCSI targets) are provisioned for each database.

In this scenario, you would want all your MYISAM files (*.MYI, *.MYD, *.frm) and all your InnoDB table spaces to be contained on the same iSCSI volume on a per-database basis.

Solution:

This is easily achieved by mounting the iSCSI target on a sub-folder of /mysql/databases. For example:

Mount iqn.openfiler.com:2006-06:blog-db-storage on /mysql/databases/blog

Here is a sequence of steps that need to be carried out to make this work.

in /etc/my.cnf set datastore=/mysql/databases
in /etc/my.cnf set innodb_file_per_table (http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html)
Configure SELINUX and user/group rights appropriately for MySQL. There is heaps on this on the internet but please comment if you would like me to add some detail here.
Start the mysqld service
configure the mysql root password if necessary
login to mysql as root
create the blog database (create database blog;) This creates the blog directory in /mysql/databases/blog At this point there is nothing inside it.
log out of mysql.
Mount your iscsi storage volume on /mysql/databases/blog
Configure your system so that the iscsi storage volume has a persistence and is always mounted on this location. Redhat have a great tutorial on how to do this. Basically it involves checking the WID of your iscsi target and then applying that to a udev rule that creates a consistently named block device under /dev
Now go ahead and create your tables. All the MYISAM and InnoDB tables will be created inside the /mysql/databases/blog directory.

Code / Scripts:

I am happy to provide scripts or code if people want. The following links should get you started.

Links:

MUST FOLLOW PROCESS...

2011-08-29T21:18:00.007+12:00

MUST FOLLOW PROCESS.... MUST FOLLOW PROCESS...

Starting with the premise that we need our customers to be happy with us, what do we need to do as an organisation to keep them that way?

We are working through steps to improve our customer satisfaction at work by carefully considering how our daily work routines, processes and actions impact our customers.

Consider this fairly typical example of how an organisation might respond to a customer request:

[NOTE: This is a little bit (A LOT) satirical and not meant to in anyway be a real representation of an actual process. It's designed to highlight how strict adherence to process can lead to unhappy customers.]

A customer needs a new type of widget. Your organisation sells gadgets that are very similar to widgets so they approach you to design and build one for them. They are willing to pay for the service but they need this new widget to work for them.

Luckily your organisation has a process to handle this type of request. It's called Customer Request Acceptance Process. It is a sequence of actions scraped out of a managerial text book written in the early 1900's.

So the sales rep in your organisation finds the appropriate template and begins...

The sales rep spends 2 hours working out how best to express his customer's requirements on the template. The template is designed to be all encompassing providing options for everything. Clearly (from looking at the template) anything not covered by it's fields are not valid for submission. [Templates have their place but be careful of over detailing them - they get confusing. A good place to start is a subject and details box. Leave the rest to the technical guys to nut out.]
The working group that handle inbound requests receive the completed form and instantly find gaps in information. They make assumptions about some of the more obvious omissions and request clarification from the sales rep on the others. [Leave the mind reading to the professionals]
The sales rep is very confident that he understands 90% of the questions and answers them straight away and on the other 10% he seeks clarification. [More mind reading...]
Finally the requirements are gathered (from the sales rep and via a series of mini assumptions about unimportant details)
A quote is hastily written up and the sales rep informed. The sales rep sells the solution to the customer. [the solution has to be sold??? I thought the customer asked for it in the first place. Be careful! If you have to **sell** the solution to a customer request then it's probably not what they wanted.]
The tech team receive the go-ahead and begin work on the widget. The widget is FANTASTIC. It's so damn clever, some individuals are already talking about it to their contacts in an effort to drum up even more revenue. [Woooah there Silver, Does it work for the customer? Has anyone asked them?]
The widget is delivered more or less on schedule. [Good work guys - go eat pizza and have a celebration.]
The customer finds that it sort-of does what it's supposed to do but they don't complain because they have come to expect this sort of thing from the organisation.[What??? This is acceptable to you Mr. Customer? Find another provider already!]
Your organisation sends the customer a survey asking how happy they are and soon it becomes apparent that they are not. [By sending a survey out you send a message saying that you need to know what's wrong before you can fix it. If you don't fix it your customer will assume your are being cynical about the whole survey thing and not bother answering it.]

My point with this extremely obvious example is that:

a happy customer is very simply a customer that you talk to all the time!

So while your organisation might have a process and tools to manage that process, it's useless if you don't know what your customer really wants.

Some customers are not too good at telling us what they really want.
Most of the time, I suggest that we are not placing the appropriate individuals in our organisations together. IE: Customer end user of WIDGET next to the provider technical person designing the WIDGET. It makes sense that these two should work closely together.

Providers should develop fast prototypes and be less precious about the their solutions in case they end up not being fit for purpose. Every time a mistake is corrected, your product is improved.

It is impossible to gather all the requirements in one go. Requirements have a way of evolving as the design takes shape.

No amount of technology based tools to manage your processes will help you if your processes do not include the customer at every step.

Every step in your process that does not include the customer results in your product diverging further from the customer's requirements.

We should all evaluate all the time how what we do impacts the customer. If you don't know how your job impacts your customers, you should make it your number one concern to find out. Customers pay your salary. Not your boss or your manager.

REBLOG!!! Sticky bit, suid and guid

2011-07-21T22:59:00.002+12:00

I have to reblog this article in it's full just in case the original author removes it.

Here is the original.
http://bashscript.blogspot.com/2010/03/unixlinux-advanced-file-permissions.html

Here in plain text to ** FORCE ** it into my rubbish layout.

UNIX/Linux Advanced File Permissions - SUID,SGID and Sticky Bit

After you have worked for a while with Linux you discover probably
that there is much more to file permissions than just the "rwx" bits.
When you look around in your file system you will see "s" and "t"

$ ls -ld /tmp
drwxrwxrwt 29 root root 36864 Mar 21 19:49 /tmp

$ which passwd
/usr/bin/passwd

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 22984 Jan 6 2007 /usr/bin/passwd

What is this "s" and "t" bit? The vector of permission bits is really
4 * 3 bits long. Yes there are 12 permission bits,not just 9.The first
three bits are special and are frequently zero. And you almost always
learn about the trailing 9 bits first.Some people stop there and never
learn those first three bits.

The forth permission bit is used only when a special mode of a file
needs to be set. It has the value 4 for SUID, 2 for SGID and 1 for the
sticky bit. The other 3 bits have their usual significance.

Here we will discuss about the 3 special attributes other than the
common read/write/execute:

1.Set-User-Id (SUID)
2.Set-Group-Id (SGID)
3.Sticky Bit

Set-User_Id (SUID): Power for a Moment:

By default, when a user executes a file, the process which results in
this execution has the same permissions as those of the user. In fact,
the process inherits his default group and user identification.

If you set the SUID attribute on an executable file, the process res-
ulting in its execution doesn't use the user's identification but the
user identification of the file owner.

The SUID mechanism,invented by Dennis Ritchie,is a potential security
hazard. It lets a user acquire hidden powers by running such a file
owned by root.

$ ls -l /etc/passwd /etc/shadow /usr/bin/passwd
-rw-r--r-- 1 root root 2232 Mar 15 00:26 /etc/passwd
-r-------- 1 root root 1447 Mar 19 19:01 /etc/shadow

The listing shows that passwd is readable by all, but shadow is unre-
adable by group and others. When a user running the program belongs to
one of these two categories (probably, others), so access fails in the
read test on shadow. suppose normal user wants to change his password,
How can he do that? He can do that by running /usr/bin/passwd. Many
UNIX/Linux programs have a special permission mode that lets users
update sensitive system files –like /etc/shadow --something they can't
do directly with an editor. This is true of the passwd program.

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 22984 Jan 6 2007 /usr/bin/passwd

The s letter in the user category of the permission field represents a
special mode known as the set-user-id (SUID). This mode lets a process
have the privileges of the owner of the file during the instance of
the program. Thus when a non privileged user executes passwd, the eff-
ective UID of the process is not the user's, but of root's – the owner
of the program. This SUID privilege is then used by passwd to edit
/etc/shadow.

What is effective user-id:

Every process really has two user IDs: the effective user ID and the
real user ID. (Of course, there's also an effective group ID and real
group ID.Just about everything that's true about user IDs is also true
about group IDs) Most of the time,the kernel checks only the effective
user ID. For example, if a process tries to open a file, the kernel
checks the effective user ID when deciding whether to let the process
access the file.

Save the following script under the name reids.pl and make it
executable (chmod 755 reids.pl).

#!/usr/bin/perl
# print real UID
print "Real UID: $<\n";
# print real GID
print "Real GID: $(\n";
# print effective UID
print "Effective UID: $>\n";
# print effective GID
print "Effective GID: $)\n";

check file permissions:

$ ls -l reids.pl
-rwxr-xr-x 1 venu venu 203 Mar 24 10:40 reids.pl

Note: For security reasons the s-bit works only when used on binaries
(compiled code) and not on scripts (an exception are perl scripts).
Scripts,i.e. programs that cannot be executed by the kernel directory
but need an interpreter such as the Bourne shell or Java,can have
their setuid bit set, but it doesn't have any effect. There are some
platforms that honor the s bits even on scripts ( some System V vari-
ants, for example), but most systems don't because it has proven such
a security headache - most interpreters simply aren't written with
much security in mind. Set the SUID bit on shell script is useless,
that's why I am using perl script here.

When you run the script you will see that the process that runs it
gets your user-ID and your group-ID:

$ ./reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 500
Effective GID: 500 500

Note: If you get an error like this:
Can't do setuid (cannot exec sperl)

In Debian install perl-suid using following command:
apt-get install perl-suid

In Centos install perl-suidperl using following command:
yum install perl-suidperl

Now change owner ship to another user (Do it as an administrator).

# chown king /home/venu/reids.pl
# ls -l /home/venu/reids.pl
-rwxr-xr-x 1 king venu 203 Mar 24 10:40 /home/venu/reids.pl

Now run the script again.

$ ./reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 500
Effective GID: 500 500

What you observed, the output of the program depends only on the user
that runs it and not the one who owns the file.

How to assign SUID permission:

The SUID for any file can be set (mostly by the superuser) with a
special syntax of the chmod command. This syntax uses the character s
as the permission. Now add SUID permission to the script reids.pl :

# chmod u+s /home/venu/reids.pl (Do it from root account)

Now return from the super user mode to the usual non privileged mode.

$ ls -l reids.pl
-rwsr-xr-x 1 king venu 203 Mar 24 10:40 reids.pl

To assign SUID in an absolute manner, simply prefix 4 to whatever
octal string you would otherwise use (like 4755 instead of 755).

The file reids.pl is owned by king and has the s-bit set where norma-
lly the x is for the owner of the file. This causes the file to be
executed under the user-ID of the user that owns the file rather than
the user that executes the file. If venu runs the program then this
looks as follows:

$ perl reids.pl
Real UID: 500
Real GID: 500 500
Effective UID: 503
Effective GID: 500 500

Effective user id of process is 503, this is not the venu's , but of
king's - the owner of the program. As you can see this is a very powe-
rful feature especially if root owns the file with s-bit set. Any user
can then do things that normally only root can do.

Caution: When you write a SUID program then you must make sure that
it can only be used for the purpose that you intended it to be used.
As administrator, you must keep track of all SUID programs owned by
root that a user may try to create or copy. The find command easily
locate them:

# find /home -perm -4000 -print | mail root

The extra octal bit (4) signifies the SUID mode, but find treats the
"–" before 4000 as representing any other permissions.

Set-Group_Id (SGID):

The set-group-id (SGID) is similar to SUID except that a program with
SGID set allows the user to have the same power as the group which
owns the program. The SGID bit is 2,and some typical examples could be
chmod g+s reids.pl or chmod 2755 reids.pl.
You can remove SGID bit using following commands:

$ chmod g-s reids.pl
$ chmod 755 reids.pl (Absolute manner)

It is really useful in case you have a real multi-user setup where
users access each others files. As a single homeuser I haven't really
found a lot of use for SGID. But the basic concept is the same as the
SUID,Similar to SUID, SGID also grants privileges and access rights to
the process running the command, but instead of receiving those of the
file's owner it receives those of the file's group. In other words,the
process group owner will be set to the file's group.

I explain it with an example. I have created two user accounts king
and venu with same home directory project. king belongs to king and
development groups, venu belongs to venu and development groups.

# groups king venu
king : king development
venu : venu development

venu's default group is venu and king's default group is king.

Login as king and create reids.pl file again and make it executable
(using chmod 755 reids.pl) .

$ id
uid=503(king) gid=503(king) groups=501(development),503(king)
$ ls -l reids.pl
-rwxr-xr-x 1 king development 203 Mar 25 19:00 reids.pl

Now login as venu and run the program:

$ id
uid=501(venu) gid=504(venu) groups=501(development),504(venu)
$ perl reids.pl
Real UID: 501
Real GID: 504 504 501
Effective UID: 501
Effective GID: 504 504 501

The effective GID of the process is the venu's,but not of the king's
-the owner of the program.

Now login as king and assign SGID bit to reids.pl program:

$ chmod 2755 reids.pl; ls -l reids.pl
-rwxr-sr-x 1 king development 203 Mar 25 19:00 reids.pl

Now login as venu and run the reids.pl program:

$ perl reids.pl
Real UID: 501
Real GID: 504 504 501
Effective UID: 501
Effective GID: 501 504 501

Real GID and Effective GID are different,here Effective GID is the
king's - the owner of the program.

Set SGID on a directory:

When SGID is set on a directory it has a special meaning. Files crea-
ted in a directory with SGID set will inherit the same group ownership
as the directory itself,not the group of the user who created the file.
If the SGID is not set the file's group ownership corresponds to the
user's default group.

In order to set the SGID on a directory or to remove it, use the
following commands:

$ chmod g+s directory or $ chmod 2755 directory
$ chmod g-s directory or $ chmod 755 directory

As I mentioned earlier venu and king's home directory is same that is
/home/project. I changed group ownership of /home/project directory
to development.

# ls -ld /home/project/
drwxrwxr-x 16 root development 4096 Mar 26 00:22 /home/project/

Now login as king and create a temp file.

$ whoami
king
$ pwd
/home/project/
$ touch temp; ls -l temp
-rw-r--r-- 1 king king 0 Mar 26 12:34 temp

You can see from the ls output that the group owner for project is
development, and that the SGID bit has not been set on the directory
yet. When king creates a file in project, the group for the file is
king (king's primary gid).

Set SGID bit on project directory. For that login as administrator
and set SGID bit using following command:

# chmod g+s /home/project/
# ls -ld /home/project/
drwxrwsr-x 15 root development 4096 Mar 26 12:34 /home/project/

From the ls output above, you know the SGID bit is set because of the
s in the third position of the group permission set,which replaces the
x in the group permissions.

Now login as king and create temp2 file.

$ whoami
king
$ touch temp2; ls -l temp2
-rw-r--r-- 1 king development 0 Mar 26 13:49 temp2

Notice the group ownership for temp2 file. It inherits group permiss-
ion from the parent directory.

Enabling SGID on a directory is extremely useful when you have a
group of users with different primary groups working on the same set
of files.

For system security reasons it is not a good idea to set many
program's set user or group ID bits any more than necessary,since this
can allow an unauthorized user privileges in sensitive system areas.If
the program has a flaw that allows the user to break out of the inten-
ded use of the program, then the system can be compromised.

Sticky bit:

The sticky bit(also called the saved text bit) is the last permission
bit remaining to be discussed. It applies to both regular files and
directories. When applied to a regular file, it ensures that the text
image of a program with the bit set is permanently kept in the swap
area so that it can be reloaded quickly when the program's turn to use
the CPU arrives. Previously, it made sense to have this bit set for
programs like vi and emacs. Today,machines with ultra-fast disk drives
and lots of cheap memory don't need this bit for ordinary files and
that is also useless.

However, the sticky bit become a useful security feature when used
with a directory. The UNIX/Linux system allows users to create files
in /tmp, but none can delete files not owned by him. That's possible
because sticky bit set for /tmp directory.

The /tmp directory is typically world-writable and looks like this
in a listing:

# ls -ld /tmp
drwxrwxrwt 32 root root 36864 Mar 27 12:38 /tmp

Everyone can read,write and access the directory.The t indicates that
only the user (root and owner of the directory,of course) that created
a file in this directory can delete that file.

In order to set or to remove the sticky bit, use the following
commands:

$ chmod +t directory or $ chmod 1754 directory
$ chmod -t directory or $ chmod 754 directory

Note: 754 permissions for a directory are powerful enough to guard
your directories from intruders with malicious intentions, that's why
I used 754 as default,if yow want you can change it.

Example:

I logged in as king and created a temp file.

$ whoami
king
$ pwd
/home/project/
$ touch temp; ls -l
-rw-r--r-- 1 king king 0 Mar 27 13:44 temp

Now logged in as venu and try to delete temp file.

$ whoami
venu
$ rm temp
rm: remove write-protected regular empty file `temp'? Y
$ ls temp
ls: temp: No such file or directory

So what happened? venu deleted file owned by king.

Assign sticky bit to the project directory.As a owner of the directory
or administrator.

# chmod +t /home/project
# ls -ld /home/project/
drwxrwxr-t 15 root development 4096 Mar 27 13:46 /home/project/

From the ls output above, you know the sticky bit is set because of
the t in the third position of the other permission set,which replaces
the x in the other permissions.

Now repeat same steps again,then you get the following message:

$ whoami
venu
$ ls -l temp
-rw-r--r-- 1 king king 0 Mar 27 17:36 temp
$ rm temp
rm: remove write-protected regular empty file `temp'? y
rm: cannot remove `temp': Operation not permitted

Observation: Login as normal user and create a file.
[venu@localhost ~]$ touch sample
[venu@localhost ~]$ ls -l sample
-rw-rw-r-- 1 venu venu 0 Dec 21 03:41 sample

Now change permissions to 644

[venu@localhost ~]$ chmod 644 sample
[venu@localhost ~]$ ls -l sample
-rw-r--r-- 1 venu venu 0 Dec 21 03:41 sample

Now assign SUID permission.

[venu@localhost ~]$ chmod u+s sample
[venu@localhost ~]$ ls -l sample
-rwSr--r-- 1 venu venu 0 Dec 21 03:41 sample

After setting SUID, if you see 'S' then it means that the file has no
executable permissions for that user.

Now remove SUID permission and change permissions to 744. Then assign
SUID permission. You should see a smaller 's' in the executable permi-
ssion position.

[venu@localhost ~]$ chmod u-s sample
[venu@localhost ~]$ chmod 744 sample
[venu@localhost ~]$ chmod u+s sample
[venu@localhost ~]$ ls -l sample
-rwsr--r-- 1 venu venu 0 Dec 21 03:41 sample

Same is applicable for SGID and Stickybit.

Posted by venu k at 10:51 AM
Labels: Article

Switch Google Desktop Search to search desktop by default

2011-03-01T12:14:00.002+13:00

I recently installed the latest version of Google Desktop. I found, however, that when searching either from the sidebar searchbox or the Ctrl Ctrl popup box, GD would always select Search Web by default.

This can now be changed using a small registry edit on Windows XP it is:
HKEY_CURRENT_USER\Software\Google\Google Desktop\Deskbar\ change search_type to "Search Desktop"

That's it. Hopefully this will save some of you some time.

Some programming guidlines

2011-01-05T17:35:00.003+13:00

This is a very simple post. I found these principals very useful in solving a problem I have with a particular project I am currently working on...

http://en.wikipedia.org/wiki/Unix_philosophy

1.Small is beautiful.
2.Make each program do one thing well.
3.Build a prototype as soon as possible.
4.Choose portability over efficiency.
5.Store data in flat text files.
6.Use software leverage to your advantage.
7.Use shell scripts to increase leverage and portability.
8.Avoid captive user interfaces.
9.Make every program a filter.

Have a read. It's well worth remembering these guidelines. Even if you have read them before, it might be useful to refresh your memmory.

Python Basic HTTPS webservice

2010-12-15T03:18:00.001+13:00

The basic webservice module in my previous posting does not support https or ssl encryption.

Here is how I finally managed to work it out:


# Basic https web server

import socket, ssl

host = ''
port = 8080

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(1)

while 1:
    osock, caddr = sock.accept()
    csock = ssl.wrap_socket(osock,
                                server_side=True,
                                certfile="servercert.cer",
                                keyfile="serverkey.key",
                                ssl_version=ssl.PROTOCOL_SSLv23)
                                
    cfile = csock.makefile('rw', 0)

    # Protocol exchange - read request
    msg = "GO AWAY!"
    while 1:
        line = cfile.readline().strip()
        if line == "GET /echotest HTTP/1.1":
            msg = "ECHOTEST OK"
        if line == '':
            cfile.write("HTTP/1.0 200 OK\n\n")
            cfile.write("<head><title>Eh?</title></head>")
            cfile.write("<h1>%s</h1>" % msg)
            cfile.close()
            csock.close()
            osock.close()
            break

Compile Pana (Amarok 1.4 clone) on Fedora 14 x86_64

2010-11-08T17:36:00.005+13:00

After meeting all the various dependencies, I had to make one small actual code change to the source code. I would have liked to submit it on the Pana website but could not find a way to do that.

Anyway here is the patch.


[dave@fedora junk]$ diff [source tree]/pana/src/osd.h [source tree]/pana/src/osd.h.original
40c40
<         void show( const QString &text, QImage newImage = QImage() );
---
>         void show( const QString &text, QImage newImage = QImage::QImage() );