Admin Reminders

Set CPU family for a specific VM on oVirt 3.5

2015-06-25T19:27:00.000+01:00

I have been playing around with an oVirt test lab now on and off for a couple of months and I must say I love it. The community is active and helpful (either via the users@ovirt.org mailing list or #ovirt on irc.oftc.net using IRC) and the software is continuously upgraded and improved by a well organised development team.

One of the features that is missing in version 3.5 is the ability to set the CPU family on a VM to CPU family with a lower set of features than the current hardware (actually the cluster but that is likely to be set to the same thing). Sounds daft, why would anyone want to do this, well you can test if a VM would work on a older class of hardware I suppose. What I wanted it for a bug with Windows 10 previews that throw a SYSTEM THREAD EXCEPTION NOT HANDLED which prevents installing on a SandyBridge CPU on the KVM hypervisor (the hypervisor used by oVirt). The answer is to fall back to the older CPU family of Westmere. To do that on stock 3.5 oVirt you need to set the whole cluster to that CPU family, there is not option to do just a VM.

I presumed someone else had figured a way to do this so asked on the #ovirt IRC channel, no one knew but did try to put me in touch with some who might help but they were away. I then emailed the mailing list and got back two prompt answers. First saying I need to use something called VDSM hooks and another says the feature will be available in 3.6 if I can wait or upgrade before final release.

I looked up VDSM hooks, these are (mostly python) scripts that can effect certain events on a running ovirt engine. The event I needed to look at is before_vm_start, following instructions and modifying the example script on this page I was able to customise my lab to enable CPU setting.
Create the following script on each host in the cluster.

vi /usr/libexec/vdsm/hooks/before_vm_start/50_cpufamily

Paste in the following text:

#!/usr/bin/python

import os
import sys
import hooking
import traceback


if os.environ.has_key('cpufamily'):
    try:
       domxml = hooking.read_domxml()   #here we read the VM XML into the domxml variable
       vcpu = domxml.getElementsByTagName('cpu')[0] #find and read the CPU definition in the VM XML
       sys.stderr.write('cpufamily: Changing cpu family to: %s\n' % os.environ['cpufamily'])  #sys.stderr.write is caught by vdsm and logged into vdsm.log for debugging
       e = domxml.createElement('model')
       txt = domxml.createTextNode(os.environ['cpufamily'])
       e.appendChild(txt)
       modelnode = vcpu.getElementsByTagName('model')[0]
       vcpu.replaceChild(e,modelnode)
       hooking.write_domxml(domxml)                       #and write to the altered domxml
    except:
       sys.stderr.write('cpufamily: [unexpected error]: %s\n' % traceback.format_exc())
       sys.exit(2)

Change the script to be executable:

# chmod +x /usr/libexec/vdsm/hooks/before_vm_start/50_cpufamily

If you are using the hosted engine setup on one of the hosts put the HA agents into global maintenance with:

# hosted-engine --set-maintenance --mode=global

Then on just the engine machine:

# engine-config -s UserDefinedVMProperties='cpufamily=^(Conroe|Penryn|Nehalem|Westmere|SandyBridge)$' --cver=3.5

# service ovirt-engine restart

Take the ha agents out of maintenance mode:

# hosted-engine --set-maintenance --mode=none

This should now allow you to log into the engine GUI, edit a VM (or create a new one) under advanced options go to custom properties and select the cpufamily key and set an appropriate CPU family, the next time you boot your VM will only have access to those CPU features relevant to the CPU family you have selected.

Genius open-vm-tools

2015-02-16T20:36:00.003+00:00

I have always felt that running a Debian VM in a VMware environment was a little bit of a pain, installing the vmware tools was easy enough that you could do it in a few minutes but not so easy that I could remember how to do it (hence this post).

Now VMware fully support open-vm-tools things are much easier if you are running wheezy or newer, you can simply run:

apt-get install open-vm-tools

Then everything just works.

I look forward to the day when the Debian installer detects that it is getting installed on a VMware VM and does this automatically for you at install.

Powershell script to bring up Hyper-V VMs Slowly enough that they function

2014-11-05T22:26:00.001+00:00

The below Powershell script is something I wrote out of frustration while on an Exchange Messaging course to try to improve my mood. The Hyper-V VMs we were starting for each lab took an absolute age to load up and become responsive after we reverted their state at the end of each lab. This script attempts to start machines one after another with a configurable gap in between, the idea being you could set the $vms array to the machines you want started then wander off for a bit of a break then hopefully when you return the VMs would have loaded and be responsive.

You may ask why wouldn't you just start them all together, well the machines that were running the VMs were not up to spec and each of the VMs seemed to be based off of a single VM using dereferenced disks which severely impacted performance. The VMs took so long to load that Services on each of them often failed to start because of timeouts.

I thought I would share in the hope it may save someone else pulling their hair out.

# Machines to start in order to start
$vms = "20342B-LON-DC1", "20342B-LON-CAS1", "20342B-LON-MBX1", "20342B-LON-CL1", "20342B-LON-CL2" , "20342B-LON-LY1"
# minimum uptime in seconds to check for before starting next machine
$delaytime = 45
Function StartVM
 {
     param ($Name)
     $vmquery = get-vm $Name
     if ($vmquery.state -eq "off")
     {
        Write-Host " not running, starting" -NoNewline
        start-vm $Name
        $vmquery = get-vm $Name
     }
     elseif ($vmquery.state -eq "Running")
     {
        Write-Host " already running" -NoNewline
     }
     While ($vmquery.uptime.TotalSeconds -lt  $delaytime) { $vmquery = get-vm $Name; sleep 2 ; Write-Host . -NoNewline }  
     write-host " uptime is "-nonewline
     write-host $vmquery.uptime.TotalSeconds -NoNewline
     write-host " seconds, over $delaytime so assumed to be up and running."
}
foreach ($vm in $vms) {
    write-host "Checking $vm... " -NoNewline
    StartVM $vm 
}
write-host "If $delaytime seconds was long enough your VMs should be functioning now."

I am still on the course studying for 70-341: Core solutions of Microsoft Exchange Server 2013 and 70-342: Advanced Solutions of Microsoft Exchange Server 2013, So I may well update the script as I run more labs. Ideally I would like to improve the script so it waits until the started VM is "responding" before starting the next one but this has proved difficult, if you have any suggestions please let me know.

Update: 06/02/2015, I have improved the script a little, it now copes better with machines being started from other sources or if the script is restarted and the times are in seconds as the latest tests I have endured have been on lab machines using SSDs which significantly improves performance.

Squeezing more from your Linux VMs

2014-09-09T21:22:00.000+01:00

There is a way to squeeze a little more from Linux VM's.

The theory

Linux has a number of ways of sharing it's storage IO among different processes, the norm seems to be to use the Completely Fair Queuing (CFQ) scheduler which helps to prevent a single process from using more than it's fair share of storage IO. This is usually helpful in an environment where this one Linux box is the only OS using a storage device but when it is running on an hypervisor the hypervisor is also busy trying to dish out fair access to the storage so we are effectively doubling up calculating fair access. There are other schedulers that we can choose from and for VM's it seems to make the most sense to use either the noop or the deadline schedulers. They are both more simple to calculate then CFQ.

The practice

If you are using the device sda for your drive use the following to check what scheduler you are using :

# cat /sys/block/sda/queue/scheduler

which will return something like

noop anticipatory deadline [cfq]

Which indicates that the scheduler for this device is CFQ.

To change it on the fly you just:

echo noop > /sys/block/sda/queue/scheduler

If you want to make the change work across reboots you will need to add elevator=noop to your kernel boot parameters, on a Debian system you edit /etc/default/grub and add "elevator=noop" to the GRUB_CMDLINE_LINUX line then run:

# update-grub

To update the grub configuration.

Sources: kb.vmware.com/kb/2011861, http://serverfault.com/questions/360718/kvm-low-io-performance

Remote desktop gateway connections failing when KB2592687 installed.

2013-12-15T12:05:00.002+00:00

While setting up a lab running Remote Desktop Gateway on Windows Server 2012 R2 I came across a strange problem. I was able to connect through the RD Gateway using some machines but not others.
On the machines that were failing I got the following message when trying to connect:
"Your computer can't connect to the remote computer because an error occurred on the remote computer that you want to connect to. Contact your network administrator for assistance."

After much pulling of hair I narrowed it down to something to do with the KB2592687 update. This update installs RDP version 8.0 on Windows 7 SP1 machines. This update was present on both the machine I was trying to connect to and on the clients that were failing. There is a list of known problems in knowledge base article but none of them applied to my setup.

More searching found loads of red herrings, then I discovered this post. It mentions LAN manager authentication level settings (Local security policy->Local Policies->Security Options->Network Security: LAN Manager authentication level). On the failing client it was set to "Send LM & NTLM - user NTLMv2 session security if negotiated." Changing it to "Send NTLMv2 response only" (which seems is the default on Vista and above) made the connection work.

Extending disks on a Windows VM

2013-06-16T16:22:00.000+01:00

Extend disks on a Windows Server 2008 R2 machine (and probably Windows 7 as well).

Extend the hard drive in VM Settings, this can be done while Windows is running but I was unable to get Windows to see the new size until after I rebooted the VM.

GUI
Server manager->Storage->Disk management, right click on the drive and choose "Extend Volume..." select size to expand to then click next.

Command line way
Start->Run and type diskpart.
At the disk part prompt type:

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online           40 GB  5121 MB

This will show you a list of the disks in the system, once you have identified the disk you want by it's number (I only have disk 0 (zero) in the example above).
We now need the volume number:

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D   CD           CDFS   DVD-ROM      112 MB  Healthy    
  Volume 1         System Rese  NTFS   Partition    101 MB  Healthy    System
  Volume 2     C                NTFS   Partition     40 GB  Healthy    Boot

Now "select" the volume you want to extend (in this case I am extending my C drive so it is volume 2).

DISKPART> select volume 2

Volume 2 is the selected volume.

Now tell diskpart to extend the volume into the all the free space on disk 0:

DISKPART> extend disk=0

DiskPart successfully extended the volume.

That is it, all done.

Another easy way to extend disk sizes (as well as many other useful functions) is to run a live CD of gparted. This will also works with other operating systems such as Linux. In fact it should work any file system that gparted understands, check out there features page.

Setting up a branch office VPN between a Watchguard XTM505 and a Debian linux box.

2013-05-13T12:00:00.000+01:00

Lets us say you have two sites and want to build a VPN between them. The first site has a Watchguard XTM device has a public IP address of 99.99.99.99 which protects a private IP address range of 10.0.1.0/24 and your second site has just a Debian Linux box which has a dynamic IP address that protects a private IP address range of 10.0.2.0/24

Debian side

Install needed packages:

# apt-get install ipsec-tools racoon

Choose "direct" for racoon setup

Edit /etc/ipsec-tools.conf

#!/usr/sbin/setkey -f

# NOTE: Do not use this file if you use racoon with racoon-tool

# utility. racoon-tool will setup SAs and SPDs automatically using

# /etc/racoon/racoon-tool.conf configuration.

## Flush the SAD and SPD

# flush;

# spdflush;

spdadd 10.0.1.0/24 10.0.2.0/24 any -P in ipsec

esp/tunnel/99.99.99.99-10.0.2.254/require;

spdadd 10.0.2.0/24 10.0.1.0/24 any -P out ipsec

esp/tunnel/10.0.2.254-99.99.99.99/require;

Edit /etc/racoon/psk.txt and add the line

99.99.99.99 Somerandomkey

Edit /etc/racoon/racoon.conf

# Simple racoon.conf

# Please look in /usr/share/doc/racoon/examples for

# examples that come with the source.

# Please read racoon.conf(5) for details, and alsoread setkey(8).

# Also read the Linux IPSEC Howto up at

# http://www.ipsec-howto.org/t1.html

path pre_shared_key "/etc/racoon/psk.txt";

path certificate "/etc/racoon/certs";

remote 99.99.99.99 {

exchange_mode aggressive,main;

dpd_delay 20;

dpd_maxfail 5;

my_identifier user_fqdn "something@company.com";

proposal {

lifetime time 28800 second;

encryption_algorithm 3des;

hash_algorithm sha1;

authentication_method pre_shared_key;

dh_group modp1024;

}

sainfo address 10.0.2.0/24 any address 10.0.1.0/24 any {

lifetime time 28800 second;

pfs_group modp1024;

encryption_algorithm 3des;

authentication_algorithm hmac_sha1;

compression_algorithm deflate;

}

Watchguard side

I have simply taken screen shots for the configuration for the watchguard side:

You will need to ensure that the preshared random key is the same as you used in the /etc/racoon/psk.txt file you created above.

VMs, snapshots and domain computer accounts.

2013-03-19T00:58:00.000+00:00

Have you ever had the problem where you have reverted a VM's snapshot to find that it's computer password is out of sync with the domain? I have, loads of times. This is often seen with the following message when you attempt to log in: "Windows cannot connect to the domain, either because the domain
controller is down or otherwise unavailable, or because your computer
account was not found."

The problem is that Windows machines on a Domain change their computer account passwords with a Domain controller every 30 Days. If a machine changes it's computer password with a domain controller and you then revert to a snapshot that was taken before the password was changed the computer account will no longer be able to authenticate on the network and domain users won't be able to logon.

How do you get around this? One way is in Microsoft's KB article 154501.

This can reduce the security on your domain, or at least the security between the DC and the workstation you make the following registry changes on but if you have a testing setup like I do this is not much of a problem and the convenience easily outweighs any security issues (in my opinion :).
You can set the DisablePasswordChange registry entry to 1 in :

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

That has now stopped this machine from changing it's domain password every 30 days. Now we need to get the machine back onto the domain, to achieve this do one of the following:

A. Remove the computer from the domain and add it back in again. Easy enough you say. I have a lot of test machines floating about that don't always have the same local administrator password. So it is a good idea to make sure that you know a local administrators password before you remove the machine from the domain otherwise you will end up with a machine you cannot access via windows (all is not necessarily lost, trinity rescue kit can help here).

B. Remove and rejoin the machine to the domain using the netdom.exe command:

netdom remove <machine name> /Domain:<domain name> /userd:<domain name>\<domain administrator account name> /passwordd:<domain administrator password>

Wait for the response:

The command completed successfully.

Then run:

netdom join <machine name> /Domain:<domain name> /userd:<domain name>\<domain administrator account name> /passwordd:<domain administrator password>

Again we are looking for the response:

The command completed successfully.

It is probably best to reboot the machine at this point, it is a windows machine after all we were messing with the domain membership and I simply would trust a machine in this state to function as expected unless it is rebooted.

ESXi not accepting previously used volume

2012-11-14T17:29:00.000+00:00

I recently had a problem on an ESXi 5.0 box I was trying to add a iscsi target to. I had set up a 6 TB iSCSI LUN but everytime I tried adding it to ESXI I got the following error:

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastoresystem" on ESXi "ESX HOSTNAME" failed.

It turned out that the setup wizard on the device had happily created a volume and formatted it with ext4 before I realised what it was doing. Although I removed this volume the partition information wasn't removed from the array and this was upsetting ESXi.

How do you fix it? As there was no obvious way to wipe the partition info from within the Array's management interface I decided to do it from a Debian VM running on that host.

Firstly I removed the discovery information from the ESXI server so it isn't trying to interfere then on the Debian VM I installed open-iscsi with:

apt-get install open-iscsi

Start the open-iscsi daemon with:

/etc/init.d/open-iscsi start

Query the iSCSI target on the storage device:

iscsiadm -m discovery -t st -p 192.168.1.99

Which should return something like this:
192.168.1.99:3260,0 iqn.2010-12.com.manufacturer:nasdevice.name

If you have set up iSCSI authentication on your storage device you will need to run something like the following commands using the iqn in the above response.

iscsiadm --mode node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" -p 192.168.0.99:3260 --op=update --name node.session.auth.authmethod --value=CHAP
iscsiadm --mode node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" -p 192.168.0.99:3260 --op=update --name node.session.auth.username --value=username
iscsiadm --mode node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" -p 192.168.0.99:3260 --op=update --name node.session.auth.password --value=password

Logon to the storage device:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --login

All being well this should now create a SCSI device as if you had attached a hard drive directly to the system. I looked at the bottom of the output from the dmesg command to find out which device (/dev/sdb).

I used the following command to write zeros to the first half a MB of the disk which will overwrite any partition table information (care should be taken that you have the correct device when using this command, it will eat your drive):

dd if=/dev/zero of=/dev/sdb bs=512 count=1024

Now logout of the storage device with the following command:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --logout

ESXi should now happily accept the iSCSI LUN when you attempt to add it.

Local security policy on Windows 8

2012-11-12T12:01:00.000+00:00

The Local Security Policy MMC in Windows 8 how now been moved further away from the user (perhaps with good cause) and is available by searching for "secpol.msc."

This is reminiscent of the move to hide the Component services mmc in Windows 7/Windows Server 2008 R2 which can me accessed by searching for "comexp.msc."

Installation of .net 3.5 on Windows Server 2012

2012-11-06T16:33:00.000+00:00

I struggled installing .net 3.5 on Windows 2012 Server. After going through the add features and selecting .net 3.5 I got the following Warning:

The request to add or remove feature on the specified server failed.
Installation of one or more roles, role services, or features failed. - The source files could not be downloaded.
Use the /source option to specify the location of the files that are required to restore the feature. The file location should be either the root directory of a mounted image or a component store that has the Windows Side-by-Side directory as an immediate subfolder.

From a administrative command prompt run the following:

dism.exe /online /enable-feature /featurename:NetFX3 /Source:d:\sources\sxs /LimitAccess /all

This will install .net 2.0 and 3.5.

"Remote Downlevel Document" instead of jobname in Samba

2012-07-13T12:35:00.004+01:00

To save the planet I implemented a print to PDF server for our Quality Assurance department at work. It consists an installation of Samba and a hand full of custom scripts.

I recently upgraded this server and added it to our domain (up until now there were no sensitive documents being processed but as we planned on adding these it became necessary to restrict access to some shares).

I have just spent far too much time trying to work out why windows was not passing the document name from the printing application correctly. It seems that the "jobname" parameter being passed from Samba to my script was always "Remote Downlevel Document" when previously it was something more helpful like the title of a document.

The offending line in the configuration was:

disable spoolss = yes

Once this was removed everything started working as before.

Using screen on a linux server

2012-06-21T15:09:00.002+01:00

Once in a while I need to leave a process running for a long period of time on a Linux server I will use the screen command. Screen allow you to run a process in a session then disconnect your SSH session but leave the process running, then log back in later and check in on the process from another SSH session.

Once screen has been installed (sudo apt-get install screen on a Debian system) you can simply type screen followed by the command line for the process you want to run. So if you want to leave a network trace running you would use something like:

# screen tcpdump -ieth0 host hostname.domain.com -w ./host.cap

This would start up tcpdump and run it in a screen session.

If you only have one screen session running on a machine then to get to it you can use:

# screen -r

However if you plan on running more than one session then you may want to give the sessions names, so for example you can run :

# screen -S capture-session tcpdump -ieth0 host hostname.domain.com -w ./host.cap

Then you can access that specific session with:

# screen -r capture-session

Once you are in a session it will perform like an ordinary SSH connection. If you wish to disconnect but leave the session running you will need to press <CTRL>+a, then d.

Installing Windows 8 x64 Release Preview on an ESXI VM

2012-06-19T21:17:00.000+01:00

I simply thought that it would be a case of installing Windows 8 release preview and I would be able to have a tinker. Unfortunately not.

Firstly make sure you are on ESXi 5.0 update1. Create a virtual machine and set the OS to "Microsoft Windows 8 (64-bit)". At the end of creating the VM, us the check box"open setting dialog"

In the VM's settings:

Change the video card to Auto detect settings and check the "Enable 3D support"
go to the Options->Boot Options and change the firmware to EFI

Now proceed with the installation of Windows 8 Release Preview and all should work.

I didn't modify the settings on the VM the first time I tried to install Windows 8 and ended up having to reinstall after making the above changes.

MySQL Server Replication

2012-06-06T20:29:00.000+01:00

Recently had cause to check out MySQL's replication capabilities. One of the programmers wanted a wiki for his department's to use. We already used Mediawiki on a Debian box for another department so I thought we could beef up the box with some RAM then add a new wiki to it, change the backup scripts and we would be done. However I got thinking that this might be an opportunity to provide a more resilient set up where the MySQL server running the wiki could replicate to an offsite location. This could then provide a read-only back up to the existing machine in case of failure. I will of course still need to take backups (just in case some Flump deletes the lot).

The following is how I set up replication between a production mediawiki server with the MySQL server installed locally.

It kind of goes without saying that you will need at least two machines each running MySQL, in my set up both machines were Debian 6 (Squeeze).

On the Production machine we will need to to configure MySQL so it knows it is going to be part of a replication group. Edit my.cnf and add or modify the following:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log

We also need to get MySQL to bind to the network address of the NIC

bind-address = 0.0.0.0

On the Offsite machine we need to tell it to be a slave:

server-id = 2
log_bin = /var/log/mysql/mysql-bin.log

We need to create an account on the master (production) machine for the slave to be able to log on to the master for replication.

$ mysql -u root -p

mysql> CREATE USER 'replicationos'@'%' IDENTIFIED BY 'password';

Query OK, 0 rows affected (0.10 sec)

mysql> GRANT REPLICATION SLAVE ON *.* TO 'replicationos'@'%';

Query OK, 0 rows affected (0.00 sec)

We now need to get the Replication master binary log components. If this is done on a production machine then I suggest you do it quickly because you will stop certain transactions from occurring while the tables are locked.

open two sessions to mysql, in the first run:

mysql> FLUSH TABLES WITH READ LOCK;

Make a note of the"File" and "Postition":

Exit mysql client in the second session (still leaving the first logged in). Now dump the data out ready to be imported by the slave with:

$ mysqldump -p --all-databases --lock-all-tables >/root/dbdump.db

Copy the dbdump.db to the slave machine with scp then on the slave machine start mysql with the --skip-slave-start option:

mysqld --skip-slave-start

Import the data with:

mysql -p < /root/dbdump.db

Now we need to tell the slave machine about the master using the "File" and "Position" we noted in a earlier step with the following command:

mysql> CHANGE MASTER TO MASTER_HOST='wiki.company.com', MASTER_USER='replicationos', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=21381;
Now start the slave process:
mysql> start slave;

Wait for a few moments then check the status of replication with the following:

mysql> SHOW SLAVE STATUS\G

*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: wiki.company.com
Master_User: replicationos
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000027
Read_Master_Log_Pos: 37742
Relay_Log_File: mysqld-relay-bin.000511
Relay_Log_Pos: 7333
Relay_Master_Log_File: mysql-bin.000027
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 37742
Relay_Log_Space: 7489
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)

The \G in the "SHOW SLAVE STATUS\G" is short hand for "ego" in the MySQL client and it means "Send command to mysql server, display result vertically." Check that the "Slave_IO_State" is "Waiting for master to send event" and that "Seconds_Behind_Master" is 0 and that means that your slave is synchronised with your server (unless you have only just lost network connection to the master).

Linux giving out wrong MAC address

2012-04-20T22:41:00.000+01:00

I recently had a problem where a Linux box with multiple NICs running a iptables firewall script was not accepting connections when it should. I ran tcpdump to find that ARP lookups received by the box were actually giving out the a MAC address from a different NIC. This seemed bizarre at first, but knowing Linux's networking strength I realised that I was missing something and that I probably needed to tweak the kernel to behave the way I wanted.

A quick Google found that I probably wanted to turn on arp_filter in the kernel. So I ran the following:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

Immediately tcpdump showed that the kernel was now behaving the way I wanted it to and handing out only the MAC address for the relevant NIC.

To make the change permanent (i.e. survive a reboot) you could add the above line to a init script or add the following line to /etc/sysclt.conf

net.ipv4.conf.all.arp_filter = 1

NTFS alternate data streams and zip files

2012-04-17T22:29:00.001+01:00

If you download a zip file onto a NTFS partition Internet Explorer will store zone information in the file using something called alternate data streams. If the zone information is stored with the file it will be "blocked" within Windows. The file can easily be unblocked by clicking the "Unblock" button in the files properties (if your user account has the correct permissions).

File properties showing the unblock button

We recently had a problem where we were downloading a zip file to a customer's machine, unzipping the archive and when we tried to run the executable that was in the archive it was failing. It seems that if you do not "unblock" the zip file then all of the files extracted from the archive will all be in the same blocked state. This was causing problems for our support team and it seems the answer to this problem is to simply unblock the zip before unpacking the archive (I suppose you could unblock each individual file after the extraction but we had a lot of files).

To see this behaviour download a file with IE and look at the properties. The alternate data stream is called Zone.Identifier, you can see the contents of the alternate data stream with the following command:

more < fullfilename:Zone.Identifier

Seeing the zone information saved with the alternate data stream on a downloaded file

Copy and paste in the vSphere client

2012-02-24T18:07:00.000+00:00

I assumed that the ability to copy and paste text between your desktop and virtual machines had been removed entirely from the vSphere client. It turns out that it has just been disabled by default for security reasons and that it can be enabled per VM or per ESX host.

I found a need for this when setting up machines with little or no contact with the outside world, but that you want to make complicated configuration changes that you have previously documented.

Information on how to enable copy and paste in the vSphere client is available in VMware's KB article 1026437.

Converting a Debian VM from Xenserver to vSphere

2011-12-16T15:47:00.000+00:00

You can transfer the data for the machine from one environment to another using clonezilla. Once it is transferred I found that the machine does not boot. On further investigation I realised that there are many differences between a Debian install on Xenserver and on vSphere.

If you get the message "Operating system not found" you will need to install grub to the virtual hard disk. To do this boot a liveCD (such as clonezilla).

As root run grub, at the grub> prompt type:

grub> find /boot/grub/stage1

This will return the location of grubs stage one, make a note of this location for the next command. Type

grub> root (hd0,0) < this is the location returned by the last command

Now install grub to the master boot record (MBR) with:

grub> setup (hd0)

then quit grub with:

grub> quit

Reboot to get the grub menu but if you let it try and continue using the default settings it will very likely hang because it will try to use a block device name that begins with xvd, these are xenserver specific and will not exist on the VMware VM. To get it to continue booting you need to hit 'e' on the grub selection screen, then edit the boot line to replace the device that looks like /dev/xvda1 and made it look like /dev/sda1, also remove the "console=hvc0" otherwise you will not be able to see interact with your OS via the console (hvc0 is another xenserver specific device name).

Once we have booted the system we need to go about preparing the system to boot correctly when left to it's own devices (geddit?), firstly ensure the keyboard keymap is set up correctly with:

# dpkg-reconfigure console-data

Edit /etc/fstab to change all references to xvd devices to sda in /etc/fstab

On many of the machines I transferred using this method I found that they had no swap partitions after the transfer. So I simply set the disk size slightly bigger than the source disk when initially creating the VM in vSphere, then added a partition with fdisk. I usually set the swap partition as /dev/sda2.
You then need to make that partition a swap partition with (I have found often that a reboot is required before the command works):

# mkswap /dev/sda2

Then you can mount it with:

# swapon /dev/sda2

And see if the swap space is available with:

# free

Don't forget to update /etc/fstab so it is pointing to the correct device for the swap partition so it is automounted at boot.

The CDRom is set to /dev/hdc unless you have changed the VM's hardware settings, update this in /etc/fstab as well.

We need to change grubs configuration so that it automatically boots a with appropriate parameters, this information is stored in /boot/grub/menu.list. You can achieve this by either running the following commands:

sed -i 's/xvd/sd/g' /boot/grub/menu.lst
sed -i 's/console=hvc0//g' /boot/grub/menu.lst

Or edit /boot/grub/menu.lst, find the line that begins # kopt, remove the console=hvc0 from the end and change the "xvd" device to the correct boot device. move down to the botton where the menu choices are configured and change the device and remove the console=hvc0 from each of the one you will likely use (the top one only in my case).

Edit /etc/inittab and comment out the line that begins "co:"

Remove the xenserver specific packages from your apt sources list with:

rm /etc/apt/sources.list.d/citrix.list

Update your packages list:

apt-get update

Remove all the kernels that are installed with:

apt-get remove linux-image*

You will get a warning asking about the removal of the running linux kernel, say "No" to this as we will install another one that we can get headers for in the next step.

Install a more appropriate kernel with:

apt-get install linux-image-2.6-686

Reboot to run the newly installed kernel.

Install VMware tools, details of how in my post here, then reboot to test.

To get the most out of the VM you should switch hardware to paravirutal drivers for SCSI controllers and NICs.

Convert a Windows machine to a VMware VM with Clonezilla

2011-12-15T23:20:00.000+00:00

I have had need to do manual conversions of physical and virtual machines for several reasons previously. Some of these reasons have included:

There is not enough free space on the machine's disk(s), (also know as "I haven't got time to wait for the machine's administrator to tidy up")
VMware Convert no longer supports the conversion of that operating system.
We can't afford PlateSpin
I wonder if I could do a conversion the hard way.

This takes me back to my first ESX Project. The company I worked for had about 8 test machines sitting along one office wall. The situation was already out of hand and I was being asked for more test machines. I decided we needed a to think big and decided on and got approved ESX on a couple of big (at least for us) servers to virtualise the test environment. I managed to simply move all the test machines by installing a fresh OS and installing and configuring all the software my self, with the help of the relevant departments. I then came to our Windows NT 4.0 software build machine.... No-one, not even the developer that put it together knew what was on there. We were building releases every couple of weeks at the time. It goes without saying this is not a good place to be, and baring a complete hardware failure on the physical machine there was no way anyone was going to be rebuilding the machine. I scratched my head and had previously used dd and tar to move installs from one machine to another on Linux, so I had a crack at it, and to my surprise a few hours later I had run a P2V by hand with a Linux live CD. I had even managed convince Windows NT to let me switch the drives from IDE to SCSI.

Having used Clonezilla to store a few images from machines that were shipped without restore media in the past, I wondered how easy it would be to back up an image of a machine and restore it into a new virtual environment. As it turns out with Clonezilla I don't even need to create an image, the tools are there to transfer the data from one machine's disk and write it directly to the disk of another.

The conversion here is from a Xenserver VM to a VMware VM but the technique used here will probably work else where.

Put a Clonezilla live CD in both your (virtual) machines and boot them.
The default boot option for Clonezilla worked fine for me on both machines, your mileage may vary.
Select your language options as necessary

On the source machine:

Select "Start_Clonezilla"
Select "device-device"
Select "Beginner"
Select "disk_to_remote_disk"
Select "dhcp" or assign an IP address with "static"
Now it should show you a list of disks in the system, select the one you wish to transfer.
Select "Skip checking/repairing source file system"
Clonezilla will now show you the command it is about to run, press "Enter."
You will be asked to confirm at a few important stages that you wish to proceed.
When you see "Waiting for the target machine to connect..." you have finished with the source machine for now, except take a note of the commands it is telling you run on the destination machine (See below image.)

On the destination machine:

Select "Enter_shell"
Select (2)
Type: sudo su - (to change to the root user)
Type: ocs-live-netcfg (and setup the networking)
Type: ocs-onthefly -s 192.168.1.1 -t sda (replacing 192.168.1.1 and sda with the relevant IP address and device name)
You will again be asked to confirm at poignant stages.

You should see a progress bar which includes an estimate for the finish time.

When this finishes you should now be able to boot the destination VM and install the VMWare tools to make everything pretty and efficient. Don't forget to install the paravirtual SCSI and network card drivers if your OS is supported.

If you have success (or failure) with this migration method please leave a comment below, it certainly helped me out of a hole.

Virtual Center Template for Debian

2011-12-05T17:30:00.001+00:00

When using a windows template, VMware's Virtual Center can help you customise the resulting virtual machine with a wizard, unfortunately it doesn't do this for Debian which happens to be my Linux distribution of choice.

To create the template I just installed a nice minimal installation onto a new VM, added the vmware tools, changed the VM to use paravirtual hardware then converted it to a template.

After each "Deploy from template" I simply run through the following:

It would be potential security risk to have all the VMs using the same SSH keys so I regenerate SSH keys for the VM with:

# rm /etc/ssh/ssh_host*

# dpkg-reconfigure openssh-server

Update packages

# apt-get update

# apt-get upgrade

Set the IP address (well it is likely to be a server) with:

# vi /etc/network/interfaces

Change host name as it isn't helpful to have duplicate host names due to the confusion it can cause me:

# vi /etc/hostname

Amend hosts file (to match the host name)

# vi /etc/hosts

Do a quick confidence reboot then go about installing the software onto the VM as usual to make it fit for the task you need it for.

Paravirtualizing vSphere guests

2011-12-01T20:19:00.001+00:00

Although just setting up a virtual machine on a ESX server works well there are ways to improve the VM's performance and reduce some of the CPU overhead required when the VM interacts with the virtual hardware. The two ways I have used are installing paravirtualized drivers for network interface and for the storage controller.

Windows
If you are installing Windows from media or an ISO image:

Edit the VM settings, replace any networks adapters with ones that are of the type "VMXNET 3."
Change the SCSI controller type to "Paravirtual"
On the floppy drive choose "use existing floppy image in datastore:", click browse, then find the relevant image in vmimages\floppies for your version of windows. Dont forget to "select connect at startup"
Hit F6 during the initial installation to add the paravirtual SCSI drivers from the floppy image.
The OS wont recognise the network card until the VMware tools are installed.

If the VM is already set up and running Windows with VMware tools the drivers for the paravirtualised SCSI adapter are also installed with VMware tools but you will need to add a second SCSI adapter of the type "Paravirtual" and boot before Windows will be happy to with the first (boot) SCSI adapter being set to "Paravirtial."

Edit the VM settings, replace any networks adapters with ones that are of the type "VMXNET 3"
Using the Vshpere client Edit the settings of your Windows VM, Click the "Add..." button, choose "Hard Disk", Choose "Create a new Virtual disk" and set a small size like 8 MB.
If your VM already has a SCSI controller you will need to add an additional one. To get another controller you will need to choose a virtual device node which is from the second SCSI controller. I usually just select SCSI(1:0).
Check device manager to ensure the the PVSCSI controller has been detected.

Shutdown the VM, remove the hard drive you have just added and change the remaining SCSI adapter to type "Paravirtual"
Start the machine, if all is well (you will see a blue screen if it isn't) you will have successfully changed windows to use a more efficient way to talk to it's boot drive.

Converting IDE drives to SCSI
I struggled with Windows XP box that I had imported using VMware Converter that only had an IDE drive. I tried deleting the hard drive (I was careful not to delete the underlying files), then I went to add the hard drive and set it to SCSI it would only allow IDE. I overcame this by shutting down the virtual machine, editing the relevant .vmdk file and changing the ddb.adapterType parameter to lsilogic like so:
ddb.adapterType = "lsilogic"
You should now be able to add the hard drive back in specifying the existing .vmdk file and set the SCSI adapter to "Paravirtual"

Debian 6
On a system that is already running and the VMware tools are installed. I ran all updates and it was simply a case of removing the existing network adapter and adding a new one of the type "VMXNET 3" and changing the SCSI adapter to "Paravirtual" and everything worked.

Update: Also check out this post on changing the IO scheduler on linux VMs squeezing more from your Linux VMs

VMware CPUID masks for AES-NI and PCLMULQDQ

2011-11-30T21:06:00.001+00:00

Today I have been messing with with CPUID masks in ESXi/vSphere. Most people in my position wouldn't normally have to worry about this, but I work for a small company so the budget especially these days is necessarily lean so we chose go for vSphere 5 Essentials PLUS which does not include Enhanced vMotion Compatibility (EVC)

The problem I had was there were two servers that were brought at different times and as is often the case had slightly different specification based on the deals the vendors were offering at the time. Basically the Intel processors had slightly different capabilities Xeon E5530 and E5620 The problem came up when I tried to vMotion VMs from one box to the other, Virtual Center complained that the older server lacked the PCLMULQDQ and AES-NI CPU features:

Host CPU is incompatible with the virtual machine's requirments at CPUID level 0x1 register 'ecx'

So working my way through the KB articles on the VMware website I realised that we were not licensed for EVC so KB 1993 applied. It took me a while to get my head around what was going on but you need to tell the VM when it starts up to ignore the flags for those 2 CPU features so that it isn't using them, so it can happily be moved between host servers that do and don't support those CPU instructions.

So down to the nitty gritty, how do you actually disable these features. Connect the vSphere client, to either your vCenter server or the ESX server itself. Right click the VM you wish to make more mobile and choose "Settings", then on the "Options" tab select CPUID Mask then click on the "Advanced..." tab. Scroll down to the "Level 1" section and for "ecx" the mask you need is (presented here for your copy and paste delight):

---- --0- ---- ---- ---- ---- ---- --0-

This was explained in the KB 1993 article but they refer to the "Level" as "a" which is not mentioned in the ESXi 5.0/vSphere 5 configuration screens.

After I changed this setting all the VMs were happy to v|Motion back and forth between the servers.

Exchange 2007 certificate expired

2011-11-15T11:50:00.001+00:00

Once a year Outlook starts to moan about the certificate on the Exchange server. This is because the certificate on the Exchange server was set to expire one year after creation.

To check weather this is actually your problem run the following command at an Exchange PowerShell prompt.

Get-ExchangeCertificate | list

This will show you a list of the certificates used by Exchange, the one we are interested in has IIS mentioned in it's list of services. The NotAfter field will tell you when the certificate expires. For me it shows a time about half an hour ago, so we need a new certificate. To create a new one run the following as an Exchange Server Administrator:

New-ExchangeCertificate

This command will ask if you want to overwrite the existing default SMTP certificate, answer yes to this. It should now display (along with other info) a thumbprint for the newly created certificate. Rather than have to re-type this, take a copy of it and use it to paste into the next command. We need to enable the newly created certificate for the IIS service, we do this with:

Enable-ExchangeCertificate -Thumbprint <thumbprint from previous command> -Service IIS

You can remove old certificates from the exchange certificate store with the following command:

Remove-ExchangeCertificate -Thumbprint <thumbprint of old certificate>

If you need a list of certificates details including their thumbprints re-run:

Get-ExchangeCertificate | list

After running the above I noticed that the certificate is now set to be valid for 5 years instead of 1 year. On further investigation it appears that Exchange 2007 SP2 changed the default for self signed certificates from 1 year to 5 years, woohoo!

Setting an “out of office” message for a user on Exchange 2007

2011-11-08T18:46:00.001+00:00

To set an out of office message for someone else you will need give permission to the user you want to enable or modify the out of office message using the PowerShell extensions for Exchange At the power shell prompt issue the following command:

Add-MailboxPermission <user’s mailbox name> -AccessRights FullAccess -user <administrators account name>

Now you have issued the correct permissions to allow you to log into outlook web access (OWA) as the above administrator account and switch to the above user mailbox and set their out of office messages..

Using Internet explorer, go to https://<FQDN for your exchange server>/owa and login as the administrator's account you specified in the above PowerShell command.
Once you have logged in (may take a while if it hasn’t been used recently due to the way Exchange dominates a server) click the username with admin rights that you logged in with in the top right and type in the name of the user you want to set the out of office reply for. Once logged in as them, select options (top right), then select Out of office autoreply from the list on the left and complete as necessary.