Some tips en thoughts about IT

Use hpacucli to find a failed physical disk drive

2012-02-17T15:10:00.001+01:00

We use a couple of HP MDS disk cabinet systems to provide massive storage. Sometimes a drive fails and it comes in handy to have this specific drive to blink it's led. Much easier to eyeball the drive in question.

First we find out which controller to use. Second we filter out the failed drive and third we enable the led for this drive to blink.

$ hpacucli "ctrl all show"
Smart Array P700m in Slot 2               (sn: P60820E9SWC05U)

$ hpacucli "ctrl slot=2 pd all show status" | grep -i --color fail
   physicaldrive 54:1:14 (port 54:box 1:bay 14, 1 TB): Failed

$ hpacucli "ctrl slot=2 physicaldrive 54:1:14 modify led=on"

GnuTLS error -12: A TLS fatal alert has been received

2012-02-06T12:54:00.003+01:00

We use several vsftp based FTP service setups hosted on Redhat EL 6 servers with TLS. This week some FileZilla clients where not able to connect. After the FTP welcome message, this error popped up : GnuTLS error -12: A TLS fatal alert has been received.
Common factor was the FileZilla version in use: 3.5.3 . Prior versions still worked like expected. It seems FileZilla changed the cipher suite they support and which default vsftp offers. Seen there is no option so far in FileZilla to permit weaker cipher suites, we had yo see if it's possible to modify the vsftp setup in order to offer a stronger cipher suite.
Adding option 'ssl_ciphers=HIGH' to vsftpd.conf and doing a service restart/reload solved to problem:

$ vim /etc/vsftpd/vsftpd.conf
ssl_ciphers=HIGH
$ service vsftpd reload
Shutting down vsftpd:                                      [  OK  ]
Starting vsftpd for vsftpd:                                [  OK  ]
Starting vsftpd for vsftpd_int:                            [  OK  ]

Update 1) This is the config part I use to enable SSL :

ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=NO
#force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
rsa_cert_file=/etc/vsftpd/vsftpd.pem
require_ssl_reuse=NO
#require_ssl_reuse=YES
ssl_ciphers=HIGH

ipcalc and Fedora

2012-02-03T10:20:00.002+01:00

After switching from Ubuntu to Fedora, one small annoyance was the different output of the ipcalc utility. Ubuntu's Ipcalc gives all info you need (and more) :

]$ ipcalculator 10.20.30.0/23
Address:   10.20.30.0           00001010.00010100.0001111 0.00000000
Netmask:   255.255.254.0 = 23   11111111.11111111.1111111 0.00000000
Wildcard:  0.0.1.255            00000000.00000000.0000000 1.11111111
=>
Network:   10.20.30.0/23        00001010.00010100.0001111 0.00000000
HostMin:   10.20.30.1           00001010.00010100.0001111 0.00000001
HostMax:   10.20.31.254         00001010.00010100.0001111 1.11111110
Broadcast: 10.20.31.255         00001010.00010100.0001111 1.11111111
Hosts/Net: 510                   Class A, Private Internet

Well under Fedora:

]$ ipcalc 10.20.30.0/23
Usage: ipcalc [OPTION...]
  -c, --check         Validate IP address for specified address family
  -4, --ipv4          IPv4 address family (default)
  -6, --ipv6          IPv6 address family
  -b, --broadcast     Display calculated broadcast address
  -h, --hostname      Show hostname determined via DNS
  -m, --netmask       Display default netmask for IP (class A, B, or C)
  -n, --network       Display network address
  -p, --prefix        Display network prefix
  -s, --silent        Don't ever display error messages

Help options:
  -?, --help          Show this help message
  --usage             Display brief usage message

Ok, something different with options?

$ ipcalc -4bmnp 10.20.30.0/23
NETMASK=255.255.254.0
PREFIX=23
BROADCAST=10.20.31.255
NETWORK=10.20.30.0

I'm not impressed to say at the least, but ipcalculator comes to the rescue:

]$ ipcalculator 137.224.54.0/23
Address:   137.224.54.0         10001001.11100000.0011011 0.00000000
Netmask:   255.255.254.0 = 23   11111111.11111111.1111111 0.00000000
Wildcard:  0.0.1.255            00000000.00000000.0000000 1.11111111
=>
Network:   137.224.54.0/23      10001001.11100000.0011011 0.00000000
HostMin:   137.224.54.1         10001001.11100000.0011011 0.00000001
HostMax:   137.224.55.254       10001001.11100000.0011011 1.11111110
Broadcast: 137.224.55.255       10001001.11100000.0011011 1.11111111
Hosts/Net: 510

So they just switched names, sigh nice all those different distro's and their interpretation how things should be ;-(

Linux cifs mount and umask

2012-01-16T16:33:00.005+01:00

The master node on our HPC environment among others mounts a share from a Redhat EL 6.2 server. In order to make it possible for the Windows 2008 nodes and Linux clients to use this share, I had to add following settings to the smb.conf to have the right create mask for new files and directories.

force create mode = 7777
force directory mode = 7777

Even when using special mount options on the client, still the Linux umask setting overrides the mount and smb.conf mask settings. But setting the force directive finally preserved the desired mask.

Fix Dual Monitors in GNOME 3 and some tips

2012-01-10T12:32:00.003+01:00

After enabling Gnome 3 on my dual monitor system, I did not like it's behavior when switching work spaces. One screen always showed the main screen while the other indeed switched to an other work space. Well Greg Cordts blog (thanks!) helped me out on this one:
http://gregcor.com/2011/05/07/fix-dual-monitors-in-gnome-3-aka-my-workspaces-are-broken/

Download gconf-editor – this is probably available in your distribution’s package manager.
Start gconf-editor – the “Run Command” menu is still available in GNOME 3 by pressing Alt-F2.
In the tree on the left, navigate to /desktop/gnome/shell/windows
UNCHECK the box next to workspaces_only_on_primary

You’ll need to log out or restart to see the changes, and some commenters in the above post seem to note some small issues with the setup, but all-in-all this has been completely functional for me; and best of all, I can use both monitors again!

Further more, this link came in handy to find out some tricks using Gnome 3:
https://live.gnome.org/GnomeShell/CheatSheet

VIM auto # and slow

2012-01-05T13:08:00.003+01:00

On a new RedHat EL 6.2 server with a big, busy nfs file system, I ran into some annoying issues when using vim. First vim was really slow on opening files and second if I pasted text it got automatically prefixed with a hash sign eg #.
Adding following three options to .vimrc helped me out:

set swapsync=
set nofsync
set paste

The sync options stop vim from forced flushing the file, the paste option stops the intelligence when pasting text.

MacBook Ubuntu or Fedora?

2011-12-31T15:19:00.000+01:00

Well that one was very simple:
Fedora 16 booted a bitt slow from the live cd, but installed flawless until the first boot: nothing. After reading this
http://fedora64.org/Members/jmontleon/installing-fedora-16-on-macbooks-using-grub2-efi
I turned in despear tot Ubuntu, and Ubuntu 11.10 installed and booted without any extra instructions or trouble. Also bluetooth is configured out of the box.

If you really want to make it interesting, try a dual boot system when using Filevault 2 ;-) Thanks to "knutser" on InsanelyMAC I got it working : http://www.insanelymac.com/forum/index.php?showtopic=264528

Maybe one day I'm feeling brave or geek enough to give Fedora a new shot ;-) But it stays a bit disappointing to find out Fedora 16 is not capable of configuring hardware which is already around for some years.

OS X hot corners not working

2011-12-24T15:10:00.000+01:00

One big annoying problem I have since upgrading to OS X Lion (10.7.2 at this very moment), is a problem with hot corners. I've assigned one corner to the screen saver, but no matter how deep I push the mouse pointer into that corner, it just doesn't kick in.
It seems I am not the only one with this problem seen the hits on this subject on Google and Bing:

https://discussions.apple.com/thread/3203909?start=30&tstart=0

I did not restart in rescue mode, but just openend the "Disk Utility" and choose to "Repair Disk Permissions" on the OS X volume. So far so good, the hot corner is back but I'm not so confident it stays that way seen the discussion on this subject.

Simple SSH logon message script

2011-12-21T16:02:00.002+01:00

Ubuntu has a nice feature to create rather dynamic motd files via landscape tooling. I could not find an alternative for Redhat, so in the end build a simple bash script to show some usefull messages during login via ssh.

All you have to do is to create a file named /etc/sshd/sshrc with some scripting of choice.

#!/bin/bash 
ACCOUNT=`whoami `
PROCCOUNT=`ps -u $ACCOUNT -U $ACCOUNT u | wc -l`
PROCCOUNT=`expr $PROCCOUNT - 1`

echo "
 System and User Information as of `date`

 Hostname = `hostname`   Address = 10.0.0.1
 Kernel   = `uname -r`   Memory = `cat /proc/meminfo | grep MemTotal | awk {'print $2'}` kB
 Uptime   = `uptime | sed 's/.*up ([^,]*), .*/1/'`


 Username = $ACCOUNT
 Sessions = `who | grep $USER | wc -l` Processes = $PROCCOUNT of `ulimit -u` max


 Helpful Information 

 * Documentation : https://server/wiki

"
echo -e "\033[0;31m Maintenance Information.\033[0m"
echo -e " `cat /etc/motd-maint`\n\n"

At the end of the script. "Maintenance Information" is printed in red and any info is read from the file /etc/motd-maint

Oops now find out how to skip this script if sftp is used because sftp dies with an error.

Bash shell script one character input

2011-12-14T17:44:00.001+01:00

Today I had to move some files from a postfix queue, but I wanted to have a choice whether to move the file or not based on the content. So a little bash shell scripting based on the read function:

for i in `ls -1` ; do postcat $i | more ; read -n 1 -p "Move $i (y/n)?" ; [ "$REPLY" == "y" ] && mv $i /newdir; done

RedHat 5/6 source packages

2011-12-06T13:52:00.001+01:00

As of RedHat 5 and later on 6, it's a bit different then before to install source packages.
This is the official RedHat knowledge base instruction:

The source code packages and download links have been provided by Red Hat, but they are outside
the scope of our posted Service Level Agreements (https://www.redhat.com/support/service/sla/)
and support procedures.

You can use the yumdownloader command with the --source option to download source RPM packages.
In order to use the yumdownloader utility, make sure that you have the yum-utils package installed on the machine.

# rpm -q yum-utils

If you do not have the yum-utils package installed, you can use the yum utility to install the package:

# yum install yum-utils

The next step is to configure a yum repository to download source packages.
To create a new repository, open a new file /etc/yum.repos.d/rhel-src.repo and add the following configuration directive:

[rhel-src]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=ftp://ftp.redhat.com/pub/redhat/linux/enterprise/$releasever/en/os/SRPMS/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Now download the source package(s) using yumdownloader:

# yumdownloader --source openldap

By default, the source package(s) is stored under the current working directory. If you want to specify another destination directory for the download, use the --destdir option:

# yumdownloader --source --destdir /tmp openldap

Table 'mysql.servers' doesn't exist

2011-12-05T09:16:00.001+01:00

When I moved a Mysql database setup from a CentOS 5.7 to a RedHat 6.1 server, I thought it would go smooth because it's a move from version 5.0.77 to 5.1.52. It went smooth except for when I tried to flush privileges:

$ tail /var/lib/mysql/server.err

" Table 'mysql.servers' doesn't exist"

Found some suggestions via google :
http://linuxhostingsupport.net/blog/problem-adding-a-database-user-in-plesk-or-restarting-mysql-table-mysql-servers-doesnt-exist

Neither of them worked, but it put me on the right track:

$ mysql --database=mysql < /usr/share/mysql/mysql_fix_privilege_tables.sql

Failed to allocate internet-domain X11 display socket

2011-12-01T11:19:00.001+01:00

Today on a Redhat 5.7 x64 server (fully patched 2011-12-01), I was not able to start a X-Forwarding session. A simple test with xeyes failed:

$ xeyes

Error: Can't open display:

Looking at the /var/log/secure log gave a hint:

sshd[3698]: error: Failed to allocate internet-domain X11 display socket.

Via google found a solution on a Debian bug blog. I had to add an extra line to the sshd config file:

$ vim /etc/ssh/sshd_config
AddressFamily inet
$ service sshd reload

It seems to be related to disabling ipv6 (which I did) and some ipv4/ipv6 mess:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=422327

Ubuntu 11.10 and HP Data Protector

2011-11-24T13:28:00.001+01:00

After installing a brand new Ubuntu 11.10 oneiric boot from SAN server on a ProLiant BL460c G7, all was running well apart from a bit flaky HP system management homepage.
Even installation of HP-DP 6 client software went smooth until I did a check "Check Installation..." from HP Data Protector manager:

[Critical]  <server>: Unsupported architecture/OS type
 (Linux servername 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux) Skipping client!

After digging the HP-DP logs on the Ubuntu server, I decided to replace /bin/uname with a shell wrapper script to trick HP-DP:


$ mv /bin/uname /bin/uname.org

$ vim /bin/uname

#!/bin/bash

# Original uname moved to /bin/uname.org 

case $1 in

"-a"*)

   echo "Linux servername 2.6.35-30-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux"

   ;;

"-A"*)

   /bin/uname.org -a

   ;;

*)

   /bin/uname.org $1 $2 $3

 ;;

esac

$ chmod +x /bin/uname

$ chattr +i /bin/uname

Now when running a check "Check Installation..." it reports:

[Normal]  Client <server> (Linux 2.6.35-30-server) OK.

So reporting a Ubuntu 10.10 maverick kernel version instead of the brand new 3 kernel did the trick.

I'm now also able to patch this client via the installation server. Off course we did a lot of backup and restore testing, and it all seems functioning very well.

Tshark ssl snif with private key

2011-11-20T15:31:00.001+01:00

In case you need to monitor SSL traffic on your own network and you have the private key, this is a small howto. So no fancy arp spoofing of MIT ;-)

I need to monitor an IIS website, so I've exported the certificate with private key and no password to a PFX/PKCS#12 file. Now I need to convert it to PEM to use it with tshark:

$ openssl pkcs12 -in w.pfx -out key.pem -nodes -nocerts

This only extracts the private key from w.pfx to the file key.pem .
And now start tshark:

$ tshark -i eth1 -o ssl.keys_list:192.168.1.1,443,http,"key.pem" -o ssl.debug_file:"/tmp/ssldebug.txt" -V -R http

Potential 0-day on Bind 9

2011-11-18T11:21:00.001+01:00

Quote from the Internet Strom Centre:
"Internet System Consortium has published an alert earlier as they are investigating a potential vulnerability on Bind 9. There are reports of the DNS server software crashing while generating log entry - "INSIST(! dns_rdataset_isassociated(sigrdataset))" The details on this is rather limited at this point, aside from DoS effect, it's unknown whether code execution is possible at this point. "
Today Redhat released errata RHSA-2011:1458 - Security Advisory, so quickly patched related servers. Ubuntu also released updated bind9 packages.

Vpshere 4 nfs datastore slow

2011-11-14T12:53:00.001+01:00

When trying out our first Vsphere nfs datastore, the data transfer rates were very poor: around 6-8 MB/s. Well that's a bit abnormal when the datastore is hosted on a HP X9000 storage environment. But finally we found the most optimum export options :

$ cat /etc/exports
/fs01/nfs01/vsphere/vmdata_001 10.1.1.1(rw,no_root_squash,sync,crossmnt,subtree_check,fsid=2)

Now data transfer rates are at a average of 35MB/s, still not blazing fast but much better. So still some tweaking todo.

iOS 5.0.1 via software update

2011-11-10T22:34:00.001+01:00

Just updated my iPhone 4 in just over 12 minutes. Via software upate over wireless, great! Now let's see if battery life has indeed improved with this version.

Ubuntu detailed kernel version

2011-11-09T14:36:00.001+01:00

Our Ubuntu 10.10 production servers are running a 2.6.35-30-server kernel, and recently after updating the kernel it still showed the same kernel version when running uname -a :

$ uname -a
Linux myserver 2.6.35-30-server #61-Ubuntu SMP Tue Oct 11 18:09:44 UTC 2011 x86_64 GNU/Linux

But having a look at the kernel package revealed some more info. Before upgrading:

$ dpkg -l `dpkg -S /boot/vmlinuz-2.6.35-30-server | cut -d":" -f1`

||/ Name                                        Version                                     Description
+++-===========================================- 
ii  linux-image-2.6.35-30-server                2.6.35-30.54                                Linux kernel image for version 2.6.35 on x86_64
t

And after upgrading the kernel package:

$ dpkg -l `dpkg -S /boot/vmlinuz-2.6.35-30-server | cut -d":" -f1`

||/ Name                                        Version                                     Description
+++-===========================================-
 ii  linux-image-2.6.35-30-server                2.6.35-30.61                                Linux kernel image for version 2.6.35 on x86_64

So a new revision, from 54 to 61 for kernel 2.6.35-30

Lock a Linux user

2011-11-07T14:41:00.000+01:00

The Linux passwd command seems to draw a distinction. It has -l and -u to lock and unlock an account. It does this by adding a !! to the front of the encrypted password to lock, and removing that !! to unlock. There is a -d for disable and it replaces the encrypted password with just a !! and obviously this is not so easily reversible.

To check all local Linux accounts :

$ sudo for i in `cat /etc/passwd | cut -d":" -f1` ;do passwd -S $i ; done


account1 PS 2011-05-17 0 99999 7 -1 (Password set, MD5 crypt.)


account2 LK 2011-09-02 0 99999 7 -1 (Password locked.)

Remove TuneUP under Lion 10.7.2

2011-11-05T14:19:00.002+01:00

Also fed up with TuneUp when opening iTunes? On an Apple support forum found the following tip to remove it safe. Open a Terminal an run the following applet from you home directory eg /Users/your account :

$ Library/Application Support/TuneUp_Support/TuneUp_Uninstaller.app/Contents/MacOS/applet

iOS 5.0.1 beta and iPad 1 multitasking gestures back?

2011-11-05T14:03:00.004+01:00

I did not upgrade our iPad 1 to iOS 5 because iOS5 only supports multitasking gestures on the iPad 2. Today had a look on the iOS 5.0.1 beta for my iPhone 4, and much to my surprise it seems Apple is planning to put those gestures back in iOS 5 also for the iPad 1?!

Read Me Before Downloading Hide details

iOS 5.0.1 beta contains improvements and other bug fixes including:

Fixes bugs affecting battery life
Adds Multitasking Gestures for original iPad
Resolves bugs with Documents in the Cloud
Improves voice recognition for Australian users using dictation
Contains security improvements

Nmap 5.61 : show only down hosts

2011-11-02T15:33:00.004+01:00

To find (possible) free ip addresses in a subnet:

$ nmap -sn -oG - 10.11.12.0/24 -v | grep Down

Ubuntu 10.10 Server x64 Boot from SAN

2011-11-02T10:55:00.002+01:00

After contacting Canonical EMEA about installing Ubuntu on a HP-Proliant BL460-G7 blade, it was clear that it is now supported to have a system disk from SAN (HP EVA 8400). A few years ago I made an attempt with Ubuntu server 8.04 and succeeded, but only after a lot of script hacking. So that did not feel safe enough to implement on production servers.

August 2011 Conanical gave the green light with regards to a G6 model with ubuntu 10.04 TLS server x64. It turned out that when using Ubuntu 10.10 server x64 on the latest G7 model, the NIC's where detected automatic on installation, beside our customer prefers 10.10 over 10.04. So that's why I decided to have it a go with 10.10 server.

It's running for some months now, and it's rock stable, survives path failures but not a total hick up like all paths failing for a few seconds.

So this is not an official Ubuntu boot from SAN guide, but just the way to success I followed. I guess a lot of it can be done via an installation script, but that's up to you.

At install time I was not able to activate multipath correct with the mpath feature. So I just installed the OS on the first path visible out of 4 paths (eg /dev/sda , b , c and d). First I setup a very basic system (via HP ILO) with ssh server so the rest of the configuration can be done via a ssh session which makes it easy to use copy and paste.

Follow the next steps to activate the multipath configuration:

- Reboot after base installation of Ubuntu
- Logon via ssh with the first account created during setup

- Apply latest patches and install some useful tools:

$ sudo su -

$ apt-get autoclean ; apt-get update ; apt-get dist-upgrade

$ apt-get install aptitude ethtool vim nfs-client cifs-utils sysv-rc-conf \ chkconfig sysstat iotop -y

- Reboot the server to activate the new kernel.
- Setup multipath (HP EVA-8400 SAN) :


$ apt-get install multipath-tools* -y


$ vim /etc/multipath.conf


defaults {


        udev_dir                      /dev


        polling_interval              5


        selector          ;        "round-robin 0"


        path_grouping_policy    failover


# Redhat       getuid_callout          "/lib/udev/scsi_id -g -u -s"


        getuid_callout         "/lib/udev/scsi_id -g -u -d /dev/%n"


        prio_callout                    none


# Redhat      path_checker            readsector0


        path_checker          directio


        rr_min_io              1000


        rr_weight              uniform


        failback               manual


        no_path_retry         fail


        user_friendly_names     yes


        #user_friendly_names   no


        bindings_file       "/etc/multipath_bindings"


}


blacklist {


       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"


       devnode "^hd[a-z]"


       devnode "^cciss!c[0-9]d[0-9]*"


}


devices {


 device {


  vendor “HP”


  product "HSV2[01]0|HSV300|HSV4[05]0"


# rh  getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"


  getuid_callout          "/lib/udev/scsi_id -g -u -d /dev/%n"


  prio_callout "/sbin/mpath_prio_alua /dev/%n"


  hardware_handler "0"


  path_selector "round-robin 0"


  path_grouping_policy group_by_prio


  failback immediate


  rr_weight uniform


  rr_min_io 100


  no_path_retry 18


  path_checker tur


 }


}

- Setup the qlogic hba driver:

$ vim /etc/modprobe.d/qla2xxx.confalias scsi_hostadapter cciss

alias scsi_hostadapter1 qla2xxx

alias scsi_hostadapter2 usb-storage

options qla2xxx ql2xmaxqdepth=16 ql2xloginretrycount=5 qlport_down_retry=1

options lpfc lpfc_lun_queue_depth=16 lpfc_nodev_tmo=14 lpfc_discovery_threads=32

- Modify /boot mount point in /etc/fstab: replace UID by /dev/mapper/mpath0-part1

/dev/mapper/mpath0-part1 /boot       ext2    defaults   0 2

#UUID=0b398b5b-a5fa-4cf2-a174-c554197175e7 /boot   ext2  defaults  0 2

- Have some more on screen output during boot time:

$ vim /etc/defaults/grub

GRUB_CMDLINE_LINUX_DEFAULT=""

#GRUB_CMDLINE_LINUX_DEFAULT="quiet"

$ update-grub

- lvm by default scans all san paths so sees in our case duplicate volume group names. Adjust the lvm.conf filter option :

$ vim /etc/lvm/lvm.conf

52:    # By default we accept every block device:

    # filter = [ "a/.*/" ]

    #  LVM on multipath so skip 'real' scsi devices detection

    filter = [ "r/sd.*/", "r:disk/by.*:", "a/.*/" ]

$ rm /etc/lvm/cache/.cache

- And final create a new kernel image and reboot the system to have all changes in effect:

$ dpkg-reconfigure multipath-tools-boot

$ reboot

Chrome Extension: FlashBlock

2011-10-31T18:17:00.001+01:00

This one comes in handy to have a choice to play flash content or not:
http://www.chromeextensions.org/appearance-functioning/flashblock/

When opening a page which contains flash content, FlashBlock prevents the flash content from executing.
It shows instead the Flash logo in a button.

You can click on the button to start playing ... or not ;-)