Linux Stories

Raid Tips & Tricks

2011-05-16T11:39:00.000+03:00

Increasing the rebuild speed

Sometimes when you’re quite lazy or bored and don’t like the noticeable amount of free resources on your server, you may like to increase the raid building and resyncing process speed.

# echo 250000 > /proc/sys/dev/raid/speed_limit_max
# echo 250000 > /proc/sys/dev/raid/speed_limit_min

And this would spin up the hards faster to their maximum extent saving almost half of the time!
Defaults were:

# cat /proc/sys/dev/raid/speed_limit_max 200000
# cat /proc/sys/dev/raid/speed_limit_min 1000

And looks like my hards have 100M/s speed.

#hddtemp /dev/sda

/dev/sda: ST3750330AS: 31°C

# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 280 MB in 3.01 seconds = 93.09 MB/sec

Checking and repairing

The larger the hard drive is the greater the probability of having its blocks corrupted. Its better to check for consistency against any bad blocks or md superblocks.

# cat /sys/block/md0/md/sync_action
idle
# echo check > /sys/block/md0/md/sync_action

And if there are any, repair them.

# echo repair > /sys/block/md0/md/sync_action

Setting up RAID alerts to send emails about any errors

Red Hat family distros come with built in service daemon mdmonitor which runs ‘mdadm –monitor’ in a daemon mode.

# mdadm --detail --scan >> /etc/mdadm.conf
# echo "MAILADDR some-email-address" >> /etc/mdadm.conf
# chkconfig mdmonitor on
# service mdmonitor restart

Remember to add “DEVICE partitions” or partitions being components of an md array to top of file /etc/mdadm.conf.

Monitoring all RAID events and changing default email template

Bonding on RedHat 5 / CentOS 5

2011-02-17T10:31:00.001+02:00

In the modprobe.conf file add the following:

alias bond0 bonding

options bond0 miimon=100 mode=1

Be sure to add this before any of the network aliases modes:

mode=0 (Balance Round Robin)

mode=1 (Active backup)

mode=2 (Balance XOR)

mode=3 (Broadcast)

mode=4 (802.3ad)

mode=5 (Balance TLB)

mode=6 (Balance ALB)

In the /etc/sysconfig/network-scripts/ directory create ifcfg-bond0

DEVICE=bond0

IPADDR=

NETMASK=

NETWORK=

BROADCAST=

GATEWAY=

ONBOOT=yes

BOOTPROTO=none

USERCTL=no

Change the ifcfg-eth0 to:

DEVICE=eth0

ONBOOT=yes

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

Change the ifcfg-eth1 to:

DEVICE=eth1

ONBOOT=yes

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

Check the status of the bond.

cat /proc/net/bonding/bond0

You can use multiple bond interface but for that you must load the bonding module as many as you need. Presuming that you want two bond interface you must configure the /etc/modprobe.conf as follow:

alias bond0 bonding

alias bond1 bonding

options bonding max_bonds=2

Also add in /etc/sysconfig/network-script/ifcfg-bond0 & bond1 below line

BONDING_OPTS="miimon=100 mode=1"

Zero Configuration Network (ZEROCONF)

2011-02-14T11:17:00.001+02:00

Most Linux distributions utilise the Zero Configuration Network (ZEROCONF) automation suite. This is an IETF workgroup that planned and coordinated a series of dynamic configuration protocols to allow many operating systems to automatically configure themselves and communicate on a network without the need of DHCP or DNS servers. ZEROCONF utilises the 169.254.0.0/16 network address to autoconfigure using a series of unanswered "ARP" queries and then assumes an address if the queries yield an empty result.

A route to the ZEROCONF network is added to the routing table by the network initscripts.

[bash]# route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.214.64.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1      <-- ZEROCONF default IP route
0.0.0.0         10.214.64.254   0.0.0.0         UG    0      0        0 eth0

ZEROCONF can be turned off by adding the following entry to the "/etc/sysconfig/network" configuration file.

[bash]# vi /etc/sysconfig/network

NOZEROCONF=yes

The value for the "NOZEROCONF" parameter can actually be set to any value, the initscripts only check to determine whether the parameter has zero length. So setting "NOZEROCONF=no" will have the same effect as setting it to "yes". You will need to comment or remove the variable to reactive ZEROCONF.

The networking service will need to be restarted before the changes will take effect.

[bash]# /etc/init.d/network restart

Checking the network routing table again will identify the ZEROCONF route has been disabled and removed from the routing table.

[bash]# route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.214.64.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         10.214.64.254   0.0.0.0         UG    0      0        0 eth0

ZEROCONF is also commonly referred to as IPv4 Link-Local (IPv4LL) and Automatic Private IP Addressing (APIPA).

Replacing A Failed Hard Drive In A Software RAID1 Array

2010-08-23T18:03:00.001+03:00

Replacing A Failed Hard Drive In A Software RAID1 Array

Version 1.0
Author: Falko Timme
Last edited 01/21/2007

This guide shows how to remove a failed hard drive from a Linux RAID1 array (software RAID), and how to add a new hard disk to the RAID1 array without losing data.

I do not issue any guarantee that this will work for you!

1. Preliminary Note

In this example I have two hard drives, /dev/sda and /dev/sdb, with the partitions /dev/sda1 and /dev/sda2 as well as /dev/sdb1 and /dev/sdb2.

/dev/sda1 and /dev/sdb1 make up the RAID1 array /dev/md0.
/dev/sda2 and /dev/sdb2 make up the RAID1 array /dev/md1.

/dev/sda1 + /dev/sdb1 = /dev/md0

/dev/sda2 + /dev/sdb2 = /dev/md1

/dev/sdb has failed, and we want to replace it.

2. How Do I Tell If A Hard Disk Has Failed?

If a disk has failed, you will probably find a lot of error messages in the log files, e.g. /var/log/messages or /var/log/syslog.

You can also run

cat /proc/mdstat

and instead of the string [UU] you will see [U_] if you have a degraded RAID1 array.

3. Removing The Failed Disk

To remove /dev/sdb, we will mark /dev/sdb1 and /dev/sdb2 as failed and remove them from their respective RAID arrays (/dev/md0 and /dev/md1).
First we mark /dev/sdb1 as failed:

mdadm --manage /dev/md0 --fail /dev/sdb1

The output of

cat /proc/mdstat

should look like this:

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
md0 : active raid1 sda1[0] sdb1[2](F)
24418688 blocks [2/1] [U_]

md1 : active raid1 sda2[0] sdb2[1]
24418688 blocks [2/2] [UU]

unused devices:

Then we remove /dev/sdb1 from /dev/md0:

mdadm --manage /dev/md0 --remove /dev/sdb1

The output should be like this:

server1:~# mdadm --manage /dev/md0 --remove /dev/sdb1

mdadm: hot removed /dev/sdb1

And

cat /proc/mdstat

should show this:

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]

md0 : active raid1 sda1[0]
24418688 blocks [2/1] [U_]

md1 : active raid1 sda2[0] sdb2[1]
24418688 blocks [2/2] [UU]

unused devices:

Now we do the same steps again for /dev/sdb2 (which is part of /dev/md1):

mdadm --manage /dev/md1 --fail /dev/sdb2

cat /proc/mdstat

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
md0 : active raid1 sda1[0]
24418688 blocks [2/1] [U_]

md1 : active raid1 sda2[0] sdb2[2](F)
24418688 blocks [2/1] [U_]

unused devices:

mdadm --manage /dev/md1 --remove /dev/sdb2

server1:~# mdadm --manage /dev/md1 --remove /dev/sdb2

mdadm: hot removed /dev/sdb2

cat /proc/mdstat

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]

md0 : active raid1 sda1[0]
24418688 blocks [2/1] [U_]

md1 : active raid1 sda2[0]
24418688 blocks [2/1] [U_]

unused devices:

Then power down the system:

shutdown -h now

and replace the old /dev/sdb hard drive with a new one (it must have at least the same size as the old one - if it's only a few MB smaller than the old one then rebuilding the arrays will fail).

4. Adding The New Hard Disk

After you have changed the hard disk /dev/sdb, boot the system. The first thing we must do now is to create the exact same partitioning as on /dev/sda. We can do this with one simple command:

sfdisk -d /dev/sda | sfdisk /dev/sdb

You can run

fdisk -l

to check if both hard drives have the same partitioning now.

Next we add /dev/sdb1 to /dev/md0 and /dev/sdb2 to /dev/md1:

mdadm --manage /dev/md0 --add /dev/sdb1

server1:~# mdadm --manage /dev/md0 --add /dev/sdb1

mdadm: re-added /dev/sdb1

mdadm --manage /dev/md1 --add /dev/sdb2

server1:~# mdadm --manage /dev/md1 --add /dev/sdb2

mdadm: re-added /dev/sdb2

Now both arays (/dev/md0 and /dev/md1) will be synchronized. Run

cat /proc/mdstat

to see when it's finished.

During the synchronization the output will look like this:

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]

md0 : active raid1 sda1[0] sdb1[1]
      24418688 blocks [2/1] [U_]
      [=>...................]  recovery =  9.9% (2423168/24418688) finish=2.8min speed=127535K/sec

md1 : active raid1 sda2[0] sdb2[1]
      24418688 blocks [2/1] [U_]
      [=>...................]  recovery =  6.4% (1572096/24418688) finish=1.9min speed=196512K/sec

unused devices:

When the synchronization is finished, the output will look like this:

server1:~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]

md0 : active raid1 sda1[0] sdb1[1]
24418688 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
24418688 blocks [2/2] [UU]

unused devices:

That's it, you have successfully replaced /dev/sdb!

Disabling IPv6 in RHEL 5.5 and Fedora 12, 13

2010-04-30T15:01:00.004+03:00

To disable IPv6 in RHEL 5.5:

1. Add the following lines to /etc/modprobe.conf:

alias net-pf-10 off options ipv6 disable=1

2. It is also recommended to change /etc/sysconfig/network to read:

NETWORKING_IPV6=no

3. Finally, be sure to prevent the IPv6 firewall from coming back up on the next reboot by doing a:

/sbin/chkconfig ip6tables off

4. Then reboot.

To disable IPv6 in Fedora 12, 13

1. Create a file under /etc/modprobe.d (e.g. “blacklist-ipv6.conf”) with the following 2 lines, in the order shown:

install ipv6 /bin/true blacklist ipv6

2. Then make sure /etc/sysconfig/network reads as described above.
You’ll also want to turn off ip6tables and use chkconfig to keep it off as described above (chkconfig ip6tables off).

3. Reboot once you’re done and all should be well in the world again.

Squid: WARNING! Your cache is running out of filedescriptors

2010-04-30T12:56:00.003+03:00

So you have a LAN with 50+ users and you set up a nice Squid w3cache as a transparent proxy with 100GB of space reserved for the cache (hdds are so cheap nowadays…). Weeks pass and suddenly you notice that something is messing up your web experience as Firefox suddenly decides to run painfully slow. About 30 minutes wasted on finding the culprit (like changing your DNS servers, clearing browser cache, etc.) until you decide to check the router and then the Squid with its logs. And then you find something fishy:

2007/01/01 17:51:19| WARNING! Your cache is running out of filedescriptors
2007/01/01 17:51:35| WARNING! Your cache is running out of filedescriptors
2007/01/01 17:51:51| WARNING! Your cache is running out of filedescriptors
(...)

I won’t be explaining why this happens. Others have done it before. What I’m going to do is present you with a solution that does not require a complete Squid recompilation/reinstallation procedure.

RedHat/Fedora

/etc/init.d/squid stop

nano /etc/squid/squid.conf
  max_filedesc 4096

nano /etc/init.d/squid
  # add this just after the comments (before any script code)
  ulimit -HSn 4096

/etc/init.d/squid start

Debian

nano /etc/defaults/squid
  SQUID_MAXFD=4096

/etc/init.d/squid restart

Ubuntu

nano /etc/default/squid
  SQUID_MAXFD=4096

/etc/init.d/squid restart

And now watch the /var/log/squid/cache.log for a similar line:

2007/01/01 18:32:27 With 4096 file descriptors available

If it still says 1024 file descriptors available (or similarly low value) you are out of luck (or you’ve just messed something up).

How to block Skype with Squid

2010-04-30T12:19:00.001+03:00

Create access list in /etc/squid/squid.com

# Block Skype
acl skype url_regex ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+
acl connect method CONNECT

Before line "http_access deny all" in squid.conf add below records

# SKYPE BLOCK
http_access deny connect skype all

If you have complete all preview steps reload or restart squid.

How to setup Windows Update through Squid

2010-04-30T12:02:00.007+03:00

Create below access list in /etc/squid/squid.conf

# Microsoft Windows Update
acl WindowsUpdate dstdomain -i "/etc/squid/windowsupdate.txt"
no_cache deny WindowsUpdate

Next step is create file /etc/squid/windowsupdate.txt and add below records

.microsoft.com
.windowsupdate.com
runonce.msn.com

Before line "http_access deny all" in squid.conf add below records

http_access allow WindowsUpdate

If you have complete all previews steps now reload or restart squid service.

Changing the policies on SeLinux

2010-04-19T11:14:00.003+03:00

Extension	File type	Notes
.te	Type enforcement	See http://www.nsa.gov/selinux/papers/policy2/t1.html for details
.fc	File contexts	Default file contexts
.mod	Policy module	Binary of compiled type enforcment policy
.pp	Policy package	The module and optional additonal files (contexts, users, etc)

In order to change the customise the running policy you can either:

(Recommended) Generate a module policy and load the new module. The selinux sources are not required. This can be done via new versions of audit2allow

$ cat /var/log/audit/audit.log | audit2allow -M local
Generating type enforcment file: local.te
Compiling policy: checkmodule -M -m -o local.mod local.te
Building package: semodule_package -o local.pp -m local.mod

******************** IMPORTANT ***********************

In order to load this newly created policy package into the kernel,
you are required to execute

semodule -i local.pp
$ semodule -i local.pp

The policy module will be stored in /etc/selinux/targeted/modules/active/modules and should still work after reboots. It can also be done manually

$ cat /var/log/audit/audit.log | audit2allow -m local > local.te
$ cat local.te
module local 1.0;

require {
role system_r;

class fifo_file {  getattr ioctl };

type cupsd_config_t;
type unconfined_t;
};

allow cupsd_config_t unconfined_t:fifo_file { getattr ioctl };


Building module policy manually

# Compile the module
$ checkmodule -M -m -o local.mod local.te
# Create the package
$ semodule_package -o local.pp -m local.mod
# Load the module into the kernel
$ semodule -i local.pp

Generate a new monolithic policy: You will need to load the selinux sources, e.g. selinux-policy-targeted-sources (or selinux-policy-devel). This allows you to make changes, roll them out and activate the new policy straight away. They will be located in /etc/selinux/targeted/src. The audit2allow program reads the dmesg output and creates exceptions for any avc denied messages it finds. It can be run without any parameters, but it is best if you copy and paste the few entries into a temporary file and run audit2allow over it:
```
cd /etc/selinux/targeted/src/policy/domains/misc/
less /var/log/messages
vi temp
audit2allow -i temp -o local.te
```
The local.te file will be integrated into the policy when you build it:
```
cd /etc/selinux/targeted/src/policy
make
make load
```

The new customised policy will now be active. Try to start the daemon, and again follow an iterative process to eliminate deny messages. Customising the running policy should only be used as a last resort measure, it is not a good idea generally to create a lot of policy exceptions.

Configure VirtualBox USB support to work in Fedora 12

2009-12-03T12:02:00.008+02:00

Configure usbfs to allow vboxusers access to USB devices and add usbfs mount to /etc/fstab

Configure VirtualBox USB

Obtain the GID of the vboxusers group (your GID may different then 501):

# cat /etc/group |grep vboxusers |awk -F : '{print $3}'
501

Create a directory to mount usbfs in:

# mkdir /vbusbf

Add mount to /etc/fstab and change “devgid” value to the GID you obtained from the command above:

# echo none /vbusbf usbfs rw.devgid=501,devmode=664 0 0

Re-mount /etc/fstab mounts:

# mount -a

Add your user to the vboxusers group (change “username” to your username):

# usermod -a -G vboxusers username

Enable USB controller in VirtualBox for VM Host:

Settings > USB

Upgrade From Fedora 11 To Fedora 12.

2009-11-27T15:19:00.006+02:00

1. First we must upgrade the rpm package:

yum update rpm

2. Then we install the latest updates:

yum -y update

3. Next we clean the yum cache:

yum clean all

4. If you notice that a new kernel got installed during yum -y update, you should reboot the system now:

reboot

5. After the reboot, log in as root again, either directly or with the help of

su -

6. Now we come to the upgrade process. We can do this with preupgrade (preupgrade will also take care of your RPMFusion packages).

Install preupgrade...

yum install preupgrade

... and call it like this:

preupgrade

7. The preupgrade wizard will then start on your desktop. Select Fedora 12 (Constantine). Afterwards the system is being prepared for the upgrade.

At the end, click on the Reboot Now button.

Restore spam email amavis with amavisd-release

2009-11-06T12:55:00.004+02:00

To restore email marked as spam by amavis we going to use the command amavisd-release

Check the log file for the serial of the mail, eg:

amavis[29297]: (29297-01-6) Blocked SPAM, … <...> -> <...>,
quarantine: spam/U/UM3XM3XDbN52.gz,
^^^^^^^^^^^^^^^^^^^^^^
Message-ID:<...>, mail_id: UM3XM3XDbN52, Hits: 13.365,

and to recover that specific message run the command:

amavisd-release spam-UM3XM3XDbN52.gz mail@example.com

where mail@example.com is the recipient address that will receive the recovered email

Check disk space

2009-10-27T15:21:00.010+02:00

#!/bin/sh
#
# Put below line your email
#
#ADMIN= email@address
#
STATUS=90 # Set here Alert level
STATUS_F=99 # Set here Warning Level
#
#-----------------------------------------
#
# Please uncomment below line for debug
#set -x
#-----------------------------------------
#
# Do not change anything from below lines
#
#=========================================
#
/bin/df -HP | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )
partition=$(echo $output | awk '{ print $2 }' )

if [ $usep -ge $STATUS_F ]; then
#
echo "Running disk space is FULL \"$partition ($usep%)\" on $(hostname) as on $(date +%d-%m-%Y)" |
mail -s "!!!WARNING: Disk space is FULL !!! $usep%" $ADMIN
#
else
if [ $usep -ge $STATUS ]; then
#
echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date +%d-%m-%Y)" |
mail -s "Alert: Almost out of disk space $usep%" $ADMIN
#
fi
fi
done