Well, look no further! (real soon now)

Under construction

• Self hosted (Fedora 16)
• 2 node KVM/RH Cluster (for hosting KVM Guests/VMs)
• 2 KVM instances running Fedora 15 (soon to be upgraded)
• Keepalived (IPVS) to ensure seamless connectivity from external hosts
• 2nd Cluster between the two KVM guest instances
• Guests/VMs running IMAPD (Dovecot 2) based webmail instances (Multi-instance or “Active/Active”)
• Postfix 2.x based SMTP instances (Multi-instance or “Active/Active”)
• Replicated Master-Master MySQL instances (Multi-instance or “Active/Active”)
• Apache 2.x instance in each KVM (Multi-instance or “Active/Active”)
• Roundcube and Horde as webmail options for users

I also happen to have a few Apple products deployed in my environment. Basically, I got tired of tinkering with Linux for things like music/media/etc and just wanted a plug ‘n play ecosystem where the components “just worked” together with no interaction from me.

I also have a home automation system (Vera2 from www.micasaverde.com) which I have integrated with the Google Calendar service to wake me up in the morning (the lights in the bedroom are gradually made brighter based on a trigger – alert – from Google Calendar). So, I obviously need to be able to edit my calendar alerts based on when I need/want to get up in the morning. And I don’t want to have to bring a laptop to bed in order to do so.

Also, since the wake-up alerter isn’t really an appointment I’d like to keep in my regular calendar, I created a secondary private Google Calendar instance for this purpose.

Then I attempted to make my iPhone access this secondary Google Calendar…

It took a little searching (there are a lot of incorrect, old, articles on this subject it seems), but I managed to find the following link which helps me select which of the calendars should be visible on an Apple iPad or iPhone device.

A quick request on the #selinux-fedora IRC channel on freenode.net clarified the whole thing, but I thought it worth posting so I won’t forget.

You may have realized, if you’ve read any of the other posts in this blog, that I have a multi-instance Apache Server configuration that shares a single GFS2 file system between two OS instances (they’re actually VMs) and I hosts a few different WordPress blogs and a Photo Gallery using this configuration. The pictures and source files for the blogs and gallery are mounted at a non-standard location (non-standard in a Fedora/httpd context) and since I recently re-enabled SELinux in my environment and set it to “Enforcing”, I’ve had to make some additions to the file context (fcontext) portions of the SELinux policy so as to ensure that any “.autorelabel” and # restorecon commands will set the wordpress and gallery source files to the correct file context (fcontext).

Basically, I’ve been creating file context regular expressions that align with my web server layout using the # semanage fcontext -a -t <context> “<file path regex>” command. At the time I created the contexts, I never noticed the fact that I was using two different contexts (they both worked, so it wasn’t obvious to me that I’d done this);

httpd_sys_rw_context_t

httpd_sys_content_rw_t

But, when I ran ” #restorecon -RFv <path to html root>” from both of my OS instances, I noticed that they were both labeling the same (shared) directories with the two different fcontexts. And that made me wonder…  What on earth was/is the difference between the two (and, secondarily, why do they both exist)?

To answer the first question first; “What on earth is the difference between the httpd_sys_rw_content_t fcontext and the httpd_sys_content_rw_t fcontext?”;

None! Zip! Nada! They’re aliases of each other and have the exact same meaning. Thus, if you have a directory where you wish to let the httpd (Apache) server write to – for instance an “upload” folder – you can set it to either context and SELinux will let the HTTPD Daemon write to the location as long as the remaining UNIX permissions are correct.

The 2nd question is harder to answer (or, in other words; I don’t have an authoritative answer to that question), but I can speculate that it’s a compatibility “play” to not break previously set fcontexts (“previously” being “fcontexts set by other selinux-policy versions and/or previous versions of the Fedora distribution itself”).

Two of the 1TB drives are mirrored in my iSCSI RAID. The iSCSI RAID array exports this volume to my backup master host and I use this iSCSI volume as 1/2 of a MD based Raid-1 mirror that hosts the backup spool for my environment. Every morning, I’ve got a script to stop the backup software, sync the drive and unmount the ext4 file system hosting the spool before splitting the MD Raid-1 mirror and disconnecting the eSATA connected 3d drive. I can then safely remove the drive, put it in protective packaging and bring it to my office where I swap this 3rd drive with a 4th that was stored in my office overnight.

At night, the reverse process is scripted to connect the “office-drive” to the backup volume. Scanning the bus for the eSATA device triggers a udev action to join the office drive to the /dev/md device and thus start the resync of the two volumes. This is fully non-disruptive, if you ignore the performance ‘hit’, and the goal is that before the morning “split” job starts running, the mirror will be merged and a new fresh copy can go to the office.

I recognize the fact that the /usr/local/bin/{add,drop}-device scripts are very quick & dirty and can stand to be cleaned up quite a bit. For instance, the retry logic could be pulled out in bash functions and called in a loop with a configurable number of retries, etc, etc, etc. But, the point is it was “quick and dirty”.

In /etc/mdadm.conf

MAILADDR root@sjolshagen.net
DEVICE /dev/raid-mirror0-p1
DEVICE /dev/raid-backup4-p1
ARRAY /dev/md/127 metadata=1.2 UUID=56a8ce1e:294930a4:72073552:3f6dd2a7 name=virt1-backup.sjolshagen.net:127

To start the host mirror (RAID-1) containing the iSCSI volume & the eSATA drive (/dev/md127 in my case) I’ve included the following in /etc/rc.local and disabled the BackupPC init service (# chkconfig backuppc stop ):

/sbin/mdadm /dev/md127 --assemble --force --auto=yes --bitmap=/var/lib/md-bitmaps/md127.bitmap /dev/raid-mirror0-p1
mount /dev/md127 /var/lib/BackupPC
res=$?
if [ ${res} -ne 0 ]
then
     echo "Mount of /dev/md127 to /var/lib/Backup failed!"
     exit 1
fi
service backuppc start

Additionally, I’ve created the following udev rules to add the eSATA drive to the mirror when the drive(s) are detected by the kernel:

# Saved in /etc/udev/rules.d/10-local.rules
#
KERNEL=="*[0-9]", IMPORT{parent}=="ID_*"

# ESATA drives (using their serial numbers/IDs)
KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="WD-WMATV3081234", SYMLINK+="raid-backup0-p%n", RUN+="/usr/local/bin/attach_raid.sh /dev/md127 /dev/raid-backup0-p%n"
KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="WD-WMATV3131170", SYMLINK+="raid-backup1-p%n", RUN+="/usr/local/bin/attach_raid.sh /dev/md127 /dev/raid-backup1-p%n"

# For the iSCSI volume
KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="WD-WCATR0362494", SYMLINK+="raid-iscsi-mirror-p%n", RUN+="/usr/local/bin/attach_raid.sh /dev/md127 /dev/raid-iscsi-mirror-p%n"

The /usr/local/bin/add-device script that runs multiple times on weekday nights:

#!/bin/bash
#
DATE=$(date +%c)
echo "**************** ${DATE} *******************"
/bin/ls /dev/raid-backup* >>/dev/null 2>&1
res=$?

if [ ${res} -ne 0 ]
then
	echo "The drive is not connected yet"
	echo "- - -" > /sys/class/scsi_host/host6/scan
	echo "- - -" > /sys/class/scsi_host/host7/scan
	exit 0
else
	echo "The drive is already connected - no need to look (scan) for it"
        exit 0
fi

The /usr/local/bin/drop-device script that runs every weekday morning:

#!/bin/bash
#
DEV=$(ls -l /dev/raid-backup?-p1 | awk '{ print $11 }' | cut -c 1-3)
MD_DEV=$(cat /proc/mdstat | grep md |grep -v bitmap | awk '{ print $1 }')

DATE=$(date +%c)
MAIL_ADM="root@host.com"

# Add "header"
echo "*********** ${DATE} ***********"

/bin/ls /dev/raid-backup* >>/dev/null 2>&1
res=$?

if [ ${res} -ne 0 ]
then
     echo "Drive not connected - nothing to do"
     echo "Exiting"
     exit
fi

echo "---------------------------------------"
/bin/cat /proc/mdstat
echo "---------------------------------------"

RAID_STATE=$(/sbin/mdadm --misc -t --detail /dev/${MD_DEV} >/dev/null)
res=$?

if [ ${res} -ne 0 ]
then
        echo "The array is not clean & rebuilt"
        echo "Rescheduling and will try again in 15 minutes"
        /usr/bin/at -f /usr/local/bin/drop-device now + 15 minutes >> /var/log/drop-device.log 2>&1
        exit 1
fi

/sbin/service backuppc stop
res=$?

if [ ${res} -ne 0 ]
then
        echo "Unable to stop BackupPC service - Exiting"
        exit 1
fi

sync ; sync ; sync
umount /var/lib/BackupPC
res=$?

if [ ${res} -ne 0 ]
then
        echo "Failed to unmount ${BACKUP_LOC}"
        echo "Restarting service and quitting"
        service backuppc start
        exit 1
fi

/sbin/mdadm /dev/${MD_DEV} --fail /dev/${DEV}1
res=$?

if [ ${res} -ne 0 ]
then
        echo "Failed to remove /dev/${DEV}1 from /dev/${MD_DEV}"
        sleep 5
        echo "Retrying...."
        /sbin/mdadm /dev/${MD_DEV} --fail /dev/${DEV}1
        res=$?

        if [ ${res} -ne 0 ]
        then
                echo "Removal of /dev/${DEV}1 from /dev/${MD_DEV} failed" | mail -s "Failed mirror operation" ${MAIL_ADM}
                exit 1
        fi
fi

sleep 15
/sbin/mdadm /dev/${MD_DEV} --remove /dev/${DEV}1
res=$?

if [ ${res} -ne 0 ]
then
        echo "Failed to hotplug /dev/${DEV}1 from /dev/${MD_DEV}"
        sleep 5
        echo "Retrying...."
        /sbin/mdadm /dev/${MD_DEV} --fail /dev/${DEV}1
        res=$?

        if [ ${res} -ne 0 ]
        then
                echo "Still no luck on the hotplug operation"
                echo "giving up and quitting"
                echo "Unplug of /dev/${DEV}1 from /dev/${MD_DEV} failed" | mail -s "Failed mirror operation" ${MAIL_ADM}
                exit 1
        fi
fi

/bin/mount /dev/${MD_DEV} /var/lib/BackupPC
res=$?

if [ ${res} -ne 0 ]
then
        echo "Unable to mount /dev/${MD_DEV} to /var/lib/BackupPC"
        exit 1
else
        /sbin/service backuppc start
fi

# Remove the block device from the system
echo 1 > /sys/block/${DEV}/device/delete

The cron entry for the two scripts:

# sudo crontab -l
45 05 * * 1-5 /usr/local/bin/drop-device >> /var/log/drop-device.log 2>&1
0,15,30,45 17-23 * * 1-5 /usr/local/bin/add-device >> /var/log/add-device.log 2>&1

We tried everything. Different bonding algorithms (balanced-alb, 802.3ad, balanced-rr, etc), different IO testing tools, different network configurations, messing with the rr_io_min settings, tuning the network stack itself (increasing read/write buffer sizes) and then we got desperate and changed the IO schedulers (For the record: I consider that “an act of desperation” since I would never expect an IO scheduler change to give us a 100% increase).

Since we didn’t configure the physical aspects of the network and we’re playing with equipment hosted in a rather large and very established Lab SAN, it took me a while to think of and accept that it might be a physical problem and not a host configuration one. It was probably about time to walk to the switches and get a visual on the port status for the host links, the Controller port lights and any ISLs or (if present) stacking links.

Guess what… the ISL between the switch connected to the test host and the switch connected to the EqualLogic group was operating at 1/2 its intended capacity.

“Mystery” solved.

You know, sometimes the blinking lights are the most important troubleshooting tool. Sometimes, the color of the blinking lights are the difference between wasting 3 hours and not. And, sometimes, a person with more than 10 years experience designing, configuring and reviewing SANs in mission critical environments can be reminded – in very embarrassing ways – that you should always verify the physical aspects of the configuration first and that simple is more often than not better than the alternative(s). Even if the simple approach includes a hike, a flight of stairs and entering into the noisy data center.

This will permit the libvirt virtualization (management) abstraction interface to easily build “briges of bridges” that in turn let a Kernel Virtual Machine (KVM) guest get it’s own “public” (only in quotes because I happen to think the average bear would not be so silly as to put their Linux/KVM host directly onto the internet. Right???) IP address and route its traffic directly onto the ether (via the lower levels of the IP stack of the host environment).

There are, as is the case with all things Linux or UNIX, a couple of ways to skin this particular bear (sorry, that’s bad!), but the one that makes the most sense to me is to have init take care of the configuration as part of the system boot process (when the network service executes). And doing that, although in its simplest form requires access to a terminal window and a text editor on the Fedora host, is actually very simple, once you know what you’re doing. Hopefully, the following will help you learn (if you don’t already know and are only reading this because you’re looking around and are a very bored individual).

At a high level, all you have to do is edit the /etc/sysconfig/network-scripts/ifcfg-ethX file for the Ethernet (eth) device you want to bridge, specify that the device will belong to a bridge named <bridgename> and then create a /etc/sysconfig/network-scripts/ifcfg-<bridgename> file that configures the bridge (assigning it a way to obtain an IP address – probably static, routing and DNS configuration information).

Prior to configuring the system to use bridging, I had configured static IP addresses on the two physical interfaces I will be creating bridges for. Since that configuration was being obsoleted in order to use the same interfaces for guest-to-iSCSI-SAN traffic, I edited and created the above mentioned configuration files. All of the edited items are in bold. Before you edit and create these files, life gets a little easier if you:

1. Back up the original ifcfg-eth[N] files to some other location than /etc/sysconfig/network-scripts/
2. # ifdown eth[N]

Then, as an example, the two-to-four files you need (ifcfg-eth[N], ifcfg-eth[N+1] and ifcfg-bridge[N] and ifcfg-bridge[N+1]).

/etc/sysconfig/network-scripts/ifcfg-eth2:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth2
BOOTPROTO=none
HWADDR=00:15:17:6C:97:94
<strong>#IPADDR=X.X.Z.231
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes</strong>
ONBOOT=yes
TYPE=Ethernet
<strong>#DNS1=[DNS-SERVER1]
#DNS2=[DNS-SERVER2]</strong>
IPV6INIT=no
USERCTL=no
IPV4_FAILURE_FATAL=yes
NAME="System eth2"
<strong>BRIDGE=iscsi-bridge0</strong>
<strong>MTU=9000</strong>

/etc/sysconfig/network-scripts/ifcfg-eth3:

# Intel Corporation 82571EB Gigabit Ethernet Controller
DEVICE=eth3
<strong>#IPADDR=X.Y.Z.232
#NETMASK=255.255.255.0
#PREFIX=24
#DEFROUTE=yes
#IPV4_FAILURE_FATAL=yes</strong>
HWADDR=00:15:17:6C:97:95
ONBOOT=yes
BOOTPROTO=none
TYPE=Ethernet
<strong>BRIDGE=iscsi-bridge1</strong>
IPV6INIT=no
USERCTL=no
NAME="System eth3"
<strong>MTU=9000</strong>

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge0:

<strong>DEVICE=iscsi-bridge0
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.231
NETMASK=255.255.255.0
STP=off
MTU=9000
DELAY=0</strong>

/etc/sysconfig/network-scripts/ifcfg-iscsi-bridge1:

<strong>DEVICE=iscsi-bridge1
ONBOOT=yes
TYPE=Bridge
IPADDR=X.Y.Z.232
NETMASK=255.255.255.0
MTU=9000
STP=off
DELAY=0</strong>

Although there’s a perfectly valid way of achieving the same thing through the virsh/libvirtd management interface to libvirt as well as with the Network Manager tools, my preference is to make this configuration “stick” using the old network init service. The problem(s) I see with the NetworkManager/libvirtd approach is twofold:

• Timing of NetworkManager start-up (not all that early) relative to the Open-iSCSI stack startup (early)
• Timing of libvirtd start-up (one of the last services to get called) relative to other iSCSI volumes needing to be available for the host environment.

So, for this example, disable NetworkManager as a boot service and enable the network service:

# service NetworkManager stop
# chkconfig NetworkManager off
# chkconfig network on
# ifup iscsi-bridge0
# ifup iscsi-bridge1

And, Bob’s yer uncle (or, at least, he should be!). To verify that everything is working properly, ping an IP target that should be reachable from the bridge device interface(s) only:

# ping -I iscsi-bridge0

# ping -I iscsi-bridge1

By the way: If you use bridged interfaces, no iSCSI volumes on the host (for guests only, in other words) and have iptables enabled on the host (which you should), make sure to configure your host iptables to leave the bridged interfaces alone. For details, see the – soon to be created, I promise – post about performance tuning the Linux IPv4 environment for iSCSI-initiators on this site. Alternatively, you can grab the information pertaining to the relevant bridge sysctl.conf entries from the KVM scalability white paper Red Hat published (and I provided most of the content for in my previous career).

But before we get into the nitty-gritty of that, I just wanted to touch on a problem I’ve been having in setting up multipath over IPv4 on my Fedora 13 host. Now, keep in mind that the Fedora host I’m using has been “tweaked” and “messed with” (a lot), so it ain’t no “fresh install” here… That said, I’ve been having problems getting both of my array-facing NICs to successfully ping the EqualLogic Group IP address.

So, if you’re able to ping the EqualLogic array’s group IP address from both Ethernet NICs, setting up the iSCSI initiator as well as the DM multipath module(s) is very much a “straight forward” exercise.

Create two new open-iscsi initiator interfaces

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># iscsiadm --mode iface --op=new --interface iscsi-0</span></span>

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># iscsiadm --mode iface --op=new --interface iscsi-1</span></span>

Validate that the new interfaces are stored

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># iscsiadm --mode iface</span></span>

<span style="font-family: courier new,courier;">default tcp,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
iser iser,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
iscsi-0 tcp,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
iscsi-1 tcp,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
</span>

Bind the physical NICs to the newly created initiator interfaces

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># iscsiadm --mode iface --op=update --interface iscsi-0 --name=iface.net_ifacename --value=eth2</span></span>

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># iscsiadm --mode iface --op=update --interface iscsi-1 --name=iface.net_ifacename --value=eth3</span></span>

Validate the bindings

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># iscsiadm --mode iface --interface iscsi-0</span></span>

<span style="font-family: courier new,courier;"># BEGIN RECORD 2.0-872
iface.iscsi_ifacename = iscsi-0
<em><strong>iface.net_ifacename = eth2</strong></em></span> <span style="font-family: courier new,courier;">
iface.ipaddress = &lt;empty&gt;
iface.hwaddress = &lt;empty&gt;
iface.transport_name = tcp
iface.initiatorname = &lt;empty&gt;
# END RECORD</span>

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># iscsiadm --mode iface --interface iscsi-1</span></span>

<span style="font-family: courier new,courier;"># BEGIN RECORD 2.0-872
iface.iscsi_ifacename = iscsi-1
<em><strong>iface.net_ifacename = eth3</strong></em></span> <span style="font-family: courier new,courier;">
iface.ipaddress = &lt;empty&gt;
iface.hwaddress = &lt;empty&gt;
iface.transport_name = tcp
iface.initiatorname = &lt;empty&gt;
# END RECORD</span>

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># iscsiadm --mode iface</span></span>

<span style="font-family: courier new,courier;">default tcp,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
iser iser,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;,&lt;empty&gt;
iscsi-0 tcp,&lt;empty&gt;,&lt;empty&gt;,eth2,&lt;empty&gt;
iscsi-1 tcp,&lt;empty&gt;,&lt;empty&gt;,eth3,&lt;empty&gt;</span>

Now that we’ve got the initiator interfaces configured and bound to the physical 1 or 10 GigE NICs, we need to make sure every initiator NIC is logged into the available targets. By the way, you can also configure your Fedora host environment to use bridged interfaces if you’ve got a limited number of NICs in the system and want to use KVM/libvirt to import LUNs directly to a virtualized guest instance.

The easiest way to log in all of the interfaces you’ve defined above to the available (the LUNs/Volumes are “available” and “seen” because you’ve set up the ACL on the LUN/Volume to permit access from this host, right?) targets  is to simply list^[1] the target(s) you’ve discovered previously and re-issue the # iscsiadm –mode node –login –target iqn.2001-05.com.equallogic:<ID of Volume/LUN> command.

The next step now is to edit/add the contents of the /etc/multipath.conf (or, if you’re so inclined: /etc/multipath/multipath.conf which can represent a symlink target for /etc/multipath.conf) file, assign an alias that can be access from /dev/mapper/<alias> for a given (multipathed) device and “do something” (use as an LVM2 volume, a raw device, host a file system on or whatever)

So, before we “do the work” to configure the Multipath environment, we’ll need to either add (or ensure) the following information is present in the /etc/multipath.conf file (by the way, the multipath utilities assume [expect] its configuration file to be readable from the /etc/multipath.conf location!):

## Use user friendly names, instead of using WWIDs as names.
defaults {
     user_friendly_names yes
     find_multipaths yes
}

## Blacklist local devices (no need to have them accidentally "multipathed")
blacklist {
     wwid scsi-SATA_WDC_WD800AAJS-1_WD-WMAM9ANC0895
     devnode "^sd[a]$"
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z]"
     devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

## For EqualLogic PS series RAID arrays
devices {
     device {
          vendor                  "EQLOGIC"
          product                 "100E-00"
          path_grouping_policy    multibus
          getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
          features                "1 queue_if_no_path"
          path_checker            readsector0
          path_selector           "round-robin 0"
          failback                immediate
          rr_min_io               10
          rr_weight               priorities
     }
}

## Assign human-readable name  (aliases) to the World-Wide ID's (names) of the devices we're wanting to manage
## Identify the WWID by issuing <span style="color: #0000ff;"># scsi_id --whitelisted --page=0x83 /dev/&lt;path to sdX for one of the device paths&gt;</span>

multipaths {
     multipath {
          wwid                    36090a01850e08bd883c4a45bfa0330ba
          alias                     benchmark2
     }
     multipath {
          wwid                    36090a01850e04bd683c4745bfa03b07c
          alias                     benchmark1
     }
}

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># service multipathd start
# chkconfig multipathd on</span></span>

Verify that the device mapper multipath configuration is ‘complete’ with the following commands:

<span style="color: #0000ff;"><span style="font-family: courier new,courier;"># multipath -v2
# multipath -ll</span></span>

<span style="font-family: courier new,courier;"><strong>benchmark2 (36090a01850e08bd883c4a45bfa0330ba) dm-0 EQLOGIC,100E-00</strong>
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
   |- 6:0:0:0 sdd 8:48 active ready running
   `- 5:0:0:0 sdb 8:16 active ready running
<strong>benchmark1 (36090a01850e04bd683c4745bfa03b07c) dm-1 EQLOGIC,100E-00</strong>
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
   |- 4:0:0:0 sde 8:64 active ready running
   `- 3:0:0:0 sdc 8:32 active ready running</span>

[1] = # iscsiadm –mode node

So, as part of playing in my lab environment to configure Fedora 13 and an iSCSI RAID array (EqualLogic PS Series array) for multipathing, I ran into a bit of a snag:

Successful ping to [group-ip] from first iSCSI NIC

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># ping -I eth2 [group-ip]</span>
PING [group-ip] ([group-ip]) from [eth2-ip] eth2: 56(84) bytes of data.
64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms
64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms
64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms
^C
--- [group-ip] ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2927ms
rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms</span>

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># tcpdump -i eth2 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply</span>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
08:30:55.601799 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 1650, seq 1, length 64
08:30:55.601933 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 1650, seq 1, length 64
08:30:56.602616 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 1650, seq 2, length 64
08:30:56.602693 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 1650, seq 2, length 64
08:30:57.602755 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 1650, seq 3, length 64
08:30:57.602831 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 1650, seq 3, length 64</span>

But failed ping to [group-ip] from second iSCSI NIC

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># ping -I eth3 [group-ip]</span>
PING [group-ip] ([group-ip]) from [eth3-ip] eth3: 56(84) bytes of data.
^C
--- [group-ip] ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2630ms</span>

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># tcpdump -i eth3 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply</span>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
08:31:01.496852 IP [eth3-ip] &gt; [group-ip]: ICMP echo request, id 1906, seq 1, length 64
08:31:01.496970 IP [group-ip] &gt; [eth3-ip]: ICMP echo reply, id 1906, seq 1, length 64
08:31:02.496568 IP [eth3-ip] &gt; [group-ip]: ICMP echo request, id 1906, seq 2, length 64
08:31:02.496660 IP [group-ip] &gt; [eth3-ip]: ICMP echo reply, id 1906, seq 2, length 64
08:31:03.496591 IP [eth3-ip] &gt; [group-ip]: ICMP echo request, id 1906, seq 3, length 64
08:31:03.496650 IP [group-ip] &gt; [eth3-ip]: ICMP echo reply, id 1906, seq 3, length 64</span>

Being, what appears to be, above average dense about this, I spent hours trying to figure out what was wrong. I’m still not convinced I figured out why this is happening (root cause), but basically the sysctl tunable net.ipv4.conf.{all,eth*,default}.rp_filter seemingly defaulting to 1 in Fedora 13, running version 2.6.33.6-147.fc13.x86_64 of the kernel, results in only one of the two NICs connected and configured for the dedicated iSCSI network in my lab environment responding to ICMP or TCP traffic (responses) from the group IP.

For some reason, if my Fedora host communicates with another (any?) host on the same subnet, it exhibits none of this behavior!

To work around this problem set you’ve got a few commands to issue:

The least work intensive (yet most disruptive and a whole lot like using a sledge hammer for the finishing touches!) option is to change net.ipv4.conf.default.rp_filter = 1 to 0 (zero) in /etc/sysctl.conf, and reboot. One (big) problem with this is the case where you’ve got other multi-homed networks connected to the same host and you want traffic filtering between NICs on the same subnet (not sure why you’d want that and you’re not using the bonding driver, but…).

Alternatively, you can issue the following command for each of your iSCSI network-facing NICs (# sysctl -w net.ipv4.conf.eth{0,1..N}.rp_filter=0) and keep your systems and workloads running. Basically, issue the above command for each of your eth, bridge or bond device attached to your dedicated IP network (and expected to read/write to/from the EqualLogic group) and voila(!) the NICs will “permit” packets from the group IP again. NOTE: This (# sysctl -w …) isn’t persistent across reboots!)

Successful ping to [group-ip] from second iSCSI NIC

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># ping -I eth2 [group-ip]</span>
PING [group-ip] ([group-ip]) from [eth2-ip] eth2: 56(84) bytes of data.
64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms
64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms
64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms
^C
--- [group-ip] ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2927ms
rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms</span>

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># tcpdump -i eth2 icmp[icmptype] == icmp-echo or icmp[icmptype] ==  icmp-echoreply</span>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
08:11:38.006766 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 19057, seq 1, length 64
08:11:38.006861 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 19057, seq 1,length 64
08:11:39.007030 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 19057, seq 3, length 64
08:11:39.007098 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 19057, seq 2,length 64
08:11:40.006615 IP [eth2-ip] &gt; [group-ip]: ICMP echo request, id 19057, seq 3, length 64
08:11:40.006688 IP [group-ip] &gt; [eth2-ip]: ICMP echo reply, id 19057, seq 3,length 64</span>

AND successful ping to [group-ip] from second iSCSI NIC

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># ping -I eth3 [group-ip]</span>
PING [group-ip] ([group-ip]) from [eth3-ip] eth3: 56(84) bytes of data.
64 bytes from [group-ip]: icmp_seq=1 ttl=255 time=0.103 ms
64 bytes from [group-ip]: icmp_seq=2 ttl=255 time=0.075 ms
64 bytes from [group-ip]: icmp_seq=3 ttl=255 time=0.093 ms
^C
--- [group-ip] ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2927ms
rtt min/avg/max/mdev = 0.075/0.090/0.103/0.013 ms</span>

<span style="font-family: courier new,courier;"><span style="color: #0000ff;"># tcpdump -i eth3 icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply</span>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
08:11:44.074841 IP [eth3-ip] &gt; [group-ip]: ICMP echo request, id 19313, seq 1, length 64
08:11:44.074938 IP [group-ip] &gt; [eth3-ip]: ICMP echo reply, id 19313, seq 1,length 64
08:11:45.075025 IP [eth3-ip] &gt; [group-ip]: ICMP echo request, id 19313, seq 2, length 64
08:11:45.075123 IP [group-ip] &gt; [eth3-ip]: ICMP echo reply, id 19313, seq 2, length 64</span>

Update just to make sure I dot the i’s, etc; The sysctl settings do not cascade up to bridges built on top of a physical interface that’s had a specific setting, like net.ipv4.conf.[NIC].rp_filter configured. Thus you have to set .rp_filter=0 for all bridges as well.

It’s pretty straight forward, really.

At its simplest and  highest(ish) level:

1. Install the iSCSI initiator management utilities for Fedora 13 (# yum install iscsi-initiator-utils)
2. Identify (change) the iqn for your Fedora 13 host (# {cat|vi} /etc/iscsi/initiatorname.iscsi)
3. Configure your target(s), define LUN(s) and set up ACLs for the LUNs (Volumes) you’re wanting to use in your EqualLogic group
4. Scan the iSCSI target (EqualLogic group) (# iscsiadm –mode discovery –type sendtargets –portal <ip for EQL group>:3260)
5. Log in to the target & volume (LUN) you’re wanting to use (# iscsiadm –mode node –login –target <iqn for LUN/Volume on EQL array> )
6. Repeat for all of the volumes/LUNs you’re wanting to access from the Fedora 13 host
7. Use the volumes/LUNs you’ve “imported”
8. Optionally (I recommend doing this for clarity), log out and remove (delete) all of the Volumes (LUNs) you’re not using on the Fedora host
   1. List all of the discovered volumes/LUNs from the target: (# iscsiadm –mode node)
   2. Log out the volumes/LUN’s you do not intend to access from the host (# iscsiadm –mode node –logout –target <iqn of Volume/LUN> )
   3. Delete the volume(s)/LUN(s) you don’t want: (# iscsiadm –mode node –target <iqn of Volume/LUN> –op delete )

If you don’t like “long” option names, substitute;

• –mode with -m
• –target with -T
• –login with -l
• –logout with -u
• –op with -o
• ‘sendtargets‘ with ‘st‘

From now on, whenever the iscsid daemon is started during boot, the host will log on to the volume(s)/LUNs during boot, the /dev/ entries will be created – but not be persistent across boots, so make sure you use these LUNs with either LVM or use the UUID= or file system LABEL=<name> when mounting file systems hosted on these Volume(s)/LUNs! – and the devices will be accessible from the host.  More on creating UDEV rules for your iSCSI Volume(s)/LUN(s) in the next  installment of the series.

Of course, now there’s configuring for Volume/LUN path availability, etc, etc. Multipath configuration which is documented by DELL in a tech report paper on our EqualLogic support web site (requires a valid support contract/warranty for login)- and I’ll summarize in another post.

I’ve got a configuration in mind which I’ll document over time, probably. Sorry for being so incredibly vague on the time-line, but my job isn’t to document Linux and EqualLogic configurations so I’m only doing this “for fun” between learning about the PS Series arrays (have to do a live demo “soon”) and various other job tasks.

[...] when someone can take storage out of a box, “install, use and protect it” on any O/S, virtualization environment with nothing more than a poster with 5 to 7 blocks as an guide, such “Apple-like” storage will have arrived.

From RayOnStorageBlog

Who else thinks this should be a goal? Why? Why not?