shanks

GlusterFS quota on volumes serving virtual machines.

2013-10-07T20:46:00.002+05:30

Experiment is to see how virtual machines behave when glusterfs quota is set on a volume which serves virtual machine qcow2 images and limit is exceeded.

I create a glusterfs volume and set virt group options.

[root@ninja ~]# gluster vol info

Volume Name: vmstore
Type: Replicate
Volume ID: c96de15d-024e-416d-a1c5-ff5fef44b25b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.68:/rhs1/vmstore
Brick2: xx.yy.zz.56:/rhs1/vmstore
Options Reconfigured:
features.quota-deem-statfs: on
features.quota: on
storage.owner-gid: 107
storage.owner-uid: 107
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

Enable quota and deem-statfs on the above volume. features.quota-deem-statfs is necessary if you consider to estimate filesystem size using df command.

# gluster volume quota vmstore enable
# gluster vol set vmstore features.quota-deem-statfs on

[root@ninja ~]# gluster vol info vmstore

Volume Name: vmstore
Type: Replicate
Volume ID: c96de15d-024e-416d-a1c5-ff5fef44b25b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.68:/rhs1/vmstore
Brick2: xx.yy.zz.56:/rhs1/vmstore
Options Reconfigured:
features.quota-deem-statfs: on
features.quota: on
storage.owner-gid: 107
storage.owner-uid: 107
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

[root@ninja ~]#

Ensure that the volume is mounted on the client
[root@client37 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-LogVol01 3.6T 2.1G 3.4T 1% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 485M 40M 421M 9% /boot
ninja.shanks.com:vmstore 195G 5.2G 190G 3% /var/lib/libvirt/images
[root@client37 ~]#

Now set quota on the volume to limit usage to 20G.

[root@ninja ~]# gluster volume quota vmstore limit-usage / 20GB
volume quota : success
[root@ninja ~]#

Check on the client:
[root@client37 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-LogVol01 3.6T 2.1G 3.4T 1% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 485M 40M 421M 9% /boot
ninja.shanks.com:vmstore 20G 19G 1.8G 92% /var/lib/libvirt/images
[root@client37 ~]#

Now keep adding data so that the quota usage limit is reached. You would see I/O error within the instance and the virtual machine becomes read-only

8<
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 1624644
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 1624645
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 1624646
lost page write due to I/O error on dm-2
Buffer I/O error on device dm-2, logical block 1624647
...
Aborting journal on device dm-2-8.
end_request: I/O error, dev vda, sector 118888896
end_request: I/O error, dev vda, sector 118898976
end_request: I/O error, dev vda, sector 118899984
EXT4-fs error (device dm-2) in ext4_reserve_inode_write: Journal has aborted
...
EXT4-fs (dm-2): Remounting filesystem read-only
EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 3244 pages, ino 13; err -30
...
JBD2: Detected IO errors while flushing file data on dm-2-8
[root@localhost home]# __ratelimit: 155 callbacks suppressed
__ratelimit: 19894 callbacks suppressed
Buffer I/O error on device dm-0, logical block 1081871
lost page write due to I/O error on dm-0
JBD2: Detected IO errors while flushing file data on dm-0-8
Aborting journal on device dm-0-8.
EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (dm-0): Remounting filesystem read-only
Buffer I/O error on device dm-0, logical block 6324224
lost page write due to I/O error on dm-0
JBD2: I/O error detected when updating journal superblock for dm-0-8.
8<

Post reboot, kernel panic is detected.

8<
dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
[] ? panic+0xa7/0x16f
[] ? do_exit+0x862/0x870
[] ? fput+0x25/0x30
[] ? do_group_exit+0x58/0xd0
[] ? sys_exit_group+0x17/0x20
[] ? system_call_fastpath+0x16/0x1b

Resolution

On the hypervisor,

[root@client37 ~]# virsh destroy domain

On the gluster node, increase the limit-usage

[root@ninja ~]# gluster volume quota vmstore limit-usage / 40GB

volume quota : success

[root@ninja ~]#

On the client, verify that the new value is reflected.

[root@client37 ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-LogVol01 3.6T 2.1G 3.4T 1% /

tmpfs 7.8G 0 7.8G 0% /dev/shm

/dev/sda1 485M 40M 421M 9% /boot

ninja.shanks.com:vmstore 40G 21G 20G 51% /var/lib/libvirt/images

[root@client37 ~]#

[root@client37 ~]# virsh start domain

And you are good to GO!!!

Live virtual machine on GlusterFS geo-replicated volume.

2013-10-03T21:08:00.000+05:30

Pre-requisites and Geo-replication configuration are still the same as identified here.

Except that geo-replication was configured for vmstore instead of snapstore. I also copied over the virtual machine xml to geo2 hypervisor.

Note: That this was tested on a local lab network, hence, network bandwidth should be considered if on the internet.

Validation

1. [root@geo1 ~]# virt-install --connect=qemu:///system --network=bridge:br0 --initrd-inject=rhel.ks --extra-args="ks=file:/rhel.ks console=tty0 console=ttyS0,115200" --name=shanks-rhel1 --disk path=/var/lib/libvirt/images/shanks-rhel1.qcow2,device=disk,format=qcow2,bus=virtio,cache=writeback,io=threads --ram 1024 --vcpus=1 --check-cpu --accelerate --hvm --location=http://download.shanks.com/RHEL-6/6.4/Server/x86_64/os/ --nographics

Here, /var/lib/libvirt/images is a gluster mount which is geo-replicated.

2. After the installation in complete, I login to the virtual machine and create a 600MB file. Generate the md5sum of it to have it validated against the geo-replicated image.

3. scp the virtual machine xml to geo2 hypervisor.

4. poweroff the virtual machine in geo1.

5. Ensure that the sync is complete:

[root@ninja ~]# gluster volume geo-replication vmstore gladiator.lab.eng.blr.redhat.com::vmstore status detail

MASTER: vmstore SLAVE: gladiator.lab.eng.blr.redhat.com::vmstore

NODE HEALTH UPTIME FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING

---------------------------------------------------------------------------------------------------------------------------------

ninja.lab.eng.blr.redhat.com Stable 1 day 17:22:49 56 0 0Bytes 0

vertigo.lab.eng.blr.redhat.com Stable 1 day 17:22:34 0 0 0Bytes 0

[root@ninja ~]#

6. virsh start the virtual machine in geo2.

7. Login and run md5sum on the 600MB file.

Result: Virtual machines boots up fine and the md5sum matches.

Got to admit, it was a great exercise!

And this opens a whole bunch of new scenarios that can be tried and tested with virtualization integrated with glusterfs geo-replication.

-shanks

Disaster recovery of VM snapshot store using GlusterFS Geo Replication.

2013-10-02T03:25:00.003+05:30

My setup looks like the following:

Couple of pre-requisites:
1. SELinux booleans are set on both the hypervisors. Geo1 and Geo2 in this example.

setsebool -P sanlock_use_fusefs on
setsebool -P virt_use_sanlock on
setsebool -P virt_use_fusefs on

2. Gluster volumes are create on both master and slave. For simplicity I have use the same volume names. Example of my gluster volumes from one of the node:

[root@ninja ~]# gluster vol info

Volume Name: vmstore

Type: Replicate

Volume ID: c96de15d-024e-416d-a1c5-ff5fef44b25b

Status: Created

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: xx.yy.zz.68:/rhs1/vmstore

Brick2: xx.yy.zz.56:/rhs1/vmstore

Options Reconfigured:

storage.owner-gid: 107

storage.owner-uid: 107

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

Volume Name: snapstore

Type: Replicate

Volume ID: ad7529d8-242b-40be-a741-aaf331e0fb80

Status: Created

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: xx.yy.zz.68:/rhs2/snapstore

Brick2: xx.yy.zz.56:/rhs2/snapstore

Options Reconfigured:

storage.owner-uid: 107

storage.owner-gid: 107

[root@ninja ~]#

My typical workflow looks like:

Configure Geo-replication

With the above setup and workflow in mind, I configure Geo-replication environment between master and slave.

[root@ninja ~]# gluster system:: execute gsec_create

Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub

Verify common_secret.pem.pub file is created

[root@ninja ~]# ls -l /var/lib/glusterd/geo-replication/common_secret.pem.pub

-rw------- 1 root root 912 Oct 1 13:10 /var/lib/glusterd/geo-replication/common_secret.pem.pub

Push pem to the slave

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore create push-pem

Creating geo-replication session between snapstore & gladiator.shanks.com::snapstore has been successful

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status

NODE MASTER SLAVE HEALTH UPTIME

----------------------------------------------------------------------------------------------------------------------

ninja.shanks.com snapstore gladiator.shanks.com::snapstore Not Started N/A

vertigo.shanks.com snapstore gladiator.shanks.com::snapstore Not Started N/A

[root@ninja ~]#

Mount vmstore from master on geo1 hypervisor. Note to update /etc/fstab appropriately.

[root@geo1 ~]# mount -t glusterfs ninja.shanks.com:vmstore /var/lib/libvirt/images/

[root@geo1 ~]# df -hT

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-LogVol01 ext4 3.6T 2.1G 3.4T 1% /

tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm

/dev/sda1 ext4 485M 40M 421M 9% /boot

ninja.shanks.com:vmstore fuse.glusterfs 195G 33M 195G 1% /var/lib/libvirt/images

[root@geo1 ~]#

Installation of virtual machine

Before proceeding any further, let's create a qcow2 file and install OS.

[root@geo1 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/shanks-rhel1.qcow2 10G

Formatting '/var/lib/libvirt/images/shanks-rhel1.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536

[root@geo1 ~]# qemu-img info /var/lib/libvirt/images/shanks-rhel1.qcow2

image: /var/lib/libvirt/images/shanks-rhel1.qcow2

file format: qcow2

virtual size: 10G (10737418240 bytes)

disk size: 193K

cluster_size: 65536

[root@geo1 ~]#

[root@geo1 ~]# virt-install --connect=qemu:///system --network=bridge:br0 --initrd-inject=rhel.ks --extra-args="ks=file:/rhel.ks console=tty0 console=ttyS0,115200" --name=shanks-rhel1 --disk path=/var/lib/libvirt/images/shanks-rhel1.qcow2,device=disk,format=qcow2,bus=virtio,cache=writeback,io=threads --ram 1024 --vcpus=1 --check-cpu --accelerate --hvm --location=http://download.shanks.com/RHEL-6/6.4/Server/x86_64/os/ --nographics

Ensure that the virtual machine is running

[root@geo1 ~]# virsh list

Id Name State

----------------------------------------------------

4 shanks-rhel1 running

[root@geo1 ~]#

Now we mount snapstore volume on geo1

[root@geo1 ~]# mount -t glusterfs ninja.shanks.com:snapstore /var/lib/libvirt/qemu/snapshot

[root@geo1 ~]# df -hT

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-LogVol01 ext4 3.6T 2.1G 3.4T 1% /

tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm

/dev/sda1 ext4 485M 40M 421M 9% /boot

ninja.shanks.com:vmstore fuse.glusterfs 195G 2.4G 192G 2% /var/lib/libvirt/images

ninja.shanks.com:snapstore fuse.glusterfs 195G 33M 195G 1% /var/lib/libvirt/qemu/snapshot

[root@geo1 ~]#

Create external snapshot

Time to create an external snapshot

[root@geo1 ~]# virsh snapshot-create-as shanks-rhel1 shanks-rhel1-snap1 --disk-only --atomic

Domain snapshot shanks-rhel1-snap1 created

[root@geo1 ~]#

[root@geo1 ~]# qemu-img info /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1

image: /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1

file format: qcow2

virtual size: 10G (10737418240 bytes)

disk size: 960K

cluster_size: 65536

backing file: /var/lib/libvirt/images/shanks-rhel1.qcow2

[root@geo1 ~]#

Copy this read-only backing file to snapstore

[root@geo1 ~]# cp shanks-rhel1.qcow2 /var/lib/libvirt/qemu/snapshot/shanks-rhel1/

You would also need to copy the xml file for it for the first time. This is not required if you can directly configure/create a xml. I copied to save some time :)

[root@geo1 images]# cp /etc/libvirt/qemu/shanks-rhel1.xml /var/lib/libvirt/qemu/snapshot/shanks-rhel1/

[root@geo1 ~]# ls /var/lib/libvirt/qemu/snapshot/shanks-rhel1/

shanks-rhel1.qcow2 shanks-rhel1-snap1.xml shanks-rhel1.xml

[root@geo1 ~]#

Mount the slave volumes on geo2 hypervisor

[root@geo2 ~]# mount -t glusterfs gladiator.shanks.com:vmstore /var/lib/libvirt/images/

[root@geo2 ~]# mount -t glusterfs gladiator.shanks.com:snapstore /var/lib/libvirt/qemu/snapshot/

[root@geo2 ~]# df -hT

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-LogVol01 ext4 3.6T 2.1G 3.4T 1% /

tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm

/dev/sda1 ext4 485M 40M 421M 9% /boot

xx.yy.zz.68:/vmstore fuse.glusterfs 200G 33M 200G 1% /var/lib/libvirt/images

gladiator.shanks.com:vmstore fuse.glusterfs 200G 33M 200G 1% /var/lib/libvirt/images

gladiator.shanks.com:snapstore fuse.glusterfs 200G 33M 200G 1% /var/lib/libvirt/qemu/snapshot

[root@geo2 ~]#

All set to start geo-replication!

From master initiate the geo-replication start command

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore start

Starting geo-replication session between snapstore & gladiator.shanks.com::snapstore has been successful

status should show as "Initializing..."

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status

NODE MASTER SLAVE HEALTH UPTIME

--------------------------------------------------------------------------------------------------------------------------

ninja.shanks.com snapstore gladiator.shanks.com::snapstore Initializing... N/A

vertigo.shanks.com snapstore gladiator.shanks.com::snapstore Initializing... N/A

[root@ninja ~]#

and then later on as "Stable"

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status

NODE MASTER SLAVE HEALTH UPTIME

-------------------------------------------------------------------------------------------------------------------

ninja.shanks.com snapstore gladiator.shanks.com::snapstore Stable 00:01:12

vertigo.shanks.com snapstore gladiator.shanks.com::snapstore Stable 00:01:09

[root@ninja ~]#

detailed status should show that files are synced

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status detail

MASTER: snapstore SLAVE: gladiator.shanks.com::snapstore

NODE HEALTH UPTIME FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING

---------------------------------------------------------------------------------------------------------------------------

ninja.shanks.com Stable 00:01:28 4 0 0Bytes 0

vertigo.shanks.com Stable 00:01:26 0 0 0Bytes 0

[root@ninja ~]#

You should be able to access the files from geo2 hypervisor snapshot mount

[root@geo2 ~]# ls /var/lib/libvirt/qemu/snapshot/shanks-rhel1/

shanks-rhel1.qcow2 shanks-rhel1-snap1.xml shanks-rhel1.xml

[root@geo2 ~]#

Copy the virtual machine xml file to the default location and edit the image source path.

[root@geo2 ~]# cp /var/lib/libvirt/qemu/snapshot/shanks-rhel1/shanks-rhel1.xml /etc/libvirt/qemu/

Copy the backing file to vmstore (/var/lib/libvirt/images) on geo2 hypervisor

[root@geo2 ~]# cp /var/lib/libvirt/qemu/snapshot/shanks-rhel1/shanks-rhel1.qcow2 /var/lib/libvirt/images/

[root@geo2 ~]#

Define and start the virtual machine

[root@geo2 ~]# virsh define /etc/libvirt/qemu/shanks-rhel1.xml

[root@geo2 ~]# virsh start shanks-rhel1

Once it is verified that the virtual machines boots up fine, its time to delete the external snapshot. This is not necessary, however, I am deleting it for simplicity.

[root@geo1 ~]# virsh snapshot-list shanks-rhel1

Name Creation Time State

------------------------------------------------------------

shanks-rhel1-snap1 2013-10-01 19:01:25 +0530 disk-snapshot

[root@geo1 ~]#

[root@geo1 ~]# virsh blockpull --domain shanks-rhel1 --path /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1 --verbose --wait

Block Pull: [ 100 %]

Pull complete

Note that blockpull populates the snap disk with backing file.

[root@geo1 ~]# qemu-img info /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1

image: /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1

file format: qcow2

virtual size: 10G (10737418240 bytes)

disk size: 1.3G

cluster_size: 65536

[root@geo1 ~]#

[root@geo1 ~]# virsh snapshot-delete shanks-rhel1 shanks-rhel1-snap1 --metadata

Domain snapshot shanks-rhel1-snap1 deleted

[root@geo1 ~]# virsh snapshot-list shanks-rhel1

Name Creation Time State

------------------------------------------------------------

[root@geo1 ~]#

Let's test this workflow:

1. From geo1 hypervisor login to the virtual machine and touch a file.

[root@geo1 ~]# virsh console shanks-rhel1

Connected to domain shanks-rhel1

Escape character is ^]

[root@localhost ~]# touch a

[root@localhost ~]# md5sum a

d41d8cd98f00b204e9800998ecf8427e a

[root@localhost ~]#

2. Create external snapshot and copy the backing file to snapstore mount (/var/lib/libvirt/qemu/snapshot)

[root@geo1 ~]# virsh snapshot-create-as shanks-rhel1 shanks-rhel1-snap2 --disk-only --atomic

Domain snapshot shanks-rhel1-snap2 created

[root@geo1 ~]#

[root@geo1 ~]# qemu-img info /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap2

image: /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap2

file format: qcow2

virtual size: 10G (10737418240 bytes)

disk size: 8.4M

cluster_size: 65536

backing file: /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1

[root@geo1 ~]#

[root@geo1 ~]# cp /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap1 /var/lib/libvirt/qemu/snapshot/shanks-rhel1/

[root@geo1 ~]#

3. Check the geo-replication status in detail and ensure that the image file is fully synced

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status detail

MASTER: snapstore SLAVE: gladiator.shanks.com::snapstore

NODE HEALTH UPTIME FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING

---------------------------------------------------------------------------------------------------------------------------

ninja.shanks.com Stable 00:52:10 5 1 1.3GB 0

vertigo.shanks.com Stable 00:52:07 0 0 0Bytes 0

[root@ninja ~]#

[root@ninja ~]# gluster volume geo-replication snapstore gladiator.shanks.com::snapstore status detail

MASTER: snapstore SLAVE: gladiator.shanks.com::snapstore

NODE HEALTH UPTIME FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING

---------------------------------------------------------------------------------------------------------------------------

ninja.shanks.com Stable 01:00:27 6 0 0Bytes 0

vertigo.shanks.com Stable 01:00:24 0 0 0Bytes 0

[root@ninja ~]#

4. Replace the existing source image of the virtual machine and start the instance to verify the md5sum of file "a'.

[root@geo2 ~]# cp /var/lib/libvirt/qemu/snapshot/shanks-rhel1/shanks-rhel1.shanks-rhel1-snap1 /var/lib/libvirt/images/shanks-rhel1.qcow2

cp: overwrite `/var/lib/libvirt/images/shanks-rhel1.qcow2'? y

[root@geo2 ~]#

[root@geo2 ~]# virsh start shanks-rhel1

Domain shanks-rhel1 started

[root@geo2 ~]# virsh console shanks-rhel1

[root@localhost ~]# md5sum a

d41d8cd98f00b204e9800998ecf8427e a

[root@localhost ~]#

Great!

5. Finally, blockpull the snapshot and delete its metadata.

[root@geo1 ~]# virsh blockpull --domain shanks-rhel1 --path /var/lib/libvirt/images/shanks-rhel1.shanks-rhel1-snap2 --verbose --wait

Block Pull: [100 %]

Pull complete

[root@geo1 ~]#

[root@geo1 ~]# virsh snapshot-delete shanks-rhel1 shanks-rhel1-snap2 --metadataDomain snapshot shanks-rhel1-snap2 deleted

[root@geo1 ~]#

freeIPA integrated with GlusterFS to automount user home directory

2013-08-14T15:09:00.000+05:30

freeIPA integrated with GlusterFS so that the user home directories are auto mounted upon login using nfs/glusterfs.

Setup IPA Server and Client

Install freeIPA:

# yum install freeipa-server bind bind-dyndb-ldap

# ipa-server-install

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the FreeIPA Server.

This includes:
* Configure a stand-alone CA (dogtag) for certificate management
* Configure the Network Time Daemon (ntpd)
* Create and configure an instance of Directory Server
* Create and configure a Kerberos Key Distribution Center (KDC)
* Configure Apache (httpd)

To accept the default shown in brackets, press the Enter key.

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Do you want to configure integrated DNS (BIND)? [no]: yes

Existing BIND configuration detected, overwrite? [no]: yes
Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
.
Example: master.example.com.

Server host name [rhsauto007.shanks.com]:

Warning: skipping DNS resolution of host rhsauto007.shanks.com
The domain name has been determined based on the host name.

Please confirm the domain name [shanks.com]:

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [shanks.COM]:
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password:
Password (confirm):

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password:
Password (confirm):

Do you want to configure DNS forwarders? [yes]:
Enter the IP address of DNS forwarder to use, or press Enter to finish.
Enter IP address for a DNS forwarder: xx.yy.zz.2
DNS forwarder xx.yy.zz.2 added
Enter IP address for a DNS forwarder:
Do you want to configure the reverse zone? [yes]:
Please specify the reverse zone name [zz.yy.xx.in-addr.arpa.]:
Using reverse zone zz.yy.xx.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname: rhsauto007.shanks.com
IP address: xx.yy.zz.237
Domain name: shanks.com
Realm name: shanks.COM

BIND DNS server will be configured to serve IPA domain with:
Forwarders: xx.yy.zz.2
Reverse zone: zz.yy.xx.in-addr.arpa.

Continue to configure the system with these values? [no]: yes

...

Restarting the web server
==============================================================================
Setup complete

Next steps:
1. You must make sure these network ports are open:
TCP Ports:
* 80, 443: HTTP/HTTPS
* 389, 636: LDAP/LDAPS
* 88, 464: kerberos
* 53: bind
UDP Ports:
* 88, 464: kerberos
* 53: bind
* 123: ntp

2. You can now obtain a kerberos ticket using the command: 'kinit admin'
This ticket will allow you to use the IPA tools (e.g., ipa user-add)
and the web user interface.

Be sure to back up the CA certificate stored in /root/cacert.p12
This file is required to create replicas. The password for this
file is the Directory Manager password

On the client machine install freeipa-client:

# yum install freeipa-client

Also install IPA clients on all the GlusterFS nodes.

A typical ipa-client-install would look like:

# ipa-client-install --force-ntpd --mkhomedir
Provide the domain name of your IPA server (ex: example.com): shanks.com
Provide your IPA server name (ex: ipa.example.com): rhsauto007.shanks.com
The failure to use DNS to find your IPA server indicates that your resolv.conf file is not properly configured.
Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: yes
Hostname: rhsauto011.shanks.com
Realm: shanks.COM
DNS Domain: shanks.com
IPA Server: rhsauto007.shanks.com
BaseDN: dc=shanks,dc=com

Continue to configure the system with these values? [no]: yes
User authorized to enroll computers: admin
Synchronizing time with KDC...
Password for admin@shanks.COM:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=shanks.COM
Issuer: CN=Certificate Authority,O=shanks.COM
Valid From: Tue Aug 13 21:02:53 2013 UTC
Valid Until: Sat Aug 13 21:02:53 2033 UTC

Enrolled in IPA realm shanks.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm shanks.COM
trying https://rhsauto007.shanks.com/ipa/xml
Forwarding 'env' to server 'https://rhsauto007.shanks.com/ipa/xml'
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Forwarding 'host_mod' to server 'https://rhsauto007.shanks.com/ipa/xml'
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.

Lets do a quick check of IPA server and client setup before proceeding:

Create user in IPA server
# ipa user-add
First name: shanks
Last name: shanks
User login [sshanks]: shanks
-------------------
Added user "shanks"
-------------------
User login: shanks
First name: shanks
Last name: shanks
Full name: shanks shanks
Display name: shanks shanks
Initials: ss
Home directory: /home/shanks
GECOS field: shanks shanks
Login shell: /bin/sh
Kerberos principal: shanks@shanks.COM
Email address: shanks@shanks.com
UID: 1661400001
GID: 1661400001
Password: False
Member of groups: ipausers
Kerberos keys available: False
[root@rhsauto007 ~]# ipa passwd shanks
New Password:
Enter New Password again to verify:
----------------------------------------------------
Changed password for "shanks@shanks.COM"
----------------------------------------------------

From client try to login as user shanks and you should be prompted to reset password set on server:

[root@rhsauto011 ~]# ssh -l shanks $HOSTNAME
shanks@rhsauto011.shanks.com's password:
Password expired. Change your password now.
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for user shanks.
Current Password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Connection to rhsauto011.shanks.com closed.
[root@rhsauto011 ~]#

[root@rhsauto011 ~]# ssh -l shanks $HOSTNAME
shanks@rhsauto011.shanks.com's password:
Last login: Tue Aug 13 17:21:01 2013 from rhsauto011.shanks.com
-sh-4.2$ pwd
/home/shanks
-sh-4.2$

All set on the IPA Server and IPA client side.

Install IPA client on GlusterFS node:

# ipa-client-install
DNS discovery failed to determine your DNS domain
Provide the domain name of your IPA server (ex: example.com): shanks.com
Provide your IPA server name (ex: ipa.example.com): rhsauto007.shanks.com
The failure to use DNS to find your IPA server indicates that your resolv.conf file is not properly configured.
Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: yes
Hostname: rhsbb1.shanks.com
Realm: shanks.COM
DNS Domain: shanks.com
IPA Server: rhsauto007.shanks.com
BaseDN: dc=shanks,dc=com

Continue to configure the system with these values? [no]: yes
User authorized to enroll computers: admin
Synchronizing time with KDC...
Unable to sync time with IPA NTP server, assuming the time is in sync. Please check that 123 UDP port is opened.
Password for admin@shanks.COM:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=shanks.COM
Issuer: CN=Certificate Authority,O=shanks.COM
Valid From: Tue Aug 13 21:02:53 2013 UTC
Valid Until: Sat Aug 13 21:02:53 2033 UTC

Enrolled in IPA realm shanks.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm shanks.COM
trying https://rhsauto007.shanks.com/ipa/xml
Hostname (rhsbb1.shanks.com) not found in DNS
Failed to update DNS records.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Forwarding 'host_mod' to server u'https://rhsauto007.shanks.com/ipa/xml'
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.
[root@localhost ~]#

You should now be able to find the above added GlusterFS node in IPA server:

[root@rhsauto007 ~]# ipa host-show rhsbb1.shanks.com
Host name: rhsbb1.shanks.com
Certificate: MIIESDCCAzCgAwIBAgIBDTANBgkqhkiG9w0BAQsFADBBMR8wHQYDVQQKExZMQUIuRU5HLkJMUi5SRURIQVQuQ09NMR4wHAYDVQQDExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTMwODEzMjEyMzIyWhcNMTUwODE0MjEyMzIyWjBJMR8wHQYDVQQKExZMQUIuRU5HLkJMUi5SRURIQVQuQ09NMSYwJAYDVQQDEx1yaHNiYjEubGFiLmVuZy5ibHIucmVkaGF0LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALGFGgrmDT8qDAmL2uRXvJfZ6r+DMQszHxVuQxBJqNMjibuUOYp4GJsjNo525uK9h2WKINBxYxVmnHf+TMvlI2pPeCbyVit/u05TKsSKzRf9742tawUlQXXzpbZLoJntG5hUFFWXiid1xtzQUStYKJru1rBeEed+ofhIK+ZpoYQ10ljepeRAHPur4syHjqZ+7r2syUEstdvsL+TVra450kF3dEuOESKilOYbAcQqj47eNSP1Z/enxL0XadNiO8QafjH/V7WKqsnCUpaCIBF6kC4750zPgJhJ34uCRg9PBgZP0mZ7D9ASnbs6MMAad7FpQrYxwi25D0VnbBh8WTdLfSECAwEAAaOCAUEwggE9MB8GA1UdIwQYMBaAFGA8rBYfLswr1bGMo3j0Y8WAb1t9MEgGCCsGAQUFBwEBBDwwOjA4BggrBgEFBQcwAYYsaHR0cDovL2lwYS1jYS5sYWIuZW5nLmJsci5yZWRoYXQuY29tL2NhL29jc3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjCBgQYDVR0fBHoweDB2oD6gPIY6aHR0cDovL2lwYS1jYS5sYWIuZW5nLmJsci5yZWRoYXQuY29tL2lwYS9jcmwvTWFzdGVyQ1JMLmJpbqI0pDIwMDEOMAwGA1UEChMFaXBhY2ExHjAcBgNVBAMTFUNlcnRpZmljYXRlIEF1dGhvcml0eTAdBgNVHQ4EFgQUPzq8RWuhwGm2HiIMjKAYxhNWLxQwDQYJKoZIhvcNAQELBQADggEBAMUbHcITmz99UoJhVGnc8VBJ5Kyy1BDlxcHXD7HxrrLbvVw5Bb5TYDj0DwEjsWcxCTx2s3GlKdhwjoy69swD9X3N6yXwtlSmpFvmdvG5E9uL6bceA46oehYx7ITsDXGRxbURKPsGvp1MSlNWbJ6hAYBKgq6Mc7qTkjCONUW09Hrcx0IFXSva39GsZ4eJttpPzGZpljuAdWHJlDBTbr0ABdePdXSGMftQErWgjIDiNpaGtBOmo+0ozX86ZKkY5CenVmLazwTlckIzbjseUfKdeYyOyv7MmqA/uSKz2wGWdkcfLcn3H+8xRqun154yQDZ0NSmx904gKCleyvkTpU/7DtY=
Principal name: host/rhsbb1.shanks.com@shanks.COM
Password: False
Keytab: True
Managed by: rhsbb1.shanks.com
Subject: CN=rhsbb1.shanks.com,O=shanks.COM
Serial Number: 13
Serial Number (hex): 0xD
Issuer: CN=Certificate Authority,O=shanks.COM
Not Before: Tue Aug 13 21:23:22 2013 UTC
Not After: Fri Aug 14 21:23:22 2015 UTC
Fingerprint (MD5): 4b:a4:4d:e3:ba:fc:f5:51:72:7e:10:97:4c:84:02:ce
Fingerprint (SHA1): 31:8e:44:56:a9:7d:79:57:22:66:0b:87:0c:be:e1:7d:9b:60:1c:dd
SSH public key fingerprint: D0:F4:B9:0B:2B:4E:FD:53:A2:C4:7F:E5:1E:2D:5A:A2 (ssh-dss),
9C:72:ED:73:2C:C9:BB:85:D3:11:00:60:A2:86:A0:38 (ssh-rsa)
[root@rhsauto007 ~]#

Create GlusterFS volume

# gluster vol create export replica 2 xx.yy.zz.76:/rhs/bricks/export xx.yy.zz.72:/rhs/bricks/export xx.yy.zz.66:/rhs/bricks/export xx.yy.zz.67:/rhs/bricks/export
volume create: export: success: please start the volume to access data
# gluster vol start export
volume start: export: success

setup automount with -fstype=nfs

Once automount maps and keys are added it should look like:

[root@rhsauto007 ~]# ipa automountlocation-tofiles default
/etc/auto.master:
/- /etc/auto.direct
/home /etc/auto.home
---------------------------
/etc/auto.direct:
---------------------------
/etc/auto.home:
* -fstype=nfs rhsbb1.shanks.com:/export/&

maps not connected to /etc/auto.master:
[root@rhsauto007 ~]#

In your client machine, execute ipa-client-automount

[root@rhsauto011 ~]# ipa-client-automount --location=home
Searching for IPA server...
IPA server: DNS discovery
Location: home
Continue to configure the system with these values? [no]: yes
Configured /etc/nsswitch.conf
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs
[root@rhsauto011 ~]#

Verifying our automount of user home directory using GlusterFS

On IPA server
[root@rhsauto007 ~]# ipa user-add user02
First name: user
Last name: 02
-------------------
Added user "user02"
-------------------
User login: user02
First name: user
Last name: 02
Full name: user 02
Display name: user 02
Initials: u0
Home directory: /home/user02
GECOS field: user 02
Login shell: /bin/sh
Kerberos principal: user02@shanks.COM
Email address: user02@shanks.com
UID: 1661400004
GID: 1661400004
Password: False
Member of groups: ipausers
Kerberos keys available: False
[root@rhsauto007 ~]#

[root@rhsauto007 ~]# mount -t nfs rhsbb1.shanks.com:/export/ /home/
[root@rhsauto007 ~]# cd /home/
[root@rhsauto007 home]# ls

[root@rhsauto007 home]# mkdir user02
[root@rhsauto007 home]# chown 1661400004.1661400004 user02
[root@rhsauto007 home]# cp /etc/skel/.* user02/
cp: omitting directory ‘/etc/skel/.’
cp: omitting directory ‘/etc/skel/..’
[root@rhsauto007 home]#

[root@rhsauto007 ~]# umount /home

Note: pam_oddjob_mkhomedir (or pam_mkhomedir) fails to create user home directory.

On the client side

[root@rhsauto011 ~]# su - user02
-sh-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora_rhsauto011-root 45G 2.0G 41G 5% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 412K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
tmpfs 2.0G 12K 2.0G 1% /tmp
/dev/vda1 477M 69M 379M 16% /boot
rhsbb1.shanks.com:/export/user02 100G 17G 84G 17% /home/user02
-sh-4.2$

In autofs log you should see something the following:
attempting to mount entry /home/user02
lookup_mount: lookup(sss): looking up user02
lookup_mount: lookup(sss): user02 -> -fstype=nfs rhsbb1.shanks.com:/export/&
parse_mount: parse(sun): expanded entry: -fstype=nfs rhsbb1.shanks.com:/export/user02
parse_mount: parse(sun): gathered options: fstype=nfs
parse_mount: parse(sun): dequote("rhsbb1.shanks.com:/export/user02") -> rhsbb1.shanks.com:/export/user02
parse_mount: parse(sun): core of entry: options=fstype=nfs, loc=rhsbb1.shanks.com:/export/user02
sun_mount: parse(sun): mounting root /home, mountpoint user02, what rhsbb1.shanks.com:/export/user02, fstype nfs, options
mount_mount: mount(nfs): root=/home name=user02 what=rhsbb1.shanks.com:/export/user02, fstype=nfs, options=
mount_mount: mount(nfs): nfs options="", nobind=0, nosymlink=0, ro=0
get_nfs_info: called with host rhsbb1.shanks.com(xx.yy.zz.76) proto 6 version 0x40
get_nfs_info: called with host rhsbb1.shanks.com(xx.yy.zz.76) proto 6 version 0x70
get_nfs_info: nfs v3 rpc ping time: 0.000733
get_nfs_info: host rhsbb1.shanks.com cost 732 weight 0
get_nfs_info: called with host rhsbb1.shanks.com(xx.yy.zz.76) proto 17 version 0x70
prune_host_list: selected subset of hosts that support NFS3 over TCP
mount_mount: mount(nfs): calling mkdir_path /home/user02
mount_mount: mount(nfs): calling mount -t nfs rhsbb1.shanks.com:/export/user02 /home/user02
spawn_mount: mtab link detected, passing -n to mount
mount_mount: mount(nfs): mounted rhsbb1.shanks.com:/export/user02 on /home/user02
dev_ioctl_send_ready: token = 172
mounted /home/user02
st_expire: state 1 path /home
expire_proc: exp_proc = 140336447108864 path /home
expire_proc_indirect: expire /home/user02

setup automount with -fstype=glusterfs

Once automount maps and keys are added, it should look like
[root@rhsauto007 ~]# ipa automountlocation-tofiles default
/etc/auto.master:
/- /etc/auto.direct
---------------------------
/etc/auto.direct:
/home -fstype=glusterfs rhsbb1.shanks.com:/export

maps not connected to /etc/auto.master:
[root@rhsauto007 ~]#

In your client machine, execute ipa-client-automount

[root@rhsauto011 ~]# ipa-client-automount --location=home
Searching for IPA server...
IPA server: DNS discovery
Location: home
Continue to configure the system with these values? [no]: yes
Configured /etc/nsswitch.conf
Configured /etc/sysconfig/nfs
Configured /etc/idmapd.conf
Started rpcidmapd
Started rpcgssd
Restarting sssd, waiting for it to become available.
Started autofs
[root@rhsauto011 ~]#

Verifying our automount of user home directory using GlusterFS

On IPA server
[root@rhsauto007 ~]# ipa user-add user03
First name: user
Last name: 03
-------------------
Added user "user03"
-------------------
User login: user03
First name: user
Last name: 03
Full name: user 03
Display name: user 03
Initials: u0
Home directory: /home/user03
GECOS field: user 03
Login shell: /bin/sh
Kerberos principal: user03@shanks.COM
Email address: user03@shanks.com
UID: 1661400005
GID: 1661400005
Password: False
Member of groups: ipausers
Kerberos keys available: False
[root@rhsauto007 ~]#

[root@rhsauto007 ~]# mount -t nfs rhsbb1.shanks.com:/export/ /home/
[root@rhsauto007 ~]# cd /home/
[root@rhsauto007 home]# mkdir user03
[root@rhsauto007 home]# chown 1661400005.1661400005 user03
[root@rhsauto007 home]# cp /etc/skel/.* user03/
cp: omitting directory ‘/etc/skel/.’
cp: omitting directory ‘/etc/skel/..’
[root@rhsauto007 home]# cd
[root@rhsauto007 ~]# umount /home/

Note: pam_oddjob_mkhomedir (or pam_mkhomedir) fails to create user home directory.

On the client side

[root@rhsauto011 ~]# su - user03
-sh-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora_rhsauto011-root 45G 2.0G 41G 5% /
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 412K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
tmpfs 2.0G 12K 2.0G 1% /tmp
/dev/vda1 477M 69M 379M 16% /boot
rhsbb1.shanks.com:/export 100G 17G 84G 17% /home
-sh-4.2$ pwd
/home/user03
-sh-4.2$

In autofs logs you should see the following:
attempting to mount entry /home
lookup_mount: lookup(sss): looking up /home
lookup_mount: lookup(sss): /home -> -fstype=glusterfs rhsbb1.shanks.com:/export
parse_mount: parse(sun): expanded entry: -fstype=glusterfs rhsbb1.shanks.com:/export
parse_mount: parse(sun): gathered options: fstype=glusterfs
parse_mount: parse(sun): dequote("rhsbb1.shanks.com:/export") -> rhsbb1.shanks.com:/export
parse_mount: parse(sun): core of entry: options=fstype=glusterfs, loc=rhsbb1.shanks.com:/export
sun_mount: parse(sun): mounting root /home, mountpoint /home, what rhsbb1.shanks.com:/export, fstype glusterfs, options
do_mount: rhsbb1.shanks.com:/export /home type glusterfs options using module generic
mount_mount: mount(generic): calling mkdir_path /home
mount_mount: mount(generic): calling mount -t glusterfs rhsbb1.shanks.com:/export /home
spawn_mount: mtab link detected, passing -n to mount
>> extra arguments at end (ignored)
mount_mount: mount(generic): mounted rhsbb1.shanks.com:/export type glusterfs on /home
dev_ioctl_send_ready: token = 176
mounted /home

Here we are mounting the entire /export on /home instead of per user as in the case of -fstype=nfs.

Launching a Windows instance in Openstack Grizzly stored on GlusterFS volume and attaching Cinder volume.

2013-08-13T15:57:00.000+05:30

This blog is about how to launch Windows instance and attach cinder volume to it. For howto integrate Openstack with GlusterFS, refer http://gsr-linux.blogspot.in/2013/07/glusterfs-integration-with-openstack.html

My setup:
+--------------+
| +-|-----------------+
| D | | |
| I | | GlusterFS Node 1|
| S | | |
| T +-|-----------------+
+---------|-+ R +-|-----------------+
| | | I | | |
|---------|-| B | | GlusterFS Node 2|
| | | U | | |
| | | T +-|-----------------+
|openstack| | E +-|-----------------+
| | | | | |
|grizlly | | R | | GlusterFS Node 3|
| | | E | | |
| | | P +-|-----------------+
| | | L +-|-----------------+
+---------|-+ I | | |
| C | | |
| A | | GlusterFS Node 4|
| T | | |
| E +-|-----------------+
| |
+--------------+

GlusterFS nodes exporting distribute replicate volume to store glance images and cinder volume for Openstack Grizzly

GlusterFS volume information:
# gluster vol info cinder-vol
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 2f4edaef-678b-492a-b972-bd95c1c490a3
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
[root@dhcpzz-183 ~]#

# gluster vol info glance-vol

Volume Name: glance-vol
Type: Distributed-Replicate
Volume ID: eedd5254-e0ca-4173-98eb-45eaef738010
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick3/glance-vol
Brick2: xx.yy.zz.223:/rhs/brick3/glance-vol
Brick3: xx.yy.zz.183:/rhs/brick4/glance-vol
Brick4: xx.yy.zz.223:/rhs/brick4/glance-vol
Options Reconfigured:
storage.owner-gid: 161
storage.owner-uid: 161
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

Downloading and launching instance

Let's launch this instance.

~(keystone_admin)]# nova boot --flavor 3 --image cc1170ea-fdc4-4c35-9a29-4bee70d9fa99 "Windows Server 2012 instance2"

+-------------------------------------+--------------------------------------+

| Property | Value |

+-------------------------------------+--------------------------------------+

| status | BUILD |

| updated | 2013-08-13T03:28:02Z |

| OS-EXT-STS:task_state | scheduling |

| OS-EXT-SRV-ATTR:host | None |

| key_name | None |

| image | Windows2012 |

| hostId | |

| OS-EXT-STS:vm_state | building |

| OS-EXT-SRV-ATTR:instance_name | instance-0000000b |

| OS-EXT-SRV-ATTR:hypervisor_hostname | None |

| flavor | m1.medium |

| id | 330dc6fb-92fa-4d17-9a6f-56891da8fef5 |

| security_groups | [{u'name': u'default'}] |

| user_id | 00a3bbfa022f4a0abba711a619b88bee |

| name | Windows Server 2012 instance2 |

| adminPass | sVePJ9jY6ZBA |

| tenant_id | ac0bd9733a08420ab6830223a8e8252b |

| created | 2013-08-13T03:28:02Z |

| OS-DCF:diskConfig | MANUAL |

| metadata | {} |

| accessIPv4 | |

| accessIPv6 | |

| progress | 0 |

| OS-EXT-STS:power_state | 0 |

| OS-EXT-AZ:availability_zone | nova |

| config_drive | |

+-------------------------------------+--------------------------------------+

Once the "status" is "active", access the console to change the password of the "Administrator" user from the Horizon dashboard.

Creating and attaching cinder volume

Create a cinder volume that should be attached to this Windows instance.

~(keystone_admin)]# cinder create --display-name shanks-windows-vol 5
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-08-13T03:51:15.285543 |
| display_description | None |
| display_name | shanks-windows-vol |
| id | 9bea8bce-19ff-4a97-81d0-b7619e3d0ca3 |
| metadata | {} |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+

Attach Openstack cinder volume to Windows instance

~(keystone_admin)]# nova volume-attach 330dc6fb-92fa-4d17-9a6f-56891da8fef5 9bea8bce-19ff-4a97-81d0-b7619e3d0ca3 auto

+----------+--------------------------------------+

| Property | Value |

+----------+--------------------------------------+

| device | /dev/vdb |

| serverId | 330dc6fb-92fa-4d17-9a6f-56891da8fef5 |

| id | 9bea8bce-19ff-4a97-81d0-b7619e3d0ca3 |

| volumeId | 9bea8bce-19ff-4a97-81d0-b7619e3d0ca3 |

+----------+--------------------------------------+

Launch "Computer Management" in the Windows instance and you should see unallocated "Disk 1" which is also offline because of policy set by administrator.

Launch command prompt in Windows and execute "diskpart"

Clear the read-only attribute

Time to get the disk online

Once the disk in online, re-launch "Computer Management" from Administrative Tools and navigate to "Disk Management", you should see the "Initialize Disk" pop-up.

You should be able to perform a quick format on this disk now.

You have a drive that can be used which actually resides on GlusterFS volume.

Encryption using dm-crypt/LUKS on disk served for GlusterFS bricks.

2013-08-07T18:01:00.000+05:30

Experiment with Linux Unified Key Setup (LUKS) which is a specification for block device encryption. Here we will try to encrypt a logical volume and then have xfs filesystem on it to serve GlusterFS bricks.

Create block devices you want to encrypt:

[root@localhost ~]# pvcreate --dataalignment 2560k /dev/vdb

Physical volume "/dev/vdb" successfully created

[root@localhost ~]# vgcreate RHS_vg1 /dev/vdb

Volume group "RHS_vg1" successfully created

[root@localhost ~]# lvcreate -l 100%FREE -n RHS_lv1 RHS_vg1

Logical volume "RHS_lv1" created

Format the device as a dm-crypt/LUKS encrypted device

[root@localhost ~]# cryptsetup luksFormat /dev/mapper/RHS_vg1-RHS_lv1

WARNING!

========

This will overwrite data on /dev/mapper/RHS_vg1-RHS_lv1 irrevocably.

Are you sure? (Type uppercase yes): YES

Enter LUKS passphrase:

Verify passphrase:

[root@localhost ~]#

[root@localhost ~]# cryptsetup isLuks /dev/mapper/RHS_vg1-RHS_lv1 && echo success

success

To see a summary of the encryption information for the device

[root@localhost ~]# cryptsetup luksDump /dev/mapper/RHS_vg1-RHS_lv1

LUKS header information for /dev/mapper/RHS_vg1-RHS_lv1

Version: 1

Cipher name: aes

Cipher mode: cbc-essiv:sha256

Hash spec: sha1

Payload offset: 4096

MK bits: 256

MK digest: 37 41 be c4 72 8f 57 a4 35 0d 4b 97 7d de 8e 86 40 a8 bb 99

MK salt: 12 ab 41 2c ee 8f 96 f7 8b 5d b4 b9 49 99 ad 9d

39 3a bd 77 a3 cd 03 d6 3b bf 31 20 2c 01 d9 1a

MK iterations: 24250

UUID: 6f2db449-7096-4f59-a4f2-ccd3d6b30216

Key Slot 0: ENABLED

Iterations: 97366

Salt: b3 d7 06 16 8b 2e 6a 4b bd 11 60 c7 71 e2 0e 9d

34 72 b1 91 24 17 39 fe bc bf 0b b0 a5 05 05 5e

Key material offset: 8

AF stripes: 4000

Key Slot 1: DISABLED

Key Slot 2: DISABLED

Key Slot 3: DISABLED

Key Slot 4: DISABLED

Key Slot 5: DISABLED

Key Slot 6: DISABLED

Key Slot 7: DISABLED

[root@localhost ~]#

[root@localhost ~]# cryptsetup luksOpen /dev/mapper/RHS_vg1-RHS_lv1 bricks

Enter passphrase for /dev/mapper/RHS_vg1-RHS_lv1:

[root@localhost ~]#

[root@localhost ~]# dmsetup info bricks

Name: bricks

State: ACTIVE

Read Ahead: 256

Tables present: LIVE

Open count: 0

Event number: 0

Major, minor: 253, 4

Number of targets: 1

UUID: CRYPT-LUKS1-6f2db44970964f59a4f2ccd3d6b30216-bricks

[root@localhost ~]#

Formatting the disk with xfs filesystem and mounting on a directory that would serve as a basedir for GlusterFS bricks.

[root@localhost ~]# mkfs.xfs -i size=512 /dev/mapper/bricks

[root@localhost ~]# mkdir -p /rhs/bricks

[root@localhost ~]# mount /dev/mapper/bricks /rhs/bricks/

Mapping information to /etc/crypttab

/etc/crypttab:

bricks /dev/mapper/RHS_vg1-RHS_lv1 none

Updating /etc/fstab

/dev/mapper/bricks /rhs/bricks xfs defaults 1 2

Verifying:

reboot:

Setting hostname localhost.localdomain: [ OK ]

device-mapper: multipath: version 1.5.0 loaded

Setting up Logical Volume Management: 1 logical volume(s) in volume group "RHS_vg1" now active

2 logical volume(s) in volume group "VolGroup" now active

[ OK ]

Enter passphrase for /dev/mapper/RHS_vg1-RHS_lv1:

(Enter passphrase to continue)

[root@localhost ~]# blkid -o full

/dev/vda1: UUID="4f42eeb2-3a08-4efe-afd6-749be6e027f2" TYPE="ext4"

/dev/vda2: UUID="32e98b72-cd13-4195-b16b-054827775c6c" TYPE="crypto_LUKS"

/dev/vdb1: UUID="RdGJcQ-FH9o-oPqo-yBON-0WNy-lOCE-xx0X0w" TYPE="LVM2_member"

/dev/mapper/luks-32e98b72-cd13-4195-b16b-054827775c6c: UUID="3sB1VE-go5j-oBMW-U5tV-A0m6-CJtJ-Rg674c" TYPE="LVM2_member"

/dev/mapper/VolGroup-lv_root: UUID="68773eaa-0cba-4f1d-ac3e-a5b98c905bcc" TYPE="ext4"

/dev/mapper/VolGroup-lv_swap: UUID="0549be15-4779-4c65-a6f1-8d2433d217f8" TYPE="swap"

/dev/mapper/RHS_vg1-RHS_lv1: UUID="6f2db449-7096-4f59-a4f2-ccd3d6b30216" TYPE="crypto_LUKS"

/dev/mapper/bricks: UUID="fdc1a50d-27af-47d5-8dc9-d5cd11df683e" TYPE="xfs"

[root@localhost ~]#

Automatically unlocking LUKS encrypted drives with a keyfile during boot

[root@localhost ~]# dd if=/dev/urandom of=/root/keyfile-bricks bs=32 count=1

1+0 records in

1+0 records out

32 bytes (32 B) copied, 5.922e-05 s, 540 kB/s

[root@localhost ~]#

[root@localhost ~]# cryptsetup luksAddKey /dev/mapper/RHS_vg1-RHS_lv1 /root/keyfile-bricks

Enter any passphrase:

[root@localhost ~]#

[root@localhost ~]# chmod 600 /root/keyfile-bricks

Update /etc/crypttab as:

bricks /dev/mapper/RHS_vg1-RHS_lv1 /root/keyfile-bricks luks

Create GlusterFS volume as you would normally create

creating first volume for cinder
# gluster vol create shanks-vol replica 2 \
xx.yy.zz.183:/rhs/bricks/shanks-vol \
xx.yy.zz.223:/rhs/bricks/shanks-vol \
xx.yy.zz.183:/rhs/bricks/shanks-vol \
xx.yy.zz.223:/rhs/bricks/shanks-vol

# gluster vol start shanks-vol
volume start: shanks-vol: success

Multiple GlusterFS backend driver integrated with OpenStack Cinder.

2013-08-01T19:19:00.001+05:30

Addressing how to configure multiple GlusterFS backend driver to an existing OpenStack setup.

Few assumptions:

1. that you already have GlusterFS nodes installed.
2. that you already have an existing Openstack setup.

GlusterFS setup:

create 2 distributed replicate volumes.

creating first volume for cinder
# gluster vol create cinder-vol replica 2 \
xx.yy.zz.183:/rhs/brick1/cinder-vol \
xx.yy.zz.223:/rhs/brick1/cinder-vol \
xx.yy.zz.183:/rhs/brick2/cinder-vol \
xx.yy.zz.223:/rhs/brick2/cinder-vol

# gluster vol set cinder-vol group virt
# gluster vol set cinder-vol storage.owner-uid 165
# gluster vol set cinder-vol storage.owner-gid 165

# gluster vol info cinder-vol
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 2f4edaef-678b-492a-b972-bd95c1c490a3
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

# gluster vol start cinder-vol
volume start: cinder-vol: success

Creating second volume for cinder
# gluster vol create cinder-vol2 replica 2 \
xx.yy.zz.183:/rhs/brick1/cinder-vol2 \
xx.yy.zz.223:/rhs/brick1/cinder-vol2 \
xx.yy.zz.183:/rhs/brick2/cinder-vol2 \
xx.yy.zz.223:/rhs/brick2/cinder-vol2

# gluster vol set cinder-vol2 group virt
# gluster vol set cinder-vol2 storage.owner-uid 165
# gluster vol set cinder-vol2 storage.owner-gid 165

# gluster vol info cinder-vol2
Volume Name: cinder-vol2
Type: Distributed-Replicate
Volume ID: a3d4831a-d754-4072-9a3a-c6641759ec09
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol2
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol2
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol2
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol2
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

# gluster vol start cinder-vol2
volume start: cinder-vol2: success

From your allinone openstack node verify that the volumes are available to rule out any network issue.
# showmount -e xx.yy.zz.223
Export list for xx.yy.zz.223:
/cinder-vol *
/cinder-vol2 *

Configuring openstack to use multiple GlusterFS drivers

Defining the names of the groups for multiple volume backends
# openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends GLUSTERFS_DRIVER1,GLUSTERFS_DRIVER2

Specifying cinder.volume.drivers.glusterfs options in /etc/cinder/cinder.conf for both the drivers

GLUSTERFS_DRIVER1:
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 glusterfs_shares_config /etc/cinder/cinder-volume1.conf

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 glusterfs_mount_point_base /var/lib/cinder/cinder-volume1

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER1 volume_backend_name GLUSTERFS_cinder_volume1

GLUSTERFS_DRIVER2:
# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 glusterfs_shares_config /etc/cinder/cinder-volume2.conf

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 glusterfs_mount_point_base /var/lib/cinder/cinder-volume2

# openstack-config --set /etc/cinder/cinder.conf GLUSTERFS_DRIVER2 volume_backend_name GLUSTERFS_cinder_volume2

Adding GlusterFS volumes in shares config
# cat > /etc/cinder/cinder-volume1.conf << EOF
> xx.yy.zz.183:cinder-vol
> EOF

# cat > /etc/cinder/cinder-volume2.conf << EOF
> xx.yy.zz.223:cinder-vol
> EOF

Just restart cinder-volume service
# service openstack-cinder-volume restart

# mount | grep cinder
xx.yy.zz.183:cinder-vol on /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
xx.yy.zz.223:cinder-vol2 on /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

All set!!!

# cinder-manage host list
host zone
2013-08-01 18:02:29 DEBUG [cinder.utils] backend <module 'cinder.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.pyc'>
dhcpzz-146.shanks.com nova
dhcpzz-146.shanks.com@GLUSTERFS_DRIVER1 nova
dhcpzz-146.shanks.com@GLUSTERFS_DRIVER2 nova

NOTE: The volume_type must be defined in the volume_type table in the database. When creating a volume for a particular backend (or group of > 1 backends) the volume_type is passed in. There can be > 1 backend per volume_type, and the capacity scheduler kicks in and keeps the backends of a particular volume_type even. In order to group a volume_type to a backend, you must define extra specs for the type. src: https://wiki.openstack.org/wiki/Cinder-multi-backend#Volume_Type

# cinder type-create GLUSTERFS_cinder_volume1
# cinder type-create GLUSTERFS_cinder_volume2

(Thanks to Shilpa M Jagannath for point out an error above which is now fixed.)

# cinder type-key glusterfs_cinder_volume1 set volume_backend_name=glusterfs_cinder_volume1
# cinder type-key glusterfs_cinder_volume2 set volume_backend_name=glusterfs_cinder_volume2

# cinder extra-specs-list
+--------------------------------------+--------------------------+-------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+--------------------------+-------------------------------------------------------+
| 50b7f795-29c0-4fe8-8ca3-cf66e2f34886 | glusterfs_cinder_volume1 | {u'volume_backend_name': u'glusterfs_cinder_volume1'} |
| cea39f0e-cb83-4c13-b712-ae29551da19f | glusterfs_cinder_volume2 | {u'volume_backend_name': u'glusterfs_cinder_volume2'} |
+--------------------------------------+--------------------------+-------------------------------------------------------+

Usage and tests

(keystone_admin)]# cinder create --display-name shanks-vol1 --volume-type glusterfs_cinder_volume1 3
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-08-01T13:22:16.810225 |
| display_description | None |
| display_name | shanks-vol1 |
| id | 312d61b8-9abc-4c7c-915a-7e9dfff0b098 |
| metadata | {} |
| size | 3 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | glusterfs_cinder_volume1 |
+---------------------+--------------------------------------+

# ll -h /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
-rw-rw-rw-. 1 root root 3.0G Aug 1 18:52 /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098

# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
getfattr: Removing leading '/' from absolute path names
# file: var/lib/cinder/cinder-volume1/586c24173ac3ab5d1d43aed1f113d9f6/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098
trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol-dht> (<REPLICATE:cinder-vol-replicate-1> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol/volume-312d61b8-9abc-4c7c-915a-7e9dfff0b098>))"

(keystone_admin)]# cinder create --display-name shanks-vol2 --volume-type glusterfs_cinder_volume2 4
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-08-01T13:23:47.948160 |
| display_description | None |
| display_name | shanks-vol2 |
| id | 152fcb35-31ca-4e66-a87e-4b921e3171aa |
| metadata | {} |
| size | 4 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | glusterfs_cinder_volume2 |
+---------------------+--------------------------------------+

# ls -lh /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
-rw-rw-rw-. 1 root root 4.0G Aug 1 18:53 /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa

# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
getfattr: Removing leading '/' from absolute path names
# file: var/lib/cinder/cinder-volume2/2277a5d6ce46f3cd52771e1314f74ce5/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa
trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol2-dht> (<REPLICATE:cinder-vol2-replicate-1> <POSIX(/rhs/brick2/cinder-vol2):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol2/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa> <POSIX(/rhs/brick2/cinder-vol2):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol2/volume-152fcb35-31ca-4e66-a87e-4b921e3171aa>))"

Add more GlusterFS integrated OpenStack Cinder nodes

2013-07-31T18:45:00.000+05:30

Add more GlusterFS integrated OpenStack Cinder nodes to an existing OpenStack setup.

GlusterFS setup

Continuing from http://gsr-linux.blogspot.in/2013/07/glusterfs-integration-with-openstack.html. I wanted to scale out OpenStack Cinder nodes integrated with new GlusterFS volume.

Creating a new distributed-replicate volume to store cinder images.

~]# gluster vol create cinder-vol2 replica 2 \

xx.yy.zz.183:/rhs/brick1/cinder-vol2 \

xx.yy.zz.223:/rhs/brick1/cinder-vol2 \

xx.yy.zz.183:/rhs/brick2/cinder-vol2 \

xx.yy.zz.223:/rhs/brick2/cinder-vol2

volume create: cinder-vol2: success: please start the volume to access data

~]# gluster vol set cinder-vol2 group virt

volume set: success

~]# gluster vol set cinder-vol2 storage.owner-uid 165

volume set: success

~]# gluster vol set cinder-vol2 storage.owner-gid 165

volume set: success

Verifying newly created cinder volume (cinder-vol2)

~]# gluster vol info cinder-vol2

Volume Name: cinder-vol2

Type: Distributed-Replicate

Volume ID: a3d4831a-d754-4072-9a3a-c6641759ec09

Status: Created

Number of Bricks: 2 x 2 = 4

Transport-type: tcp

Bricks:

Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol2

Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol2

Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol2

Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol2

Options Reconfigured:

storage.owner-gid: 165

storage.owner-uid: 165

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

~]# gluster vol start cinder-vol2

volume start: cinder-vol2: success

~]# gluster vol status cinder-vol2

Status of volume: cinder-vol2

Gluster process Port Online Pid

------------------------------------------------------------------------------

Brick xx.yy.zz.183:/rhs/brick1/cinder-vol2 49156 Y 16793

Brick xx.yy.zz.223:/rhs/brick1/cinder-vol2 49156 Y 10807

Brick xx.yy.zz.183:/rhs/brick2/cinder-vol2 49157 Y 16804

Brick xx.yy.zz.223:/rhs/brick2/cinder-vol2 49157 Y 10818

NFS Server on localhost 2049 Y 16816

Self-heal Daemon on localhost N/A Y 16824

NFS Server on xx.yy.zz.223 2049 Y 10830

Self-heal Daemon on xx.yy.zz.223 N/A Y 10837

There are no active volume tasks

Configuring OpenStack Cinder to user new GlusterFS volume

On the new Cinder node do:

~]# yum install openstack-utils openstack-cinder openstack-selinux
~]# yum install glusterfs glusterfs-fuse

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf
~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes

~]# cat > /etc/cinder/shares.conf << EOF
> xx.yy.zz.183:cinder-vol2
> EOF

Configuring to connect to exiting Cinder database

Configuration to connect this new node to an existing cinder database and also to an existing AMQP broker.

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT rpc_backend cinder.openstack.common.rpc.impl_qpid

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT qpid_hostname xx.yy.zz.146

Note: Here xx.yy.zz.146 is my OpenStack setup installed using "packstack --allinone" option.

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT qpid_username guest

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT qpid_password notused

~]# openstack-config --set /etc/cinder/cinder.conf DEFAULT sql_connection mysql://cinder:794153107f0a44d7@xx.yy.zz.146/cinder

Note: sql_connection value can be found in your /etc/cinder/cinder.conf of default cinder node.

All set!!!

~]# service openstack-cinder-volume start

Verifying your addition of new GlusterFS integrated OpenStack Cinder node

[root@dhcpzz-211 ~(keystone_admin)]# mount

/dev/mapper/vg_dhcpzz211-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/vda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

xx.yy.zz.183:cinder-vol2 on /var/lib/cinder/volumes/c1c93c8c420380b0c5bdea9f2eacd8b8 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

[root@dhcpzz-211 ~(keystone_admin)]#

The logs should indicate (provided verbose=true in cinder.conf) that qpid is connected to AMQP server as:

==> /var/log/cinder/volume.log <==

2013-07-31 17:15:31 INFO [cinder.service] Starting 1 workers

2013-07-31 17:15:31 INFO [cinder.service] Started child 3338

2013-07-31 17:15:31 AUDIT [cinder.service] Starting cinder-volume node (version 2013.1.2)

2013-07-31 17:15:32 INFO [cinder.volume.manager] Updating volume status

2013-07-31 17:15:32 INFO [cinder.openstack.common.rpc.impl_qpid] Connected to AMQP server on xx.yy.zz.146:5672

Now cinder-manage should list both the nodes:

~(keystone_admin)]# cinder-manage host list

host zone

dhcpzz-146.shanks.com nova

dhcpzz-211.shanks.com nova

Lets test by creating cinder volumes

[root@dhcpzz-146 ~(keystone_admin)]# cinder create --display-name vol1 20

+---------------------+--------------------------------------+

| Property | Value |

+---------------------+--------------------------------------+

| attachments | [] |

| availability_zone | nova |

| bootable | false |

| created_at | 2013-07-31T11:43:30.979520 |

| display_description | None |

| display_name | vol1 |

| id | 0a3bf28b-0313-4f8f-b49d-d1337bcddac8 |

| metadata | {} |

| size | 20 |

| snapshot_id | None |

| source_volid | None |

| status | creating |

| volume_type | None |

+---------------------+--------------------------------------+

In ==> /var/log/cinder/volume.log <== of your first/default cinder node you should see (provided verbose=true in cinder.conf):

2013-07-31 17:23:03 INFO [cinder.volume.manager] volume volume-35e1f918-9c82-4e13-86c4-affe1d9f7091: creating

2013-07-31 17:23:03 INFO [cinder.volume.drivers.glusterfs] casted to xx.yy.zz.183:cinder-vol

2013-07-31 17:23:03 INFO [cinder.volume.manager] volume volume-35e1f918-9c82-4e13-86c4-affe1d9f7091: created successfully

[root@dhcpzz-146 ~(keystone_admin)]# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/volumes/586c24173ac3ab5d1d43aed1f113d9f6/volume-35e1f918-9c82-4e13-86c4-affe1d9f7091

getfattr: Removing leading '/' from absolute path names

# file: var/lib/cinder/volumes/586c24173ac3ab5d1d43aed1f113d9f6/volume-35e1f918-9c82-4e13-86c4-affe1d9f7091

trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol-dht> (<REPLICATE:cinder-vol-replicate-1> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol/volume-35e1f918-9c82-4e13-86c4-affe1d9f7091> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol/volume-35e1f918-9c82-4e13-86c4-affe1d9f7091>))"

[root@dhcpzz-211 ~(keystone_admin)]# cinder create --display-name vol2 20

+---------------------+--------------------------------------+

| Property | Value |

+---------------------+--------------------------------------+

| attachments | [] |

| availability_zone | nova |

| bootable | false |

| created_at | 2013-07-31T11:54:00.245068 |

| display_description | None |

| display_name | vol2 |

| id | c384cc7b-f888-43e5-979c-915898fc70e8 |

| metadata | {} |

| size | 20 |

| snapshot_id | None |

| source_volid | None |

| status | creating |

| volume_type | None |

+---------------------+--------------------------------------+

In ==> /var/log/cinder/volume.log <== of your newly added GlusterFS integrated OpenStack Cinder node you should see (provided verbose=true in cinder.conf):

2013-07-31 17:24:03 INFO [cinder.volume.manager] volume volume-c384cc7b-f888-43e5-979c-915898fc70e8: creating

2013-07-31 17:24:03 INFO [cinder.volume.drivers.glusterfs] casted to xx.yy.zz.183:cinder-vol2

2013-07-31 17:24:04 INFO [cinder.volume.manager] volume volume-c384cc7b-f888-43e5-979c-915898fc70e8: created successfully

[root@dhcpzz-211 ~(keystone_admin)]# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/volumes/c1c93c8c420380b0c5bdea9f2eacd8b8/volume-c384cc7b-f888-43e5-979c-915898fc70e8

getfattr: Removing leading '/' from absolute path names

# file: var/lib/cinder/volumes/c1c93c8c420380b0c5bdea9f2eacd8b8/volume-c384cc7b-f888-43e5-979c-915898fc70e8

trusted.glusterfs.pathinfo="(<DISTRIBUTE:cinder-vol2-dht> (<REPLICATE:cinder-vol2-replicate-0> <POSIX(/rhs/brick1/cinder-vol2):dhcpzz-223.shanks.com:/rhs/brick1/cinder-vol2/volume-c384cc7b-f888-43e5-979c-915898fc70e8> <POSIX(/rhs/brick1/cinder-vol2):dhcpzz-183.shanks.com:/rhs/brick1/cinder-vol2/volume-c384cc7b-f888-43e5-979c-915898fc70e8>))"

GlusterFS integration with openstack-cinder and openstack-glance (Grizzly) using packstack --allinone

2013-07-30T16:54:00.000+05:30

Setup GlusterFS server and integrate with openstack-cinder and openstack-glance (Grizzly) :

GlusterFS setup:

Creating a distributed-replicate gluster volume for cinder

# gluster vol create cinder-vol replica 2 \
xx.yy.zz.183:/rhs/brick1/cinder-vol \
xx.yy.zz.223:/rhs/brick1/cinder-vol \
xx.yy.zz.183:/rhs/brick2/cinder-vol \
xx.yy.zz.223:/rhs/brick2/cinder-vol

volume create: cinder-vol: success: please start the volume to access data

Creating a distributed-replicate gluster volume for glance

# gluster vol create glance-vol replica 2 xx.yy.zz.183:/rhs
/brick3/glance-vol xx.yy.zz.223:/rhs/brick3/glance-vol xx.yy.zz.183:/rhs/brick
4/glance-vol xx.yy.zz.223:/rhs/brick4/glance-vol

volume create: glance-vol: success: please start the volume to access data

Setting appropriate volume options for cinder and glance volume

cinder-vol:
# gluster vol set cinder-vol group virt
volume set: success

# gluster vol set cinder-vol storage.owner-uid 165
volume set: success

# gluster vol set cinder-vol storage.owner-gid 165
volume set: success

glance-vol:
# gluster vol set glance-vol group virt
volume set: success

# gluster vol set glance-vol storage.owner-uid 161
volume set: success

# gluster vol set glance-vol storage.owner-gid 161
volume set: success

Verifying the volume before starting it

# gluster vol info cinder-vol
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 2f4edaef-678b-492a-b972-bd95c1c490a3
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick1/cinder-vol
Brick2: xx.yy.zz.223:/rhs/brick1/cinder-vol
Brick3: xx.yy.zz.183:/rhs/brick2/cinder-vol
Brick4: xx.yy.zz.223:/rhs/brick2/cinder-vol
Options Reconfigured:
storage.owner-gid: 165
storage.owner-uid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
[root@dhcpzz-183 ~]#

# gluster vol info glance-vol

Volume Name: glance-vol
Type: Distributed-Replicate
Volume ID: eedd5254-e0ca-4173-98eb-45eaef738010
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: xx.yy.zz.183:/rhs/brick3/glance-vol
Brick2: xx.yy.zz.223:/rhs/brick3/glance-vol
Brick3: xx.yy.zz.183:/rhs/brick4/glance-vol
Brick4: xx.yy.zz.223:/rhs/brick4/glance-vol
Options Reconfigured:
storage.owner-gid: 161
storage.owner-uid: 161
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

Starting both the volumes

# gluster vol start cinder-vol
# gluster vol start glance-vol

Verifying the cinder volume status

# gluster vol status cinder-vol

Status of volume: cinder-vol

Gluster process Port Online Pid

------------------------------------------------------------------------------

Brick xx.yy.zz 183:/rhs/brick1/cinder-vol 49152 Y 3824

Brick xx.yy.zz 223:/rhs/brick1/cinder-vol 49152 Y 3689

Brick xx.yy.zz 183:/rhs/brick2/cinder-vol 49153 Y 3835

Brick xx.yy.zz 223:/rhs/brick2/cinder-vol 49153 Y 3700

NFS Server on localhost 2049 Y 3916

Self-heal Daemon on localhost N/A Y 3924

NFS Server on xx.yy.zz 223 2049 Y 3779

Self-heal Daemon on xx.yy.zz 223 N/A Y 3786

There are no active volume tasks

Verifying the glance volume status

# gluster vol status glance-vol
Status of volume: glance-vol
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick xx.yy.zz 183:/rhs/brick3/glance-vol 49154 Y 3893
Brick xx.yy.zz 223:/rhs/brick3/glance-vol 49154 Y 3756
Brick xx.yy.zz 183:/rhs/brick4/glance-vol 49155 Y 3904
Brick xx.yy.zz 223:/rhs/brick4/glance-vol 49155 Y 3767
NFS Server on localhost 2049 Y 3916
Self-heal Daemon on localhost N/A Y 3924
NFS Server on xx.yy.zz 223 2049 Y 3779
Self-heal Daemon on xx.yy.zz 223 N/A Y 3786

There are no active volume tasks

Configuring Openstack Cinder to use GlusterFS volume:

Installing required packages

# yum install -y openstack-utils openstack-cinder openstack-glance

# yum install -y glusterfs-fuse glusterfs

Setting glusterfs default config parameters in cinder.conf

# openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf

# openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes

Creating /etc/cinder/shares.conf with gluster volume details:

# cat > /etc/cinder/shares.conf << EOF
xx.yy.zz.183:cinder-vol
EOF

Configuring Openstack Glance to use GlusterFS volume:

Ensure filesystem_store_datadir is specified as:

<snip>

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store

# writes image data to

filesystem_store_datadir = /var/lib/glance/images/

</snip>

Create directory under glance as images:

mkdir /var/lib/glance/images

Mount gluster volume on filesystem_store_datadir:

mount -t glusterfs xx.yy.zz.183:/glance-vol /var/lib/glance/images

Update /etc/fstab for persistent mount of glance volume across reboot.

Installing openstack using packstack with --allinone option

# packstack --allinone --debug

Welcome to Installer setup utility

Installing:

Clean Up... [ DONE ]

Setting up ssh keys... [ DONE ]

OS support check... [ DONE ]

Adding pre install manifest entries... [ DONE ]

Adding MySQL manifest entries... [ DONE ]

Adding QPID manifest entries... [ DONE ]

Adding Keystone manifest entries... [ DONE ]

Adding Glance Keystone manifest entries... [ DONE ]

Adding Glance manifest entries... [ DONE ]

Adding Cinder Keystone manifest entries... [ DONE ]

Installing dependencies for Cinder... [ DONE ]

Checking if the Cinder server has a cinder-volumes vg...[ DONE ]

Adding Cinder manifest entries... [ DONE ]

Adding Nova API manifest entries... [ DONE ]

Adding Nova Keystone manifest entries... [ DONE ]

Adding Nova Cert manifest entries... [ DONE ]

Adding Nova Conductor manifest entries... [ DONE ]

Adding Nova Compute manifest entries... [ DONE ]

Adding Nova Scheduler manifest entries... [ DONE ]

Adding Nova VNC Proxy manifest entries... [ DONE ]

Adding Nova Common manifest entries... [ DONE ]

Adding Openstack Network-related Nova manifest entries...[ DONE ]

Adding Quantum API manifest entries... [ DONE ]

Adding Quantum Keystone manifest entries... [ DONE ]

Adding Quantum L3 manifest entries... [ DONE ]

Adding Quantum L2 Agent manifest entries... [ DONE ]

Adding Quantum DHCP Agent manifest entries... [ DONE ]

Adding Quantum Metadata Agent manifest entries... [ DONE ]

Adding OpenStack Client manifest entries... [ DONE ]

Adding Horizon manifest entries... [ DONE ]

Adding Swift Keystone manifest entries... [ DONE ]

Adding Swift builder manifest entries... [ DONE ]

Adding Swift proxy manifest entries... [ DONE ]

Adding Swift storage manifest entries... [ DONE ]

Adding Swift common manifest entries... [ DONE ]

Preparing servers... [ DONE ]

Adding Nagios server manifest entries... [ DONE ]

Adding Nagios host manifest entries... [ DONE ]

Adding post install manifest entries... [ DONE ]

Installing Dependencies... [ DONE ]

Copying Puppet modules and manifests... [ DONE ]

Applying Puppet manifests...

Applying xx.yy.zz.146_prescript.pp

xx.yy.zz.146_prescript.pp : [ DONE ]

Applying xx.yy.zz.146_mysql.pp

Applying xx.yy.zz.146_qpid.pp

xx.yy.zz.146_mysql.pp : [ DONE ]

xx.yy.zz.146_qpid.pp : [ DONE ]

Applying xx.yy.zz.146_keystone.pp

Applying xx.yy.zz.146_glance.pp

Applying xx.yy.zz.146_cinder.pp

xx.yy.zz.146_keystone.pp : [ DONE ]

xx.yy.zz.146_glance.pp : [ DONE ]

xx.yy.zz.146_cinder.pp : [ DONE ]

Applying xx.yy.zz.146_api_nova.pp

xx.yy.zz.146_api_nova.pp : [ DONE ]

Applying xx.yy.zz.146_nova.pp

xx.yy.zz.146_nova.pp : [ DONE ]

Applying xx.yy.zz.146_quantum.pp

xx.yy.zz.146_quantum.pp : [ DONE ]

Applying xx.yy.zz.146_osclient.pp

Applying xx.yy.zz.146_horizon.pp

xx.yy.zz.146_osclient.pp : [ DONE ]

xx.yy.zz.146_horizon.pp : [ DONE ]

Applying xx.yy.zz.146_ring_swift.pp

xx.yy.zz.146_ring_swift.pp : [ DONE ]

Applying xx.yy.zz.146_swift.pp

Applying xx.yy.zz.146_nagios.pp

Applying xx.yy.zz.146_nagios_nrpe.pp

xx.yy.zz.146_swift.pp : [ DONE ]

xx.yy.zz.146_nagios.pp : [ DONE ]

xx.yy.zz.146_nagios_nrpe.pp : [ DONE ]

Applying xx.yy.zz.146_postscript.pp

xx.yy.zz.146_postscript.pp : [ DONE ]

[ DONE ]

**** Installation completed successfully ******

Verifying that glusterfs mount was successful

# mount | grep gluster

xx.yy.zz.183:/glance-vol on /var/lib/glance/images type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

xx.yy.zz.183:cinder-vol on /var/lib/cinder/volumes/586c24173ac3ab5d1d43aed1f113d9f6 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

Verifying cinder volume is mounted on glusterfs volume

~(keystone_admin)]# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/cinder/volumes/586c24173ac3ab5d1d43aed1f113d9f6
getfattr: Removing leading '/' from absolute path names
# file: var/lib/cinder/volumes/586c24173ac3ab5d1d43aed1f113d9f6
trusted.glusterfs.pathinfo="((<DISTRIBUTE:cinder-vol-dht> (<REPLICATE:cinder-vol-replicate-0> <POSIX(/rhs/brick1/cinder-vol):dhcpzz-183.shanks.com:/rhs/brick1/cinder-vol/> <POSIX(/rhs/brick1/cinder-
vol):dhcpzz-223.shanks.com:/rhs/brick1/cinder-vol/>) (<REPLICATE:cinder-vol-replicate-1> <POSIX(/rhs/brick2/cinder-vol):dhcpzz-223.shanks.com:/rhs/brick2/cinder-vol/> <POSIX(/rhs/brick2/
cinder-vol):dhcpzz-183.shanks.com:/rhs/brick2/cinder-vol/>)) (cinder-vol-dht-layout (cinder-vol-replicate-0 0 2147483646) (cinder-vol-replicate-1 2147483647 4294967295)))"

Verifying glance image is created on glusterfs volume

~(keystone_admin)]# glance image-create --name="test" --is-public=true --container-format=ovf --disk-format=qcow2 < f17-x86_64-openstack-sda.qcow2

+------------------+--------------------------------------+

| Property | Value |

+------------------+--------------------------------------+

| checksum | 1f104b5667768964d5df8c4ad1d7cd27 |

| container_format | ovf |

| created_at | 2013-07-30T12:56:05 |

| deleted | False |

| deleted_at | None |

| disk_format | qcow2 |

| id | a66213ee-1a76-4d4a-959d-5df3f8f320ac |

| is_public | True |

| min_disk | 0 |

| min_ram | 0 |

| name | test |

| owner | 84e2f0fac93d402287a8eb97b6ba9711 |

| protected | False |

| size | 251985920 |

| status | active |

| updated_at | 2013-07-30T12:56:49 |

+------------------+--------------------------------------+

~(keystone_admin)]# ls -l /var/lib/glance/images/

total 246080

-rw-r-----. 1 glance glance 251985920 Jul 30 18:26 a66213ee-1a76-4d4a-959d-5df3f8f320ac

~(keystone_admin)]# getfattr -d -etext -m. -n trusted.glusterfs.pathinfo /var/lib/glance/images/a66213ee-1a76-4d4a-959d-5df3f8f320ac

getfattr: Removing leading '/' from absolute path names # file: var/lib/glance/images/a66213ee-1a76-4d4a-959d-5df3f8f320ac

trusted.glusterfs.pathinfo="(<DISTRIBUTE:glance-vol-dht> (<REPLICATE:glance-vol-replicate-1> <POSIX(/rhs/brick4/glance-vol):dhcpzz-223.shanks.com:/rhs/brick4/glance-vol/a66213ee-1a76-4d4a-959d-5df3f

8f320ac> <POSIX(/rhs/brick4/glance-vol):dhcpzz-183.shanks.com:/rhs/brick4/glance-vol/a66213ee-1a76-4d4a-959d-5df3f8f320ac>))"

virt-install example with qcow2 disks

2013-01-24T21:55:00.000+05:30

qemu-img create -f qcow2 /home/vms/vm-disk/vm-disk.img 20G
qemu-img create -f qcow2 /home/vms/vm-disk/vm-1-disk.img 100G
qemu-img create -f qcow2 /home/vms/vm-disk/vm-2-disk.img 100G
qemu-img create -f qcow2 /home/vms/vm-disk/vm-3-disk.img 100G
qemu-img create -f qcow2 /home/vms/vm-disk/vm-4-disk.img 100G

virt-install --connect=qemu:///system \
--network=bridge:br0 \
--extra-args="ks=http://kickstart-server.com/kickstart/20-virtio.ks console=tty0 console=ttyS0,115200" \
--name vm-disk \
--disk path="/home/vms/vm-disk/vm-disk.img,device=disk,bus=virtio" \
--disk path="/home/vms/vm-disk/vm-1-disk.img,device=disk,bus=virtio" \
--disk path="/home/vms/vm-disk/vm-2-disk.img,device=disk,bus=virtio" \
--disk path="/home/vms/vm-disk/vm-3-disk.img,device=disk,bus=virtio" \
--disk path="/home/vms/vm-disk/vm-4-disk.img,device=disk,bus=virtio" \
--ram 4096 \
--vcpus=4 \
--check-cpu \
--accelerate \
--location=http://OS-location.com/ \
--hvm \
--nographics

Resize root partition online

2013-01-23T18:57:00.000+05:30

# pvscan
PV /dev/sda2   VG vg_rhsgpsrv10   lvm2 [29.32 GiB / 0    free]

# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/vg_rhsgpsrv10-lv_root
                      5.7G 1.8G 3.6G 34% /
tmpfs                  24G     0   24G   0% /dev/shm
/dev/sda1             485M   38M 422M   9% /boot

# fdisk -l | grep sdb
Disk /dev/sdb: 32.0 GB, 32017047552 bytes

# fdisk /dev/sdb
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3892, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-3892, default 3892):
Using default value 3892
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created

# vgextend -v vg_rhsgpsrv10 /dev/sdb1
    Checking for volume group "vg_rhsgpsrv10"
    Archiving volume group "vg_rhsgpsrv10" metadata (seqno 3).
    Wiping cache of LVM-capable devices
    Adding physical volume '/dev/sdb1' to volume group 'vg_rhsgpsrv10'
    Volume group "vg_rhsgpsrv10" will be extended by 1 new physical volumes
    Creating volume group backup "/etc/lvm/backup/vg_rhsgpsrv10" (seqno 4).
Volume group "vg_rhsgpsrv10" successfully extended

# pvscan
PV /dev/sda2   VG vg_rhsgpsrv10   lvm2 [29.32 GiB / 0    free]
PV /dev/sdb1   VG vg_rhsgpsrv10   lvm2 [29.81 GiB / 29.81 GiB free]

# lvresize -r -L +29G /dev/mapper/vg_rhsgpsrv10-lv_root
Extending logical volume lv_root to 34.70 GiB
Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_rhsgpsrv10-lv_root is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 3
Performing an on-line resize of /dev/mapper/vg_rhsgpsrv10-lv_root to 9096192 (4k) blocks.
The filesystem on /dev/mapper/vg_rhsgpsrv10-lv_root is now 9096192 blocks long.

# df -hT
Filesystem    Type    Size Used Avail Use% Mounted on
/dev/mapper/vg_rhsgpsrv10-lv_root
              ext4     35G 1.9G   31G   6% /
tmpfs        tmpfs     24G     0   24G   0% /dev/shm
/dev/sda1     ext4    485M   38M 422M   9% /boot

GlusterFS: {path} or a prefix of it is already part of a volume

2012-09-23T17:35:00.000+05:30

For the directory that was formerly part of a volume:

setfattr -x trusted.glusterfs.volume-id $brick-path
setfattr -x trusted.gfid $brick-path
rm -rf $brick-path/.glusterfs

Remove Comments and Empty Lines With Grep

2012-08-07T12:16:00.002+05:30

$ grep -v -e ^# -e ^$ file.txt

Install Google Chrome with YUM on Fedora 17

2012-07-19T12:42:00.000+05:30

Create yum repository:

cat /etc/yum.repos.d/google-chrome.repo

[google-chrome]

name=google-chrome

baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64

enabled=1

gpgcheck=1

shanks ~]$ sudo yum install google-chrome-stable

Boot hangs at "starting udev" - Cannot use 2 vCPUs

2012-07-07T14:30:00.002+05:30

Try adding the following to your grub command line or /etc/grub.conf:


clocksource=acpi_pm

yum versionlock

2012-06-29T16:04:00.000+05:30

versionlock plugin for YUM is for you to prevent YUM from updating the versions or "lock the version" of some selected packages.

yum install yum-plugin-versionlock
yum versionlock 389-ds-base{,-devel,-libs}

This updates /etc/yum/pluginconf.d/versionlock.list file. Once this plugin is installed, every time when YUM starts, it checks for the existence of /etc/yum/pluginconf.d/versionlock.list file.

(28)No space left on device: Parent could not create lock file /etc/httpd/ipa Configuration Failed

2012-06-29T13:40:00.000+05:30

[Fri Jun 29 03:46:59 2012] [crit] (28)No space left on device: Parent could not create lock file /etc/httpd/ipa
Configuration Failed

Above message was detected in /var/log/httpd/error_log and there was definitely enough space.

Problem: There were lots of semaphore arrays owned by apache.
# ipcs -s | grep apache

Solution: for semId in $(ipcs -s | grep apache | cut -f 2 -d ' '); do ipcrm -s $semId; done

Fedora 16 to 17

2012-06-08T01:13:00.000+05:30

For the first time ever with fingers crossed I used "preupgrade" to move from Fedora 16 to Fedora 17.

[root@shanks ~]# yum install preupgrade

[root@shanks ~]# preupgrade

Follow the instructions and stand-by. The total time depends on the speed of your internet connection.

Once the preupgrade is done, it prompts you for a reboot.
Select the "upgrade" option from the grub menu.
The packages get installed

And Fedora 17 is ready!!! :) Wow!!! that was as smooth as a breeze.

Suspend laptop on lid close in fedora

2012-05-25T20:06:00.001+05:30

root@qemu$ gsettings list-schemas | grep -i power


root@qemu$ gsettings list-keys org.gnome.settings-daemon.plugins.power | grep batter
critical-battery-action
idle-dim-battery
lid-close-battery-action
sleep-display-battery
sleep-inactive-battery
sleep-inactive-battery-timeout
sleep-inactive-battery-type
root@qemu$ 


root@qemu$ gsettings get org.gnome.settings-daemon.plugins.power lid-close-battery-action
'suspend'
root@qemu$

find files larger than 1GB

2012-05-03T20:00:00.001+05:30

find . -type f -size +1G -exec ls -lht {} \;

No IPA winsync without "surname" in AD.

2011-11-25T19:07:00.002+05:30

Nice:

- Windows server LDAP schema has objectclass Person which does not have 'surname' attribute as mandatory. I.e. you can have users without surnames.

- RFC4519 defines Person object class that MUST have 'sn' attribute (surname). I.e. all traditional LDAP database persons must have surnames.

As result, if you sync users between the two with, say, FreeIPA winsync plugin, those users without surnames will not be synced.

No unnamed soldiers.

2011-09-15T15:26:00.001+05:30

#!/bin/bash
# Author: Gowrishankar Rajaiyan
# Email: gowrishankar.rajaiyan@gmail.com
# Date: Thu Sep 15 11:46:01 EDT 2011
# Purpose: Script to check if IPv6 is supported in your OS.

test -f /proc/net/if_inet6 && echo "Running kernel is IPv6 ready"
    if [ $? -ne 0 ]; then
        echo "Running kernel is not IPv6 ready"
    fi

lsmod |grep -w 'ipv6' && echo "IPv6 module successfully loaded"
    if [ $? -ne 0 ]; then
        echo "IPv6 module is not loaded"
        echo "execute: modprobe ipv6"
    fi

/sbin/ifconfig -? 2>& 1|grep -qw 'inet6' && echo "utility 'ifconfig' is IPv6-ready"
        if [ $? -ne 0 ]; then
                echo "utility 'ifconfig' is not IPv6-ready"
        fi

/sbin/route -? 2>& 1|grep -qw 'inet6' && echo "utility 'route' is IPv6-ready"
        if [ $? -ne 0 ]; then
                echo "utility 'route' is not IPv6-ready"
        fi

/sbin/ip 2>&1 |grep -qw 'inet6' && echo "utility 'ip' is IPv6-ready"
        if [ $? -ne 0 ]; then
                echo "utility 'ip' is not IPv6-ready"
        fi

# EOF

URL for IPv6 testing

2011-09-15T15:14:00.000+05:30

http://www.kame.net/

If the turtle on this page is animated, the connection is via IPv6, otherwise the turtle is static.

Memory utilization chart using sar and perl.

2011-08-29T16:19:00.000+05:30

Author: Vikrant Telkar.

Memory utilization chart using sar -r and perl script.

Required Perl module:-

use Net::SSH::Expect;
use GD::Graph::lines;

Steps:-

1. Copy following code in cpugraph.html file and put it under apache RootDocument so that you can access it through webbrowser.

cpugraph.html:-

</head>

<p>Please enter userid on server, hostname(or IPaddress) and its password to view graphical view of memory utilization (sar -r)</p>

<form method="post" name="form" action="/cgi-bin/connect_to_server.pl"

target=_top>

<p>userid          <input

name=userid></p>

<p>hostname     <input name="hostname"></p>

<p>password     <input type="password" name="pwd"></p>

</form></p></body></html>

2. Install Net::SSH::Expect Perl moudule so that you can login to required server using usreid and password which you will mentioned in cpugraph.html page

perl -MCPAN -e shell

cpan> install Net::SSH::Expect

3. Install use GD::Graph module which will genrate png image which will give graphical presentation of "sar -r" command.

perl -MCPAN -e shell

cpan> install GD::Graph

4. Once both the perl modules are installed you are half done as these perl modules play major role. Now copy following code into connect_to_server.pl script, put it under /var/www/cgi-bin/ directory and make it excutetable.

connect_to_server.pl:-

#!/usr/bin/perl

# Author: Vikrant Telkar

# Purpose: To connect to remote server and generate report.

# This program is free software: you can redistribute it and/or

# modify it under the terms of the GNU General Public License

# as published by the Free Software Foundation, either version 3

# of the License, or (at your option) any later version.

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied

# warranty of MERCHANTABILITY or FITNESS FOR A

# PARTICULAR PURPOSE. See the GNU General Public

# License for more details.

# You should have received a copy of the GNU General Public

# License along with this program. If not, see

# <http://www.gnu.org/licenses/>.

use Net::SSH::Expect;

use CGI ':standard';

local ($buffer, @pairs, $pair, $name, $value, %FORM);

# Read in text

$ENV{'REQUEST_METHOD'} =~ tr/a-z/A-Z/;

if ($ENV{'REQUEST_METHOD'} eq "POST")

{

read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});

}else {

$buffer = $ENV{'QUERY_STRING'};

}

# Split information into name/value pairs

@pairs = split(/&/, $buffer);

foreach $pair (@pairs)

{

($name, $value) = split(/=/, $pair);

$value =~ tr/+/ /;

$value =~ s/%(..)/pack("C", hex($1))/eg;

$FORM{$name} = $value;

}

#hostname, userid and password extracted from webpage.

my $HOST = $FORM{hostname};

my $USER = $FORM{userid};

my $PWD = $FORM{pwd};

open OUT, "> /var/www/cgi-bin/out.txt" or dir $!;

my $ssh = Net::SSH::Expect->new (

host => "$HOST",

password=> "$PWD",

user => "$USER",

raw_pty => 1

);

#Connect to server

$ssh->login();

my $sar_r = $ssh->exec("sar -r ");

print "Content-type:text/html\r\n\r\n";

print "<html>";

print "<head>";

print "<title>Hello - Second CGI Program</title>";

print "</head>";

print "<body>";

#Print output of sar -r in OUT file

print OUT $sar_r;

print "<meta http-equiv='Refresh' content='5;url=http://webserver/cgi-bin/cpugraph.pl' />";

print "</body>";

print "</html>";

5. create out.txt file under /var/www/cgi-bin directory and give appropriate permission so that create in step 4 should be able to write and read the file.

6. So far we connected to server of which we want memory utilization and collected the "sar -r" data also. We in this step will draw graph using sar -r output. So copy following code in cpugraph.pl script, make it excuteable and put it under /var/www/cgi-bin/ directory.

cpugraph.pl :-

#!/usr/bin/perl -w

# Author:- Vikrant Telkar

# Purpose: To connect to remote server and generate report.

# This program is free software: you can redistribute it and/or

# modify it under the terms of the GNU General Public License

# as published by the Free Software Foundation, either version 3

# of the License, or (at your option) any later version.

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied

# warranty of MERCHANTABILITY or FITNESS FOR A

# PARTICULAR PURPOSE. See the GNU General Public

# License for more details.

# You should have received a copy of the GNU General Public

# License along with this program. If not, see

# <http://www.gnu.org/licenses/>.

# Requirement:- This script requires GD::Graph module to be install.

# How to run:- Start your apache server with cgi-bin module.

# Save this file in /var/www/cgi-bin directory with graph.pl file

# extension and access this file as http://localhost/cgi-bin/graph.pl

#use CGI ':standard';

use GD::Graph::lines;

use GD::Graph::hbars;

use strict;

my (@data,@time,@utl) ;

my $i = 0 ;

open FILE, "cat /var/www/cgi-bin/out.txt |" || die $!;

# Both the arrays should same number of entries.

while(<FILE>) {

next if($_ =~ /^\D/);

next if($_ =~ m/@/);

my @word = split(/\s+/, $_ );

next if ($word[5] =~ m/^%/);

$time[$i] = $word[0] ;

$utl[$i] = $word[4] ;

$i++ ;

}

@data = ([@time], [@utl]);

my $mygraph = GD::Graph::lines->new(1200, 600);

$mygraph->set(

y_label => 'Memory Utilization in percentage',

x_label => 'Time',

title => 'Memory Utilization chart',

# Draw datasets in 'solid', 'dashed' and 'dotted-dashed' lines

line_types => [1, 2, 4],

# Set the thickness of line

line_width => 2,

# Set colors for datasets

dclrs => ['red'],

x_label_skip => 3,

show_values => 1,

values_format => "%55555f",

) or warn $mygraph->error;

$mygraph->set('x_labels_vertical'=> 1);

$mygraph->set_legend_font(GD::gdMediumBoldFont);

$mygraph->set_legend('Sar Output');

my $myimage = $mygraph->plot(\@data) or die $mygraph->error;

print "Content-type: image/png\n\n";

print $myimage->png;

7. Now last and best step. excute the script. http://servername/cpugraph.html

Note:-

1. Make sure you have install perl_mod.so to excute perl script with apache.

2. Make sure perl scripts are excutelable and all scripts are under proper path.

3. Make sure out.txt file is read and write by apache or webserver user.

Permanently delete files in Linux

2011-08-19T03:25:00.002+05:30

how to delete a file in linux securely and permanently, so it can never be recovered?

$ yum install srm
$ srm myfile.txt

srm removes each specified file by overwriting, renaming, and truncating it before unlinking.
This prevents other people from undeleting or recovering any information about the file from the command line.

Instead of just overwriting your files with random data, it uses a special process – a combination of random data, zeros, and special values developed by cryptographer Peter Gutmann – to really, really make sure your files are irrecoverable.