Linux How - To's - Free Consultancy

POSTS MOVED

2013-04-10T12:26:00.002+05:30

ALL POSTS HAS BEEN MOVED TO:

https://kasimani.wordpress.com/

Kernel Panic - Fallback to Old Kernel

2013-03-13T17:02:00.000+05:30

Few days back, i was testing a kernel module, unfortunately it didn't work and instantly after reboot, kernel Panic occurs. Well i was clueless what goes wrong, but now I cannot select the older kernel version to boot. But, there is a way out, if we choose these first before reboot (off course after installing new kernel), and this is inbuilt in GRUB. This mechanism is called Grub Fall-back. In this case you can set your older/tested kernel as the Fallback kernel in case new kernel panics. And here is how you do it:All you have to do is to open your /boot/grub/grub.conf and add a few lines in it like this

—————————————————————————————————————
default=saved

timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
fallback 0 1

title CentOS LIDS (2.6.18-128)
root (hd0,0)
kernel /vmlinuz ro root=/dev/VolGroup00/LogVol00
initrd /initrd-lids-2.6.18-128.img
savedefault fallback

title CentOS (2.6.18-128.el5xen)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-128.el5.img
savedefault

—————————————————————————————————————

If the above grub.conf is booted, then first Grub.conf will try to boot the system with “CentOS Testing” and if and only if the kenrel panics, it will call “CentOS Working” and boot the system

Hope this quick howto helps.

Citrix XenServer – iSCSI LUN

2013-02-14T12:36:00.000+05:30

Hi all, here is my new Mini-Howto for Citrix XenServer – Storage Repository Over Software iSCSI LUN

Topic contains how to and about iSCSI, NFS, on Linux. How to mount them on Citrix Xen Server using Command line and GUI (that includes Screen Shots)

Let’s starts with iSCSI creation on linux box, I will use CentOS 6 for this.

Packages Required: scsi-target-utils

Service Name: tgtd

Configuration File: /etc/tgt/targets.conf

I will Use LVM, as i am using a large partition on this box. So proceeding with it:

# fdisk /dev/sda

Press ‘n’ for new partition

Press ‘L’ for logical partition

First Cylinder size: press enter

Type ‘w’ for save and quit

# init 6 (Restart the System)

Note: do not format partition:

LVM Setup on newly created partition:

Now create Physical Volumes (PV) on newly created partition.

# pvcreate /dev/sda5

# pvscan

# pvdisplay

Now create Volume Group (VG) : on created PV

# vgcreate sr0_vg1 /dev/sda5

# vgscan

# vgdisplay

Now we can create one or more Logical Volumes (LV’s) on VG

# lvcreate --size +30G --name sr0_vg1_lv100 sr0_vg1

# lvscan

# lvdisplay

This part is done: Now to LUN creation

For this we have to ensure we have required package:

# rpm -qa | grep -i scsi

iscsi-initiator-utils-6.2.0.872-34.el6.x86_64

scsi-target-utils-1.0.14-4.el6.x86_64 (this is main package required for iSCSI server)

If do not have, the install it either using directly from RPM if you have, or YUM

# yum install -y scsi-target-utils

After the successful installation, we have to edit the config for SAN:

# vi /etc/tgt/targets.conf

# Example: the next line would override default driver type.

# driver iser

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv100

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv101

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv102

#initiator-address 192.168.23.113 (by proving IP, we can restrict the storage to specific IPs only)

#initiator-address 192.168.23.83

</target>

Save and Quit

Now start the service of iSCSI target:

# service tgtd start

#chkconfig tgtd on

# tgt-admin -s (this will display information about iscsi target device)

I have disabled IPTABLES on target server, in my case.

This done now i will show you how to mount the available LUN on Citrix Xen Server using GUI.

Later I will add command prompt, currently I am facing some issue, suggest me if you have steps

Open XenCenter à Right Click on any XenServer a Choose New SR....

In Next window, choose Software iSCSI and click Next

Name the New SR and click Next

Provide IP of Target server (iSCSI server) and click on Discover IQNs, choose your IQN and then click on Discover LUNs and choose your LUN that you had created during LVM create.

Click on Finish and the XenCenter automatically mounts it on Server

And finally here is output/details of mounted SR.

Now LUN is ready and mounted on XenServer, start using as your need.

Please let me know for suggestions/corrections.

Kernel Panic - Fallback to Old Kernel

2013-01-03T15:41:00.000+05:30

Few days back, i was testing a kernel module, unfortunately it didn't work and instantly after reboot, kernel Panic occurs. Well i was clueless what goes wrong, but now i can not select the older kernel version to boot. But, there is a way out, if we choose these first before reboot (off course after installing new kernel), and this is inbuilt in GRUB. This mechanism is called Grub Fall-back. In this case you can set your older/tested kernel as the Fallback kernel in case new kernel panics. And here is how you do it:All you have to do is to open your /boot/grub/grub.conf and add a few lines in it like this

——————————————————————————————————————-
default=savedtimeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
fallback 0 1title CentOS LIDS (2.6.18-128)
root (hd0,0)
kernel /vmlinuz ro root=/dev/VolGroup00/LogVol00
initrd /initrd-lids-2.6.18-128.img
savedefault fallbacktitle CentOS (2.6.18-128.el5xen)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-128.el5.img
savedefault
——————————————————————————————————————-
If the above grub.conf is booted, then first Grub.conf will try to boot the system with “CentOS Testing” and if and only if the kenrel panics, it will call “CentOS Working” and boot the system
Hope this quick howto help.

Citrix XenServer – iSCSI LUN

2012-08-18T15:15:00.000+05:30

Hi all, here is my new Mini-Howto for Citrix XenServer – Storage Repository Over Software iSCSI LUN

Topic contains:
How to Create NFS and iSCSI LUN on Linux (I will CentOS)
How to mount them on Citrix Xen Server using Command line and GUI (that includes Screen Shots)

Let’s starts with iSCSI creation on linux box, I will use CentOS 6 for this.

Packages Required: scsi-target-utils

Service Name: tgtd

Configuration File: /etc/tgt/targets.conf

I will Use LVM, as i am using a large partition on this box. So proceeding with it:

# fdisk /dev/sda

Press ‘n’ for new partition

Press ‘L’ for logical partition

First Cylinder size: press enter

Type ‘w’ for save and quit

# init 6 (Restart the System)

Note: do not format partition:

LVM Setup on newly created partition:

Now create Physical Volumes (PV) on newly created partition.

# pvcreate /dev/sda5

# pvscan

# pvdisplay

Now create Volume Group (VG) : on created PV

# vgcreate sr0_vg1 /dev/sda5

# vgscan

# vgdisplay

Now we can create one or more Logical Volumes (LV’s) on VG

# lvcreate --size +30G --name sr0_vg1_lv100 sr0_vg1

# lvscan

# lvdisplay

This part is done: Now to LUN creation

For this we have to ensure we have required package:

# rpm -qa | grep -i scsi

iscsi-initiator-utils-6.2.0.872-34.el6.x86_64

scsi-target-utils-1.0.14-4.el6.x86_64 (this is main package required for iSCSI server)

If do not have, the install it either using directly from RPM if you have, or YUM

# yum install -y scsi-target-utils

After the successful installation, we have to edit the config for SAN:

# vi /etc/tgt/targets.conf

iqn.2012-08.web18:vmsan>

# Example: the next line would override default driver type.

# driver iser

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv100

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv101

backing-store /dev/ sr0_vg1/ name sr0_vg1_lv102

#initiator-address 192.168.23.113 (by proving IP, we can restrict the storage to specific IPs only)

#initiator-address 192.168.23.83

Save and Quit

Now start the service of iSCSI target:

# service tgtd start

# chkconfig tgtd on

# tgt-admin -s

(this will display information about iscsi target device)

I have disabled IPTABLES on target server, in my case.

This done now i will show you how to mount the available LUN on Citrix Xen Server using GUI.

Later I will add command prompt, currently I am facing some issue, suggest me if you have steps

Open XenCenter àRight Click on any XenServer aChoose New SR....

In Next window, choose Software iSCSI and click Next

Name the New SR and click Next

Provide IP of Target server (iSCSI server) and click on Discover IQNs, choose your IQN and then click on Discover LUNs and choose your LUN that you had created during LVM create.

Click on Finish and the XenCenter automatically mounts it on Server

And finally here is output/details of mounted SR.

Now LUN is ready and mounted on XenServer, start using as your need.

Please let me know for suggestions/corrections.

commands

2012-05-21T00:40:00.000+05:30

rkhunter --check --rwo --summary > /tmp/rkhunter_final.log
cat /tmp/archival_system/delJS_downtime.log | grep "^[0-9]" | awk '{sum=sum+$1} END{print sum}'
grep "proxy\:" /var/log/apache_front_error_log | awk '{print $NF}' |grep -v resid | sort | uniq -c | sort -n
for i in `ls -lrt | awk '{print $7"\t"$9}' | grep ^31 | awk '{print $2}'`; do cat $i | tai64nlocal | grep "m.jannath\_perthous100\@yahoo.com"; cat $i | tai64nlocal | grep "mahesh_f_b@yahoo.co.in"; cat $i | tai64nlocal | grep "annaisai1986@yahoo.co.in"; cat $i | tai64nlocal | grep "anfaz_m@hotmail.com"; cat $i | tai64nlocal | grep "sujitha2012@gmail.com"; cat $i | tai64nlocal | grep "desiginer@gmail.com"; done
for i in `ls -lrt | awk '{print $7"\t"$9}' | grep ^31 | awk '{print $2}'`; do cat $i | tai64nlocal | grep "m.jannath\_perthous100\@yahoo
cat /tmp/list | awk '{print $5}' | xargs -i egrep "adityaseth09@gmail.com|kvenk111@yahoo.co.in|knvs.prasad@gmail.com" /home/qmail-logs/2010/{}
ls -ltra | awk '{print $5," ",$9}' | awk '$1 < 3000' | awk '/gif|png|jpg/ {print $2}' | xargs -I % cp -rp % /web/R/images/
to massively clean lines containing x’s in front some css declarations in a group of files ls | awk '/.css$/' | xargs sed -i.bk -e '/^ *x.*$/d' next... find . -type f -exec grep -i '^ *x' /dev/null {} + | awk '!/svn|htdocs/' | cut -c 3- | awk '!/^#/' | awk -F ':' '{print $1}' | awk '/.css$/'| xargs sed -i.bk -e '/^ *x.*$/d\'
seq -w 1 30 | xargs -i -t zcat in_Feb2011/in_Employer-{}Feb2011.gz | egrep 'resume.html?resid=3304847|resume.html?resid=30780899|resume.html?resid=31313778|resume.html?resid=31628106'

sar - Collect, report, or save system activity information.
      -b :Report I/O and transfer rate statistics.
      -B :Report paging statistics
      -d :Report activity for each block device (kernels 2.4 and newer only)
      -q :Report queue length and load averages
      -r :Report memory and swap space utilization statistics
      -u :Report CPU utilization.

iostat - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.
       -d : device utilization report
       -c : CPU usage report
       -x : Display extended statistics

iostat

    * rrqm/s : The number of read requests merged per second that were queued to the hard disk
    * wrqm/s : The number of write requests merged per second that were queued to the hard disk
    * r/s : The number of read requests per second
    * w/s : The number of write requests per second
    * rsec/s : The number of sectors read from the hard disk per second
    * wsec/s : The number of sectors written to the hard disk per second
    * avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
    * avgqu-sz : The average queue length of the requests that were issued to the device
    * await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
    * svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
    * %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.


Device:     rrqm/s     wrqm/s     r/s     w/s     rsec/s       wsec/s     rkB/s         wkB/s         avgrq-sz     avgqu-sz     await     svctm     %util
cciss/c0d0     2269.60 132.60     296.20     77.60     20521.60 1681.60     10260.80     840.80         59.40         4.15         11.18     2.65     99.20

Citrix Xen VM:
Open properties of VM,, select "Startup Options", change "OS Boot Parameters" to "rw init=/bin/bash", graphical utf8reboot, you get # prompt... if want to edit some files...graphical utf8

graphical utf8

run this command:
# mount -w -o remount /
# now edit files you want... and reboot...

#### mysql query to backup table contents into a file.
mysql#> select id,user,subject,content,status,sentdate,last_update from tickets where type='PRODUCT' into outfile '/tmp/product_ticket' fields terminated by '~' lines terminated by '|~|';

Mysql check
mysqlcheck -u root -p --auto-repair -c -o -A

Repair Mysql MYISAM files..
/myisamchk --force --sort_buffer_size=64M --key_buffer_size=16M --read_buffer_size=8M --write_buffer_size=8M ../data/phplists/phplist_linktrack.MYI

mysql INNODB related
mysql> #SELECT TABLE_SCHEMA, TABLE_NAME, CONCAT(ROUND(data_length / ( 1024 * 1024 ), 2), 'MB') DATA, CONCAT(ROUND(data_free / ( 1024 * 1024 ), 2), 'MB')FREE from information_schema.TABLES where TABLE_SCHEMA='man' and table_name='result10' and Data_free > 0;
mysql> select TABLE_SCHEMA, TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='man';

ALTER TABLE `pass11` ENGINE = InnoDB   (this will also refragment innodb table)

ext4 disable journal

At one high loaded web project I needed a very fast file system. I decided to use Ext4 with disabled journal (As a google:))).

# Create ext4 fs on /dev/sda10 disk
mkfs.ext4 /dev/sda10

# Enable writeback mode. This mode will typically provide the best ext4 performance.
tune2fs -o journal_data_writeback /dev/sda10

# Delete has_journal option
tune2fs -O ^has_journal /dev/sda10

# Required fsck
e2fsck -f /dev/sda10

# Check fs options
dumpe2fs /dev/sda10 |more

For more performance add fstab opions: data=writeback,noatime,nodiratime
i.e:
/dev/sda10 /opt ext4 defaults,data=writeback,noatime,nodiratime 0 0

Tested at non-boot partition ;)

Redirect port request internally using IPTABLES

# iptables -I PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

# rpm -q --provides openssl

To check which users are connected to ftp servers, make the below entrt in vsftpd.conf
setproctitle_enable=YES

restart vsftpd  and run command 
# ps -fe | grep ftp


to restrict users to time based login in system and ftp server

edit /etc/pam.d/vsftpd and add below line just after ending auth section

session    optional     pam_keyinit.so    force revoke
auth       required    pam_listfile.so item=user sense=deny file=/etc/vsftpd/ftpusers onerr=succeed
auth       required    pam_shells.so
auth       include    system-auth
account    required     pam_time.so
account    include    system-auth
session    include    system-auth
session    required     pam_loginuid.so


now edit /etc/security/time.conf and add below line at end.

vsftpd;*;mayankg;!Al1000-1900


Done

STORE SSH KEYS IN LDAP

2012-03-13T16:21:00.000+05:30

We need password less ssh login, so here i have tried to make the keys centralized by storing them in LDAP

LDAP Server Setup

[root@ldap]# yum install openldap{,-clients,-devel,} nss_ldap

make entry in /etc/hosts

192.168.1.3 directory.domain.com

Run this Command and copy output

[root@ldap]# slappasswd

Note: Enter desired password here

pass:

confirm:

Output will like this, copy it

{SSHA}AuOU1S01Nj+gQ9FIHf8gCompENETugiT

Now edit slapd.conf

[root@ldap]# vi /etc/openldap/slapd.conf

check for this similar entry and edit as follows

database bdb

suffix "dc=directory,dc=domain,dc=com"

rootdn "cn=Manager,dc=directory,dc=domain,dc=com"

rootpw {SSHA}AuOU1S01Nj+gQ9FIHf8gCompENETugiT #(paste the encrypted password here)

directory /var/lib/ldap

[root@ldap]# service ldap start

Create a file named /etc/openldap/init.ldif and below line

dn: cn=Manager,dc=directory,dc=domain,dc=com

ou: People

description: All People in Organisation

objectClass: organizationalUnit

dn: ou=People,dc=directory,dc=domain,dc=com

ou: People

objectclass: organizationalUnit

dn: ou=Groups,dc=directory,dc=domain,dc=com

ou: Groups

description: All People in Organisation

objectClass: organizationalUnit

now you have to add that file in LDAP server

[root@ldap]# ldapadd -x -D "cn=Manager,dc=directory,dc=domain,dc=com" -W -f /etc/openldap/init.ldif

It will now ask for password, paste or type the password that you typed for "slappasswd"

Now import any user from linux passwd file

[root@ldap]# useradd manish

[root@ldap]# passwd manish

[root@ldap]# grep manish /etc/passwd > passwd.manish

[root@ldap]# /usr/share/openldap/migration/migrate_passwd.pl passwd.manish manish.ldif

[root@ldap]# ldapadd -x -D "cn=Manager,dc=directory,dc=domain,dc=com" -W -f manish.ldif

It will ask for the rootdn passwd, provide that.

TEST LDAP SERVER: On LDAP Server

[root@ldap]# ldapsearch -x -b "dc=directory,dc=domain,dc=com"

------------------------------------------------------------------------------------------

LDAP Client Setup

[root@ldap-client]# yum install nss_ldap openldap-clients

make entry in /etc/hosts

192.168.1.3 directory.domain.com

now run this command: (This is to enable client authentication on SSH logins using LDAP server)

[root@ldap-client]# authconfig --useshadow --usemd5 --enableldap --enableldapauth --ldapserver=directory.domain.com --ldapbasedn="dc=directory,dc=domain,dc=com" --enablemkhomedir --updateall

Edit /etc/ldap.conf and add below lines

ssl no

tls_cacertdir /etc/openldap/cacerts

pam_password md5

uri ldap://directory.ibnlive.com/

base dc=directory,dc=ibnlive,dc=com

edit /etc/openldap/ldap.conf and add below lines

ssl no

tls_cacertdir /etc/openldap/cacerts

pam_password md5

uri ldap://directory.ibnlive.com/

base dc=directory,dc=ibnlive,dc=com

Now Test LDAP Client for auth:

1: ldapsearch -x (if successful the also run 2 one)

2: getent passwd | grep manish (though the user manish do not exists in client system, you should get answer for this)

3: ssh -l manish 192.168.1.3

4: su - manish (though the user manish do not exists in client system, you should get login for this)

TLS (LDAP Server) – Secure LDAP Server

[root@ldap]# mkdir /etc/ssl/openldap/

[root@ldap]# cd /etc/ssl/openldap/

[root@ldap]# openssl genrsa -out ldap.key 2048

[root@ldap]# openssl req -new -key ldap.key -out ldap.csr

[root@ldap]# openssl genrsa -out ca.key 2048

[root@ldap]# openssl req -new -x509 -days 3650 -key ca.key -out ca.cert

[root@ldap]# openssl x509 -req -in ldap.csr -out ldap.cert -CA ca.cert -CAkey ca.key -CAcreateserial -days 3650

[root@ldap]# openssl x509 -in ldap.cert -text -noout

Edit /etc/openldap/slapd.conf on LDAP Server, append these lines: (If there is already below entries, comment that)

TLSCertificateFile /etc/ssl/openldap/ldap.cert

TLSCertificateKeyFile /etc/ssl/openldap/ldap.key

TLSCACertificateFile /etc/ssl/openldap/ca.cert

Now edit /etc/openldap/ldap.conf on LDAP CLIENT: remove all lines... and enter these

HOST directory.domain.com

PORT 636

tls_cacertdir /etc/openldap/cacerts

TLS_REQCERT demand

pam_password md5

uri ldap://directory.domain.com/ ldaps://directory.domain.com:636

base dc=directory,dc=domain,dc=com

[root@ldap]# rm -rf /etc/ldap.conf

[root@ldap]# ln -s /etc/openldap/ldap.conf /etc/ldap.conf

------------------------------------------------------------------------------------------

Replication: LDAP Server to LDAP Standby Client Server

On Second Server: (Standby)

1. Follow the same above procedure for LDAP.

2. Create /etc/rsyncd.conf and enter below lines

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

lock file = /var/run/rsync.lock

[ldap_standby]

path=/var/lib/ldap/

comment=ldap sync

uid = root

gid = root

read only = no

list = no

auth users = root

secrets file = /etc/ldap-admin

strict modes = false

Save and exit

Now create a file /etc/ldap-admin

[root@ldap-standby]# echo "root:syncldapadmin" > /etc/ldap-admin

[root@ldap-standby]# rsync --daemon

Now add this in /etc/rc.local at end

[root@ldap-standby]# vi /etc/rc.local

rsync --daemon

Now edit /etc/sysconfig/iptables and add below line before "COMMIT"

[root@ldap-standby]# vi /etc/sysconfig/iptables

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j ACCEPT

save and exit

[root@ldap-standby]# /etc/init.d/iptables restart

This is done

Now on LDAP Server.

create a file under /etc/ by name pass-ldap

[root@ldap]# echo "syncldapadmin" > /etc/pass-ldap

[root@ldap]# chmod 700 /etc/pass-ldap

now add a line in crontab

[root@ldap]# crontab -e

* * * * * /usr/bin/rsync -az /var/lib/ldap/ root@::ldap_standby/

save and exit

------------------------------------------------------------------------------------------

SSH Keys – LDAP Store

[root@ldap]#yum -y install gcc openldap-devel pam-devel zlib-devel openssl-devel

[root@ldap]#wget -qO - http://ftp.jaist.ac.jp/pub/OpenBSD/OpenSSH/portable/openssh-5.4p1.tar.gz | tar zxf -

[root@ldap]#cd openssh-5.4p1

[root@ldap]#wget -q http://openssh-lpk.googlecode.com/svn/trunk/patch/contrib/contrib-openssh-lpk-5.4p1-0.3.13.patch

[root@ldap]#patch < contrib-openssh-lpk-5.4p1-0.3.13.patch

[root@ldap]#./configure --with-ldap --sysconfdir=/etc/ssh --prefix=/usr --with-pam

If error comes for pam, the issue this command

[root@ldap]#yum install pam-devel

[root@ldap]#make

Now take backup of /etc/init.d/sshd and /etc/ssh/

[root@ldap]#mkdir –p /opt/sshback

[root@ldap]#mv /etc/init.d/sshd /opt/sshnack/

[root@ldap]#mv /etc/ssh /opt/sshback/

[root@ldap]#yum -y erase openssh-server

[root@ldap]#make install

[root@ldap]#cp /opt/sshback/sshd /etc/init.d/

Now edit the “sshd_config”

[root@ldap]#vi /etc/ssh/sshd_config

UseLPK yes

LpkLdapConf /etc/openldap/ldap.conf

LpkServers ldap://localhost/

LpkUserDN ou=People,dc=domain,dc=com

LpkGroupDN ou=Groups,dc=domain,dc=com

LpkBindDN cn=Manager,dc=domain,dc=com

LpkBindPw controlman

LpkServerGroup tech

LpkForceTLS no

LpkSearchTimelimit 3

LpkBindTimelimit 3

LpkPubKeyAttr sshPublicKey

Now add a group in ldap

Create a file maingroup.ldif and add below lines, and add all users in that file as below

dn: cn=tech,ou=Groups dc=domain,dc=com

cn: tech

gidnumber: 504

memberuid: user1

memberuid: user2

memberuid: user3

memberuid: user4

objectclass: posixGroup

[root@ldap]#ldapadd -x -D "cn=Manager,dc=domain,dc=com " -w -f maingroup.ldif

It should display like this:

adding new entry "cn=web18,ou=Groups,dc=domain,dc=com"

[root@ldap]#mkdir -p /var/empty/sshd/etc

[root@ldap]#wget -q http://openssh-lpk.googlecode.com/files/openssh-lpk_openldap.schema -O /etc/openldap/schema/openssh-lpk_openldap.schema

Now edit the slapd.conf

[root@ldap]#vi /etc/openldap/slapd.conf

Include /etc/openldap/schema/openssh-lpk_openldap.schema

Save and exit

------------------------------------------------------------------------------------------

PHPLDAPADMIN (frontend for LDAP Management)

Download phpldapadmin from
[root@ldap]#wget http://downloads.sourceforge.net/project/phpldapadmin/phpldapadmin-php5/1.2.2/phpldapadmin-1.2.2.tgz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fphpldapadmin%2Ffiles%2Fphpldapadmin-php5%2F1.2.2%2F&ts=1331543024&use_mirror=nchc
[root@ldap]#unzip phpldapadmin-1.2.2.zip
[root@ldap]#mv phpldapadmin /var/www/html/
[root@ldap]#chown wwwrun:www /var/www/html/phpldapadmin
Now go to phpldapadmin/config/
[root@ldap]#mv config.php.example config.php
now access that from browser:
http:///phpldapadmin/

user: cn=Manager,dc=domain,dc=com
pass:

------------------------------------------------------------------------------------------

Now log into phpLDAPadmin and browse to the user you want to attach an SSH key to:

Now, under the user's "objectClass" entry, click "Add Value":

You should now see the following screen:

Select "ldapPublicKey" from the list and then hit "Add new ObjectClass".

Now that the ldapPublicKey ObjectClass is available to this user, you can add the sshPublicKey attribute to the user and paste their key in. Just hit "Add new attribute" as shown below, and you will get a dropdown menu.

In the dropdown menu, select "sshPublicKey" and you will see the dropdown change into the following:

Paste the user's key in here, then hit "Update Object", and that's it! You're done...the user now has their Public SSH key stored in your LDAP directory, and as long as their companion private key is available on the machine they're connecting from, they will never need to use a password to log into any machine that's pointed at your LDAP server.

Now restart SSHD and ldap app.

# /etc/init.d/sshd restart

#/etc/init.d/ldap restart

Generate public keys: (do not provide any password)

# ssh-keygen -q -f ~/.ssh/id_rsa -t rsa

Enter passphrase (empty for no passphrase): …

Enter same passphrase again: …

# cat ~/.ssh/id_rsa.pub

Expecting: like below

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAoLpYk/1bOPiQC8tP0aB7g5YfcmniEAdEEitGwRrjc3XZptLQm3syl11bXdGbBCXb8JT1r3R8OVsnN3Ml/zTy3f7GBIWDH47ytyBxhQcMaCYk9Kk6pG6jbJCoikkxYggD0BexeVCK8qNyi9uohLh6PWCWRX29wF2zqiRjWyy5t8WK8oBgahdx18nqRoQRxyLsk0YIiff8n/WFQ3lWW2H2AlHWRus5cLVZ6J1safgLChOX5BIGt4A/UHsKdXHJd/zcSt+xkEXO7WqzIlMF/hh62NdfV2oR2AQuhAvGcYeDJhMLqbf3GqBdhL/zBF+pxhu8IU/7wPBjpJV865XRI0/JTQ== root@vm-machine

Above line has to copy in sshpublic keys.

DNS QUERY TIMEOUT ON LINUX CLIENT

2012-03-01T12:31:00.000+05:30

Hi all, here i have come with a situation, when we have multiple DNS, basically Nameserver entry in resolve.conf at client, and sometimes we also face dam slow resolves in queries, this is may also due to DNS server issue, may be DNS has gone away or some other issues that is making it slow to resolve queries.By default, linux client attempts 3 times to Nameserver for queries. Here it make the process slow.
So what we can do, is to set timeout in resolve.conf, this will help client to choose another Nameserver for AnswersHere we go:

Edit the file resolve.conf:

# vi /etc/resolve.conf

nameserver 192.168.1.71
nameserver 192.168.1.72
nameserver 192.168.1.73
nameserver 192.168.1.74
options timeout:1
options attempts:1

Here you can see, i have set options value in file.

options timeout:1 --> will tell client to skip this server, if there is delay in Answer
options attempts:1 --> will tell client to do just single attempt per server

Now you can check the difference.

Multiple Purpose Expect Script

2011-12-23T21:25:00.000+05:30

DISK ALERT - Expect ScriptHi All, here's I have bring a expect script, that this we can Run command on Remote Linux Servers, copy file using SSHSo here it is:
[root@virtual-vm ~]# yum install expect (if expect package is not installed)
[root@virtual-vm ~]# vi /opt/scripts/diskspace-alerts.expDISK ALERT - Expect Script

#!/usr/bin/expect

set logfile [open /opt/scripts/iplist]

set ipaddrs [read $logfile]

foreach line [split $ipaddrs \n] {

set timeout -1

spawn ssh cube@$line /sbin/ifconfig eth0 | /bin/grep 'inet addr' | /bin/awk '{print \$2}' | /bin/cut -d: -f2

set pass "ssh-password"

set ya "yes"

expect {

password: {send "$pass\r"; exp_continue}

connecting {send "$ya\r"; exp_continue}

}

spawn ssh cube@$line /bin/df -hlP | sed G

set pass "ssh-passowrd"

expect {

password: {send "$pass\r"; exp_continue}

}

/opt/scripts/iplist: contains IP address of servers, on which you may want to run the commands

Set this in Cron:
expect /opt/scripts/diskspace-alerts.exp | egrep -v 'spwan|cube|ssh' > /tmp/diskspace-alert.txt ; /usr/sbin/postsuper -d ALL; /etc/init.d/postfix start; mail -s "Disk space alert - IBNLive.com" delhi.manish@gmail.com < /tmp/diskspace-alert.txt ; sleep 10; /etc/init.d/postfix stop

Use your own choice of Mail Server

MPM-Apache-settings

2011-11-14T13:43:00.000+05:30

Worker based

StartServers             5
ServerLimit             16
ThreadLimit             35
MaxClients             560
MinSpareThreads         25
MaxSpareThreads         75
ThreadsPerChild         35
MaxRequestsPerChild 25000

Prefork Based:

StartServers          15
MinSpareServers       15
MaxSpareServers       35
MaxClients           100
MaxRequestsPerChild 2500

StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild 0

# perchild MPM
# NumServers: constant number of server processes
# StartThreads: initial number of worker threads in each server process
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# MaxThreadsPerChild: maximum number of worker threads in each server process
# MaxRequestsPerChild: maximum number of connections per server process

NumServers           5
StartThreads         5
MinSpareThreads      5
MaxSpareThreads     10
MaxThreadsPerChild 20
MaxRequestsPerChild 0

# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum number of requests a server process serves

ThreadsPerChild 250
MaxRequestsPerChild 0

# BeOS MPM
# StartThreads: how many threads do we initially spawn?
# MaxClients:   max number of threads we can have (1 thread == 1 client)
# MaxRequestsPerThread: maximum number of requests each thread will process

StartThreads               10
MaxClients                 50
MaxRequestsPerThread       10000

# NetWare MPM
# ThreadStackSize: Stack size allocated for each worker thread
# StartThreads: Number of worker threads launched at server startup
# MinSpareThreads: Minimum number of idle threads, to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# MaxThreads: Maximum number of worker threads alive at the same time
# MaxRequestsPerChild: Maximum number of requests a thread serves. It is
#                      recommended that the default value of 0 be set for this
#                      directive on NetWare. This will allow the thread to
#                      continue to service requests indefinitely.

ThreadStackSize      65536
StartThreads           250
MinSpareThreads         25
MaxSpareThreads        250
MaxThreads            1000
MaxRequestsPerChild      0
MaxMemFree             100

# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process,
#                  to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process
# MaxRequestsPerChild: Maximum number of connections per server process

StartServers           2
MinSpareThreads        5
MaxSpareThreads       10
MaxRequestsPerChild    0

URL Rewriting - Examples

2011-11-10T13:33:00.000+05:30

Apache Rewrite Rules can be used to transform a pretty ugly looking URL into a beautiful, SEO-friendly and meaningful URL. I would like to list a couple of real-world examples below to explain how Rewrite Rules can be set up and how they work. Rewrite Rules use Regular Expressions, so this article is closely related to my Regular Expressions article.

Time-Dependent Rewriting

RewriteCond %{TIME_HOUR}%{TIME_MIN} >0700
RewriteCond %{TIME_HOUR}%{TIME_MIN} <1900
RewriteRule ^test\.html$ test.day.html [L]
RewriteRule ^test.html$ test.night.html

Redirect Failing URLs to Another Web Server

RewriteCond /your/docroot/%{REQUEST_FILENAME} !-f
RewriteRule ^(.+) http://webserverB.com/$1 [R]

Browser Dependent Content

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/3.*
RewriteRule ^foo\.html$ foo.NS.html [L]
RewriteCond %{HTTP_USER_AGENT} ^Lynx/.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[12].*
RewriteRule ^foo\.html$ foo.20.html [L]
RewriteRule ^foo\.html$ foo.32.html [L]

Blocking of BAD Proxied IP Addresses

RewriteMap hosts-deny txt:/etc/apache2/conf/host.deny
RewriteCond ${hosts-deny:%{HTTP:True-Client-IP}|NOT-FOUND} !=NOT-FOUND
RewriteRule ^/.* - [R=401,L]

Redirecting a URI to a New Format

RewriteCond %{QUERY_STRING} !marker
RewriteCond %{QUERY_STRING} id=([-a-zA-Z0-9_+]+)
RewriteRule ^/?human\.php$ /%1? [R,L]
RewriteRule ^/?([-a-zA-Z0-9_+]+)$ human.php?marker&id=$1 [L]

Force some pages to use Secure Server

RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^/?(page1|page2|page3|page4|page5)$ https://www.secure-server.com/%1 [R,L]

Redirect UserAgent for IPAD, IPHONE, And Anroid OS

RewriteCond %{HTTP_USER_AGENT} !^.*iPhone.*$ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.*iPad.*$ [NC]
RewriteCond %{HTTP_USER_AGENT} !^.*Android.*$ [NC]
RewriteCond %{REQUEST_URI} ^/tab

RewriteRule .* /index.html [R=301,L]

Block Bad Referrer

RewriteCond %{HTTP_REFERER} (tvchannelsfree) [NC,OR]
RewriteCond %{HTTP_REFERER} (onionz) [NC]
RewriteRule ^/.* - [R=401,L]

Make Fancy URLS

Ex. What to What:

http://domain.com/index.php?id1=3456&text=humanbeing_nonsense&sub_id=14 TO

http://domain.com/senses/humanbeing_nonsense/3456/14.html

RewriteRule ^senses/([a-z_]+)/([0-9]+)/([0-9]+)\.html$ /index.php?id1=$2&text=$1&sub_id=$3

Redirect Page Request to Other Server

RewriteCond %{REQUEST_URI} /username/register.php
RewriteRule .* http://domain2.com/username/register_new.php [R=301,NC]
RewriteCond %{REQUEST_URI} /username/entry.html
RewriteRule .* http://domain2.com/username/entry.html [R=301,NC]

Load Balance Mysql Slaves Using MySQLProxy

2011-11-01T18:31:00.000+05:30

I am sure you all have read about MySQL replication, like Master-Master, Single Master - Multiple Slaves and their topologies.
Here I am going to describe how you can load balance several MySQL Slaves and query logging.

Required packages:

· MysqlProxy

· Lua

· Yum (readline readline-devel libtermcap-devel ncurses-devel libevent libevent-devel mysql-devel openssl krb5-libs glib2-devel)

Our Servers:

· 192.172.1.1 (Mysql Proxy Server’s IP)

· 192.172.1.2 (Mysql Slave)

· 192.172.1.3 (Mysql Slave)

· 192.172.1.4 (Mysql Slave)

· 192.172.1.5 (Mysql Slave)

Install Mysql Proxy:

I am using MysqlProxy Ver. 0.6.1, download this version from your own way or use the below lines.

# cd /mnt/download

# wget http://dev.mysql.com/get/Downloads/MySQL-Proxy/mysql-proxy-0.6.1.tar.gz/from/http://www.mirrorservice.org/sites/ftp.mysql.com/

# tar zxvf mysql-proxy-0.6.1.tar.gz

Download Lua, if you want logging:

# wget http://www.lua.org/ftp/lua-5.1.3.tar.gz

# tar xzf lua-5.1.3.tar.gz

# cd lua-5.1.3

# yum install readline readline-devel libtermcap-devel ncurses-devel libevent libevent-devel

# make linux

# make install

Lua Done

Now go to mysql-proxy under download directory

# cd /mnt/download/ mysql-proxy-0.6.1

# yum install mysql-devel openssl krb5-libs glib2-devel

# ./configure --prefix=/opt/mysql-proxy --with-lua LDFLAGS="-lm -ldl" LUA_CFLAGS="-I/usr/lib64" LUA_LIBS=/usr/lib64/liblua5.1.a

# make

# make install

MysqlProxy Installation Done:

Now we have to create a Bash File for automating task:

# vi /scripts/runproxy.sh (I put all my script under /scripts/, so change accordingly)

#!/bin/bash

BASE_DIR=/usr/local/

BIN_DIR=${BASE_DIR}/sbin

pkill -f mysql-proxy

sleep 1

${BIN_DIR}/mysql-proxy \

--proxy-backend-addresses=192.172.1.2:3306 \

--proxy-backend-addresses=192.172.1.3:3306 \

--proxy-backend-addresses=192.172.1.4:3306 \

--proxy-backend-addresses=192.172.1.5:3306 &

--proxy-lua-script=/scripts/simple-log.lua &

sleep 1

echo "Testing Query: select * FROM speed limit 1;"

echo "select * FROM speed limit 1;" | /usr/local/mysql/bin/mysql --host=192.172.1.1 --port=4040 --user=root --password=mysql-password test || { echo "${sqlStatement}: failed (is that server up?)"; exit 1; }

# vi /scripts/simple-log.lua

local log_file = '/var/log/mysql-proxy/mysql.log'

local fh = io.open(log_file, "a+")

function read_query( packet )

if string.byte(packet) == proxy.COM_QUERY then

local query = string.sub(packet, 2)

fh:write( string.format("%s %6d -- %s :IP %s :USER: %s\n",

os.date('%Y-%m-%d %H:%M:%S'),

proxy.connection.server.thread_id,

query,

proxy.connection.client.address,

proxy.connection.client.username,

error_status))

fh:flush()

end

Now time to execute the script, but still some work before running the script.

Have to edit IPTABLES :

# vi /etc/sysconfig/iptables (On MysqlProxy Server)

-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4040 -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4041 -j ACCEPT

# vi /etc/sysconfig/iptables (On Mysql Slave Servers)

-A RH-Firewall-1-INPUT -s 192.172.1.2 -d 192.172.1.1 -p tcp -m state --state NEW,ESTABLISHED,RELATED -m tcp --dport 4040 -j ACCEPT

Apply IPTABLES Rules, restart IPtables

Now run the script:

# chmod 755 /scripts/runproxy.sh

# /bin/sh /scripts/runproxy.sh

Or Just add this file to : /etc/rc.local

# vi /etc/rc.local

#!/bin/sh

# This script will be executed *after* all the other init scripts.

# You can put your own initialization stuff in here if you don't

# want to do the full Sys V style init stuff.

/var/lock/subsys/local

/bin/sh /scripts/runproxy.sh

Check MysqlProxy is running:

# netstat -tnlp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:4040 0.0.0.0:* LISTEN 3778/mysql-proxy

tcp 0 0 0.0.0.0:4041 0.0.0.0:* LISTEN 3778/mysql-proxy

Report Developer's About Error/Warning in Codes from Apache Logs

2011-08-12T21:38:00.000+05:30

This is just a scenario i got it in mind, Reporting Warning/Error to Developer(s), using a PERL script, this you can also be done using SHELL Script.

#!/usr/bin/perl

use Net::SMTP;
my $date=`date +%a" "%b" "%d`;
my $warn="Warning:";
my $line1=`cat /var/log/apachelogs/error_log | grep -i 'php warning' | grep '$date' | awk -F'$warn' '{print \$2}' | sort | uniq -c | sed G > /tmp/get-PHP-errors.txt`;

my $line=`cat /tmp/get-PHP-errors.txt`;

my $ServerName = "mail2.mailer.com";

my $RealNameFrom = "PHP Warnings";

my $MailFrom = "noreply\@tech-suppport.com";

my $RealNameTo = "Web Coders";

my $MailTo = "delhi.manish\@gmail.com";

my $MailSubject = "Apache Error Log On Stagging Server";

$smtp = Net::SMTP->new($ServerName);

die "Couldn't connect to server" unless $smtp;

$smtp->mail( $MailFrom );
$smtp->to( $MailTo );

$smtp->data();

$smtp->datasend("To: $RealNameTo <$MailTo>\n");
$smtp->datasend("From: $RealNameFrom <$MailFrom>\n");
$smtp->datasend("Subject: $MailSubject\n\n");

$smtp->datasend("$line");
$smtp->dataend();

$smtp->quit();

Monitor PostgreSQL with Queries

2011-03-15T13:11:00.000+05:30

(Not All written by me.... )Some time just monitoring a DB servers from monitoring tools is just not enough, in that case we need some tool that let us know, what is DB is holding, processing, parameters etc..

So here are some PSQL Queries that will let us know some expected results:

Create Aggregate function

# psql

CREATE AGGREGATE array_accum (anyelement)
(
    sfunc = array_append,
    stype = anyarray,
    initcond = '{}'
);

--------------------------------------------------------------------------------------------------------
The function pg_stat_get_backend_idset provides a convenient way to generate one row for each active server process. For example, to show the PIDs and current queries of all server processes:

# psql -c "SELECT pg_stat_get_backend_pid(s.backendid) AS procpid, pg_stat_get_backend_activity(s.backendid) AS current_query FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s;"
--------------------------------------------------------------------------------------------------------

Sample query to pull the number of rows, indexes, and some info about those indexes for each table. (Only works on 8.3; ditch the pg_size_pretty if you’re on an earlier version)

postgres=# SELECT
pg_class.relname,
pg_size_pretty(pg_class.reltuples::bigint) AS rows_in_bytes,
pg_class.reltuples AS num_rows,
count(indexname) AS number_of_indexes,
CASE WHEN x.is_unique = 1 THEN 'Y'
ELSE 'N'
END AS UNIQUE,
SUM(case WHEN number_of_columns = 1 THEN 1
ELSE 0
END) AS single_column,
SUM(case WHEN number_of_columns IS NULL THEN 0
WHEN number_of_columns = 1 THEN 0
ELSE 1
END) AS multi_column
FROM pg_namespace
LEFT OUTER JOIN pg_class ON pg_namespace.oid = pg_class.relnamespace
LEFT OUTER JOIN
(SELECT indrelid,
max(CAST(indisunique AS integer)) AS is_unique
FROM pg_index
GROUP BY indrelid) x
ON pg_class.oid = x.indrelid
LEFT OUTER JOIN
( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns FROM pg_index x
JOIN pg_class c ON c.oid = x.indrelid
JOIN pg_class ipg ON ipg.oid = x.indexrelid )
AS foo
ON pg_class.relname = foo.ctablename
WHERE
pg_namespace.nspname='public'
AND pg_class.relkind = 'r'
GROUP BY pg_class.relname, pg_class.reltuples, x.is_unique
ORDER BY 2;

------------------------------------------------------------------------------------------------------

Query to check Database Size, Cache, Hit Success

# psql -c "select a.datname, pg_size_pretty(pg_database_size(a.datid)) as size, cast(blks_hit/(blks_read+blks_hit+0.000001)*100.0 as numeric(5,2)) as cache, cast(xact_commit/(xact_rollback+xact_commit+0.000001)*100.0 as numeric(5,2)) as success from pg_stat_database a order by a.datname"--

--------------------------------------------------------------------------------------------------------
Table & index sizes along which indexes are being scanned and how many tuples are fetched.

postgres=# SELECT
t.tablename,
indexname,
c.reltuples AS num_rows,
pg_size_pretty(pg_relation_size(t.tablename)) AS table_size,
pg_size_pretty(pg_relation_size(indexrelname)) AS index_size,
CASE WHEN x.is_unique = 1 THEN 'Y'
ELSE 'N'
END AS UNIQUE,
idx_scan AS number_of_scans,
idx_tup_read AS tuples_read,
idx_tup_fetch AS tuples_fetched
FROM pg_tables t
LEFT OUTER JOIN pg_class c ON t.tablename=c.relname
LEFT OUTER JOIN
(SELECT indrelid,
max(CAST(indisunique AS integer)) AS is_unique
FROM pg_index
GROUP BY indrelid) x
ON c.oid = x.indrelid
LEFT OUTER JOIN
( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns, idx_scan, idx_tup_read, idx_tup_fetch,indexrelname FROM pg_index x
JOIN pg_class c ON c.oid = x.indrelid
JOIN pg_class ipg ON ipg.oid = x.indexrelid
JOIN pg_stat_all_indexes psai ON x.indexrelid = psai.indexrelid )
AS foo
ON t.tablename = foo.ctablename
WHERE t.schemaname='public'
ORDER BY 1,2;
----------------------------------------------------------------------------------------------------------
Shared Locks

# psql -c "select pg_stat_activity.datname,pg_class.relname,pg_locks.transactionid, pg_locks.mode, pg_locks.granted, pg_stat_activity.usename,substr(pg_stat_activity.current_query,1,30), pg_stat_activity.query_start, age(now(),pg_stat_activity.query_start) as "age", pg_stat_activity.procpid from pg_stat_activity,pg_locks left outer join pg_class on (pg_locks.relation = pg_class.oid) where pg_locks.pid=pg_stat_activity.procpid order by query_start;"

----------------------------------------------------------------------------------------------------------

Finding unused indexes

#psql -c "SELECT *, pg_size_pretty(pg_relation_size(indexrelname)) FROM pg_stat_all_indexes WHERE schemaname = 'public' ORDER BY pg_relation_size(indexrelname) DESC, idx_scan ASC;
select indexrelid::regclass as index, relid::regclass as table from pg_stat_user_indexes JOIN pg_index USING (indexrelid) where idx_scan = 0 and indisunique is false;"

Finding duplicate indexes
# psql -c "select indrelid::regclass, array_accum(indexrelid::regclass) from pg_index group by indrelid, indkey having count(*) > 1;"

Finding useless indexes
# psql -c "select starelid::regclass, indexrelid::regclass, array_accum(staattnum), relpages, reltuples, array_accum(stadistinct) from pg_index join pg_statistic on (starelid=indrelid and staattnum = ANY(indkey)) join pg_class on (indexrelid=oid) where case when stadistinct < 0 then stadistinct > -.8 else reltuples/stadistinct > .2 end and not (indisunique or indisprimary) and (relpages > 100 or reltuples > 1000) group by starelid, indexrelid, relpages, reltuples order by starelid ;"

-----------------------------------------------------------------------------------------------------------

Count number of current connections:
SELECT COUNT(*) FROM pg_stat_activity;

Number of commits and rollbacks for each database:
SELECT datname, xact_commit, xact_rollback FROM pg_stat_database;

Summaries of number of inserts/updates/deletes:
SELECT SUM(n_tup_ins), SUM(n_tup_upd), SUM(n_tup_del) FROM pg_stat_all_tables;

Summaries of query plans:
SELECT SUM(seq_scan), SUM(seq_tup_read), SUM(idx_scan), SUM(idx_tup_fetch) FROM pg_stat_all_tables;

Summaries of locks held:
SELECT mode, COUNT(mode) FROM pg_locks GROUP BY mode ORDER BY mode;

#############Summary of disk I/O in terms of blocks read:
SELECT SUM(heap_blks_read) FROM pg_statio_user_tables;
SELECT SUM(idx_blks_read) FROM pg_statio_user_tables;
SELECT SUM(toast_blks_read) FROM pg_statio_user_tables;
SELECT SUM(tidx_blks_read) FROM pg_statio_user_tables;
Swap `read' for `hit' to get the number of blocks read from memory.
Caution!
This does not take the operating system's disk cache into account!
pg_database_size(name)
pg_tablespace_size(name)

-----------------------------------------------------------------------------------------------
Check Master-Slave Sync Status in SR Mode

On Master
postgres=# select pg_current_xlog_location();
pg_current_xlog_location
--------------------------
7B/3CB4CBD0
(1 row)

On Slave
postgres=# select pg_last_xlog_replay_location();
pg_last_xlog_replay_location
------------------------------
7B/3CB4CBD0
(1 row)

Some Useful Quotes

2011-02-22T12:34:00.000+05:30

Block Size:
The block size specifies size, that the file system, will use to Read and Write data. Larger block sizes will help improve disk I/O performance when using large files, such as Databases.
Mostly of size 1024, need to be 4096 for bigger file support like in database server case.

Block Size: Different Terms
1: Hardware Block Size = Sector Size
2: File System Block Size = Block Size
3: Kernel Buffer Block Size = Block Size
4: Partition Table Block Size = Cylinder Size
----------------------------------------------------------------------------------------------

MyISAM tables support concurrent inserts. If a MyISAM table has no holes in the middle resulting from deleted or updated records, inserts always take place at the end of the table and can be performed while other clients are reading the table. Concurrent inserts can take place even for a table that has been read-locked explicitly if the locking client acquired a READ LOCAL lock rather than a regular READ lock. If a table does have holes, concurrent inserts cannot be performed. However, you can remove the holes by using OPTIMIZE TABLE to defragment the table.
----------------------------------------------------------------------------------------------

Clustered Index: Clustered index physically rearrange the data that users inserts in your tables. It is nothing but a dictionary type data where actual data remains.
Non-Clustered Index: It Non-Clustered Index contains pointers to the data that is stored in the data page. It is a kind of index backside of the book where you see only the reference of a kind of data.
----------------------------------------------------------------------------------------------

To change the default size limit for MyISAM tables, set the myisam_data_pointer_size, which sets the number of bytes used for internal row pointers. The value is used to set the pointer size for new tables if you do not specify the MAX_ROWS option. The value of myisam_data_pointer_size can be from 2 to 7. A value of 4 permits tables up to 4GB; a value of 6 permits tables up to 256TB.

You can check the maximum data and index sizes by using this statement:

SHOW TABLE STATUS FROM db_name LIKE 'tbl_name';
----------------------------------------------------------------------------------------------

Both InnoDB and PostgreSQL - as well as many other databases - use a technique called multi-version concurrency control (MVCC) to provide transaction isolation: transactions should not see the work of other, uncommitted transactions. MVCC means that, when a row is updated, the database stores both the old and new versions of the row.
----------------------------------------------------------------------------------------------

todays quote

2011-02-08T14:31:00.000+05:30

Block Size:
The block size specifies size, that the file system, will use to Read and Write data. Larger block sizes will help improve disk I/O performance when using large files, such as Databases.
Mostly of size 1024, need to be 4096 for bigger file support like in database server case.

Block Size: Different Terms
1: Hardware Block Size = Sector Size
2: File System Block Size = Block Size
3: Kernel Buffer Block Size = Block Size
4: Partition Table Block Size = Cylinder Size
----------------------------------------------------------------------------------------------------------------------------------------------------
MyISAM tables support concurrent inserts. If a MyISAM table has no holes in the middle resulting from deleted or updated records, inserts always take place at the end of the table and can be performed while other clients are reading the table. Concurrent inserts can take place even for a table that has been read-locked explicitly if the locking client acquired a READ LOCAL lock rather than a regular READ lock. If a table does have holes, concurrent inserts cannot be performed. However, you can remove the holes by using OPTIMIZE TABLE to defragment the table.
----------------------------------------------------------------------------------------------------------------------------------------------------
Clustered Index: Clustered index physically rearrange the data that users inserts in your tables. It is nothing but a dictionary type data where actual data remains.
Non-Clustered Index: It Non-Clustered Index contains pointers to the data that is stored in the data page. It is a kind of index backside of the book where you see only the reference of a kind of data.
----------------------------------------------------------------------------------------------------------------------------------------------------

To change the default size limit for MyISAM tables, set the myisam_data_pointer_size, which sets the number of bytes used for internal row pointers. The value is used to set the pointer size for new tables if you do not specify the MAX_ROWS option. The value of myisam_data_pointer_size can be from 2 to 7. A value of 4 permits tables up to 4GB; a value of 6 permits tables up to 256TB.

You can check the maximum data and index sizes by using this statement:

SHOW TABLE STATUS FROM db_name LIKE 'tbl_name';
----------------------------------------------------------------------------------------------------------------------------------------------------

Both InnoDB and PostgreSQL - as well as many other databases - use a technique called multi-version concurrency control (MVCC) to provide transaction isolation: transactions should not see the work of other, uncommitted transactions. MVCC means that, when a row is updated, the database stores both the old and new versions of the row.
----------------------------------------------------------------------------------------------------------------------------------------------------

Purge Bin_Logs, Save Disk Space - Mysql

2011-01-24T11:36:00.000+05:30

Purge Bin_Logs Save Disk Space

Oftenly i meet with a question in mysql that, "How can I remove old mysql-bin log file in log directory?"

So here is the way... sort, good and practically proved.

There is two way to purge the bin-logs
1: Automatic way
2: Manually way, and (May be next step in-case.)

To purge bin-logs automatic, you can add a "expire_logs_days" Variable in my.cnf during the configuration of replication server.

expire_logs_days = 7

It will purged binary logs older than 7 days.The old logs will be purged during the next bin-log switch

Now one condition, if you have deleted some binary logs earlier manually and hasn't updated the ".index" file, then you have to run the below MySQL query.

Note: The "expire_logs_days" settings won't work if you have already deleted some bin logs manually.
It will always tries to find the old deleted bin logs, unless you manually update the index file and delete the binlogs file name which is deleted.

# mysqladmin -u root -p flush-logs

OR

mysql> show slave status \G
check here for current "Master_Log_File"

Like this:
Slave_IO_State: Waiting for master to send event
Master_Host: xxx.xxx.xxx.xx (IP of master server)
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysqlcluster1-bin.000233
Read_Master_Log_Pos: 929859308
Relay_Log_File: mysql-repl-lb-relay-bin.000050
Relay_Log_Pos: 929859453
Relay_Master_Log_File: mysqlcluster1-bin.000233
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:

goto in data dir of mysql slave:

ls -lht on the bases of this: Run the below

mysql> PURGE BINARY LOGS BEFORE '2010-02-27 11:02:30';

but before you purge please make sure that slave is on sync with master or confirm the current status from slaves and also confirm that, you are leaving 2-3 days bin-logs on server.

Block Proxied IPs From Apache

2011-01-21T00:33:00.000+05:30

Few days back i was fighting with a issue.. some one leached our website and showing my website contents on his domain.
Now my seniors start worrying about their product, and it was obvious.
I started digging out some methods.. and started playing with some settings.. like
some apache directives..

Order Deny,Allow
Deny from
Didn't worked...
My websites are Akamaied, so the real IPs i did not able to get...So at end i decided to get Real IPs (Actual IPs - True Client IP)So at first i tried, i should get the actual IP who was hitting by enabling access log .

Here is some settings to get the log in Virtual Host.

LogFormat "%{True-Client-IP}i \"%r\" \"%{Referer}i\"" combined

CustomLog "|/usr/local/sbin/cronolog /usr/local/apache2/logs/domain.com-access-%d.log" combined

# tail -f /usr/local/apache2/logs/domain.com-access-20.log | grep

After getting the IP in access log for culprit domain, added that IP in a file, called in my case host.deny under
/usr/local/apache2/conf/

and in vhost conf, some more entries i made

RewriteEngine On

RewriteMap hosts-deny txt:/usr/local/apache2/conf/host.deny
RewriteCond ${hosts-deny:%{HTTP:True-Client-IP}|NOT-FOUND} !=NOT-FOUND
RewriteRule ^/.* - [R=401,L]

So final configuration is here:
ServerAdmin manish.singh@domain.com
DocumentRoot /var/www/web/html/
ServerName domain.com
RewriteEngine On

LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%V\" \"%{User-Agent}i\"" combined

CustomLog "|/usr/local/sbin/cronolog /usr/local/apache2/logs/domain.com-access-%d.log" combined

RewriteMap hosts-deny txt:/usr/local/apache2/conf/host.deny
RewriteCond ${hosts-deny:%{HTTP:True-Client-IP}|NOT-FOUND} !=NOT-FOUND
RewriteRule ^/.* - [F,L]

So what these blue lines doing here?
RewriteMap hosts-deny txt:/usr/local/apache2/conf/host.deny (RewriteMap Tells Apache to read a source file of type Text File)

RewriteCond ${hosts-deny:%{HTTP:True-Client-IP}|NOT-FOUND} !=NOT-FOUND (RewriteCond tells Apache, this is the condition, read parameter from hosts-deny and meet the condition)

Third line tell Apache, if the above condition meets, the Access Denied the request.

Now your time to test the above at your end.

PostgreSql Backup Script - Bash

2010-11-27T22:34:00.000+05:30

If you are one who is using PostgreSql as your Back-end Database Server for your work, then obviously you need a backup solution, so here this is my try for a small solution for Automated Backup Job in Bash Shell Script.#!/bin/bash
DIR=/data/backups/db_backup/
DT=`date +%Y-%m-%d`
mkdir $DIR/$DT
[ ! $DIR ] && mkdir -p $DIR || :
LIST=$(psql -l | awk '{ print $1}' | grep -vE '^-|^List|^Name|template[0|1]|^\(')
for d in $LIST
do
pg_dump $d | gzip -c > $DIR/$DT/$d.gz
done

Hope this works for you.

Note: If you have compiled your postgresql from source and installed in a different directory, then you may need to SET path for "Bin" directory of postgresql in Environment Variables.

Read Only User In Postgresql

2010-11-21T21:53:00.000+05:30

Read only users in databases servers are frequently required, and generally the method to create such user in different Database servers are very well known, such like in Mysql, Oracle, MSSQL etc...But making a read only user in Postgresql is bit different, so here is a short and smart way.

DB name is "mydb" (i am taking a DB name as mydb)

So just login with Postgres user and DB name on which you want to set read-only permission.

# psql -U postgres mydb

#####Revoke default permissions from public group############

mydb=# REVOKE CREATE ON SCHEMA public FROM PUBLIC;

mydb=# REVOKE USAGE ON SCHEMA public FROM PUBLIC;

##### Add back permissions for your database owner ############

mydb=# GRANT CREATE ON SCHEMA public TO readwriteuser;

mydb=# GRANT USAGE ON SCHEMA public TO readwriteuser;

mydb=# \q

##### Now create a unprivileged User   ######

psql -U postgres -t -c "create role readonlyuser password 'abc123' NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;
mydb=# GRANT USAGE ON SCHEMA public TO readonlyuser;

mydb=# \q

# psql -U postgres -qAt -c "select 'grant select on ' || tablename || ' to \"readonlyuser\";' from pg_tables where schemaname = 'public'" mydb| psql -U postgres mydb

mydb=# select * from pg_namespace where nspname='public';
(Result below:)
nspname | nspowner | nspacl

---------+----------+------------------------------------------------------------------

public     | 10        | {postgres=UC/postgres,mydbuser=UC/postgres,mydbuser_ro=U/postgres}

Now test the "readonlyuser" by creating a table in database mydb

# psql -U readonly mydb

mydb=# CREATE TABLE films (
code        char(5) CONSTRAINT firstkey PRIMARY KEY,
title       varchar(40) NOT NULL,
did         integer NOT NULL,
date_prod   date,
kind        varchar(10),
len         interval hour to minute
);

ERROR: permission denied for schema public

Above you can see that now "readonlyuser" is now not able to create a table even.... It can only perform "SELECT" statements.

Note: These above is just for a specified schema, if you have created a different schema by other name, then change the settings accordingly.

Now Restrict the readonlyuser from pg_hba.conf

add the below line in pg_hba.conf

host    mydb    readonlyuser      192.168.23.72/32          trust

By this, a extra level of security you can force.

Comparing MySQL and Postgres 9.0 Replication

2010-11-20T13:00:00.000+05:30

Hi,

As this post is not mine... but i am glad to share this post with every one.
This one is amazing about mysql vs postgresql replication way.

Click below Link:
Comparing MySQL and Postgres 9.0 Replication

Postgresql 9.0 Replication - Step by Step

2010-10-14T01:18:00.000+05:30

Postgresql 9.0 Streaming Replication - A Big Competition

Now Postgresql 9.0 has come with in-built replication features. NOW A BIG COMPETITOR FOR MYSQL

This method for those who wants to satisfy them by implementing Asynchronous mode of replicationWho wants Synchronous method, and then have to wait for 9.1 to come in existence.I tried to build a Postgresql Master-Slave replication system with new feature “Streaming Replication (SR)”, and this I have to say, this is very easy and stable method for asynchronous method for replication.While building the SR, I felt, this type of SR is like Mysql Master-slave replication, Slaves are getting updates from master from log files (binlog in cas e of mysql and xlog in postgres)

The above statement can be taken as rough idea when comparing Mysql and Postgresql replication method

Steps Included:

1.       Download and build, install Postgresql from source.

2.     Initialize database on master

3.       Some tweaks in postgresql.conf (On Master)

4.       Start backup mode on master

5.       Take snapshot of Data (/usr/local/postgres/data) directory on master

6.       Sync snapshot to slave and remove postmaster.pid on slave

7.       Some tweaks in postgresql.conf and recovery.conf (On Slaves)

8.       At end start the Database on Master and slave

Download the Database: (On Master)

# wget http://wwwmaster.postgresql.org/redir/198/h/source/v9.0.0/postgresql-9.0.0.tar.bz2
# tar xzf postgresql-9.0.0.tar.bz2

If you have fresh CentOS/other linux version, then maybe you have to download and install required packages… In my case I am Using CentOS 5.3 64 Bit

Run the below command as “ROOT”

# yum install -y glibc gcc libtermcap-devel readline-devel zlib-devel
# useradd postgres
# chown –R postgres:postgres postgresql-9.0.0
# su – postgres
# cd postgresql-9.0.0 (extracted directory)

Run the following command to compile and install Postgresql

# ./configure –prefix=/usr/local/postgres –with-perl
# gmake
# su –
# gmake install
# chown -R postgres:postgres /usr/local/postgres
# mkdir /usr/local/postgres/data
# chown postgres /usr/local/postgres/data
# su - postgres
# /usr/local/postgres/bin/initdb -D /usr/local/postgres/data

After the initializing the data directory, we have to tweak---

“/usr/local/postgres/data/postgresql.conf“ file

I am showing you required things only

listen_addresses = '*'
max_connections = 500
shared_buffers = 32MB
wal_level = hot_standby
archive_mode = on
archive_command = 'cp %p /usr/local/postgres/data/pg_xlogarch/%f'
max_wal_senders = 1 ####( this setting for 1 slave can connect with Master)
wal_keep_segments = 32
datestyle = 'iso, mdy'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
default_text_search_config = 'pg_catalog.english'

# mkdir /usr/local/postgres/data/pg_xlogarch/
# su – postgres
# cd /usr/local/postgres/
# vi data/pg_hba.conf (add the below line)

host replication all 172.32.1.2/32 trust

Now start the Database Instance on Master:

As user postgres

# /usr/local/postgres/bin/pg_ctl -D /usr/local/postgres/data -l logfile start

After starting the database Instance, start the backup mode on DB Master

# cd/usr/local/postgres/bin/
# ./psql –c "SELECT pg_start_backup('initial backup for SR')" = database name for which you want replication

# cd /usr/local/postgres/

# tar -czf data.tar.gz data/

Exit from Backup mode
# /usr/local/postgres/bin/psql –c "SELECT pg_stop_backup()"

# rsync –avz data.tar.gz postgres@172.32.1.2:/usr/local/postgrsql/ (run this command after the slave’s Postgresql installation)

On SLAVE:

Same as master download and install Postgresql on slave

Just do not initialize the database:

As user Postgres
# cd /usr/local/postgrsql
# tar xzf data.tar.gz
# chown –R postgres:postgres /usr/local/postgres/data/pg_xlogarch

# cd data

# rm –rf postmaster.pid

# vi /usr/local/postgrsql/data/Postgresql.conf

In short:

listen_addresses = '*'          # Which IP address(es) to listen on
max_connections = 500        # (change requires restart)shared_buffers = 32MB    #min 128kB
hot_standby = ondatestyle = 'iso, mdy'lc_messages = 'en_US.UTF-8'   # locale for system error message
lc_monetary = 'en_US.UTF-8'        # locale for monetary formatting
lc_numeric = 'en_US.UTF-8'          # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting
default_text_search_config = 'pg_catalog.english'

Now create a “recovery.conf” under /usr/local/postgres/data/ and put the below lines: (In Slave)

standby_mode = 'on'
primary_conninfo = 'host=172.32.1.1 port=5432 user=readonly'
restore_command = 'cp /usr/local/postgres/data/pg_xlog/%f %p'

That's it we are done with configurations.

Start the database instance on slave machine

# /usr/local/postgres/bin/pg_ctl -D /usr/local/postgres/data -l logfile start

If all goes well, the on master, you will see like this on Master:

LOG: replication connection authorized: user=readonly host=172.32.1.2 port=55811

And on slave’s log file (logfile)

LOG: streaming replication successfully connected to primary

If fails, check "logfile" and send me error (if in case you feel so)

New Addition: If you want to check, which pgsql server is Master or Slave, then run this command:

# /usr/local/postgres/bin/pg_controldata /usr/local/postgres/data/   (running on Master)
pg_control version number:            903
Catalog version number:               201008051
Database system identifier:           5572911747953590263
Database cluster state:               in production (This shows, it is master server)
pg_control last modified:             Fri 06 May 2011 05:09:20 PM IST
Latest checkpoint location:           2/97482E60
............

On Slave:

Same Line will show as:

Database cluster state: in archive recoveryYou can also use below one: to check

Slave Status:

Postgres=# select pg_last_xlog_replay_location();
pg_last_xlog_replay_location
------------------------------
7B/35F312B0
(1 row)

isten_addresses = '*'          # what IP address(es) to listen on;
max_connections = 500                   # (change requires restart)
shared_buffers = 32MB                   # min 128kB
hot_standby = on
datestyle = 'iso, mdy'
lc_messages = 'en_US.UTF-8'                     # locale for system error message
lc_monetary = 'en_US.UTF-8'                     # locale for monetary formatting
lc_numeric = 'en_US.UTF-8'                      # locale for number formatting
lc_time = 'en_US.UTF-8'                         # locale for time formatting

Linux Commands - Not Usual

2010-10-14T01:16:00.000+05:30

Linux Command Tips

Check Validity of a Certificate File (PEM File)

# openssl x509 -in Certificates.pem -inform PEM -text -noout -enddate

Certificate:
Data:

Version: 3 (0x2)

Serial Number:

5b:84:3e:8f:d3:1e:fc:67

Signature Algorithm: sha1WithRSAEncryption

Issuer: C=US, O=Sabe Inc., OU=Sabe Relations, CN=Sabe Certification Authority

Validity

Not Before: Oct 1 11:56:10 2011 GMT
Not After : Sep 30 11:56:10 2012 GMT

Sometime Apache process, keeps on execution (Seems like Hangs), so generally trying to get the exact PHP file that is running by Apache Process, So here is my Try.

I used Strace to get the opened files by the apache process. (Get PID of

Apache process that is taking time, though you can also get it From top command)

# pstree -p -n | grep http

(This will show each files that is being processed by that Apache Proc)

# strace -p

The list of files could also be get using lsof, but that could not be of full use, as you need the files continuusly using by Apache Process

Create many Files Sequentially

# seq -w 1 30 | xargs -i -t zcat in_Feb2011/in_Files-{}May2011.gz | grep -E 'name.html?secsid=3304847|name.html?secsid=30780899'

Delete Empty Directories

# find folder/ -type d -empty | xargs -i -t rm -rf {}

# find folder/ -type d -empty -delete

Tail, Vmstat and Date in Loop, Output every 10 Sec

# vmstat 1 1;for ((;;));do date; vmstat 10 2 | tail -n1;done

Real Time Monitoring on Linux
# watch -n1 --difference "echo "Uptime"; uptime; echo \n ; ps -eo pcpu,pid,args | sort -k 1 -r |grep -v watch | head -10; echo "\n" ; tail /var/log/cron | grep "check_load"

Find files based and sorted on Size
# find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -nrk 2,2

Check Memory Fault

# dd if=/dev/urandom bs=768304 of=/tmp/memtest count=1050

# md5sum /tmp/memtest; md5sum /tmp/memtest; md5sum /tmp/memtest

Repair Mysql MYISAM File
# myisamchk --force --sort_buffer_size=64M --key_buffer_size=16M --read_buffer_size=8M --write_buffer_size=8M ../data/phplists/phplist_linktrack.MYI

DELETE mail Que from Qmail

Steps:

# qmail stop

# find /var/qmail/queue/mess -type f -exec rm {} \;

# find /var/qmail/queue/info -type f -exec rm {} \;

# find /var/qmail/queue/local -type f -exec rm {} \;

# find /var/qmail/queue/intd -type f -exec rm {} \;

# find /var/qmail/queue/todo -type f -exec rm {} \;

# find /var/qmail/queue/remote -type f -exec rm {} \;

# qmail start

Display sorted process taking most CPU in descending order
# ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

Download FTP files Recursivly using ncftpget

Syntax:

ncftpget -R -v -u "" -p " "

# ncftpget -R -v -u "ibnlive" -p "ibn123" ftpserver /temp/ /www.domain.com/2009_Feb/*

Command to make Services off in Defined Level on Linux

# chkconfig --list | awk '{print $1}' | cut -d: -f1 | grep -vE '^crond|^network|^sshd|^syslog|^iptables' | awk '{print $1}' | while read line; do chkconfig --level 3 $line off; count=`expr $count + 1`; echo $count $line;done

or Use this... One lineer

# chkconfig --list | awk '{print $1}'| grep -vE '^crond|^network|^sshd|^syslog|^iptables' | xargs -i chkconfig --level 3 {} off

Counting Hits from Web Server Access log

# awk '{print $1}' /opt/indian.com/access_log | grep -vE '^:|^common|^-' | sort | uniq -c | sort -nr > /var/www/reports/ips/indian.txt

or # awk '$1>10000 {print $1}' /opt/indian.com/access_log | uniq -c | sort -nr > /var/www/reports/ips/indian.txt

PERL Search and Replace Text Pattern using Perl On linux Platform
# find . -type f -name "*.html" | xargs perl -pi~ -e 's/\/js\/active18\//\/read\/js\/active18\//'

Killing processes in one Line
# kill -9 `ps -ef | grep rsync| grep -v grep| awk '{print $2}'`

Check CPU Temperature
# echo `date +%b-%d-%H:%M:%S` | tr -d '\ 012' ; echo -n ' '; sensors | awk '/CPU Temp:/{ print $3 }'
Check those commands which have been used most
# history|awk '{print $2}' |awk '{print $1}' | sort | uniq -c | sort -rn | head -10
Mount / partition in "Repair Mode" Repair filesystem
# mount -w -o remount /

Free Memory on Linux at Runtime
# sync

# echo 3 > /proc/sys/vm/drop_caches

What is IaaS or Private Cloud ?

2010-10-14T01:15:00.001+05:30

What is IaaS or Private Cloud? And why Enterprise should consider it?

The shortest definition could be. ‘Virtualization on top of an HPC’. Let’s define it like this. What do you think is the biggest limitation to virtualization?

Its, the flexibility of scaling a virtual machine. The scaling will be always limited to the maximum resources available on one particular physical server. So for instanced if let’s say I have a server with 24 Cores and 32GB of Ram. I cannot have a virtual machine which can have more that or even equal to 24Cores and 32GB of RAM.

But let’s say I have 50 such servers. So technically I have 1200 Cores and 1.6TB of RAM. But if for a day even I need a machine with let’s say 50 cores and 64GB of RAM. I can’t have it. I have to buy a new server which has more than 50 cores and 50GB of RAM and then either use it as a physical box or create a virtual machine accordingly on top of it.

But with IaaS, what we do is that we first aggregate our entire datacenter as a single pool of resources and then start provisioning virtual machines on top of it. This is something like first creating a HPC (high performance computing) platform which integrates all our existing resources in one single pool. And then running a virtualization layer on top of it.

Other key benefits
First, the redundancy is inbuilt. So you don’t have to bother about creating failsafe’s, etc. Second. It inherits the functionalities of a Public cloud architecture which makes it elastic in nature. By elastic I mean scaling up and down a virtual machine on-the-fly by adding or removing resources from the cloud pool and that to in an automatic manner and at a need basis. So for instance today a machine needs 15 cores instead of 3. My cloud infrastructure just finds 12 free cores from any server inside my datacenter and allocates it to the VM. And once the need is fulfilled, it releases back the 12 cores to the cloud pool so that they can be used by some other VM.

And third, is the most important benefit is. It gives us the flexibility to add resources from any supporting public cloud if my infrastructure is not able to fulfill a requirement. So, for instance when my VM wanted those 12 extra cores. My Private cloud was only able to find 10 free cores at that time, so it went to Amazon’s EC2 cloud and borrowed the rest of the 2 cores. Used it and when the requirement was fulfilled or 2 more cores were available in my private cloud, it just released the 2 cores back to Amazon. And you will be only charged by Amazon for those 2 cores for that duration of time when you actually used it. This is called a Hybrid Cloud.

Monitor File Activities - Using iWatch

2010-10-14T01:15:00.000+05:30

Monitor File Activities - iWatch

We frequently meet situation, like some files or directories that supposed to secure are having some operations on it, like read, open, close, write etc...

And many times people would like to have a monitoring tool to monitor files activities.

Here’s a solution for this, I got a tool “iWatch”, written in Perl and can also work in Daemon mode.

This tool can be used as to monitor/detect unwanted manipulation on file system and report it to system admins.

For installation this tool requires:

· Linux kernel Version >= 2.6.13

· Perl modules

§ Linux::Inotify2

§ Event

§ Mail::Sendmail

§ XML::Simple

These below are the syntax used by the iwatch:

In the daemon mode, iWatch has following options:

Usage: iwatch [-d] [-f ] [-v] [-p ]

-d Execute the application as daemon.

-f

Specify an alternate xml configuration file.

-p

Specify an alternate pid file (default: /var/run/iwatch.pid)

-v Verbose mode.

And in the command line mode:

Usage: iwatch [-c command] [-C charset] [-e event[,event[,..]]] [-h|--help] [-m ]

[-r] [-s <on|off>] [-t ] [-v] [--version] [-x exception]

[-X ]

Target is the directory or file you want to monitor.

-c command

Specify a command to be executed if an event occurs. And you can use

following special string format in the command:

%c Event cookie number

%e Event name

%f Full path of the filename that gets an event.

%F The old filename in case moved_to event.

%p Program name (iWatch)

%v Version number

-C charset

Specify the charset (default is utf-8)

-e event[,event[,..]]

Specify a list of events you want to watch. Following are the possible events you can use:

access : file was accessed

modify : file was modified

attrib : file attributes changed

close_write : file closed, after being opened in writeable mode

close_nowrite : file closed, after being opened in read-only mode

close : file closed, regardless of read/write mode

open : file was opened

moved_from : File was moved away from.

moved_to : File was moved to.

move : a file/dir within watched directory was moved

create : a file was created within watched directory

delete : a file was deleted within watched directory

delete_self : the watched file was deleted

unmount : file system on which watched file exists was unmounted

q_overflow : Event queued overflowed

ignored : File was ignored

isdir : event occurred against dir

oneshot : only send event once

all_events : All events

default : close_write, create, delete, move, delete_self and move_self.

-h, --help

Print this help.

-m

Specify the contact point's email address.

-r Recursivity of the watched directory.

-s <on|off>

Enable or disable reports to the syslog (default is off/disabled)

-t

Specify a filter string (regex) to compare with the filename or directory name.

-v verbose mode.

--version

Print the version number.

-x exception

Specify the file or directory which should not be watched.

-X

Specify a regex string as exception

And I have used this combination to monitor a Directory (in Recursive mode)

Command:

# iwatch -r -e access,modify -c "ls -lht %f | awk '{print \$3}'|xargs echo '%f was changed by user'" >> /tmp/watched_files.txt -X /home/user1/error/ /home/user1 –d

OUTPUT in watched_files.txt

[ 7/Sep/2010 14:26:59] IN_ACCESS /home/user1/button.php

/home/user1/fbutton.php was changed by user webuser

[ 7/Sep/2010 14:27:09] IN_MODIFY /home/user1/button.php

/home/user1/fbutton.php was changed by user webuser

[ 7/Sep/2010 14:27:10] IN_MODIFY /home/user1/button.php

/home/user1/fbutton.php was changed by user webuser

And at the end, you can set a cron, to send mail to system administrator.

(I am setting cron that will run at 11:55 and send mails)

55 23 * * * /bin/mail -s "Watched Files Activities for DAY `date +%d-%m-%Y`" "delhi.manish@gmail.com" < /tmp/watched_files.txt

Linux How - To's - Free Consultancy

POSTS MOVED

Kernel Panic - Fallback to Old Kernel

Citrix XenServer – iSCSI LUN

Kernel Panic - Fallback to Old Kernel

Citrix XenServer – iSCSI LUN

commands

ext4 disable journal

STORE SSH KEYS IN LDAP

DNS QUERY TIMEOUT ON LINUX CLIENT

Multiple Purpose Expect Script

MPM-Apache-settings

URL Rewriting - Examples

Load Balance Mysql Slaves Using MySQLProxy

Report Developer's About Error/Warning in Codes from Apache Logs

Monitor PostgreSQL with Queries

Some Useful Quotes

todays quote

Purge Bin_Logs, Save Disk Space - Mysql

Purge Bin_Logs Save Disk Space

Block Proxied IPs From Apache

PostgreSql Backup Script - Bash

Read Only User In Postgresql

Comparing MySQL and Postgres 9.0 Replication

Postgresql 9.0 Replication - Step by Step

Linux Commands - Not Usual

Check Validity of a Certificate File (PEM File)

# openssl x509 -in Certificates.pem -inform PEM -text -noout -enddate

Certificate:Data:

Not Before: Oct 1 11:56:10 2011 GMTNot After : Sep 30 11:56:10 2012 GMT

What is IaaS or Private Cloud ?

Monitor File Activities - Using iWatch

Monitor File Activities - iWatch

Certificate:
Data:

Not Before: Oct 1 11:56:10 2011 GMT
Not After : Sep 30 11:56:10 2012 GMT