Linux System Administration

MySQL MyISAM tables Recovery

2011-11-07T19:01:00.002+05:30

Sorry Readers i was busy from last few days. Now I am here to discuss few MySQL related issues .

One of the common problem MySQL MyISAM tables faces are Crashing. Once MyISAM tables crash it take long time to recover. To improve recovery time following parameters can be used

myisam_sort_buffer_size and
myisam_max_sort_file_size
If above given parameters not set correctly then, recovery done using key_cache which may take longer time.

The first we must know about myisam_sort_buffer_size is that , this is a per session buffer. This buffer memory assigned per connection/thread. myisam_sort_buffer_size variable used when no index present in table to help the sorting .

LINUX BOOTING SEQUENCE TROUBLE SHOOTINGS

2011-10-02T14:27:00.001+05:30

Friends last few days were so busy days for most of my seniors.

Actually one of our client has deleted (accidently) some important configuration files from server. We did lot’s of troubleshooting to make server up.

On those trouble shootings I got opportunity to learn some precious things. For me it was like practical review of LINUX BOOTING SEQUENCE TROUBLE SHOOTINGS.

Let me share some boot troubleshooting topics.

Linux boot process

a) Bios initialization

b) Boot loader

c) Kernel initialization

d) init

a) BIOS is the interface between hardware & software in very basic level. It runs POST, and then it looks for peripherals & a device to boot from. at the end of POST , a boot device is selected from list of detected boot device.(it may be floopy,hddcd-rom,NIC)

b) BIOS reads & executes the 1^st physical sector of chosen boot media on system. Usually this is contained in the 1^st 446 byte of hard disk.(in case of GRUB)

1) So BIOS 1^st pass control to IPL within MBR.(This stage is called 1^st stage of booting of Linux)

Note: - If in system more than one OS is installed then boot loader must be configured to pass control to other desired OS.

2) If 1^st stage of booting pass control to Linux then it seeks /boot partition & by finding grob.conf,initrd & other files it completed 2^nd stage of booting.

c) Kernel initialization: - lots of process generated for compiling device driver & to attempt to locate their corresponding drivers.

You can see /var/log/dmesg that contains snapshoot of kernel messages taken just after control pass to init.

Note: - if essential (need to boot) drivers have been compiled as modules instead of into the kernel, and then they must be include in initrd.img. This is then temporarily mounted by kernel on RAM disk to make module available for initialization process.

After loading of all essential drivers, kernel mounts root file system (/) read-only & pass the control to 1^st process (init).

d) INIT:-init reads /etc/inittab (because now / is read-only) & execute all that written in this file as follows

1) Selection of desired run level

2) Execute /etc/rc.d/rc.sysinit

A) Activate udev & selinux

B) Set kernel parameter from /etc/sysctl.conf

C) Set system clock

D) Enable swap partition

E) Set hostname

F) Checking root file system & remount (in read write mode)

G) Activate RAID & LVM

H) Enable disk Quota

I) Check & mount other file system read write mode

J) Clean up stale lock & PID files

3) Runs /etc/rc.d/rc?.d/

(/etc/rc.d/init.d has soft link for corresponding run level)

4) It runs /etc/rc.d/rc.local

5) It starts xinetd service.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

This is the complete booting sequence of Linux. While troubleshooting Linux booting problem it helps to find actual cause for booting failure

Linux booting troubleshooting is as follows

LINUX BOOTING SEQUENCE TROUBLESHOOTING

Case 1: No boot loader splash screen or prompt appears

Cause:

1) Grub.conf miss configures

2) Initrd misplaced or deleted

3) MBR cruuppet

4) /boot partition miss

Grub.conf miss configures

Remedy: try to pass boot location initrd location & kernel module location as follows

Step a) Grub > root (hd0,0)

Note: here hd0,0 means boot partition is in 1^st partition of 1^st hdd.(please use hd for SATA & SCSI hdd also )

Step b) grub > kernel /vmlinuz-$(uname -r) root=LABEL=/ rhgb quiet

Step c) grub > initrd /initrd-$(uname -r).img

Step d) grub > boot

Then after rebooting try to recreate grub.conf file

Initrd misplaced or deleted

Case a) system is up

This is the most fortunate situation for system admins who are managing the server remotely because now once the system will down it will not be up without rescue mode

Remedy:

[root@dh-localhost ~]# mkinitrd /boot/ initrd-$(uname -r).img $(uname -r)

Case b) system is down

Then boot from system via DVD & start it by rescue mode

Remedy:

Step 1) Boot: linux recue

After some process & ittrective prompt you will be on sh prompt

Step 2) chroot /mnt/sysimage

cd /boot

mkinitrd /boot/ initrd-$(uname -r).img $(uname -r)

Then reboot the system by HDD

Note: - in case of fstab is also misconfigured (ie / & other partition will not mount after boot) then chroot command will not work. So in this case your 1^st step would be to correct fstab then perform above steps as follows

Step a) boot from DVD or other bootable media

Step b) chroot /mnt/sysimage (you will find chroot error here )

Step c) mkdir /test

e2label /dev/sda1 (if it will show boot them mount it otherwise try to search by e2label /dev/sda2 ownwards)

mount /dev/sda1 /test

cd /test

cd grub

vi grub.conf (please make it correct)

mkdir /data

mount /dev/sda2 /data (mounting / partition on /data )

cd /data

cd /etc

vi fstab (please correct it )

Then reboot the system & follow previous step

Mbr corrupt

Case a) system is up

Method 1

[root@dh-localhost ~]# /sbin/grub-install /dev/sda

Method 2

[root@dh-localhost ~]# grub

grub> root (hd0,0)

grub > setup (hd0)

grub > quit

Case b) system is down

Boot system by rescue mode

Sh #Chroot /mnt/sysimage

Sh # /sbin/grub-install /dev/sda

Note : A smart system admin always take backup of MBR as follows

[root@dh-localhost ~]# dd if=/home/mbr of=/home/mbr bs=1 count 500

For restoration for MBR

[root@dh-localhost ~]# dd if=/home/mbr of=/home/mbr bs=1 count 500

/boot partition miss

In this case if you have backup of /boot then you can recover it otherwise you need to rebuild the server again.

What in NDR Spam ?

2011-08-14T10:36:00.001+05:30

Redirect http to https

2011-07-19T10:37:00.000+05:30

Suppose you have a requirement to divert your http traffic to https. There are many ways to achieve this , but as a Apache web administrator i prefer to apply following Rewrite rule for my site

RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://abc.com/$1 [R,L]

Here in above example example site is abc.com

Caching Configuration in Apache

2011-07-19T10:23:00.001+05:30

The goad of caching in Apache could be

-Reduce Number of request in man cases

-Eliminate need to send full response in many cases

For former we use Expiration mechanism and for later we use validation mechanism.

For Apache mod_expires and mod_headers handles cache control through http headers.

mod_expires module control setting of expires and cache-control http header in server response.

For example to enable mod_expires for all .html files, we can use following syntax

ExpiresActive On

ExpiresDefault A300

Expires A86400

In Above example, default caching is for 5 minutes(A300) but for html files caching is 1Day(A86400).

Now suppose we want to disable caching for dynamic pages located in folder /var/www/phpfiles

Header Set Cache-Control "max-age=0,no-store"

Prevent DDOS attack on web site

2011-05-30T10:14:00.001+05:30

There can a number of idea to prevent Distributed Denial of Service attack on website . Here i am sharing a basic idea that i use now a days to mitigate DDOS from malicious web crawler .As I noticed when a malicious crawler access web site we found log entry by a particular User Agent in access-log. What i do is rewrite request from such User Agent to Forbidden messages using Rewrite module

RewriteCond %{HTTP_USER_AGENT} Malicious_User_Agent [NC]

RewriteRule .* - [F,L]

http traffic load

2011-05-30T09:42:00.000+05:30

On some occasion , we need to calculate http traffic of our system. Although there are a number of tools and utilities but i avoid to install and use them in small size set-up.

I decided to use tcpdump command , tcpdump command is very useful command that help to decode network traffic. For example for above given requirement i applied tcpdump command in following format.

tcpdump -s 1500 -Svni eth0 tcp and port 80

Above given command will display http packets (tcp packet of port number 80). In output of command we can get total length which is equal to IP Header Length + TCP Header Length + Payload Length . As we know IP Header + TCP Header length will be 52 , so we can calculate payload length which will be Total Length -52.

Hack Attempt

2011-05-23T10:52:00.001+05:30

Today i noticed ../../../../../../../../. entries in my access log file of one of my web site.

I wonder is it a hack attempt ? If some hacker trying to doing scanning to find vulnerability of my system?

Yes my doubt was genuine , It was a hack attempt. It was Local File Intrusion vulnerability. The attempt was to check if /proc/self/environ is accessible, if anyway it access that then do attempt to inject code and get access to shell.

Still i am working to way to mitigate such attack, immediate solution for me was to block ip which was doing such attempt.

log-rotate in Linux

2011-05-18T05:37:00.001+05:30

Today my client claimed to reset the log rotation of access_log to 1 day and 1 log of all his 532 websites hosted in a server .

Also he claimed that all rotated log will be mailed to him .

Then I did :->

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Step 1:-> create a file

# vi logger.conf

/var/www/vhosts/*/statistics/logs/access_log

{daily

rotate 1

mail amit@abc.com

compress

}

:wq!

Then I simply type

Step2:-> logrotate logger.conf

++++++++++++++++++++++++++++++++++++++++++++

There are other options here :->


compress	This is used to compress the rotated log file with gzip.
nocompress	This is used when you do not want to compress rotated log files.
copytruncate	This is used when processes are still writing information to open log files. This option copies the active log file to a backup and truncates the active log file.
nocopytruncate	This copies the log files to backup, but the open log file is not truncated.
create mode owner group	This rotates the log file and creates a new log file with the specified permissions, owner, and group. The default is to use the same mode, owner, and group as the original file.
nocreate	This prevents the creation of a new log file.
delaycompress	When used with the compress option, the rotated log file is not compressed until the next time it is cycled.
nodelaycompress	This overrides delaycompress. The log file is compressed when it is cycled.
errors address	This mails logrotate errors to an address.
ifempty	With this, the log file is rotated even if it is empty. This is the default forlogrotate.
notifempty	This does not rotate the log file if it is empty.
mail address	This mails log files that are cycled to an address. When mail log files are cycled, they are effectively removed from the system.
nomail	When mail log files are cycled, a copy is not mailed.
olddir directory	With this, cycled log files are kept in the specified directory. This directory must be on the same filesystem as the current log files.
noolddir	Cycled log files are kept in the same directory as the current log files.
prerotate/endscript	These are statements that enclose commands to be executed prior to a log file being rotated. The prerotate and endscript keywords must appear on a line by themselves.
postrotate/endscript	These are statements that enclose commands to be executed after a log file has been rotated. The postrotate and endscript keywords must appear on a line by themselves.
daily	This is used to rotate log files daily.
weekly	This is used to rotate log files weekly.
monthly	This is used to rotate log files monthly.
rotate count	This specifies the number of times to rotate a file before it is deleted. A count of 0 (zero) means no copies are retained. A count of 5 means five copies are retained.
tabootext [+] list	This directs logrotate to not rotate files with the specified extension. The default list of extensions is .rpm-orig, .rpmsave, v, and ~.
size size	With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem)

For more detail you can read man page of logrotate .

Logging Bind Queries

2011-04-24T21:04:00.002+05:30

To log DNS queries on linux, we use logging feature of Bind server.Suppose you want to log dns queries in file /var/log/querylog.log in this case make following entries in /etc/named.conf.

  logging {
   channel querylog{
   file "/var/log/querylog";
   severity debug 3;
   print-category yes;
   print-time yes;
   print-severity yes;
   };
   category queries { querylog;};
   };

After putting above lines in /etc/named.conf , It is advisable to create log file manually

root#touch /var/log/querylog

And and make it writeable by user named

root#chown named.named /var/log/querylog

Restart bind server to apply changes

root#/etc/init.d/named restart

Do some queries to dns server using dig or nslookup

View query now in file /var/log/querylog.

root#tail /var/log/querylog

Sendmail Spamming Prevention

2011-04-16T20:37:00.004+05:30

Sendmail is defame for security holes. But this is also a fact that sendmail is one of popular MTA. Its not possible to avoid spamming through sendmail completely but yes sendmail can be configured in better way to limit spamming though it.

I am here discussing some of the options which can be useful in this context. Although this is a subjective discussion that these options are how much useful in different scenario.

confMAX_DAEMON_CHILDREN is one of such option in sendmail.mc which specify number of Daemon's children. But these children process can handle incoming and outgoing traffic both.

confMAX_QUEUE_RUN option specify how many queued message to process every time queue is run.

confQUEUE_LA and confREFUSE_LA options are also very useful. confQUEUE_LA limit at what system load mail will queued for later processing and confREFUSE_LA specify at what load sendmail will reject will (Even will not queue for later processing).

Authentication in single user mode

2011-04-12T00:28:00.002+05:30

After a long time i am here with an security trick. Many occasion we need to move into single user mode , which is very simple. But this can be security risk also to overcome this either we can use password in grub boot loader or we can go with following simple solution

---------------------------------------------------

open /etc/inittab file and put following line in this

~~:S:wait:/sbin/sulogin

Extended Globbing

2011-03-07T19:53:00.001+05:30

As we know Wildcard in bash refer as pathname expansion and also as globbing. The use of * , ? etc in command parameter it represent something. For example following command

root#echo *

What you guessing? It should print * , but no it will not print * but instead it will list you files and folders in current directory.

But *, ? and [] are not form of wildcard supported by bash. The other form are refer by Extended Globbing. Unlike normal globbing Extended Globbing has to be enable before using. To enable extended globbing use following command.

  root# shopt -s extglob

Example: Suppose you need to delete all subfolder except the folder ABC. Following simple command will serve this purpose
  root# shopt -s extglob
  root# rm -rf !(ABC)

What is LUN?

2011-03-05T16:18:00.000+05:30

If suppose we got a large storage array, and requirement is to not allow one server to use all storage spaces, so it need to divided into logical units as LUN(Logical Unit Number). So LUN allow us slice storage array into usable storage chunks and present same to server. LUN basically refer to either a entire physical volume or subset of larger physical disk or volume. LUN represent logical abstraction or you can say virtual layer between physical disk and application. A LUN is scsi concept.

Restrict telnet login by user

2011-03-05T09:25:00.000+05:30

As we know ssh and telnet can be used for remote login by user. It is very simple to disable telnet service so that no one can login via telnet. But what if we want some user to log in via telnet and others not. For example suppose it want to allow user vishesh to log in via telnet and other users completely not allowed login via telnet

pam_succeed_if pam module should be configured in /etc/pam.d/telnet file to achieve this. Put following entries in /etc/pam.d/telnet

auth required pam_succeed_if.so user=vishesh quiet

My advice is create a group by name telnet and add user to that group to allow telnet login, in this scenario put following line as 2nd line in /etc/pam.d/telnet

auth required pam_succeed_if.so quiet user in group telnet

Recover MySql root Password

2011-03-04T18:31:00.001+05:30

I am not a mysql expert but use mysql as a database for storing data generated by applications. I stuck today when i forgot root password of mysql. After spending few minutes i got following work around.

root#mysqld_safe --skip-grant-tables&

Above command started mysqld daemon without grant tables, so no password will be asked if i need to go to mysql prompt

root#mysql

   mysql>use mysql
   mysql>update user set password=PASSWORD('abcd') where User='root'
   mysql>flush privileges
   mysql>quit

Now i restarted the mysqld server and problem get solved.

Remove usb device from command line

2011-03-04T10:26:00.003+05:30

As we use Eject option for usb devices in graphical , what if we need to do same thing using command line. Following command can do this

root# echo 1>/sys/bus/usb/devices/usb1/remove

To poweroff the usb device, hdparm command can also be used in following way

root# hdparm --command=stop /dev/sda1

If usb device has to be put in sleep mode, following command can be used

root#hdparm -Y /dev/sda1

In above given examples /dev/sda1 refering plugged usb device.

Recover Slave OpenLdap Server

2011-03-03T12:44:00.000+05:30

If slurpd based slave openldap server become inconsistent. Following steps can be followed for immediate recovery

Put the master slapd server in read only mode
Completely stop slapd on slave
Clear the content of data directory, (/var/lib/ldap)
Copy fresh dump from master and paste it into slave server
Use slapadd to rebuild the directory on slave.
Start the slave slapd server in normal mode
Switch master slapd server in previous (read/write) mode.

Bandwidth testing using command

2011-02-28T12:28:00.000+05:30

Sometime we need to test bandwidth between two linux system. Here in this post i am showing by example to test bandwidth without any network monitoring tool.
Suppose you need to test bandwidth between two linux hosts one in Host1 and another is Host2. Execute following command on Host1
   root@Host1# nc -u -l 7654 >/dev/null
On Host2 run following command
   root@Host2# dd if=/dev/zero bs=1MB |nc -u Host1 7654 & pid=$(pidof dd)
   while((1)) ; do kill -USR1 $pid; sleep 2; done

Now analyze the output, How much data copied in how many seconds and calculate your network bandwidth.

Incremental Backup using tar

2011-02-26T21:50:00.002+05:30

GNU tar currently provide following options to handle incremental backup

--listed-incremental=snapshot file (-g snapshot file)

--incremental (-G)

Examples

root# tar --create --file=archive1.tar listed-incremental=/var/log/usr.snar /usr

Above given command will create archive1.tar as incremental backup of /usr filesystem, additional metadata will stored in file /var/log/usr.snar file . If /var/log/usr.snar file not exist it will get created, the created archive will then level 0 backup.

Now suppose for the same above given example, if /var/log/usr.snar exist. Then above given will check which files get modified and only those files stored in archive, so that will be level 1 backup.

So the best option is take level 0 (full) backup first as following

root# tar --create --file archive1.tar --listed-incremental=/var/log/usr.snar

copy that file

root#cp /var/log/usr.snar /var/log/usr.snar-1

Then take incremental backup as follows

root#tar --create --file archive2.tar --listed-incremental=/var/log/usr.snar

To extract content from backup we also have to follow same steps which followed at the time of backup. In our example the procedure will be as follows
root# tar --extract --listed-incremental=/dev/null --file=archive1.tar
Followed by
root#tar --extract --listed-incremental=/dev/null --file=archive2.tar

rsync in Batch mode

2011-02-26T14:34:00.001+05:30

Batch mode can be used to apply same set of changes to many identical system. Suppose same of tree of a filesystem has to be replicated a number of hosts. Updates in one tree has be propagated to many hosts. In order to achieve this rsync can be used in batch mode. The --write-batch option tell rsync client to store in a batch file all the information needed to repeat this information against the other.

To apply the recorded change rsync has to be run with --read-batch option by specifying the same batch file. One thing also need to keep in mind that one bash file also get created in this process. Let us put this in an example command

root#rsync --write-batch=lists -a host:/source/dir /testdir

root#rsync --read-batch=lists -a /datadir

What is archive mode in rsync ?
As you know when i apecify -a option with rsync that means i want to apply rsync command in archive mode. Archive mode considered as most suitable for taking backup. Putting -a basically a shortcut of bunch of switches which includes follwings
-r recursive
-l copy symlinks as symlinks
-p preserve permission
-t preserve modification time
-g preserve group
-o preserve owner
-D device specials
And which exclude followings
-H preserve hard link
-A preserve ACLs
-X preserve extended attribute

So how concise and smart to use -a instead of mentioning above given bunch of switches.

Applying filter in rsync

2011-02-26T13:48:00.003+05:30

As we know rsync command is very helpful in taking backup. Sometime need to backup of some folder by including some sub folders and excluding some of them. by an e synchronizing with remote system we can mention folders which to be included and which to be excluded as command line options. Let us understand it by an example, suppose one need to take take backup of / by excluding /home and by including /home/user2 .

Following command help out

root# rsync -av --filter=+home --filter=+/home/user2 --filter=-/home/* / user1@remotehost:/folder

Without the delete option per directory options are only relevant on sending side, to exclude the merge files themselves without affecting the transfer. The -e modifier can do this easily. For example
root# rsync -av --filter=':e .mp3' host:src/dir /dst

Nmap vs Nessus

2011-02-24T23:40:00.000+05:30

Nmap and Nessus both are network vulnerability scanner The history of vulnerability scanner is very exciting. In initial days Telnet was used to find open port stat. Over time a set of scripts was developed to make vulnerability scanning simple, one of such script set was SATAN (Security Administrator Tool for Analyzing Network). After SATAN some of popular commercial tool was ISS ( Internet Security System) .

As the Open Source movement became popular, in network security field Nmap was released in 1997 and Nessus released in 1998 both was open source. Nessus became proprietary in 2005 although for personal use this product is still free.

As per as use Nmap use is concern, it is very helpful in

Find the status of host (up or down)
Find the open ports on a particular hosts
OS and its version on hosts (windows xp or linux ?)
Presence of firewall
List of network services running on host

Nessus can do almost all which Nmap do, other than that Nessus can find CVE(Common Vulnerability Exposures) using its plug in. Nessus should be used in you have following security needs

Security audit
Vulnerability Scanning and analysis
Sensitive data discovery
Open port scanner (like Nmap)
Asset & Process profiling

One point to be noted that Nmap can work more effectively if we use its Scripting Engine feature

VirtualHosts and SSL

2011-02-23T11:18:00.000+05:30

Due to limitation of SSL protocol, It is impossible to host more than one SSL Virtualhosts on the same ip address and port. The limitation of SSL protocol is that. Apache needs to know the name of host in order to select correct certificate. The name of host part encapsulated inside HTTP Request Header , which is encrypted content. Since host name part will not revealed unless encryption channel get established so it's not possible to create multiple SSL Virtualhost on one ip and port.

Apache allow to configure name-based virtual host, but it always use configuration of first configured virtual host ssl settings for rest of virtual hosts. Logically you can say SSL create encryption layer between client and server on which traffic for all the virtual hosts can move. In reality it doesn't create any sort of encryption channel between client and virtual host. All the virtual hosts of configured apache server will use same SSL certificate for encryption, it may acceptable in many circumstances. But the need of independent SSL Certificate for virtual hosts are also very common.

Background Flushing

2011-02-22T09:38:00.000+05:30

Background flushing in linux happens when either of following condition reached

Too much written data is pending (/proc/sys/vm/dirty_background_ratio)
Timeout of pending writes is reached(/proc/sys/vm/dirty_expire_centisecs)

In above circumstances, unless another hit ( /proc/sys/vm/dirty_ratio) is not reached ,more written data may be cached and further writes will block.

In theory this will create a background process without disturbing other process but in practical it does disturbance to the other process which are doing uncached reading or synchronized writing. The reason of this may be background process writing with the 100% device speed, as a result other process halt. I got the fact that clearing dirty caches may solve this issue

Clear Cache from memory

root# echo 3>/proc/sys/vm/drop_caches

And its recommended to run 'sync' command before clearing cache