Tune2Linux

rsync with delete option and different ssh port

2017-03-21T19:49:00.002-07:00

How to rsync e.g PIPELINE dir from Source to Destination?

#rsync -avzr --delete-before -e "ssh -p $portNumber" /local/path/. user@remoteip:/path/to/files/

#rsync -avzr --delete-before -e "ssh -p 5555" /root/PIPELINE root@192.168.10.203:~/

How to rsync from Destination to Source?

#rsync -avzr -e "ssh -p $portNumber" user@remoteip:/path/to/files/ /local/path/.

#rsync -avzr -e "ssh -p 5555" root@192.168.10.203:~/PIPELINE

iptables ratelimit connection limiting

2016-12-06T01:55:00.001-08:00

Steps:

1. Create a file iptables.rules with below content.
2. Load rules #iptables-restore < iptables.rules
3. Also Start http service. #service httpd restart
4. cd /var/www/html and create a index.html file.
5. From other system try to ping firewall machine- Observe what happens?
6. From other system try $curl firewall machine> - Observe what happens after 10 requests.

Happy firewalling ;)

# cat iptables.rules

# Generated by iptables-save v1.4.7 on Wed Jun 8 00:51:34 2016
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:ICMPCHK - [0:0]
:tcpchk - [0:0]
:ssh - [0:0]
:http - [0:0]

# Rule will pass to ssh chain
-A INPUT -p tcp -m tcp --dport 5555 -j ssh

#Rule will pass to ICMPCHK chain
-A INPUT -p icmp --icmp-type echo-request -j ICMPCHK

#Rule will pass to http chain
-A INPUT -p tcp -m tcp --dport 80 -j http

# Any traffic to port 80 will be accepted
#-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

# Drop all packets
-A INPUT -j DROP

# Output chain
-A OUTPUT -p udp -m udp --dport 5555 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 5555 -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -j DROP

-A ssh -p tcp -m tcp --dport 5555 -m state --state NEW -m recent --set --name DEFAULT --rsource
-A ssh -p tcp -m tcp --dport 5555 -m state --state NEW -m recent --rcheck --seconds 60 --hitcount 2 --name DEFAULT --rsource -m limit --limit 2/min -j LOG --log-prefix "SSH-ATTACK : " --log-level 5
-A ssh -p tcp -m tcp --dport 5555 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 --name DEFAULT --rsource -j REJECT --reject-with tcp-reset
-A ssh -p tcp -m tcp --dport 5555 -j ACCEPT
-A ssh -j RETURN

#Rule for counting of incoming connections.
-A http -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --set --name DDOS

#Rule to write log
-A http -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --update --name DDOS --seconds 30 --hitcount 10 -j LOG --log-prefix "DDOS : " --log-level 5

# Rule check number of connection made from an ip to port 80. If count is >10, Client is made to wait for 30 seconds and allowed.
-A http -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --update --name DDOS --seconds 30 --hitcount 10 -j DROP

# Generally accept http
-A http -j ACCEPT
-A http -j RETURN

#Rule to limit pings
-A ICMPCHK -p icmp --icmp-type echo-request -m limit --limit 3/min --limit-burst 5 -j ACCEPT
-A ICMPCHK -p icmp --icmp-type echo-request -m limit --limit 3/min --limit-burst 5 -j LOG --log-prefix "ICMP-CHK-LIMITED: "
-A ICMPCHK -p icmp -j DROP
-A ICMPCHK -j RETURN

COMMIT
*nat
:PREROUTING ACCEPT [15030:976242]
:POSTROUTING ACCEPT [5423:372090]
:OUTPUT ACCEPT [4987:357161]
COMMIT

creating tar archive by excluding file

2016-12-02T07:21:00.002-08:00

how can i exclude files when creating tar archive?

tar -zcv -f target.tar.gz --exclude-from='exclude.txt' source/

Now, in your exclude.txt file you can place the paths to exclude:

logs/*
errors
data/*.bin

xsos tool for sysadmins and support techs

2016-11-03T04:42:00.001-07:00

a tool for sysadmins and support techs

curl -Lo ./xsos bit.ly/xsos-direct;

chmod +x ./xsos;

./xsos -ya

Refer: https://access.redhat.com/discussions/469323

linux dns client cache ttl

2015-11-06T03:31:00.002-08:00

Checking TTL's DNS

Incidentally to check what the TTL is for a given entry you can use dig with the following switches.

Example

$ dig +nocmd www.google.com +noall +answer | tail -1
www.google.com. 137 IN A 74.125.225.82

$ dig +nocmd www.google.com +noall +answer | tail -1
www.google.com. 135 IN A 74.125.225.115

So the TTL for this response is 137 seconds. Waiting ~2 seconds and running it again shows 135 seconds. The TTL means how much time is left until the DNS entry expires, and we need to go query the authoritative server for the domain.

Checking Max TTL's

If we were to query the authoritative server.

$ dig @ns1.google.com +nocmd www.google.com +noall +answer | tail -1
www.google.com. 300 IN A 74.125.225.210

So the actual TTL for this entry is 300 seconds.

NOTE: The authoritative server is also known as the SOA - Start of Authority.

SOA information

You can query the domain further for SOA information.

$ dig +nocmd dyndns.org any +multiline +noall +answer
dyndns.org. 596 IN SOA ns1.dyndns.org. hostmaster.dyndns.org. (
863998266 ; serial
600 ; refresh (10 minutes)
300 ; retry (5 minutes)
604800 ; expire (1 week)
600 ; minimum (10 minutes)
)
dyndns.org. 85904 IN NS ns5.dyndns.org.
dyndns.org. 85904 IN NS ns1.dyndns.org.
dyndns.org. 85904 IN NS ns2.dyndns.org.
dyndns.org. 85904 IN NS ns3.dyndns.org.
dyndns.org. 85904 IN NS ns4.dyndns.org.
dyndns.org. 12268 IN MX 10 mail.dyndns.com.
dyndns.org. 12268 IN MX 20 mx2.mailhop.org.
dyndns.org. 179 IN A 204.13.248.116

Changing TTLs

The only way to change a DNS entry's TTL (outside of some sort of API that your registrar might provide) is through the server.

Example

Within Bind you could setup your zone file like so:

;Zone file for exampleweb.com
$TTL 14400
@ 86400 IN SOA ns.exampleweb.com. admin.exampleweb.com. (
2009022402 ; serial, todays date+todays
86400 ; refresh, seconds
7200 ; retry, seconds
3600000 ; expire, seconds
86400 ) ; minimum, seconds
exampleweb.com. 86400 IN NS ns.exampleweb.com.
exampleweb.com. 86400 IN NS ns1.exampleweb.com.
exampleweb.com. IN A 209.59.139.21
localhost IN A 127.0.0.1
exampleweb.com. IN MX 0 exampleweb.com.
mail IN CNAME exampleweb.com.
www IN CNAME exampleweb.com.
ftp IN A 209.59.139.21
cpanel IN A 209.59.139.21
webmail IN A 209.59.139.21
The above macro, $TTL would set the TTL to 14400 seconds for any entries, unless it get's overridden for particular entries.

Linux add swap

2015-09-17T23:41:00.000-07:00

sudo dd if=/dev/zero of=/swap bs=1M count=1024

sudo mkswap /swap

sudo swapon /swap

Shrewsoft vpn disconnects frequently

2015-06-10T18:19:00.000-07:00

The workaround is to set

Phase1 key life time to 180 sec while keeping 
Phase2 key life time on default 28800.

This will force a rekey before the SA is deleted from the SRX. Tunnel connectivity is not disrupted and the tunnels stays up.

Good luck!

email previous day log files

2015-06-08T00:35:00.005-07:00

#!/bin/bash

cd /var/log

fil=$(mktemp)
stat -c "%y|%n" rpmpkgs-*.gz | grep `date --date="1 days ago" +%Y-%m-%d` | awk -F"|" '{print $2}' > $fil

for i in `cat $fil`;
do
/usr/bin/uuencode $i | mail -s "$i" youremail@example.com
done

rm $fil

Have a great day!

zabbix trigger on changed values

2015-04-18T22:02:00.001-07:00

[03:37:20] root@simba:[/etc/zabbix]
# zabbix_get -s 192.168.1.51 -k redis[llen,storeapp:failed]
525
[05:19:44] root@simba:[/etc/zabbix]
# zabbix_get -s 192.168.1.52 -k redis[llen,storeapp:failed]
1470

changed trigger for failed jobs:

{redis.host:redis[llen,storeapp:failed].change()}<>0

so it will fire up when Items in Failed Queue is changed, not related on how many items we do have in queue.

ss command socket stats

2015-04-12T22:19:00.001-07:00

Check socket stats on e.g ip 192.168.1.10

ss -tnao dst 192.168.1.10

http://www.binarytides.com/linux-ss-command/

mysql binary logs

2015-01-12T22:21:00.003-08:00

Please do not just delete them in the OS.

You need to let mysqld do that for you. Here is how mysqld manages it:

The file mysql-bin.index keeps a list of all binary logs mysqld has generated and auto-rotated. The mechanisms for cleaning out the binlogs in conjunction with mysql-bin.index are:

PURGE BINARY LOGS TO 'binlogname';
PURGE BINARY LOGS BEFORE 'datetimestamp';

These will clear all binary logs before the binlog or timestamp you just specified.

For example, if you run

PURGE BINARY LOGS TO 'mysql-bin.000223';

this will erase all binary logs before 'mysql-bin.000223'.

If you run

PURGE BINARY LOGS BEFORE DATE(NOW() - INTERVAL 3 DAY) + INTERVAL 0 SECOND;

this will erase all binary logs before midnight 3 days ago.

If you want to have binlog rotated away automatically and keep 3 days woth, simply set this:

mysql> SET GLOBAL expire_logs_days = 3;

then add this to /etc/my.cnf

[mysqld]
expire-logs-days=3

and mysqld will delete them logs for you

SHOW SLAVE STATUS\G

This is critical. When you run SHOW SLAVE STATUS\G, you will see two binary logs from the Master:

Master_Log_File
Relay_Master_Log_File

When replication has little or no lag these are usually the same value. When there is a lot of replication lag, these values are different. Just to make it simple, choose whatever Relay_Master_Log_File is, and go back to the Master and run

PURGE BINARY LOGS TO 'Whatever Relay_Master_Log_File Is';

That way, replication is not interrupted.

Give it a Try !!!

how to redirect stdout & stderr to logger?

2015-01-12T18:26:00.001-08:00

logger come in handy

* * * * * cd /usr/local/bin; sh test_hi.sh 2>&1 | logger

# cat test_hi.sh
#!/bin/bash

echo "Hi testing logger.. thanks"

Test

# tail -f /var/log/messages

Jan 13 02:20:01 simba-c1 root: Hi testing logger.. thanks

If you don't need any log then you may simple use like

* * * * * cd /usr/local/bin; sh sh test_hi.sh >/dev/null 2>&1

Kernel parameters for high-load Linux

2015-01-11T23:35:00.001-08:00

Kernel parameters for high-load Linux

Edit two config files:


% vi /etc/sysctl.conf

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.domainname=servername
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_fin_timeout = 3
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.somaxconn = 16384
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 3000
#net.ipv4.tcp_congestion_control = htcp #for centos 5
net.ipv4.tcp_congestion_control = cubic #for centos 6
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.neigh.default.unres_qlen = 6
net.ipv4.neigh.default.proxy_qlen = 96
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 10
fs.file-max = 360000


% vi /etc/security/limits.conf

* soft nofile 360000
* hard nofile 360000

and reboot…
http://wenxueliu.github.io/blog/10/10/2014/linux-tuning/

What is Overcommit? And why is it bad?

2014-11-26T22:40:00.002-08:00

What is Overcommit? And why is it bad?

From kernel source/Documentation:
When this flag is 0, the kernel attempts to estimate the amount
of free memory left when userspace requests more memory.
When this flag is 1, the kernel pretends there is always enough
memory until it actually runs out.
When this flag is 2, the kernel uses a "never overcommit"
policy that attempts to prevent any overcommit of memory.
Note that user_reserve_kbytes affects this policy.
This feature can be very useful because there are a lot of
programs that malloc() huge amounts of memory "just-in-case"
and don't use much of it.
The default value is 0.

There are a lot of misunderstandings of memory management on Linux, leading to a lot of bad software that fails to robustly handle low-memory conditions. This all stems from a basic myth:

On Linux, malloc never fails. It always returns a pointer to allocated memory, but later your application might crash attempting to access that memory if not enough physical memory is available.

Where does this myth come from? Something called overcommit.
When a system such a Linux utilizing virtual memory allocates memory to a userspace process (via brk or mmap), there is no fixed correspondence between the virtual memory created in the process's virtual address space and the physical memory of the machine. In fact, before the memory is first used, it's likely that there's not any correspondence at all to physical memory; only after the first attempt to access the memory does the kernel have to setup a physical memory corresponding to it.
Overcommit refers to the practice of giving out virtual memory with no guarantee that physical storage for it exists. To make an analogy, it's like using a credit card and not keeping track of your purchases. A system performing overcommit just keeps giving out virtual memory until the debt collector comes calling — that is, until some program touches a previously-untouched page, and the kernel fails to find any physical memory to instantiate it — and then stuff starts crashing down.
What happens when “stuff starts crashing down”? It can vary, but the Linux approach was to design an elaborate heuristic “OOM killer” in the kernel that judges the behavior of each process and decides who’s “guilty” of making the machine run out of memory, then kills the guilty parties. In practice this works fairly well from a standpoint of avoiding killing critical system processes and killing the process that’s “hogging” memory, but the problem is that no process is really “guilty” of using more memory than was available, because everyone was (incorrectly) told that the memory was available.
Suppose you don’t want this kind of uncertainty/danger when it comes to memory allocation? The naive solution would be to immediately and statically allocate physical memory corresponding to all virtual memory. To extend the credit card analogy, this would be like using cash for all your purchases, or like using a debit card. You get the safety from overspending, but you also lose a lot of fluidity. Thankfully, there’s a better way to manage memory.
The approach taken in reality when you want to avoid committing too much memory is to account for all the memory that’s allocated. In our credit card analogy, this corresponds to using a credit card, but keeping track of all the purchases on it, and never purchasing more than you have funds to pay off. This turns out to be the Right Thing when it comes to managing virtual memory, and in fact it’s what Linux does when you set the vm.overcommit_memory sysctl parameter to the value 2. In this mode, all virtual memory that could potentially be modified (i.e. has read-write permissions) or lacks backing (i.e. an original copy on disk or other device that it could be restored from if it needs to be discarded) is accounted for as “commit charge”, the amount of memory the kernel as committed/promised to applications. When a new virtual memory allocation would cause the commit charge to exceed a configurable limit (by default, the size of swap plus half the size of physical ram), the allocation fails.
Unfortunately, a lot of application developers like overcommit, presumably for two reasons:

It allows you to allocate a ridiculous amounts of memory as long as you know you’ll only make sparse use of it. In our credit card analogy, this is like a contractor going to a building supply store and buying twice the amount of materials they expect to need using a credit card, knowing they’ll be safe as long as they go back and return the unused materials for a refund before the credit card bill is due.
It gives you an excuse to be lazy handling errors. You can rationalize ignoring the return value of malloc on the basis that, due to overcommit, even if you check the return value you can’t be sure to avoid crashing later when the kernel doesn’t have enough physical memory to instantiate your virtual memory.

The first reason is actually fairly legitimate, but overcommit is not the right solution. Instead, applications which want to use large amounts of virtual memory without getting charged for it should use the right protections to ensure that memory that won’t be written is mapped read-only. This is robust and portable, unlike relying on overcommit.
The second reason is pure laziness and foolishness. The fact that you can’t detect all errors on a system that’s configured in a non-robust way (the Linux default) is not an excuse for failing to detect other errors (like exhausting virtual address space) and crashing on systems that were intentionally configured for maximum robustness. In fact, even in the default configuration, Linux tries to avoid severe overcommit; it just doesn’t do detailed accounting.
Overcommit is harmful because it encourages, and provides a wrong but plausible argument for, writing bad software. While the number of applications that completely ignore the failure of malloc seems to be shrinking, plenty of applications and even libraries intended for use in serious software utilize “xmalloc” wrappers that abort (!!) the caller when malloc returns a null pointer, and the justification is almost always that, since the program could OOM-crash anyway if allocation fails, it’s no worse to abort. And of course this line of reasoning completely neglects systems that were intentionally configured to be robust under memory exhaustion.
Correct software, especially library code which may be used in applications requiring maximum reliability, should be written to assume the operating system does not overcommit, and to handle failure of malloc by backing out whatever operation was in place and reporting failure. This approach will ensure safety on systems configured to do commit accounting, and will give behavior no worse than the operating system’s default on systems that do overcommit.

sendmail STARTTLS=client version=TLSv1/SSLv3, verify=FAIL

2014-10-28T10:21:00.005-07:00

 STARTTLS=client version=TLSv1/SSLv3, verify=FAIL

Another way is to edit the current sendmail.cf file and make sure
 this line is in it and uncommented:
 
  O TLSSrvOptions=V
 
 This tells sendmail to NOT request the client's certificate.  Note that
 this option is not safe and is only present in sendmail V8.12.x.

Really a bad idea to edit the sendmail.cf directly. Each service restart
will delete such a manual direct setting in the .cf file! And editing
the sendmai.cf directly often causes errors. Even the OP did set that
with define(`confTLS_SRV_OPTIONS', `V') in the sendmail.mc file.

Reclaim disk space in Linux

2014-07-05T20:06:00.000-07:00

If you can't kill your application, you can truncate instead of deleting the log file to reclaim the space. If the file was not open in append mode (with O_APPEND), then the file will appear as big as before the next time the application writes to it (though with the leading part sparse and looking as if it contained NUL bytes), but the space will have been reclaimed.

To truncate it: #> /path/to/the/file.log

If it was already deleted, on Linux, you can still truncate it by doing:

#> "/proc/$pid/fd/$fd"

Where $pid is the process id of the process that has the file opened, and $fd one file descriptor it has it opened under (which you can check with lsof -p "$pid".

If you don't know the pid, and are looking for deleted files, you can do:

#lsof -nP | grep '(deleted)' Or (on Linux):

#find /proc/*/fd -ls | grep '(deleted)'

Or to find the large ones with zsh: ls -ld /proc/*/fd/*(-.LM+1) | grep '(deleted)' An alternative, if the application is dynamically linked is to attach a debugger to it and make it call close(fd) followed by a new open("the-file", ....).

1TB USB Seagate external harddrive does not detect in ubuntu

2014-02-23T05:25:00.000-08:00

Just plugged in a newly purchased USB external hard drive 1TB - Stunned with my Ubuntu OS simply did not detect!

dmesg's

[ 130.433880] scsi8 : uas
[ 151.820010] scsi 8:0:0:0: uas_eh_abort_handler tag -1
[ 151.820024] scsi 8:0:0:0: uas_eh_device_reset_handler tag -1
[ 151.820030] scsi 8:0:0:0: uas_eh_target_reset_handler tag -1
[ 151.820036] scsi 8:0:0:0: uas_eh_bus_reset_handler tag -1
[ 151.824027] usb 1-1.1: URB BAD STATUS -71
[ 151.892010] usb 1-1.1: reset high speed USB device number 6 using ehci_hcd
[ 151.984903] scsi 8:0:0:0: Device offlined - not ready after error recovery
[ 302.577810] usb 1-1.1: USB disconnect, device number 6
[ 307.634478] usb 1-1.1: new high speed USB device number 7 using ehci_hcd

Steps to resolve the issue:

Disconnect the drive

$rmmod uas

Create a new file and like :

$cat /etc/modprobe.d/blacklist-uas.conf
blacklist uas;

Connect the drive it simply works! Voila... ;)

CentOS 6 installation on software RAID1 and LVM

2014-02-07T20:20:00.001-08:00

Recover deleted partitions using Testdisk in Ubuntu #11

2014-02-07T19:09:00.001-08:00

Data Recovery on a Formatted Drive with TestDisk by Britec

2014-02-07T19:05:00.001-08:00

AWS EC2 Instance types

2013-12-12T19:40:00.002-08:00

http://aws.amazon.com/ec2/instance-types/

snmp setup on CentOS

2013-12-12T19:01:00.000-08:00

Install the SNMP service on CentOS.

yum install net-snmp

mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.org
vim /etc/snmp/snmpd.conf

rocommunity public 192.168.x.x

P.S. - Replace 192.168.x.x with the IP address of the server that you want to allow SNMP lookups from:

/etc/init.d/snmpd start
chkconfig snmpd on

In firewall open UDP port 161 to your SNMP lookup server.

And Test -
snmpwalk -v 2c -c public 192.168.x.x

Git-Server in 2min with SCM-Manager

2013-12-04T20:28:00.001-08:00

NFS Active Passive using DRBD Heartbeat Cent OS 5

2013-10-24T10:56:00.001-07:00

NFS Active Passive using DRBD Heartbeat Cent OS 5

yum repos used - epel , rpmforge and

wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo

VIP - 192.168.1.70 -- see in haresources file below (Heartbeat section)

[root@a-log1]cat /etc/hosts

192.168.1.71    a-log1 
192.168.1.72    a-log2

DRBD setup

[root@a-log1 ~]# rpm -qa | grep drbd
kmod-drbd83-8.3.15-3.el5.centos
drbd83-8.3.15-2.el5.centos

config file 

[root@a-log1 ~]# cat /etc/drbd.conf 
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd83/drbd.conf
#

include "/etc/drbd.d/global_common.conf";
include "/etc/drbd.d/*.res";

[root@a-log1 ~]# cat /etc/drbd.d/res0.res
resource r0 {
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout     30; }
disk { on-io-error detach; no-disk-barrier; no-disk-flushes; no-md-flushes; no-disk-drain; }
net { cram-hmac-alg "sha1"; shared-secret "nit3fl!rtZ"; sndbuf-size 512k; rcvbuf-size 512k; max-epoch-size 8000; max-buffers 8000; unplug-watermark 16;}
syncer { rate 180M; al-extents 3389; }

device    /dev/drbd1;
disk      /dev/sdb1;
meta-disk internal;
on a-log1 {
    address   192.168.1.71:7789;
}
on a-log2 {
    address   192.168.1.72:7789;
}
}

Let us prepare raw sdb on a-log1 and a-log2 servers i,e  

Now we have to create the metadata on both servers:

a-log1# drbdadm create-md r0

a-log2#drbdadm create-md r0

and we can now start DRBD on both servers at the same time:

[root@a-log1]# service drbd start

Starting DRBD resources: [ d(main) s(main) n(main) ].

[root@a-log2]# service drbd start

Starting DRBD resources: [ d(main) s(main) n(main) ].

You have two ways that I know to verify that DRBD is running properly:

# service drbd status

# cat /proc/drbd

As you can see they say that it is connected but ro is 
Secondary/Secondary meaning that we haven’t told the system which one is
 the Primary server (master) that contains the block to be replicated. Once we tell the system who is the master it will start the synchronization.

Primary server -- Study drbd webpage - you can hardcode in config

a-log1 # drbdsetup /dev/drbd1 primary –o

The
 synchronization started and it will take a little while to be 
completed. Please wait until it is done and move to the next step.

0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

 ns:41941724 nr:0 dw:0 dr:41942388 al:0 bm:2560 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Now that our servers are in sync we can create format our /dev/drbd0 with our preferred file system. In my case I will use ext4.

On primary a-log1:

a-log1# mkfs.ext4 /dev/drbd1

Configuring NFS exports for Heartbeat integration

Okay, now we have our DRBD up and 
running. Great!! Next step is to set up out NFS export and then install 
heartbeat so we can have an automatic fail over system.

Lets prepare our NFS first:

On both servers:

# mount /dev/drbd1 /data    # We will mount it 

# mkdir /data/main           # Create it to store data         

 [root@a-log1 ~]# cat /etc/exports 
/data/main 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,fsid=11280987)

 [root@a-log1 ~]# ls -l /data/
total 36
drwx------  2 root root 16384 Oct 23 13:09 lost+found
drwxr-xr-x 14 root root 12288 Oct 23 21:55 main
drwxr-xr-x  5 root root  4096 Oct 24 22:22 nfs

The above nfs came like this....

[root@a-log2 ~]# cat /etc/exports 
/data/main 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,fsid=11280987)

 NFS stores some information about your NFS mounts at /var/lib/nfs and 
since those information will have to be mirrored we will have to move 
them to the DRBD device:

 # mv /var/lib/nfs/ /data/

this might generate an error such as:

mv: cannot remove `/var/lib/nfs/rpc_pipefs/cache': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/nfsd4_cb': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/statd': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/portmap': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/nfs': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/mount': Operation not permitted

mv: cannot remove `/var/lib/nfs/rpc_pipefs/lockd': Operation not permitted

but do not worry about it because it will create the directories anyways.

# mv /var/lib/nfs /var/lib/nfsBackup

Then symlink /var/lib/nfs to our /data directory:

# ln -s /data/nfs/ /var/lib/nfs

# umount /data

On server2 only:  a-log2 

# mv /var/lib/nfs/ /var/lib/nfsBackup

# mkdir /data

# ln -s /data/nfs/ /var/lib/nfs

The symbolic link will be broken since the /dev/drbd1 is not mounted. This will work in case of NFS fail-over.

Heartbeat installation and configuration

[root@a-log1 ~]# rpm -qa | grep heart
heartbeat-3.0.3-2.3.el5
heartbeat-libs-3.0.3-2.3.el5

[root@a-log1 ~]# rpm -qa | grep pace
pacemaker-1.0.12-1.el5.centos

 [root@a-log1 ha.d]# cat ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
warntime 5
deadtime 10
initdead 30
bcast eth0
ping 192.168.1.1
node    a-log1
node    a-log2

[root@a-log1 ha.d]# cat authkeys  

 auth 1
1 sha1 n11f

[root@a-log1 ha.d]# cat haresources 
a-log1 IPaddr::192.168.1.70/24/eth0:0 drbddisk::r0 Filesystem::/dev/drbd1::/data::ext4 nfslock nfs

Passive server a-log2

[root@a-log2 ~]# cat /etc/ha.d/ha.cf 
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
warntime 5
deadtime 10
initdead 30
bcast eth0
ping 192.168.1.1
node    a-log1
node    a-log2

[root@a-log2 ha.d]# cat authkeys  

 auth 1

1 sha1 n11f

[root@a-log2 ~]# cat /etc/ha.d/haresources
a-log1 IPaddr::192.168.1.70/24/eth0:0 drbddisk::r0 Filesystem::/dev/drbd1::/data::ext4 nfslock nfs

Final Important - Using nfs version 4

[root@a-log2 ~]# ps aux | grep nfs
root      9975  0.0  0.0      0     0 ?        S<   23:05   0:00 [nfsd4]
root      9976  0.0  0.0      0     0 ?        S    23:05   0:00 [nfsd]
root      9977  0.0  0.0      0     0 ?        S    23:05   0:00 [nfsd]
root      9978  0.0  0.0      0     0 ?        S    23:05   0:00 [nfsd]
root      9979  0.0  0.0      0     0 ?        S    23:05   0:00 [nfsd]

Both servers - open # vim /etc/init.d/nfs

#killproc nfsd -2 - comment and change to
killproc nfsd -9

Test

 Primary a-log1 

[root@a-log1 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: [ OK ]

Check a-log2 server  -  VIP failover is successfull

[root@a-log2 ~]# ip addr
1: lo: mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:4a:db:2f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.72/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.70/24 brd 192.168.1.255 scope global secondary eth0:0
    inet6 fe80::a00:27ff:fe4a:db2f/64 scope link
       valid_lft forever preferred_lft forever
3: sit0: mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

In Less than 10 seconds - NFS services will be failover from a-log1 to a-log2 server

Added a NFS client a5-web server

[root@a5-web ~]# cat /etc/fstab

192.168.1.70:/data/main /mnt/storage nfs noatime,nodev 0 0

Perform continuous write's

[root@a5-web ~]# for i in `seq 1 10000`;do echo "hello how r u" >> /mnt/storage/ashok/nf.txt;sleep 1;done

Perform continuous read's

[root@a5-web ~]# for i in `seq 1 10000`;do cat /mnt/storage/ashok/nf.txt | wc -l;sleep 2 ; done
13175

In primary server - you can stop heartbeat and watch what happens in a5-web nfs client.

Later, start heartbeat and watch everything works as expected ;)

Another test - Download a file.

[root@a5-web ~]# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fedelivery.oracle.com" "http://download.oracle.com/otn-pub/java/jdk/6u38-b05/jdk-6u38-linux-i586.bin"

Try coping this file to /mnt/storage/

While it is copying - stop heartbeat in primary server ... it should continue to copy and finish!

Check md5sum of both files.

[root@a5-web ~]# md5sum /mnt/storage/ashok/jdk-6u38-linux-i586.bin
5bae3dc304d32a7e3c50240eab154e24 /mnt/storage/ashok/jdk-6u38-linux-i586.bin
[root@a5-web ~]# md5sum jdk-6u38-linux-i586.bin
5bae3dc304d32a7e3c50240eab154e24 jdk-6u38-linux-i586.bin
[root@a5-web ~]#

Ofcourse - please watch magic happening in logs in both servers ;)

[root@a-log1 ~]# tail -f /var/log/messages /var/log/ha-*

[root@a-log2 ~]# tail -f /var/log/messages /var/log/ha-*

Cool :)

reference -

http://www.linux-ha.org/wiki/Haresources

http://www.gossamer-threads.com/lists/linuxha/dev/59880

http://www.linuxnix.com/2010/01/heartbeat-clustering.html

ssh tunnel

2013-09-25T21:51:00.002-07:00

How do I view ntop stats without opening port 3000?

Setup simple tunnel using ssh, enter the following on your local UNIX / Linux desktop system:

$ ssh -L 3000:localhost:3000 -N -f user@server.yourcorp.com

Now open browser and type the following command:

http://localhost:3000/