In part 1, we setup a CentOS container which we could enter/exit but we didn’t do anything else on the network. What we want to do is set it up so we have the ability to bridge our hosts’ network and define static IP for network interface.


yum install bridge-utils

vzctl set 101 --netif_add eth0 --save
echo '
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=dhcp
' > /vz/private/101/etc/sysconfig/network-scripts/ifcfg-eth0

After this just start and you should be good to go.

For my purposes, I started with very basic installation of CentOS 5.5. I have done this on existing installations of CentOS without any issues as well. Ok so let’s start with getting the OpenVZ repository added to our system. This means that we will be going down “yum” path instead of “rpm” path. This is the quickest and easiest way to get OpenVZ installed.

cd /etc/yum.repos.d
wget http://download.openvz.org/openvz.repo
rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ

Before we move on, let’s turn off selinux:

vi /etc/selinux/config
:0,$s/=enforcing/=disabled/g
:wq

Go ahead and reboot the box and check:

[root@tooncent ~]# getenforce
Disabled

Now since we have the OpenVZ repository added and selinux disabled, we can go ahead and move on with installation part:
yum install vzkernel.x86_64 #install kernel with openvz support

echo '
# ----------added for OpenVZ installation----------------
# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
' >> /etc/sysctl.conf

Note: by executing above snippet, you will end up creating duplicate entries. It’s always a good idea to clean up duplicates to avoid confusion/mistakes later.

Following step is optional and mostly for clarity. Edit your grub.conf and change CentOS to say OpenVZ:

vi /etc/grub.conf

Change the first “title CentOS” to “title OpenVZ” (without the quotes). For example:

title CentOS (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

becomes:

title OpenVZ (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

Ok at this point we are ready to reboot and use the new kernel. Issue a “reboot”. Let’s install the utilities to manage OpenVZ:
yum install vzctl.x86_64 vzquota.x86_64

We need to start up vz manually once.

/etc/init.d/vz start

Alright! we got OpenVZ running. Now, let’s get a precreated template from OpenVZ site.

cd /vz/template/cache/
wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

Ok now comes the fun part; let’s get our first container up and running:
Note: ostemplate name is the file you downloaded minus .tar.gz. centos-5-x86_64.tar.gz

[root@tooncent cache]# vzctl create 101 --ostemplate centos-5-x86_64
Creating container private area (centos-5-x86_64)
Performing postcreate actions
Container private area was created
[root@tooncent cache]#

Container files are created under /vz/private/101 < – 101 is the id you give it. You can put any numeric number above 100. Documentation says not to use anything below 101 since its a reserved range.

Let’s set a hostname so there is no confusion:

vzctl set 101 --hostname toontest.tooncent.com --save
Set hostname: toontest.tooncent.com
Saved parameters for CT 101

To start your new container:

[root@tooncent ~]# vzctl start 101
Starting container ...
Container is mounted
Setting CPU units: 1000
Set hostname: toontest.tooncent.com
Container start in progress...
[root@tooncent ~]#

To access your container:

vzctl enter 101

To exit the container, type: exit

To stop the container:

# vzctl stop 101
Stopping container ...
Container was stopped
Container is unmounted

Congratulations! You have successfully setup a container.

As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be running the exact same version of the
build, it makes sense that I should only allow a user of my deployment
scripts to link a build which exists on all servers. But how do you
accomplish this in bash?

In most other languages, you have access to numerous array helping
functions that allow you to perform intersects, uniqs, and merges. My goal
was to do the same thing in bash without forking out to any external
binary. So how do you ensure that a particular thing exists on N number of
servers? Here it is:

function in_array() {
 local x
 ENTRY=$1
 shift 1
 ARRAY=( "$@" )
 [ -z "${ARRAY}" ] && return 1
 [ -z "${ENTRY}" ] && return 1
 for x in ${ARRAY[@]}; do
   [ "${x}" == "${ENTRY}" ] && return 0
 done
 return 1
}

MASTER=()
CURRENT=()
FIRST=1
for SERVER in ${SERVERS}; do
 # collect all builds from server and populate CURRENT list
 COMMAND="${LS} -1fd ${WEBROOT}/${SITE}.*"
 BUILDS=`${SSH} ${SSHOPTS} root@${SERVER} "${COMMAND}"`
 for BUILD in ${BUILDS}; do
   CURRENT=( ${CURRENT[@]-} ${BUILD} )
 done

 # if this is our first time around, copy CURRENT to MASTER
 if [ ${FIRST} -eq 1 ]; then
   MASTER=( ${CURRENT[@]} )
   FIRST=0
 fi

 # now we do a compare between MASTER and CURRENT to see what builds
 # are common
 INTERSECT=()
 for ENTRY in ${CURRENT[@]}; do
   in_array "${ENTRY}" "${MASTER[@]}"
   RET=$?
   if [ "${RET}" -eq 0 ]; then
     INTERSECT=( ${INTERSECT[@]-} ${ENTRY} )
   fi
 done
 MASTER=( ${INTERSECT[@]} )

 # clear the CURRENT array
 CURRENT=()
done

Let me take a moment to explain the code above:

• In order to check for array intersection, you need an in_array()
  function
  • The first argument as the “needle” and the second is the
    “haystack”
  • We verify that both parameters were passed
  • We simply loop through the haystack checking for the needle
  • If we find it, return success. Otherwise, eventually return
    false
• We need to loop through each server eventually, but we’ll start with
  the first one
  • Run an SSH command to get a listing of builds
  • Populate an array ($CURRENT) with the builds that were found
  • Since the first server has no previous server to compare with, so we
    just copy it to $MASTER
• We then loop to the 2nd server, and put the result of getting builds
  into $CURRENT
  • Now that we have the first server’s builds in $MASTER, we perform an
    intersect with $CURRENT
  • We realize the need for an $INTERSECT array to hold the intersections
    found above
  • $INTERSECT becomes $MASTER since it only contains similar builds from
    the 1st and 2nd server
• Looping to the 3rd server, we get the builds and put them in $CURRENT
  • Since $MASTER contains only the similar builds thus far, we again
    compare it with $CURRENT
  • The intersect can now be used to compare against builds on the 4th
    server, and so on
• Once you finish looping through all servers, your $MASTER should
  contain only similar builds

There are a few guides out there which show you how to do this via
forking, but I thought someone may appreciate the elegance of using 100%
bash to accomplish this. I hope this helps someone else out there!

libtool: link: cannot find the library `/usr/local/src/httpd-2.2.14/srclib/apr-util/xml/expat/lib/libexpat.la' or unhandled argument `/usr/local/src/httpd-2.2.14/srclib/apr-util/xml/expat/lib/libexpat.la'

I was able to fix this error by adding: --with-expat=builtin

Remember to do make clean or untar again before doing configure. Good luck!

for i in `find -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

I am sure there are many different ways to show file count for each folder in a directory and I am curious to see what people do so please do post comments with what you do.

Above command is pretty simple and can be expanded to do whatever you need.  For example, you can throw it into a bash script and be able to pass parameters.  For example:  count_files /home/  In this case your command line would look like:

for i in `find /home/ -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

only difference would be that /home/ would be argument you passed and therefore will be $1.  Here is a sample script for above example:

#!/bin/bash
for i in `find $1 -maxdepth 1 -type d`; do
echo -n $i " ";
find $i|wc -l;
done

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

./configure –prefix=/usr/local/curl –disable-ipv6

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

One of the things you can do to prevent getting “infected” is to not click on any links on facebook. Even if it comes from friends you trust.  It would also be a good idea to arm your self with the best virus protection software.  If you do click on a link, make sure you don’t enter any login/passwords  on any sites you end up on.

Here is an article on eHow which talks about avoiding the virus:  how to avoid mygener.im on facebook

EDIT:  some people have reported that destination url has changed at least once so you may end up on a site which looks like aother “legit” site.

http://tinyurl.com/cosuzr

Watch where you hang out and if you are feeling any symptoms, go see a doctor.

Here is some info from  wikipedia:

Mexican officials state that since March 2009 there have been over 1600 reported cases and put the death toll at 149, with 20 confirmed to be linked to a new swine influenza strain of influenza A virus subtype H1N1.

In the US, twenty-eight cases have been confirmed among New York students, seven in California, two in Texas, two in Kansas and one in Ohio; all have recovered. Kansas state health officials confirmed two cases of swine flu in Kansas on April 25, 2009, just minutes after eight school children in New York City were believed to be infected after a school trip to Mexico. The New York case had been confirmed as influenza A virus, which met the Centers for Disease Control and Prevention (or CDC) definition of a probable case of swine flu. Tests have now confirmed that this flu virus is the new strain of H1N1, according to reports on UK BBC News. On April 25, 2009, Texas closed a high school near San Antonio indefinitely after a third student showed symptoms of swine flu.

In Canada, the virus has now been confirmed in six cases. There are also three confirmed cases outside North America. The first confirmed case in Europe has occurred in a man in Spain, the second and third in Scotland in the UK. In Scotland there are two confirmed cases being treated in a hospital in Airdrie, Lanarkshire. The results from their tests came in at 6.10pm on 27 April 2009 and were confirmed as positive. These are the first two confirmed cases in the UK and the second and third in Europe. On April 26, 2009, students from Rangitoto College and Northcote College, both on the North Shore of Auckland, New Zealand, returned from school trips to Mexico and began exhibiting signs of influenza. Like the US students, they tested positive for influenza A which means swine flu is highly likely, however definite results are pending. There have also been reports of suspected infection in Australia, Brazil, Chile, France and Israel. Airports worldwide are on alert, with passengers from Mexico entering Japan being screened for the virus.

The head of the World Health Organization will lead the agency’s efforts against a deadly swine flu outbreak. The World Health Organization warns this new swine flu has the potential to become an international influenza pandemic. On April 25, 2009, the World Health Organization issued a document called the Swine influenza frequently asked questions. On April 24, 2009, the World Health Organization agreed that the current situation constitutes a public health emergency of international concern. The EU are advising against non essential travel to Mexico and the USA. Experts warn that use of face masks does not provide adequate protection. Face masks only work when dry; after an hour or two the breath causes the fabric to become damp and then it does not provide filtration of infection. A face mask must be replaced every two hours and the one used disposed of or cleaned. The WHO have increased the level pandemic alert to phases 4 on a 6 point scale. Phase 4 indicates a “significant increase in risk of pandemic”.

So without further delay, following command will telnet to local memcached server on port 11211 and issue one of the memcached commands, stats

(sleep .5;echo stats) | telnet localhost 11211
You may have to play with the sleep timer to get it to work for your environment but in our .5 was the sweet spot. Good luck and let me know if you have another shell command. Obviously we can do this from perl, php, python, etc but the beauty of this is that you do not need any other dependencies plus its a very short command.

Since “jsled” commented about nc (thanks jsled), here is the syntax to do the same thing with nc:

echo "stats" | nc localhost 11211

I needed to redirect a url:  http://www.example.com/testpage.php?foo=bar to http://www.example.com/

So I added this to .htaccess file:

RewriteRule ^testpage\.php / [R=301,L]

This did do a 301 redirect as I wanted but query string passed so my redirected url looked like:  http://www.example.com/?foo=bar

Obviously this is not what I wanted so in order for me to fix it, I had to take a rather lame approach.  My new redirect is:

RewriteRule ^testpage\.php /? [R=301,L]

The only difference is the “?” after / for target url.  I am just curious to know why query-string was passed.  I have not used rewrites for long time but I do not remember this behavior.

Assumptions:  you will be connecting as user “root” to svn server located at “10.0.0.1”.  All your files will be saved at c:\ including your svn command line utility

First we will have to generate a key.  We can accomplish this by using a free utility called puttygen.  Run puttygen and click on “Generate” button.  You will have a key similar to below example:

Example of a key generated by puttygen

Copy this, you will need it in few mins.  At this point, go ahead and create a private key by clicking on:  “Save private key”.  Save this as private.ppk on your C:\. 

Now let us log in to the svn server and add this public key to the authorized_keys2 (see setting up keys for step by step instructions).  I will assume you are using “root” as login.

vi /root/.ssh/authorized_keys2

Make sure when you paste, it is not broken into different lines.  All of the key should be one line.

Ok now back to your Windows machine.  Now we need to set up ssh tunnel to our server.  There are few ways of doing this but for our purpose, we will use another free program provided by the same developers as puttygen: download plink to C:\.  If you do not do this step, you will get following error:

svn: Can't create tunnel: The system cannot find the file specified.

Ok let us set the variables for svn.  Go to command prompt and type (you can also set this in your scripts and inside windows environment.  But since this post is to show you an example, we will just do this):

set SVN_SSH="/plink.exe" -i /private.ppk -l root

Now let’s run this one time manually to cache key:

/plink.exe -i /private.ppk -l root 10.0.0.1

Press “y” when it asks you to save.  Type exit and get back to your prompt.

Ok now we can test our svn utility.

/svn info svn+ssh://10.0.0.1/svn/testrepo/trunk/

This should display output similar to:

Path: trunk
URL: svn+ssh://10.0.0.1/svn/testrepo/trunk
Repository Root: svn+ssh://10.0.0.1/svn/testrepo
Repository UUID: b9143312-b1a1-11ba-a111-11cdcd1d2222
Revision: 10
Node Kind: directory
Last Changed Author: root
Last Changed Rev: 4
Last Changed Date: 2008-11-18 15:18:47 -0800 (Tue, 18 Nov 2008)

Now you are ready to script your checkouts, do checkin’s with out having to type in your password, etc.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Note:  you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz
tar zxpfv fuse-*.gz
cd fuse*
./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found
configure: error:
*** Please specify the location of the kernel source with
*** the '--with-kernel=SRCDIR' option
configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/
total 4.0K
drwxr-xr-x 18 root root 4.0K Oct  7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64
make && make install
cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz
tar zxpfv sshfs*
cd sshfs-fuse-*
./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

OR

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0’ found
No package ‘gthread-2.0’ found

You need to install glib2.  Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure
make && make install

After installation is done, we can move on with testing the installation:

cd /mnt
mkdir test
sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse
FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt
sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
...
sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test
...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

It is wonderful when famous people who can help make a difference do things to help a cause.  One of those people is Lance Armstrong, winner of Tour de France a record-breaking seven consecutive years.  He is also a cancer survivor.   Today I found out that Lance Armstrong announced that he is coming back to professional cycling in order to raise cancer awareness. Below is the full press release from livestrong.com website.  You can also visit the site to watch the video:

“I am happy to announce that after talking with my children, my family and my closest friends, I have decided to return to professional cycling in order to raise awareness of the global cancer burden. This year alone, nearly eight million people will die of cancer worldwide. Millions more will suffer in isolation, victims not only of the disease but of social stigma. After the passage of Proposition 15 in Texas, a $3 billion investment in the fight against cancer which is helping to make this disease part of the national dialogue in America, it’s now time to address cancer on a global level.”

wget http://www.whatismyip.org

Well that does the trick.  But being lazy as I am, I did not want to have to cat the output file to find out the ip (plus there is no point of creating extra files and doing extra work to remove them).  So if you are ssh’ed in, you can issue following command (I am sure there is another way of doing it, but this is the quickest way I could think of):

wget -q -O - http://www.whatismyip.org

-O tells wget to redirect output to the following file (- being the standard out ).  So it basically echo’s output to our console.

-q makes wget run in  quiet mode so you do not see all of the connection/download/etc output.

That is it!  I am curious to know what other ways people use to get the same information.  Please share your way if possible.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mkdir temp
cd temp
touch Foo FooO "Foo Bar" "FOO BAaR"
\ls | while read -r FILENAME
do
mv -v "$FILENAME" `echo $FILENAME| tr ' ' '_'| tr '[A-Z]' '[a-z]'`
done

Note:  I intentionally have slash in front of ls (\ls).  \ls means that we want to make sure there is no ls alias overwriting our command. This is needed if your system has alias setup to display ls in a different way instead of default listing.  mv -v shows us the filenames being renamed as your script goes through the whole dir.  Your output should be like:

`Foo' -> `foo'
`FOO BAaR' -> `foo_baar'
`Foo Bar' -> `foo_bar'
`FooO' -> `fooo'

One of the very powerful commands in this post is the “tr” command. This command is not as popular as sed or awk but it is very useful and simple to use (read more about tr).

If you only needed to convert spaces to underscores and you are using CentOS/Fedora/Redhat, you can use this simpler method. NOTE: this command is not available on all distributions: rename " " "_" *

Learn more about rename command
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Install perl modules:

install Term::ReadKey
install File::Tail
install Time::HiRes

Now copy paste the script below and save it in a location and set +x attribute on it so you can execute it.  On my setup, I have the script under /usr/bin/httptop:

#!/usr/bin/perl -w
use Time::HiRes qw( time );
use File::Tail (  );
use Term::ReadKey;
use Getopt::Std;
use strict;
### Defaults you might be interested in adjusting.
my $Update = 2; # update every n secs
my $Backtrack = 250; # backtrack n lines on startup
my @Paths = qw(
%
/title/%/logs/access_log
/var/log/httpd/%/access_log
/usr/local/apache/logs/%/access_log
);
my $Log_Format = "combined";
my %Log_Fields = (
combined => [qw/ Host x x Time URI Response x Referer Client /],
vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
);
### Constants & other thingies. Nothing to see here. Move along.
my $Version = "0.4.1";
sub by_hits_per (  ) { $b->{Rate} <=> $a->{Rate} }
sub by_total (  ) { $b->{Total} <=> $a->{Total} }
sub by_age (  ) { $a->{Last} <=> $b->{Last} }
my $last_field = "Client";
my $index = "Host";
my $show_help = 0;
my $order = \&by_hits_per;
my $Help = "htlwufd?q";
my %Keys = (
h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
w => [ "Show remote host" => sub { $index = "Host" } ],
u => [ "Show requested URI" => sub { $index = "URI" } ],
f => [ "Show referring URL" => sub { $index = "Referer" } ],
d => [ "Show referring domain" => sub { $index = "Domain" } ],
'?' => [ "Help (this thing here)" => sub { $show_help++ } ],
q => [ "Quit" => sub { exit } ]
);
my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
my @Record_Fields = qw/ Host URI Referer Domain /;
my $Max_Index_Width = 50;
my $Initial_TTL = 50;
my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
my %Term = (
HOME => "\033[H",
CLS => "\033[2J",
START_TITLE => "\033]0;", # for xterms etc.
END_TITLE => "\007",
START_RV => "\033[7m",
END_RV => "\033[m"
);
my ( %hist, %opt, $spec );
$SIG{INT} = sub { exit };
END { ReadMode 0 };
### Subs.
sub refresh_output
{
my ( $cols, $rows ) = GetTerminalSize;
my $show = $rows - 3;
my $count = $show;
my $now = (shift || time);
for my $type ( values %hist ) {
for my $peer ( values %$type ) {
# if ( --$peer->{_Ttl} > 0 ) {
my $delta = $now - $peer->{Start};
if ( $delta >= 1 ) {
$peer->{ Rate } = $peer->{ Total } / $delta;
} else {
$peer->{ Rate } = 0
}
$peer->{ Last } = int( $now - $peer->{ Date } );
# } else {
# delete $type->{$peer}
# }
}
}
$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values %{$hist{$index}};
my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
my $first = 0;
$first = ( $first <= $_ ? $_ + 1 : $first ) for map { $_ ? length($_->{$index}) : 0 } @list;
$first = $Max_Index_Width if $Max_Index_Width < $first;
print $Term{START_TITLE}, "Monitoring $spec at: ", scalar localtime, $Term{END_TITLE} if $ENV{TERM} eq "xterm"; # UGLY!!!
my $help = "Help/?";
my $head = sprintf( "%-${first}s %6s %4s %4s %s (%d total)",
$index, qw{ Hits/s Tot Last }, $last_field,
scalar keys %{$hist{$index}}
);
#
# Truncate status line if need be
#
$head = substr($head, 0, ($cols - length($help)));
print @Term{"HOME", "START_RV"}, $head, " " x ($cols - length($head) - length($help)), $help, $Term{END_RV}, "\n";
for ( @list ) {
# $_->{_Ttl}++;
my $line = sprintf( "%-${first}s %6.3f %4d %3d %s",
substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last }, $last_field)} );
if ( length($line) > $cols ) {
substr( $line, $cols - 1 ) = "";
} else {
$line .= " " x ($cols - length($line));
}
print $line, "\n";
}
print " " x $cols, "\n" while $count++ < $show;
}
sub process_line
{
my $line = shift;
my $now = ( shift || time );
my %hit;
chomp $line;
@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /"([^"]+)"|\[([^]]+)\]|\s/o, $line ) );
$hit{ URI } =~ s/HTTP\/1\S+//gos;
$hit{ Referer } = "<unknown>" if not $hit{Referer} or $hit{Referer} eq "-";
( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
$hit{ Client } ||= "<none>";
$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
$hit{ Client } =~ s/[^\x20-\x7f]//gos;
# if $now is negative, try to guess how old the hit is based on the time stamp.
if ( $now < 0 ) {
my @hit_t = ( split( m![:/\s]!o, $hit{ Time } ))[ 0 .. 5 ];
my @now_t = ( localtime )[ 3, 4, 5, 2, 1, 0 ];
my @mag = ( 3600, 60, 1 );
# If the hit didn't parse right, or didn't happen today, the hell with it.
return unless $hit_t[2] == ( $now_t[2] + 1900 )
and $hit_t[1] eq $Months[ $now_t[1] ]
and $hit_t[0] == $now_t[0];
splice( @hit_t, 0, 3 );
splice( @now_t, 0, 3 );
# Work backward to the UNIX time of the hit.
$now = time;
$now -= (shift( @now_t ) - shift( @hit_t )) * $_ for ( 3600, 60, 1 );
}
$hit{ Date } = $now;
for my $field ( @Record_Fields ) {
my $peer = ( $hist{$field}{$hit{$field}} ||= { Start => $now, _Ttl => $Initial_TTL } );
@$peer{ @Display_Fields } = @hit{ @Display_Fields };
$peer->{ Total }++;
}
}
sub display_help {
my $msg = "httptop v.$Version";
print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, "\n\n";
print " " x 4, $_, " " x 8, $Keys{$_}[0], "\n" for ( split "", $Help );
print "\nPress any key to continue.\n";
}
### Init.
getopt( 'frb' => \%opt );
$Backtrack = $opt{b} if $opt{b};
$Update = $opt{r} if $opt{r};
$Log_Format = $opt{f} if $opt{f};
$spec = $ARGV[0];
die <<End unless $spec and $Log_Fields{$Log_Format};
Usage: $0 [-f <format>] [-r <refresh_secs>] [-b <backtrack_lines>] <logdir | path_to_log>
Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
End
for ( @Paths ) {
last if -r $spec;
( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
}
die "No access_log $ARGV[0] found.\n" unless -r $spec;
my $file = File::Tail->new(
name => $spec,
interval => $Update / 2,
maxinterval => $Update,
tail => $Backtrack,
nowait => 1
) or die "$spec: $!";
my $last_update = time;
my ( $line, $now );
# Backtracking.
while ( $Backtrack-- > 0 ) {
last unless $line = $file->read;
process_line( $line, -1 );
}
$file->nowait( 0 );
ReadMode 4; # Echo off.
print @Term{"HOME", "CLS"}; # Home & clear.
refresh_output;
### Main loop.
while (defined( $line = $file->read )) {
$now = time;
process_line( $line, $now );
while ( $line = lc ReadKey(-1) ) {
$show_help = 0 if $show_help;
$Keys{$line}[1]->(  ) if $Keys{$line};
}
if ( $show_help == 1 ) {
display_help;
$show_help++; # Don't display help again.
} elsif ( $now - $last_update > $Update and not $show_help ) {
$last_update = $now;
refresh_output( $now );
}
}

Save/exit and make sure you make it executable by setting it to +x (chmod +x httptop)

Now you can run httptop by typing:  httptop -f combined -r 1 /usr/local/apache2/logs/access_log

NOTE:  Your access_log file might be in different location.  Point to the right location.  This sets the refresh rate to 1 sec (-r 1).  Now you can run httptop any time you want to checkout how your http traffic is doing.  Remember to press “?” to get help once you are in.

—————

DISCLAIMER: As always, if you find any inaccurate information, please comment and let me know. When you do comment, make sure you give me some references to confirm.

Here are the instructions on how to get sphinxse compiled with your mysql installation on CentOS x64.  I am sure same instructions will work for other flavors but I have not tested it.  I will be compiling the most current version of sphinx (0.9.8) with most current stable version of mysql (5.0.51b) at the time of the writing.  Let’s get the appropriate packages first:

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8.tar.gz
wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/
tar zxpf sphinx*
tar zxpf mysql*

You will also need “bison”, “patch”, “automake” and “libtool” installed.  Let us just do a yum install for it.

yum -y install bison patch automake libtool

NOTE:  if you don’t install bison, you will get the following error:
sed '/^#/ s|y\.tab\.c|sql_yacc.cc|' y.tab.c >sql_yacc.cct && mv sql_yacc.cct sql_yacc.cc
sed: can't read y.tab.c: No such file or directory
make[2]: *** [sql_yacc.cc] Error 2

Let us continue with patching mysql source with sphinx storage engine (sphinxse) code and compile/install our new binaries.

cd mysql*
patch -p1 < ../sphinx-0.9.8/mysqlse/sphinx.5.0.37.diff #Make sure everything succeeded.
BUILD/autorun.sh
mkdir sql/sphinx
cp ../sphinx-0.9.8/mysqlse/* sql/sphinx
./configure --prefix=/usr/local/mysql --with-sphinx-storage-engine
make
make install

Now start your mysql installtion and check if engine support is compiled in:

mysql> show engines\G
Engine: SPHINX
Support: YES
Comment: Sphinx storage engine 0.9.8

To read more about how to use Sphinx storage engine, please refer to:  Sphinx documentation for using sphinx storage engine

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Following command will search for the string you specified. Generally this will give you all of the packages which has specified string in title or description. Most of the time you will have to look through a lot of output to find what you are looking for.

yum search string

Probably one of the most important options for yum is provides/whatprovides. If you know what command you need, you can find out what package you have to install in order to have that command available to you.

yum provides (or whatprovides) command

Following command is same as above but with less output.

yum -d 1 provides command

So for example if you are trying to figure out what you need to install to use bunzip2, type:

yum -d 1 provides bunzip2

you will get a similar output as below.

# yum -d 1 provides bunzip2
bzip2.x86_64 1.0.2-13.EL4.3 base
bzip2.x86_64 1.0.2-13.EL4.3 base
man-pages-fr.noarch 0.9.7-13.el4 base
man-pages-ja.noarch 20050215-2.EL4.3 base
man-pages-pl.noarch 0.23-5 base

As you can see bunzip2 is part of bzip2 package. So now you can you just install bzip2.x86_64 to get bunzip2.

To learn more about what else is available, read man yum.

So if you just want to replace these with appropriate symbols, you may do it with a simple sql query. Note that below queries are without where clause. You may what to test it with one of your rows before making changes to the whole table. Of course, you should always backup your data before you try this out. If you have a dev system, that is even better. I put all my sql queries into a file ex: fix.sql and sourced it with mysql client.

vi fix.sql

update table_name set fieldname = replace(fieldname, '’', '\'');
update table_name set fieldname = replace(fieldname, '…','...');
update table_name set fieldname = replace(fieldname, '–','-');
update table_name set fieldname = replace(fieldname, '“','"');
update table_name set fieldname = replace(fieldname, '”','"');
update table_name set fieldname = replace(fieldname, '‘','\'');
update table_name set fieldname = replace(fieldname, '•','-');
update table_name set fieldname = replace(fieldname, '‡','c');

Save/exit.

# mysql
mysql> source fix.sql;

I am not sure if I am missing any other chars. If you know of any other chars, please comment with them and I will add on to the script here.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

svn co svn+ssh://192.168.1.10/svn/myrepos/ /home/mycheckout/

Now when they do “svn update” inside the their /home/mycheckout/ directory, they get an error:

We needed to point the checkout to the new ip. Easiest way to do this is to delete your checkout and re-checkout. Unfortunately, some of the developers had a lot of modified files which wasn’t checked in yet. I fixed it by issuing:

find /home/mycheckout -name "entries"|xargs /usr/bin/perl -w -i -p -e "s/192.168.1.10/10.1.1.10/g"

Find command helps us in finding all the files with name “entries” and xargs takes the filename and passes it to perl. To understand what perl command is doing, see this post.

Another method which may be preferred as mentioned in comments is: svn switch Only downside I see with this is that you have to remember what you used originally. If you did checkout as user@192.168.1.10, you would have to pass that to the command below.

Syntax is:

svn switch --relocate svn+ssh://192.168.1.10 svn+ssh://10.1.1.10

I would suggest at this time you switch to using hostname instead of ip.

# /usr/local/apache2/bin/apachectl -t -D DUMP_MODULES
Loaded Modules:
...
deflate_module (static)
...
Syntax OK

If you don’t have have deflate_module, you would have to recompile your apache with “–enable-deflate” option.

Going forward, I am going to assume you have deflate_module. Add the following to your apache conf file:

<Location />
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4\.0[678] no-gzip\
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary
</Location>

The main thing you need to configure is the line which says “no-gzip dont-vary” also in bold above. This tells apache to not compress certain type of files. I have noticed on some of my sites that swf (flash) files do not work as expected if they are compressed. So if you have swf files in your site, you may want to add |swf right after png.

This is all what it takes for you to enable gzip compression in Apache2. Once you restart your apache so it reads the conf file, you can test if your site is getting compressed or not by using this tool: http://www.gidnetwork.com/tools/gzip-test.php

Here are the results for my blog:

Results for: http://crazytoon.com
Web page compressed? Yes
Compression type? gzip
Size, Markup (bytes) 57,337
Size, Compressed (bytes) 11,666
Compression % 79.7

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Following are the instructions on how to install innotop on CentOS x64/Fedora/RHEL (Redhat enterprise). Most probably same instructions can be used on all flavors of Linux. If not, leave me a comment and I will research a solution for you. Let us start with downloading innotop. I used version 1.6.0 which is the latest at the time of writing.

wget http://internap.dl.sourceforge.net/sourceforge/innotop/innotop-1.6.0.tar.gz

Now let us go ahead and unzip and create the MakeFile to get it ready for install

tar zxpf innotop-1.6.0.tar.gz
cd innotop-1.6.0
perl Makefile.PL

At this point if you get the following output, you are good to continue:

Checking if your kit is complete...
Looks good
Writing Makefile for innotop

If you get something similar to following, you will need to take care of the prerequisites:

Looks good
Warning: prerequisite DBD::mysql 1 not found.
Warning: prerequisite DBI 1.13 not found.
Warning: prerequisite Term::ReadKey 2.1 not found.
Writing Makefile for innotop

Just because they are warnings does not mean you ignore them. So let us install those prerequisites. We will use perl’s cpan shell to get this installed (visit my post on how to install perl modules for more details). If it is your first time starting this up, you will have to answer some questions. Defaults will work fine in all cases.

perl -MCPAN -eshell
install Term::ReadKey
install DBI
install DBD::mysql

Note: you must install DBI before you can install DBD::mysql.

If you get an error like following when you are installing DBD::mysql:

Error: Can't load '/root/.cpan/build/DBD-mysql-4.007/blib/arch/auto/DBD/mysql/mysql.so' for module DBD::mysql: libmysqlclient.so.15: cannot open shared object file: No such file or directory at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.

You will have to create a symlink to the object file in your lib64 (or lib if you are not using x64 version) folder:

ln -s /usr/local/mysql/lib/mysql/libmysqlclient.so.15 /lib64/

Once all prerequisites are done, type perl Makefile.PL and you should have no warnings. Continue the install:

make install

At this point you should have innotop installed on your system. Let us do some quick set up so you can start using innotop. We start with configuring your .my.cnf to include connection directives.

vi ~/.my.cnf

Add the following (edit to reflect your install) and save/exit

[mysql]
port = 3306
socket = /tmp/mysql.sock

Start up innotop by typing innotop at your shell prompt. First prompt will ask you to “Enter a name:”. I just put localhost since this will be used to connect locally. Next prompt asks you about DSN entry. I use: DBI:mysql:;mysql_read_default_group=mysql

This tells innotop to read .my.cnf file and use group [mysql] directives. Next prompt is optional (I just press enter). Next two prompts you enter information if you need to.

At this point your innotop installation / testing is complete. You can read man innotop to get more details on how to use innotop.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

So today we decided that we are going to migrate one of our master database servers to new hardware.  Since we got the hardware this morning and we wanted to move on to it asap, we decided that we will take our slave down, copy data from it, and bring it up on future master server.  At that point, we will let it run as slave to the current master server until its time for us to take it down.  Reason we did that instead of mysqldump/import was to avoid the lag mysqldump creates on our server.

After we did all this and put up the new master server, we started to notice odd issues.  After looking around and comparing old db with new, we found out that new db was missing data.  How it happened is beyond me and is the reason why I am writing this.  We never had issues with the slave which would cause data to be lost; so what happened to those missing rows?  Is this something which is common?  Can we not trust our slave enough to use it as master if master died?  Can we not run backups off the slave with confident that our data is protected and up to date so to keep load down on our master?  All these questions which keep me awake and wondering…

If any of the phpbb guys read this, Thank you and keep up the good work! 🙂

Check out the forums: http://crazytoon.com/forum/ and feel free to post  questions, comments, suggestions, rants, etc.