Linux A-Z

2012-03-16T15:10:00.001+06:00

SNMP and MRTG on Sarge quick start

There appear to be no uncomplicated introductions to the subject of traffic monitoring on the internet anywhere. So here is one. The objective is to get traffic graphs for the primary interface on your server, workstation or firewall quickly and efficiently.

The system is fairly simple and consists of these parts:

The SNMP server. This allows access via the SNMP protocol to the system's network interface statistics and other data.
The MRTG (Multi router traffic grapher). This is a large Perl script which polls the SNMP server and accumulates information about network usage. This runs periodically from cron and generates graphs at defined intervals

Please note that you can collect and graph anything with these - they are not limited to network statistics. Not only that, you can manage many aspects of your server with SNMP.

Please make sure you have apache or apache2 installed for this to work.

SNMP server configuration

Firstly, you need an SNMP server to provide network interface statstics on demand:

# apt-get install snmpd

You need to edit the configuration for this as it does not allow any connections by default. With your favourite editor, edit:

/etc/snmp/snmpd.conf

Comment out the following (prefix with #):

com2sec paranoid default public

Insert the following underneath the commented out section:

com2sec readonly default public

That gives anyone with access to the SNMP server read-only access to the public community. This is the one that contains the interface statistics.

To apply the changes, restart snmpd:

/etc/init.d/snmpd restart

Make sure you firewall off any SNMP related ports so that you don't get any unwanted visitors (check netstat and /etc/services for port information).

Installation of MRTG

MRTG is the main collection and graphing component of the traffic monitoring solution I am presenting here. Firstly, install MRTG:

# apt-get install mrtg

You can manually or automatically generate the configuration file for mrtg. I would recommend doing it automatically as it is a lot easier. Issue the following command:

# cfgmaker --global 'WorkDir: /var/www/mrtg' \
    --output /etc/mrtg.cfg public@127.0.0.1

This will generate the configuration file. You then need to make an index file which contains a list of all of your interfaces. Issue the following command:

# indexmaker /etc/mrtg.cfg --columns=1 \
    --output /var/www/mrtg/index.html

You will now need to execute mrtg manually 3 times to create the required database files. Issue the following command 3 times sequentially. On the third run, you should see no errors being reported:

# mrtg

This is executed every 5 minutes by cron. The cron job was added by dpkg for you so you do not have to configure it.

Conclusion

Finally, inspect your results! You will not see any reasonable graphs for quite some time so sit back end relax for a bit!

Browse to http://your-server-name/mrtg/

Setting Up Unison File Synchronization Between Two Servers On Debian Squeeze

2012-01-01T16:52:00.003+06:00

This tutorial shows how to set up file synchronization between two Debian Squeeze servers with Unison. Unison is a file-synchronization tool similar to rsync, but the big difference is that it tracks/synchronizes changes in both directions, i.e., files changed on server1 will be replicated to server2 and vice versa.

I do not issue any guarantee that this will work for you!

1 Preliminary Note

In this tutorial I use the following two Debian Squeeze servers:

server1.example.com with the IP address 192.168.0.100
server2.example.com with the IP address 192.168.0.101

I want to synchronize the directory /var/www between the two servers. I will run Unison as the root user in this tutorial so that Unison has sufficient permissions to synchronize user and group permissions.

2 Installing Unison

server1/server2:

Unison has to be installed on server1 and server2; since we connect from server1 to server2 using SSH, we also need the SSH packages. This can be achieved as follows:

apt-get install unison openssh-server ssh

3 Creating A Private/Public Key Pair On server1

server1:

Now we create a private/public key pair on server1.example.com:

ssh-keygen -t dsa

root@server1:~# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): <-- ENTER
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): <-- ENTER
Enter same passphrase again: <-- ENTER
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
1b:95:bc:4a:f4:9f:d8:ea:24:31:0f:c9:72:d5:a7:80 root@server1.example.com
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|         o o     |
|        E * . .  |
|       o = o o   |
|      . S o .    |
|       + O + .   |
|        + + +    |
|         o .     |
|         .o      |
+-----------------+
root@server1:~#

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit ENTER!

Next, we copy our public key to server2.example.com:

ssh-copy-id -i $HOME/.ssh/id_dsa.pub root@192.168.0.101

root@server1:~# ssh-copy-id -i $HOME/.ssh/id_dsa.pub root@192.168.0.101
The authenticity of host '192.168.0.101 (192.168.0.101)' can't be established.
RSA key fingerprint is 25:d8:7a:ee:c2:4b:1d:92:a7:3d:16:26:95:56:62:4e.
Are you sure you want to continue connecting (yes/no)? <-- yes (you will see this only if this is the first time you connect to server2)
Warning: Permanently added '192.168.0.101' (RSA) to the list of known hosts.
root@192.168.0.101's password: <-- server2 root password
Now try logging into the machine, with "ssh 'root@192.168.0.101'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

root@server1:~#

Now check on server2 if server1's public key has correctly been transferred:

server2:

cat $HOME/.ssh/authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAACBAPhiAexgEBexnw0rFG8lXwAuIsca/V+lhmv5lhF3BqUfAbL7e2sWlQlGhxZ8I2UnzZK8Ypffq6Ks+lp46yOs7MMXLqb7JBP9gkgqxyEWqOoUSt5hTE9ghupcCvE7rRMhefY5shLUnRkVH6hnCWe6yXSnH+Z8lHbcfp864GHkLDK1AAAAFQDddQckbfRG4C6LOQXTzRBpIiXzoQAAAIEAleevPHwi+a3fTDM2+Vm6EVqR5DkSLwDM7KVVNtFSkAY4GVCfhLFREsfuMkcBD9Bv2DrKF2Ay3OOh39269Z1rgYVk+/MFC6sYgB6apirMlHj3l4RR1g09LaM1OpRz7pc/GqIGsDt74D1ES2j0zrq5kslnX8wEWSHapPR0tziin6UAAACBAJHxgr+GKxAdWpxV5MkF+FTaKcxA2tWHJegjGFrYGU8BpzZ4VDFMiObuzBjZ+LrUs57BiwTGB/MQl9FKQEyEV4J+AgZCBxvg6n57YlVn6OEA0ukeJa29aFOcc0inEFfNhw2jAXt5LRyvuHD/C2gG78lwb6CxV02Z3sbTBdc43J6y root@server1.example.com

4 Running Unison

server1:

We can now run Unison for the first time to synchronize the /var/www directory on both servers. On server1 run:

unison /var/www ssh://192.168.0.101//var/www

Output will be similar to this one - you might have to answer a few questions as this is the first time Unison is being run:

root@server1:~# unison /var/www ssh://192.168.0.101//var/www
Contacting server...
Connected [//server1.example.com//var/www -> //server2.example.com//var/www]
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
        /var/www
        //server2.example.com//var/www
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.

Update detection may take a while on this run if the replicas are
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison

  Waiting for changes from server| webalizer
Reconciling changes

local          server2.e...
dir      ---->            apps  [f]
file     ---->            index.html  [f]
link     ---->            ispconfig  [f]
dir      ---->            php-fcgi-scripts/apps  [f]
dir      ---->            webalizer  [f]
link     ---->            webmail  [f]

Proceed with propagating updates? [] <-- y
Propagating updates

UNISON 2.32.52 started propagating changes at 14:25:02 on 26 Jul 2011
[BGN] Copying apps from /var/www to //server2.example.com//var/www
[BGN] Copying index.html from /var/www to //server2.example.com//var/www
[BGN] Copying ispconfig from /var/www to //server2.example.com//var/www
[BGN] Copying php-fcgi-scripts/apps from /var/www to //server2.example.com//var/www
[BGN] Copying webalizer from /var/www to //server2.example.com//var/www
[BGN] Copying webmail from /var/www to //server2.example.com//var/www
[END] Copying ispconfig
[END] Copying webmail
[END] Copying apps
[END] Copying webalizer
[END] Copying index.html
[END] Copying php-fcgi-scripts/apps
UNISON 2.32.52 finished propagating changes at 14:25:03 on 26 Jul 2011

Saving synchronizer state
Synchronization complete at 14:25:03  (6 items transferred, 0 skipped, 0 failed)
root@server1:~#

Check the /var/www directory on server1 and server2 now, and you should find that they are in sync now.

Of course, we don't want to run Unison interactively, therefore we can create a preferences file (/root/.unison/default.prf) that contains all settings that we otherwise would have to specify on the command line:

vi /root/.unison/default.prf

# Roots of the synchronization
root = /var/www
root = ssh://192.168.0.101//var/www

# Paths to synchronize
#path = current
#path = common
#path = .netscape/bookmarks.html

# Some regexps specifying names and paths to ignore
#ignore = Path stats    ## ignores /var/www/stats
#ignore = Path stats/*  ## ignores /var/www/stats/*
#ignore = Path */stats  ## ignores /var/www/somedir/stats, but not /var/www/a/b/c/stats
#ignore = Name *stats   ## ignores all files/directories that end with "stats"
#ignore = Name stats*   ## ignores all files/directories that begin with "stats"
#ignore = Name *.tmp    ## ignores all files with the extension .tmp

#          When set to true, this flag causes the user interface to skip
#          asking for confirmations on non-conflicting changes. (More
#          precisely, when the user interface is done setting the
#          propagation direction for one entry and is about to move to the
#          next, it will skip over all non-conflicting entries and go
#          directly to the next conflict.)
auto=true

#          When this is set to true, the user interface will ask no
#          questions at all. Non-conflicting changes will be propagated;
#          conflicts will be skipped.
batch=true

#          !When this is set to true, Unison will request an extra
#          confirmation if it appears that the entire replica has been
#          deleted, before propagating the change. If the batch flag is
#          also set, synchronization will be aborted. When the path
#          preference is used, the same confirmation will be requested for
#          top-level paths. (At the moment, this flag only affects the
#          text user interface.) See also the mountpoint preference.
confirmbigdel=true

#          When this preference is set to true, Unison will use the
#          modification time and length of a file as a `pseudo inode
#          number' when scanning replicas for updates, instead of reading
#          the full contents of every file. Under Windows, this may cause
#          Unison to miss propagating an update if the modification time
#          and length of the file are both unchanged by the update.
#          However, Unison will never overwrite such an update with a
#          change from the other replica, since it always does a safe
#          check for updates just before propagating a change. Thus, it is
#          reasonable to use this switch under Windows most of the time
#          and occasionally run Unison once with fastcheck set to false,
#          if you are worried that Unison may have overlooked an update.
#          The default value of the preference is auto, which causes
#          Unison to use fast checking on Unix replicas (where it is safe)
#          and slow checking on Windows replicas. For backward
#          compatibility, yes, no, and default can be used in place of
#          true, false, and auto. See the section "Fast Checking" for more
#          information.
fastcheck=true

#          When this flag is set to true, the group attributes of the
#          files are synchronized. Whether the group names or the group
#          identifiers are synchronizeddepends on the preference numerids.
group=true

#          When this flag is set to true, the owner attributes of the
#          files are synchronized. Whether the owner names or the owner
#          identifiers are synchronizeddepends on the preference
#          extttnumerids.
owner=true

#          Including the preference -prefer root causes Unison always to
#          resolve conflicts in favor of root, rather than asking for
#          guidance from the user. (The syntax of root is the same as for
#          the root preference, plus the special values newer and older.)
#          This preference is overridden by the preferpartial preference.
#          This preference should be used only if you are sure you know
#          what you are doing!
prefer=newer

#          When this preference is set to true, the textual user interface
#          will print nothing at all, except in the case of errors.
#          Setting silent to true automatically sets the batch preference
#          to true.
silent=true

#          When this flag is set to true, file modification times (but not
#          directory modtimes) are propagated.
times=true

The comments should make the file self-explaining, except for the path directives. If you specify no path directives, then the directories in the root directives will be synchronized. If you specify path directives, then the paths are relative to the root path (e.g. root = /var/www and path = current translates to /var/www/current), and only these subdirectories will be synchronized, not the whole directory specified in the root directive.

You can find out more about the available options by taking a look at Unison's man page:

man unison

Now that we have put all settings in a preferences file (especially the root (and optionally the path) directives), we can run Unison without any arguments:

unison

5 Creating A Cron Job

server1:

We want to automate synchronization, that is why we create a cron job for it on server1.example.com:

crontab -e

*/5 * * * * /usr/bin/unison &> /dev/null

This would run Unison every 5 minutes; adjust it to your needs (see

man 5 crontab

). I use the full path to unison here (/usr/bin/unison) just to go sure that cron knows where to find unison. Your unison location might differ. Run

which unison

to find out where yours is.

6 Links

Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Debian: http://www.debian.org/

Lan Management System (LMS) On Debian Squeeze

2012-01-01T16:41:00.000+06:00

LMS (Lan Management System) is a good system for small ISPs made in Poland. Documentation for LMS GUI is available in english here. But installation, configuration and integration with firewall or traffic shaping mechanisms could take a lot of time. Here you can try my scripts for express-installation of LMS. The scripts were tested in several companies.

First download and install Debian Squeeze in netinstall version i386 or amd64. Install it with basic system only (no X GUI, no services except ssh). Choose eth0for your primary interface and configure network settings (IP address, netmask, gateway and DNS servers). Make sure you have a second interface described aseth1. Next log into your root account (via ssh by PuTTY or directly on the console) and type the magic three lines for i386 architecture:

wget http://files.v-smart.pl/v-smart-2.0/install-vsmart-2.0-en-32bit.sh
chmod +x install-vsmart-2.0-en-32bit.sh
./install-vsmart-2.0-en-32bit.sh

and for amd64 architecture:

wget http://files.v-smart.pl/v-smart-2.0/install-vsmart-2.0-en-64bit.sh
chmod +x install-vsmart-2.0-en-64bit.sh
./install-vsmart-2.0-en-64bit.sh

The scripts will download necessary packages from debian repositories and my deb packages:

linux kernel 2.6.32 with patches: layer-7, imq, esfq

iptables 1.4.8 with patches: layer-7 and imq

iproute 20101221 with esfq patch

ppp 2.4.3 with mppe and mppc

pppoe 3.10 with mppe, mppc and kernel plugin

pptpd 1.3.4 with mppe and mppc

All the packages are available for independent download from:

http://files.v-smart.pl/v-smart-2.0/

You may view the scripts before executing to see what they exactly do. You have to write down the MySQL root password and type it when the install script ask for. After reboot you can go to the router GUI via browser. Simply open the router IP address in the browser. First time LMS will ask you for creating an admin account. Don't forget to check full access option for admin. Example configuration is available for view after installation. You have to set up your WAN bandwidth in the /router/router.conf file in kilobits-per-second. Default is 10Mbps.

How does it work? Network administrator adds clients, computers and tariffs (download and upload speed) into LMS. There is my daemon running in the background which checks if something was changed in the GUI configuration. If so, the daemon will update the configuration file for the firewall (/router/lms.conf) and reload firewall, NAT and traffic shaping. Firewall scripts and configs are in the /router directory. LMS GUI is installed in the /var/www directory. Other stuff (messages, daemon, etc.) are in /var/v-smart directory. Network configuration you can find in /etc/rc.local script.

Installed LMS is pure and unmodified. In the database there is vsmart table with to-do records that are read by the daemon in 3-second period. I added MySQL triggers to follow changes in the LMS tables. The triggers will update to-do records when something is changed in customers' devices configuration. Then the daemon makes a decision about reloading firewall, traffic shaper and NAT. Finally - changes in LMS GUI are set in the router almost instantly. This is the main idea of my project.

In the crontab there are periodicaly run some LMS scripts (stats, payments, host alive checking and other). Feel free to view or adjust /etc/cron.d/vsmart file.

List of router main functions:

- Dynamic traffic shaping on WAN port using IMQ with HTB/esfq and service priority,
- Static traffic shaping on LAN port (LMS tariffs),
- MAC + IP authorization for clients,
- DHCP server,
- DNS server,
- PPPoE server,
- PPtP server (Windows VPN),
- Messages: payment reminder, total block, no authorization,
- LMS GUI - see manual,
- LMS functions: customers, computers, networks, network devices, network map, tariffs, invoices, helpdesk, calendar,
- LMS USERPANEL - access via http://router_ip/userpanel,
- Night tarrifs for LAN and WAN,
- Port forward (/router/forward.conf)
- local speedtest in http://router_ip/freemeter taken from http://speed-meter.net

Technical solutions

1. How to add new network(s) to my LAN?

Let us consider new LAN network: 192.168.102.0/24 with 192.168.102.254 gateway address on the eth1 interface. In LMS GUI (IP Networks -> New network) we add:

Network name: LAN2

Network addres/mask: 192.168.102.0 / 24 (256-addresses)

Interface: eth1

Gateway: 192.168.102.254

DNS servers: 192.168.102.254, 8.8.8.8

In the file /etc/rc.local we add before /usr/sbin/ip link set eth1 up:

/usr/sbin/ip a a 192.168.102.254/24 brd 192.168.102.255 dev eth1

In the file /etc/rc.local we add on the bottom:

/usr/sbin/pppoe-server -I eth1 -L 192.168.102.254 -N 1000 -k

In the file /router/router.conf we add variable with value:

INTNET2=192.168.102.0/24

In the file /router/scripts/firewall.sh and /router/scripts/nat.sh we find all lines that include $INTNET1 variable and we copy them bellow changing $INTNET1 for $INTNET2. For example:

$IPTABLES -A INPUT -s $INTNET1 -m state --state NEW -p tcp --sport 1024: --dport 53 -j ACCEPT
$IPTABLES -A INPUT -s $INTNET2 -m state --state NEW -p udp --sport 1024: --dport 53 -j ACCEPT

etc..

Tip: If you want to use public subnet on LAN you have to comment MASQUERADE for this subnet in /router/scripts/nat.sh:

#$IPTABLES -t nat -A POSTROUTING -s $INTNET2 -o $EXTDEV -j MASQUERADE

After reboot everything should work fine.

Tiny Web Proxy And Content Filtering Appliance (Version 1.2) With Debian Squeeze

2012-01-01T16:19:00.002+06:00

Set up a small virtual machine to speed up and secure your home / small enterprise web surfing network using Debian 6, Squid 3.1 and QuintoLabs Content Security 1.2 applications deployed in a VMWare Virtual Player running on Windows 7 x64 as a host operating system. This howto is targeted at novice users and may sometimes seem too thorough for more advanced gurus.

While preparing this howto I have tried several different Linux distributions to act as the base for the virtual appliance we are going to create and found out that the one easiest to deploy, having smallest memory footprint and latest packages available is Debian Linux 6. In case you disagree please adjust the instructions of this howto for your favorite distribution and mention the needed changes in comments. I will try to integrate them into the next version of the howto.

Step 1. Download And Install Debian 6

Go to http://www.debian.org and get the smallest ISO image of the latest version of Debian 6 32-bit. The site says the recommended version for a modern server is x64 bit but as we are trying to create a small virtual machine the 32bit will suffice for our purposes.

Start up the VMWare Virtual Player and create a new virtual machine with the following hardware parameters: name - virtual-proxy, hard disk - 8Gb. Press the "Customize the hardware" button and delete the floppy, USB controller, printer and sound card, set the amount of memory to 256Mb. Switch the network adapter from "NAT" mode into "Bridged". Point the virtual CDROM to the ISO image that you have downloaded earlier and start the virtual machine.

Follow the steps of the Debian install wizard mostly accepting the defaults. Deselect all packages in the "Software Selection" wizard step except for "SSH server" and "Standard system utilities". Configure machine hostname as "proxy" and root password as "P@ssw0rd" (without quotation marks). Now wait a little until the installation is complete and then reboot the system.

The following screenshots of the installation show most important steps of the installation process.

After reboot check the amount of free memory available and note that it is around 200Mb (kind of cool comparing to default CentOS 5 installation with 140Mb of free memory available after similar install).

Step 2. Perform Post Install Configuration Of Debian

The IP address of the network interface card of our freshly installed Debian server is set to DHCP. While it is possible to leave it like this it means that every time the server is restarted it may be assigned a different IP address so accessing this server later by IP address may become a little irritating. The easiest way is to set the IP address to static one. To do it we must change the file located at/etc/network/interfaces by typing the following in the terminal

# nano /etc/network/interfaces

Find the line with iface eth0 inet dhcp in the file, remove or comment it out and add the following:

# iface eth0 inet dhcp
iface eth0 inet static
                address 192.168.1.2
                netmask 255.255.255.0
                network 192.168.1.0
                broadcast 192.168.1.255
                gateway 192.168.1.1

NOTE: The provided above settings (IP addresses) are valid for my environment that connects to the ISP using the LinkSys Wireless N Broadband Router (with custom dd-wrt firmware) that has a DHCP server build in that gives out internal IP addresses from the 192.168.1.* private subnet. Your router may give other addresses so please beware :) !

One more thing left to do is adjust the DNS settings. Run the following command in the terminal and add the IP address of the DNS server that runs on router:

# nano /etc/resolv.conf

nameserver 192.168.1.1

Now restart your networking by typing #/etc/init.d/networking in the root terminal or just restarting the virtual machine. After restart confirm that the networking functions correctly by typing in the terminal (there should not be any errors in the outputs on these commands):

ping -c 3 192.168.1.1
nslookup google.com

Next small step prior to installing squid is to update the system to the latest version of packages. This will require disabling Debian repositories that refer to CDROM that we used to install the system from and enabling the online repositories.

Type the following in the root terminal #nano /etc/apt/sources.list then find and comment all lines that start with deb cdrom:. Save the file by pressing CTRL+o followed by CTRL+x and exit nano. Then perform system update by typing

# apt-get update && apt-get upgrade

in the root terminal. Wait until the system downloads all the updates.

Step 3. Install Squid Web Caching Proxy

Next we need to install the latest version of the squid proxy server. In order to do that type the following in the root terminal:

# apt-get install squid3

The squid packages is downloaded from the Internet and installed automatically. The only thing to do is to let the external users from our network access the squid. Open the squid configuration file by typing

# nano /etc/squid3/squid.conf

and uncomment the following lines: acl localnet src 192.168.0.0/16 (approximately line 651) and http_access allow localnet (approximately line 786).

Save the file and restart the squid by typing

# service squid3 restart

To verify that squid runs correctly point your browser to the IP address of the proxy server (192.168.1.2) and surf to some of your favorite websites.

Step 4. Install Apache 2

t is also a good idea to have a web server installed on the virtual machine. This web server will later host the status and report information for Squid and Content Security for Squid Proxy. In order to have the Apache2 installed issue the following commands in the root terminal:

# apt-get install apache2 libapache2-mod-php5

Then open your browser and navigate to http://192.168.1.2. You should see the “It Works!” greetings from Apache.

Step 5. Install QuintoLabs Content Security 1.2

Next step would be to install the Content Security 1.2 for Squid from QuintoLabs (I will refer to it as qlproxy further in text). NOTE: this part was upgraded from the previous version of this howto. The 1.2 version of qlproxy now supports ICAP based integration with Squid and is much easier to install.

For those who do not know, qlproxy is a content filtering server to be used as a companion to the Squid web proxy that (citation) "allows the administrator to filter/block web downloads, remove advertisements and banners and control web site usage by the proxy clients" (i.e. prohibit explicit and adult content).

Unfortunately QuintoLabs does not yet have online package repository with qlproxy but I heard it will change in the future :). So until it happens we have to get the Debian 6 package of qlproxy manually from the QuintoLabs web site at http://www.quintolabs.com/qlicap_download.php using your favorite browser (thorough your new squid of course :) ) and upload the package to the system using scp. Another (much easier) way would be to type the following commands in the root terminal:

# wget http://www.quintolabs.com/qlproxy/binaries/1.2.217.0/qlproxy-1.2-debian-1.2.217_i386.deb

Wait a little until the download completes (approx. 20Mb) and run the following command to install the downloaded package

# dpkg –install qlproxy-1.2-debian-1.2.217_i386.deb

Installer will run and after a short while the program will be installed into /opt/quintolabs/qlproxy.

Now we need to configure it and integrate it with Squid. The configuration files are plain text and stored in /opt/quintolabs/qlproxy/etc/*.conf and rather simple to modify with a handful of comments inside. I am going to perform the following modifications:

Change the default blocked template to the minimal by commenting out the blocked_page=/opt/quintolabs/qlproxy/redirect/blocked.html and uncommentingblocked_page=/opt/quintolabs/qlproxy/redirect/blocked_minimal.html. This is done to get the empty blank page on the screen when qlproxy blocks something. The idea for this is to have nothing on the browser screen instead of default text “this item was blocked…”. When QuintoLabs adds a default 1x1 pixel for blocked ads in the next version this step would be irrelevant… but for now it is ok.
I personally do not like excessive advertizing on the web sites so as I often browse through Russian and German web sites I will also enable adblock filtering by uncommenting the corresponding Russian and German adblock subscriptions in /opt/quintolabs/qlproxy/etc/adblock.conf file. I also do not like sites tracking me so I usually uncomment easy_privacy subscription in the same file.
My kids sometimes play online games on my computer so I prefer to set the level of adult blocking heuristics to high in the /opt/quintolabs/qlproxy/etc/adultblock.conf file –by changing fromheuristics_level = normal to heuristics_level = high. If anything is falsely blocked by the qlproxy I can later add it to the exceptions.conf file to have it passed through.
I heard that worms, trojans and other malware related software often connect to the world by IP addresses so I put a magic regexp into the /opt/quintolabs/qlproxy/etc/httpblock.conf file to filter them outurl = http://\d+\.\d+\.\d+\.\d+/.*

Good for now, let us issue a restart command to make the qlproxyd daemon reload the configuration:

#/etc/init.d/qlproxy restart

Next we need to integrate it with Squid. As the qlproxy daemon now supports the shiny ICAP protocol this is a little bit different from the url_rewrite_program integration described in the previous version of this howto. By the way, README file in /opt/quintolabs/qlproxy/ contains instructions on how to do that. Anyway here are the steps required:

Open the /etc/squid3/squid.conf in nano by typing
#nano /etc/squid3/squid.conf
in the root terminal
Find and change the icap_enable off directive to icap_enable on
Uncomment icap_preview_enable on directive
Uncomment and change icap_preview_size -1 directive to icap_preview_size 4096
Uncomment directive icap_persistent_connections on
Uncomment and change directive icap_send_client_ip off to icap_send_client_ip on
Uncomment and change directive icap_send_client_username off to icap_send_client_username on
Find the icap_service section with commented out samples for different services and add somewhere there two lines icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod andicap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod
Find the adaptation_access section and add the following two lines adaptation_access qlproxy1 allow all and adaptation_access qlproxy2 allow all
Write the changes to file and close nano by typing Ctrl+o and then Ctrl+x

Now restart the squid by typing

# service squid3 restart

in the root terminal. After restart try surfing the same sites with your browser and see how nicely ads are blocked. Another useful test is to go to the eicar.com web site and try to download a sample artificial eicar.com virus to see that com files are blocked by the download filter.

Note: for those of you who must stick with squid 2.6 for performance reasons the url rewriter integration is quite straightforward. Open /etc/squid3/squid.conf and find the url_rewrite_program section. Add the following url_rewrite_program /opt/quintolabs/qlproxy/sbin/qlproxyd_redirector --config_path=/opt/quintolabs/qlproxy/etc/qlproxyd.conf

The last thing to do is to integrate the qlproxy with Apache to be able to see the reports on user activities generated once a day. This is actually quite easy, open the /etc/apache2/sites-enabled/default file and add the following to it near the </VirtualHost> directive:

Alias /reports /opt/quintolabs/qlproxy/reports
   <Directory /opt/quintolabs/qlproxy/reports >
        Options FollowSymLinks
        AllowOverride None
   </Directory>

Now reload the apache by typing in the terminal #service apache2 restart.

You can navigate to http://192.168.1.2/reports to see the generated reports. The funny thing is that qlproxy blocks access by the IP address according to our settings in httpblock.conf described earlier. Solution would be to add the 192.168.1.2 as entry to the /opt/quintolabs/qlproxy/etc/exceptions.conf or just tell the browser not to use proxy for this address.

Resume

Finally everything is in place to start the accelerated secure web surfing without adverts – point your browser to 192.168.1.2 port 3128 and browse to your favorite website and see the difference. The IP addresses in URLs are blocked and explicitly adult content sites too. The VMWare takes not more than 256 MB and surfing experience is quite acceptable. The system is automatically updated once a day for the latest url block list and advert subscriptions and requires minimal additional maintenance.

Used Documentation Links

Program to monitor log-files

2012-01-01T11:26:00.000+06:00

xlogmaster

xlogmaster - A program to monitor logfiles

Xlogmaster can monitor an almost infinite number of logfiles and all devices that can be read via "cat" like the /proc ones.On a regular expression base, you can hide, raise or lower lines in the logfile and set triggers, which can launch scripts or pop-up a window

Install xlogmaster in Debian

#apt-get install xlogmaster

httpd logfile analysis package

2012-01-01T11:24:00.006+06:00

wwwstat
wwwstat - httpd logfile analysis package

wwwstat processes a sequence of httpd Common Logfile Format access_log files and prior summary outputs, and then outputs a summary of the access statistics in a nice HTML format.

splitlog processes a sequence of httpd Common Logfile Format access_log files (or CLF with a one-field prefix) and splits the entries into separate files according to the requested URL and/or vhost prefix.

Install wwwstat in Debian

#apt-get install wwwstat

Real-time user logins monitoring tool

2012-01-01T11:24:00.003+06:00

whowatch

whowatch - Real-time user logins monitoring tool

whowatch is a ncurses who-like utility that displays information about the users currently logged on to the machine, in real-time. Besides standard information (login name, tty, host, user's process), the type of the connection (ie. telnet or ssh) is shown.

Install whowatch in Debian
#apt-get install whowatch

VA RAID monitor

2012-01-01T11:24:00.000+06:00

varmon

varmon - VA RAID monitor

A text-based tool to monitor DAC 960 RAID controllers. This includes Mylex RAID cards from the 960 and 1100 series, eXteremeRAID 2000 & 3000, AcceleRAID 352 & 170.

Install varmon in Debian

#apt-get install varmon

Display informations about a local user

2012-01-01T11:23:00.002+06:00

userinfo

userinfo - display informations about a local user

Userinfo is a small console utility to display as much information about a local user as possible. It currently supports: password file info, home directory mode, mail forwards/aliases/last read time/new time/folder size, utmp info and more

Install userinfo in Debian

#apt-get install userinfo

Report system security vulnerabilities

2012-01-01T11:22:00.002+06:00

tiger
tiger - Report system security vulnerabilities

TIGER, or the 'tiger' scripts, is a set of Bourne shell scripts, C programs and data files which are used to perform a security audit of UNIX systems. TIGER has one primary goal: report ways 'root' can be compromised.

Debian's TIGER incorporates new checks primarily oriented towards Debian distribution including: md5sums checks of installed files, location of files not belonging to packages, check of security advisories and analysis of local listening processes.

Install tiger in Debian

#apt-get install tiger

Fast, Flexible and Detailed log analysis for the Squid Proxy

2012-01-01T11:21:00.002+06:00

srg

srg - Fast, Flexible and Detailed log analysis for the Squid Proxy

SRG provides fast and flexible analysis of squid proxy log files while retaining a high level of detail, including down the the exact URLs visited by users and the times at which they were visited. SRG also contains support for generated PHP reports that can be integrated with your existing authentication system to provide granular access to report output.

Install srg in Debian

#apt-get install srg

Data integrity and host intrusion alert system

2012-01-01T11:20:00.005+06:00

samhain

samhain - Data integrity and host intrusion alert system

Samhain is an integrity checker and host intrusion detection system that can be used on single hosts as well as large, UNIX-based networks. It supports central monitoring as well as powerful (and new) stealth features to run undetected on memory using steganography.

Main features

* Complete integrity check
+ uses cryptographic checksums of files to detect modifications,
+ can find rogue SUID executables anywhere on disk, and
* Centralized monitoring
+ native support for logging to a central server via encrypted and authenticated connections
* Tamper resistance
+ database and configuration files can be signed
+ logfile entries and e-mail reports are signed
+ support for stealth operation

Install samhain in Debian

#apt-get install samhain

Tool for auditing Cisco router configuration

2012-01-01T11:20:00.002+06:00

router-audit-tool

router-audit-tool - Tool for auditing Cisco router configuration

The Router Audit Tool (rat) audits router configurations. It will log into the routers specified using provided login
information, download the configuration and audit against a set of provided rules.

This package also provides the NSA's Cisco Router Configuration Guide

Install router-audit-tool in Debian

#apt-get install router-audit-tool

Undelete files on ext2 partitions Recover

2012-01-01T11:19:00.005+06:00

recover

recover - Undelete files on ext2 partitions

Recover automates some steps as described in the ext2-undeletion howto. This means it seeks all the deleted inodes on your hard drive with debugfs. When all the inodes are indexed, recover asks you some questions about the deleted file. These questions are:

* Hard disk device name
* Year of deletion
* Month of deletion
* Weekday of deletion
* First/Last possible day of month
* Min/Max possible file size
* Min/Max possible deletion hour
* Min/Max possible deletion minute
* User ID of the deleted file
* A text string the file included (can be ignored)
If recover found any fitting inodes, it asks to give a directory name and dumps the inodes into the directory. Finally it asks you if you want to filter the inodes again (in case you typed some wrong answers).

Note that recover works only with ext2 filesystems - it does not support ext3.

Install recover in Debian
#apt-get install recover

Tk-based printer configuration tool

2012-01-01T11:19:00.002+06:00

printtool

printtool - Tk-based printer configuration tool

This is an enhanced version of the Red Hat printtool, with autodetection of printers and PPD support.

Install printtool in Debian

#apt-get install printtool

Tool to tune system for optimal performance

2012-01-01T11:18:00.005+06:00

powertweak

powertweak - Tool to tune system for optimal performance

Powertweak is a tool for tweaking Linux systems to peak performance. It tunes PCI devices to use optimal settings and enables performance enhancing features of the CPU(s).

This package contains no files. It merely ensures that you install the following packages: powertweakd, powertweak-extra, powertweak-gtk

Install powertweak in Debian
#apt-get install powertweak

The GNU Parted disk partition resizing program

2012-01-01T11:18:00.002+06:00

parted

parted - The GNU Parted disk partition resizing program

GNU Parted is a program that allows you to create, destroy, resize, move and copy hard disk partitions. This is useful for creating space for new operating systems, reorganising disk usage, and copying data to new hard disks. This package contains the Parted binary and manual page.

Parted currently supports DOS, Mac, Sun, BSD, GPT, MIPS and PC98 disklabels/partition tables, as well as a 'loop' (raw disk) type which allows use on RAID/LVM. Filesystems which are currently fully supported are ext2, ext3, fat (FAT16 and FAT32), ReiserFS (with libreiserfs) and linux-swap. Parted can also detect and remove HFS (Mac OS), JFS, NTFS, UFS (Sun and HP), XFS and ASFS/AFFS/APFS (Amiga) filesystems, but cannot create, resize or check these filesystems yet.

Note that ReiserFS support is only enabled if you install the libreiserfs0.3-0 package. Since libreiserfs0.3-0 has been removed from sarge, ReiserFS support is not compiled in the default package.

The nature of this software means that any bugs could cause massive data loss. While there are no known bugs at the moment, they could exist, so please back up all important files before running it, and do so at your own risk.

Install parted in Debian
#apt-get install parted

GUI for searching packages and viewing package information

2012-01-01T11:17:00.005+06:00

packagesearch

packagesearch - GUI for searching packages and viewing package information

This tool is aimed to help you search the packages you need. It should make the task of searching a pleasant experience.

Search can be done by

* pattern
* tags (based on the debtags system, a new way of categorizing Debian packages)
* files
* installed status
* orphaned packages
Additionally a lot of information about the packages is displayed, including the files belonging to them.
It is possible to install or remove packages.

Install packagesearch in Debian

#apt-get install packagesearch

Remote network security auditor, the server

2012-01-01T11:17:00.002+06:00

nessusd

nessusd - Remote network security auditor, the server

The Nessus Security Scanner is a security auditing tool. It makes possible to test security modules in an attempt to find vulnerable spots that should be fixed.

It is made up of two parts: a server, and a client. The server/daemon, nessusd, is in charge of the attacks, whereas the client, nessus, provides the user a nice X11/GTK+ interface.

This package contains the nessusd server, which must be run as root.

Install nessusd in Debian

#apt-get install nessusd

Remote network security auditor, the client

2012-01-01T11:16:00.005+06:00

nessus

nessus - Remote network security auditor, the client

The Nessus Security Scanner is a security auditing tool. It makes possible to test security modules in an attempt to find vulnerable spots that should be fixed.

It is made up of two parts: a server, and a client. The server/daemon, nessusd, is in charge of the attacks, whereas the client, nessus, provides the user a nice X11/GTK+ interface.

This package contains the GTK+ client, which exists in other forms and on other platforms, too.

Install nessus in Debian

#apt-get install nessus

Distributed password cracking system

2012-01-01T11:16:00.002+06:00

medussa
medussa - Distributed password cracking system

Medussa is a distributed password cracking system that can attempt various types of attacks to crypted passwords distributing the work on many machines.

Install medussa in Debian

#apt-get install medussa

Hardware monitoring without kernel dependencies (text client)

2012-01-01T11:15:00.005+06:00

mbmon

mbmon - Hardware monitoring without kernel dependencies (text client)

(x)mbmon allows you to monitor hardware status using your motherboard's sensors. This information can be: temperatures, voltages and/or fan speeds.This package contains the console client.

Install mbmon in Debian

#apt-get install mbmon

List all SCSI devices (or hosts) currently on system

2012-01-01T11:15:00.002+06:00

lsscsi

lsscsi - list all SCSI devices (or hosts) currently on system

Uses information in sysfs (linux kernels 2.6.0 and later) to list all scsi devices (or hosts) currently attached to the system. Options can be used to control the amount and form of information provided for each device.

Install lsscsi in Debian
#apt-get install lsscsi

Automagically remove unnecessary locale data

2012-01-01T11:14:00.001+06:00

localepurge

localepurge - Automagically remove unnecessary locale data

This is just a simple script to recover diskspace wasted for unneeded locale files and localized man pages. It will
automagically be invoked upon completion of any apt installation run.

Please note, that this tool is a hack which is *not* integrated with Debian's package management system and therefore is not for the faint of heart. This program interferes with the Debian package management and does provoke strange, but usually harmless, behaviour of programs related with apt/dpkg like dpkg-repack, debsums, reportbug, etc. Responsibility for its usage and possible breakage of your system therefore lies in the sysadmin's (your) hands.

Please definitely do abstain from reporting any such bugs blaming localepurge if you break your system by using it. If you don't know what you are doing and can't handle any resulting breakage on your own then please simply don't use this package.

Install localepurge in Debian

#apt-get install localepurge

output

localepurge: Disk space freed in /usr/share/locale: 32672K

Distributed resource monitor

2012-01-01T11:13:00.002+06:00

jmon

jmon - distributed resource monitor

Resource monitoring for large networks. The jMon system allows for the real time monitoring of CPU, memory and swap usage. The system makes use of a small daemon running on each of the machines that are to be monitored.

The client makes a TCP/IP connection with each of the servers specified in the configuration file. The statistics of all the host machines are displayed in an ncurses window on the console.

Install jmon in Debian
#apt-get install jmon