Use Perl to enhance ModSecurity

Anatoliy Dimitrov — Thu, 23 May 2013 11:00:00 GMT

ModSecurity is the most popular open source web application firewall. Its built-in, basic functionality simply blocks a client's request if it matches a security rule. However, you can extend ModSecurity in various ways, including through your own custom Perl scripts. Let's see how.

I'm going to assume you have already installed and configured ModSecurity. If not, read how to Protect and audit your web server with ModSecurity. I'll also create a script you can use, and make it as simple as possible so that you can understand the logic and rewrite the code in any programming language if you so choose.

How to extract the needed information

By default, any request that triggers a ModSecurity rule is logged in the Apache's error log (/var/log/httpd/error_log). An example ModSecurity entry looks like this:

[Mon May 06 09:16:12 2013] [error] [client 10.0.0.1] ModSecurity: Warning. Operator LT matched 5 at TX:inbound_anomaly_score. [file "/etc/httpd/conf/crs/modsecurity_crs_60_correlation.conf"] [line "33"] [id "981203"] [msg "Inbound Anomaly Score (Total Inbound Score: 2, SQLi=, XSS=): Request Missing an Accept Header"] [hostname "www.example.org"] [uri "/index.php"] [unique_id "BOlcOH8AAAEAABlxPboAAAAJ"]

This entry provides a lot of information, such as time of request (Mon May 06 09:16:12 2013), remote client's IP address (client 10.0.0.1), and the triggered ModSecurity rule ID (id "981203"). You can focus on this information and extract it with a regular expression.

A good regular expression for this purpose is

\[ $date .*\[client ([0-9|\.]*)\] ModSecurity.*\[id "([0-9]{6})"\]

How do you interpret that? First, one variable is used – date. It determines the time range for which information will be extracted. It is in the format Day Month Day-Of-Month Hour:Minute:Seconds Year. You don't have to specify the full date; use only the part you want and the rest is matched automatically. For example, if you need information only for one day, you should specify Mon May 06. Everything after that – the hours, minutes, and seconds – is matched automatically because of the "match all" combination (.*). If you want to get today's date automatically in suitable format, you can use the common Linux /bin/date command with this special formatting: date +"%a %b %d".

Continuing further with the regular expression, the two matches that will be referenced later are in brackets. The expression inside the brackets of the first one, \[client ([0-9|\.]*)\], is the simplest way to match the IP address of the remote client. The second bracketed expression, \[id "([0-9]{6})"\], matches the ID of the ModSecurity rule. All officially distributed ModSecurity rules are six digits. If you have rules with different numbering, make sure to adjust the expression.

The entire regular expression will match every IP address and ModSecurity ID from the log. If you have a lot of traffic and many ModSecurity rules, you may end up flooded with information, so you should apply some filters.

First, try to whitelist legitimate bots and disregard them. Most legitimate bots send valid requests and usually do not trigger ModSecurity alerts, but there are exceptions, such as some monitoring tools and less popular search engines. If you know of such an IP address – let's say for example 10.1.1.1 – you can whitelist it directly in ModSecurity's configuration file /etc/httpd/conf/crs/whitelists.conf with the directive SecRule REMOTE_ADDR "^10.1.1.1" phase:1,nolog,allow,ctl:ruleEngine=off.

However, often you deal with a large array of IP addresses and it's not feasible to whitelist them all. You should therefore find a way to check whether reported addresses are legitimate and avoid blocking them. One way to do that is to check whether the address has an authentic reverse DNS resolution. If it does, you can check whether its domain matches a list of domains you have whitelisted. You can see how this is implemented with a Perl subroutine called legitimate_bot_check in the example script at the end of the article.

To further reduce false-positive ModSecurity alerts, it is useful to establish some thresholds, and disregard IP addresses or ModSecurity IDs whose counters are below the threshold. This is necessary because even the most compliant client and regular visitor may trigger a complex ModSecurity rule, which makes it impractical to investigate every triggered rule. You can generally disregard IP addresses and rules with few hits. You can see just how to do this in the example script.

How to use the accumulated information

If you use this regular expression, your script will have a list of suspicious IP addresses and ModSecurity IDs and some counters for them. Here are a few ideas about how to make good use of this information:

Examine an activity report – Receive regular email reports about the accumulated information to get an overall view of the attacks against your server. You'll see how in the example script.
Block attackers – Once the counter for an IP address exceeds a threshold, you should block it. Aggressive attackers not only pose security risks but also consume system resources. You can block an address either manually or by programming it to happen automatically. Iptables is always a good choice for this purpose. A command such as /sbin/iptables -I INPUT -s 10.0.0.1 -p TCP --dport 80,443 -j DROP denies access to ports 80 and 443 for the IP address 10.0.0.1.
Improve ModSecurity rules – After evaluating the counters for each ModSecurity rule, you may find that some IDs are commonly blocking valid requests. Investigate every ID that has been triggered more than 100 times and make sure it's applicable for your environment. If one is not, disable it so it cannot block valid requests. You can do this with the ModSecurity directive SecRuleRemoveById [ID1] [ID2] ... in the a file /etc/httpd/conf/crs/exceptions.conf.

An example script

Here's an example script that implements all of the above features and sends an email message with a report. You should be able to use this script after tailoring it slightly to your environment. Set it to run as a cron job just before midnight so that it analyzes the information for a full day.

#!/usr/bin/perl

use warnings;
use strict;
use Socket;

# Define your admin email. This is where you will receive the reports
my $admin_mail = 'admin_reports@example.org';

# My from email. This is the email that mails will appear from

my $from_mail = 'sender@example.org';

# Define either the first argument as the Modsecurity Log or hardcode it (default)
# my $file = $ARGV['0'];
my $file = '/var/log/httpd/error_log';

# Create the hashes for the attackers and the ModSecurity ids.
# The keys for the attackers hashes are the remote client IPs. The values are the number of times this IP has appeared in the ModSecurity log with a triggered rule.
# The keys for the sec_ids hash are the IDs of the triggered ModSecurity ids. Their values are the number of times each of these rules has been triggered. 

my %attackers = ();
my %sec_ids = ();

# Get the current date in the format 'Fri Feb 17'. This is necessary for log files.
# You can also put a static date for a certain period of time

my $date = `date +"%a %b %d"`;
#my $date = 'Sun May 05'; #you can also specify a custom date here
# Remove the new line after the end of the date string
chomp($date);

# The search pattern for the ModSecurity rules from the Apache error log
my $search_pattern = '\['.$date . '.*\[client ([0-9|\.]*)\] ModSecurity.*\[id "([0-9]{6})"\]';

# From the log we're counting IPs and rules hit. To avoid showing all of them we're setting some thresholds. 
# $ips_threshold - the threshold of triggered ModSecurity rules above which IPs should be accounted for.
# $rules_threshold - the threshold of counting the IDs of triggered rules
# Increase these thresholds if you have a lot of rules and many legitimate non-complying clients

my $ips_threshold = 100;
my $rules_threshold = 100;

# Enable blocking or use only reporting. Specify 1 if you want any enabled.
my $blocking = 0;
my $reporting = 1;

# How ips should be blocked. Use iptables rule and the INPUT chain. In most cases just blocking the default web ports 80 and 443 is sufficient
sub block_command{
    my $ip = $_[0];
    system("/sbin/iptables -I INPUT -s ".$ip." -p TCP --dport 80,443 -j DROP");
}

# Create some safeguard against blocking legitimate IPs.
# For example, to avoid blocking legitimate bots, search engines and monitoring tools whitelist them here.
# In this subroutine whitelist search engines and make sure there is reverse resolving for them.

sub legitimate_bot_check {
    # Take the first argument of the routine as the remote IP. For example 66.249.66.1
    my $ip = $_[0];
    
    # Confirm there is a proper reverse DNS record or exit with false. For the IP 66.249.66.1 this should return crawl-66-249-66-1.googlebot.com.
    my $iaddr = inet_aton($ip) or return 0;
    
    # Confirm the previously obtained reverse DNS record (crawl-66-249-66-1.googlebot.com) points to the same IP (66.249.66.1) or exit with false
    my $name  = gethostbyaddr($iaddr, AF_INET) or return 0; 
    
    # Once you prove the IP has an authentic reverse DNS, check if it is among popular bots. 
    # If yes, exit with 'true' meaning that the IP shouldn't be counted
    if (gethostbyname($name) eq $iaddr) {
            if ($name =~ m/(yandex\.com$|search\.msn\.com$|googlebot\.com$|yahoo\.com$|baidu\.jp$|baidu\.com$|pingdom\.com$|monitorengine\.com$)/) {
                return 1;
            }
    } else {
        return 0; 
    }
}

# Open the log file for reading and prepare it for search
open LOGFILE, '<', $file or warn "Unable to open log file: $file!\n";

# Create an array from the contents of the file so that we can iterate through it
my @logarray = <LOGFILE>;

for (@logarray) {
    # Create a conditional if a row matches our search pattern
    if ($_ =~ /$search_pattern/) {
        # For each match increase the number of hits for each remote IP
        $attackers{$1}++;
        # For each match increase the number of hits for each unique ModSecurity ID
        $sec_ids{$2}++;
    }
}

# Close the log file safely
close LOGFILE;

# Iterate through the attackers hash
while ( my ($attacker_ip,$number_of_hits) = each %attackers ) {    
    # Apply the thresholds and filter the data
    # First filter the attackers.
    # Besides applying the threshold make sure you also apply some logic for whitelisting legitimate IPs such as the legitimate bots subroutine.    
    if ($number_of_hits < $ips_threshold or &legitimate_bot_check($attacker_ip)) {
        delete($attackers{$attacker_ip});
    }    
}

# Apply the threshold for the ModSecurity rules.

while ( my ($rule_id,$counter) = each %sec_ids ) {
    delete($sec_ids{$rule_id}) if $counter < $rules_threshold;    
}
        
# If blocking is enabled block the suspicious IPs.
if ($blocking) {
    while ( my ($attacker_ip,$number_of_hits) = each %attackers ) {
        &block_command($attacker_ip);
    }
}

# Send a report if enabled. Pipe the mail text to sendmail.

if ($reporting) {
    open(MAIL, "|/usr/sbin/sendmail -t");

    print MAIL "From: $from_mail\n";
    print MAIL "To: $admin_mail\n";
    print MAIL "Subject: Mod Security Stats\n\n";

    print MAIL "Attacking IPs and number of hits:\n";
        while ( my ($attacker_ip,$number_of_hits) = each %attackers ) {
            print MAIL "$attacker_ip => $number_of_hits\n";
    }
    print MAIL "\nModsecurity IDs and their counters:\n";
        while ( my ($rule_id,$counter) = each %sec_ids ) {
            print MAIL "Rule ID $rule_id hit $counter number of times.\n";
    }
    close(MAIL);
}

Once you're running the script daily, you will save time on manual jobs such as analyzing the logs and blocking intruders. By digging through the logs and implementing a few additional features, you can turn ModSecurity into a comprehensive intrusion detection and protection system. All it takes is some simple programming and certain logic.

The secret to great reporting with Drupal 7

Juliet Kemp — Mon, 20 May 2013 11:00:00 GMT

Drupal, the popular CMS platform, doesn't just allow you to create and manage content. You can also set up reporting options to find out more about how your site is being used and visited. Data on user access, content creation, user interaction, and other metrics can help you to modify your site to provide a better experience for users. Reports also support you in assessing how well you're meeting your goals for the site. Let's set up some basic reporting in Drupal 7, and see how to use the View and Google Analytics modules to get more report data.

The core Drupal 7 install has a Reports tab, but it has fairly limited functionality. The handful of things it does cover includes update reports and log messages. More usefully, it also summarizes the "access denied" and "page not found" errors your site reports, which can alert you to potential problems, as well as the top search phrases used, which can help you tweak site content to better suit your users.

If you want to get more site information, a couple of report modules are available, but unfortunately neither is a great fit. Report, which tracks user data and content activity, is not available (yet?) for Drupal 7; and Report Builder is unsupported and no longer under active development, so it's not a great choice for a production site.

Google Analytics and traffic tracking

Luckily, the Google Analytics Reports module for Drupal 7 provides graphical reporting of tracking data. After it connects your site to a Google account that is set up with Google Analytics, it can show you where your visitors are clicking through from and which pages and links they're visiting. It's easy to set up, but you must do a few things before you can get it running.

You'll need to install the PHP cURL library on your server (sudo apt-get install php5-curl libcurl4 libcurl4-dev on Ubuntu or Debian, or sudo yum install curl curl-devel on Red Hat and CentOS). You must also install a few Drupal modules from the Drush command shell:

drush dl google_analytics_reports 
drush dl chart 
drush dl oauth

Once that's done, go to the Modules admin page on your site and enable Google API, OAuth, and Charts. Reload the page, then click Settings next to the Google API module to configure it. You'll need to connect the module with a Google account that has Google Analytics already set up, and authorize it. Once that's all set up you can go to Reports -> Google Analytics Summary to get a report of site traffic.

As a bonus, the Charts API you've set up also gives you a Charts view of various aspects of site activity, and is also available from the Reports tab.

Custom reports with Views

If you want to set up your own custom reports, the easiest way to do this for most sites is to use the Views module. Install it (and the Advanced Help module, to access detailed Views help via the Drupal admin console) with drush:

drush dl views 
drush dl advanced_help

Enable the Views module in the Modules tab, then go to Structure -> Views to set up your views. You might want to set up a few default Views (such as Archive, Backlinks, and Recent Comments) for your site, but the only one that's of interest to us for report purposes is Tracker. Enable it, then click Edit to see what the view looks like. Note that while by default it is viewable by anyone, if you click on Access you can apply permission/role access to limit its use to only administrators, for example.

Almost certainly, though, what you want is a custom view, to report on exactly the information you're interested in. Effectively, a view is an SQL query that pulls information from the Drupal back end database. You can query by field (the SQL SELECT query) and relationships (INNER JOIN), filter the data (WHERE), and sort it (ORDER BY). You can also choose how it's displayed and to whom.

Let's try a basic example. We'll look at which tags are on the most posts, and which get the most comments, which may indicate what sort of content is most popular on your site (and whether the popular content is the sort of content that you're actually producing most of!):

Click "new view" and give the new view a name, such as "Tag Report."
In the Show box, select "Taxonomy terms," and type "Tags." Choose "Table" for display format, then click the Continue button.
You'll now see the detailed setup of the View. Click the Advanced dropdown, over on the right, to reveal the Relationships option, and add a relationship: "Taxonomy term: Content using Tags."
Under Other, click Aggregate.
Now go back to Fields, on the left side. This is where you choose the fields to show in your display table. Add three fields:
1. Taxonomy term: Name; set aggregation to "Group results together."
2. Content: Nid; set aggregation to "Count."
3. Content: Comment count; set aggregation to "Sum."
The default filter is to show only posts that are published. You can change this if you want to, but here it's unlikely to be helpful, as unpublished posts won't have comments.
Finally, you can choose to sort either on comment count or on node (page/article) count.

At the bottom of the page you'll see a preview of what the View will look like, and it is updated whenever you make a change. If you're happy with what you see, you can save it, and consider other relevant settings, such as whether to restrict access to administrators only. Reload the View page and you can see which of your tags are the most popular. (Note, however, that this setup won't show pages that have no tags at all, so be aware of that when you consider the data.)

This is only a basic example of how to work with Views and generate site information and reports. You can also change the way information is shown to you, and chop and slice it in various ways that fit your needs. You can do a bunch of other things with Views, especially when using Relationships. For more on the details of Relationships and how they work, see the Relationships page on the Views Advanced Help; basically, if you include a relationship to the base query into your View, more information pertaining to that relationship becomes available. In this way you can access other tables in addition to the base one of your query; hence the similarity with JOIN in SQL.

If the default Views options don't cut it for you, you can use the Views API to generate plugins and handlers. It's a very flexible system that's well worth taking the time to get to know.

A more colorful LibreOffice unveiled

Federico Kereki — Thu, 16 May 2013 11:00:00 GMT

In the first article in this series I pointed out some problems I and other LibreOffice users have with the standard LibreOffice color palette, and I talked about how colors are specified, both on computers in general and within LibreOffice .soc files. Let's now see how to generate a new set of colors.

The process requires finding some base colors to start with and scripting some code to build darker and lighter variations of them. Since I'm not particularly gifted (rather the contrary!) in artistic color-related work, I needed to get my colors from someplace. The World Wide Web Consortium (W3C) provides a list of color names as used in CSS (cascading style sheets). However, that list is in alphabetic order, and I'd rather group colors in families (reds, greens, blues, browns, and so on). A Wikipedia article provides a suitable table. With some cutting, pasting, and minor editing, I ended with the following text file:

REDS
IndianRed CD 5C 5C 205 92 92
LightCoral F0 80 80 240 128 128
Salmon FA 80 72 250 128 114
DarkSalmon E9 96 7A 233 150 122
LightSalmon FF A0 7A 255 160 122
Red FF 00 00 255 0 0
Crimson DC 14 3C 220 20 60
FireBrick B2 22 22 178 34 34
DarkRed 8B 00 00 139 0 0

PINKS
Pink FF C0 CB 255 192 203
LightPink FF B6 C1 255 182 193
HotPink FF 69 B4 255 105 180

...many lines snipped out...

DimGray 69 69 69 105 105 105
LightSlateGray 77 88 99 119 136 153
SlateGray 70 80 90 112 128 144
DarkSlateGray 2F 4F 4F 47 79 79
Black 00 00 00 0 0 0

I left blank lines for clarity; our script will ignore them. Lines with a single name identify color families, whose names I took from the web page. (You'll see why I kept those lines in a little bit.) Other lines have seven fields: a color name, the three hexadecimal RGB values, and their three decimal equivalents; we won't be using the hex values, but leaving them was simpler than removing them! We'll use this basic color table as a basis for our color variations, after we tweak it a bit.

The W3C also defines the 16 basic colors supported with by Windows VGA palette. I added them to the top of the colors table. Finding the decimal equivalents is easy, since the only hex numbers that appear are 00, 80, C0, and FF, which are 0, 128, 192, and 255 in base 10. I call this family "BASIC," and I opted to give its colors all-uppercase names. These colors are duplicated elsewhere in the X11 colors table, but being able to find, say, red, at two different places is no big deal.

BASIC
WHITE FF FF FF 255 255 255
SILVER C0 C0 C0 192 192 192
GRAY 80 80 80 128 128 128
BLACK 00 00 00 0 0 0
RED FF 00 00 255 0 0
MAROON 80 00 00 128 0 0
YELLOW FF FF 00 255 255 0
OLIVE 80 80 00 128 128 0
LIME 00 FF 00 0 255 0
GREEN 00 80 00 0 128 0
AQUA 00 FF FF 0 255 255
TEAL 00 80 80 0 128 128
BLUE 00 00 FF 0 0 255
NAVY 00 00 80 0 0 128
FUCHSIA FF 00 FF 255 0 255
PURPLE 80 00 80 128 0 128

Finally, after some experimenting, I noted that a couple of nice LibreOffice colors, Pale Yellow and Pale Green (actually, more like Pale Cyan) were missing. Looking at the standard LibreOffice colors list I found that Pale Yellow was (255, 255, 204), and Pale Green was (204, 255, 255). Noticing the pattern, I also tried out the (255, 204, 255) combination, and got a Pale Rose. Since I like these three colors, I added them in the Yellow, Pinks, and Blues families; 204 is CC in hexadecimal, if you are curious.

PaleYellow FF FF CC 255 255 204
PalePink FF CC FF 255 204 255
PaleCyan CC FF FF 204 255 255

The full list is in the downloadable file colors.x11. At this point, we have a satisfactory table with all the basic X11 colors plus some additions, from which we can derive as many variations as we please. Time for some scripting!

Getting color variants

Let's say we want to produce some variations of color (240,64,128), a sort of dark pink. We want to get both lighter and darker versions of it, so we have to modify its R, G and B values appropriately.

Suppose we want to create four darker variations, ending up in black. Given that the R component must go from 0 (darkest) to 240 (the original color) in four steps, the three intermediate colors we'll need will have R values of 60, 120 and 180; the G component will be 16, 32, and 48, and the B component 32, 64, and 96. Thus, the three colors we'll produce are (60, 16, 32), (120, 32, 64), and (180, 48, 96). Similarly, the three lighter colors will be (244, 112, 160), (248, 160, 192), and (252, 208, 224).

Of course, you could have any number of variations other than four, and you need not have the same number of darker and lighter versions. Also note that not all variations are really interesting; as you go to the extremes, they become either too dark or too light, so you'll probably want to pick just a few of all the available colors, most likely centered around the original one.

Now that we know how to produce variations from a given color, we can script the process with awk. Awk is an interpreted programming language, standard in Unix and Linux environments, often used for processing text files. Its name comes from the last names of its creators: Aho, Weinberger, and Kernighan. An awk program consists in a series of conditions and actions. Each input line is tested, and if a condition is satisfied, awk executes the corresponding actions, programmed in a C-like language. Conditions usually involve pattern-matching to regular expressions, which makes awk suitable for text file processing. Two special conditions, BEGIN and END, are respectively satisfied at the beginning of the program run and when no more input is available. You can execute short awk programs – one-liners – directly from the command line for simple transformations.

Producing the actual tables

Since the transformations needed to build a color palette from the color list aren't that hard, the full program turns out to be just a few lines long. Given the .soc format that we saw in the last article, what we basically require is:

At the beginning, output two header lines, the ones beginning with "<?xml..." and "<ooo:color-table..."
For each color in our list, output several "<draw:color" lines
Output a closing "</ooo:color-table>" footer line

How many variations of each color should we produce? LibreOffice shows colors eight in a row, so producing a multiple of eight colors would look good. Using another number would cause LibreOffice to split the variations over different lines – not so clear. I started by splitting the range from the extremes (black and white) to each color in six steps. After examining the resulting colors, I decided to keep four lighter variations, the original color, and three darker variations; other variations end up looking too similar to white or black. Here's the first part of the colors.awk script; you can download the complete script.

BEGIN {
  # We'll create 8 variations of each color; 4
  # lighter ones, plus the original color, plus 3
  # darker ones. We'll split the range between the
  # color and black or white in 6 steps, so there
  # will be a margin between the extremes and our
  # variations.
  VARIATIONS = 8
  INITIAL = 4
  STEPS = 6

  # The following constants come in handy
  # to shorten printf lines for publication
  DC = "draw:color"
  DN = "draw:name"
  UR = "urn:oasis:names:tc:opendocument:"
  XM = "xmlns:"

  # Print the header for the color table
  printf "<?xml version='1.0' " \
    "encoding='UTF-8'?>\n" \
    "<ooo:color-table\n" \
    XM "office='" UR XM "office:1.0' \n" \
    XM "draw='" UR XM "drawing:1.0'\n" \
    XM "xlink='http://www.w3.org/1999/xlink'\n" \
    XM "svg='http://www.w3.org/2000/svg'\n" \
    XM "ooo='http://openoffice.org/2004/office'>\n"
}

Generating color variations is simple math, but how shall we name the colors we produce? I incorporated the family name of each color to produce names such as REDS/LightCoral, PINKS/HotPink, and PURPLES/Thistle, because then by just typing the first letters of a family color, LibreOffice immediately moves to the colors in that family. To distinguish colors, I added "+1," "+2," etc., to the lighter variations, and "-1," "-2," etc., to the darker ones. I also decided not to make variations of the basic VGA colors, given that they are duplicated elsewhere in the color list.

The main loop of the awk script matches:

empty lines, which are just ignored
lines with a single field (family names), which are saved for later
lines in the BASIC family, which produce just a single color, no variations at all
lines in other families, which produce a number of variations.

In the code, $1, $2, etc., refer to the fields in the line, so $1 is a color name, and $5 to $7 its RGB components. We need a round() function of our own, because awk doesn't provide one. Here's the body of the script:

NF==1 {
  colorPrefix = $1
}

NF==7 && colorPrefix=="BASIC" {
  printf "<" DC " " DN "='BASIC/%s' " \
    DC "='#%02x%02x%02x'/>\n", \
    $1, $5, $6, $7
}

NF==7 && colorPrefix!="BASIC" {
  for (i=INITIAL; i>INITIAL-VARIATIONS; i--) {
    if (i<0) {
      red = $5 + round(i * $5 / STEPS)
      green = $6 + round(i * $6 / STEPS)
      blue = $7 + round(i * $7 / STEPS)
    } else {
      red = $5 + round(i * (255-$5) / STEPS)
      green = $6 + round(i * (255-$6) / STEPS)
      blue = $7 + round(i * (255-$7) / STEPS)
    }

    if (i==0) {
      printf "<" DC " " DN "='%s/%s' " \
        DC "='#%02x%02x%02x'/>\n", \
        colorPrefix, $1, red, green, blue
    } else {
      printf "<" DC " " DN "='%s/%s%+i' " \
        DC "='#%02x%02x%02x'/>\n", \
        colorPrefix, $1, i, red, green, blue
    }
  }
}

function round(r) {
  return int(0.5 + r);
}

We print hex digits with the '#%02x%02x%02x' format string. After this code, we just have to output the required XML footer for the color list:

END {
  # Print the footer for the color table
  print "</ooo:color-table>\n"
}

Running this script is simple: awk -f colors.awk <colors.x11 >pick.a.name.soc. I called my file colors.soc.

Copy the newly produced file to your LibreOffice directory and rename it standard.soc so LibreOffice will use it by default, after first renaming the original standard.soc file to original.soc, just in case you want to go back to it.

After you look at your new color table, you might consider going down to four colors if eight are too many. For less stark color graduations, increase the number of steps; fewer steps produce higher contrasts. You might pare down the GRAYS and WHITES families. Finally, you might want to include the "Chart" colors LibreOffice uses, with no variations (as with the BASIC color family) and no renaming.

In conclusion

Open standards mean that you'll rarely be stuck with a problem if you are willing to do some work. In this instance, we were able to extend LibreOffice with a full color library, to allow us to work with presentations, spreadsheets, and other documents using a more attractive palette.

Toward a more colorful LibreOffice

Federico Kereki — Mon, 13 May 2013 11:00:00 GMT

Recently, I was working on an Impress presentation, and I needed more colors than LibreOffice provides with its standard color picker. Specifically, I needed several light colors, but the standard color list includes only a few light blue possibilities. I could have defined my own colors manually using the Color Picker, but "mixing" colors by specifying RGB or CMYK numbers is a cumbersome, hit-or-miss process. LibreOffice also lets users load predefined color libraries, but none of the ones I found suited me. Instead, by working with web-standard colors plus a little bit of awk scripting, I managed to create my own color list, providing well-organized colors, grouped in families, with as many color variations as I wanted.

The Impress and Draw components let you load a new color list and add colors to it, which is an option if you just need one or two new colors and know how to define them. (An aside: Any color changes you make in Impress or Draw become available for all LibreOffice components.) Open Impress or Draw, go to Format -> Area -> Colors, and you'll see the table of all available colors, with their RGB and CMYK equivalents, along with commands for editing the list:

Modifying the standard color table is straightforward. To add a new color, enter a name for it, select its R, G, and B values, and click the Add button. Your new color is added at the bottom of the table. To modify an existing color, click on it, change its R, G, and B values, and click on Modify. You can select a color several different ways by clicking on the Edit button (see figure below). Finally, to delete a color from the color table, select it and click on Delete.

LibreOffice or OpenOffice?
LibreOffice was forked off OpenOffice in 2010, but the color lists you develop apply equally well to both, though you might need to make some adjustments to file names and locations.

If you were skilled enough, you could create a whole set of colors in the standard table by using these tools. LibreOffice saves its standard color table in the file standard.soc, whose location varies from distribution to distribution; openSUSE saves it in $HOME/.config/libreoffice/4-suse/user/config, but the standard place is $HOME/.libreoffice/4/user/config. You can learn the location by clicking on the Load Color List icon in the Format -> Colors dialog and taking note of the directory it shows.

(By the way, different distributions not only change the location of the file, but also its contents. For instance, Ubuntu includes three special colors – Ubuntu Orange, Ubuntu Yellow, and Ubuntu Red – and Red Hat includes Red Hat 1 through Red Hat 5.)

LibreOffice also provides alternate color lists, with different sets of colors. Click on the Load Color List icon in the Format -> Colors dialog to see a list of available *.soc files. (I assume, but couldn't verify, that "soc" stands for "set of colors.")

I tried out many tables, but despite getting more options than with the standard table, I still wasn't satisfied. I finally decided to produce my own color list, with enough variations to suit me, grouped in color families, so finding the right shade would be easier. I'll walk you through the process, but first you need to know a little about how computers represent colors.

Color models: RGB and CMYK

Since computers are digital machines, colors must be represented as numbers, and a color model is a way of doing just that. RGB is an additive model, based on red, green, and blue lights, which added together in varying intensities produce colors. Colors in the RGB model are specified by three numbers, from 0 (minimum) to 255 (maximum). Zero intensity for all lights produces no color at all (black) while maximum intensity for all colors gives white. If intensities are not all the same, you get variously colored hues. Higher component values mean lighter colors (all the way from black to white) so to get a lighter version of a RGB color, you have to increase its R, G, and B values, and for a darker version, you decrease those values.

Computers use RGB for screens, but for printed output CMYK is standard. CMYK is a subtractive color model based on four ink colors: cyan, magenta, yellow, and key (black). Using no inks at all is plain white. Each ink absorbs certain frequencies of light, so mixing inks in specific proportions can produce other colors. Mixing cyan, magenta, and yellow inks in maximum proportions (100%) should produce black, but actually gets you a dark muddy brown, and that's why CMYK uses a fourth ink, black. CMYK colors are given by four percentages, so a green color could be produced by something around (15%, 0%, 35%, 35%) meaning 15% intensity of cyan, no magenta at all, and 35% intensities of both yellow and black.

To produce darker and lighter variations, CMYK works the opposite way from RGB: Higher values (more ink) mean darker colors, and to get a lighter variation of a color (less ink) you have to lower its C, M, Y, and K values.

By default, LibreOffice uses RGB to specify screen colors, and we'll use it when we create our color table. In binary, each color is represented by a 24-bit number, so there are 2^24 possible variations, or 16,777,216 colors in all.

What's a .soc like?

So much for the way colors are represented. The next step is to determine the format for .soc files. While downloading the full source code for the LibreOffice suite and searching around for the correct description was one way to figure that out, I felt that the file format shouldn't be too hard to deduce, and when I looked at a few .soc files on my own, that proved to be the case.

Color lists are XML files that contain a <draw:color> element for each defined color, given by its RGB value in hexadecimal:

<?xml version="1.0" encoding="UTF-8"?>
<ooo:color-table xmlns:office=
"urn:oasis:names:tc:opendocument:xmlns:office:1.0"
xmlns:draw=
"urn:oasis:names:tc:opendocument:xmlns:drawing:1.0"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:ooo="http://openoffice.org/2004/office">

<draw:color
    draw:name="Black" draw:color="#000000"/>
<draw:color
    draw:name="White" draw:color="#ffffff"/>
<draw:color
    draw:name="Blue" draw:color="#000080"/>
<draw:color
    draw:name="Green" draw:color="#008000"/>

...many lines snipped out...

<draw:color
    draw:name="Chart 10" draw:color="#ff950e"/>
<draw:color
    draw:name="Chart 11" draw:color="#c5000b"/>
<draw:color
    draw:name="Chart 12" draw:color="#0084d1"/>
</ooo:color-table>

Since XML files are text files, producing a table of such elements should be easy for a script – you just need to decide how to automatically generate variations for a table of colors and then write some awk code to do the actual work.

So how are we to produce color variations? Given a base color, we can produce darker or lighter variations of it by modifying its R, G, and B values, but not haphazardly. If we make a large variation in the R component it should be accompanied by proportionally large variations in the G and B components, or we'll end up with a totally different hue.

Of course, in order to accomplish this, we need a list of starting colors, which we can modify to get variations, and then do some calculations to actually produce the .soc file. To see how, read part two of this series.

Flexible administration with Puppet's Facter and templates

Anatoliy Dimitrov — Thu, 09 May 2013 11:00:00 GMT

One of the strengths of Puppet, the IT automation application, is the flexibility it offers administrators to manage multiple servers through the use of customized templates and remote system facts, which are essentially variable values from remotely managed nodes. Thanks to this flexibility you can administer each Puppet node with instructions tailored for its specific environment – something you cannot do with simpler multiserver administration tools.

As I talk about how to administer nodes with Puppet, I'll assume you are already acquainted with the software and have a working Puppet environment configured. If not, read Multiserver administration with Puppet to get started.

How to find out information about nodes

The better you know a Puppet node, the more optimally you can administer it. For example, when you define the global Apache configuration, it makes a big difference which nodes run which different operating systems and which have what number of CPUs and RAM capacity, so you can adapt the values for Apache's daemon options to match each node's system resources.

You can get that kind of information with Facter, a Puppet library and a command-line tool (/usr/bin/facter). Facter works with hashes of key and value pairs such as memorytotal => 4 GB, which is the hash for the total RAM. By default, it gives you more than 50 facts, from the type of the operating system and its architecture to the size of the RAM.

To see all the available facts for a node, run the command /usr/bin/facter without parameters. The number of values may vary depending on nodes' software and hardware specifications. For example, if a node has more than one NIC, Facter will report a fact hash for each NIC.

If you want to get the value for a specific fact, specify this fact as the first argument of the command. For example, to find out if the node is a virtual or a bare-metal server, run /usr/bin/facter is_virtual. This capability can be handy even outside of Puppet. Since Puppet and Facter run under multiple Linux and BSD flavors, you can use Facter instead of different OS-specific commands to find out information you need.

Furthermore, Facter is extendable – you can teach it to work with your own facts. One way to do this is by exporting a Bash variable with a command such as export FACTER_custom_fact=`cat /etc/redhat-release`. In this example, custom_fact is the name of the variable and it is the result of the execution of the shell command cat /etc/redhat-release, which contains the Red Hat release information for the current version of CentOS. Once you've defined a custom fact, Facter works with it just as it does with any other fact. For more options and information, check Puppet's documentation about custom facts.

How to use templates

You can combine Puppet facts with another useful Puppet feature – templates. Puppet templates are based on the Ruby templating system (ERB) and can be used in Puppet manifests (configuration files). Templates allow Puppet to use variables inside static text files, which makes them suitable for global configuration files that are adapted locally for a node's specific environment.

By convention, you should place your Puppet templates in the subdirectory "templates" inside the directory of the module to which the template belongs. For example, for the NTP module, with which you manage the ntp service on the nodes, the path is /etc/puppet/modules/ntp. In this case you should place the templates in the directory /etc/puppet/modules/ntp/templates. Thus, when you refer to this template, you need only two pieces of information: the name of the module and the file of the template separated by a slash, such as test/example.erb.

You can instruct Puppet to use a template by using the template function. For example, you can generate one of the files (/etc/example.conf) for a module called "test" from a template by using the following code inside any Puppet manifest file:

file {'/etc/example.conf':
      content => template('test/example.conf.erb'),
     }

Templates are interpreted on each node during catalog compilation, when the node receives the Puppet master's instructions and prepares them for local execution. At that time Puppet substitutes all template variables with their corresponding local values, and the template example.conf.erb becomes the static file /etc/example.conf.

How to write templates

Template files are regular static text files with pieces of ERB code inside them. Don't worry if you are not acquainted with ERB because the basics are simple. In fact it is important to understand just two main concepts in order to start writing templates.

The first concept is how to reference variables and Facter facts. Just like any other ERB variable inside the current scope, facts are referenced by placing the at-sign (@) in front of their names. ERB variables are referenced in such a way only inside templates, and you should not confuse them with the standard variables referenced by the dollar sign ($). Understanding this basic concept is enough to get you started. You can read more about the language scope if you want to get an idea about variable scoping and isolation.

The second essential concept is how to specify ERB code inside templates with static text content. Denote the start and end of the ERB code with two different tags. The first tag is <%= code %>. The code inside this tag will be evaluated and the code block will be substituted with the result. This is useful for assigning configuration parameters from variables. For example, with this tag you can assign the ServerName parameter in Apache's configuration using the fully qualified domain name of the node as provided by Facter with the code ServerName <%= @fqdn %>. When the template is processed and the code is evaluated you will end up with something like ServerName somenode.example.org in the node's Apache configuration.

The second tag for denoting start and end of ERB code is <% code %>. Unlike the first tag, this code does not provide output by default but rather is used for executing Ruby code. Having this powerful option gives you an abundance of options limited only by your programming skills. Most frequently you can use it to evaluate conditionals or go through iterations. For example, try a conditional where you want a specific configuration option to be placed only for virtual servers:

<% if @is_virtual == "true" -%>
Here goes the specific configuration option
<% end -%>

Notice that since this is a multiline statement, instead of its regular closing tag %> you must specify an additional dash – -%>.

With the example above, the specific configuration option will be placed only if the fact is_virtual evaluates true on a node.

To see a complete example that incorporates facts and customized templates, download a module such as ntp from Puppet Forge and install it on the Puppet master with the command puppet module install puppetlabs/ntp. You can then check the file /etc/puppet/modules/ntp/manifests/init.pp to see how different templates are used depending on the remote host operating system. After that check any of the files inside /etc/puppet/modules/ntp/templates/ to see what a complete template looks like.

Knock for OpenSSH

Federico Kereki — Mon, 06 May 2013 11:00:00 GMT

If an attacker knows that your server is listening on a given port, potentially ready to allow entrance, he can target it for a security breach. An open port can be seen as a door to your system, and a password would be its key. But what would happen if the door were hidden, invisible, and you needed to use something special to make it visible again? A technique called port knocking keeps ports closed until users provide a secret knock sequence that only then allows them access to your site.

It should be obvious that you can't use port knocking for a public service, because nobody would be able to use it. However, if you want to allow access to resources on your network to only a few people in the know, port knocking can help. The usual security methods, including passwords and certificates, are still mandatory, but they enter into play only after a user has provided the appropriate sequence of "knocks."

More secure OpenSSH
OpenSSH is a well-known suite of security tools that comprises the utilities ssh, scp, sftp, and more. You should configure OpenSSH to be as safe as possible, in addition to using port knocking. Consider:

Using a port other than the default port of 22, to give script kiddies a harder time.
Disallowing root connections.
Using certificates instead of passwords.
Adding software such as DenyHosts to fight brute-force attacks.

Check the OpenSSH documentation for more information on how to use these features.

The main point behind hiding ports is that, even though ports in your firewall may be closed, you can still recognize an attempt at contacting them. Whenever the server recognizes a specific, secret sequence of attempts (an "open sesame!", if you will) it can open a port to let the outside user attempt logging in through OpenSSH. However, while the sequence is underway, the server sends no answer whatsoever to the remote user, making the listener itself undetectable. For extra safety, if a user takes too long, the port can be closed again. And after a user logs out, the port should also be closed. Finally, when the port is opened, it's only for the original user; other users won't be able to take advantage of the port if they do not provide the appropriate knock sequence.

You can make the knock sequence as complex as you like, ranging from a simple list of ports (such as "try first TCP port 8000, then TCP port 8010, and finally TCP port 8006") to use of a list of one-time-only sequences, which are discarded after each use and never again accepted. (For cryptography buffs, this is equivalent to one-time pads, the most secure encryption method.) Given that there are more than 60,000 available ports (65,535 possible ports, less the ones already assigned), there are roughly 13 million million million port-knocking combinations for a hacker to try – brute-force methods won't help here! Of course, a proficient hacker might resort to more sophisticated methods, and that's why we say that you should never use port knocking as your only protection, but rather one more hoop for hackers to jump through.

Setting up port knocking

If you are good at iptables you can implement port knocking by yourself, but there's a ready-made alternative: the knockd daemon. If you use it you don't have to change any other settings or configurations; the daemon will work with the firewall, not interfering with OpenSSH. Knockd is available for all distributions; install it on any server you wish to protect. You'll have to edit a configuration file, /etc/knockd.conf (see below), possibly forward some ports from the firewall to your server (so knockd can recognize the attempts), and finally start knockd; that's it!

[options]
  interface= eth0  
  usesyslog
  logfile= /var/log/knockd.log
  pidfile= /var/run/knockd.pid

[opencloseSSH] 
  sequence= 8000,8010,8006
  tcpflags= syn 
  seq_timeout= 15 
  cmd_timeout= 30 
  start_command= /usr/sbin/iptables -s %IP% -I input_ext 1 -p tcp --dport 22 -j ACCEPT 
  stop_command= /usr/sbin/iptables -s %IP% -D input_ext -p tcp --dport 22 -j ACCEPT

The configuration file has an options section with global parameters, and one or more additional sections, each of which defines a knocking sequence. Parameters include what interface to use (eth0 is the default), how to log events, and whether to provide a file with the PID of the knock program; if knockd were to crash, the system would be completely inaccessible from a remote location, so it's convenient to set up a cron task to use the PidFile option to check if a restart is needed.

One-time sequences
Always using the same sequence is a potential security problem. You can improve security by using one-time sequences – a list of sequences to be used only once and then erased. With this approach, even if an attacker learned a sequence, the system wouldn't be compromised. Check knockd's man page for more on this and other options.

Each sequence must start with a name. Set a somewhat short time (15 seconds here) in which the sequence must be received; if all knocks aren't given in that time, everything goes back to zero. Set another timeout for the login attempt itself (30 seconds); again, if the remote user hasn't logged in in that time, all ports are closed again. When someone successfully provides the knock sequence, knockd runs the "start command"; in our case, opening port 22 (OpenSSH) but only for the IP address from which the knock attempts came, as specified by %IP%. When the connection is closed, the "stop command" gets things back to normal, closing port 22. You'll have to determine the appropriate iptables commands for your own setup; the ones shown here work fine on my openSUSE machine, but may differ for other distributions.

The knockd program itself has a few options:

Knockd options
Parameter	Meaning
-c	What configuration file to use, instead of the standard /etc/knockd.conf
-d	Run knockd in daemon mode
-D	Provide debugging output
-h	Provide help on syntax and options
-i	Specify what interface to watch
-l	Enable DNS lookup for log entries. This isn't a good idea, since it makes your server originate DNS traffic, thus losing stealthiness.
-v	Be more verbose
-V	Show the program version

Port knocking in use

If you just close all ports, and don't run knockd, you won't be able to connect with OpenSSH:

ssh your.site.url -o ConnectTimeout=15 
ssh: connect to host your.site.url port 22: Connection timed out

Doing more work
Knockd is generally used for opening and closing ports, but you can have it listen for different knock sequences and do jobs other than working with the firewall. The start_command can be any script, so you are not limited in possible usages for port knocking!

However, if you start knockd (run either sudo /etc/init.d/knockd start or sudo knockd -d) then knock at the defined ports in order (8000, 8010, 8006) you will be able to log in. For knocking you can use the knock program, or you can make do with telnet instead, along the lines of telnet your.site.url 8000; since telnet is usually available everywhere, you should never have any problems opening a remote port:

> knock your.site.url 8000 
> knock your.site.url 8010
> knock your.site.url 8006 
> ssh your.site.url -o ConnectTimeout=15
Password:

Port knocking allows you to raise the security level of your server by totally hiding even the fact that it could accept OpenSSH connections. It's a useful tool that can make hackers work harder, which is always better. On the down side, handling the security aspects of the knocking method can be a (minor) problem, and a hacker may recognize it by analyzing packet flows, so don't forget to make the rest of your system as safe as possible.

Get more out of phpMyAdmin

Mayank Sharma — Thu, 02 May 2013 11:00:00 GMT

phpMyAdmin is popular with both individuals and enterprise users who want a graphical interface for administering MySQL databases. Although the app has an expansive list of features, most people don't use it for much beyond basic tasks such as creating new databases. Here are some features tucked beneath phpMyAdmin's folds that can make you more efficient.

Manage users, check statistics

One of the most important tasks for a MySQL server administrator is to manage which users can access the database server. Although MySQL offers a flexible permissions system, mastering it isn't a trivial task.

You can save yourself the hassle of constantly referring to the MySQL documentation by creating and managing users via phpMyAdmin. The Users tab gives you an overview of existing users and their privileges. From this section, you can add and delete users and employ a simple series of checkboxes to set and modify user privileges.

MySQL's SHOW STATUS command displays all aspects of the server along with statistics on network traffic and database queries. You can get ahold of this information and more via phpMyAdmin from the Status tab, which lists all the run-time information about the server. Under Status, the All status variables tab lets you browse through all the variables that the server tracks to assess things such as cache performance and memory usage. The Monitor tab shows all the statistics as live charts that can help you track down resource-intensive queries. The Advisor tab analyzes the different variables and provides recommendations to tune and optimize the server.

Create a Control user

In addition to these features hidden in its graphical interface, phpMyAdmin also has some advanced features. For example, you can use phpMyAdmin to bookmark complex SQL queries and execute them later, or keep a log of executed queries, and even track changes made to tables and databases. To use these additional features you need to create a database known as the phpMyAdmin configuration storage or pmadb to store the configuration information for these additional features.

But to administer this database, you first need to create a control user. The control user is a special MySQL user that's set up with limited permissions and only has read-only access to the user and db tables of the database named mysql.

To create the user, head to phpMyAdmin's SQL tab, where you can run SQL queries on your server, or you can run the following commands from the shell using the MySQL command-line client. To use "pma" as the control user and "pmapass" as this user's password, run the commands:

GRANT USAGE ON mysql.* TO 'pma'@'localhost' IDENTIFIED BY 'pmapass';
GRANT SELECT (
    Host, User, Select_priv, Insert_priv, Update_priv, Delete_priv,
    Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv,
    File_priv, Grant_priv, References_priv, Index_priv, Alter_priv,
    Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv,
    Execute_priv, Repl_slave_priv, Repl_client_priv
    ) ON mysql.user TO 'pma'@'localhost';
GRANT SELECT ON mysql.db TO 'pma'@'localhost';
GRANT SELECT ON mysql.host TO 'pma'@'localhost';
GRANT SELECT (Host, Db, User, Table_name, Table_priv, Column_priv)
    ON mysql.tables_priv TO 'pma'@'localhost';

Finally, specify the details for the control user in phpMyAdmin's configuration file, config.inc.php, available under the directory you've installed the software in. Look for the strings that read $cfg['Servers'][$i]['controluser'] and $cfg['Servers'][$i]['controlpass'] and replace them with $cfg['Servers'][$i]['controluser'] = 'pma'; and $cfg['Servers'][$i]['controlpass'] = 'pmapass';.

Enable configuration storage

After you create the control user you need to create a special database to store the settings for the phpMyAdmin configuration storage, and ensure that this database can be accessed only by the control user.

To create the database, change into the script/ or examples/ directory inside phpMyAdmin's install directory; you'll have one or the other depending on how you have installed phpMyAdmin. Look for a file called create_tables.sql. If you're running a more recent version than MySQL version 4.1.2, first run mysql < upgrade_tables_mysql_4_1_2+.sql, then import create_tables.sql into your MySQL server with the command mysql < create_tables.sql. This will create a database called "phpmyadmin" with tables for each feature.

Next, allow the control user to make changes to this database. You can either do this graphically via phpMyAdmin Users tab or enter the following SQL command:

 GRANT SELECT, INSERT, UPDATE, DELETE ON phpmyadmin.* TO 'pma'@'localhost';

To enable an advanced feature, such as the query bookmark feature, you need to specify the name of the table that controls that feature in config.inc.php. To do that, first specify the name of the configuration storage database by hunting for the string $cfg['Servers'][$i]['pmadb'] and replacing it with $cfg['Servers'][$i]['pmadb'] = 'phpmyadmin';.

Now if you wish to use the bookmark function, enter the bookmarktable name in $cfg['Servers'][$i]['bookmarktable'] such as $cfg['Servers'][$i]['bookmarktable'] = 'pma_bookmark';.

Browse the phpmyadmin database and make sure you enter the correct table name in the configuration file, then log out and log into phpMyAdmin again to bring the new configuration into effect.

Track changes

After activating the phpMyAdmin configuration storage mechanism you should enable the track changes feature, which keeps track of every phpMyAdmin-executed MySQL command. You can then create multiple versions of individual tables. You can create a version of a table, for example, before you manipulate its data, so that you have a reliable copy to revert to in case the changes corrupt the table. Each version is a snapshot of the table and logs all data manipulation statements (such as INSERT, UPDATE, and DELETE) and data definition statements (such as CREATE, DROP, and ALTER) and links these commands with the version number.

To enable the tracking mechanism, edit the config.inc.php file and replace the $cfg['Servers'][$i]['tracking'] string with $cfg['Servers'][$i]['tracking'] = 'pma_tracking';.

After the feature has been enabled you'll get an additional Tracking tab when browsing a database. When you click on the tab you can select which data definition and manipulation statements you'd like to track for that particular table.

After you've enabled tracking for a table and created a version, the tracking page will show all versions for all tables. From here you can also view a complete report for every version. You can revert any table to a specific version.

Securing phpMyAdmin

While you might have gone to great lengths to secure your MySQL server and secure your web server, it all comes to naught if you haven't secured phpMyAdmin.

The good news is that securing phpMyAdmin doesn't take much effort. phpMyAdmin supports a variety of authentication methods. If you use the simple "config" authentication type, anyone with the URL of your phpMyAdmin can log in without a password and have the same rights to your data as you! This authentication mechanism works best if your server is behind a firewall or you limit access to the data using your web server's access mechanisms, such as Apache's .htaccess.

If you use the "cookie" authentication type (which is the default authentication mechanism), only authentic MySQL users can access the phpMyAdmin interface. There's also "http" authentication, which prompts for a MySQL username and password but only uses basic HTTP access authentication, and the "signon" authentication, to integrate phpMyAdmin with single sign-on systems.

Another way to secure phpMyAdmin is to block access to it by constructing rules that allow or deny access after verifying the IP address of the machine from which the request is received. To enable this option, edit the config.inc.php configuration file. Look for the host authentication order string $cfg['Servers'][$i]['AllowDeny']['order'] = 'allow,deny';. This denies access to everyone except for the users you've allowed in the authentication rules.

Or you can change the order to $cfg['Servers'][$i]['AllowDeny']['order'] = 'deny,allow';, which tells MySQL to apply all deny rules first, followed by allow rules. If a case is not mentioned in the rules, then access will be granted. Then, to block a user from logging into phpMyAdmin, look for host authentication rules, and change it to read:

$cfg['Servers'][$i]['AllowDeny']['rules'] = array(
'deny some_user from all',
);

This will deny access to the user called some_user. You can specify a list of rules to allow only a very limited number of users (even ones from remote hosts) while denying access to the rest (including some from local hosts).

Another important aspect to securing phpMyAdmin is to deny access to the almighty root user. To prevent the root user from logging into phpMyAdmin, add this line to the configuration file: $cfg['Servers'][$i]['AllowRoot'] = FALSE;.

Backup databases

Although taking backups of databases is no easy task, you can use phpMyAdmin to easily take a snapshot of a database or individual tables and export them in more than a dozen formats, including CSV, Open Document, Excel, JSON, YAML, and PDF.

To export a database, head to the Export tab. By default the Quick export mode is toggled, which exports all the databases from the current server in the format you select from the pull-down list. If you switch to the Export tab after selecting a database, the Quick export mode allows you to select the tables you want to export (by default all are selected) in addition to the export format.

If you want more control over the export process, select the Custom export method under the Export tab. This gives you lots of choices, including the option to compress the output, export just the table structures, just the data, or both, and a lot more.

phpMyAdmin packs in a lot of functionality. I hope these tips help you exploit the tool to its full potential.

Image annotation in GIMP, Dia, and OpenOffice Draw

Colin Beckingham — Mon, 29 Apr 2013 11:00:00 GMT

The GIMP is a wonderful image editor, but it might be overkill if all you want to do is annotate an image. If you want to highlight a part of an image, so that for example the audience for your presentation can focus on a particular aspect, you'll probably find it easier and more intuitive to do that in a program such as Dia or OpenOffice Draw. Let's see how to annotate an image in all three programs.

As an example, we'll highlight an image by circling an area and creating an arrow to point at the circle. All three of the tools mentioned allow you to work in different layers, separating the edits of the background from the annotations. We'll work with an image of coats of arms that dates from 1876, and highlight the coat of arms for Siam (Thailand), which shows an elephant. You could ask an audience to count down four rows and over two columns, but if you draw a circle around the item, the audience can quickly find it immediately.

With the image loaded in the GIMP, drill down through the menu using Tools -> Selection Tools -> Ellipse Select. Click with the mouse at the top left of the Siam entry and then again at the bottom right. This creates a "marching ants" outline for the circle. Using Windows -> Dockable Dialogs -> Colors set the foreground (drawing) color to a distinctive color to make the circle stand out against the image.

Choose Edit -> Stroke Selection to change line parameters. Choose "Stroke line" to begin testing, with a width of 3 px. It will use the current foreground color and draw a line of the thickness you specify where the marching ants are. You should now have a red circle around the Siam coat of arms. Choose Select -> none to cancel the circle selection. The marching ants will stop and your circle will remain in place.

To draw an arrow, choose Tools -> Paint Tools -> Pencil or Brush. Set the properties of the drawing tool to suit your purposes. The result should be something like Figure 1.

One of the drawbacks of using the GIMP is that the result is not easily editable. You can edit the circle and arrow as a bunch of pixels separate from the underlying image if you paint them onto their own separate layer, but any advanced editing, such as moving the arrow relative to the circle or changing its color, unless each is on its own layer, is not easy.

Annotation in Dia

Now let's attempt the same thing using Dia. Load the image into the main Dia window, then select the ellipse or perfect circle object from the diagram editor and click it onto the base image. Figure 2 shows the initial result of this operation.

You can use the eight green points to adjust the position and size of the circle over the correct part of the large image. You can also click and drag as needed.

Next, right-click on the circle and choose Properties from the dialog box. Here you can change the color of the circle and its line width. Change "Draw background" to NO; this controls opacity, and the background it refers to is the white background of the circle object. Saying no allows the coat of arms image to show through. In Dia, opacity of objects is either 0 percent or 100; if you need more varied opacity then you should choose a different editor.

At this point you have two objects, the circle and the base image, both sitting on one layer. Since they are both on the same layer, Dia may try to associate them. You may not notice an effect now, but when you add an arrow, Dia will probably try to associate the arrow with the base image. If you see some strange behavior where you cannot place the end of an arrow where you want it, you should remove your objects from the layer, add a new layer through the Layer menu option, and paste the objects onto the new layer. Dia now will not try to associate the annotations with the underlying image if they are on different layers, but will associate them with each other, which is what you want.

To add an arrow, click the line object from the diagram editor palette and click again somewhere in the image outside the circle. This produces a line with an arrowhead. Click on the ends to adjust its position. Dragging the arrow end into the circle lights up the circle to show that the program recognizes that the circle and the arrow should be associated. Release the mouse button and you'll find that the arrow is now connected to the circle.

The two objects can now be treated as a group or separately. If you drag or resize the circle, the arrow will follow it. You can edit each object by selecting it and right-clicking to produce the dialog box.

Figure 3 shows a detail view of the final output. Note that at high resolution there is a little bleeding of the lines into the base image. This would not be noticeable at a distance; the thinner the lines, the worse the bleeding.

If you export this set of objects as an image, Dia will not preserve the object structure. Make sure to save the structure as a Dia file if you need to come back and edit the objects separately.

Annotation in OpenOffice Draw

OpenOffice Draw, which is just as intuitive as Dia, uses a similar process. You can easily select objects once they have been created, and change their properties at any time. Draw does not have the same library of technical shapes Dia comes with, but it is quite capable with more general shapes.

Also, Draw does not associate or connect objects; that is up to the drawer. In some cases it is more efficient and effective to allow a program such as Dia do this, since the result will usually look more professional, and it makes subsequent editing easier.

In Draw, add a circle in the same way as in Dia, by selecting one of the standard objects. You can add an arrow either by placing a plain line and then editing the line to show an arrow at one end, or select an object from the Arrow tool box, which you can make viewable from the View -> Toolbars menu item.

Conclusion

When you need to annotate images, clearly you have a choice of tools from which you can pick. The GIMP is the way to go if you want to learn just one application that provides all the bells and whistles required to edit your base image as well as add annotation. However, the GIMP does not treat annotations as objects that can be linked together, so it is not easy to edit annotations once they have been created, but you can build them in layers and delete and recreate layers as needed. Also, the GIMP's menu structure is quite complicated; remembering the steps and which menu is where is a challenge unless you are a regular user of the program.

Dia is a good choice if you need to make basic adjustments to an image and need access to a library of technical object shapes to add to your exposition. Most of Dia's commands and buttons are readily visible.

OpenOffice Draw is an excellent midpoint between the GIMP and Dia. It creates annotations as objects as in Dia, but lacks Dia's library of technical object shapes. It does however have a large gallery of fills, which, while not relevant in this case since the background is part of the presentation, would be helpful in a more general context.

Solr, Drupal 7, and faceted search

Juliet Kemp — Thu, 25 Apr 2013 11:00:00 GMT

It's easy to get started with the search server Apache Solr and the popular CMS platform Drupal, as described in our previous tutorial on setting up Solr 4.2 with Drupal 7. Straight out of the box Solr handles basic text searching, but you can increase its power by adding faceted search – the ability to filter on specific facets or aspects of the data on a site. For example, on a Drupal site you might want to be able to filter your searching by author, date, or tags. Solr's faceted search allows users to combine this type of filtering with text searching to find the information they're after faster. Read on for the lowdown on setting up and configuring basic faceted search using Solr 4.2 and Drupal 7, running on Apache on Linux.

The Facet API module for Solr on Drupal will get you set up with faceted search. It doesn't come by default with the Solr Drupal module; you have to install it independently. You can download it from the website and install it manually into sites/all/modules in your Drupal install, or install it using Drush, a command-line interface to Drupal. Drush is worth installing if you have more than one or two modules on your site, as it makes it quicker to work with them from the command line. To get Facet API running, you also need to install its dependency, the Chaos Tools module. From the command line (as root/sudo) type:

drush dl facetapi 
drush dl ctools

Once the modules are installed, go to the Drupal admin console and click the Modules tab (http://localhost/?q=admin/modules). Select Chaos Tools in the Chaos Tools Suite section, and Current Search Blocks and Facet API in the Search section, then click Save Configuration.

To configure the facets you want users to be able to search on, go to the Apachesolr module configuration on Drupal (http://localhost/?q=admin/config/search/apachesolr/settings) and click the Facets link next to localhost. You'll see the list of default facet blocks that you can enable: author, content type, language, post date, tags, and update date. Enable only the ones that you wish to have available to users, because facet indexing requires system resources, and indexing facets that you won't use may adversely affect system performance.

Once you've enabled the facets, if you reload the http://localhost/?q=admin/config/search/apachesolr/settings page, you should see a configuration drop-down link next to each enabled facet:

Configure display controls how the facet links show up on the search page (as links or links with checkboxes, which can help users keep track of what they're searching on), how they are sorted, and whether you use AND or OR when applying this facet.
Configure dependencies controls under what circumstances the facet is shown. You can hide facets on pages or articles, or when specific users are logged in.
Configure filters controls whether certain items are shown, and how many levels are displayed for multilevel facets such as tags.
Export configuration shows you the code associated with the current configuration so you can save it.

Set these up to meet the needs of your specific site and save the configuration. You can then go to the search page and see how your settings appear. At this point they'll only show up on the search page, so you can filter your search results once they come back. Try them out to make sure everything is working as expected.

Showing facets outside of search pages

What if you want facets and facet search to show on non-search pages? For example, many sites show a tag cloud or dates of articles in a sidebar. These are both examples of faceted search, and you can use Facet API and Solr to set them up for your Drupal site.

To show facets on all pages, go back to the Facets settings admin page (http://localhost/?q=admin/config/search/apachesolr/settings/solr/facets) and click "Show facets on non-search pages." You'll then see a box that allows you to enter the paths of the pages on which you want facets to be displayed. Note, however, that you cannot just use * to display facets on all paths; this leads to a bug where faceted searches will fail with no results. Instead, you need to list specific paths. A good starting point is:

node
node/* 
user/*

This will cover the default front page (node), all pages and articles (node/*), and all user pages (user/*). You may of course wish to add other paths for your site; just don't add search/* or you'll run into the same bug!

Once you've enabled the facets, you need to decide where to display them on the page; until you do they won't be visible. Go to the Blocks admin page (http://localhost/?q=admin/structure/block/) and scroll down to the Disabled section, then click Configure for each facet you wish to display, and choose where you wish to display each facet. You could, for example, put all your enabled facet searches in the second sidebar. Save the configuration, then check your site to see the new facets in the place you've chosen to display them.

Other facet-related modules

A lot of sites these days show tags as a cloud rather than as a list, where the font size of each tag increases along with how often it appears. The facetapi_tagcloud Drupal module allows you to display your Solr facets this way. Install it manually or with Drush, and enable it from the Modules admin page. Then go to the Facet configuration page, click on tag (http://localhost/?q=admin/config/search/facetapi/apachesolr%40solr/block/im_field_tags/edit), and choose "Tagcloud links" as the display widget. Note that this module is in beta, so use with care.

Another interesting module, facetapi_multiselect, which gives you a multi-select view for faceted search, is unfortunately not yet available as a recommended release for Drupal 7. You could however download the development release and try that at your own risk.

You might also consider building your own facets if the built-in ones don't meet the needs of your site. Unfortunately the module apachesolr_facetbuilder, which allows you to build facets without writing code, isn't available for Drupal 7 (and doesn't appear as if it's being updated) so you'll have to dig into the code to do this. The built-in facets, though, are good enough to meet the needs of many Drupal sites, so take the time to get them set up!

Using FreeNAS' new full disk encryption for ZFS

Gary Sims — Mon, 22 Apr 2013 11:00:00 GMT

Last month's release of FreeNAS 8.3.1 adds new functionality that allows system administrators of the open source-based network attached storage solution to encrypt entire disks while using ZFS. ZFS has been the primary filesystem for FreeNAS since FreeNAS 8, and has supplanted FreeBSD's UFS as the project's focus. The new security functionality applies only to ZFS and is the first time that FreeNAS has supported encryption.

Why use disk encryption? Modern operating systems such as FreeBSD, which is at the heart of FreeNAS, are designed to protect against unauthorized data access, and if you employ a good security policy, it should be hard for outside attackers to gain access to sensitive data stored under FreeBSD. If an attacker has physical access to a server, however, he can remove a disk from a server, take it away, and examine it at will – but not if it's encrypted.

There is also the problem of disposing of obsolete or failed disks that contain sensitive data. Unless you physically break it (which some companies do) there is always a risk that someone could extract data from a discarded device – unless it's encrypted.

With encryption enabled, all the data written to a disk is inaccessible without the right credentials. The newest security features in the latest version of FreeNAS support full-disk encryption, meaning that the encryption is done at the disk level, below the ZFS filesystem. FreeNAS employs GELI, a block device-layer disk encryption subsystem written for FreeBSD, which makes it more efficient than any add-on encryption tools would be.

Considerations

Before you use FreeNAS' new disk encryption facilities, you should understand a few important things:

The disk encryption in FreeNAS 8.3.1 and later is not the same as the disk encryption designed by Oracle in ZFS v30. This means that you cannot import encrypted volumes from systems running Oracle's ZFS into FreeNAS.
If you lose or forget the master key for your encrypted disk, all the data on the disk will be irretrievable.
Encrypting data in the fly can be costly in terms of performance. Intel, in its i5 and i7 CPUs, and AMD, in its Bulldozer and Piledriver cores, have added a new AES encryption instruction set to some of their processors. Without these AES instructions you will suffer an approximate 20 percent performance hit – more if you have multiple disks.
You cannot convert an existing unencrypted volume into an encrypted one. To encrypt the data from an existing volume, you must back it up and restore it to a drive that has encryption enabled.

Getting started

To use disk encryption you need to be running at least FreeNAS 8.3.1-RELEASE-p2 – that is, the second patched version of the initial 8.3.1 release, which had a few problems. The FreeNAS site has good documentation, including a step-by-step installation guide.

After installation you administer FreeNAS via a web interface. To create an encrypted volume, drill down through the tree menu on the left side of the web UI Storage -> Volumes -> Volume Manager. Select the disks you want to use to form the pool, select ZFS, and ensure that "Enable full disk encryption" is checked. There is also the option to initialize the disk with random data, which is useful if the disk was previously used.

If you are using more than one disk, select the type of volume they should comprise: stripe, mirror, or RAID-Z:

Striping doesn't offer any redundancy but does improved performance. It is the equivalent of RAID-0, but it isn't generally recommended, as the failure of a single disk would result in the loss of all the data on the volume.
Mirroring is the same as RAID-1. All the data is duplicated exactly onto one or more drives.
RAID-Z needs at least three drives and offers the best trade-off between performance and redundancy. Data is shared among all the drives, and the volume will survive a disk failure.

Once you create a volume you must set a passphrase for the master key. Click the Create Passphrase button (the first icon with a key on it) and enter the passphrase. It can contain spaces, and can be a sentence or phrase and not necessarily just a word.

It is prudent to create a recovery key so that in case you forget the passphrase you can use the associated recovery key instead. When you click the "Add recovery key" icon FreeNAS generates a recovery key and pushes the key file to you via the browser.

You should also download a backup copy of the master key by clicking the Download Key icon. The GELI configuration is separate from the FreeNAS configuration, and it is best to have a backup copy in case the GELI key information is ever lost or destroyed. If, for example, you ever need to reinstall FreeNAS, due perhaps to a failure of the operating system disk, you can use the backup master key to gain access to the existing disks. Without the backup, you'd lose the encryption key information if the operating system disk failed.

Whenever you reboot the FreeNAS server you will need to enter the passphrase before you can access the encrypted volume. Click Storage -> Volumes -> View Volumes, click the unlock icon (the only key-type icon shown), enter the passphrase, and ensure that the relevant network access services (such as CIFS and NFS) are marked for restart so that the volume becomes available to those services. If you forget the passphrase you can use the recovery key by uploading it instead of entering the passphrase. In such situations you should reset the passphrase by clicking on the Change Passphrase icon, in place of the Create Passphrase icon, and regenerating the recovery key.

Here's something to watch out for: Locked volumes are available to the volume manager for use in new volumes. This lets you reuse previously encrypted disks when necessary, but it also mean that these volumes could be destroyed by mistake. It can seem a little disconcerting to see, considering that the volumes contain valuable data! Once a volume is unlocked its disks are no longer available to the volume manager, so you should unlock the volumes as soon as you can.

It should almost go without saying that you must protect the passphrase, master key, and recovery key. If an attacker is able to physically access the disks and also has in his possession the passphrase or the recovery key, your encryption is rendered useless.

Replacing a failed encrypted disk

The ZFS mirror and RAID-Z options provide disk redundancy and let the filesystem survive the failure of a single hard disk in a volume. If a hard disk fails, click the Volume status icon on the View Volumes page, then take the disk offline by clicking Offline for the faulty disk. Physically replace the failed disk; depending on the hot-swap capabilities of your hardware, you may need to shut down FreeNAS to do this. On the View Volumes page click Replace next to the disk that has been changed and select the new disk. Enter the encryption passphrase for the volume – without the passphrase you cannot replace the disk.

You cannot use the recovery key when you're replacing a disk. If you lose the passphrase and a disk fails, the only option is to use the recovery key to unlock the degraded volume, then change the passphrase before replacing the disk.

Extending an encrypted volume

One of the many advantages of the ZFS filesystem is that it lets you extend volumes by simply adding disks. Due to the nature of ZFS you can't increase the number of disks in an existing set, but instead you need to add more disks to the volume itself. For example, the way to expand a volume made up of three disks in a RAID-Z configuration is to add another three disks also working in RAID-Z. The result will be a volume with two sets of RAID-Z disks. The extra disks don't alter the original RAID-Z disks but rather expand the volume. It isn't possible to add a forth disk to the existing three.

The advantage of this approach is that you can add disks of different sizes than the original disks to a volume. The disadvantage is that you cannot mix different configurations. In other words, you can expand a volume using RAID-Z only by adding another set of disks in a RAID-Z configuration, and so on.

The same behavior applies when adding disks to an encrypted pool, but with the added complication that adding disks removes the existing encryption; you have to generate the passphrase and recovery key afresh once the new disks have been added.

To add new disks to a volume, use the Volume Manager page. Select new member hard disks, matching the existing configuration – stripe, mirror, or RAID-Z array – and choose the volume to extend. The user interface will automatically select the filesystem type and disable the volume name field (since the volume already has a name and is just being extended). Click Extend Volume to get the FreeNAS volume manager to add the new hard disk (or disks) into the existing pool.

Once you've added the disks, click on the Create passphrase icon and enter a passphrase to reestablish encryption, download the master key as a backup, and create a recovery key.

Conclusion

Encryption can guard against situations where an attacker physically gets hold of a hard disk. When used with a CPU that supports the AES instruction set encryption costs just a negligible performance impact. The FreeNAS implementation, although new, uses the proven FreeBSD disk encryption subsystem and a web-based UI that grants easy access to its functionality. But remember, you must guard the keys because, just like in the real world, if a thief has keys then he has full access.

Create distributed storage with Gluster

Anatoliy Dimitrov — Thu, 18 Apr 2013 11:00:00 GMT

If you're looking for Linux-based, hardware-agnostic storage software, check out Gluster, an open source project for creating a distributed filesystem. It provides fast performance, high availability, and horizontal scalability by spreading storage volumes over redundant cluster nodes. Here's how you can build a Gluster distributed storage system yourself.

Gluster's storage is build up by what it calls bricks, which are exported directories allocated on the cluster nodes. Cluster nodes are united in trusted pools that together provide storage services and share disk resources.

Gluster doesn't require any special hardware. You need at least two servers to act as nodes in the filesystem, to provide redundancy and make use of Gluster's essential features for performance and recovery. Each node must have at least 1GB of RAM; having more RAM may allow you to provide in-memory caching for the storage, thus speeding the I/O operations. Each node needs at least 1Gbps network connectivity and at least two disks – one for the operating system and one more for the storage. The higher the I/O speed of the media, of course, the better storage performance. Gluster runs best on CentOS but is also supported on other 64-bit Linux distributions. No 32-bit distributions are supported.

Install Gluster from its own yum repository, which always provides the latest Gluster version. First, download Gluster's repository with the command wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo. This command places the repo file in the directory /etc/yum.repos.d/, thus enabling the repo.

Now you can install the packages glusterfs-fuse, the userspace program for managing GlusterFS, and glusterfs-server, the server daemon, with the command yum install glusterfs{-fuse,-server}.

Finally, start Gluster's daemon for the first time with the command service glusterd start. To make it start and stop automatically with the system, run the command chkconfig glusterd on.

Follow the same process on each Gluster server.

Configure the network and the firewall

For optimal performance and security you should run the Gluster cluster inside a private and secured network. In such a network you can disable the default CentOS firewall because the network should be accessible only by trusted clients. To do this, use the command chkconfig iptables off && service iptables stop.

If you cannot provide a private network for the storage cluster, you can use iptables to allow only predefined clients to access Gluster's services. To do so, allow the following with iptables:

RPC incoming connectivity – Gluster's processes communicate using RPC, so RCP's port, TCP and UDP port 111, has to be allowed for incoming connections. Use the commands iptables -I INPUT -m state --state NEW -m tcp -p tcp -s X.X.X.X/24 --dport 111 -j ACCEPT; iptables -I INPUT -m state --state NEW -m udp -p udp -s X.X.X.X/24 --dport 111 -j ACCEPT;.
Gluster's own services – For Gluster to access its bricks on each cluster node, you have to allow TCP ports 24007, 24008, and 24009, plus an additional number of ports in sequence equivalent to the number of bricks across all volumes. Thus if you have two bricks in the cluster, you have to allow ports from 24007 to 24011 (24009 plus 2). The command for this is iptables -A INPUT -m state --state NEW -m tcp -p tcp -s X.X.X.X/24 --dport 24007:24011 -j ACCEPT.
Other access – Allow other remote connections if you want to use NFS, iSCSI, or other connectivity. You don't have to allow other ports if you plan to mount volumes with the native GlusterFS filesystem.

The above rules allow all cluster nodes and clients from the X.X.X.X C-class network to communicate. You can also apply these rules on a per-host basis by using Y.Y.Y.Y for the IP address of each host instead of X.X.X.X/24 for the network.

Don't forget to save the iptables configuration once you've added the new rules by using the command service iptables save.

Prepare disks for Gluster storage

To comply with best practices, use one disk for the operating system and dedicate one or more others for Gluster's storage only. Also, format the future Gluster storage disk with the XFS filesystem, which is fast, scalable, and reliable for distributed storage.

So far you've installed Gluster on one disk and have just attached another for Gluster storage. The new disk is not formatted and appears under the name /dev/sdb. You can acknowledge the disk with the command fdisk -l, which should show the new disk with valid partitions.

To prepare the disk, first use fdisk to create a primary partition on the new disk. Run the command fdisk /dev/sdb and select "n" for new partition, "p" for primary, "1" for first. Then write the new partition table to the disk by selecting "w." You should now have a new primary partition under the name /dev/sdb1.

Next, format the new partition with the XFS filesystem with the command mkfs.xfs /dev/sdb1. Create a new directory to be used as a mount point for the /dev/sdb1 partition, such as /export/brick1. Brick1 is a good name because this partition will be used for the first brick, which is to say the first exported directory found on the first node.

Finally, make sure the new partition /export/brick1 is automatically mounted during system boot time by adding a new row to /etc/fstab: /dev/sdb1 /export/brick1 xfs defaults 1 2.

Set trusted storage pools

Gluster unites cluster nodes into a trusted pool, and all nodes inside this pool share disk resources and offer storage services. Acting as one logical entity, a trusted pool provides redundancy, better storage performance, and scalability.

To manage a new trusted pool, and in fact manage all parts of Gluster, use the Gluster console manager at /usr/sbin/gluster. This command-line tool accepts arguments for options; there is no GUI.

A storage pool is created automatically when you install and start Gluster on the first cluster node. From the first node you can add the rest of the nodes after you install and start Gluster on them.

For example, suppose you installed Gluster on a node with the IP address 10.1.1.1 and you want to add to the pool a node with IP 10.1.1.2. To accomplish this use the Gluster console manager with the arguments peer probe [host], as in /usr/sbin/gluster peer probe 10.1.1.2.

Similarly, you can remove servers from the storage pool with the command /usr/sbin/gluster peer detach [host].

After adding servers to or removing them from the trusted storage pool, you should check the pool's status. Use the command /usr/sbin/gluster peer status to confirm you have the correct number of peers and their statuses.

How to manage Gluster volumes

A Gluster volume is a logical entity composed of bricks, which are exported directories from servers inside the trusted pool. One pool can support numerous volumes. You can start managing Gluster volumes as soon as you have a trusted storage pool set up.

Gluster supports many advanced options for its volumes. For instance, you can create a distributed, striped, and replicated volume. Such a volume performs well, first because it's distributed – reads and writes happen on multiple servers. Second, the fact that it is striped ensures that large files are striped across multiple bricks and servers and thus accessed faster. High availability is ensured by the fact that the volume and its data is replicated and redundant, so if one node fails another covers for it and there is no interruption in service.

To create a distributed, striped, and replicated volume use the command /usr/sbin/gluster volume create volumename [stripe count] [replica count] host:/brickname. An example of the command would look like: /usr/sbin/gluster volume create testvolume stripe 2 replica 2 10.1.1.1:/export/brick1 10.1.1.2:/export/brick2. This creates a volume called "testvolume" striped across two bricks, replicated twice. The volume consists of two nodes and their bricks – 10.1.1.1:/export/brick1 and 10.1.1.2:/export/brick2.

Once you have a Gluster volume you can easily and seamlessly change its size without interrupting the storage operations. For example, if you want to increase the size of your volume you can add more cluster nodes with more bricks attached to them. First, add each additional node with the command /usr/sbin/gluster peer probe host. Then use the command /usr/sbin/gluster volume add-brick volumename host:/brickname. Finally, rebalance and redistribute the volume evenly across all the bricks with the command /usr/sbin/gluster volume rebalance volumename start.

You need to keep a few concepts in mind for proper volume management and configuration. First, you should use only one brick per server in a given volume so that in case of a crash it can be covered for by its replicating peer. Second, the number of bricks should be a multiple of the replica number – that is, how many times data should be replicated – so that valid replication can be established. For example, if you want data to be replicated twice, you should build your cluster from any number of bricks that's a multiple of two. Last, the number of stripes should be equal to the number of bricks so that you can gain performance by reading from and writing to multiple locations simultaneously.

Connect a client to the storage

The best way to connect to Gluster storage is through GlusterFS, which allows clients to take advantage of Gluster's clustering and high reliability features. GlusterFS offers better performance and reliability than non-native communication methods such as NFS or iSCSI, and GlusterFS allows clients to communicate simultaneously with multiple cluster nodes.

GlusterFS is based on filesystem in userspace (FUSE). Prior to installing GlusterFS you have to install the fuse and fuse-libs packages. In CentOS these prerequisite packages can be installed through the official repository with the command yum install fuse fuse-libs. You must run this installation only on the clients from which you will connect to the storage; FUSE is automatically installed with the Gluster server software.

Restart the server after installing the fuse package in order to load the fuse kernel module, or if you want to postpone the restart you can manually load the module using the command modprobe fuse.

Next, to install GlusterFS, you have to add Gluster's repository just as you added it on the cluster nodes, with the command wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo, and install glusterfs-fuse with the command yum install glusterfs-fuse.

Once you've installed all the necessary software you can mount a GlusterFS volume. You can use the /bin/mount command with "glusterfs" as parameter for the filesystem (-t option) and a few other GlusterFS-specific parameters; for example, mount -t glusterfs -o backupvolfile-server=10.1.1.2 10.1.1.1:/testvolume /media/gluster.

This example mounts the volume "testvolume" from the Gluster node 10.1.1.1. The option backupvolfile-server instructs the client to work with the replicating Gluster node 10.1.1.2 in case 10.1.1.1 fails.

Once the volume is mounted, Linux treats it just as any other disk resource. Multiple clients can mount the same volume and work with it at the same time.

Gluster is a powerful distributed storage system that helps meet the growing need for better Linux-native storage. It's easy to implement and work with, limited only by the system resources you can dedicate to it, and doesn't require you to buy any expensive vendor-specific hardware.

How to set up Solr 4.2 on Drupal 7 with Apache

Juliet Kemp — Mon, 15 Apr 2013 11:00:00 GMT

Solr is an open source search server based on Apache Lucene. Lucene provides Java-based indexing and a search library, and Solr extends it to provide a variety of APIs and search functionality, including faceted search and hit highlighting, and handles Word and PDF document searching. It also provides caching and replication, making it scalable, robust, and very fast.

Happily, Solr also plays nicely with Drupal, the popular CMS platform. If you want fast and effective search on your Drupal site, installing Solr is a straightforward way of getting it quickly. Until this month, the Apachesolr Drupal module didn't support the current Solr 4.x schemas, but as of the very latest version of the Apachesolr module, 7.x-1.2, you can now set up Solr 4.x on your Drupal 7 site. This tutorial assumes that you're running Drupal 7.22 (the most up-to-date version) under Apache on a Linux box.

Installing Apachesolr

You need two components to run Solr with Drupal: the Apachesolr Drupal module, and the Solr server itself. The server runs independently of your Drupal/Apache installation when it indexes the data. The module links that independent Solr server with your Drupal install, passing data into Solr to index, and then enabling Drupal to serve up the search results.

You need to start by installing the Drupal Apachesolr module, as it has some config options you need to set up the Solr server. You can download the most recent version of the module from the bottom of the Drupal Apachesolr web page. Important note: As of the version of 13 April 2013, this module should work with 4.x Solr; older versions only work with 3.x Solr. Unpack the tarball and move the unpacked apachesolr directory to your modules directory, which might be sites/all/modules or /sites/all/contrib, depending on your setup.

Now log into your Drupal install from the web dashboard as admin, and go to the Modules tab. You may need to manually run the "check for updates" task, but after that, you should see the Apache Solr modules available at the bottom of the modules list. Click the boxes to enable them.

Installing Solr

Next you'll need to install the Solr server itself. Go to http://www.apache.org/dyn/closer.cgi/lucene/solr/ to find your closest mirror, and download the most recent release (4.2.1 at time of writing). Unpack it somewhere outside your web directory; /opt/solr or /usr/local/solr are good choices.

The next step is to configure Solr to talk to your Drupal site. There's a decent example Solr setup in /opt/solr-4.2.1/example/. Copy that across to your own Solr directory before you start making changes:

cp -rf example/ my-drupal-solr/ cd my-drupal-solr/

By default, Solr looks for its configuration in solr/solr.xml. If that file is missing, it assumes the config is kept instead in solr/collection1/. We will put our Drupal config (taken from the Apachesolr Drupal module) in that collection1/ directory, and move aside the solr.xml config:

 
cd solr/ 
mv solr.xml solr.bak 
cp /var/www/sites/all/modules/apachesolr/solr-conf/solr-4.x/* collection1/conf/

Now start your Solr server with java -jar start.jar, and go to http://localhost:8983/solr to check the Solr admin console and make sure that your Solr server is running.

Indexing and checking Solr with Drupal, and other setup notes

To confirm that Solr is talking to Drupal, and that the indexing is working correctly, go to the Drupal console, and go to Configuration -> Apache Solr search. Here you can submit all pages for indexing, and then run the indexing immediately rather than waiting for the next cron job. It will take a couple of minutes for the indexing to complete. Once it does, go to http://localhost/?q=search/site and enter a term in the search box. You should get a successful search result. If you don't, check that you've got the Apachesolr configuration files in the right place for your Solr install to pick them up, as described in the section above.

Right now if you try searching using the Content rather than the Node search tab, or from the default Drupal front page search box, you won't get any results. To fix this, go back to the Apache Solr module configuration page and click the Pages/Blocks tab. You'll see Core Search listed, and a Taxonomy Search. Click Clone next to Core Search, then edit the cloned search. Rename it Content, edit the Description and Title fields appropriately, and change the Path field to search/node. Save this and try the Content search again, and all should work fine.

You may also wish to set Apache Solr as the default search from the Search module configuration page.

Run Solr automatically

At this point you are running Solr manually, which you probably don't want to do on a production site. To get it to start on startup, you need a startup script. Here's a basic one that should do the trick. Save it as /etc/init.d/solr and make it executable by root:

#!/bin/sh

# Prerequisites:
# 1. Check SOLR_DIR value is correct
# 2. daemon needs to be installed
# 3. Script needs to be executed by root

# This script will launch Solr in a mode that will automatically respawn if it
# crashes. Output will be sent to $LOG. A pid file will be 
# created in the standard location.

NAME=solr 
SOLR_DIR=http://www.openlogic.com/opt/solr-4.2.1/my-solr-drupal 
COMMAND="java -jar start.jar" 
LOG=http://www.openlogic.com/var/log/solr/solr.log

start () {
    echo -n "Starting solr..."

    # start daemon
    daemon --chdir=$SOLR_DIR --command "$COMMAND" --respawn --output=$LOG --name=$NAME --verbose

    RETVAL=$?
    if [ $RETVAL = 0 ]
    then
        echo "done."
    else
        echo "failed. See error code for more information."
    fi
    return $RETVAL
}

stop () {
    # stop daemon
    echo -n "Stopping $NAME..."

    daemon --stop --name=$NAME  --verbose
    RETVAL=$?

    if [ $RETVAL = 0 ]
    then
        echo "done."
    else
        echo "failed. See error code for more information."
    fi
    return $RETVAL
}


restart () {
    daemon --restart --name=$NAME  --verbose
}


status () {
    # report on the status of the daemon
    daemon --running --verbose --name=$NAME
    return $?
}


case "$1" in
    start)
        start
    ;;
    status)
        status
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    *)
        echo $"Usage: $NAME {start|status|stop|restart}"
        exit 3
    ;; esac

exit $RETVAL

You can run it manually with /etc/init.d/solr start to check that it's working correctly. Once it is, add it to your standard startup scripts with update-rc.d solr defaults on Ubuntu, or if you're on CentOS or another Red Hat-based distro, add the line

# chkconfig: 2345 64 36

to the script (note that it's a comment!), then run chkconfig --add solr. Now when you reboot, Solr should start automatically.

This is a very basic Solr setup, but it's enough to get you started. If you want to utilize the full power of Solr you can look into faceted search and filtering, further text analysis configuration, or exciting options like geospatial search.

Caching web service results can enhance Apache application performance

Federico Kereki — Thu, 11 Apr 2013 11:00:00 GMT

Since in most web services, reads outnumber updates by far, you can enhance the performance of a RESTful server by implementing a local cache, so repeated requests can be fed with local data and don't actually require more expensive processing. Caches are usually associated with static content such as images or videos that don't change over time, but in this article we'll see how to configure Apache to work as a reverse proxy, caching dynamic requests to reduce the load on your server.

Before we start, you may wonder: Should you ever cache your service results? Won't stale data negatively affect users? The answer depends upon your application. For example, in an online retail website, you might have a service return the top 10 bestsellers over the last few hours; is it really worthwhile to recalculate those results for every request? Caching the information will improve responsiveness for your users, and it's quite likely that sending the same results for a certain period of time won't negatively affect the users. Along the same lines, book recommendations for users could be recalculated only once a day or a week. There's no single, simple answer for every application. You should consider each service call and think about whether caching would detract from the value of the information returned.

In this article we'll keep working with the services we defined in our previous Apache and RESTful Web Services articles, which provide REST access to a world database, with information about countries, their regions, and their cities. As most of this data will almost never change, caching is a good solution.

To begin, configure Apache to enable caching. You have to enable the mod_cache module, plus at least one storage module; we'll go with mod_cache_disk, which stores the cache data in the filesystem, but check the Apache Caching Guide for more options. You'll also have to set some specific headers so data will be cached, and that requires the mod_headers module. To accomplish these tasks on my openSUSE system, I had to edit the /etc/sysconfig/apache2 file and include the new options for the APACHE_MODULES variable, as in APACHE_MODULES="actions ... rewrite cache disk_cache headers". You might have to edit a different file if you use another distribution.

You must also turn caching on and specify what URLs will get cached and where cached files should be stored. In my case, I edited /etc/apache2/default-server.conf to include the lines:

CacheEnable disk /rest_rewrite
CacheEnable disk /rest_alias
CacheRoot   /var/cache/apache2

What gets cached?
Section 13 of the HTTP definition (RFC 2616) specifies in detail the conditions that allow content to be cached, and Apache follows those rules. Basically, only GET requests may be cached, because they shouldn't create any server-side effects; our services implementation satisfies this condition. (HEAD requests can also be cached, but we didn't include any in our example.) If GET requests include parameters, the server must provide an expiration time, or the request will be considered non-cacheable. PUT, POST, and DELETE requests, which are expected to produce server changes, won't ever be cached. There are some extra rules – for example, only requests that return HTTP status codes 200, 203, 300, 301, or 410 may be cached. You can see the complete rules in the "What can be cached?" section of the Apache caching guide.

Each CacheEnable line specifies a URL path to cache; everything beneath becomes a candidate for caching. In our original example we provided two paths for each service, so I've specified two URLs that should be cached. You must refer to the original URL without considering the effects of URL rewriting. The CacheRoot line specifies the caching directory.

You must also specify the default expiration time for files, or some requests won't be cached even if it's otherwise possible; see "What gets cached?" You can use the FilesMatch directive to be more selective. I arbitrarily decided that for our example only countries requests would get cached, and would expire 10 seconds afterward. (This is obviously a ridiculously quick expiration time, but very good for impatient article writers who don't want to wait when testing their code.) You must also provide a Cache-Control header. You could provide it through code, but it's better not to modify the services themselves, and instead set all the caching configuration without any code changes, at the Apache level:

#
# By default everything expires right now.
# Arbitrarily, only the "countries" service is cached.
#
ExpiresActive  On
ExpiresDefault A0
<FilesMatch "countries">
  ExpiresDefault A10
  Header append Cache-Control "public"
</FilesMatch>

After you make these configuration changes, restart or reload the Apache server to make the changes take effect.

Is it working?

There are several ways you can see whether services are actually being cached. For instance, you can provide some kind of timestamp in the services results or their HTTP headers, or you can configure Apache to do some extra logging. For the former, I modified the common_setup.php code to send a Pragma header with the microtime(1) value, the current Unix timestamp including microseconds. (Pragma headers are implementation-specific values, which are ignored by applications to which they don't apply, so using this header is safe enough.) All service results will include this data, but it will just be ignored.

function sendResponseAndExitIf($condition, $code, $description, $extraHeader=false, $xmlResponse=false) {
    if ($condition) {
        header("HTTP/1.1 {$code} {$description}");
        header("Pragma: " . microtime(1));
        ...
    }
}

We can test whether caching is working by making a certain request repeatedly, and checking timestamps:

fkereki@fkereki-desktop:/srv/www/htdocs/services> wget 'http://127.0.0.1/rest_rewrite/countries/UY' -O /dev/tty --save-headers
--2013-04-09 22:32:00--  http://127.0.0.1/rest_rewrite/countries/UY
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1714 (1.7K) [text/xml]
Saving to: `/dev/tty'
HTTP/1.1 200 OK
Date: Wed, 10 Apr 2013 01:32:00 GMT
Server: Apache/2.2.22 (Linux/SUSE)
X-Powered-By: PHP/5.4.13
Pragma: 1365557520.4992
Content-Length: 1714
Cache-Control: max-age=10, public
Expires: Wed, 10 Apr 2013 01:32:10 GMT
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/xml; charset=UTF-8

<?xml version='1.0' encoding='UTF-8'?>
<countries><country><link rel='Uruguay' href='http://192.168.1.200/rest_rewrite/countries/UY' /><code>UY</code><name>Uruguay</name><regions><link rel='Artigas' href='http://192.168.1.200/rest_rewrite/regions/UY/01' />
<link rel='Canelones' href='http://192.168.1.200/rest_rewrite/regions/UY/02' />
...many lines snipped...
...<link rel='Treinta y Tres' href='http://192.168.1.200/rest_rewrite/regions/UY/19' />
100%[================================================================================================================================================================>] 1,714       --.-K/s   in 0s      

2013-04-09 22:32:00 (265 MB/s) - `/dev/tty' saved [1714/1714]

> wget 'http://127.0.0.1/rest_rewrite/countries/UY' -O /dev/tty --save-headers
--2013-04-09 22:32:05--  http://127.0.0.1/rest_rewrite/countries/UY
...
Pragma: 1365557520.4992
...
Expires: Wed, 10 Apr 2013 01:32:10 GMT
Age: 5
...

<?xml version='1.0' encoding='UTF-8'?>
<countries><country><link rel='Uruguay' href='http://192.168.1.200/rest_rewrite/countries/UY' /><code>UY</code><name>Uruguay</name><regions>
<link rel='Artigas' href='http://192.168.1.200/rest_rewrite/regions/UY/01' />
...many lines snipped...
...<link rel='Treinta y Tres' href='http://192.168.1.200/rest_rewrite/regions/UY/19' />
100%[================================================================================================================================================================>] 1,714       --.-K/s   in 0s      

2013-04-09 22:32:05 (152 MB/s) - `/dev/tty' saved [1714/1714]

> wget 'http://127.0.0.1/rest_rewrite/countries/UY' -O /dev/tty --save-headers
--2013-04-09 22:32:07--  http://127.0.0.1/rest_rewrite/countries/UY
...
Pragma: 1365557520.4992
...
Expires: Wed, 10 Apr 2013 01:32:10 GMT
Age: 6
...

<?xml version='1.0' encoding='UTF-8'?>
<countries><country><link rel='Uruguay' href='http://192.168.1.200/rest_rewrite/countries/UY' /><code>UY</code><name>Uruguay</name><regions>
<link rel='Artigas' href='http://192.168.1.200/rest_rewrite/regions/UY/01' />
...many lines snipped...
...<link rel='Treinta y Tres' href='http://192.168.1.200/rest_rewrite/regions/UY/19' />
100%[================================================================================================================================================================>] 1,714       --.-K/s   in 0s      

2013-04-09 22:32:07 (269 MB/s) - `/dev/tty' saved [1714/1714]

As you can see, the actual service code was called the first time, and the two next requests were fulfilled from the cache. In the latter cases the service adds the "Age:" header. After a wait that depends on the Cache-Control: max-age value the timestamp changes, so we know the PHP code was called again.

fkereki@fkereki-desktop:/srv/www/htdocs/services> wget 'http://127.0.0.1/rest_rewrite/countries/UY' -O /dev/tty --save-headers
--2013-04-09 22:37:08--  http://127.0.0.1/rest_rewrite/countries/UY
Connecting to 127.0.0.1:80... connected.
...
Pragma: 1365557828.7668
...
Expires: Wed, 10 Apr 2013 01:37:18 GMT
...

<?xml version='1.0' encoding='UTF-8'?>
<countries><country><link rel='Uruguay' href='http://192.168.1.200/rest_rewrite/countries/UY' /><code>UY</code><name>Uruguay</name><regions>
<link rel='Artigas' href='http://192.168.1.200/rest_rewrite/regions/UY/01' />
...many lines snipped...
...<link rel='Treinta y Tres' href='http://192.168.1.200/rest_rewrite/regions/UY/19' />
100%[================================================================================================================================================================>] 1,714       --.-K/s   in 0s      

2013-04-09 22:37:08 (237 MB/s) - `/dev/tty' saved [1714/1714]

Adding logging

The method above is useful for a quick spot check, but adding logging is a better approach, because it allows you to more thoroughly evaluate the results of your caching. The mod_log_config module is usually loaded by default; if it's not, edit the APACHE_MODULES variable as above to include it. By adding a few lines to the server configuration, we can ensure all cache requests are logged:

#
# Logging: mod_cache runs before mod_env,
# so on a cache hit CACHE_MISS won't be set
# and a dash will be shown instead.
#
SetEnv     CACHE_MISS      "Miss"
LogFormat  "%h %l %u %t \"%r\" %>s %b %{CACHE_MISS}e" common-cache
CustomLog  /var/log/apache2/cache.log common-cache

The CACHE_MISS variable will be set only if the cache wasn't used; for successful cache lookups it will be shown just as a dash, while unsuccessful ones will show "Miss." You can check the online documentation for a full explanation of the format parameters. We specify the log file in the CustomLog line. After our tests, the file looks like:

...
127.0.0.1 - - [09/Apr/2013:22:32:00 -0300] "GET /rest_rewrite/countries/UY HTTP/1.1" 200 1714 Miss
127.0.0.1 - - [09/Apr/2013:22:32:05 -0300] "GET /rest_rewrite/countries/UY HTTP/1.1" 200 1714 -
127.0.0.1 - - [09/Apr/2013:22:32:07 -0300] "GET /rest_rewrite/countries/UY HTTP/1.1" 200 1714 -
127.0.0.1 - - [09/Apr/2013:22:37:08 -0300] "GET /rest_rewrite/countries/UY HTTP/1.1" 200 1714 Miss

Tricking the cache
If in some case you absolutely require fully up-to-date data, you can trick the cache into calling the actual service by adding an irrelevant parameter to the request; a timestamp does nicely. For example, if you request the service at http://127.0.0.1/rest_rewrite/countries/UY?irrelevant=20130409231846.1532, the added parameter won't match any cached data, so the cache will be forced to call the service code. Of course, if you were always to need the latest information, then you shouldn't be caching that particular service at all!

The first line shows the original request, which had to be fulfilled by calling the PHP service code. The next two requests were satisfied out of the cache, while the last call, after the expiration time of the cache data, also was considered a miss. You can set logging like this and study it to see how much the cache setup is helping speed up your server.

Cleaning up

All the configuration we've seen so far is fine, but there's an ugly little problem lurking in the shadows: The cache directory will grow forever until it takes over all of your disk space, because the caching module never deletes anything. You could fix this by simply setting up a cron task that would rm -rf everything in the cache directory, but there's a cleaner way (pun intended) to do this, with htcacheclean. By running a command along the lines of htcacheclean -d360 -i -n -t -p/var/cache/apache2 -l10M (with appropriate permissions) you can clean out old entries from your cache directory and keep it trimmed below a maximum size. The parameters for that command are described below; for more possibilities check the documentation.

Parameter	Meaning
-d360	Run in daemon mode, every so many minutes (360 in this case).
-i	Run in "intelligent" mode, only if there were any cache changes.
-n	Be "nice" to other programs; favor other processes, even if it delays cleaning a bit.
-t	Remove empty directories.
-p	Specify the path to the cache directory.
-l10M	Limit the cache size to the given size (10 megabytes in this example).

By the way, you can clean just a specific URL to nullify (delete) a cached entry in order to force a refresh if you know you have updated data.

Adding caching to your server so fresh dynamic content won't be needlessly reevaluated can enhance the performance of your server, so consider this kind of caching for your web services.

Website editor showdown: CKEditor vs. TinyMCE

Matt Hartley — Mon, 08 Apr 2013 11:00:00 GMT

What's the best WYSIWYG editor for your website? CKEditor and TinyMCE are two popular JavaScript-powered WYSIWYG editors that make editing text on your web pages simple. CKEditor is a powerful, robust editor, while TinyMCE is more of a lightweight, minimalist option. Which should you use? It comes down to which editor has the most relevant functionality to address your needs. Let's look at what each editor has to offer, and see how well each meets a range of use cases.

Both CKEditor and TinyMCE are easy to install. Both are available as plugins for most content management systems (CMS), and some applications even bundle support for one or the other. WordPress for example, uses TinyMCE as the default editor. If a given application doesn't support one or the other out of the box, you can probably add support through a plugin or extension. If you choose to do that, you should install only one WYSIWYG editor on one application.

Both editors offer the standard HTML formatting tools, such as bold, italic, underline, lists, and various text alignments. Both provide cross-platform support and properly render text in all the popular web browsers.

Individual strengths

CKEditor offers extensive spell-checking functions, anchor text linking to other areas of the same page, and the ability to expand the editing area as you fill up the box with text. TinyMCE does not.

CKEditor offers a text editor control panel in WordPress through the CKEditor WordPress plugin; TinyMCE does not. CKEditor, as both a WordPress plugin and standalone, lets you customize the editor's output by making minor changes to things such as color and formatting in the embedded JavaScript code without editing complex configuration files. Making similar changes with TinyMCE is more complex, requiring you to edit your themes/advanced/skins/ui.css file.

Another way the two differ is in the way they handle a copied page. I browsed over to the Wazi homepage, pressed Ctrl-A to copy the content, then pasted the copied content into each editor. CKEditor kept most of the images and formatting, but TinyMCE managed to grab only the text, without the images or formatting.

The two editors offer different levels of user interface control to users without forcing them to work at the code level. Both editors let you change things such as the appearance of the text editor or how it behaves when formatting text, but only CKEditor offers you most of this control within an easy-to-use GUI for WordPress users. By contrast, TinyMCE doesn't offer a GUI control panel to make changes to its settings. To gain access to GUI control panel for TinyMCE, WordPress users need to install a plugin called TinyMCE Advanced, which offers a control panel under the WordPress settings where you can add or remove additonal TinyMCE formatting features. In standalone installations, neither CKEditor nor TinyMCE offers any sort of control panel.

CKEditor offers one enterprise-friendly feature that TinyMCE lacks. With the easy-to-use CKBuilder tool, you can create a custom-designed CKEditor that you can integrate into your website. You can choose a basic, standard, or full set of toolbar icons, select from more than 100 available plugins, and specify the language for your editor's interface.

By contrast, if you want to integrate advanced functionality with plugins for TinyMCE, you will need to code them yourself using this page as a rough guide.

One simple task indicates the level of customizability of the two editors. If you want to change the interface color for CKEditor, you would have to replace the <textarea> elements on the page you want to modify with a CKEditor instance and include the desired color of the user interface. For example, you can use the code below to specify the uiColor property for the editor embedded on any given page:

<script>

			// Replace the <textarea id="editor"> with an CKEditor
			// instance, using default configurations.
			CKEDITOR.replace( 'editor1', {
				uiColor: '#14B8C4',
				toolbar: [
					[ 'Bold', 'Italic', '-', 'NumberedList', 'BulletedList', '-', 'Link', 'Unlink' ],
					[ 'FontSize', 'TextColor', 'BGColor' ]
				]
			});

		</script>

It's a little easier if you have a WordPress website that uses the CKEditor plugin. You can simply log in to WordPress, browse over to CKEditor -> Basic Settings and make the change using the GUI.

For a standalone installation of TinyMCE, you change the appearance of the UI by tweaking the stylesheet settings in the TinyMCE/css/content.css file. Edit the file and look for this entry:

body {
	background-color: #FFFFFF;

Change #FFFFFF to another RGB color code, save the file, and your TinyMCE editor's background will reflect the changes to the background you've selected.

When it comes to support, CKSource, the company that offers CKEditor, supports its own product. TinyMCE partners with a third-party vendor for support. Both CKEditor's and TinyMCE's support options are paid services. TinyMCE's support vendor provides assistance after you've filled out a support request, asking them to reach out to you by phone or email. For CKEditor support, you must have a paid commercial license, and you can get support by email only through the CKSource contact page.

Selecting the best rich text editor

If you're looking to have control over a custom website environment, then CKEditor is a clearly the better choice. It's powerful out of the box and simple to customize, while remaining easy enough for any novice to use.

TinyMCE also offers a decent user experience, but it's better suited for developers looking for a minimalist user interface.

How to write CentOS initialization scripts with Upstart

Anatoliy Dimitrov — Thu, 04 Apr 2013 11:00:00 GMT

On Linux systems, initialization (init) scripts manage the state of system services during system startup and shutdown. When the system goes through its runlevels, the System V init system starts and stops services as configured. While this tried-and-true technology has been around since the dawn of Unix, you can now create modern and efficient CentOS 6 init scripts by using Upstart, an event-based replacement for System V init.

Until its latest release, CentOS used the System V init system by default. SysV init scripts are simple and reliable, and guarantee a certain order of starting and stopping.

Starting with version 6, however, CentOS has turned to a new and better init system – Upstart. Upstart is faster than System V init because it starts services simultaneously rather than one by one in a certain order. Upstart is also more flexible and robust, because it is event-based. Upstart generates events at various times, including while going through the system runlevels, similar to the SysV init system. However, Upstart may also generate custom events. For example, with Upstart you can generate an event that requires certain services to be started, regardless of the runlevel. And Upstart not only generates events, it also handles them – so, for example, when it acknowledges the event for starting a service it will do so. This event-based behavior is robust and fast.

Upstart supports SysV init scripts for compatibility reasons; most service init scripts in CentOS 6 continue to be SysV-based. You might someday have to create an init script yourself if you write custom software. If you do, you should write your new init scripts with Upstart in mind so you can benefit from the new init system's faster performance and additional features.

Beginning the Upstart init script

Upstart keeps init scripts in the /etc/init/ directory. A script's name should correspond to the name of the service or job it controls, with a .conf extension. The init script for the Tomcat service, for example, should be named /etc/init/tomcat.conf.

Like SysV init scripts, Upstart init scripts are regular Bash scripts, but extended with some Upstart-specific directives, which are called stanzas in Upstart. In SysV init scripts you commonly see the line . /etc/init.d/functions, which provides access to additional necessary SysV functions. Upstart scripts are more sophisticated and complete; you don't have to include any additional functions or libraries.

Just as in any Bash script, comments in Upstart scripts start with #. Put descriptive comments at the beginning of each script to explain its purpose, and in other places where the code may need explanation. You can use two special stanzas, author and description, for documentation.

Defining when a service starts

Tasks and services
Upstart manages two types of jobs: tasks and services. Tasks are short-lived processes that are expected to start, complete a task, then die. One example for such a task job is defined in /etc/init/control-alt-delete.conf in CentOS 6. It restarts the computer when a user presses the Control, Alt, and Delete keys.

In contrast to a task job, a service job handles a daemon or service, such as the Apache web service. This article focuses on service jobs.

After the introductory comments you can define when a service should start and stop using the special stanzas stop on and start on. These two stanzas can be used with a recognized Upstart event such as when the system enters a runlevel.

Usually administrators configure service jobs to start and stop with the server. By convention, in CentOS you should configure a service job to start at runlevels 2, 3, 4, and 5 and stop at runlevels 0, 1, and 6. In an Upstart init script this is written like this:

start on runlevel [2345]
stop on runlevel [06]

This instructs Upstart to start and stop the service whenever the system enters the runlevel in brackets.

Upstart also lets you start or stop services based on other types of events, such as the starting or stopping of other services. For example, suppose you have an Apache web server integrated with a Varnish caching web server, as described in the article Varnish improves web performance and security. In such a scenario you should make sure that Varnish starts whenever Apache starts, so the configuration stanza for Varnish should look like:

start on starting httpd
stop on stopped httpd

The latter stanza is an unique feature of Upstart; Upstart init scripts can stop services at the same time as other services are stopped, while SysV init scripts depend solely on runlevels.

Another difference is that you configure SysV init scripts when to start and stop by placing symlinks to them in the corresponding runlevels' directories in /etc/rcX.d/, where X is the runlevel number. The command chkconfig does this automatically for you in CentOS. While chkconfig continues to manage most of the init scripts in CentOS 6, it does not work with Upstart, and you cannot manage Upstart jobs with it.

Preparing for an Upstart job

To prepare and customize your environment for an Upstart service job you can use a few additional parameters, each on a new line in the job's .conf file:

respawn – When you use this parameter the service process will be restarted if it dies unexpectedly. Without this Upstart parameter you might have to write a dedicated wrapper program to start a service and ensure its proper and constant operation, such as mysqld_safe for starting MySQL.
expect fork – Every service job should be expected to fork in as a background process. When you specify this parameter Upstart obtains the new process's PID, which it can use later to send signals to it, such as to shut down or reload configuration.
kill timeout [seconds] – This is the number of seconds before the process may be forcibly killed. You should specify enough time (say, 120 seconds) so that interruption-sensitive services such as MySQL are able to complete any pending operations and shut down safely.

You can also adjust a few Bash variables in the job's .conf file. For example, you can configure the umask (permissions) with which new files will be created. A secure choice is umask 007, which means that new files will be created with restrictive permissions 770, which allow only the user itself and the members of its user group to manipulate the newly created files.

A special Upstart stanza pre-start allows you to specify a command or inline script to be run right before Upstart actually starts the job. This stanza is suitable for specifying any sanity checks, such the existence of a necessary file. You may also define prerequisite tasks, such as cleaning of the caching directory of a caching proxy. If the pre-start procedure fails – that is, if it exits with a code other than zero – then the whole job fails and the service is not started.

To see how this works, here's a pre-start directive to remove PHP accelerator eAccelerator's previously cached files, as you would want to do when eAccelerator is integrated with Apache: pre-start exec rm /var/cache/eaccelerator/* -rf. A longer example with a whole inline script looks like:

pre-start script
    # check if Apache's binary is executable or fail 
    [ -x /usr/sbin/httpd ]
    # clear the /tmp directory from old sessions
    rm /tmp/sess_*
end script

Several other stanzas are similar to the pre-start stanza:

post-start – specifies a procedure to run after starting the service. This is usually useful for complex services that may need additional attention after startup, such as MySQL.
pre-stop – specifies actions used in preparing for the service shutdown. It is rarely used.
post-stop – may be regarded as an alternative to the pre-start stanza; in some cases it makes more sense to take some actions right after service shutdown instead of waiting for its next start.

Configuring the Upstart start command or script

Once you've configured your environment, the last thing you must do is define the job's command or script using the stanzas script ... end script to write a regular Bash script inline, or just exec to simply execute a command with arguments. Here's an example that uses the script stanza for the rsyslog service in the rsyslog.conf file:

script
    . /etc/default/rsyslog
    exec rsyslogd $SYSLOGD_OPTIONS
end script

The above directive first sources (includes) the content of the file /etc/sysconfig/rsyslog, where the variable SYSLOGD_OPTIONS is defined. This variable is then used to start the rsyslogd service. This is a convenient way to start a service that requires complex or custom configuration, and it's why such script stanzas are suitable for services such as MySQL or Apache.

Alternatively, the exec stanza lets you specify an executable file and any additional arguments it may need. It's suitable for simpler services; for example, you can start the CUPS daemon with the directive exec /usr/sbin/cupsd -F.

How Upstart stops and reloads services

If you're familiar with SysV init scripts, you may wonder how you configure the commands to stop a service or reload its configuration. The answer is that you don't have to; with Upstart you only configure the start command for a service. When Upstart starts a service it keeps track of its PID and the PIDs of the process forks. When Upstart later needs to shut down a service, it does so with the native Unix signals. Upstart first sends a PID the SIGTERM signal to gracefully shut it down. If the process ignores SIGTERM, Upstart sends SIGKILL to forcibly kill it. Similarly, when the configuration needs to be reloaded, Upstart sends the SIGHUP signal. Upstart's simple architecture removes the needs to specify procedures for stopping, restarting, and reloading a service, though if the shutdown procedure for a service requires more than just sending SIGHUP or SIGTERM signals, you can use the pre-stop and post-stop stanzas.

One last and important difference between Upstart and SysV inits is how you manually start and stop jobs. Upstart works with the command /sbin/initctl, the init daemon control tool. It accepts as a first argument stop, start, restart, or reload, and changes the state of the service correspondingly. The second argument is the name of the service. For example, to start MySQL's Upstart job manually you would run the command initctl start mysqld.

I hope you can see the advantages of Upstart's powerful and sophisticated features. Today most default service jobs in CentOS 6 today remain SysV-based, though that will probably change over time. In Ubuntu, for example, Upstart was introduced in 2009, and today most of the init scripts have been migrated to Upstart.

Write your own Ant tasks

Juliet Kemp — Mon, 01 Apr 2013 11:00:00 GMT

Apache Ant is a Java-based build tool, and because it's based on Java, it is entirely cross-platform. It comes with a huge list of available tasks that you can run simply by including them in a build.xml file, but if there's something you want it to do and no existing task can do it, it's easy to extend Ant just by writing a new Java class. Here's how to write your very own Ant tasks.

Before you get started, it's always best to check the list of existing Ant tasks, and the list of external (not included in default Ant) tasks submitted by developers outside the Ant project and not directly supported by the Ant developers. While writing a new task can be fun, there's no need to reinvent the wheel if someone has already done the work for you. If they haven't, because you've found something new it would be handy to do or have unusual specifics you need to manage, you can extend Ant for yourself.

A very basic new task

To write a new task, you need to extend the Ant Task class. (You can theoretically manage without doing so, but extending Task gives you access to a bunch of useful methods and setup, so it's sensible to use it.) Let's see how with a simple HelloWorld example. Create a new directory hellotask, and a file src/HelloTask.java that contains:

 
import org.apache.tools.ant.Task;

public class HelloTask extends Task {
  public void execute() {
    String message = getProject().getProperty("ant.project.name");

    log("Project: " + message);
    log("Hello from " + getLocation());
  }
}

The only method you absolutely must have to extend Task is execute(). Here, we're calling the logger twice to output two lines. getProject(), getProperty(), and getLocation() are all inherited from the Task class, and do what you'd expect.

To run and compile this file, obviously we want to use Ant! Here's the build.xml file:

<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="HelloTask" basedir="." default="jar">

    <property name="src.dir" value="src" />
    <property name="classes.dir" value="classes" />

    <target name="clean" description="Delete all generated files">
        <delete dir="${classes.dir}"/>
        <delete file="${ant.project.name}.jar"/>
    </target>

    <target name="compile" description="Compile HelloTask">
        <mkdir dir="${classes.dir}"/>
        <javac srcdir="${src.dir}" destdir="${classes.dir}"/>
    </target>

    <target name="jar" description="Create JAR" depends="compile">
        <jar destfile="${ant.project.name}.jar" basedir="${classes.dir}"/>
    </target>

    <target name="hello" description="Run HelloTask" depends="jar">
        <taskdef name="hellotask" classname="HelloTask" classpath="${ant.project.name}.jar"/>
        <hellotask />
    </target>

</project>

Most of this should be familiar if you read my tutorial on Ant buildfiles; it cleans things up, compiles, and creates a JAR from the source code. Run ant jar to build HelloTask into a JAR.

The final section is the bit that actually runs the task. The taskdef (task definition) line tells Ant what to do when it sees <hellotask /> later on. (With predefined tasks, you don't need this line because Ant already knows how to handle them.) Note that in this setup, where we might be deleting and recompiling the JAR, the taskdef line must be inside the target block. If you put the definition earlier in the file, and HelloTask.jar has been deleted (by running ant clean), Ant won't find the JAR and you'll get an error. Later, if you were to import the JAR elsewhere and just run the task, you could put this line at the top of the file.

Finally, we set up the hello target to run <hellotask/>. Type ant hello and you should see output a bit like this:

Buildfile: /home/juliet/coding/ant/mytask/build.xml

compile:

jar:

hello:
[hellotask] Project: HelloTask
[hellotask] Hello from /home/juliet/coding/ant/mytask/build.xml:24: 

BUILD SUCCESSFUL Total time: 0 seconds

Congratulations – you've created your first task!

Handling properties in a task

Currently, this task doesn't take parameters. Let's rewrite it so we can pass something in – the name of the task's owner, for example:

public class HelloTask extends Task {
    private String owner;

    public void execute() {
        String projectName = getProject().getProperty("ant.project.name");

        log("Project: " + projectName);
        log("Hello to " + owner + " from " + getLocation());
    }

    public void setOwner(String owner) {
        this.owner = owner;
    }
}

Note that the setter method (setOwner()) must match the name of the variable you wish to set. Ant then generates a matching getter method at compile time, and uses this when the task is run. (You could of course also write your own getter method if you prefer.) Edit build.xml to specify the parameter:

<target name="hello" description="Run HelloTask" depends="jar">
  <taskdef name="hellotask" classname="HelloTask" classpath="${ant.project.name}.jar"/>
  <hellotask owner="Juliet"/>
</target>

Run ant hello and your output should look like this:

 
Buildfile: /home/juliet/coding/ant/mytask/build.xml

compile:
    [javac] Compiling 1 source file to /home/juliet/coding/ant/mytask/classes

jar:
      [jar] Building jar: /home/juliet/coding/ant/mytask/HelloTask.jar

hello:
[hellotask] Project: HelloTask
[hellotask] Hello to Juliet from /home/juliet/coding/ant/mytask/build.xml:24: 

BUILD SUCCESSFUL Total time: 0 seconds

A more complicated task

Finally, let's take a look at a more complicated real-world example – an Ant task I wrote to deal with LaTeX files. This is the first, ultra-basic version, which just generates DVI output from a given LaTeX input file. Save this as src/MakeLatex.java:

public class MakeLatex extends Task {
  private String input;

  public void execute() {
    log("Building file: " + input);
    String runLatex = "latex " + input;
    try {
      String line;
      Process p = Runtime.getRuntime().exec(runLatex);
      BufferedReader input = new BufferedReader(
          new InputStreamReader(p.getInputStream()));
      while ((line = input.readLine()) != null) {
        System.out.println(line);
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  public void setInput(String input) {
    this.input = input;
  }
}

The setter method is familiar. In execute(), we send a log message, then create the command to run from the input provided. This command is then run externally with Runtime.getRuntime().exec(), and the output collected and printed to screen with a BufferedReader and the while loop.

To run this, edit the build.xml file to include this new target:

<target name="latex" description="Compile LaTeX" depends="jar">
    <taskdef name="makelatex" classname="MakeLatex" classpath="${ant.project.name}.jar"/>
    <makelatex input="test"/>
</target>

The existing jar target builds the new class into the HelloTask.jar file. Obviously if you were using this for real you'd want to build it into its own jarfile and save it somewhere else so you could use it in future.

You also need a test .tex file. (Note: The test file is saved as test.tex, but you don't need the .tex extension when setting input in build.xml; LaTeX will assume it.):

\documentclass[a4paper]{article}
\begin{document} Test!
\end{document}

Run ant latex to build your file, then look at test.dvi in your preferred DVI viewer to see the output.

While this approach generates a test.dvi file, it doesn't open it for the user to look at, and it generates a lot of output noise. Also, these days most people prefer PDF output to DVI output – I certainly do! So here's an improved version, which takes an output and a verbosity value:

 
public class MakeLatex extends Task {
  private String input;
  private String output = "pdf";
  private Boolean verbose = false;
  private String latexCommand;
  private String showOutputCommand;

  public void execute() {
    log("Building file: " + input);
    if (output.equals("pdf")) {
      latexCommand = "pdflatex";
      showOutputCommand = "xpdf " + input + ".pdf";
    } else if (output.equals("pdf")) {
      latexCommand = "latex";
      showOutputCommand = "xdvi " + input + ".dvi";
    } else {
      log("Don't know how to handle output " + output + "!");
      log("Generating and showing you the dvi instead");
      latexCommand = "latex";
      showOutputCommand = "xdvi " + input + ".dvi";
    }
    String runLatex = latexCommand + " " + input;
    try {
      String line;
      Process p = Runtime.getRuntime().exec(runLatex);
      if (verbose) {
        BufferedReader input = new BufferedReader(
            new InputStreamReader(p.getInputStream()));
        while ((line = input.readLine()) != null) {
          System.out.println(line);
        }
      }
      p = Runtime.getRuntime().exec(showOutputCommand);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  public void setInput(String input) {
    this.input = input;
  }
  public void setOutput(String output) {
    this.output = output;
  }
  public void setVerbose(Boolean verbose) {
    this.verbose = verbose;
  }
}

And here's the accompanying buildfile, which specifies the output format and verbosity. If your buildfile doesn't specify output or verbosity values, the task defaults to PDF and to non-verbose:

<target name="latex" description="Compile Latex" depends="jar">
   <taskdef name="makelatex" classname="MakeLatex" classpath="${ant.project.name}.jar"/>
   <makelatex input="test" output="dvi" verbose="true"/>
</target>

Ant will now generate the sort of output you prefer, and fire up the specified viewer.

Finally, one last improvement: Let's pass in a filename from the command line. Ant can do this if you add this code to build.xml:

<condition property="params.set">
  <isset property="input"/>
</condition>

<target name="latex" description="Compile Latex" depends="jar">
  <fail unless="params.set">
     Must specify input file
  </fail>
  <taskdef name="makelatex" classname="MakeLatex" classpath="${ant.project.name}.jar"/>
  <makelatex input="${input}" verbose="false"/>
</target>

This sets a condition property, params.set, based on whether the "input" value is set. When we build the latex target, Ant first checks the status of params.set, and fails the build (stopping there) unless it is true. If it is true, the value is passed into the task, and off we go. You call this like so (note the -D argument):

 ant latex -Dinput='test'

As it stands, this task is very basic. You could add more options to it; for example, you could add other output formats, or a bibtex option. You could also do some error-checking on the input filename; currently if you set input as test.tex rather than test, the task would fail. Experiment a little with the code to see what else you can do with it, and to get a better grasp of writing a task to do exactly what you want it to.

How to convert Apache rewrites for nginx

Riccardo Capecchi — Thu, 28 Mar 2013 11:00:00 GMT

Apache is still by far the most widely deployed HTTP server, according to the latest Netcraft web server survey, but nginx has been slowly, steadily gaining market share, thanks to its blazing speed. If you want to try a faster web server and move from Apache to nginx, you'll probably have to change some of your websites' configurations, starting with rewrite directives. To migrate rewrite rules from Apache to nginx, start with these tips and tricks.

The Apache mod_rewrite module provides powerful and sophisticated tools for nearly all types of URL rewriting. It is, however, somewhat complex, and may be intimidating to beginners. In fact, however, rewrite rules are not magical incantations, though to understand them you need some understanding of regular expressions.

Even if you have never heard of mod_rewrite, you may still be using it. Popular applications such as WordPress, Drupal, and Magento are shipped with .htaccess files that contain standard configurations that make these applications work properly, and these usually include one or more rewrites, so to properly move your website to an nginx web server you have to "translate" the Apache mod_rewrite directives into equivalent rules for nginx's HttpRewriteModule.

If you want to see whether your website is using mod_rewrite, you can use this approach:

In your document root, the directory where you have all your HTML files, add a file named .htaccess, or modify it if it already exists, and make sure it contains these lines:

<IfModule mod_rewrite.c>
    // Tell PHP that the mod_rewrite module is ENABLED.
    SetEnv HTTP_MOD_REWRITE On
</IfModule>

Then add a second file in your document root named check_rewrite.php with the following content:

<?php

if (is_mod_rewrite_enabled()) {
  print "The apache module mod_rewrite is enabled.
\n";
} else {
  print "The apache module mod_rewrite is NOT enabled.
\n";
}

/**
 * Verifies if the mod_rewrite module is enabled
 *
 * @return boolean True if the module is enabled.
 */
function is_mod_rewrite_enabled() {
  if ($_SERVER['HTTP_MOD_REWRITE'] == 'On') {
    return TRUE;
  } else {
    return FALSE;
  }
}
?>

Now browse to http://yourservername.com/check_rewrite.php, and you can see whether mod_rewrite is enabled.

As general guide you can follow these simply steps to move your rules from Apache to nginx configuration file:

Replace the word "RewriteRule" in your Apache configuration files (or .htaccess) with "rewrite"
Replace the character "^" with "^/"
Replace the options [L] or [QSA,L] with "last;" (every line in nginx configuration files must end with ";")
If a rule contains braces – that is, { or } – you have to surround the regular expression with quotation marks.

Let's take a look at some examples. Bear in mind that in Apache directives can be written in a .htaccess file in your document root or in the virtual host file configuration. Usually they are located in /etc/apache2/sites-enabled (if you're running Debian or Ubuntu) or /etc/httpd/conf.d/ (under CentOS or Red Hat). In nginx, however, all the configurations regarding a virtual host, including the rewrite rules, are located in its server section, which is usually /etc/nginx/sites-enabled (Debian/Ubuntu) or /etc/nginx/conf.d/ (CentOS, Red Hat).

Rewrite rule to change a domain

Sometimes you want to rewrite a URL adding (or removing) "www." This might happen if you first published your website without www and later want to change the main URL, but you don't want to lose all the links that are coming from search engines and other websites. So, for example, you could rewrite all the URLs that arrive at linuxaria.com to www.linuxaria.com. In Apache mod_rewrite you would use something similar to these directives:

<Virtualhost>
Servername www.linuxaria.com
ServerAlias linuxaria.com
RewriteCond  %{HTTP_HOST}  linuxaria.com
RewriteRule  (.*)          http://www.linuxaria.com$1
...

The easiest way to achieve the same result with nginx is to create two server sections for your virtualhost. In the one for the domain that you want to redirect, include just a rewrite directive, and in the one for the real website, put all the directives you need for your website. So, for the first server, the one you want to redirect, use:

server {
    listen       80;
    server_name  linuxaria.com;
    return       301 http://www.linuxaria.com$request_uri;
}

And for the second server, the real website:

server {
    listen       80;
    server_name  www.linuxaria.com;
    ...
}

In this example we have not translated the rewrite rules, but rather changed our approach to solve the problem. We'll use the same principle in the next example.

WordPress permalinks

WordPress permalinks are permanent URLs that link to individual blog posts or categories and other lists of blog postings. Other people can use permalinks to link to your articles, or you might use one to send a link to someone in an email message. To produce "pretty" permalinks, WordPress uses mod_rewrite in Apache, and you should put the following standard rules in your .htaccess file if you use Apache (WordPress will not do this automatically):

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

You could translate these statements to "If the requested file is not a real file (!-f) or a directory (!-d), then use the page index.php."

A "literal" translation of these rules for nginx could be:

if (!-f $request_filename){
set $rule_1 1$rule_1;
}
if (!-d $request_filename){
set $rule_1 2$rule_1;
}
if ($rule_1 = "21"){
rewrite /. /index.php last;
}

But this is not an efficient solution with nginx. It's better to change the logic:

if (!-e $request_filename)
    {
        rewrite ^(.+)$ /index.php?q=$1 last;
    }

Here the logic says, "If the requested file doesn't exist (!-e) then use the page index.php." This change allow us to get the same result with fewer lines in the configuration file, because Apache mod_rewrite doesn't have this option, but nginx HttpRewriteModule does.

Modify a URL extension with nginx

Suppose you want to add by default a .html extension to all the URLs of your website, so http://linuxaria.com/myurl would become http://linuxaria.com/myurl.html. This could become necessary if you changed your blog platform, moving from WordPess to Jekyll or another static CMS. In Apache you can get the result with these rules:

RewriteCond %{REQUEST_URI} !^.*\.html$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ $1.html [L,R=301]

The nginx approach is similar to the one that we saw above. First we verify that the URL doesn't already contains .html. Next we verify that the requested filename with the extension .html exists. If it does, we rewrite the URL, adding .html to the end:

location / {
        # break if URI has .html extension
        if ($request_filename ~* ^.+.html$) {
          break;
        }
        # add .html to URI and serve file, directory, or symlink if it exists
        if (-e $request_filename.html) {
          rewrite ^/(.*)$ /$1.html last;
          break;
        }
      }

Useful resources

To learn more about rewrite rules in nginx, read the Nginx HttpRewriteModule documentation page, which provides examples and conditions that you can use in your expressions.

You can also visit two websites that attempt to translate your existing rules for you, though they are not always 100 percent accurate. The htaccess to nginx converter was conceived as a mod_rewrite-to-nginx converter, but it also allows you to convert some other instructions that you might have reason to port from Apache to nginx. A second script to convert Apache htaccess to nginx is much more "literal" in its translation, but can be useful sometimes.

This is just a brief introduction to some common rewrite tasks that you might use on your site and that you'll probably need to consider when migrating from Apache to nginx. Rewrites are powerful, and can be complex, but learning how to use rewrites effectively can save your day, and add functionality to your website without the use of other software.

Three ways to integrate Tomcat and Apache for best performance and features

Anatoliy Dimitrov — Mon, 25 Mar 2013 11:00:00 GMT

Tomcat, the most popular open source implementation of the Java Servlet and JavaServer Pages specifications, is frequently integrated with Apache, the most popular open source web server. Here's a tutorial that shows you three ways to use Tomcat and Apache together.

For this article I installed Tomcat and Apache on CentOS 6, but the information here should apply to other Linux distributions too. I'm assuming you already have a basic setup of Apache on Linux; if not, on CentOS run the command yum install httpd.

Pros and cons for integrating Tomcat with Apache

Tomcat can serve clients perfectly fine by itself, but by integrating it with Apache you get:

Faster static content delivery – Apache serves static content better than Tomcat and supports better caching mechanisms.
High availability – Apache allows load balancing and clustering of multiple Tomcat servers behind it, thus providing high availability.
Better security – Apache protects Tomcat through its built-in security features and through advanced third-party modules such as ModSecurity.
Extensibility – Apache provides an abundance of modules for just about anything from URL rewriting (ModRewrite) to GeoIP services. With Apache you can use these modules to extend Tomcat's functionality.

Still, there are a few drawbacks for using Tomcat with Apache. First, dynamic content is delivered a little bit more slowly because data has to pass through Apache instead of flowing directly between the client and Tomcat. Second, you have to support an additional service (Apache), thus increasing the complexity of the setup and the maintenance burden.

Obviously the pros outnumber the cons, and that's why Tomcat is usually integrated with Apache or another web server.

Proper Java setup

Before you start working with Tomcat and Apache, you should be sure Java is set up properly. By default on most Linux distributions, OpenJDK (the open source Java Development Kit) is installed and supported through the official packet manager. To check if this is the case on your system run java -version and look for output like:

java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.8.0.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

OpenJDK is fine in most cases since it shares most of the code base with Oracle's commercial JDK. However, for better compatibility and trouble-free Java support you can use Oracle's JDK. To do so, download the latest JDK from Oracle's site. For CentOS look for the package jdk-7u17-linux-x64.rpm (64-bit architecture) or jdk-7u17-linux-i586.rpm (32-bit architecture). Once you download the installation package, use the command rpm -ivh to install it.

If you had OpenJDK installed before you installed Oracle's version you should make sure that Oracle's JDK is the default Java environment. Use the command /usr/sbin/alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 200000 to specify that Oracle's Java binary (/usr/java/latest/jre/bin/java) should be with the highest priority (200000) compared to OpenJDK (by default it has priority 170009).

You can also use the command alternatives --config java to change the default Java environment. This command lists all alternatives for Java; choose /usr/java/latest/jre/bin/java as the default one.

Confirm you have the correct Java by running java -version again. You should see output like:

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Tomcat installation

Tomcat 7, the current version, provides the latest features and performance enhancements. In most cases this version should be your choice, but to confirm it suits your business requirements check Tomcat's version comparison page.

Usually Tomcat 7 has to be installed from source because most Linux distributions (including CentOS 6) support the previous stable version, Tomcat 6, through their official package managers. To install Tomcat 7 first create a user and group for running the Tomcat service with the commands

groupadd tomcat
useradd -s /sbin/nologin -g tomcat tomcat

The first command creates the tomcat group; the second creates the user tomcat as part of the tomcat group, and for security reasons does not allow it to log in to the system.

After that, download the latest Tomcat installation files from the official download page. Extract the files to /usr/share with the command tar zxvf apache-tomcat-7.0.37.tar.gz -C /usr/share/. Change the ownership of the newly extracted files to the tomcat user and group with the command chown tomcat: /usr/share/apache-tomcat-7.0.37/ -R.

Now you can create the startup script for Tomcat 7 in /etc/init.d/tomcat7:

#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
CATALINA_HOME=http://www.openlogic.com/usr/share/apache-tomcat-7.0.37/bin

case $1 in
start)
/bin/su -s /bin/sh tomcat $CATALINA_HOME/startup.sh
;;
stop)
/bin/su -s /bin/sh tomcat $CATALINA_HOME/shutdown.sh
;;
restart)
/bin/su -s /bin/sh tomcat $CATALINA_HOME/shutdown.sh
/bin/su -s /bin/sh tomcat $CATALINA_HOME/startup.sh
;;
esac
exit 0

The above Bash script first sets the home directory for Catalina, Tomcat's servlet container, to /usr/share/apache-tomcat-7.0.37/bin. Then the script defines three scenarios (cases). Depending on the first argument passed to the script (start, stop, or restart) the script runs the corresponding Tomcat start and stop scripts under the tomcat user.

Make the script executable by changing its permissions to 755 with the command chmod 755 /etc/init.d/tomcat7. After that you can start Tomcat 7 for the first time with the command service tomcat7 start. You should also make sure Tomcat starts and stops with the system by running chkconfig tomcat7 on.

To confirm Tomcat 7 has been properly installed and started try to access it at its server's address and TCP port 8080 – for example, http://192.168.1.66:8080. You should see the default Tomcat 7 page.

Integrating Tomcat with Apache through ModProxy and ModProxyAJP

There are a few ways to integrate Apache with Tomcat, all involving Apache modules that allow Apache to act as an intermediary. The simplest approach is to use ModProxy or ModProxyAJP. The difference between them is mainly in the protocol the modules use to communicate with Tomcat. ModProxy is more transparent and works only with the HTTP protocol, while ModProxyAJP uses Apache JServ Protocol version 1.3 (AJP13), a binary communication protocol designed to enhance performance.

Both ModProxy and ModProxyAJP are installed and enabled by default with the httpd package on CentOS. Configuring them is relatively simple; here are two examples. First, to use ModProxy to forward all the requests through Apache to Tomcat, use the directives ProxyPass and ProxyPassReverse:

ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

In CentOS, Apache automatically includes any file with the extension .conf inside the directory /etc/httpd/conf.d/, so place the above lines in a file such as tomcat.conf inside this directory. These two directives tell Apache to provide the necessary proxy functionality to and from Tomcat. Any URL request is forwarded to Tomcat, as configured by the argument / for root directory after the ProxyPass and ProxyPassReverse directives. Here Apache and Tomcat are on the same server, as requests are proxied to localhost at the default Tomcat port 8080.

Besides listening for http requests on port 8080, Tomcat also listens for the AJP13 protocol on port 8009 by default. To configure Apache to communicate with Tomcat over the AJP13 protocol and port 8009, edit the file /etc/httpd/conf.d/tomcat.conf to look like this:

ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

This proxy configuration is almost identical to the previous one, with adjustments only for the protocol (AJP13) and the port (8009). After the adjustments the address of the destination server becomes ajp://localhost:8009/

To see the difference between the performance of AJP13 and HTTP, try a few tests with the Apache Benchmark tool (ab). Ab is part of the CentOS package httpd-tools; you can install it with yum install httpd-tools.

When designing your benchmark tests, make sure you run enough iterations to make your data representative. You should have enough if you send 100,000 requests, 50 concurrent at a time. For target URL you can use one of Tomcat's example Java server pages. You can run such a test with the command /usr/bin/ab -n 100000 -c 50 http://192.168.1.66/examples/jsp/jsp2/el/basic-comparisons.jsp.

Try this command with the two alternative proxy configurations and compare the results. On the average, your results should be 20 percent faster with AJP13 (ModProxyAJP) than with http (ModProxy).

Advanced integration of Tomcat and Apache through mod_jk

The mod_jk module is an alternative to ModProxy and ModProxyAJP and can be used in their place. It provides the most advanced functionality to connect Apache to Tomcat and provides load balancing and failure detection.

To install mod_jk, first make sure that you have its dependencies installed. For CentOS 6 you can install the required packages and their dependencies with the command yum install gcc gcc-c++ httpd-devel. This installs the GNU compiler collection with C++ support (gcc, gcc-c++) and Apache's development tools, including the required apxs tool.

Next, download mod_jk's latest source from Tomcat's connectors page and extract it to a temporary directory. Inside the source directory compile and install the software with the commands ./configure --with-apxs=http://www.openlogic.com/usr/sbin/apxs && make && make install. This should create mod_jk and place it in Apache's modules directory under the name /usr/lib64/httpd/modules/mod_jk.so.

Once mod_jk is installed you can configure it, via two files. The first, /etc/httpd/conf.d/tomcat.conf, should look like this:

LoadModule    jk_module  /usr/lib64/httpd/modules/mod_jk.so
JkWorkersFile /etc/httpd/conf.d/workers.properties
JkLogFile     /var/log/httpd/mod_jk.log
JkLogLevel    info
JkMount /* localserver

The LoadModule directive instructs Apache to load the newly compiled module from /usr/lib64/httpd/modules/mod_jk.so. JkWorkersFile configures the essential mod_jk configuration file, as we'll see in a moment. After that come the logging directives: JkLogFile specifies the log file and JkLogLevel the logging level.

The last directive, JkMount, has the structure JkMount URLprefix Workername. In this example, any URL request (/*) is forwarded to the worker named localserver. The localserver worker is defined in the file workers.properties, which is the second configuration file we need for mod_jk. It contains the back ends' configuration. Here is an example configuration with only one back-end server, worker:

worker.list=localserver
worker.localserver.port=8009
worker.localserver.host=localhost
worker.localserver.type=ajp13

It define the name of the worker, localserver, along with a port (8009), host (localhost), and type (ajp13).

If you try the previous benchmark tests after installing mod_jk you should see good performance results similar to the ones for ModProxyAJP and AJP13 that complement mod_jk's advanced features for complex setups with multiple workers. In such setups mod_jk provides robust load balancing with failure detection, session stickiness, and work quotas (how much traffic a worker should get according to its predefined load balancing factor). For more information check the official worker properties documentation.

Considering the above, it's fair to say that for a simple setup with a pair of Tomcat and Apache servers on the same machine you should use ModProxyAJP and AJP13. However, if you want robust load balancing, it's worth investing in setting up mod_jk.

Nine all-purpose plugins for Vim

Bruce Byfield — Thu, 21 Mar 2013 11:00:00 GMT

In its 22 years of existence, the Vim editor has become the center of an ecosystem of plugins. Whether you are coding or writing a text document, you can probably find a plugin that will make you more productive.

If you're coding in a specific programming language, you can find multiple plugins that will customize Vim to transform it into a dedicated editor for your language, but you don't have to be so specialized to benefit from many Vim plugins. Consider the following.

Pathogen

The more plugins you use in Vim, the harder it is to keep them updated or delete them. A number of plugin managers for Vim can help. The oldest and most popular, Pathogen, simplifies plugin management by requiring that each plugin have its own subdirectory. It is popular among those who prefer to manage plugins for themselves, but appreciate having a tool that simplifies the process.

Vundle ("Vim bUndle") uses the same directory structure as Pathogen, but takes the additional step of requiring modifications to the .vimrc directory in order to allow automatic updates and provide the ability to turn off a plugin without actually deleting it.

NERDTree

Switching from Vim to a file manager can break your concentration, especially if you are working in a maximized terminal. NERDTree solves the problem by adding a directory tree in a separate pane inside Vim. The tree highlights files, directories, symbolic links, read-only files, and binaries in different colors for easy identification. You can also bookmark files so that you can jump directly to them.

When you start NERDTree with the command :NERDTree, it shows the current directory. If you prefer, you can specify a directory or a bookmarked file as the starting point instead by adding its path to the end of the commandYou can also use the command :NERDTreeFind [String]. to search for a specific file and jump to it.

If you like the idea of having a file manager within Vim but don't love NERDTree, you have other options. Easytree, for instance, has fewer features, while Vimpanel and wimanager have more.

YankRing

"Yank" is Vim's term for copying or cutting and pasting. A yankring is a retrievable history of yanks, similar to the one provided by Klipper on the KDE desktop.

By default, Vim stores the last nine yanks. However, with the YankRing plugin, you can reconfigure Vim to store more or fewer. Even more importantly, YankRing allows you to copy or cut in a separate pane or with a series of commands that give you the choice of yanking everything from the current word or line to a range of lines. If you store the ring in a file, you can use it with multiple instances of Vim, which may be especially handy on a network.

Surround

Surround is useful with many programming languages because it deals with delimiters – characters that appear in pairs to define a string, such as quotation marks, brackets, parentheses, and complete tags in markup languages like HTML and XML. Surround adds, deletes, and replaces delimiters via a series of keyboard commands.

If you want to compare Surround with similar alternatives, closetag and ragtag offer something of the same functionality, specifically for markup languages.

Showmarks

Marks are bookmarks within a Vim document. You can set a mark by pressing m followed by another letter that designates the mark. To jump to a mark, enter ' followed by the mark's letter.

The great weakness of marks is that they are invisible. This limits the number you can use to however many you can remember, and you can easily accidentally overwrite an existing mark by creating another with the same name.

Showmarks allows you to toggle the visibility of marks off and on – and that tiny functionality is enough to increase the usefulness of marks several times over.

NERD-commenter

NERD-commenter is a general-purpose plugin designed for writing code or using markup languages. Even if you are not writing code or markup, comments are still helpful when you're collaborating with others on a document. You may also find comments more convenient than storing thoughts in a separate buffer, as the Notes plugin does.

NERD-commenter installs with a series of keyboard shortcuts for adding and removing comments based on various selections of words or lines or of cursor position. It supports nested comments, the use of alternate delimiters to mark comments, and selecting the positioning of delimiters.

Vim-abolish

Vim-abolish is so elegant that you wonder why no one thought of it before, but it's hard to describe. It has aspects of a word processor's spell checker or autocorrect, but might best be described as a configurable search and replace tool. What makes Vim-abolish so powerful is that it allows you not only to search and replace one word or spelling for another, but also to include all instances of a word. Upper case, lower case, noun and adverb, past and present tense, participles – all can be added to the search and replaced with a few dozen characters.

Admittedly, you might take a while to learn how to think in the terms necessary to set up a Vim-abolish command, and learning how to construct a command may take some time too. However, once you understand how Vim-abolish works, you will probably find it an invaluable proofreading tool.

VimOutliner

VimOutliner is exactly what its name suggests: a way to create hierarchial outlines. It is so popular that most major distributions package it, but you must add two lines to your .vimrc file before you can use it:

filetype plugin indent on
syntax on

Once you have VimOutliner up and running, creating a new level is as simple as starting a new line with an additional indent and adding a heading. Start a line below a heading with a colon (:), and text will be wrapped automatically and formatted when you export the current document to another format; start a line with a semicolon (;) and you can insert a line break. You can also collapse or open outline levels as useful, with everything color-coded for convenience.

You can save a VimOutiner document to a file with an .otl extension. However, VimOutliner also includes a series of Python scripts to save to awk, PDF, DocBook, HTML, and LibreOffice Impress formats – the latter presumably chosen because its Outline view is the best outline tool available in the office suite.

Gundo

Vim supports powerful undo functionality that tracks not only changes, but branches of changes. However, good luck finding a particular change with vanilla Vim so you can revert to it. If you don't want to simply revert one step at a time until you find what you are looking for, you can try to jump according to your best guess as to the time the change was made. Of course, if you guess wrong, or fail to keep track of what branch you are on, you are more likely to be confused than successful.

Gundo eliminates this confusion with a text-based representation of the undo tree and a few simple tools for navigation. At all times, you know exactly where in the undo tree you are, and that allows the undo functions to really come into their own.

Learning curves and alternatives

Add a few of these plugins, and functionally Vim soon begins to rival a word processor. However, no matter how many plugins you add, Vim is never going to match the convenience of an office suite, even if you use GUI-Vim. Some of these plugins include a lengthy list of keyboard commands that take practice and repeated use to master. However, the control and power they offer makes the effort worthwhile, especially if you work regularly without a display manager.

Clonezilla vs. FOG: The clone wars

Mayank Sharma — Mon, 18 Mar 2013 11:00:00 GMT

Computer cloning, also referred to as ghosting or imaging, involves setting up the operating system, drivers, software, and data on one computer, then automatically replicating the same setup on other computers. Clonezilla Server Edition and FOG are the most popular open source cloning systems. While both do similar jobs – clone and restore machines over the network using tools and services such as partimage, tftp, and PXE – they go about it very differently. Which is right for you depends on your network's configuration and composition.

The most visible difference between the two solutions is in their user interfaces. Clonezilla has an archaic ncurses-based interface, while FOG provides a web-based interface that you can access from any computer on the network. It also has a special UI that lets you deploy images from mobile devices.

While both FOG and Clonezilla SE can handle multiple images, FOG is better designed to manage multiple images. With FOG you can logically arrange images in groups, such as Accounts Department or Labs or Lounge. FOG also scales better; it features a storage manager that can control a group of NFS servers that all host images. Multiple servers can serve images, taking some of the stress off the main FOG server and speeding up cloning, especially in large networks.

In an enterprise where you have a large number of computers to deploy, the best course is to deploy images across the network, which requires setting up a dedicated cloning server. For Clonezilla SE, this means setting up a Diskless Remote Boot in Linux (DRBL) server by installing it on top of a Debian or CentOS server. If you manage a small network and don't want to earmark a computer as a permanent DRBL server, you can use the DRBL live CD to convert any machine into a server temporarily to broadcast images created by Clonezilla and stored on a local disk to any computer on the network.

To use a cloning solution effectively, it's best if computers on your network have similar hardware components. If your network is made up of computers with different components, you'd need to manage lots of different images.

As for FOG, its installation script creates the components required for an imaging server on top of a standard LAMP server. Unlike Clonezilla's DRBL server, the FOG server is designed to integrate with existing network infrastructure such as DHCP and DNS servers, although it can also set these up on its own. Setting up a FOG cloning server is pretty straightforward, but if you're using an existing DHCP server you must tweak it to forward PXE traffic to the FOG server, and configure the firewall to pass traffic to the server.

If all of the machines you want to image are not on your network, you need a solution that can clone offline machines as well. To do that, you create a master image, copy it on to a removable USB disk, and deploy from that. Clonezilla lets you save an image to a disk and then clone it either over the network or by physically going over to the computer. FOG doesn't – you can clone only from an imaging server.

After an image has been deployed, both Clonezilla SE (with DRBL-Winroll) and FOG (with its client service) can integrate the cloned machines into an Active Directory domain.

Other cloning options
Clonezilla SE and FOG are designed for cloning and deploying multiple computers over a network, but they both require setting up a dedicated server. This is overkill for situations where you just need to quickly image a disk, as you would for example when switching to a bigger hard drive. In such a situation, you might consider using one of these open source disk cloning applications:

Mondo Rescue – This tool is available in the repos of major desktop distros and can clone your computer to tapes, hard disks, USB devices, and NFS mounts. Unlike other tools, Mondo creates a specialized recovery live CD/DVD based on its own Mindi distro, customized for the computer being backed up.
Redo Backup and Recovery – Perhaps the easiest point-and-click cloning solution with a good-looking graphical interface. It can save cloned images on a disk attached to a computer or a network folder. The live CD also has tools such as testdisk and photorec to recover files. Its only limitation is that it's very sensitive to hardware changes and will restore only to identical hardware.
Trinity Rescue Kit – Popularly known as TRK, the live distro is primarily designed for rescuing and repairing Windows installations. It's chock-full of tools that can reset forgotten passwords, recover accidentally deleted files and partitions, and scan for viruses and rootkits. Additionally, you can also use the distro to clone Windows machines over the network.

While that's just about the extent of Clonezilla's capabilities, the FOG service can do a lot more. It can schedule tasks, such as deploying images, from its web interface, and perform several tasks other than imaging, such as installing and managing printers, tracking access to cloned machines, powering up a computer remotely using Wake-On-LAN, and installing applications remotely via a feature called snap-ins.

With snap-ins you can directly push the executable installer for a simple application to a Windows computer. For a larger application such as Microsoft Office that requires multiple files for installation, you need to use a tool like SFX Maker instead. FOG snap-ins don't have to be associated with an imaging task, so they can be used for things such as pushing software updates to client computers.

If you don't need to keep track of machines after cloning them, as might be the case, for example, in a repair or service center, Clonezilla gives you an edge over FOG. With Clonezilla you don't need to register a host with the server before it can be cloned. In contrast, FOG, by default, not only requires hosts to be registered before they are imaged, it also needs to register clients before it deploys images to them – though by using the Capone plugin you can override FOG's default behavior and clone a computer without first registering it with the server. Registering hosts has advantages, however, especially on enterprise networks. While registering hosts, FOG records the unique MAC address of their network cards, which reduces the chance of mistakenly deploying an image to the wrong machine.

While FOG is a fine enterprise imaging solution, it also has many of the features you'd want from a network management app. For example, in addition to options for registering and deploying an image, FOG's menu offers options to wipe a disk, restore deleted files, scan a disk for bad blocks, and run a virus scan using ClamAV.

Final word

If you're choosing between Clonezilla and FOG and you have machines that aren't connected to a network, you have no option but to use Clonezilla. You have to take an image with you and deploy it on each machine, but that's still faster than preparing each disk and installing and setting up an operating system on each individual client.

Clonezilla SE makes the most sense for networks with clients that don't require much administration after being imaged, such as those in Internet cafes, libraries, and school labs. If you need to look after machines after they've been cloned, it's best to go with FOG. Its post image options are excellent, and you can deploy machines without leaving your own desk.

The bottom line is that, while Clonezilla works best for offline computers and individuals, FOG has all the features you need to manage and deploy images in an enterprise.

Migrating from MySQL to PostgreSQL

Anatoliy Dimitrov — Thu, 14 Mar 2013 10:00:00 GMT

MySQL is a fine database, but the open source community has been nervous about its future since it was acquired by Oracle in 2009. Some users have sought alternatives in MySQL forks such as MariaDB. Others have considered migration to PostgreSQL, a mature database with a track record of more than 15 years. If you want to explore PostgreSQL, here's how to get started.

You can install and run PostgreSQL on the same server as MySQL, which makes the migration easier. A stable version of PostgreSQL is available through the package managers of all major Linux distributions. If you want to be sure to get the latest features and best performance, you can install the most recent version, which is usually available from third-party sources.

To install the current version of PostgreSQL, 9.2, on CentOS, first add PostgreSQL's repository to your .repo file with the command rpm -ivh http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/pgdg-redhat92-9.2-7.noarch.rpm. You can then install the PostgreSQL packages with the command yum groupinstall "PostgreSQL Database Server PGDG".

Before you start PostgreSQL for the first time, you have to have to initialize a new server instance and its database with the command service postgresql-9.2 initdb. After that you can start the server with the command service postgresql-9.2 start.

To configure PostgreSQL to start and stop with the system, run the command chkconfig postgresql-9.2 on.

Migrating data from MySQL to PostgreSQL

Working with PostgreSQL
PostgreSQL provides a command-line client, /usr/bin/psql, that's similar to the MySQL client /usr/bin/mysql. By default, PostgreSQL allows only the system user postgres to access the server, and not the root user. To use /usr/bin/psql you should switch to the postgres user with the command su postgres.

If you try to use /usr/bin/psql as root, you will get the error psql: FATAL: role "root" does not exist. This means the root user has not been given a PostgreSQL role, meaning it has no rights. While you could create a role for root, it's not recommended from a security point of view; stick to the special postgres user. For more information, check the PostgreSQL documentation for database roles.

Once you have PostgreSQL installed you can start your migration. To illustrate how you can migrate data from MySQL to PostgreSQL I will use the freely available sample employees database, which offers large amounts of diverse data. To import it into MySQL, first download the full archive, extract it, and run the command mysql -t < employees.sql.

Plenty of tools are available for converting MySQL data to PostgreSQL. mysql2pgsql, for instance, is a Perl script that takes as input a MySQL data dump file and converts it to PostgreSQL-compatible format. You can find other tools in the official PostgreSQL documentation about converting from other databases.

To begin the conversion, first create a dump of your MySQL data. For the employees database, run the command mysqldump employees > employees_mysql.sql. When the dump completes, start the PostgreSQL conversion with the command ./mysql2pgsql.perl employees_mysql.sql employees_postgre.sql. This command creates a new file called employees_postgre.sql that contains all the statements needed to import your old data into PostgreSQL.

To perform the data import, first create the employees database in PostgreSQL with the Linux command /usr/bin/createdb employees. Then use the command /usr/bin/psql employees < employees_postgre.sql to import the previously converted dump.

Once the import completes you should find that the databases in MySQL and PostgreSQL are the same. You can make a simple check by looking at the number of rows in the salaries table. First check it in MySQL:

mysql> select count(*) from salaries;
+----------+
| count(*) |
+----------+
|  2844047 |
+----------+

Then in PostgreSQL:

employees=# select count(*) from salaries;
  count  
---------
 2844047

You can try more complex comparisons, such as looking for specific records inside the employees table. In MySQL an example query looks like:

mysql> select * from employees where first_name = 'Ioana' and last_name = 'Tsukuda';
+--------+------------+------------+-----------+--------+------------+
| emp_no | birth_date | first_name | last_name | gender | hire_date  |
+--------+------------+------------+-----------+--------+------------+
| 496865 | 1961-09-17 | Ioana      | Tsukuda   | M      | 1994-10-13 |
+--------+------------+------------+-----------+--------+------------+

Its analogue in PostgreSQL with exactly the same syntax should return the same result:

employees=# select * from employees where first_name = 'Ioana' and last_name = 'Tsukuda';
 emp_no | birth_date | first_name | last_name | gender | hire_date  
--------+------------+------------+-----------+--------+------------
 496865 | 1961-09-17 | Ioana      | Tsukuda   | M      | 1994-10-13

Connecting your application to PostgreSQL

Once you ensure your data is properly migrated to PostgreSQL you can proceed with connecting your application to it.

Some applications, such as Drupal, support PostgreSQL by default, while others, such as WordPress, require you to install an additional plugin. In any case, PostgreSQL is widely supported, and it should be relatively easy to find support for your application and its programming language.

When you create applications that use a database, you should design them to allow later developers to easily migrate from one database to another. If your applications use a database abstraction level, you should be able to just change the database driver and the connection string and start using a different database, such as PostgreSQL. For example, if you are using PHP on CentOS, install the php-pgsql package to get the PostgreSQL database module (driver) for PHP. Once you have it installed (yum install -y php-pgsql), you can establish a PostgreSQL connection in PHP with a connection string similar to $connection=pg_connect('dbname=employees user=test password=example');

If your application lacks a database abstraction layer, all SQL statements might be hard-coded with MySQL specifics. In such a case you'll have a hard time moving away from MySQL because almost every SQL statement may require adjustment. But nothing is impossible with enough determination, time, and budget. Even if immediate migration looks difficult, it's good to know that there is an alternative such as PostgreSQL to which you can turn in future.

Three simple tools for backing up MySQL databases

Mayank Sharma — Mon, 11 Mar 2013 10:00:00 GMT

One of the most important responsibilities of any administrator is to make regular backups. But unlike backing up regular data, backing up databases isn't a straightforward affair. Designing an effective strategy for taking backups of production MySQL servers depends on a lot of factors. Depending on your situation, your best MySQL backup tool might be mysqldump, AutoMySQLBackup, or MySQL Workbench.

The easiest and safest way to back up a database is to shut down the MySQL server. This ensures you have a consistent copy of the data, since you don't have to worry about parts of the database being modified while it's being backed up. The backups themselves will be completed faster than on a production server, as the server isn't handling requests from other applications.

That said, taking a database server offline isn't a practical solution for most enterprise deployments. So you'll want to use a solution that lets you take backups of a production server without taking it offline.

Types of backups

Hot backup – A hot backup is a backup of a running database. During a hot backup, neither reads nor writes to the database are blocked. This is the best option for huge databases that can take days or weeks to back up, as well as deployments that can't afford any downtime. However, it's risky, and if not implemented properly it can negatively affect performance.
Warm backup – A warm backup is also a backup of a database that is still running, but during a warm backup, only read queries are allowed; write queries are blocked from modifying a database. This option works well for most deployments, and you can use a combination of tools to speed up the backup process for large databases.
Cold backup – A cold backup is a backup performed while the database is shut down. This makes it easy to make a consistent copy of your data. The biggest disadvantage of this method is that the server is not accessible during the time the backup is performed. This approach works best for databases that hold mostly static information, such as a database of employees.

Then there's the question of how you want to back up the database. There are two major ways to back up MySQL's data: either by creating a logical backup or by copying the raw files. A logical backup saves information that represents the database structures. It's made up of SQL statements such as CREATE DATABASE and INSERT. The obvious advantage of such a backup is that you can make changes to the data with a text editor or using tools such as grep. A raw backup, on the other hand, is a backup of the disk partitions of the database server.

Both methods have advantages and disadvantages. Logical backups are simpler, you can restore them by simply piping them into the MySQL server, and logical backups are more compatible between different versions of MySQL. It's actually a standard practice to restore a database from a logical backup when upgrading from one MySQL release to another. But depending on the size and complexity of the database, taking a raw backup and restoring it can be much faster than working with a logical backup.

Take backups with mysqldump

The default backup tool that ships with MySQL server is the command-line mysqldump program. With it you can create a logical backup of all the databases on a server, or individual databases, or only particular tables.

The output dumped by the tool contains all the commands required to recreate the database, complete with tables filled with the original data.

Using the tool is pretty simple:

$ mysqldump --user=root --password --all-databases > dump.sql

This command creates a file named dump.sql in the current directory that contains all the SQL commands to recreate the database and all its tables and fill it with the original data. You can browse the contents of the file with the less command. Here's a brief snippet:

$ less dump.sql
-- MySQL dump 10.13  Distrib 5.5.29, for linux2.6 (i686)
--
-- Host: localhost    Database: 
-- ------------------------------------------------------
-- Server version       5.5.29

--
-- Current Database: `drupal7`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `drupal7` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `drupal7`;

--
-- Table structure for table `actions`
--

DROP TABLE IF EXISTS `actions`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `actions` (
  `aid` varchar(255) NOT NULL DEFAULT '0',
  `parameters` longblob NOT NULL,
  `label` varchar(255) NOT NULL DEFAULT '0',
  PRIMARY KEY (`aid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Stores action information.';
/*!40101 SET character_set_client = @saved_cs_client */;

The tool has an extensive list of options, some more commonly used than others. If you want to back up only particular databases, instead of the --all-databases option use the --databases option and specify the name of the databases you want to back up. To ensure the backups are consistent, use the --lock-all-tables option to lock all tables across all databases.

The mysqldump tool works on all the MySQL storage engines. It can do a warm backup at acceptable backup speeds but drags its feet when restoring the database. It works best for smaller databases, and its biggest advantage is that it's scriptable.

Script backups with AutoMySQLBackup

If mysqldump works for your MySQL setup, you can either script it for taking automated backups, or use AutoMySQLBackup. It's a clever little script, based on mysqldump, that does a lot of heavy lifting. It can back up a single database or multiple databases and save them in separate compressed files. Using the script you can easily set up incremental backups that are automatically rotated.

To use the script, download and unpack it. It has an install.sh script that installs the script under /etc/automysqlbackup directory. Edit the configuration file under this directory as per your setup and requirements. The various configuration options are explained in the script itself, but here are the most important ones.

To make sure the script can connect to the MySQL server, make sure you customize the following variables as per your setup:

CONFIG_mysql_dump_username='root'
CONFIG_mysql_dump_password='not$telling'
CONFIG_mysql_dump_host='localhost'

If you want you can also change the directory where the backups are stored by specifying the location in the CONFIG_backup_dir variable. Similarly, if you want only a particular list of databases to be backed up instead of everything, specify them with the CONFIG_db_names variables. Or, you can specify the databases and tables you wish to exclude with the CONFIG_db_exclude and CONFIG_table_exclude variables.

To set up a rotation policy, you need to tweak the appropriate variables. For example, if you want to do monthly backups on the 28th of every month, weekly backups on Sundays, and keep daily backups for seven days, weekly backups for 14 days, and monthly backups for 30 days, change the variables thusly:

CONFIG_do_monthly="28"
CONFIG_do_weekly="7"
CONFIG_rotation_daily=7
CONFIG_rotation_weekly=14
CONFIG_rotation_monthly=30

Once the file has been configured, test it by invoking it manually with # automysqlbackup /etc/automysqlbackup/dbserver.conf, assuming dbserver.conf is the name of our AutoMySQLBackup configuration file. It should create compressed logical backups of the databases in the appropriate directories.

When you're satisfied with the command, you can automate the process by creating a backup script and use cron to run that script daily. The README file bundled with AutoMySQLBackup includes a simple but effective backup script.

Backups with MySQL Workbench

Both mysqldump and AutoMySQLBackup rely on command-line tools to restore the dumps. If you prefer a graphical user interface, there's a GUI backup tool you can use to restore logical backups – among many other things.

If you work with MySQL databases, chances are you use the graphical MySQL Workbench database design tool. In addition to querying and designing databases, you can also use the tool to create (and restore) logical backups of the databases.

MySQL Workbench's main interface is divided into three categories based on function. The backup options are housed under the Server Administration section. To back up your databases, connect to the server from under this category. You will then be switched to a tab that lists all the server administration tools.

Click on the Data Export option in the left pane. Select the databases and tables you want to back up from the list shown in the right pane. When specifying the destination of the backups, MySQL Workbench lets you specify whether you want to export the backup to a folder (where each table is written to a separate file) or to a single self-contained file.

To restore databases, switch to the Data Import/Restore option, point to the location where you've stored the backup dumps, and Workbench will display the backed-up databases and tables. You can select the tables you wish to import and they will be restored.

Other backup options

These three options for creating warm logical backups of databases should work for most enterprise users, but if you need to create a consistent hot backup of your databases, you can turn to the snapshot functionality of your filesystem.

Filesystems such as FreeBSD's ZFS have native support for creating snapshots, and you can also create them using the Logical Volume Manager (LVM) on Linux.

But filesystem snapshots are still not an ideal solution for taking hot backups. For that, you'd need a commercial product such as MySQL Enterprise Backup or Zmanda Recovery Manager.

Replacing MySQL with MariaDB

Manolis Tzanidakis — Thu, 07 Mar 2013 11:00:00 GMT

For years, MySQL has been the king of open source database servers. It powers a large part of the web and numerous applications worldwide. However, concerns about the future of MySQL since its acquisition by Oracle, combined with an increasing demand for performance and scalability, have driven people to consider alternative options, such as PostgreSQL and MongoDB. Switching to either of those alternatives, however, is not a simple proposition. MariaDB, by contrast, offers enhanced performance in a DBMS that can be a drop-in replacement for MySQL.

MySQL's creator, Michael "Monty" Widenius, started working on MariaDB in 2009 as a fork of MySQL, and supports it via his company, Monty Program Ab. In late 2012, the nonprofit MariaDB Foundation was founded to support the development of the database and manage the MariaDB trademark. All code for the project is open source, distributed under a GPLv2, LGPL, or BSD license, and the Foundation is commited to keep the development open to the community. Besides transparency, the project also aims to improve the quality of the database with timely bug fixes and stability improvements. The next versions of Fedora and openSUSE will include both MySQL and MariaDB, but will offer MariaDB as the default.

Compatible, yet improved

The biggest strength of MariaDB is, of course, its compatibility with MySQL. It is a drop-in replacement for the same version of MySQL; that is, if an application works with MySQL 5.5, it should work with MariaDB 5.5 without any modification. Even the names of the command tools are the same – you interact with MariaDB databases via the mysql command, and perform backups with mysqldump. Developers and system administrators should feel right at home, with minimal adjustments to their familiar workflow. I've tested some open source applications by replacing MySQL with MariaDB – including WordPress, Drupal, phpMyAdmin, Adminer, and software I've written – without problems. The project offers a list of open source applications that officially support MariaDB and a detailed compatibility list.

The core of any database system is its data storage engine, and MariaDB offers several powerful engines to choose from, including the standard MyISAM, Blackhole, CSV, Memory, and Archive engines. A new storage engine, Aria (previously known as Maria), was developed with the goal of becoming the default non-transactional and transactional engine for MariaDB, and might be included in future versions of MySQL too. The current version of Aria offers a crash-free alternative to MyISAM, and support for transactions and additional features is planned for the next version. The Percona XtraDB engine is included as a replacement for InnoDB, MySQL's default engine since 5.5. XtraDB is a faster version of InnoDB, designed for more efficient scalability on modern multicore systems. It is also backward-compatible with InnoDB. The current development version, MariaDB-10.0, also includes the Apache Cassandra NoSQL storage engine, and more NoSQL storage options are promised for future versions. You can find all the storage engines included in MariaDB in the list of feature differences between the two databases.

MySQL is not a slow database, and MariaDB aims to improve performance further. The project's blog offers some benchmarks comparing MariaDB 5.5 and 10.0 with MySQL 5.5 and 5.6. MariaDB outperforms MySQL in most of the tests. A post by Asher Feldman of Wikimedia largely verifies the results of that benchmark.

One feature of MariaDB that can greatly improve performance is the option of running server threads in a thread pool. This feature is available in the commercial, closed source versions of MySQL as a plugin, but the MariaDB implementation is more efficient and built-in. Thread pools offer better system resource management and can make a big difference in most work loads, especially on busy servers. To enable a thread pool, just add thread_handling = pool-of-threads to the MariaDB configuration file, my.cnf, on Unix-like systems. It is enabled by default in the Windows version.

MariaDB also includes user statistics, a diagnostic feature that is also available as a patch for MySQL 5.5. Combined with an automated, scripted process, userstat can be a powerful tool for monitoring server activity. To enable statistics to be gathering, add userstat = 1 in my.cnf. Userstat adds several new information schema tables and new FLUSH and SHOW commands. Enter SELECT * FROM INFORMATION_SCHEMA.USER_STATISTICS\G or SHOW USER_STATISTICS\G at the mysql prompt to view information about users' activity on your server.

Installation

Future versions of most Linux distributions and BSDs will ship MariaDB packages in their official repositories, but at the moment, distro availability is limited. However, the project offers prebuilt packages for popular distributions and a nice repository configuration tool to help you select the right repository for your distribution. Considering the slow pace of updates in server-oriented distributions, including RHEL and CentOS, I don't expect to see MariaDB RPMs in official repos soon. Luckily, MariaDB offers long-term support for updates on RHEL, CentOS, and Ubuntu LTS. If you just want to test how MariaDB behaves on your current setup without disrupting your installation, you can install it alongside MySQL.

It's simple to install MariaDB on a CentOS 6 (64-bit) server. If MySQL is already installed on the server and you want to replace it with MariaDB, you should back up /etc/my.cnf and /var/lib/mysql beforehand, then uninstall MySQL with yum erase mysql-server mysql. Using the repo config tool, create a repo entry for yum and saved it as /etc/yum.repos.d/MariaDB.repo:

[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

After that, just run yum install MariaDB-server MariaDB-client. You can also try yum install mysql to install MariaDB or just yum update if MySQL is already installed, but currently doing so installs the Galera version of MariaDB, which I'll discuss in a moment, so it is better to be explicit about what you want. If you prefer to create your own RPM packages from source, you can follow the project's detailed instructions.

To migrate your previous MySQL installation, overwrite the fresh /etc/my.cnf with your backup. Optionally, you can also break your my.cnf file into multiple configuration files in /etc/my.cnf.d. After starting the server process you should run mysql_upgrade, as you would with any MySQL updates between major versions. You cannot undo this last step, so make sure you have backups of your databases before proceeding. Running mysql -e 'SHOW GLOBAL VARIABLES LIKE "version"' will show you the version of your newly installed MariaDB server.

MariaDB for the enterprise

Large database installations often utilize multiple servers with replication, for extended availability and increased performance. The standard replication mechanisms of MySQL and PostgreSQL are, mainly, asynchronous. Writes to the database occur on a master server initially, and then propagate to one or more slave servers, which are used only for reading data. Another approach is the multi-master topology, which can offer seamless failover, at least in theory. However, due to the asynchronous nature of current replication methods, it is easy for servers to get out of sync and cause more problems.

The MariaDB Galera cluster aims to fix this problem by offering true synchronous replication between multiple master servers. Any changes to the database are committed to all nodes simultaneously, via certification-based replication. Synchronous replication offers great benefits in enterprise installations. Applications with many client write transactions can scale to previously unimagined levels by distributing the transactions across the slave nodes of the cluster. Even traditional master-slave designs can benefit from synchronous replication with zero slave lag.

The Galera cluster is still under development, although at the time of this writing it is considered release-candidate quality. To test it, you need at least three server nodes to form the database cluster. Instead of the MariaDB-server package you need to install MariaDB-Galera-server, along with the galera package, which offers the replication engine (wsrep). The current implementation supports only InnoDB and XtraDB.

Besides speed and scalability, enterprise database installations require premium support services. An increasing number of service providers offer support for MariaDB. Corporations with large investment in MariaDB can also support the Foundation and help move the development of the database forward.

MariaDB has already made its mark in the database ecosystem as a worthy replacement for MySQL, and you can expect its adoption to keep increasing in the future. After all, having a tuple of high-quality open source databases to choose from is not a bad thing.

BackTrack and its tools can protect your environment from remote intrusions

Anatoliy Dimitrov — Mon, 04 Mar 2013 11:00:00 GMT

Attackers commonly look for remote vulnerabilities in order to compromise the resources on your network. BackTrack Linux, a security-testing distribution, can help you inspect your network and servers for remote security weaknesses and potential vulnerabilities.

BackTrack, which is based on Ubuntu, bundles all the tools necessary for penetration testing and security audits. You don't really have to run BackTrack to use the tools it provides, but booting up a BackTrack live CD lets you get started in no time.

A word of caution before you start: Scanning your network may take system and network resources. Make sure to fully coordinate your actions inside your company and inform your management.

Exploring Your Environment With Nmap

One of the first tools you'll want to try is Nmap, a powerful, smart, command-line network scanner. In essence, Nmap shows open ports and some information about the services listening on these ports.

Nmap comes with a lot of useful options enabled by default. For instance, port randomization, which randomizes the order of the ports being scanned, prevents simple intrusion detection and protection systems from detecting and blocking you while you're probing. Some other useful options include:

--script – performs the scan with the aid of a set of scripts. Nmap comes bundles with a few sets of scripts categorized according to their use, such as for DOS, brute force, and other weakness detection. You can find the full list on the home page of the Nmap Scripting Engine. If you are not sure what scripts to use, choose the default, which is more informative and less intrusive than some other options.
-p – specifies which ports you are interested it. Most legitimate services run within the port range of 1:10000 for UDP and TCP protocols. Scanning a larger range requires more time and resources.
-sV – shows service and version information for the open ports. This is useful to show whether a service is listening on a non-default port, such as Apache listening on TCP port 8080, and the version information may indicate outdated software and potential vulnerabilities. If you detect such outdated applications with Nmap, make sure to patch them as soon as possible.
target – defines the target host or network as the final argument of your Nmap command.

Put the above options together and you get a command like:

nmap --script=default -p U:1-10000,T:1-10000 -sV 192.168.1.0/24

In this case the target is an internal, private network (192.168.1.0/24). If you are looking for truly remote vulnerabilities you should perform the scan from outside your local network. Still, internal scans are also useful and can show unnecessary exposure of services, which is always a security risk.

The output of this example is similar to:

...
Nmap scan report for example.org (192.168.1.102)
Host is up (0.0017s latency).
Not shown: 9997 closed ports
PORT     STATE SERVICE     VERSION
22/tcp   open  ssh     OpenSSH 6.0p1 Debian 3 (protocol 2.0)
...
80/tcp   open  http    Apache httpd 2.2.22 ((Debian))
|_http-methods: No Allow or Public header in OPTIONS response (status code 200)

...
3306/tcp open  mysql   MySQL 5.5.28-1
...

The above excerpt from the Nmap output shows information for SSH, HTTP, and MySQL services running on 192.168.1.102. It clearly shows the name and the version of each software – OpenSSH 6.0p1, Apache httpd 2.2.22, and MySQL 5.5.28-1.

Limiting the network exposure

For each exposed service, you should decide whether it's really necessary for it to be exposed. If not, one option is to make sure the service listens only on the local loopback interface (127.0.0.1). This is a good option for the MySQL service when it serves only local requests. To do that for MySQL, edit the my.cnf file. When MySQL listens on all interfaces, the bind-address directive looks like this: bind-address = 0.0.0.0. To make it listen only on the local interface, change it to bind-address = 127.0.0.1.

Furthermore, MySQL lets you explicitly limit the remote hosts from which a user is allowed to connect as part of MySQL's connection verification process. Such an additional security restriction helps guard against brute-force attacks but usually does not help against remote software vulnerability exploits.

You can also use a firewall to restrict access to a service. When properly configured, a firewall can allow only certain hosts to connect to a given service. An example iptables command for MySQL looks like this: iptables -I INPUT -s 192.168.1.1 -p TCP --dport 3306 -j ACCEPT; iptables -I INPUT -p TCP --dport 3306 -j DROP. This sample command allows MySQL connections only from the IP address 192.168.1.1.

As a last resort, you may decide to change the default port for a service. This is usually feasible for sensitive services that have to remain relatively hidden but still fully accessible from outside. This is often the case with SSH when system administrators want to have access from everywhere. You can change the SSH port to a port above the commonly scanned range (1-10000). For example, if you set the SSH daemon to listen on port 19999, it's less likely to be detected, but you will be still able to access the service from anywhere as long as you know this port. To change the SSH port, edit the directive Port in the file /etc/ssh/sshd_config and restart the service. Don't forget to specify this port when connecting later with your SSH client (ssh -p in Linux shell).

These are the simplest ways to limit the remote connectivity and protect from external attacks. If you are interested in more advanced techniques, consider port knocking as one interesting idea and option.

Limiting the exposed information

After dealing with the SSH and MySQL services from our example, only the web service remains. It is supposed to be fully accessible for the outside world on the standard HTTP TCP port 80, and we cannot hide it under a different port nor restrict the access to it. Our only choice is to disclose as little information about it as possible.

According to the above example, the Apache version is Apache httpd 2.2.22 ((Debian)). This gives an attacker plenty of information – web server name, version, and even operating system. You can limit this information by changing the Apache configuration and setting the server token setting as ServerTokens ProductOnly. After this, when you run the scan again, you'll see for version only Apache httpd.

Even with our basic Nmap example we got more information than just the version. For Apache, Nmap also showed that the OPTIONS header is allowed and there is no restriction on the http methods – |_http-methods: No Allow or Public header in OPTIONS response (status code 200). From a security point of view it's important to restrict the allowed HTTP methods to only the ones your site really needs and thus give attackers fewer options.

For example, the TRACE HTTP method echos back user input. Clearly, this is a feature useful for debugging but not needed in a production web server. The TRACE method is used in certain attacks by allowing access to sensitive information, so it should almost always be disabled. In fact, most average web servers should support only two HTTP methods: GET and POST. To forbid the rest, use the Apache directive limitexcept like this:

    Order deny,allow
    Deny from all

Hiding the server token and forbidding unneeded communication methods is a good start, but it's far from enough. You also need an application firewall, which offers more thorough protection. Almost all publicly accessible services have such a firewall solution available. In the case of Apache, check the article How to protect your web server with ModSecurity.

Examine your Nmap output and proceed similarly for any publicly exposed services. You should be able to foil the usual generic untargeted attacks that randomly scan the Internet for outdated or misconfigured software.

Delving deeper into penetration testing with Nessus

Nmap is powerful and lets you do some serious penetration testing of your environment, but it's not as easy to use and complete as some more advanced vulnerability scanners. Nessus, for instance, is a commercial vulnerability scanner that allows limited free use for home users. To start using it in BackTrack you have to register for a license key first and follow a few preliminary steps as described in the official Nessus on BackTrack guide.

Once you have Nessus up and running in BackTrack, you control it from an intuitive web interface accessible at https://localhost:8834. The web interface allows you to easily configure your penetration test and generate a professional-looking report upon completion.

Nessus detects more than 50,000 vulnerabilities on all exposed levels, from misconfigured services to outdated software. Once it detects a vulnerability, it reports all the related information and, most importantly, suggests a solution. This saves you time, ensures that you follow the best practices for resolving a problem, and that you comply with the highest security standards.

Nessus is the most preferred penetration testing solution for enterprise users. Still, Nmap remains the Swiss Army Knife for network scanning, as it provides the fastest and simplest path to exploring your environment without binding to licenses and spending significant amounts of money.

No matter what application you choose, the most important thing is to understand the concept of remote security. Don't unnecessarily expose services, nor any information that is not explicitly required. That should ensure you don't attract publicly circulating threats that may exploit zero-day vulnerabilities.

Ant buildfiles: A look under the carapace

Juliet Kemp — Thu, 28 Feb 2013 11:00:00 GMT

You may already use the Apache Ant Java-based build tool to build projects with a presupplied build.xml file. But what if you want to do something more complicated? Knowing how Ant works under the hood will let you make the most of its capabilities. Let's dissect an Ant buildfile and examine its components.

Ant operates in a different way than other command-line build tools. Instead of using shell commands to extend your tool, you extend Ant by writing Java classes to run particular tasks – although Ant comes with a huge list of available tasks. Instead of constructing a shell command to run a particular task configuration, you configure Ant in XML files. The XML creates a tree of tasks to be executed to build particular defined targets, and each task is run by its own object. The crucial implication of this is that Ant is entirely cross-platform, unlike make and its ilk.

I'll assume that you already have Ant installed for this tutorial; if not, get it from its website or your package manager.

Basic Ant building

If you just type ant on the command line in an empty directory, you'll get an error message telling you that there's no build.xml file. build.xml is the file Ant needs in order to know what it's supposed to do. Let's look at the structure of a basic buildfile:

<?xml version="1.0" encoding="UTF-8"?>
<project name="HelloAnt" default="compile" basedir=".">
  <property name="src.dir" value="."/>
  <property name="build.dir" value="build/"/>

  <target name="clean">
     <echo>Cleaning up ${build.dir}</echo>
     <delete dir="${build.dir}"/>
  </target>

  <target name="init" depends="clean">
     <echo>Creating build directory ${build.dir}</echo>
     <mkdir dir="${build.dir}"/>
  </target>

  <target name="compile" depends="init">
     <echo>Compiling source from ${src.dir} to ${build.dir}</echo>
     <javac srcdir="${src.dir}" destdir="${build.dir}"/>
  </target>
</project>

The root element for an Ant buildfile is project: the rest of the file forms part of this element. The project element itself has a name, default target, and base directory, none of which are required. The buildfile then defines source and build directory properties, which it refers to later in the file. This means that if you change your directory structure, you only need to alter one place in the file. The actual work is done by three targets: clean (which deletes the build directory), init (which recreates the build directory), and compile.

Each target comprises a number of tasks. Ant has a huge list of built-in tasks available, and you can define more of your own by writing a Java file, though here we're just using built-in ones. echo, delete, mkdir, and javac all do what you'd expect; you can check out the documentation for more information. Note that the javac task will fail if the destination directory doesn't already exist, hence the need to create it in the init target.

You'll also notice the depends attribute on two of the tasks. This sets up a chain of tasks; if you tell Ant to execute Task 1 which is dependent on Task 2, Ant will run Task 2 for you first. You can also chain multiple dependencies, as here, where compile depends on init which depends on clean. They'll all be executed in the correct order. Tasks can also have multiple dependencies, and Ant will sort out everything for you.

To test this buildfile, create a basic HelloWorld.java file in your base directory, then type ant. Since our default target is compile, it should clean, init, and compile your program.

Properties

Instead of having all the properties at the start of build.xml, you can instead keep them in a properties file. This is usually called build.properties, and ours would look like this:

 
src.dir=. 
build.dir=build
test.dir=${build.dir}/test

As you can see, you can refer to earlier properties within the file itself. You then include this build.properties file in build.xml:

<project name="HelloAndroid" default="help">
  <property file="build.properties"/>
  ...
</project>

This modular structure is more maintainable when you have a significant number of properties, so it's good to get into the habit of using it.

You can also override properties on the command line. Try this command (note the lack of a space between the flag and the value):

ant -Dbuild.dir=test

Your project will be rebuilt into test/. Note that in this case, the old classfiles in build/ will not be deleted, because the clean target will be run on test/ instead.

More on targets

Ant lets you create "hidden" targets. For example, let's add a couple of targets:

<project name="HelloAnt" basedir="." default="compile">
  ...
  <property name="jar.dir" value="jar/"/>
  <property name="jar.jarfile" value="HelloAnt.jar"/>

  ...

  <target name="-jarinit">   
    <tstamp>   
      <format property="TODAY_UK" pattern="yyyy-MM-dd_HH.mm" locale="en_GB" />
    </tstamp>   
  </target>   

  <target name="jar" depends="-jarinit"   
          description="Creates the binary distribution.">   
    <mkdir dir="${jar.dir}/${TODAY_UK}" />   
    <jar basedir="${build.dir}"   
         destfile="${jar.dir}/${TODAY_UK}/${jar.jarfile}" />   
  </target>   
</project>

The target -jarinit can't be called from the command line, which makes sense, as all it does is to set up a property, TODAY_UK, which is a timestamp with a particular format. However, the target jar, which can be called from the command line, depends on -jarinit, which sets up the directory the jar file will be saved in. If you remove that depends="-jarinit" attribute, the task will still run, but it'll use the directory ${TODAY_UK} rather than substituting in the property. (This is one property that has to stay in build.xml rather than build.properties.)

In fact, jar really should depend on compile as well as jarinit, since you can't create a jarfile from a nonexistent build directory and don't want to create one from an old build directory! Change the depends property to read depends="-jarinit,compile" and run ant jar again. You'll see Ant run -jarinit, then traverse the tree of tasks to run clean, init, compile, and then jar.

Help!

Now try typing ant -projecthelp. You should get output like this:

 
Buildfile: /home/juliet/coding/ant/build.xml

Main targets:

 jar  Creates the binary distribution. 
Default target: compile

Only one of our tasks shows up! This is because the jar task is the only one with a description property. Add descriptions to the other tasks, and try ant -projecthelp again:

 
Buildfile: /Users/juliet/coding/ant/build.xml

Main targets:

 -jarinit  Creates a timestamp.
 clean     Cleans up old build directory.
 compile   Compiles source.
 init      Creates build directory.
 jar       Creates the binary distribution. 
Default target: compile

Note that even the hidden task will show up if you give it a description.

It's also good practice to create a help target – something like this:

<target name="help" depends="usage" />   

<target name="usage" description="Display usage information.">   
  <echo message="  Execute 'ant -projecthelp' for build file help." />   
  <echo message="  Execute 'ant -help' for Ant help." />   
</target>

This shows another feature of Ant targets: You can have a "target alias," meaning a target that only invokes another target. Here, all help does is invoke usage. This means that the user will get help from either ant help or ant usage. You could also alias build to compile, for example. This also shows another way of using the echo task.

It's good practice to change your default to help or to something else that doesn't do much; if you make the default be compile, a user may be unhappy when Ant blows away old build files, or takes ages to run for a large project.

This tutorial should give you enough information to get started editing an Ant build.xml file if you want to tweak your builds a little. Once you feel comfortable, you can also try setting up an Ant file for other tasks you do regularly; there's no need to limit yourself to Java compilation!