VladGh.com blogs

Varnish 3 and Apache in Ubuntu 12.04 LTS Precise

Vlad — Mon, 06 Aug 2012 23:33:31 +0000

This article is a Partner Post by the UK based PartyCasino.com. Being one of the best ranked Internet casinos, they offer various slot games. Even if I am not an avid player, I can admit that the revenue is tempting and I understand why this is such a fast growing and profitable industry.

Varnish is an HTTP accelerator, which serves static content from disk or RAM memory. Unlike Squid or other similar software that was created to be mainly a proxy, Varnish has resolved the problems of having data in the virtual memory, and was created from ground up as an HTTP accelerator. It is also very configurable allowing a multitude of uses.

You will probably ask how is this different from memcached, used in conjunction with nginx. The answer is that they are two completely different systems, varnish is an accelerator while memcached is an in-memory key-value store for small chunks of arbitrary data.

Now let's install a typical LAMP system, that everybody blames as being slow and memory consuming. By simply adding varnish as a buffer in front of it, you will notice not only a more linear memory consumption but also an increase in the speed.

We will start by installing MySQL:

apt-get install mysql-server mysql-client

Now bring the example configuration. The path might be different in your case though.

cp -f /usr/share/doc/mysql-server-5.5/examples/my-medium.cnf /etc/mysql/my.cnf
/etc/init.d/mysql restart

Install apache and enable a few modules:

apt-get install apache2-mpm-prefork apache2-utils

a2enmod rewrite

a2enmod expires

a2enmod headers

Slow it down by limiting the max clients in the /etc/apache2/apache2.conf file:

<IfModule mpm_prefork_module>

        StartServers 10

        MinSpareServers 10

        MaxSpareServers 20

        ServerLimit 50

        MaxClients 50

        MaxRequestsPerChild 5000
</IfModule>

Install PHP and a few extra modules, such as suhosin for increased security and APC, an opcode cache:

apt-get install php5 php5-mysql libapache2-mod-php5 php5-suhosin php5-dev php-pear php5-curl php5-cli php5-gd php-apc

Enable the APC engine:

echo"apc.enabled = 1

apc.shm_size = 128

apc.shm_segments=1

apc.write_lock = 1

apc.rfc1867 = On

apc.ttl=7200

apc.user_ttl=7200

apc.num_files_hint=1024

apc.mmap_file_mask=/tmp/apc.XXXXXX

apc.enable_cli=1

apc.slam_defense = Off

" >> /etc/php5/conf.d/apc.ini

And now Varnish:

curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
echo "deb http://repo.varnish-cache.org/debian/ lucid varnish-3.0" >> /etc/apt/sources.list
apt-get update
apt-get install varnish

Now, before you start the servers, let's adjust Apache and Varnish to work together. First change the default port for your virtual host in apache:

vi /etc/apache2/sites-available/default

Change from <VirtualHost *:80> into <VirtualHost *:8008> (this is just a preference)

The configuration daemon for Varnish is located in /etc/default/varnish. Here you can pass the startup options:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256M \
             -u www-data \
             -g www-data"

-f /etc/varnish/default.vcl is the configuration file, and -s malloc,256M means that I allowed Varnish to use 256M of my RAM. Further configurations are very well documented in the comments of this file. For me, setting Varnish to use the disk instead of the virtual memory wasn't too much of an improvement. Of course if you have more memory to spare, you can allow it to use more. -a 80 is the port on which varnish listens, which is the default 80 (we want it in front of Apache), and -T localhost:6082 is the administration panel.

Now let's get to the configuration file (/etc/varnish/default.vcl). This is a very simple use of Varnish which strips the cookies of some static files, like jpg and png, allowing them to be cached. In my case this works best, and it also gives me a header information of what was cached and what not (if you check the headers with something like FireBug). Of course you can find even better configurations available. Having a specific configuration for your platform, like Drupal or WordPress, is not a bad thing either.

backend apache {

  .host="127.0.0.1";

  .port="8008";

  .connect_timeout= 600s;

  .first_byte_timeout= 600s;

  .between_bytes_timeout= 600s;
}
acl purge {

  "localhost";
}
acl ban {

  "localhost";
}
sub vcl_recv {

  // Strip cookies for static files:

  if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm|xml)$") {

    unset req.http.Cookie;

  }
  // Remove specific cookies:

  set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");

  set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");

  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

  if (req.http.cookie ~ "^ *$") { unset req.http.cookie; }

  if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; }

  if (req.http.Cookie == "") { remove req.http.Cookie; }
  // Normalize Accept-Encoding header (straight from the manual: https://www.varnish-cache.org/docs/3.0/tutorial/vary.html)

  if (req.http.Accept-Encoding) {

    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {

      // No point in compressing these

      remove req.http.Accept-Encoding;

    } elsif (req.http.Accept-Encoding ~ "gzip") {

      set req.http.Accept-Encoding = "gzip";

    } elsif (req.http.Accept-Encoding ~ "deflate") {

      set req.http.Accept-Encoding = "deflate";

    } else {

      // unkown algorithm

      remove req.http.Accept-Encoding;

    }

  }
  if (req.restarts == 0) {

    if (req.http.x-forwarded-for) {

      set req.http.X-Forwarded-For = req.http.X-Forwarded-For;

    } else {

      set req.http.X-Forwarded-For = client.ip;

    }

  }
  if (req.request != "GET" &&

    req.request != "HEAD" &&

    req.request != "PUT" &&

    req.request != "POST" &&

    req.request != "TRACE" &&

    req.request != "OPTIONS" &&

    req.request != "DELETE") {

      /* Non-RFC2616 or CONNECT which is weird. */

      return (pipe);

  }
  if (req.request != "GET" && req.request != "HEAD") {

    /* We only deal with GET and HEAD by default */

    return (pass);

  }
  if (req.http.Authorization || req.http.Cookie) {

    /* Not cacheable by default */

    return (pass);

  }
  if (req.request == "PURGE") {

    if (!client.ip ~ purge) {

      error 405 "Not allowed.";

    }

    return (lookup);

  }
  if (req.request == "BAN") {

    if (!client.ip ~ ban) {

      error 405 "Not allowed.";

    }

    ban("req.http.host == " + req.http.host + "&& req.url == " + req.url);

    error 200 "Ban added";

  }
  return (lookup);
}
sub vcl_pipe {

  set bereq.http.connection = "close";

  return (pipe);
}
sub vcl_pass {

  return (pass);
}
sub vcl_hash {

  hash_data(req.url);

  if (req.http.host) {

    hash_data(req.http.host);

  } else {

    hash_data(server.ip);

  }
  // If the client supports compression, keep that in a different cache

  if (req.http.Accept-Encoding) {

    hash_data(req.http.Accept-Encoding);

  }
  return (hash);
}
sub vcl_hit {

  if (req.request == "PURGE") {

    purge;

    error 200 "Purged.";

  }
  //Listen to browser force refresh

  if (req.http.Cache-Control ~ "no-cache") {

    if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE")) {

      set obj.ttl = 0s;

      return (restart);

    }

  }
  return (deliver);
}
sub vcl_miss {

  if (req.request == "PURGE") {

    purge;

    error 200 "Purged.";

  }
  return (fetch);
}
sub vcl_fetch {

  if (beresp.ttl <= 0s ||

    beresp.http.Set-Cookie ||

    beresp.http.Vary == "*") {

      set beresp.ttl = 120 s;

      return (hit_for_pass);

  }

  return (deliver);
}
sub vcl_deliver {

  // Debugging

  if (obj.hits > 0) {

    set resp.http.X-Cache = "HIT";

  } else {

    set resp.http.X-Cache = "MISS";

  }
  // Remove some headers: PHP version

  unset resp.http.X-Powered-By;
  // Remove some headers: Apache version & OS

  unset resp.http.Server;
  return (deliver);
}
sub vcl_error {

  set obj.http.Content-Type = "text/html; charset=utf-8";

  set obj.http.Retry-After = "5";

  synthetic {"

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>

  <head>

    <title>"} + obj.status + " " + obj.response + {"</title>

  </head>

  <body>

    <h1>Error "} + obj.status + " " + obj.response + {"</h1>

    <p>"} + obj.response + {"</p>

    <h3>Guru Meditation:</h3>

    <p>XID: "} + req.xid + {"</p>

    <hr>

    <p>Varnish cache server</p>

  </body>
</html>
"};

  return (deliver);

}
sub vcl_init {

  return (ok);

}
sub vcl_fini {

  return (ok);

}

You can make sure that it works by checking the headers for: Via - 1.1 Varnish

Nagios with NginX in Ubuntu

Vlad — Fri, 07 Jan 2011 18:20:18 +0000

In this article I will show you how I installed the latest version of the Nagios from sources, and how to serve it with NginX. Nagios is an open-source monitoring application with a very strong community and a huge set of plugins that can help you monitor every aspect of your servers. At the time of writing this article the latest version is 3.4.1. The platform used is Ubuntu Precise 12.04 LTS, but except dependencies, it should be the same for older distributions. It doesn't hurt to check them first.

We will start by installing the dependencies:

apt-get install libperl-dev libpng12-dev libgd2-xpm-dev

And then create the user and group under which it will run:

/usr/sbin/useradd -M -s /bin/false nagios

Go ahead and create a directory in which we can download and compile the packages:

mkdir nagios_sources &&cd nagios_sources
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.4.1.tar.gz
wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
tar xzf nagios-3.4.1.tar.gz
tar xzf nagios-plugins-1.4.15.tar.gz

As you can see we downloaded both the core packages and the plugins. Next we will install the main program. I will use as the destination directory "/opt/":

cd nagios-3.2.3
./configure --prefix /opt/nagios \
--sysconfdir=/etc/nagios \
--with-nagios-user=nagios \
--with-nagios-group=nagios \
--with-command-user=nagios \
--with-command-group=nagios
make all
make install
make install-init
make install-config
make install-commandmode

Next step is to create a password file:

htpasswd -c /etc/nagios/htpasswd.users nagiosadmin

Next we will install the plugins:

cd ../nagios-plugins-1.4.15
./configure --prefix /opt/nagios \
--with-nagios-user=nagios \
--with-nagios-group=nagios
make && make install

Before we finish the installation you need to associate your email address with the nagiosadmin contact in the /etc/nagios/objects/contacts.cfg file.

That's it! The only thing left is to create the start-up links for Ubuntu, to test the configuration and if everything works, start the daemon:

update-rc.d -f nagios defaults
/opt/nagios/bin/nagios -v /etc/nagios/nagios.cfg
/etc/init.d/nagios start

The /etc/nagios/objects/ folder contains all the definitions pertaining your monitored machines and you can change them any way you like. Everything is very well commented so it's very easy to modify it. But before restarting the server check first with the command above if everything works as expected.

Now, for the nginx configuration, first of all apart from a functional php environment you must also have the lightweight fcgiwrap package from Ubuntu. You can install it with:

apt-get install spawn-fcgi fcgiwrap

And this is an example of the virtual host which allows normal operation under /var/www as well as aliases for the /nagios/ folder (modify it according to your configuration):

server {

        server_name example.com;
        access_log  /var/log/nginx/example.com.access.log;

        error_log   /var/log/nginx/example.com.error.log;

  auth_basic            "Restricted Nagios Area!";

  auth_basic_user_file  /etc/nagios/htpasswd.users;     

        root    /var/www;

        index   index.php index.html;
   location / {

    try_files $uri $uri/ index.php;

  }

  location /nagios {

    alias /opt/nagios/share/;

  }
  location ~ ^/nagios/(.*\.php)$ {

    alias /opt/nagios/share/$1;

    include /etc/nginx/fastcgi.conf;

    fastcgi_pass unix:/var/run/php5-fpm.sock;

  }

  location ~ \.cgi$ {

    root /opt/nagios/sbin/;

    rewrite ^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break;

    fastcgi_param AUTH_USER $remote_user;

    fastcgi_param REMOTE_USER $remote_user;    

    include /etc/nginx/fastcgi.conf;

    fastcgi_pass unix:/var/run/fcgiwrap.socket;

  }  

  location ~ \.php$ {

    include /etc/nginx/fastcgi.conf;

    fastcgi_pass unix:/var/run/php5-fpm.sock;

  }
}

You can now restart nginx and fcgiwrap and access your nagios installation by visiting example.com/nagios.

Install NginX and PHP with PHP-FPM, MySQL and APC

Vlad — Sat, 24 Jul 2010 14:52:30 +0000

*** DISCLAIMER: All content provided here including the scripts is provided without any warranty. You use it at your own risk. I can not be held responsible for any damage that may occur because of it. By using the scripts I provide here you accept this terms.

*** Please bear in mind that this method is intended for development and testing purposes only. If you care about stability and security you should use the packages provided by your distribution.

An unattended script that installs everything can be found at GitHub. Check the README file for instructions on how to run the installer.

If you found this useful and you want to contribute, please consider sending a donation (click the button below).

In this tutorial I will compile a LEMP stack with the latest NginX (1.3.10) and PHP (5.4.10) with FPM, MySQL and APC (3.1.14) on Ubuntu Quantal(12.10). For the moment, Suhosin 0.9.33 is not working with PHP 5.4.x. Apart for some version numbers for some dependencies, all the steps should be the similar for Ubuntu Lucid, Maverick, Natty and Oneiric.

I will not explain how to configure NginX here. You can find a lot of resources about that on the nginx wiki.

In order to compile these programs you will need the following dependencies (most of them should already be installed):

sudo apt-get install htop vim-nox binutils cpp flex gcc libarchive-zip-perl libc6-dev m4 libpcre3 libpcre3-dev libssl-dev libpopt-dev lynx make perl perl-modules openssl unzip zip autoconf2.13 gnu-standards automake libtool bison build-essential zlib1g-dev ntp ntpdate autotools-dev g++ bc subversion psmisc re2c

And also some libraries for PHP:

sudo apt-get install libmysqlclient-dev mysql-client libcurl4-openssl-dev libgd2-xpm-dev libjpeg-dev libpng3-dev libxpm-dev libfreetype6-dev libt1-dev libmcrypt-dev libxslt1-dev bzip2 libbz2-dev libxml2-dev libevent-dev libltdl-dev libmagickwand-dev libmagickcore-dev imagemagick libreadline-dev libc-client-dev libsnmp-dev snmpd snmp libpq-dev

Now let's create a folder in which we can play:

mkdir ~/lemp && cd ~/lemp

and start downloading the sources:

wget http://nginx.org/download/nginx-1.3.10.tar.gz
wget http://us.php.net/distributions/php-5.4.10.tar.gz
wget http://pecl.php.net/get/APC-3.1.14.tgz

You can visit each site in order to find faster mirrors. Now decompress them:

tar zxvf nginx-1.3.10.tar.gz
tar xzvf php-5.4.10.tar.gz
tar xzvf APC-3.1.14.tgz

And now let's install them one by one. I like to put everything I compile by myself in one place, the /opt directory. By doing this I know where to find things like configuration files and libraries.

cd nginx-1.3.10/
./configure \

  --prefix=/opt/nginx \

  --conf-path=/etc/nginx/nginx.conf \

  --pid-path=/var/run/nginx.pid \

  --lock-path=/var/lock/nginx.lock \

  --http-log-path=/var/log/nginx/access.log \

  --error-log-path=/var/log/nginx/error.log \

  --http-client-body-temp-path=/var/lib/nginx/body \

  --http-proxy-temp-path=/var/lib/nginx/proxy \

  --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \

  --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \

  --http-scgi-temp-path=/var/lib/nginx/scgi \

  --with-http_stub_status_module \

  --with-http_ssl_module \

  --with-http_realip_module \

  --with-http_gzip_static_module \

  --user=www-data \

  --group=www-data \

  --without-mail_pop3_module \

  --without-mail_imap_module \

  --without-mail_smtp_module

make
sudo make install

cd ..
sudo wget -O /etc/init.d/nginx https://github.com/vladgh/VladGh.com-LEMP/raw/master/init_files/nginx
sudo chmod +x /etc/init.d/nginx
sudo update-rc.d -f nginx defaults
sudo mkdir -p /var/lib/nginx/{body,proxy,fastcgi,uwsgi,scgi}

You sould also create the file /etc/logrotate.d/nginx in order to compress the logs.

sudo sh -c "echo '/var/log/nginx/*.log {

        weekly

        missingok

        rotate 52

        compress

        delaycompress

        notifempty

        create 640 root adm

        sharedscripts

        postrotate

                [ ! -f /var/run/nginx.pid ] || kill -USR1 \`cat /var/run/nginx.pid\`

        endscript

}' > /etc/logrotate.d/nginx"

You can also find a very simple configuration for nginx at https://github.com/vladgh/VladGh.com/blob/master/nginx.conf, but I strongly recommend to use the Nginx Configuration page

Starting with Ubuntu 11.04 the canonical team modified the path for some libraries so you might need to run the following set of commands in order to get the PHP compilation to work:

arch=$(dpkg-architecture -qDEB_HOST_MULTIARCH)
[ -f /usr/lib/${arch}/libjpeg.so ] && sudo ln -s /usr/lib/${arch}/libjpeg.so /usr/lib/
[ -f /usr/lib/${arch}/libpng.so ] && sudo ln -s /usr/lib/${arch}/libpng.so /usr/lib/
[ -f /usr/lib/${arch}/libXpm.so ] && sudo ln -s /usr/lib/${arch}/libXpm.so /usr/lib/
[ -f /usr/lib/${arch}/libmysqlclient.so ] && sudo ln -s /usr/lib/${arch}/libmysqlclient.so /usr/lib/
[ -d /usr/lib/i386-linux-gnu/mit-krb5 ] && sudo ln -s /usr/lib/${arch}/mit-krb5/lib*.so /usr/lib/

Now let's compile the PHP packages:

sudomkdir/var/www
sudochown-R www-data:www-data /var/www
cd php-5.4.10
./buildconf --force
./configure \

  --prefix=/opt/php5 \

  --with-config-file-path=/etc/php5 \

  --with-config-file-scan-dir=/etc/php5/conf.d \

  --with-curl \

  --with-pear \

  --with-gd \

  --with-jpeg-dir \

  --with-png-dir \

  --with-zlib \

  --with-xpm-dir \

  --with-freetype-dir \

  --with-t1lib \

  --with-mcrypt \

  --with-mhash \

  --with-mysql \

  --with-mysqli \

  --with-pgsql \

  --with-pdo-mysql \

  --with-pdo-pgsql \

  --with-openssl \

  --with-xmlrpc \

  --with-xsl \

  --with-bz2 \

  --with-gettext \

  --with-readline \

  --with-fpm-user=www-data \

  --with-fpm-group=www-data \

  --with-imap \

  --with-imap-ssl \

  --with-kerberos \

  --with-snmp \

  --disable-debug \

  --enable-fpm \

  --enable-cli \

  --enable-inline-optimization \

  --enable-exif \

  --enable-wddx \

  --enable-zip \

  --enable-bcmath \

  --enable-calendar \

  --enable-ftp \

  --enable-mbstring \

  --enable-soap \

  --enable-sockets \

  --enable-shmop \

  --enable-dba \

  --enable-sysvsem \

  --enable-sysvshm \

  --enable-sysvmsg
make
sudo make install

The next step is to configure PHP-FPM:

sudo mkdir /var/log/php-fpm
sudo mkdir -p /etc/php5/conf.d
sudo chown -R www-data:www-data /var/log/php-fpm
sudo cp -f php.ini-production /etc/php5/php.ini
sudo chmod 644 /etc/php5/php.ini
sudo cp -f /opt/php5/etc/php-fpm.conf.default /etc/php5/php-fpm.conf
sudo cp -f /opt/php5/etc/pear.conf /etc/php5/pear.conf
sudo cp -f sapi/fpm/init.d.php-fpm /etc/init.d/php-fpm
sudo chmod +x /etc/init.d/php-fpm
sudo update-rc.d -f php-fpm defaults

We should also modify the init file and the configuration file for PHP with the standard path for the pid file (/var/run):

sudo sed -i".bak" "s/php_fpm_CONF=.*/php_fpm_CONF=\/etc\/php5\/php-fpm.conf/" /etc/init.d/php-fpm
sudo sed -i "s/php_fpm_PID=.*/php_fpm_PID=\/var\/run\/php-fpm.pid/" /etc/init.d/php-fpm
sudo sed -i".bak" "s/;pid = .*/pid = \/var\/run\/php-fpm.pid/" /etc/php5/php-fpm.conf

You can make the desired modifications to /etc/php5/php-fpm.conf file. This is were you can setup the server to listen on a socket or TCP port, or you can adjust how many processes you want. The file is very well documented in the comments. A sample conf file can be found here.

The newer versions of php complain if a time zone is not set on php.ini (so we grab the system's one)

TIMEZONE=$([ -f /etc/timezone ] && cat /etc/timezone | sed "s/\//\\\\\//g")
sudo sed -i".bak" "s/^\;date\.timezone.*$/date\.timezone = \"${TIMEZONE}\" /g" /etc/php5/php.ini

Create the logrotate file for PHP with the following command:

sudo sh -c "echo '/var/log/php-fpm/*.log {

        weekly

        missingok

        rotate 52

        compress

        delaycompress

        notifempty

        create 640 www-data www-data

        sharedscripts

        postrotate

                [ ! -f /var/run/php-fpm.pid ] || kill -USR1 \`cat /var/run/php-fpm.pid\`

        endscript

}' > /etc/logrotate.d/php-fpm"

Because the executables for both PHP and NginX are not in a standard location you can add the following lines to /etc/environment so that the new path is always loaded:

export PATH="/opt/php5/bin:/opt/php5/sbin:/opt/nginx/sbin:$PATH"
sudo sh -c "echo \"PATH=\"$PATH\"\" > /etc/environment"

Now you should be able to see the version numbers for NginX and PHP:

nginx -V

php -v

And also the location of the executables:

which nginx
which php

Now let's install the latest APC:

cd ../APC-3.1.14
/opt/php5/bin/phpize -clean

./configure --enable-apc --with-php-config=/opt/php5/bin/php-config --with-libdir=/opt/php5/lib/php 
make
sudo make install

You will have to add the extension in php.ini:

sudo sh -c "echo 'extension = apc.so

apc.enabled = 1

apc.shm_size = 128M

apc.shm_segments=1

apc.write_lock = 1

apc.rfc1867 = On

apc.ttl=7200

apc.user_ttl=7200

apc.num_files_hint=1024

apc.mmap_file_mask=/tmp/apc.XXXXXX

apc.enable_cli=1

; Optional, for \"[apc-warning] Potential cache slam averted for key... errors\"

; apc.slam_defense = Off' > /etc/php5/conf.d/apc.ini"

This is it! You can now start the servers:

sudo /etc/init.d/php-fpm start
sudo /etc/init.d/nginx start

If you want a MySQL server installed you can run the following command.

apt-get install mysql-server mysql-client

This should be enough to get MySQL up and running. If you have a separate database somewhere you can only install the mysql-client package. For additional config parameters look into the /usr/share/doc/mysql-server-5.1/examples folder and you will find a few configurations. You can also get the MySQLTuner.pl script which will give a lot more information on how to fine tune your mysql server:

wget mysqltuner.pl

You should now have a fully functional LEMP platform.

An unattended script that installs everything can be found at GitHub. Check the README file for instructions on how to run the installer.

Jetty Powered MultiCore Apache Solr

Vlad — Tue, 29 Jun 2010 20:17:59 +0000

Installing the latest version of Apache Solr search engine on a Jetty powered server is very straightforward. You simply copy the packages in the right place, change a few lines to customize it and you are done. This tutorial is meant for Ubuntu users but it can be easily adapted for other distributions.

Assuming you are root (you can become by running "sudo -i" command), the first thing you need to do is get the dependencies:

apt-get update && apt-get upgrade
apt-get install openjdk-6-jdk

I recommend the java development kit from Ubuntu but if you want you can also use the one made by Sun (replace "natty" with your own distribution codename):

add-apt-repository "deb http://archive.canonical.com/ natty partner"
apt-get install sun-java6-jdk

Download and extract the Jetty server into the /opt folder (Visit http://download.eclipse.org/jetty/ for the latest version):

JETTY_VERSION=7.4.5.v20110725
cd /opt
wget http://download.eclipse.org/jetty/$JETTY_VERSION/dist/jetty-distribution-$JETTY_VERSION.tar.gz
tar xfz jetty-distribution-$JETTY_VERSION.tar.gz
mv jetty-distribution-$JETTY_VERSION jetty

Download and extract the Apache Solr application into a temporary folder. We don't need the embedded jetty because we have our own. The URL provided is just a mirror so it's best to visit http://lucene.apache.org/solr/ for the latest version and a faster download link:

SOLR_VERSION=3.3.0
cd /tmp
wget http://mirrors.kahuki.com/apache//lucene/solr/$SOLR_VERSION/apache-solr-$SOLR_VERSION.tgz
tar zxvf apache-solr-$SOLR_VERSION.tgz

Next, we need to copy the actual Solr application into Jetty:

cp apache-solr-$SOLR_VERSION/dist/apache-solr-$SOLR_VERSION.war /opt/jetty/webapps/solr.war
cp -R apache-solr-$SOLR_VERSION/example/solr /opt

Before we start the server we need to add a few options to java:

echo "JETTY_USER=jetty

JAVA_OPTIONS=\"-Dsolr.solr.home=/opt/solr\"" >> /etc/default/jetty

By modifying the java_options line you can customize/optimize even more or Java environment, but that's outside the scope of this tutorial. For example:

JAVA_OPTIONS="-Xms16m -Xmx64m -Djava.awt.headless=true -Dsolr.solr.home=/opt/solr"

We also need to create the multicore config for Solr:

mkdir-p/opt/solr/example1.com/data
mkdir-p/opt/solr/example2.com/data
cp -R /opt/solr/conf /opt/solr/example1.com
cp -R /opt/solr/conf /opt/solr/example2.com

And modify /opt/solr/solr.xml accordingly:

<cores adminPath="/admin/cores" defaultCoreName="collection1">

  <core name="Example1" instanceDir="example1.com" />

  <core name="Example1" instanceDir="example1.com" />
</cores>

One last thing is to create the user under which all this will run, set the right permissions and check if everything is in order with Jetty:

adduser --system--no-create-home--disabled-login--disabled-password--group jetty
chown -R jetty:jetty /opt/jetty
chown -R jetty:jetty /opt/solr
/opt/jetty/bin/jetty.sh check

If everything is in order you can start Jetty and even configure it to run at bootup:

/opt/jetty/bin/jetty.sh start
ln -s /opt/jetty/bin/jetty.sh /etc /init.d/jetty

update-rc.d -f jetty defaults

That's it; you now have the latest Apache Solr engine on the latest Jetty web server.

VladGh.com blogs

Varnish 3 and Apache in Ubuntu 12.04 LTS Precise

Tags:

Nagios with NginX in Ubuntu

Tags:

Install NginX and PHP with PHP-FPM, MySQL and APC

Tags:

Jetty Powered MultiCore Apache Solr

Tags: