I've split this article into compromise hacks (i.e. rootkit attacks) which are relatively easy to detect and application compromises which can be more subtle.

UPDATE: After a discussion about the article on WebHostingTalk I've incorporated a few great suggestions.

Root compromises.

This means someone has full access to the system, here are the tell tale signs in order of most likely to give you a quick feel for what's going on.

1. Have a look for system files that have changed recently

This is the first thing I would do.

find /etc /var -mtime -2

The "-2" means 2 days, i.e. show me all files modified in the last 2 days.

Now if you haven't installed any new software on your server for a while then this command will run and produce very little output. For a server I investigated there were references to postfix. clearly someone had installed a mail server probably for sending spam.

2. Run who

who  

user1 pts/2        2012-03-28 13:38 (128.114.44.209)

This should give you a list of users on the system, what you're looking for is users other than yourself especially root.

3. History

history

Login as root and run history, this will give you a list of recently run commands. If you're the only person who logs in as root you should have an idea if anything looks suspicious or not.

4. Netstat

It's a good idea at this point to see what processes are running on which ports.

netstat --listen -A inet   

tcp 0 0 *:64010 *:* LISTEN
tcp 0 0 *:http-alt *:* LISTEN 
tcp 0 0 *:ssh *:* LISTEN 
tcp 0 0 *:https *:* LISTEN 

Application Breaches

An example of an application breach might be a shopping cart that allows file uploads these uploads can be executed as say PHP scripts (This happened with oscommerce). The attacker doesn't have root access but they can certainly use up resources.

5. High load and Memory.

Sudden spikes in memory and load usage are an indication of possible root breaches but sometimes the only indication an application has been breached.

If you're a ServerMonitoringHQ.com user then you'll receive notifications of CPU and Memory spikes. (Along with bandwidth and more).

You can use the following command to track load (CPU usage). Generally values over 1 indicate the CPU is getting used quite hard.

cat /proc/loadavg

0.04 0.35 0.26 1/83 27412  

The first column 0.04 is the number to look at

6. Top

The top command is a quickly way to see what processes are consuming resources.

7. Bandwidth Spikes

As with process and memory if you don't know what you normally consume it's difficult to tell if you get a spike or not.

netstat -i 
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR 
eth0 1500 0 2937582 0 0 0 3127979 0 0 0 BMRU 
lo 16436 0 531312 0 0 0 531312 0 0 0 LRU 

The larger numbers for TX and RX are bytes transferred (roughly), run this twice, a minute apart will give you an indication of bandwidth transfer per minute.

And finally, good luck.

We've all been there, best of luck getting your server up and running again.

The Curl syntax allows you to specify sequences and sets of URL's. Say for example we're going to run a load stress test against Google we can run...

curl -s "http://google.com?[1-1000]"

This will make 1000 calls to google i.e.

http://google.com?1  
http://google.com?2  
http://google.com?3 
\... 
http://google.com?1000

So say you want to stress test your web application and it won't complain if it's fed an extra parameter, 10,000 calls could be done something like.

curl -s "http://yourappp.com/your_page_to_test.php?[1-10000]"

Multiple Pages

Easy just add each page to the command line.

curl -s "http://yourapp.com/page1.php?[1-1000]" "http://yourappp.com/page2.php?[1-1000]"

Or even...

curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]"

Timing

Using the time command we can get a view on our performance

time curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]"  

real 0m0.606s 
user 0m0.009s 
sys 0m0.008s 

Simulating consecutive users

OK, this is great for sending a whole bunch of calls one after the other but what about simultaneous calls. For this we can place the Curl calls in a script and set them running in the background. i.e. my_stress_test.sh

curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!" 
curl -s "http://yourapp.com/page{1, 2}.php?[1-1000]" &
pidlist="$pidlist $!"  

for job in $pidlist do 
  echo $job     
  wait $job || let "FAIL+=1" 
done  

if [ "$FAIL" == "0" ]; then 
  echo "YAY!" 
else 
  echo "FAIL! ($FAIL)" 
fi

Then run time my_stress_test.sh

Caveats

This does not simulate user behaviour exactly as the browser is not only downloading the page but all attached images, javascripts, stylesheet etc. You could simulate this too by adding the URL's to the url command.

There are many ways to keep a process running on linux but I haven't seen any that are as easy to implement as the script below.

Basically the script does a ps ax and then a grep for your process. If it's not running it will re-start the process.

You install the script into your crontab i.e. crontab -e

As a bonus this mechanism will re-start your process after a re-boot.

Edit your crontab

Cut and paste the following code into your crontab.

*/5 * * * * /home/path_to_the_script/make-run.sh

Make sure the cron entry is pointing to where your make-run.sh is located.

The Bash script

#!/bin/bash 
# make-run.sh 
# make sure a process is always running.  
# Add the following to the crontab (i.e. crontab -e)
# */5 * * * * /home/path_to_make_run/make-run.sh

process=servermonitoringhq 
makerun="/home/path_to_the_job_you_want_running/runjob.sh"  

if ps ax | grep -v grep | grep $process > /dev/null         
then                 
  exit         
else         
  $makerun &
fi 

Test the script on it's own first to make sure it starts your job.

A basic development iteration

I'm usually developing the following repeatedly through development/maintenace of a Ruby on Rails application.

1. Have 1 tab open in firefox with shellinabox and make file edits using Vim.
2. Have another open tab in firefox with my application running and test the changes.
3. Repeat the first 2 steps until the fix is ready.
4. Run the unit tests
5. If all is OK commit changes to git.
6. Push changes to production which in my case is Heroku.

Using Vim, screen and shellinabox I can run through this cycle quickly. After a while I realised that coding through the browser was no slower than coding in an IDE.

I also like the idea of being able to pick up from where I left off whilst moving from one computer to another.

I now code exclusively in VIM through shellinabox.

Setting up the IDE

Why did I create ServerMonitoringHQ.

After running Status2K.com (a PHP script for server monitoring) for a few months I ran into a few issues. Mainly that it's hard to maintain and spot errors with a script running on someone else's servers. And also that it's fustrasting to see copies of your script pirated on numerous file sharing sites.

So I wanted to create a hosted server monitoring and statistics solution. I looked at the other hosted solutions and most seem to concentrate on uptime, i.e. pinging servers or loading web pages. There are a few that gather statistics internally e.g. you install an agent on each of your servers. I decided to have an agent-less approach. So ServerMonitoringHQHQ can gather server statistics over SSH with nothing to install on your server. For those that don't want to give out SSH information you can also use the ServerMonitoringHQ agent script. The script gives a nice realtime web based statistics output. See the ServerMonitoringHQ Agent

About Me

I've been programming professionally since 1994 mainly for investment banks in London. I started out doing C and C++ for satellite communications companies before progressing onto Java. I installed Linux for the first time in 1996 and have been using a mixture and Windows and Linux on the server side throughout my career.

I've had numerous side projects throughout the years but ServerMonitoringHQ is the most viable and I'm going to spend a lot of time marketing this site and hopefully making it the best monitoring solution out there.

Technology

ServerMonitoringHQ is a Ruby on Rails application. After building sites in Java and PHP I've found Rails to be the most productive platform currently available.

The ServerMonitoringHQ Agent is a PHP application. PHP I believe is the most widely available scripting language on Linux so hopefully most people should find installation of the script trivial.

The site is hosted at Heroku which provides me with the ability to scale the site if required quite quickly. I actually have 4 applications installed at heroku to support this site.

1. This blog which is a basic rack application where I edit the posts by hand coding HAML pages with Vim.
2. The application itself, i.e. all the screens and data collection. ServerMonitoringHQ.com
3. Resque A queue for background jobs with a nice front end with live statistics.
4. A customised version of Fat Free CRM with which I handle all customer interaction and which I'll extend to provide a TenderApp style support channel.

Credits

I'd like to give credit to the guys over at the Micropreneur Academy the information from there has shaped a lot of what you see here.