.bash_history

Simple web server load testing

2011-05-03T22:27:00.003-04:00

I recently ran into ab, which is an Apache HTTP server benchmarking tool. The following is an invocation where I'm running a load test on a local tomcat instance which is using 1000 requests with 10 concurrent threads.

mike@shiner $ ab -n 1000 -c 10 http://localhost:8080/test-ui/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Apache-Coyote/1.1
Server Hostname:        localhost
Server Port:            8080

Document Path:          /test-ui/
Document Length:        7891 bytes

Concurrency Level:      10
Time taken for tests:   13.262 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      8054000 bytes
HTML transferred:       7891000 bytes
Requests per second:    75.40 [#/sec] (mean)
Time per request:       132.624 [ms] (mean)
Time per request:       13.262 [ms] (mean, across all concurrent requests)
Transfer rate:          593.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      12
Processing:     5  132 349.6     28    3087
Waiting:        5  130 349.4     26    3087
Total:          5  132 349.7     28    3087

Percentage of the requests served within a certain time (ms)
  50%     28
  66%     84
  75%    145
  80%    168
  90%    264
  95%    376
  98%   1450
  99%   2412
 100%   3087 (longest request)

The tool has a lot of really great features and I'll try to exemplify some of them here. The following shows how you can change the User-Agent to a variant of Internet Explorer (thanks goes to www.useragentstring.com for the value). I've also set the verbosity level to 4 so I can verify the 'User-Agent' value is set as intended. I've excluded the output for brevity.

mike@shiner $ ab  -v 4 -H "User-Agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1)"   http://localhost:8080/test-ui/

Next is an example where I set the value of a cookie. I run into situations where I need to evaluate code for cell testing. It happens to be that our cell testing code is dependent on cookie values.

mike@shiner $ ab -n 1000 -c 10 -C "cellNum=10"  http://localhost:8080/test-ui/

The following is an example of setting the -G flag, which will output a file named 'perf.out' which can be used with GnuPlot. The file doesn't have to be used with just GnuPlot. It can also be used with IDL, Mathematica, Igor, and Excel.

mike@shiner $ ab -n 1000 -c 10 -G perf.out http://localhost:8080/test-ui/

Finding the memory usage of a process

2011-04-30T23:48:00.002-04:00

I recently needed to figure out how to determine the memory usage of a process so I could log it over time. I used the following command to accomplish this.

mike@shiner $ ps -p 7557 -o pid,vsz,rss
  PID      VSZ    RSS
 7557  1983664 298536

You'll need to replace '7557' with the process id you're investigating. The meaning of 'VSZ' and 'RSS' is copy and pasted here from the man page:

     rss        resident set size
     vsz        virtual size in Kbytes (alias vsize)

'RSS' represents the amount of memory resident in RAM. The rest of the memory is held in swap space or the filesystem. 'VSZ' is the amount of virtual memory allocated to the process.

Moving multiple files

2011-04-26T22:58:00.000-04:00

I recently learned that you can move multiple files in one invocation of mv. As long as the last argument is a directory, then all the previous files/directories will be moved into it.

mike@shiner $ mv one.txt two.txt three.txt foobar

If the last argument isn't a directory, then you'll see an error.

mike@shiner $ mv one.txt two.txt three.txt 
mv: target `three.txt' is not a directory

How to convert Unix timestamps

2011-04-05T00:09:00.000-04:00

It's been a long *time* since my last post, hence I thought I'd do a post that involves time. I often run into the situation where I have a Unix timestamp and I need to determine the date. You can use the following to do this.

mike@shiner $ date -d @1252278000
Sun Sep  6 19:00:00 EDT 2009

The following shows how to get the Unix timestamp for a particular moment in time.

mike@shiner $ date -d "2009-09-06 19:00:00" "+%s"
1252278000

How to sum up the values in a file

2011-01-15T16:01:00.009-05:00

Once you know how to remove a column from a file, then a time will come where you'll want to sum up the values in a column. For this example, let's take a look at a file with numbers in it.

mike@shiner $ cat file-with-numbers
1
2
3
4
5
6

We can sum up the values in this file with the following command.

mike@shiner $ cat file-with-numbers | awk '{sum+=$1}END{print sum}'
21

We can see here that the sum of the numbers in the file total to 21.

How to remove spaces from filenames

2011-01-13T00:31:00.001-05:00

I often run into files with spaces in them and I want to rename them. Once again, there's a command that's appropriately named to do the job. I'm going to go over how to use the rename command. Let's take a look at the files we're going to work with.

mike@shiner $ ls
foo bar.txt  test this.txt

We have two files which are named 'foo bar.txt' and 'test this.txt'. We're going to use the following command to get rid of the spaces.

mike@shiner $ rename 's/ /_/' *.txt

The command takes two arguments. The first argument is a regular expression where the 's' stands for substitute. The contents between the first set of slashes represent what we're matching on. The '_' between the second set of slashes represent what we want to replace it with. The second argument, *.txt, tells rename what files will be affected. The preceding command happily renders the following result.

mike@shiner $ ls
foo_bar.txt  test_this.txt

Now let's say we want to replace the underscores with dashes. We can do this with the following command.

mike@shiner $ rename 's/_/-/' *.txt
mike@shiner $ ls
foo-bar.txt  test-this.txt

How to pipe STDOUT to vim

2011-01-03T23:57:00.000-05:00

I usually use ColorDiff when I want to get a colorized diff from the command line.

mike@shiner $ cvs diff -up | colordiff

I was talking with a friend about vim and he showed me how to utilize vim for a similar purpose.

mike@shiner $ cvs diff -up | vim -

This is great because it gives you the same syntax highlighting benefits, but you don't have to install another application.

How to make a banner from the command line

2010-12-25T10:01:00.001-05:00

I thought I'd send out some command line Christmas cheer. What better way than to use figlet.

mike@shiner $ figlet 'Merry XMAS!'
 __  __                       __  ____  __    _    ____  _ 
|  \/  | ___ _ __ _ __ _   _  \ \/ /  \/  |  / \  / ___|| |
| |\/| |/ _ \ '__| '__| | | |  \  /| |\/| | / _ \ \___ \| |
| |  | |  __/ |  | |  | |_| |  /  \| |  | |/ ___ \ ___) |_|
|_|  |_|\___|_|  |_|   \__, | /_/\_\_|  |_/_/   \_\____/(_)
                       |___/

How to delete the last lines of a file

2010-12-22T01:34:00.001-05:00

A friend of mine showed me a way to delete the last lines of a file after he read the article on how to delete the first lines of a file. The following is an example of how he did this with sed.

mike@shiner $ sed '$d' really_big_file.txt > new_file.txt

You'll need to use the single quotes to prevent the shell, such as bash, from interpreting $d as a variable. The '$' symbol matches the last line, similar to how it matches the end of the line in a regular expression. The 'd' tells sed that we want to delete the line.

Another way to do this, albeit not as elegant, is exemplified next.

mike@shiner $ tac really_big_file.txt | sed 1d | tac > new_file.txt

If you've never seen the tac command, it's similar to cat, but it's cat in reverse. I think it's kind of nifty that cat spelled backwards is tac. This helps with remembering the command.

Here's what I'm doing in each step:

1. cat the file in reverse.
2. Delete the first line.
3. cat the file in reverse again.
4. Save the output to 'new_file.txt'.

The end result is the same as before, but I can see someone arguing against it because of readability.

How to find files larger than X

2010-12-19T01:43:00.002-05:00

You can use the following command to find files that are larger than 50M.

mike@shiner $ find . -type f -size +50000k

The -type argument tells find that we want to look at files and the -size argument specifies files larger than 50,000k. We can use the following command to get a nicely formatted output with the name of the file and the size.

mike@shiner $ find . -type f -size +50000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

Alternatively, we can do the following to get the same output.

mike@shiner $ find . -type f -size +50000k | xargs ls -lh | awk '{ print $9 ": " $5 }'

xargs is a handy command that takes each line of input and passes it to the proceeding command, which in this case is ls -lh.

How to change the default ssh port

2010-12-17T22:14:00.000-05:00

On Ubuntu, we can edit the '/etc/ssh/sshd_config' file to change the port our ssh server is listening on. You should find something similar to the following in this file.

Port 22

You can change this value to any integer between 1025 and 65536 (inclusive). You should pick a value that won't conflict with any other services. If you want to limit ssh access to specific users, then you can add the users to the 'AllowUsers' setting separated by spaces. Once you make the changes, you'll need to reload ssh to use the new configurations.

mike@shiner $ sudo /etc/init.d/ssh reload
[sudo] password for mike: 
 * Reloading OpenBSD Secure Shell server's configuration sshd
   ...done.

You should stay logged in until you can test the configuration by logging in from a different machine. If we changed the 'Port' value to 4200, then we can use the following command to specify port 4200 on the remote machine for ssh.

mike@primus $ ssh -p 4200 mike@shiner

How to create a path of directories

2010-12-15T21:15:00.001-05:00

I've often times created a directory structure using a series of mkdir commands. For example,

mike@shiner $ mkdir foo
mike@shiner $ mkdir foo/bar
mike@shiner $ mkdir foo/bar/test

You can create the same directory structure as above by using the -p option.

mike@shiner $ mkdir -p foo/bar/test

How to execute vim commands from the command line

2010-12-13T21:14:00.000-05:00

I was trying to execute a regular expression over multiple lines when I ran across an example of how to execute vim commands from the command line. I happen to know how to do this from inside vim, so this was a serendipitous find. First, I'm going to show an example of how to do a simple replace. Here's the current contents of our test file

mike@shiner $ cat test_regex.txt
this is
a test

We can run the following command to replace 'this' with 'there'.

mike@shiner $ vim test_regex.txt -c '%s/this/there/' -c 'wq'
mike@shiner $ cat test_regex.txt
there is
a test

The -c arguments are executed sequentially. They're equivalent to typing : and then the contents of the argument. In the example above, we do a search/replace and then write/quit the file. If the "-c 'wq'" is omitted, then vim will remain open. Now back to what I was trying to do originally.

mike@shiner $ vim test_regex.txt -c '%s/there\_.*a/regex' -c 'wq'
mike@shiner $ cat test_regex.txt
regex test

The \_ tells vim that the . will include the new line character. Hence, the expression states that we want to match 'there' and 'a' and anything between these characters. This match is replaced with 'regex'.

How to remove a column from a file

2010-12-11T14:46:00.004-05:00

I often need to remove a column from a character delimited file so I can do some additional processing, such as counting up values. I'm going to use a file with the following contents as an example.

mike@shiner $ cat name_age_city.txt
Mike|35|New York
Jason|28|Dallas
Fox|32|Washington
Mike|28|Los Angeles
Ann|23|Denver
Dana|32|Louisville
Mike|32|Atlantic City
Ryan|30|Austin

We can use the cut command to extract the column with the age information.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 2
35
28
32
28
23
32
32
30

The -d option tells cut that the delimiter for the columns is the '|' character and the -f option specifies the second column. Let's say we want to know how many people have the same age. We can use the following command to enumerate these values for us.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 2 | sort | uniq -c
      1 23
      2 28
      1 30
      3 32
      1 35

If we need to extract multiple columns, cut gives us this functionality as well.

mike@shiner $ cat name_age_city.txt | cut -d '|' -f 1,3
Mike|New York
Jason|Dallas
Fox|Washington
Mike|Los Angeles
Ann|Denver
Dana|Louisville
Mike|Atlantic City
Ryan|Austin

Unfortunately, cut won't allow us to reorder the columns. We get the same results if we try "cut -d '|' -f 3,1". On a side note, we can use awk to give us functionality similar to cut. One advantage awk has over cut is the ability to reorder the columns.

mike@shiner $ cat name_age_city.txt | awk -F '|' '{print $3"|"$1}'
New York|Mike
Dallas|Jason
Washington|Fox
Los Angeles|Mike
Denver|Ann
Louisville|Dana
Atlantic City|Mike
Austin|Ryan

The awk syntax requires us to put the '|' character in the print output to get the same behavior as cut.

How to delete the first lines of a file

2010-12-08T23:27:00.005-05:00

I had to delete the first line of a file today, but I didn't want to open the file because it was close to 1 GB in size. A file this large will make most editors pretty unhappy. Instead of opening an editor to just delete a line, I used the following handy command.

mike@shiner $ sed 1d really_big_file.txt > new_file.txt

The '1d' tells sed that we want to delete line 1. The next command will delete the first three lines. The '1,3' is defining a range from line 1 to line 3.

mike@shiner $ sed 1,3d really_big_file.txt > new_file.txt

How to split a file

2010-12-07T00:17:00.005-05:00

I often have a file that's too big. A great way to split the file is to use.....wait for it.....the split command. I usually split a file by line count. Here's an example invocation:

mike@shiner $ split -l 100 -d test_file.txt test_file_

The -l option defines the maximum number of lines per file. If 'test_file.txt' has 951 lines, then we'll end up with 10 files where we have 9 with 100 lines each and 1 with 51 lines. The -d option tells split to use numeric suffixes. 'test_file.txt' is the file we're going to split and 'test_file_' is the prefix used for each of the files.

mike@shiner $ ls
test_file.txt  test_file_02  test_file_05  test_file_08
test_file_00   test_file_03  test_file_06  test_file_09
test_file_01   test_file_04  test_file_07