juretta.com

Disable basic authentication for CouchDB running behind an Apache reverse proxy

Sat May 30 00:00:00 +1200 2009

I am currently using CouchDB version 0.9 for quite a few different projects on the production machine. CouchDB only accepts local connections, I ususally use either an SSH tunnel or an Apache virtual host to access the Futon administration interface.

The virtual host uses basic authentication and a reverse proxy to expose the CouchDB HTTP API and Futon. Apache will pass the authentication header to CouchDB which in turn invokes its authentication handler. As I am using CouchDB using a variety of different clients (Java based web application running in Tomcat, Rabbit MQ, a CouchDB/Rails project that is being tested by a Hudson continuous integration server, CLI apps and so forth) and I am not exposing CouchDB anyway I am not really keen on using the authentication mechanism CouchDB provides.

So I basically would like to let Apache take care of the access control and not having CouchDB invoking its authentication handler.

As far as I know there are basically two approaches to this problem.

The first is to set a null_authentication_handler in your CouchDB local.ini file (usually in $INSTALL_PREFIX/etc/couchdb/local.ini):

[httpd]
authentication_handler = {couch_httpd, null_authentication_handler}

Unfortunately the 0.9 release version does not seem to contain the null_authentication_handler and works on more recent CouchDB versions only.

This basically means that the proxy in front of CouchDB needs to strip the Authentication header.

This can be done for Apache by enabling the mod_headers module (a2enmod headers on Debian for example) and using the following line to unset the Authentication header:

RequestHeader unset Authorization

The following listing shows a setting similar to the one I am using for the virtual host:

<VirtualHost *:80>
        ServerName couchdb.example.com
        ServerAdmin admin@example.com


        ProxyRequests Off
        ProxyPreserveHost Off
        AllowEncodedSlashes On
        KeepAlive Off

        <Proxy *>
            Order deny,allow
            Allow from all
        </Proxy>

        <Location />
                AuthType Basic
                AuthName "Couchdb Admin"
                AuthUserFile /etc/apache2/passwd
                Require valid-user
        </Location>

        ProxyPass / http://localhost:5984/ nocanon
        ProxyPassReverse / http://localhost:5984/
        RequestHeader unset Authorization

        ErrorLog /var/log/apache2/couchdb.example.com-error_log
        CustomLog /var/log/apache2/couchdb.example.com-access_log common
</VirtualHost>

Goodby Rails, kind of...

Fri May 29 00:00:00 +1200 2009

I started this blog about 4 years ago using a custom built Rails blog engine using Rails 0.8. A Rails based blog engine wasn't around at that time and I wanted to build something using Rails anyway.

Having a custom built blog engine has its benefits and drawbacks. The rather basic administration interface was a pain to use and running two Mongrel instances for a nearly static website seemed to be a misallocation of ressources anyway.

So instead of switching to another full-fledged blog engine I switched to using Jekyll to generate a static version of the blog.

Tom Preston-Werner (the author of Jekyll) writes about his reasons for creating Jekkyl in Blogging Like a Hacker:

I was tired of complicated blogging engines like WordPress and Mephisto. I wanted to write great posts, not style a zillion template pages, moderate comments all day long, and constantly lag behind the latest software release.

Jekyll is the complete opposite, it is a:

[...] simple, blog aware, static site generator. It takes a template directory (representing the raw form of a website), runs it through Textile and Liquid converters, and spits out a complete, static website suitable for serving with Apache or your favorite web server.

With a basic directory layout and a simple configuration file creating a static copy of your blog is merely running the jekyll command in your blog directory.

stefan@macbook-pro:~/webs/juretta.com/www [0]$ tree -L 3
.
|-- Rakefile
|-- _config.yml
|-- site
|   |-- 404.html
|   |-- _includes
|   |   `-- entry-preview.html
|   |-- _layouts
|   |   |-- default.html
|   |   `-- post.html
|   |-- _posts
|   |   |-- 2008-11-11-hudson_test_drive_part_1_rails.html
|   |   |-- 2008-12-11-hudson_at_wellrailed.html
|   |   |-- 2009-05-24-activemq-jms-stomp.markdown
|   |   `-- 2009-05-29-goodbye-rails.markdown
|   |-- about
|   |   `-- index.html
|   |-- atom.xml
|   |-- favicon.ico
|   |-- images
|   |-- index.html
|   |-- javascripts
|   |-- rss.xml
|   |-- sitemap.xml
|   `-- stylesheets
`-- web

Running jekyll in this directory will copy all the static files in site into web and will transform your content files (HTML, Markdown, Textile based) applying a series of layout files (Read more on the Jekyll Wiki pages: Usage).

Running jekyll --server will transform the files and fires up a local WEBrick server running on http://localhost:4000.

stefan@macbook-pro:~/webs/juretta.com/www (master) [0]$ rake server
(in /Users/stefan/webs/juretta.com/www)
juicer merge --force --document-root ./site ./site/stylesheets/base.css
Produced site/stylesheets/base.min.css from
  site/stylesheets/undohtml.css
  site/stylesheets/screen.css
  site/stylesheets/form.css
  site/stylesheets/syntax.css
  site/stylesheets/base.css
jekyll --server
Configuration from ./_config.yml
Building site: site -> web
Successfully generated site: site -> web
[2009-05-29 17:49:10] INFO  WEBrick 1.3.1
[2009-05-29 17:49:10] INFO  ruby 1.8.6 (2007-09-24) [i686-darwin9.2.0]
[2009-05-29 17:49:10] INFO  WEBrick::HTTPServer#start: pid=7634 port=4000

Throw in some juicer goodness and a nice git post-receive hook and your new post is only a single git push away.

ActiveMQ, Stomp, JMS and the empty message body

Sun May 24 00:00:00 +1200 2009

For a project I am currently working on I needed to hook up BlazeDS and ActiveMQ. While everything went fine and without any disruption, I ran into a minor problem after I decided to use a Ruby based script to sent notifications to a topic based message queue.

The setup looks remotely like this:

BlazeDS and ActiveMQ are using the Java Message Service (JMS) API for message exchange. The Ruby script is using the Stomp (Streaming Text Orientated Messaging Protocol) interface to ActiveMQ.

Stomp is a really simple interface that looks vaguely like HTTP - it can even be used by using a simple Telnet connection to the message queue.

A simple stomp based message frame looks like this:

SEND
destination:/queue/a

hello queue a
^@

The message frame starts with a single Stomp command name (one of SEND, SUBSCRIBE, UNSUBSCRIBE, BEGIN, COMMIT, ABORT, ACK, DISCONNECT) followed by a newline, the message header consisting of key: value pairs, a blank line and the message body followed by a null character (NUL).

The Ruby based equivalent to send a simple text message looks like this:

require 'rubygems'
require 'stomp'

conn = Stomp::Connection.open 'USER', 'PASSWORD', 'localhost', 61613, true 
conn.send '/topic/messageTopic', "#{Time.now}", :persistent => 'true'
conn.disconnect

ActiveMQ is responsible for mapping Stomp messages to JMS and this is where the trouble began.

JMS defines five message types: TextMessage, MapMessage, BytesMessage, StreamMessage and ObjectMessage. Stomp does not define distinct message types, the content of the message body can be anything as long as you take care of proper handling of the terminating null character.

The result of sending messages using the Ruby Stomp based message producer was unexpected. All the messages had an empty message body when received by the Flex based clients.

Stepping through the BlazeDS code (the method messageReceived(JMSMessageEvent evt) in flex.messaging.services.messaging.adapters.JMSAdapter is a good candidate for a breakpoint) shows that all the messages sent by the Ruby Stomp client are converted to JMS BytesMessage?

ActiveMQ uses the content-length header to determine if a Stomp message should be mapped to a JMS TextMessage or a BytesMessage (http://activemq.apache.org/stomp.html) - if a Stomp message contains a content-length header field it will be converted to a BytesMessage otherwise it will be converted to a TextMessage.

Digging into the Ruby Stomp code shows that every message that is being sent by the Ruby client will contain the content-length header. This is the reason for the conversion to the BytesMessage and the empty message body when the message is received by the BlazeDS clients.

lib/stomp/connection.rb

def _transmit(s, command, headers = {}, body = '')
  @transmit_semaphore.synchronize do
    s.puts command
    headers.each {|k,v| s.puts "#{k}:#{v}" }
    s.puts "content-length: #{body.length}"
    s.puts "content-type: text/plain; charset=UTF-8"
    s.puts
    s.write body
    s.write "\0"
  end
end

The Stomp gem actually follows the Stomp specification.

From the Stomp protocol specification:

It is recommended that SEND frames include a content-length header which is a byte count for the length of the message body. If a content-length header is included, this number of bytes should be read, regardless of whether or not there are null characters in the body. The frame still needs to be terminated with a null byte and if a content-length is not specified, the first null byte encountered signals the end of the frame.

Using the stock Ruby stomp gem and a standard BlazeDS/ActiveMQ setup seems to be a bit of a problem. You either have to take care of the message conversion yourself or you have to change the behaviour of the stomp library. As the official Stomp protocol recommends the inclusion of the 'content-length' header the current implementation seems to be correct. Changing the default behaviour therefore does not seem to be the best option. I decided to add a monkeypatch that can be used with ActiveMQ and changes the method _transmit in order to suppress the content-length header.

module Stomp
  class Connection
    def _transmit(s, command, headers = {}, body = '')
      @transmit_semaphore.synchronize do
        unless headers['suppress_content_length']
          headers['content-length'] = "#{body.length}"
        else
          headers.delete 'suppress_content_length'
        end
        s.puts command
        headers.each {|k,v| s.puts "#{k}:#{v}" }
        s.puts "content-type: text/plain; charset=UTF-8"
        s.puts
        s.write body
        s.write "\0"
      end
    end
  end
endd

conn = Stomp::Connection.open 'USER', \
    'PASSWORD', 'localhost', 61613, true 
conn.send '/topic/messageTopic', "Wellington: (#{Time.now})", \
    'suppress_content_length' => true
conn.disconnect

This fixes the problem and results in the desired behaviour. A bug has been filed and we'll see if there is a good way to enhance the Stomp gem to work properly with ActiveMQ without changing the current behaviour.

Meanwhile a fork of the stomp git repository including the ActiveMQ changes (hopefully without breaking existing client code) can be found here:

http://github.com/juretta/stomp/tree/activemq-jms-mapping

Hudson at WellRailed

Thu Dec 11 00:00:00 +1300 2008

I had the pleasure to give a short talk about Rails and Hudson at the Wellington Rails user group (WellRailed) meeting in November.

I talked about using the Hudson extensible continuous integration engine to run Rails tests and using git as the SCM.

The slides are availble here: static.coravy.com/talks/WellRailed-Hudson-Rails-git.pdf

Hudson test drive (Part 1: Rails)

Tue Nov 11 00:00:00 +1300 2008

I've been using Hudson for quite a while now and it just rocks. Gone are my CruiseControl (for Java) and CruiseControl.rb (for Ruby/Rails) installations. I even ditched most of my crontab entries and have Hudson manage the repeated tasks for me.

Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Hudson focuses on the following two jobs: 1. Building/testing software projects continuously just like CruiseControl or DamageControl. [...] 2. Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. [...]

Compared to CruiseControl configuring jobs in Hudson is really easy and straightforward. You can easily give it a test ride without even installing Hudson in your servlet container. Let's give it a try.

stefan@macbook-pro:~$ curl -OL http://hudson.gotdns.com/latest/hudson.war
  % Total    % Received % Xferd  
                                 
100 19.6M  100 19.6M    0     0  
stefan@macbook-pro:~$ java -jar hudson.war 
[Winstone 2008/10/24 12:33:11] - Beginning extraction from war file
hudson home directory: /Users/stefan/.hudson

This will start an instance of hudson which can be accessed at http://localhost:8080/

Hudson will create a .hudson directory to store files and plugins. You can set the HUDSON_HOME environment variable to use a different directory.

stefan@macbook-pro:~$ export HUDSON_HOME=/opt/local/hudson

Due to its multi purpose nature it's easy to build different kinds of software projects. If you can build your projetc using a command line script you might be able to build it using Hudson.

Hudson and Rails

Part one covers building Rails projects with Hudson. I'm going to cover building Java and Flex projects in future posts.

In order to build a Rails project as soon as a commit is made, we need a couple of plugins. I'm going to use the git scm for this example. The default Hudson installation supports SVN out of the box but with the Hudson git plugin using git as your scm is a piece of cake.

Go to Manage hudson (http://localhost:8080/manage) > Manage plugins (http://localhost:8080/pluginManager/) and click on the Available tab. Choose the following plugins:

Git Plugin
Rake plugin
Ruby Plugin

and click on the Install Button (bottom right).

Hudson needs to be restarted in order to activate the plugins. Stop the currently running proceess (CTRL+C) and start Hudson again (java -jar hudson.war).

Configure your Rails project

Hudson can run your Rake test task without further modification. But it would be really nice if the test output could be aggregated in Hudson.

We are going to install the ci_reporter gem (http://caldersphere.rubyforge.org/ci_reporter/) for this purpose.

CI::Reporter is an add-on to Test::Unit and RSpec that allows you to generate XML reports of your test and/or spec runs. The resulting files can be read by a continuous integration system that understands Ant‘s JUnit report XML format, thus allowing your CI system to track test/spec successes and failures.

stefan@macbook-pro:~$ sudo gem install ci_reporter
Successfully installed ci_reporter-1.5.1
1 gem installed
Installing ri documentation for ci_reporter-1.5.1...
Installing RDoc documentation for ci_reporter-1.5.1...

Add the following lines to your Rakefile (or clone the git repository with the Rails example project:
git clone http://github.com/juretta/hudson-example-project-rails/tree/master).

require 'rubygems'
# Install:
#  sudo gem install ci_reporter
gem 'ci_reporter'
require 'ci/reporter/rake/test_unit' # use this if you're using Test::Unit

Run the following rake task and the ci_reporter gem should create xml files in the results directory of your project.

stefan@macbook-pro:~$ rake ci:setup:testunit test:units \
    test:functionals CI_REPORTS=results

Add a new job

Go to the Hudson homepage and click on New Job (http://localhost:8080/newJob). Choose a Job name, select Build a free-style software project and click Ok.

You should now be on the project configuration page (http://localhost:8080/job/YOUR-PROJECT-NAME/configure).

In the Source Code Management section choose git. Enter your git repository URL (The Rails project used in this example is hosted on github: git://github.com/juretta/hudson-example-project-rails.git).
Click on Add build step and choose Invoke Rake. Enter db:migrate ci:setup:testunit test:units test:functionals CI_REPORTS=results RAILS_ENV=test into the Task field.
In the Post-build Actions section check Publish JUnit test result report and enter results/*.xml into the Test report XMLs field.
Click Save

You can click Build now to trigger the first build of your Rails project.

We now want to trigger the build automatically as soon as we do a git commit. Edit the post-receive file in the hooks directory of your git repository (.git/hooks/post-receive).

URL='http://localhost:8080/job/YOUR-PROJECT-NAME/build'

echo "Run Hudson build"
wget $URL > /dev/null 2>&1

Make the file executable.

stefan@macbook-pro:~$ chmod u+x .git/hooks/post-receive

The hook script now triggers a build after each commit.

The Ruby metrics plugin

The Ruby metrics plugin shows Rails stats (LOC) and rcov reports. Unfortunately with the latest version of Hudson the Ruby metrics Plugin does not seem to work.
You should definitely give it a try. If you get an HTTP 500 Status code with a org.apache.commons.jelly.JellyTagException
remove (or disable) the following line:
<st:include it="${it.owner}" page="sidepanel.jelly" /> in
$HUDSON_HOME/plugins/rubyMetrics/WEB-INF/classes/hudson/plugins/rubyMetrics/railsStats/RailsStatsBuildAction/index.jelly

Have fun with Hudson!

URI-Template gem released

Sat Mar 15 00:00:00 +1300 2008

URITemplate is a parser for URI Templates as defined in the URI Template specification (http://bitworking.org/projects/URI-Templates).

URI Templates are strings that contain embedded variables that are transformed into URIs after embedded variables are substituted. This specification defines the structure and syntax of URI Templates.

A simple example shows the usage: http://example.com/search?q={searchTerms}&num={count} can be expanded to

http://example.com/search?q=hello%20world&num=5
http://example.com/search?q=ruby&num=100

by supplying different variables for searchTerms and count.

http://bitworking.org/projects/URI-Templates/ is the homepage of the current draft. http://code.google.com/p/uri-templates/ hosts the current reference implementation in Python.

Install the Ruby gem:

macbook-pro:~ stefan$ sudo gem install uri-templates

Treetop - go look at it

After playing around with Ragel and ANTLR I decided to use the excellent Treetop (Parsing expression grammar implementation) to implement the URI template specification. Although the draft explicitly states that you do not need a full full-fledged parser, Treetop made it really easy to implement just that.

Get the source:

macbook-pro:~ stefan$ git clone git://scm.juretta.com/uri-templates.git

New Zealand Daylight Saving Time Change: Mac OS X and Java

Fri Oct 05 00:00:00 +1300 2007

It appears that Apple missed the new rules for Daylight Saving Time (DST) in New Zealand in even the most recent versions of Mac OS X 10.4 Tiger.

From this year [2007], Daylight Saving will run for a period of 27 weeks. It now commences from the last Sunday in September, when 2.00am becomes 3.00am, and ends on the first Sunday in April the following year, when 3.00am becomes 2.00am.

Mac OS X still thinks that the Daylight Saving time starts on Oct. 7th.

The knowledgebase entry is rather disappointing

Deselect (uncheck) 'Set Date & Time automatically' if it's enabled.
Manually set the correct time.

A better approach is to update the timezone database that is used to compute the correct time for your timezone in relation to UTC (Read more):
GMT_TIME + LOCAL_TIME_ZONE_OFFSET + DST_OFFSET.

First check if you have to update the timezone information (assuming you have selected NZST/NZDT as your local timezone):

macbook-pro:~ stefan$ zdump -v /etc/localtime | grep 2007
/etc/localtime Sat Mar 17 13:59:59 2007 UTC = Sun Mar 18 02:59:59 2007 NZDT isdst=1 
/etc/localtime Sat Mar 17 14:00:00 2007 UTC = Sun Mar 18 02:00:00 2007 NZST isdst=0 
/etc/localtime Sat Oct 6 13:59:59 2007 UTC = Sun Oct 7 01:59:59 2007 NZST isdst=0 
/etc/localtime Sat Oct 6 14:00:00 2007 UTC = Sun Oct 7 03:00:00 2007 NZDT isdst=1

This shows that the Mac thinks that the daylight saving time (NZDT) will start on Sun Oct 7. That's wrong.

To update the Mac OS X timezone database download the latest tzdata2007_.tar.gz from ftp://elsie.nci.nih.gov/pub/. As of Oct 1 it's ftp://elsie.nci.nih.gov/pub/tzdata2007h.tar.gz

macbook-pro:~ stefan$ curl -O ftp://elsie.nci.nih.gov/pub/tzdata2007h.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  158k  100  158k    0     0   4851      0  0:00:33  0:00:33 --:--:--  7928

Extract the archive and run the zic timezone compiler

macbook-pro:~ stefan$ mkdir tzdata; tar -xvzf tzdata2007h.tar.gz -C tzdata
macbook-pro:~ stefan$ cd tzdata; sudo zic australasia

You might want to reboot the system to enable the new timezone information.

Check again:

macbook-pro:~ stefan$ zdump -v /etc/localtime | grep 2007
/etc/localtime  Sat Mar 17 13:59:59 2007 UTC = Sun Mar 18 02:59:59 2007 NZDT isdst=1
/etc/localtime  Sat Mar 17 14:00:00 2007 UTC = Sun Mar 18 02:00:00 2007 NZST isdst=0
/etc/localtime  Sat Sep 29 13:59:59 2007 UTC = Sun Sep 30 01:59:59 2007 NZST isdst=0
/etc/localtime  Sat Sep 29 14:00:00 2007 UTC = Sun Sep 30 03:00:00 2007 NZDT isdst=1

Java Update

The JRE uses its own timezone database (unfortunately). We need to update this information as well.

Got to http://java.sun.com/javase/downloads/index.jsp and get the JDK DST Timezone Update Tool - 1.2.2.

Expand the archive and run the tzupdater.jar: sudo java -Djava.vendor="Sun Microsystems Inc." -jar tzupdater.jar -v -u

macbook-pro:~ stefan$ sudo java -Djava.vendor="Sun Microsystems Inc." -jar tzupdater.jar -v -u 
Password:
java.home: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
java.vendor: Sun Microsystems Inc.
java.version: 1.6.0-dp
JRE time zone data version: tzdata2006a
Embedded time zone data version: tzdata2007g
Extracting files... done.
Renaming directories... done.
Validating the new time zone data... done.
Time zone data update is complete.

Quick check:

Calendar cal = new GregorianCalendar(TimeZone.getTimeZone("Pacific/Auckland"));
cal.setTimeInMillis(new Date().getTime());	
System.out.format("%1$tT (%1$tB %1$td %1$tY)", cal, new Locale("de", "DE"));

javap

Thu Oct 04 00:00:00 +1300 2007

The javap command disassembles a Java(tm) class file. The options determine the output. If no options are used, javap prints out the package, protected, and public fields and methods of the classes passed to it. The javap command displays its output to stdout.

Usage:

macbook-pro:~ stefan$ javap -private java.util.Calendar
Compiled from "Calendar.java"
public abstract class java.util.Calendar extends java.lang.Object implements java.io.Serializable,java.lang.Cloneable,java.lang.Comparable{
    public static final int ERA;
    public static final int YEAR;
    public static final int MONTH;
    public static final int WEEK_OF_YEAR;
    public static final int WEEK_OF_MONTH;
    public static final int DATE;
    public static final int DAY_OF_MONTH;
    public static final int DAY_OF_YEAR;
    public static final int DAY_OF_WEEK;
    public static final int DAY_OF_WEEK_IN_MONTH;
    public static final int AM_PM;
    public static final int HOUR;
    public static final int HOUR_OF_DAY;
    public static final int MINUTE;
    public static final int SECOND;
    public static final int MILLISECOND;
    public static final int ZONE_OFFSET;
    public static final int DST_OFFSET;
    public static final int FIELD_COUNT;
    public static final int SUNDAY;

List open ports on your machine (Mac OS X)

Wed Aug 08 00:00:00 +1200 2007

netstat on Mac OS X is not what you might be used to on a Linux machine (on Linux you can use netstat -atp | grep -i "listen"). To show all open ports applications on your machine are listening to you can use this workaround:

macbook:~ sudo lsof -i -P | grep -i "listen"
netinfod    33    root    7u  IPv4 0x3f76e08      0t0    TCP localhost:1033 (LISTEN)
AppleFile  142    root   29u  IPv6 0x3973c50      0t0    TCP *:548 (LISTEN)
AppleFile  142    root   30u  IPv4 0x3f76018      0t0    TCP *:548 (LISTEN)
mysqld     175   mysql   11u  IPv4 0x415c018      0t0    TCP *:3306 (LISTEN)
slpd       214    root    2u  IPv4 0x415c710      0t0    TCP *:427 (LISTEN)
eclipse    277 stefans   23u  IPv6 0x39733a0      0t0    TCP *:8118 (LISTEN)
eclipse    277 stefans   25u  IPv6 0x3973a24      0t0    TCP *:49372 (LISTEN)
eclipse    277 stefans   27u  IPv6 0x51d20e0      0t0    TCP *:6836 (LISTEN)
eclipse    277 stefans   29u  IPv6 0x51d2764      0t0    TCP *:14685 (LISTEN)
cupsd      279    root    0u  IPv4 0x3f75c9c      0t0    TCP localhost:631 (LISTEN)
java      1037 stefans    7u  IPv6 0x3972d1c      0t0    TCP *:1527 (LISTEN)
java      1052 stefans    9u  IPv6 0x3971538      0t0    TCP *:49979 (LISTEN)
java      1052 stefans   15u  IPv6 0x3972f48      0t0    TCP *:44441 (LISTEN)
java      1052 stefans   23u  IPv6 0x51d4174      0t0    TCP *:49987 (LISTEN)
java      1118 stefans   12u  IPv6 0x51d3240      0t0    TCP *:50055 (LISTEN)
java      1118 stefans   38u  IPv6 0x568f3a0      0t0    TCP *:50061 (LISTEN)
java      1118 stefans   48u  IPv4 0x3f76394      0t0    TCP *:8000 (LISTEN)
java      3422 stefans   11u  IPv6 0x5a28de8      0t0    TCP *:51669 (LISTEN)
java      3422 stefans   13u  IPv6 0x6111990      0t0    TCP *:8080 (LISTEN)
java      3422 stefans   14u  IPv6 0x56af764      0t0    TCP *:8443 (LISTEN)
java      3422 stefans   37u  IPv6 0x5f69764      0t0    TCP *:51672 (LISTEN)

From the lsof man page:

Lsof [...] lists information about files opened by processes [...]. An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.) A specific file or all the files in a file system may be selected by path.

Monitoring network traffic with ngrep

Fri Jul 27 00:00:00 +1200 2007

There are many tools to monitor network traffic. Wading through lines and lines of tcpdump output can be cumbersome sometimes. ngrep to the rescue. From their website:

ngrep strives to provide most of GNU grep's common features, applying them to the network layer.

To install ngrep on a Mac OS X machine you can use the port command if you have darwinports installed.

macbook:~ sudo port install ngrep

To monitor network traffic using the htt protocol use the following command:

macbook:~ sudo ngrep -d en0 -W byline port 80 ## T 66.35.250.209:80 -> 10.200.1.40:50710 [AP] HTTP/1.1 200 OK. Date: Fri, 27 Jul 2007 13:45:05 GMT. Server: Apache/1.3.33 (Unix) PHP/4.3.10. Last-Modified: Tue, 28 Nov 2006 13:54:21 GMT. ETag: "219bab-160c-456c3f8d". Accept-Ranges: bytes. Content-Length: 5644. Connection: close. Content-Type: text/html. [...]

To monitor the traffic to a certain host include the host information:

macbook:~ sudo ngrep -d en0 -W byline host google.com

Hint use ifconfigto figure out which device to use (in my case it is en0, it might be eth0 for example on your machine).




Ngrep homepage: http://ngrep.sourceforge.net/


For usage information you can find more information here: http://ngrep.sourceforge.net/usage.html.