Cain Manor

Basic Samba-Active Directory troubleshooting using wbinfo

Greg Cain — Wed, 08 Sep 2010 19:25:39 +0000

This document is intended to step you through some of the basics of troubleshooting samba connectivity with your domain controller, using wbinfo. This is not intended to show you how to configure Samba to talk to your domain controller. Others have done a much better job that I can.

winbind must be running in order for wbinfo to work, and you need to be connected to your domain.

wbinfo -p
Ping to winbindd succeeded on fd 4

A simple ping test to see if your DC is alive

wbinfo -t
checking the trust secret via RPC calls succeeded

Verify that the workstation trust account created when the Samba server is added to the Windows NT domain is working. This is a good first step to make sure you can talk to your DC

wbinfo -u

This will list your domain users. Make sure the users you expect are there

wbinfo -a greg.cain%password

This is an easy way to check that your user can authenticate against the domain. However, I’d recommend against using it since it will show your password on the command line, in your history, and in the ps command if it’s executed at the right time (there are many other reasons, but those should be enough to dissuade you.) Use a dummy account if possible.

wbinfo -g

Show a list of domain groups. Again, make sure the groups you expect are present.

wbinfo -n greg.cain
S-1-5-21-XXXX789314-3545391909-2802175779-XXXX User (1)

This will get you the SID for the user you’ve specified (greg.cain in this case.)

wbinfo -s S-1-5-21-XXXX789314-3545391909-2802175779-XXXX
DOMAIN\greg.cain

Using the SID you just returned, let’s make sure the reverse mappings are correct.

wbinfo -r greg.cain
16XXX216
16XXX217
....

This will show a list of groups that the user greg.cain belongs to.

wbinfo -G 16XXX217
S-1-5-21-1540789314-3545391909-2802175779-YYY

This converts the GID to the SID.

wbinfo -Y S-1-5-21-1540789314-3545391909-2802175779-YYY
16XXX217

This is the reverse mapping of the SID to the GID

Setup logging for Varnish – The right way

Greg Cain — Fri, 30 Jul 2010 02:45:45 +0000

Initially, logging Varnish didn’t really work that well. Varnish, by default, doesn’t log (varnishncsa can do some logging, but I haven’t figured that out yet.) I’ve been parsing my logs with JAWstats, which reads the Apache logs. However, by default, varnish will replace the IP address of the originating client with the IP address of the proxy server, meaning all traffic will look like it comes from my single server. This is what an entry looks like

173.230.157.240 - - [29/Jul/2010:15:47:47 -0700] "GET / HTTP/1.1" 200 20046 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
173.230.157.240 - - [29/Jul/2010:15:47:48 -0700] "GET / HTTP/1.1" 200 20046 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
173.230.157.240 - - [29/Jul/2010:15:47:49 -0700] "GET / HTTP/1.1" 200 20046 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
173.230.157.240 - - [29/Jul/2010:15:47:50 -0700] "GET / HTTP/1.1" 200 20046 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
Yeah, I made this up, but you get the idea.

That’s no good. How will I tell that I’m famous in Sweden? I found a way to get the client IP information passed to the backend

sub vcl_recv {
       remove req.http.X-Forwarded-For;
       set    req.http.X-Forwarded-For = client.ip;
}

This is what we get from this.

207.218.231.170, 207.218.231.170 - - [29/Jul/2010:15:47:47 -0700] "GET / HTTP/1.1" 200 20046 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"

Ok, this is better, but the IP address is duplicated. I don’t know if it causes trouble with JAWstats (I assume it does), but I know that it is ugly to me. After much googling, I found this…

sub vcl_recv {
# 	Rename the incoming XFF header to work around a Varnish bug.
  if (req.http.X-Forwarded-For) {
#	Append the client IP
    set req.http.X-Real-Forwarded-For = req.http.X-Forwarded-For ", " regsub(client.ip, ":.*", "");
    unset req.http.X-Forwarded-For;
  }
  else {
    // Simply use the client IP
    set req.http.X-Real-Forwarded-For = regsub(client.ip, ":.*", "");
  }
}

And finally we have a working log file.

This is the first log file from this server

82.103.128.63 - - [04/Jul/2010:03:57:47 -0400] "GET / HTTP/1.0" 200 23301 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"

And this is the last

24.19.50.69 - - [29/Jul/2010:19:44:18 -0700] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 177 "http://cainmanor.com/wp-admin/post-new.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16"

Setup logging for Varnish

Greg Cain — Sun, 25 Jul 2010 19:10:11 +0000

So far, we’ve got Varnish installed and have a correct default.vcl. Now let’s make sure AWStats/JAWStats can read them.

In your default.vcl, make sure you have this stanza. Otherwise, the Apache log files will only show traffic from your proxy server IP address. This allows the client.ip to be forwarded to your log file.

sub vcl_recv {
  # Add a unique header containing the client address
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;
}

Now we need to setup Apache to log correctly. You need to add this line your httpd.conf to get the correct logging format. Notice the http.X-Forwarded-For we just set.

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined

Further down (or in your virtual server.conf file, if you break those out), we need to set our server to use that custom logging. I have a domain (cainmanor.com) and a subdomain (photos.cainmanor.com), so I just pointed both virtual host to the same log file.

<VirtualHost *:8080>
        DocumentRoot /wordpress
        ServerName cainmanor.com
        CustomLog /var/log/httpd/cainmanor.com-access.log varnishcombined
VirtualHost>

Once you’ve done this, you should be able to look in your custom log (/var/log/httpd/cainmanor.com-access.log) and see your traffic coming from varied IP addresses.

Now we tell AWStats where to find the custom log ( /etc/awstat/awstats.cainmanor.com.conf ) we created.

#LogFile="/var/log/httpd/access_log" <- OLD
LogFile="/var/log/httpd/cainmanor.com-access.log"

Now all you need to do is restart varnish and apache, and you should be logging successfully.

Happy Computing!!

Updated Varnish default.vcl

Greg Cain — Sun, 25 Jul 2010 18:50:19 +0000

I wasn’t able to get the varnish logs in a format awstats/jawstats could understand. After a LOT of googling, I’ve gotten this to work. I added

sub vcl_recv {
  # Add a unique header containing the client address
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;
}

as you can see below.

[ root@CainManor:/etc/varnish ] cat default.vcl
backend default {
.host = "173.230.157.240";
.port = "8080";
}
 
sub vcl_recv {
# Add a unique header containing the client address
# This is so we can forward the headers to the Apache logs.
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;
}
 
sub vcl_recv {
# Let's make sure we aren't compressing already compressed formats.
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|mp3|mp4|m4v)(\?.*|)$") {
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            remove req.http.Accept-Encoding;
        }
    }
# Remove cookies and query string for real static files
    if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.*$", "");
    }
# Remove cookies from front page
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }
}

How to use Varnish cache with WordPress

Greg Cain — Sun, 25 Jul 2010 04:41:00 +0000

I’ve been wanting to play around with a Web Cache/proxy in front of my web server, just to see how hard it would be and how well it would work.

Initially, I was going to try Squid, which is used far and wide. Pretty quickly I found out about Varnish, which is supposed to be easy to use, and quick to install. That may have been true, but getting it to work with WordPress was relatively painful for me. However, once I got it working, it made some crazy improvements in the speed of this site.

I won’t go into too much detail about the more standard parts. Should you need help with things like installing or finding packages, Varnish might not be for you. I will also assume you have a fully functional WordPress installed on Apache 2.

Install Varnish

Configure your Varnish config file (/etc/varnish/default.vcl.) Here is mine (also, check THIS post.)

backend default {
.host = "173.230.157.240";
.port = "8080";
}
 
sub vcl_recv {
# Normalize Content-Encoding
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|lzma|tbz)(\?.*|)$") {
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            remove req.http.Accept-Encoding;
        }
    }
# Remove cookies and query string for real static files
    if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") {
       unset req.http.cookie;
       set req.url = regsub(req.url, "\?.*$", "");
    }
# Remove cookies from front page
    if (req.url ~ "^/$") {
       unset req.http.cookie;
    }
}
sub vcl_fetch {
        if (req.url ~ "^/$") {
                unset obj.http.set-cookie;
        }
}

Regretfully I don’t remember where I got that particular file, but I’m going to see if I can find it again. It’s perfect for what we need to do. All you need to do is change your IP to match your host. By doing this, you’ll be pointing at your web server (called backend.) We’re going to change your web server to listing on 8080, and varnish will answer all normal port 80 traffic.

Now to configure Apache. Edit your /etc/httpd/conf/httpd.conf, and look for these lines, and add 8080 to the end (the majority were just 80.)

Listen 127.0.0.1:8080
Listen 173.230.157.240:8080
 
# for each of your Virtual host
NameVirtualHost *:8080

Now stop apache, and stop and start varnish, and start apache.

There are a few ways to see if it’s working.

varnishstat
varnishhist -d
varnishtop

Probably the most fun is to use ab (Apache Benchmark.) I did this from my desktop client against my Linode 512MB slice.

First, let’s look at our web server directly (via 8080 we changed it to.)

ab -c 100 -n 1000 http://cainmanor.com:8080/
This is ApacheBench, Version 2.3 &lt;$Revision: 655654 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking cainmanor.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
Server Software:        Apache
Server Hostname:        cainmanor.com
Server Port:            8080
 
Document Path:          /
Document Length:        0 bytes
 
Concurrency Level:      100
Time taken for tests:   54.226 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      291000 bytes
HTML transferred:       0 bytes
Requests per second:    18.44 [#/sec] (mean)
Time per request:       5422.552 [ms] (mean)
Time per request:       54.226 [ms] (mean, across all concurrent requests)
Transfer rate:          5.24 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       28   35   6.4     33      72
Processing:    89 3852 7070.4   1924   48164
Waiting:       89 3852 7070.3   1923   48163
Total:        119 3887 7071.0   1957   48202
 
Percentage of the requests served within a certain time (ms)
  50%   1957
  66%   1997
  75%   2041
  80%   2091
  90%   4021
  95%  18134
  98%  35079
  99%  41195
 100%  48202 (longest request)

Now let’s go to port 80 directly

ab -c 100 -n 1000 http://cainmanor.com/
This is ApacheBench, Version 2.3 &lt;$Revision: 655654 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking cainmanor.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
Server Software:        Apache
Server Hostname:        cainmanor.com
Server Port:            80
 
Document Path:          /
Document Length:        18654 bytes
 
Concurrency Level:      100
Time taken for tests:   3.507 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      19276653 bytes
HTML transferred:       18960147 bytes
Requests per second:    285.17 [#/sec] (mean)
Time per request:       350.665 [ms] (mean)
Time per request:       3.507 [ms] (mean, across all concurrent requests)
Transfer rate:          5368.33 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       30   75 127.0     52    1052
Processing:   121  254 153.4    203    1104
Waiting:       33   78  86.4     54     984
Total:        167  329 189.5    262    1235
 
Percentage of the requests served within a certain time (ms)
  50%    262
  66%    299
  75%    317
  80%    343
  90%    481
  95%    767
  98%   1125
  99%   1143
 100%   1235 (longest request)

I also tried to use W3 Total Cache and memcached at the same time, but I got slightly slower response times.

These are impressive improvements.
Before – Time taken for tests: 54.226 seconds
After – Time taken for tests: 3.507 seconds

Have Fun