semicomplete.com - Jordan Sissel

new keynav version available (20091108)

Sun, 08 Nov 2009 21:14 GMT

Hop on over to the keynav project page and download the new version.

The changelist from the previous announced release is as follows:

20091108:
  - Added xinerama support.
    * Default 'start' will now only be fullscreen on your current xinerama
    display. You can move between screens by using the move-* actions to move
    the current selection outside the border of the current screne.
  - All xdotool commands now return integers so we can forward their return
    status to the user.
  - Actually handle SIGCHLD now so the shell commands get reaped on exit.

Ruby metaprogramming will cost you documentation.

Sun, 08 Nov 2009 09:56 GMT

Ruby, like many other dynamic and modern languages, makes it easy for you to do fun stuff like metaprogramming.

Ruby, also like other nice languages, comes with a builtin documentation generator that scans your code for comments and makes them available in html and other formats.

... until you start metaprogramming.

Take a simple example, the Fizzler! The name of the class is unimportant; this class will simply provide a new way to define methods, simply for the sake of showing some metaprogramming and how ruby's rdoc fails on it.

class Fizzler
  def self.fizzle(method, &block)
    self.class_eval do
      define_method method, &block
    end
  end
end

class Bar < Fizzler
  # Print a worldly message
  fizzle :hello do
    puts "hello world!"
  end
  
  # A simple test
  def test
    puts "testing, 1 2 3!"
  end
end

# Now some sample code, let's invoke the new 'hello' method we generated with
# 'fizzle'.
bar = Bar.new
bar.hello

The output looks like this:

% ruby fizzler.rb 
hello world!

All is well! We are generating new methods on the fly, etc etc, all features of metaprogramming. However, we can never make this 'hello' method obviously available to the world via rdoc, at least as far as I can tell. The rdoc generated looks like this:

Note the lack of any mention of 'hello' as a method. I cannot simply do what works for lots of other normal ruby code and ask for the documentation of hello by running 'ri Bar#hello' - because rdoc simply doesn't see it.

I recall in python, if you were dynamically generating methods and classes, you could also inject their documentation by simply setting the '__doc__' property on your class or method. Ruby doesn't appear to have such a thing.

Additionally, in some metaprogramming cases, the stack traces are actually harder to read. For example, ActiveRecord makes extensive use of 'method_missing' rather than dynamically generate methods. The output is the same, but the stacktraces are now littered with 'method_missing' and references to files and lines you don't own, rather than containing stacktraces to named functions and other useful pointers. This perhaps is a feature, but for cases like method_missing, being able to add other useful data onto the stack trace would greatly aid in debugging.

So, if long term necessities like documentation and easy debuggability (stack traces, etc), are hindered by metaprogramming, at least in ruby, what are we left to do? Metaprogramming is clearly a win in some places, but the automatic losses seem to detract from any value it may have.

grok 20091103 release

Tue, 03 Nov 2009 09:54 GMT

Lots of changes since the last announced release. Grok should get some more activity now that I'm actually using it in a few places. If you find bugs or have feature requests, please file them on googlecode issue tracker (see below)

The largest changes are:

we ship with Ruby and C API.
lots of new testing code.
we now use tokyocabinet internally instead of bdb.

Grok documentation:: http://code.google.com/p/semicomplete/wiki/Grok
Download:: http://semicomplete.googlecode.com/files/grok-20091103.tar.gz
File bugs/features:: http://code.google.com/p/semicomplete/issues/list

This release has all tests passing in these configurations:

FreeBSD 7.1. tokyocabinet 1.4.30, pcre 8.00, libevent 1.4.12
Ubuntu 9.04. tokyocabinet 1.4.35, pcre 7.8-2, libevent 1.3e-3
CentOS 5.3. tokyocabinet 1.4.9-1, pcre 7.8-2, libevent 1.1a-3.2.1

Thanks to Pete Fritchman, grok also ships with an RPM spec so you can 'rpmbuild -tb grok-20091103.tar.gz' for simple build and deployment. The spec builds grok, grok-devel, and grok-ruby.

I'm using this version of grok myself with good success. It's also being used in the new logstash (log indexing tool) project for doing log parsing.

Full changelist since the last announced release:

20091103
 - New: ruby bindings are now really supported.
 - Change 'WORD' pattern to be word bounded (\b)
 - Move grok-patterns to patterns/base
 - update rpm spec to install patterns/base in /usr/share/grok

20091102
 - Add a bunch of tests, mostly in ruby, to exercise grok. This uncovered a
   few bugs which are fixed.
   All tests currently pass (both CUnit and Ruby Test::Unit) on:
   * FreeBSD 7.1. tokyocabinet 1.4.30, pcre 8.00, libevent 1.4.12
   * Ubuntu 9.04. tokyocabinet 1.4.35, pcre 7.8-2, libevent 1.3e-3
   * CentOS 5.3. tokyocabinet 1.4.9-1, pcre 7.8-2, libevent 1.1a-3.2.1
 - When making strings in ruby, we now make them tainted per ruby C docs.
 - "Too many replacements" error will now occur if you have cyclic patterns,
   such as defining 'FOO' to be '%{FOO}'. Max replacements is 500.

20091030
 - Make 'grok' main take a config for an argument.
 - Add grok rpm spec.
 - Updated Makefile to work on Linux and FreeBSD without modification.
 - Fixed bug introduced in 20091022 where capture_by_(name,subname) didn't
   work properly.
 - Add default values for match {} grok.conf blocks:
   shell: stdout
   reaction: "%{@LINE}"
 - Have grok exit nonzero if there were no reactions executed, akin to grep(1)
   not matching anything. 'reactions' are important here; matches with no
   reaction will not count as a reaction.

20091023
 - Fix libgrok accidentally sharing it's parser/lexer functions. Turns out,
   libgrok doesn't actually need to parse the grok.conf, so we don't build
   against it anymore for the library.

20091022:
 - Convert to using tokyocabinet instead of berkeley db.
   * Berkeley DB isn't easy to target across platforms (4.x versions vary
     wildly in bugs)
   * tokyo cabinet should be faster
   * tokyo cabinet is less code to write, and slightly more readable in the
     author's opinion.
   * we don't have to serialize with xdr anymore

20091019:
 - include pregenerated bison/flex output since gnu flex varies much from
 non-gnu flex, and many important platforms don't have gnu flex available
 easily from packages (freebsd, centos, etc) but come with the other flex.

 No functional changes.

Bringing test tools to Nagios monitoring

Fri, 30 Oct 2009 09:24 GMT

With all the TDD (test-driven design) and BDD (behavior-driven design) going around these days, it'd be a shame not to use these tools on monitoring applications.

You might have a boatload of tests that test your application before you roll a new version, but do you use those tests while the application is in production? Can you? Yes!

Let's take an important example of monitoring some complex interaction, like searching google and checking the results. Simple with a mouse, but perhaps complex in code. Even if you wrote a script to do it, using an existing testing framework gets you pass/fail testing automatically.

For this example, I'll use the following ruby tools: rspec and webrat. This fairly easy, though it took me a bit to find all the right documentation bits to clue me in to the right way.

require 'rubygems'
require 'webrat'

Spec::Runner.configure do |config|
  include Webrat::Methods
end

describe "google search for my name" do
  it "should include semicomplete.com in results" do
    visit "http://www.google.com/"
    webrat.response.title.should =~ /Google/
    query = "jordan sissel"
    fill_in "q", :with => query
    field_named("btnG").click
    webrat.response.title.should == "#{query} - Google Search"
    click_link "semicomplete.com"
  end
end

Now, we run this with the 'spec' tool:

% spec rspec-webrat.rb 
.

Finished in 0.578546 seconds

1 example, 0 failures

Seems ok. Let's break the test and see what happens. Change the 'visit' line to something else:

    visit "http://www.yahoo.com/"

Now rerun the test, which was checking specifically for google things in the page and will now fail on yahoo's page:

 % spec rspec-webrat.rb
F

1)
'google search for my name should include semicomplete.com in results' FAILED
expected: /Google/,
     got: "Yahoo!" (using =~)
./rspec-webrat.rb:29:

Finished in 0.186847 seconds

1 example, 1 failure

This output kind of sucks. Additionally, rspec failures seem to have exit code 1, not 2 as wanted by a nagios check reporting critical. Let's fix those. First, fixing the exit code can be hacked around directly in ruby if you want:

# Nagios checks expect exit code '2' to mean CRITICAL.
# Let's make any nonzero exit attempt always exit 2 (EXIT_CRITICAL).
EXIT_CRITICAL = 2
module Kernel
  alias :original_exit :exit
  def exit(value)
    value = EXIT_CRITICAL if value != 0
    original_exit(value)
  end
end

Fixing the output just means telling spec to use a different output format. I like the 'nested' output. Rerun that test now:

% spec -f nested rspec-webrat.rb
google search for my name
  should include semicomplete.com in results (FAILED - 1)

1)
'google search for my name should include semicomplete.com in results' FAILED
expected: /Google/,
     got: "Yahoo!" (using =~)
./rspec-webrat.rb:30:

Finished in 0.017534 seconds

1 example, 1 failure

% echo $?
2

All set.

Even better is that you can include multiple checks in the same script, if you wanted to. RSpec lets you select any test to run alone, so your nagios checks for a given web application could be a very simple:

define command {
  command_name check_google_for_semicomplete
  command_line /usr/bin/spec -f nested -e "google search for my name" mytests.rb
}

Net-SNMP and tcp-wrappers verbosity

Tue, 27 Oct 2009 03:27 GMT

I see this in my server logs quiet often:

Oct 23 05:37:48 pww-5 snmpd[23946]: Connection from UDP: [XX.XX.XX.XX]:34650 
Oct 23 05:37:48 pww-5 last message repeated 16 times
Oct 23 05:37:48 pww-5 snmpd[23946]: Connection from UDP: [XX.XX.XX.XX]:34652 
Oct 23 05:37:48 pww-5 last message repeated 24 times

Googling points out that in snmpd.conf we should use "dontLogTCPWrappersConnects" - but thet top search results claim that it doesn't work (syntax errors, etc). I tried this:

dontLogTCPWrappersConnects

This makes an error of:

/etc/snmp/snmpd.conf: line 29: Error: Blank line following dontLogTCPWrappersConnects token.

So I took a guess and changed it to:

dontLogTCPWrappersConnects 1

This works to quiet the 'Connection from UDP: ...' messages. However, it still logs things like:

Oct 22 23:17:35 pww-4 snmpd[29383]: Received SNMP packet(s) from UDP: [XX.XX.XX.XX]:42926

Fixing this requires telling snmpd to log less stuff to syslog. The '-L' logging options support upper-case versions which set the level at which it will log. Fixing syslog to not log the snmp packet info means setting this flag "-LSnd". This means we'll log at 'notice' levels and above to syslog with the daemon facility. Setting this flag seems to make snmpd less chatty in logs about packets it gets. Setting the log level to '-LSid' (info level) will make it once again log the packet receipts.

In CentOS (and other redhat variants) you'll edit this file to make this change permanent: /etc/sysconfig/snmpd.options - just change "-Lsd" (default in my version of net-snmp) to "-LSnd" and make sure the OPTIONS line is uncommented.

Ruby: Finding subclasses in your world

Tue, 29 Sep 2009 07:54 GMT

Use the ObjectSpace class to find all ancestors of a given class.

class Foo; end
class Bar < Foo; end
class Baz < Foo; end

subclasses = ObjectSpace.each_object(Class).select do |klass|
  klass.ancestors.include?(Foo) and klass != Foo
end

# prints "[Baz, Bar]"
puts subclasses

Of course, you could always override Class#inherited instead, but if you don't want to override methods, the above is a reasonable choice.

MySQL 5.0 'read-only' permits uncommitted writes

Mon, 28 Sep 2009 07:41 GMT

I recently had to do a master failover in mysql to bring up a new mysql master to replace an older one.

The switchover went awry shortly after we told the old master to start slaving off of the new master. The output of 'show slave status' indicated a halt of replication due to foreign key constraints: an auto_increment primary key had a duplicate insert attempt. How did this happen? I'm not sure yet, still digging.

This puzzle made me wonder how we got into that state given that I put the old master in 'read only' mode before doing the switch. Turns out, there are some edge cases that are permitted even in read-only mode. The docs have this to say:

When it is enabled, the server allows no updates except from users that have the SUPER privilege or (on a slave server) from updates performed by slave threads.

The above exceptions sound pretty reasonable, but I found an undocumented exception: uncommitted transactions can be committed even in read-only mode.

This breaks my expectation that setting read_only means that as soon as this setting goes 'true' all writes will fail. This means your backups aren't consistent when using read_only unless you lock all your tables during the backup. Additionally, commits made after read_only is set will bump the binlog position, meaning if your backups do "set readonly, copy master status, mysqldump", then you may break things because your mysqldump may have data that includes things in the future of the master status you recorded.

It looks like this is fixed in MySQL >=5.1.15. The docs say that later versions will cause setting read_only to block while there's a pending transactions.

Related to the original problem (replication failover), I think we should've just locked all the tables then restart the old master in read_only mode rather than simply setting read_only.

Below is an example of 'read_only' being set and a transaction commit as non-superuser resulting in a data write. I have aligned the actions side-by-side as they were executed chronologically.

Note the master status binary log position had changed after the commit. This is expected after a normal database write. You can also see the table was actually updated. I wasn't expecting any writes to succeed when read_only is set:

mysql as user 'test'	mysql as user 'root'
# As a user I created 'test': mysql> SET AUTOCOMMIT=0; mysql> CREATE TABLE foo.foo (a int) ENGINE INNODB; mysql> START TRANSACTION; mysql> SELECT * FROM foo.foo; Empty set (0.00 sec) # So far so good, let's insert. mysql> INSERT INTO foo.foo (a) VALUES (2); mysql> SELECT * FROM foo.foo; +------+ \| a \| +------+ \| 2 \| +------+ 1 row in set (0.00 sec) mysql> INSERT INTO foo.foo (a) VALUES (12345); ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement # We expected the above error, but can we commit # our previous insert before read_only was set? mysql> COMMIT; Query OK, 0 rows affected (0.01 sec)	# This is an empty result, since we haven't # yet committed. root@mysql> SELECT * FROM foo.foo; Empty set (0.00 sec) root@mysql> SET GLOBAL READ_ONLY = TRUE; root@mysql> SHOW MASTER STATUS \G ************************* 1. row **** File: mysql-bin.000001 Position: 644 Binlog_Do_DB: Binlog_Ignore_DB: 1 row in set (0.00 sec) root@mysql> SHOW MASTER STATUS \G *********************** 1. row ****** File: mysql-bin.000001 Position: 834 Binlog_Do_DB: Binlog_Ignore_DB: 1 row in set (0.00 sec) root@mysql> SELECT * from foo.foo; +------+ \| a \| +------+ \| 2 \| +------+ 1 row in set (0.00 sec)

mysql as user 'test'

mysql as user 'root'

# As a user I created 'test':
mysql> SET AUTOCOMMIT=0;
mysql> CREATE TABLE foo.foo (a int) ENGINE INNODB;
mysql> START TRANSACTION;
mysql> SELECT * FROM foo.foo;
Empty set (0.00 sec)

# So far so good, let's insert.
mysql> INSERT INTO foo.foo (a) VALUES (2);
mysql> SELECT * FROM foo.foo;
+------+
| a    |
+------+
|    2 | 
+------+
1 row in set (0.00 sec)









mysql> INSERT INTO foo.foo (a) VALUES (12345);
ERROR 1290 (HY000): The MySQL server is running 
with the --read-only option so it cannot execute 
this statement

# We expected the above error, but can we commit
# our previous insert before read_only was set?
mysql> COMMIT;
Query OK, 0 rows affected (0.01 sec)








# This is an empty result, since we haven't 
# yet committed.
root@mysql> SELECT * FROM foo.foo; 
Empty set (0.00 sec)





root@mysql> SET GLOBAL READ_ONLY = TRUE;
root@mysql> SHOW MASTER STATUS \G
*************************** 1. row ********
            File: mysql-bin.000001
        Position: 644
    Binlog_Do_DB: 
Binlog_Ignore_DB: 
1 row in set (0.00 sec)









root@mysql> SHOW MASTER STATUS \G
*************************** 1. row ********
            File: mysql-bin.000001
        Position: 834
    Binlog_Do_DB: 
Binlog_Ignore_DB: 
1 row in set (0.00 sec)

root@mysql> SELECT * from foo.foo;
+------+
| a    |
+------+
|    2 | 
+------+
1 row in set (0.00 sec)

Setuid bind(2) and switch!

Thu, 24 Sep 2009 07:20 GMT

Got a program that can't setuid but needs to listen on a priviledged port? I was hacking around with Linux's capabilities(7) tonight and came up empty trying to allow a non-priviledged user to bind to port 80 without having to start as root - after all, not everything is capable of setuid on startup, like many java programs.

Speaking of java, if you do setuid, the java hotspot monitor file thing (/tmp/hsperfdata_<user>/<pid>) is in the original user's directory, not the setuid'd user, so you can't jstack reliably. I might be PEBCAKing it, but this is the behavior I observe. Unless we can fix that, and my java-fu is weak, we'll have to find a workaround. One workaround is to run a proxy or firewall redirector that forwards the privileged port to the real port.

Linux supports this process setting that allows a special non-root process to listen on privileged ports, but I can't get it working. I gave up trying to use libcap's sucap, execcap, and setpcaps tools trying to allow nonroot processes to bind to lower ports. Let's hack it with LD_PRELOAD and execve(2).

The trick is creating a socket and binding on the correct port, then sharing that socket with your process. This comes with two steps, the socket creation, and sharing.

The creation is easy, for simplicity, I used ruby. The sharing requires LD_PRELOAD - we'll override bind(2) and have it dup our existing socket to whatever socket bind(2) is being called with. The sharing is done using environment variables to share the file descriptor number and the port number.

The end result looks like this, where we can now force programs that try to bind on our specified port to use our pre-created socket instead.

# id
uid=0(root) gid=0(root) groups=0(root)
# ./bindandswitch.rb
Usage: bindandswitch.rb [host]:port user group command [arg1 ...]
# ./bindandswitch.rb localhost:80 jls jls nc -l -p 80
setgid: 1000 (jls)
setuid: 1000 (jls)
Exec: ["nc", "-l", "-p", "80"]


# -- now in another terminal --
% lsof -i :80 | grep LISTEN 
 nc        27092  jls    4u  IPv4 8100070       TCP localhost:www (LISTEN)

See, now we have given 'nc' the ability to bind to privileged ports as non-root (user 'jls' above). Sweet!

The ruby code takes care of creating the socket and doing setuid (I was lazy and didn't want to write the C code). I also have a very short C library that simply dup2()'s the original bound socket when bind(2) is called by the new process on our specific port.

bind(2) calls for other ports than the one specified in the command line are handled normally. For example, this fails:

# ./bindandswitch.rb localhost:33 jls jls nc -l -p 80
setgid: 1000 (jls)
setuid: 1000 (jls)
Exec: ["nc", "-l", "-p", "80"]
Can't grab 0.0.0.0:80 with bind : Permission denied

The reason for this investigation was really for seeing how we could allow non-root java processes to bind to port 80. Maybe I'll use this at work if it behaves well.

Code:

bindandswitch.over (needs liboverride)
bindandswitch.rb

DNS Redirection and how it will break things

Wed, 23 Sep 2009 03:28 GMT

There's some hot debate around the implications and rightness of service providers to do things like traffic filtering, session hijacking, etc.

I'm not here to talk about that. The data below aims to focus on the technical failures induced by dns hijacking, or dns redirection. I won't bore you with the moral, political, or philosophical discussion around this topic.

Here's a summary if you don't want to read the details:

DNSBLs probably don't work anymore for Comcast users
Owned domains (semicomplete.com, google.com, etc) are also subject to hijacking. Domain owners cannot opt-out.
Down dns servers may result in full-domain hijacking by Comcast (due to dns search suffix and retry behavior)
Privacy/security/cookie leak due to domain hijacking
There is an IETF draft to formalize/standarize this destructive behavior that ignores consequences and impact, and neglects important points.
Sometimes Domain Helper malfunctions and steals valid hostnames, like www.google.com: screenshot here ; or a video
Web browsers aren't the only things using DNS.

All of the samples below are real data I observed while digging into this issue. They are not faked.

Comcast recently finished rolling out their new Domain Helper software. This tool intercepts responses from other DNS servers and replaces them with forged responses that point to comcast's search portal. It currently modifies responses that indicate 'no such hostname'; in DNS this response is called NXDOMAIN.

For example, if you try to visit 'http://www.someinvalidhost12345.com', it doesn't exist, and the DNS 'NXDOMAIN' response saying so is dropped and replaced with an A record pointing to Comcast's search site:

% host http://www.someinvalidhost12345.com
http://www.someinvalidhost12345.com has address 208.68.139.38

The domain helper documentation explains how it works, but incorrectly describes behavior and doesn't describe the consequences of the new behavior. From the docs, it sounds something "must" (per above) include "www". This is false, and is likely just out-of-date documentation, as the documentation goes on to explain that they "will eventually phase in the following pattern matches to enhance this service in the future." This probably explains why valid invalid.valid hostnames are bounced to comcast, and why invalid TLDs are bounced to comcast:

% host -t A abcdefghijklmnopww12345678.semicomplete.com
abcdefghijklmnopww12345678.semicomplete.com has address 208.68.139.38

% host -t A www.dns.is.broken
www.dns.is.broken has address 208.68.139.38

Simply having 'ww' is sufficient, even for valid domains like mine or totally invalid TLDs like 'broken'. Every NXDOMAIN is subject to hijacking. The IP address above is still comcast's search redirector.

So, what is the hard technical impact of this change?

DNSBLs may be broken

DNSBL is 'dns blacklist', check wikipedia for detail. Wikipedia says this about how DNSBLs are used:

Look up this name in the DNS as a domain name ("A" record). This will return either an address, indicating that the client is listed; or an "NXDOMAIN" ("No such domain") code, indicating that the client is not.

% host -t A 111.0.168.192.bl.spamcannibal.org
111.0.168.192.bl.spamcannibal.org.home has address 208.68.139.38
% host -t A 111.111.11.11.zen.spamhaus.org
111.111.11.11.zen.spamhaus.org.home has address 208.68.139.38

According to DNSBL de facto behavior, I should probably get NXDOMAIN given it's unlikely these IPs are in DNSBLs. Alternately, many DNSBLs reply with a special '127.0.0.xxx' address to indicate what the status of the address is if it is in the list. The responses above are neither, but are Comcast's search domain.

A Comcast employee on twitter pointed out that DNSBLs are only used by mail servers. Fine, we don't run mail servers on Comcast residental, but there are mail clients that support DNSBL for local spam filtering (thunderbird may, evolution appears to). The point is, DNSBL is not purely for 'mail servers', meaning this feature breaks DNSBL for clients.

I don't have data about how many people use DNSBL on mail clients. Point is, it's an option not limited to mail servers.

Down DNS servers are fully hijacked

When a DNS server is down, your DNS request generally times out and you get a SERVFAIL response. What happens when your local DNS client gets a SERVFAIL, though?

DNS resolvers support 'search suffixes'. In linux, you see this in /etc/resolv.conf as 'search some list of domains'. At my house, I have a 'home' dns zone managed automatically by my ddwrt router. This means I have 'search home' in my resolver.

If I try doing a dns query for 'some-invalid-host.something.com' and get NXDOMAIN or SERVFAIL, my resolver will try adding a suffix, if it's configured to do so. Here's an example of a dns server not responding and yielding SERVFAIL and the resolver's attempt to try with a suffix 'home' resulting in comcast NXDOMAIN hijacking me:

% host -t A wwwdead.evilcouncil.org 68.87.76.182
Using domain server:
Name: 68.87.76.182
Address: 68.87.76.182#53
Aliases:

wwwdead.evilcouncil.org.home has address 208.68.139.38

Sniffing my local network traffic with tshark during this lookup:
6.755838 192.168.0.97 -> 68.87.76.182 DNS Standard query A wwwdead.evilcouncil.org
6.770844 68.87.76.182 -> 192.168.0.97 DNS Standard query response, Server failure
6.771264 192.168.0.97 -> 68.87.76.182 DNS Standard query A wwwdead.evilcouncil.org.home
6.799348 68.87.76.182 -> 192.168.0.97 DNS Standard query response A 208.68.139.38

Interesting. Here's another data point:

% ping hijackmyww.evilcouncil.org
PING hijackmyww.evilcouncil.org.home (208.68.139.38) 56(84) bytes of data.
^C

Notice comcast's search portal IP repeated again and again...

If we use comcast's default search suffix for my area, hsd1.ca.comcast.net, what happens?; 'www.example.com' may SERVFAIL, then we retry as www.example.com.hsd1.ca.comcast.net, which comes back as comcast's search2.comcast.net, again, due to their NXDOMAIN intercept.

Now, any SERVFAIL may automatically direct you to through their 'dns helper' search redirect.

Security issues

The last issue is something I noticed last night while disconnecting my VPN to work. I had our monitoring site up in my browser - this page autorefreshes itself every few minutes. After disconnecting, it refreshed, and dumped me at search2.comcast.net.

Since the redirect happens in DNS, the browser will believe that you are still talking to the same server (by name) and will share cookies and http authentication freely. What happens if you visit 'www.mail.google.com' in Firefox? Let's look at the network:

snack(~) % sudo ngrep . 'port 80 and host 208.68.139.38'
[sudo] password for jls:
interface: eth0 (192.168.0.0/255.255.255.0)
filter: (ip or ip6) and ( port 80 and host 208.68.139.38 )
match: .
####
T 192.168.0.97:44422 -> 208.68.139.38:80 [AP]
  GET / HTTP/1.1..Host: www.mail.google.com..User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.3) Gecko/20090824 Firefo
  x/3.5.3..Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept-Language: en-us,en;q=0.5..Accept-Encoding: gzip,
  deflate..Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7..Keep-Alive: 300..Connection: keep-alive..Cookie: PREF=ID=0cc23ef3118faec6:U=fffb...

The above shows Firefox sending Comcast my google mail login cookie. Had I been visiting a page that uses basic HTTP authentication, my username and password would've been sent, too.

Sure, cookies have a 'secure' attribute which tells the browser to only send them over SSL connections, but does everyone use them? Nope. Bank of America, for example, doesn't set this flag, which is a security issue itself, but is leaked to Comcast on typoed URLs or during 'domain helper' malfunctions.

Firefox has no idea that 'www.mail.google.com' isn't Google, or www1234.bankofamerica.com isn't Bank of America, and a normal user won't have any idea they just leaked their credentials, private session info, and other cookie data.

Domain Helper Malfunction

Ofcourse, the worst part of this whole system is when the hijacking itself misbehaves, and you get 'www.google.com' hijacked, when it's a real hostname.

Defenses and Actions

How can we prevent some or all of this? Contact your ISP, help them understand the impact of this behavior. Pushing at least to allow domain owners to opt-out of the domain redirection might be a start or to require that providers not hijack owned domains. Not hijacking invalid TLDs would be good too, given the common usage of 'corp' and others as an intranet TLD.

If you are a comcast user and want to opt-out of Comcast's Domain Helper, you can visit dns-opt-out.comcast.net (requires login, only works when viewed from comcast). Alternately, you can point your local dns caches at comcast's non-domain-helper servers

GDB for poking at libc to test random things

Tue, 22 Sep 2009 03:22 GMT

I wanted to test something quickly out in C, but didn't want to write the 5 line of code to do it. Having done some fun ruby debugging with gdb recently, I decided to go with that.

% gdb -q `which sleep` --args `which sleep` 60000
(gdb) break nanosleep
(gdb) run
Starting program: /bin/sleep 60000
[Thread debugging using libthread_db enabled]
[New Thread 0x7f8c40bc46f0 (LWP 6504)]
[Switching to Thread 0x7f8c40bc46f0 (LWP 6504)]

Breakpoint 1, 0x00007f8c404f7ce0 in nanosleep () from /lib/libc.so.6
(gdb) call strcspn("hello world", "w")
$1 = 6

I don't know why I didn't think about this before. This is nicely useful, allowing me to easily test any simple function call unrelated.