My wife does not share this tolerance. She absolutely hates when a site she’s researching doesn’t load. The swirly browser “I’m thinking” icon sits there, taunting her, mocking her, bringing out the most vile of phrases from her mouth. If I were the Internet, I’d be hiding under the bed. For serious.

OK, not really that bad. But she’s quite vocal when it happens. I finally decided to do something about it.

Step 1 - Call Comcast. The friendly technician explained that the status information she was getting from the cable modem pointed toward a bad connection between the modem and the wireless router I used. Two possibilities there… bad router, or bad cable. I had already recently upgraded from an aging Linksys WRT54G to a D-Link DIR-655 Extreme N Wireless Router, so I more or less ruled that out. Cable? Easy enough to swap. Net result? No noticeable problems for a few days, then back to the same routine.

Step 2 - I swapped out the D-Link router for an Apple AirPort Extreme not too much later (not in an attempt to resolve this issue, but for research purposes). Problem stayed. This prompted me to explore further.

Step 3 - Question the wiring leading up to the cable modem. Took a gander in the basement. Can you spot the problem?

That’s an 8-way passive splitter. Each channel being attenuated by -11dB. That works out to each cable coming out of the splitter having less than 8% of the power of the signal that went in to the splitter. This has been this way since I bought the house, and never really paid attention to how the incoming lines were split.

I put a cheap $10 RadioShack 2-way passive splitter (each channel attenuated by -3.5dB, losing roughly 50% of the original signal strength) in front of the 8-way. One of those 2-channels heads straight to the office, the other into the 8-way to be distributed throughout the house.

The result? Too early to tell for reliability. BUT… I happened to run a speed test before and after the modification.

Cable Modem Speed with 8-way Splitter

Cable Modem Speed with 2-way Splitter

That’s an order of magnitude increase in upload speed. Boo-ya-ka-sha.

Battery Backup

The aging battery in the Back-UPS 280 on the left was no longer cutting it for keeping my network up and running when the power occasionally drops out at home. At minimum, I don’t want local network traffic or my hard drive backups over the network getting messed up if the power flickers. With the new unit on the right, I can at least get through a 10 minute outage with minimal impact.

Printers

My print volume at home is relatively low, but what would often happen using an old inkjet printer is that after several weeks of lack of use, to suddenly be expected to fire off 30 or 40 pages at a time would result in clogged print heads, dried ink, stuck gears, etc. Back in 2004, I picked up the LaserJet in the discard pile at CRA in Cambridge, MA, and used it successfully at home for a few years, then at a startup for a few years, until finally it sat idle after the startup closed down, one day hoping that I would find time to get it up and running at home.

Upgrading a 10+ year old inkjet for a 10+ year old laser? Odd, I know. That is, if you’ve never come across the marvelous workhorse that is the HP LaserJet 4M Plus. This thing just doesn’t quit. 281,000 pages printed and counting.

Now we can scratch “new color laser printer” off of the Christmas list, since between the Epson R300 inkjet for photos, and the LaserJet for the basics, we’re covered.

Degraded RAID Array

Our home file server for pictures and music has been running on a degraded RAID 1 configuration for some time now, thanks due to a failed SATA controller. I’ve had the new controller (only $13 on Newegg!) for some time now, but just needed to get it in there and restore the array.

Man, I do love software raid. This truly was as simple as:

1. Power off
2. Install RAID card
3. Hook up second drive
4. Power on
5. mdadm –manage -a /dev/md0 /dev/sdc

From there, it took a few hours to rebuild the array, but now all is well and protected in the event of a hard drive failure on our file server.

Upgrade the DVR wiring for Gigabit

We’re constantly moving large file around the home network, and 100Mbps Fast Ethernet to the DVR was becoming a noticeable factor. Everything supported Gigabit (even the switch in the basement attached to the server farm), except the old wiring. With that upgraded, it’s now Gigabit through and through. Score.

Wireless N for … Wireless N?

Ok, I know what you’re thinking. Either that I’m clueless, or that I’m swimming in enough money to swap out Wireless N routers willie-nillie (which, if you’re a reader of my wife’s blog, you know is not true).

For now, let’s just say that this is an initial foray into Zeroconf networking and Wide-area Bonjour with DynDNS.

Here’s an example shot with a Logitech QuickCam Pro 9000 webcam using the built-in microphone, standing roughly six feet away from the camera:

It’s OK, but not the greatest. Here’s the same Logitech webcam, but this time sitting in front of the camera, roughly two feet away, with the direction of speaking always toward the camera (and thus, the built-in microphone):

MUCH better! But we still want to be able to do recordings at the whiteboard, walking around, etc.

Here’s an example using the mic built-in with one of our camcorders (an Insignia HD Flash camcorder, which provides amazing video quality for the money):

What’s the ultimate solution? Well, a high-quality lavalier microphone, of course! And we have one… it’s the Audio-Technica ATR-35S.

So, why then, on our latest (and also most popular) videos, do we still have not the greatest sound in the world?

Insignia camcorder with built-in mic, but noisy environment in the middle of office, with lots of moving around by the speaker (relative to the mic):

Same Insignia camcorder and built-in mic, but quieter area with more control of what’s going on:

Same Insignia camcorder and built-in mic, but a better job of noise reduction in post-production:

So, I ask again, if we have the lavalier microphone, why haven’t we been using it? The answer is … we don’t have any place convenient to plug in the lavalier microphone when using our camcorders.

The camcorders themselves unfortunately do not have external microphone inputs.

We’ve tried using the microphone inputs for some laptops and desktops we had handy (and synching the audio and video in post-production), but the poor quality sound cards built-in clearly had some grounding issues, because a hefty hum could be heard (and often overheard) in the background. Plus, this often meant a long cable from the lavalier microphone to the computer, which was likely to be tripped over. No good.

But as of today, that changes. I just picked up a Sony ICD-UX70 digital voice recorder for the video production team, with built-in stereo microphones, an external microphone jack, and MP3 encoding of audio files. Now, we can throw this in the shirt pocket for whoever is speaking on video (when we’re in a rush), or for the best quality, just have the actress or actor hold it/put it in a pocket and wire up the lavalier microphone to walk around freely.

We’re shooting some video this week… I’ll let you know how it works out (hopefully you’ll be able to hear the difference for yourself!).

Here’s to good quality audio.

What’s not so easy is trying to secure and lock down the individual virtual servers so they cannot interfere with one another’s network connection. The efforts spent on IPv6 also exposed some issues for IPv4 and ARP that could arise when used in a VPS environment.

Note: We’re going to get down and dirty into the technical nitty gritty, and it’s beyond the scope of this post to bring all of my readers up to speed. The faint of heart may wish to turn back now.

Problem Cases for IPv6

Two main network spoofing cases come to mind that need to be protected against for running IPv6 on Xen in a virtual private server environment:

1) A user changing the IPv6 address of their network interface to an IPv6 address we haven’t authorized them to use, or adding/aliasing additional IPv6 addresses to their network interface that they did not pay for.

2) A user changing/adding/aliasing an IPv6 address that’s in use by another customer, which can wreak all sorts of routing havoc, interfering with the ability for our customers to reliably communicate with others on the Internet.

Perhaps Luke Crawford put it best with his response to my question, “Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I’d be happy to submit the changes and a guide to making it work and work well.” on the Xen developers email list:

Guides would be awesome. Also, if you made anti-spoof work for IPv6, that would also be awesome. I’ve had IPv6 going for a while with no problems (though the xen antispoof rules only work with IPv4), but I didn’t even need to enable IPv6 in the Dom0; it’s a layer 2 bridge, so it just works. In fact, stateless autoconfiguration even worked. I asked for an IPv6 allocation from my provider and my customers noticed it was working before I got around to doing any setup at all.

Problem Cases for IPv4 and ARP

When we initially went down the IPv6 road, we started paying closer attention to what was already implemented for IPv4 protection to get an idea for where to begin. In doing so, we found a couple of scenarios where the existing anti-spoof protection leaves Xen a bit too exposed for using as a VPS platform:

1) If a user aliases an IPv4 gateway IP address for the same subnet that user is on, all network connectivity can go down for that subnet. As an example, if your IP address is 1.2.3.5/24 and your gateway is at 1.2.3.1, and you run “ifconfig 1.2.3.1″ to alias the gateway IP address on your default network interface, you will now respond to ARP requests trying to find the gateway with your MAC address… meaning no other users on that subnet can reliably talk to the gateway… meaning their network connectivity is pretty much useless.

2) If a user aliases an IPv4 address that is already in use by another user, all network connectivity can go down for that other user, for the same reasons it goes down when aliasing the gateway IP address (i.e., you’ll be responding to ARP requests for their IP address).

3) Users can deliberately perform ARP poisoning attacks, by sending arbitrary ARP replies on the subnet, which can interfere with authorized communication for the same reasons as 1 and 2 above.

4) Users can see the network traffic for other domUs on the same dom0 by putting their interfaces into promiscuous mode (i.e., packet sniffing). 

My intent here is not to knock the Xen development community; they’ve put together a fantastic platform, and no one expects our business requirements for using Xen to host virtual private servers to be inline with their development requirements. I’m not even certain that we’ve squashed the issues entirely, but we have at least squashed the pressing issues for IPv6, IPv4, and ARP outlined above.

Now that we have it working in the field, I’d like to share the handful of changes with the community so they can benefit, and perhaps suggest better/more complete/more succinct ways of doing it. The solution we implemented was modeled after the current IPv4 anti-spoof protection.

I will start off with an overview of the current IPv4 anti-spoofing approach used by Xen, followed by the specific commands used to alter the behavior for what we needed for reliable and secure communication, concluded by the actual changes we made to the Xen networking scripts to enable this behavior modification.

Current IPv4 Anti-spoofing Protection for Xen

The current IPv4 anti-spoof protection scheme relies on iptables, in particular the FORWARD chain that each packet destined for a virtual server hits on its way in or out. Rules get added each time a virtual network interface comes up, and removed when the interface does down. 

For the sake of the remaining discussion, where I’ll be walking through some command examples, we’ll assume:

• peth0 is the physical network interface that the bridge connects to
• vif1.0 will be a virtual network interface connected to a domU, and the domU sees that device as eth0
• The domU is “allowed” to use IPv4 address 216.146.46.43/24, with a gateway IP of 216.146.46.1 for IPv4. 
• The domU is “allowed” to use IPv6 address 2607:f590:0:ffff::60/64, with a gateway IP of 2607:f590:0:ffff::1 for IPv6.
• The MAC address of the virtual network interfaced vif1.0 is 00:16:3E:38:B4:AC.
• The domU hostname is www.standingonthebrink.com.

Yes, you guessed it… all of the settings above are the real-world settings for the Xen VPS that is hosting this blog (with the exception of the virtual network interface name of vif1.0, which will change over time as the VPS is powered off and on).

In a nutshell, when you enable anti-spoofing as a parameter in the network-bridge script, the following iptables commands are issued when the bridge comes online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD  

  # Accept packets that are entering the bridge from the physical
  #   network interface peth0.
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

Now, when each domU virtual network interface comes online, the following iptables command is run for each IP address that the domU is allowed to use, as specified in the www.standingonthebrink.com.cfg file found in /etc/xen/:

  # Accept packets coming into the bridge from the domU if its
  #   source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

When the domU’s virtual network interface goes offline, the same command is run except the -A is changed to -D to delete the rule.

Into the Bridge? Out of the Bridge? What’s the Difference?

It’s important to point out the difference between “into” and “out of” the bridge. For the purpose of thinking about rules governing the packets that are allowed to traverse the bridge, it can be helpful to think of the bridge as a little switch connecting the physical network interface of the server to the virtual network interface of the dom0 as well as all of the virtual network interfaces for each of the domUs. The rules we write in the FORWARD chain will tell the bridge what packets to allow to enter and/or leave the bridge.

All of the rules we’ll write are from the perspective of the network bridge. Perhaps a couple examples will help convey the concept:

• A packet coming from the Internet destined for the first domU -> Comes IN peth0 and goes OUT vif1.0.
• A packet coming from the first domU destined for the second domU -> Comes IN vif1.0 and goes OUT vif2.0.
• A packet sent from the first domU to somewhere on the Internet -> Comes IN vif1.0 and goes OUT peth0.

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

Proposed Solution for ARP Poisoning Protection for Xen

Quite simply, we use arptables in a similar manner to using iptables. It can be a little tricky getting the rules right for what valid ARP traffic should be allowed, but with quite a bit of trial-and-error, we think we have it.

When the bridge comes online, we do:

  # Default policy for packets in the FORWARD chain is DROP.
  arptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  arptables -F FORWARD

When each virtual network interface comes online, we do:

  # Accept ARP requests coming from the domU into the bridge.
  arptables -A FORWARD --opcode Request --in-interface vif1.0 -j ACCEPT

  # Accept ARP requests coming out of the bridge into the domU.
  arptables -A FORWARD --opcode Request --out-interface vif1.0 -j ACCEPT

  # Accept ARP replies coming out of the bridge from the physical
  #  network into the domU.
  arptables -A FORWARD --opcode Reply --out-interface vif1.0 \
    --in-interface peth0 -j ACCEPT

In addition to the above rules that get added when the virtual network interface comes online, we do the following for each valid IPv4 address:

  # Accept ARP replies coming from the domU into the bridge if they
  #  provide a valid and authorized IP address to MAC address pair.
  arptables -A FORWARD --opcode Reply --in-interface vif1.0 \
    --source-ip 216.146.46.43 --source-mac 00:16:3E:38:B4:AC -j ACCEPT

It’s this last rule that prevents the ARP poisoning attacks and the havoc wreaking by aliasing gateway IP addresses. If invalid ARP replies are denied, as far as I can tell, no harm can be done. When the virtual network interface goes offline, the same commands are run with -D instead of -A to delete the rules.

Proposed Solution for Preventing IPv4 Packet Sniffing

As mentioned earlier, the current IPv4 rules that prevent anti-spoofing are:

  # Accept packets that are entering the bridge from the
  #  physical network interface peth0.    
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

These work great for preventing a user from using an unauthorized IPv4 address, but the rule for incoming traffic from peth0 is not sufficiently explicit, in my opinion. It allows users to see the network traffic of other users on the same dom0. By tightening up the rules a bit, we can prevent this.

The proposed rules are as follows, for the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD

For each valid IPv4 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv4
  #  address for that domU.
  iptables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 216.146.46.43 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv4 address for that domU.  
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 216.146.46.43 -j ACCEPT

Proposed Solution for IPv6 Anti-spoofing Protection for Xen

Quite simply, we use ip6tables with the exact same rule structure as shown above for IPv4. For the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  ip6tables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  ip6tables -F FORWARD

For each valid IPv6 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv6
  #  address for that domU.
  ip6tables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 2607:f590:0:ffff::60 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv6 address for that domU.  
  ip6tables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 2607:f590:0:ffff::60 -j ACCEPT

Simple, eh?

Modifications to Xen Network Scripts

So, the above outlined the original IPv4 anti-spoofing approach, and used that as a guideline to IPv6 anti-spoofing, as well as protecting against ARP poisoning attacks. Now, here are the changes to make to actually get this new behavior.

It’s worth noting that these changes were done in a manner to minimize impact to the existing networking scripts (i.e., no refactoring was done). This was done in case these changes never made it out to the community, we still need them in place to effectively use Xen for VPS hosting, and we would have to merge in community changes alongside our changes with each new community release. Thus, there’s a lot of duplicated code. Should these changes make it into Xen, I would recommend refactoring to remove this duplication!

It’s worth noting also that a method was needed for storing the IPv6 address for a domU. This was done by adding the address to the www.standingonthebrink.com.cfg file under /etc/xen/ in the following format:

  #ipv6=2607:F590:0000:FFFF:0000:0000:0000:0060

The modifications to the Xen networking scripts include the ability to read this into the xenstore for usage.

Without further ado, the network script diffs, which were performed against xen-unstable on October 8, 2008:

• vif-common.sh.diff
• vif-bridge.diff
• network-bridge.diff

I would also like to acknowledge the DynDNS team for helping to test and put together these modifications, and in particular Pierre Beaumier and Matthew Horsfall from Dynamic Network Services Inc., who discovered the vulnerabilities and greatly helped in putting a solution together.

Some of their feedback included:

• Audio required post processing, since background noise interfered with the spoken voice.
• Video was too long, and less than 3 minutes in length would be best.
• Transitions from one screen to the next were abrupt and obtrusive.
• The last slide showing the URL for more info disappeared too quickly.
• Wear a plain-colored dress shirt instead of a striped dress shirt, which will help video compression.

With their feedback in mind, I set about creating demo videos showing a nifty feature of the Spring Server platform: If you lose network connectivity to your virtual private server, you can log in to springconsole.com with any SSH client, and get an out of band console connection to your server to fix the network problem. Essentially serial console access to a VPS with an SSH client.

Originally, I prepared a single demo video, but after review with some in-house folks, came to the conclusion that this would not be a “one size fits all” situation. Some people knew what serial console was and how it could be used, and just wanted to learn how to use it here. Others would have no idea what I was talking about.

Accordingly, two videos were put together. The first is a “I’ll show you on the whiteboard” informal introduction to what serial console is and what it’s used for. The second is “I’ll show you on my laptop” informal demo on actually using it.

Whiteboard Intro: What is Spring Console?

VPS Serial Console Demo

We’re going to put these in a DynDNS-branded Flash player and on the DynDNS content delivery network before they go live on our site, but wanted to get some initial feedback from folks here on the blog (even though they’re just hosted with veoh.com above).

-Cory

The event starts off with a bit of a networking mixer in the bar area of the Royal Sonesta in Cambridge, and then into the ballroom for a set of “main dish” and “side presentations.” Main dishers get about 10 minutes to demo their latest and greatest creations (PowerPoint seems to be strictly forbidden… if you don’t have something to demo that makes people go “Oooh!” and “Aaah!”, don’t present). Side dish folks get a minute-long elevator pitch, directing people to a couple of tables in the back for demos after the presentations are over. After that, it’s basically just a bunch of techies and investors hanging out for a couple of hours over a couple of drinks… simply awesome.

There must have been at least 500 folks in attendance… far larger than any other non-conference event I’ve been to in the area, but it didn’t FEEL like a large, impersonal event. It was very open and social.

Probably the biggest shock for me was that I never once ran across a service provider pimping their own services. I’m sure they were there, but there were so many other folks that I never ran across one. Granted, the service providers are often the folks that end up sponsoring these types of events, and they keep the gears moving, but after you’ve talked to your nineteenth or twentieth IP attorney, it gets a little tiresome. If I was looking for a new patent attorney, that would be one thing, but I’ve been very happy with Brett.

I once knew a guy who said he would meet a couple of attorneys and accountants at the networking events, and then introduce them to each other with the line “I think you guys might be a good match!”, and then he would wait and see how long the “dueling salesman” scenario would play out before they figured it out! I doubt it’s a true story, but it certainly would be interesting to watch. Anywho, I digress…

The real thing that tipped me off to attending was that Artur Janc was one of the presenters. Artur and I both went to WPI, and we crossed paths about four times at various events without ever actually “meeting”. It was great to spot him (Artur’s pretty easy to spot in the crowd… he has the most bad-ass hair you’ll ever find on an entrepreneur), and catch up a bit. His new project is absolutely awesome; it’s called Lingro, and it’s “the coolest dictionary known to hombre!“.

It’s a service that let’s you visit a web page in any language, and all of the words become “clickable” to translate and/or lookup their definitions in a nice, unobtrusive panel on the page. The service is free for general usage, but if you’re someone who is actively trying to learn a new language, they’ll be offering an upgraded version of the service that will make life easier for you. Watch out Rosetta!

I thoroughly enjoyed all of the main and side dish presenters… and without further delay, here are my six-word summaries for each dish:

• Good2Gether - non-profits meet for-profits for social profit.
• PicMe Photo Sharing - flying photo stacks sharable with friends.
• Jack Cards - e-cards suck; easy hand-written cards rule.
• Traackr - measure your pimp status online. word.
  • (PS - I miss the days when people used vowels… I’m convinced folks are just trying to get a leg-up in Wheel of Fortune down the line)
• Moborazzi - visual mobile twitter… visual mobile twitter.
• Yamli - search in Arabic using English letters.
• Entrecard - blogger networking meets targeted sidebar ads.
• Flimp - these folks will make a fortune.
• StylePath - hopefully my wife never finds this!
  • Okay that one’s a joke, but it was my first thought! The real six-word summary is: visual search when words aren’t enough.

Very much looking forward to the next event, currently scheduled for July. I hope to see you there!

So, if you see this in your mailbox:

Turn to page 96, and we’re in the Five Steps to a Greener SMB article.

Click here for the full article text on PC Magazine’s web site.

Many thanks to Oliver Rist from PC Magazine for the mention!

http://code.google.com/p/rest-api-code-gen/