Here’s an example shot with a Logitech QuickCam Pro 9000 webcam using the built-in microphone, standing roughly six feet away from the camera:

It’s OK, but not the greatest. Here’s the same Logitech webcam, but this time sitting in front of the camera, roughly two feet away, with the direction of speaking always toward the camera (and thus, the built-in microphone):

MUCH better! But we still want to be able to do recordings at the whiteboard, walking around, etc.

Here’s an example using the mic built-in with one of our camcorders (an Insignia HD Flash camcorder, which provides amazing video quality for the money):

What’s the ultimate solution? Well, a high-quality lavalier microphone, of course! And we have one… it’s the Audio-Technica ATR-35S.

So, why then, on our latest (and also most popular) videos, do we still have not the greatest sound in the world?

Insignia camcorder with built-in mic, but noisy environment in the middle of office, with lots of moving around by the speaker (relative to the mic):

Same Insignia camcorder and built-in mic, but quieter area with more control of what’s going on:

Same Insignia camcorder and built-in mic, but a better job of noise reduction in post-production:

So, I ask again, if we have the lavalier microphone, why haven’t we been using it? The answer is … we don’t have any place convenient to plug in the lavalier microphone when using our camcorders.

The camcorders themselves unfortunately do not have external microphone inputs.

We’ve tried using the microphone inputs for some laptops and desktops we had handy (and synching the audio and video in post-production), but the poor quality sound cards built-in clearly had some grounding issues, because a hefty hum could be heard (and often overheard) in the background. Plus, this often meant a long cable from the lavalier microphone to the computer, which was likely to be tripped over. No good.

But as of today, that changes. I just picked up a Sony ICD-UX70 digital voice recorder for the video production team, with built-in stereo microphones, an external microphone jack, and MP3 encoding of audio files. Now, we can throw this in the shirt pocket for whoever is speaking on video (when we’re in a rush), or for the best quality, just have the actress or actor hold it/put it in a pocket and wire up the lavalier microphone to walk around freely.

We’re shooting some video this week… I’ll let you know how it works out (hopefully you’ll be able to hear the difference for yourself!).

Here’s to good quality audio.

What’s not so easy is trying to secure and lock down the individual virtual servers so they cannot interfere with one another’s network connection. The efforts spent on IPv6 also exposed some issues for IPv4 and ARP that could arise when used in a VPS environment.

Note: We’re going to get down and dirty into the technical nitty gritty, and it’s beyond the scope of this post to bring all of my readers up to speed. The faint of heart may wish to turn back now.

Problem Cases for IPv6

Two main network spoofing cases come to mind that need to be protected against for running IPv6 on Xen in a virtual private server environment:

1) A user changing the IPv6 address of their network interface to an IPv6 address we haven’t authorized them to use, or adding/aliasing additional IPv6 addresses to their network interface that they did not pay for.

2) A user changing/adding/aliasing an IPv6 address that’s in use by another customer, which can wreak all sorts of routing havoc, interfering with the ability for our customers to reliably communicate with others on the Internet.

Perhaps Luke Crawford put it best with his response to my question, “Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I’d be happy to submit the changes and a guide to making it work and work well.” on the Xen developers email list:

Guides would be awesome. Also, if you made anti-spoof work for IPv6, that would also be awesome. I’ve had IPv6 going for a while with no problems (though the xen antispoof rules only work with IPv4), but I didn’t even need to enable IPv6 in the Dom0; it’s a layer 2 bridge, so it just works. In fact, stateless autoconfiguration even worked. I asked for an IPv6 allocation from my provider and my customers noticed it was working before I got around to doing any setup at all.

Problem Cases for IPv4 and ARP

When we initially went down the IPv6 road, we started paying closer attention to what was already implemented for IPv4 protection to get an idea for where to begin. In doing so, we found a couple of scenarios where the existing anti-spoof protection leaves Xen a bit too exposed for using as a VPS platform:

1) If a user aliases an IPv4 gateway IP address for the same subnet that user is on, all network connectivity can go down for that subnet. As an example, if your IP address is 1.2.3.5/24 and your gateway is at 1.2.3.1, and you run “ifconfig 1.2.3.1″ to alias the gateway IP address on your default network interface, you will now respond to ARP requests trying to find the gateway with your MAC address… meaning no other users on that subnet can reliably talk to the gateway… meaning their network connectivity is pretty much useless.

2) If a user aliases an IPv4 address that is already in use by another user, all network connectivity can go down for that other user, for the same reasons it goes down when aliasing the gateway IP address (i.e., you’ll be responding to ARP requests for their IP address).

3) Users can deliberately perform ARP poisoning attacks, by sending arbitrary ARP replies on the subnet, which can interfere with authorized communication for the same reasons as 1 and 2 above.

4) Users can see the network traffic for other domUs on the same dom0 by putting their interfaces into promiscuous mode (i.e., packet sniffing). 

My intent here is not to knock the Xen development community; they’ve put together a fantastic platform, and no one expects our business requirements for using Xen to host virtual private servers to be inline with their development requirements. I’m not even certain that we’ve squashed the issues entirely, but we have at least squashed the pressing issues for IPv6, IPv4, and ARP outlined above.

Now that we have it working in the field, I’d like to share the handful of changes with the community so they can benefit, and perhaps suggest better/more complete/more succinct ways of doing it. The solution we implemented was modeled after the current IPv4 anti-spoof protection.

I will start off with an overview of the current IPv4 anti-spoofing approach used by Xen, followed by the specific commands used to alter the behavior for what we needed for reliable and secure communication, concluded by the actual changes we made to the Xen networking scripts to enable this behavior modification.

Current IPv4 Anti-spoofing Protection for Xen

The current IPv4 anti-spoof protection scheme relies on iptables, in particular the FORWARD chain that each packet destined for a virtual server hits on its way in or out. Rules get added each time a virtual network interface comes up, and removed when the interface does down. 

For the sake of the remaining discussion, where I’ll be walking through some command examples, we’ll assume:

• peth0 is the physical network interface that the bridge connects to
• vif1.0 will be a virtual network interface connected to a domU, and the domU sees that device as eth0
• The domU is “allowed” to use IPv4 address 216.146.46.43/24, with a gateway IP of 216.146.46.1 for IPv4. 
• The domU is “allowed” to use IPv6 address 2607:f590:0:ffff::60/64, with a gateway IP of 2607:f590:0:ffff::1 for IPv6.
• The MAC address of the virtual network interfaced vif1.0 is 00:16:3E:38:B4:AC.
• The domU hostname is www.standingonthebrink.com.

Yes, you guessed it… all of the settings above are the real-world settings for the Xen VPS that is hosting this blog (with the exception of the virtual network interface name of vif1.0, which will change over time as the VPS is powered off and on).

In a nutshell, when you enable anti-spoofing as a parameter in the network-bridge script, the following iptables commands are issued when the bridge comes online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD  

  # Accept packets that are entering the bridge from the physical
  #   network interface peth0.
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

Now, when each domU virtual network interface comes online, the following iptables command is run for each IP address that the domU is allowed to use, as specified in the www.standingonthebrink.com.cfg file found in /etc/xen/:

  # Accept packets coming into the bridge from the domU if its
  #   source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

When the domU’s virtual network interface goes offline, the same command is run except the -A is changed to -D to delete the rule.

Into the Bridge? Out of the Bridge? What’s the Difference?

It’s important to point out the difference between “into” and “out of” the bridge. For the purpose of thinking about rules governing the packets that are allowed to traverse the bridge, it can be helpful to think of the bridge as a little switch connecting the physical network interface of the server to the virtual network interface of the dom0 as well as all of the virtual network interfaces for each of the domUs. The rules we write in the FORWARD chain will tell the bridge what packets to allow to enter and/or leave the bridge.

All of the rules we’ll write are from the perspective of the network bridge. Perhaps a couple examples will help convey the concept:

• A packet coming from the Internet destined for the first domU -> Comes IN peth0 and goes OUT vif1.0.
• A packet coming from the first domU destined for the second domU -> Comes IN vif1.0 and goes OUT vif2.0.
• A packet sent from the first domU to somewhere on the Internet -> Comes IN vif1.0 and goes OUT peth0.

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

Proposed Solution for ARP Poisoning Protection for Xen

Quite simply, we use arptables in a similar manner to using iptables. It can be a little tricky getting the rules right for what valid ARP traffic should be allowed, but with quite a bit of trial-and-error, we think we have it.

When the bridge comes online, we do:

  # Default policy for packets in the FORWARD chain is DROP.
  arptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  arptables -F FORWARD

When each virtual network interface comes online, we do:

  # Accept ARP requests coming from the domU into the bridge.
  arptables -A FORWARD --opcode Request --in-interface vif1.0 -j ACCEPT

  # Accept ARP requests coming out of the bridge into the domU.
  arptables -A FORWARD --opcode Request --out-interface vif1.0 -j ACCEPT

  # Accept ARP replies coming out of the bridge from the physical
  #  network into the domU.
  arptables -A FORWARD --opcode Reply --out-interface vif1.0 \
    --in-interface peth0 -j ACCEPT

In addition to the above rules that get added when the virtual network interface comes online, we do the following for each valid IPv4 address:

  # Accept ARP replies coming from the domU into the bridge if they
  #  provide a valid and authorized IP address to MAC address pair.
  arptables -A FORWARD --opcode Reply --in-interface vif1.0 \
    --source-ip 216.146.46.43 --source-mac 00:16:3E:38:B4:AC -j ACCEPT

It’s this last rule that prevents the ARP poisoning attacks and the havoc wreaking by aliasing gateway IP addresses. If invalid ARP replies are denied, as far as I can tell, no harm can be done. When the virtual network interface goes offline, the same commands are run with -D instead of -A to delete the rules.

Proposed Solution for Preventing IPv4 Packet Sniffing

As mentioned earlier, the current IPv4 rules that prevent anti-spoofing are:

  # Accept packets that are entering the bridge from the
  #  physical network interface peth0.    
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

These work great for preventing a user from using an unauthorized IPv4 address, but the rule for incoming traffic from peth0 is not sufficiently explicit, in my opinion. It allows users to see the network traffic of other users on the same dom0. By tightening up the rules a bit, we can prevent this.

The proposed rules are as follows, for the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD

For each valid IPv4 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv4
  #  address for that domU.
  iptables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 216.146.46.43 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv4 address for that domU.  
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 216.146.46.43 -j ACCEPT

Proposed Solution for IPv6 Anti-spoofing Protection for Xen

Quite simply, we use ip6tables with the exact same rule structure as shown above for IPv4. For the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  ip6tables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  ip6tables -F FORWARD

For each valid IPv6 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv6
  #  address for that domU.
  ip6tables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 2607:f590:0:ffff::60 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv6 address for that domU.  
  ip6tables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 2607:f590:0:ffff::60 -j ACCEPT

Simple, eh?

Modifications to Xen Network Scripts

So, the above outlined the original IPv4 anti-spoofing approach, and used that as a guideline to IPv6 anti-spoofing, as well as protecting against ARP poisoning attacks. Now, here are the changes to make to actually get this new behavior.

It’s worth noting that these changes were done in a manner to minimize impact to the existing networking scripts (i.e., no refactoring was done). This was done in case these changes never made it out to the community, we still need them in place to effectively use Xen for VPS hosting, and we would have to merge in community changes alongside our changes with each new community release. Thus, there’s a lot of duplicated code. Should these changes make it into Xen, I would recommend refactoring to remove this duplication!

It’s worth noting also that a method was needed for storing the IPv6 address for a domU. This was done by adding the address to the www.standingonthebrink.com.cfg file under /etc/xen/ in the following format:

  #ipv6=2607:F590:0000:FFFF:0000:0000:0000:0060

The modifications to the Xen networking scripts include the ability to read this into the xenstore for usage.

Without further ado, the network script diffs, which were performed against xen-unstable on October 8, 2008:

• vif-common.sh.diff
• vif-bridge.diff
• network-bridge.diff

I would also like to acknowledge the DynDNS team for helping to test and put together these modifications, and in particular Pierre Beaumier and Matthew Horsfall from Dynamic Network Services Inc., who discovered the vulnerabilities and greatly helped in putting a solution together.

Some of their feedback included:

• Audio required post processing, since background noise interfered with the spoken voice.
• Video was too long, and less than 3 minutes in length would be best.
• Transitions from one screen to the next were abrupt and obtrusive.
• The last slide showing the URL for more info disappeared too quickly.
• Wear a plain-colored dress shirt instead of a striped dress shirt, which will help video compression.

With their feedback in mind, I set about creating demo videos showing a nifty feature of the Spring Server platform: If you lose network connectivity to your virtual private server, you can log in to springconsole.com with any SSH client, and get an out of band console connection to your server to fix the network problem. Essentially serial console access to a VPS with an SSH client.

Originally, I prepared a single demo video, but after review with some in-house folks, came to the conclusion that this would not be a “one size fits all” situation. Some people knew what serial console was and how it could be used, and just wanted to learn how to use it here. Others would have no idea what I was talking about.

Accordingly, two videos were put together. The first is a “I’ll show you on the whiteboard” informal introduction to what serial console is and what it’s used for. The second is “I’ll show you on my laptop” informal demo on actually using it.

Whiteboard Intro: What is Spring Console?

VPS Serial Console Demo

We’re going to put these in a DynDNS-branded Flash player and on the DynDNS content delivery network before they go live on our site, but wanted to get some initial feedback from folks here on the blog (even though they’re just hosted with veoh.com above).

-Cory

The event starts off with a bit of a networking mixer in the bar area of the Royal Sonesta in Cambridge, and then into the ballroom for a set of “main dish” and “side presentations.” Main dishers get about 10 minutes to demo their latest and greatest creations (PowerPoint seems to be strictly forbidden… if you don’t have something to demo that makes people go “Oooh!” and “Aaah!”, don’t present). Side dish folks get a minute-long elevator pitch, directing people to a couple of tables in the back for demos after the presentations are over. After that, it’s basically just a bunch of techies and investors hanging out for a couple of hours over a couple of drinks… simply awesome.

There must have been at least 500 folks in attendance… far larger than any other non-conference event I’ve been to in the area, but it didn’t FEEL like a large, impersonal event. It was very open and social.

Probably the biggest shock for me was that I never once ran across a service provider pimping their own services. I’m sure they were there, but there were so many other folks that I never ran across one. Granted, the service providers are often the folks that end up sponsoring these types of events, and they keep the gears moving, but after you’ve talked to your nineteenth or twentieth IP attorney, it gets a little tiresome. If I was looking for a new patent attorney, that would be one thing, but I’ve been very happy with Brett.

I once knew a guy who said he would meet a couple of attorneys and accountants at the networking events, and then introduce them to each other with the line “I think you guys might be a good match!”, and then he would wait and see how long the “dueling salesman” scenario would play out before they figured it out! I doubt it’s a true story, but it certainly would be interesting to watch. Anywho, I digress…

The real thing that tipped me off to attending was that Artur Janc was one of the presenters. Artur and I both went to WPI, and we crossed paths about four times at various events without ever actually “meeting”. It was great to spot him (Artur’s pretty easy to spot in the crowd… he has the most bad-ass hair you’ll ever find on an entrepreneur), and catch up a bit. His new project is absolutely awesome; it’s called Lingro, and it’s “the coolest dictionary known to hombre!“.

It’s a service that let’s you visit a web page in any language, and all of the words become “clickable” to translate and/or lookup their definitions in a nice, unobtrusive panel on the page. The service is free for general usage, but if you’re someone who is actively trying to learn a new language, they’ll be offering an upgraded version of the service that will make life easier for you. Watch out Rosetta!

I thoroughly enjoyed all of the main and side dish presenters… and without further delay, here are my six-word summaries for each dish:

• Good2Gether - non-profits meet for-profits for social profit.
• PicMe Photo Sharing - flying photo stacks sharable with friends.
• Jack Cards - e-cards suck; easy hand-written cards rule.
• Traackr - measure your pimp status online. word.
  • (PS - I miss the days when people used vowels… I’m convinced folks are just trying to get a leg-up in Wheel of Fortune down the line)
• Moborazzi - visual mobile twitter… visual mobile twitter.
• Yamli - search in Arabic using English letters.
• Entrecard - blogger networking meets targeted sidebar ads.
• Flimp - these folks will make a fortune.
• StylePath - hopefully my wife never finds this!
  • Okay that one’s a joke, but it was my first thought! The real six-word summary is: visual search when words aren’t enough.

Very much looking forward to the next event, currently scheduled for July. I hope to see you there!

So, if you see this in your mailbox:

Turn to page 96, and we’re in the Five Steps to a Greener SMB article.

Click here for the full article text on PC Magazine’s web site.

Many thanks to Oliver Rist from PC Magazine for the mention!

http://code.google.com/p/rest-api-code-gen/

… WS-* is *not* preferable to REST. WSDL creates a tight coupling between the client and service which no amount of XML versioning can hope to alleviate. Code generation is brittle, can create unnecessary data definition dependencies, and inhibits extensibility. The industry has by and large learned this well over the past few years, which is why I mentioned 2004 in my blog about this. WS-* is practically dead.

WSDL and code generation try to give distributed systems the appearance of being local, ala RPC, only for the purpose of making the developer’s life easier, but in doing they create big problems. That’s why I mentioned 1994 and pointed to the “Note on Distributed Computing” paper, which covers this quite well.

You might want to read my latest IEEE Internet Computing columns on this topic, as those who don’t know history are doomed to repeat it:

Assuming you apply it correctly, REST is far, far preferable to what you’ve described…

Here are my thoughts…

Hi Steve,

Thanks for taking the time to write back. I appreciate it. I whole-heartedly agree with each of your points:

WSDL creates tight coupling, and this is bad.

Code generation is brittle, can create unnecessary data definition dependencies, and inhibits extensibility. All bad.

I know you’ve touched upon how REST is intrinsically more scalable than WS-*. I agree.

I believe my key failing here is that I failed to adequately explain the problem I am trying to solve.

I make embedded systems. More specifically, I make a device that can remotely reboot or power control a server in the same fashion as pressing the reset or power switch on the front of the server’s chassis. The device, named the WebReboot, can also measure the temperature inside a server, and tell you if the server is powered on or off. It’s core application is the remote recovery of crashed servers.

The interface for the WebReboot is tightly coupled to the hardware. We use a relay to control the power switch or reset switch of a server. We use a thermistor to measure temperature. The hardware has remained relatively unchanged for quite a long time, largely because it is simple, reliable, and cost-effective. Since the hardware has remained relatively unchanged, the interface has remained relatively unchanged. The extent of changes are limited to adding complimentary functionality that does not affect existing functionality (for instance, if I added a humidity sensor). In this scenario, I don’t think the tight coupling forced upon you by WSDL really makes you any worse off.

Scalability is largely not a concern for this application. In practice, it is not necessary to have many folks or systems trying to reboot a crashed server or retrieve a server’s temperature simultaneously.

Since my functionality is limited to what’s implemented in relatively simple physical hardware (e.g., sensors and relays), I’ve found limited application for complicated data structures and dependencies in controlling this hardware. I’ve also found limited application for extensibility. I’ve come up with many hypothetical ways for their application, but I haven’t found a need to use them. For these reasons, I haven’t in practice found the generated code for Java, Python, PHP or .NET to be brittle.

The biggest challenge we’ve had as a small company (< 10 people) competing in a large market (IT hardware) is competing with organizations that have significantly more resources than we do (Servprise was bootstrapped, and several of my competitors are public companies). One thing I noticed was that my competitors were largely offering stove-pipe solutions that were designed only for human operators to manually use. We decided from the get-go to make ease of integration a key selling point for the solution. This selling point has been the largest contributor to our growth.

There are two main ways our customers integrate our solution:

1) The functionality is made available in a centralized system for ease of access and control (e.g., a control panel at a hosting company).

2) Monitoring solutions, such as Nagios, that can tell you whether or not a server has crashed or a server is overheating, take automated recovery steps to either correct the problem or prevent escalation of the problem by sending commands to the WebReboot.

Our customers obtain financial benefits by making the relatively simple hardware functionality of our solution accessible in larger systems such as control panels and monitoring solutions. The problem for me was enabling this integration faster and cheaper than the competition.

Our competition, for the most part, chose the route of either developing and maintaining client libraries to manage communication to and from their embedded systems in Java, PHP, Python, etc., to ease integration for their customers, or left the customer to develop solutions on their own. I did not and still do not have the resources to develop and maintain client libraries for my customers. It’s code generation that has enabled me to compete on this level.

For my application, as well as numerous other embedded applications where the interface is tightly coupled to physical hardware, where the means of control on the physical world (e.g., physical relays and sensors, embedded systems) remain largely unchanged over time (it’s very cost-prohibitive to change), where scalability is not a major concern, and where capability and ease of integration are key selling points, WSDL and SOAP may be just what the doctor ordered, even if the only reason is code generation.

Clearly, those that put forth WS-* had much grander ambitions. I firmly believe REST is better suited for obtaining those ambitions. But I also believe this approach has a real future in making it easier and more cost effective for people to bridge the gap between software and the simple embedded hardware that enables control of the physical world.

I still remember developing web applications the old-fashioned way… you know, some HTML forms posting some data to the server… doing some server-side magic, and then pushing out a hot ‘n fresh new page of HTML, ripe for the user’s viewing pleasure! Here’s a professional artist’s depiction of what I’m talking about:

The user enters their name on a form (”Bob”). Clicks submit. The data is POSTed to the server as a name=value pair (”Name=Bob”). Your choice from a number of server-side technologies then generates a new page with a response (”Hi Bob!”) and sends it to the browser.

It was easy to understand. Quick to implement. Enjoyable to tinker with. It worked, and I made money doing it. At the time, I didn’t have too much to complain about. You make a form. You post the data. You process the data. You return HTML. Rinse and repeat.

Then came the wave of AJAX. Ho-ly crap.

I remember the first time I saw an AJAX web app. It was Gmail. It was the freakin’ auto-complete fields in Gmail. I couldn’t believe it. How did it work that fast? It’s faster than… my desktop email client! Un-freakin-believable! What sorcery was at work?

Disgustingly simple, and disgustingly effective sorcery. Just a little snippet of client-side code, in the background and without interrupting the user, sent little tidbits of information to the server, and received and processed responses in return. If all you needed were these little tidbits, you didn’t even have to generate a new page on the server. Just let the client-side code manipulate the page as needed.

Brilliant. This had HUGE implications for me. I come from an embedded hardware world. A world where you need to squeeze the maximum performance possible out of your hardware to deliver the greatest feature set possible to the customer, without spending too much money on super-fast processors. When it came to web-enabled embedded hardware, this opened up many, many new doors.

I now didn’t need the embedded processors in our remote reboot hardware to spend all of their time dynamically building HTML. Instead of dynamically building HTML files line-by-line in C, I could serve static HTML and JavaScript files. The static HTML would contain the forms for the user to interact with the system. The JavaScript would just submit the data from the forms, receive the response, and display the response for the user on the same page. I went from having to dynamically process tens of kilobytes to tens of bytes. That’s three orders of magnitude less data to process.

As a result, the system was much “snappier” (as end users often describe), and I had extra processing power to burn for new features. But it didn’t stop there. See, the operating system we eventually settled on for the WebReboot Enterprise product line was the Evolution OS from Lantronix. They actually had an AJAX interface right out-of-the-box. Worked just like described above. When you were using it, it didn’t even feel like there was an embedded processor on the serving end. Just what the doctor ordered. But there was more to come…

About the same time, a bit of buzz was being generated on web services, promising brand new worlds of interoperability, making it easier to get systems to talk to each other. If I can make it easier for people to write software against my remote reboot hardware, that would surely give me a leg up in the marketplace. It wasn’t the first technology on the block to make such promises, but hey, it’s worth a look, right?

There were two major camps of web service folks at the time. The REST folks, and the SOAP folks. Without getting too technical, the main differences, as I understood them, were that REST promised a very loose, informal way of exchanging data that was very quick and easy to get up and running, while SOAP promised more rigid, explicit, and formal data exchange that took a little more effort to get started with.

Sticking with my “What’s Your Name?” web application example, someone writing code against a REST web service might access a URI like the following:

http://whatisyourname.com/set-name/Bob

Pretty simple. You just invoked this REST web service, telling the service your name is Bob. In response, the web service could return:

Hi Bob!

That’s it… I mean it could be a fully generated HTML page as a response, it could include other stuff, but really, that shows you what I mean by quick and easy. You invoke the web service by including the parameters in the URI, and the response you read back from the request is the result. You can even manually invoke the web service in your web browser just by plopping that URI in the address bar and seeing the response displayed in the browser… in fact, I find that to be one of the major appeals of this approach.

The SOAP approach, on the other hand, requires a little more setting up… see, you’re actually going to send a request to the web service formatted in XML, and then get a response back in XML.

You might send the following to the web service at the URI http://whatisyourname.com/services/NameService:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetName xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Name>Bob</q0:Name>
</q0:SetName>
</soapenv:Body>
</soapenv:Envelope>

The response would be:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetNameResponse xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Response>Hi Bob!</q0:Response>
</q0:SetNameResponse>
</soapenv:Body>
</soapenv:Envelope>

Wow. What a difference, huh? Looks like REST is the clear winner. I mean, with REST, I just access the URI containing the parameter, and get the response back with the data. With SOAP, I have to build an XML document with VERY strict formatting requirements, submit it, get an XML document back as the response, parse it, and I have my data.

Well, not for me, and not for what I wanted to accomplish. You see, there’s a lot more that goes into SOAP web services than REST web services, but you get plenty back. I chose SOAP, and never looked back.

For my SOAP web service, I documented the format of the request and response messages using the Web Service Description Language (WSDL… just another XML document). Basically, I specified that the request XML should contain a string with the name (”Bob”), and the response XML should contain a string with, well, the response (”Hi Bob!”). The WSDL in effect documents the API. It tells you what commands and data you can invoke on the service, and what you’ll receive in response.

Now here’s the absolute coolest part (at least in my opinion). Armed with the WSDL document, you can use freely available, open source tools to automatically generate stub code to send requests to and receive responses from the SOAP web service in just about any modern programming language of your choosing.

For both the client and the server.

The stub code generates and parses all of the XML. As a developer working in the language of your choice, you are completely abstracted from the creation of the code for the sending and receiving of data on the wire (I had a gross omission here… thanks owed to Steve Vinoski for pointing it out).

If I were to write Java client code against this web service, I would use the open source WSDL2Java utility, and as output I would get super easy to use setName(String name) and getResponse() Java methods to interact with the service.

I could just as easily write code against the service in Python, .NET, PHP, Perl, C/C++, JavaScript, and many more. By spending the time to formally document the API in WSDL, you can now automatically generate stub code to communicate with the web service for your language of choice. Well worth the extra effort.

To build a web application on top of a XML SOAP web service, the JavaScript client creates the XML SOAP requests and parses the XML SOAP responses, just like the AJAX described above.

For our remote reboot hardware, I decided to not use the built-in AJAX support of the Evolution OS (which relied on sending and receiving data by using POST and reading the response), and instead built an XML SOAP web service in C in the Evolution OS, and a JavaScript web service client for the web browser interface. It took a while to get up and running, but has paid dividends.

What are some examples of how we’ve benefited?

• We have complete freedom to choose the right programming tool for each job, without having to worry about maintaining multiple API libraries. We don’t need to distribute JARs and worry about compiling in Java 1.4, 1.5, or 1.6. We don’t need to distribute PHP that was developed against version 4 but is buggy against 5. We don’t impose “DLL Hell” on users. We just maintain the web service as the API, put out a guide showing people how to generate the stub code in the language of their choice, and everything else just works.
• The web browser interface, once fully loaded, is extremely responsive, considering it is an embedded processor based, SSL-enabled system.
• All functionality of the WebReboot Enterprise is regression tested using unit tests written in Java using TestNG against the XML SOAP web service. That’s right… test-driven development of an embedded system using Java, even though the embedded system was written in C. Every change to the C code that’s committed to Subversion gets checked out by Bamboo, our continuous integration server, built and deployed on actual WebReboot Enterprise hardware, and ran against a suite of checkin, verification, and functional regression tests (depending on what changed). All automatic. Cool beans.
• We are abstracted from dealing with the data on the wire. I don’t have to care about sending 0×0A hex for a reboot, and 0×0B hex for a power on… I just call my nice clean doReboot() and doPowerOn() methods in a Java client I’m integrating with the web service, and it just works.
• After implementing the server in C and the client in JavaScript, I wanted to create a reference implementation for the server in a language I can develop much faster than C in. Accordingly, I took the WSDL file, ran WSDL2Java on it to generate the server stub code, and filled in the blanks for the various requests and responses in Java. The net result is I was able to create a fully functional “mock” WebReboot Enterprise in a Java servlet container. We now distribute this for folks to get started with development without needing a full, physical hardware WebReboot Enterprise in front of them, and we also use it for our live public demonstration, enabling each user that connects to get a unique “session,” such that any configuration changes they make are confined to their session and don’t affect other users accessing the demo. Super cool beans.
• In the field, we’ve whipped together Python applications in a matter of minutes against the XML SOAP web service that automated several of the mundane processes for deploying large installations… such as setting Syslog servers, DNS servers, etc.
• XML SOAP messages are versioned by namespace. Each request contains a namespace, and each response contains a namespace. The namespaces (if you designed your service well) correspond to which version of the WSDL document (in other words, the API) they were designed for. When our web service code released in December, 2007 receives a message from a client that was written against an older version of the web service, it automatically determines this, and automatically degrades functionality. Upgrades won’t break existing systems. The more I can prevent things from breaking, the happier I am (since I don’t have to hack fixes together to get stuff working in the field), and the happier my customers are (since they don’t need to rewrite their code when they upgrade firmware… if they want the latest features made available in the latest version of the web service they’ll need to make some code changes, but they won’t break anything just by upgrading).
• In the factory, we’ve developed a test system written in Python against the XML SOAP web service to do a full hardware inspection to ensure each unit off the line meets quality standards. The test system is even integrated with the web service built-in to JIRA, our issue tracking system (those Atlassian guys are so smart), so quality issues are automatically tracked.
• We’ve developed plugins in Python for Nagios that use the data available from the XML SOAP web service on the WebReboot Enterprise (like server temperature, server power status, and server connection status) to make decisions about corrective actions that can be taken to automate the datacenter (turn off overheating servers, turn on servers that should never be turned off, reboot crashed servers, etc.).
• Our customers have integrated the WebReboot Enterprise functionality into their client control panels using everything under the sun… from ASP.NET to PHP to Java servlets. We’re talking huge datacenters filled with thousands and thousands of servers, and the folks using those servers now have nice little reboot, power control, power monitoring, and temperature monitoring displays in the control panels they were already using to monitor and control those servers. When customers are happy, those are the coolest beans of all.
  

For these reasons, and several more I plan on going into more depth on, web services have enabled me to kick more butt with less effort in the embedded hardware world. In Part 2, I’ll show you how I’ve used this approach to kick more butt with less effort for full-blown, production web applications, and Part 3 will discuss the huge benefits this approach has for test-driven development of web applications. I’ll probably wrap up with some lame “TOP 10 WAYS WEB SERVICES RULEZ!!1!!!”, since that’s the only way to get on Digg anymore.

Stay tuned!