Not Just AstLinux Stuff

findpod - Find Your Own "Packets of Death"

2013-02-20T13:00:00.002-05:00

After my original post on “packets of death” I’ve spent the last couple of weeks receiving reports from users all over the world ranging from:

- Able to reproduce on 82574L with your “packets of death”
- Not able to reproduce on 82574L or any other controller
- Not able to reproduce with your “packets of death” but experiencing identical, sporadic failures across a wide range of controllers

Of course the last category intrigued me the most. There seem to be an awful lot of people experiencing sporadic failures of their ethernet controllers but many of them (as I noted in another update) don’t have the time or tools to diagnose the issue further. In most cases the symptoms are identical to what I described in my original post:

- Ethernet controller loses link (or reports some other hardware error)
- Varying amounts of time since boot (hours, days, weeks)
- Can only be resolved by a reboot or in some cases a complete power cycle

Many of these users have been dealing with these failures in various ways but have been unable to find a root cause. I’ve created a tool to help them with their diagnosis. It’s called findpod and it’s been tested on various Debian-based Linux distributions. Findpod uses the excellent ifplugd daemon, the venerable tcpdump, and screen.

Once installed and started iflpugd will patiently wait to receive link status notifications from the Linux kernel. Once link is detected on the target interface it will start a tcpdump session running inside of screen. This tcpdump session will log all packets sent and received on that interface. Here’s the thing - many of these failures are reported after days or weeks of processed traffic - the tcpdump capture file could easily reach several gigabytes or more! Here’s where one key trick in findpod comes into play - by default findpod will only log the last 100MB sent or received on the target interface. As long as ifplugd doesn’t report any link failures tcpdump will keep writing to the same 100MB circular capture file.

What happens when the interface loses link? This is the second unique feature of findpod. When ifplugd reports a loss of link it will wait for 30 seconds before stopping the packet capture and moving the capture file to a meaningful (and known) name. If you think your ethernet controller failures could be related to the types of traffic you’re sending or receiving (as I discovered with my “packets of death”) findpod will help you narrow it down to (at most) 100MB of network traffic, even if the capture runs for weeks and your interface handles GBs of data!

Of course (more than likely) it’s even easier than that; if your link loss is being caused by a specific received packet it will be the last packet in the capture file provided by findpod and you’ll only have a 100MB capture file to work with. If your issue is anything like mine you should be able to isolate it down to a specific packet that you can feed to tcpreplay; reproducing your controller issue on demand.

Please tell me about your experiences with findpod. As always, comments and suggestions are welcome!

Packets of Death - UPDATE

2013-02-08T15:42:00.003-05:00

UPDATE - Intel has pointed me towards the successor to the 82574, which includes some of the features I suggest here. I think it's safe to say we're all looking forward to this chip hitting the streets!

The last 48 hours has been interesting, to say the least.

My original post gathered much more attention than I originally thought. I’ll always remember being on a conference call and having someone tell me “Hey, you’re on Slashdot”. Considering the subject matter I suppose I should have expected that.

As of today, here’s what I know:

Many of you have shared the results of your testing. The vast majority of tested Intel ethernet controllers do not appear to be affected by this issue.

Intel has responded with an expanded technical explanation of the issue. I also received a very pleasant and professional phone call from Douglas Boom (at Intel) to update me on their assessment of the situation and discuss any ongoing concerns or problems I may have. Thank you Doug and well done Intel! Note to other massive corporations that could be presented with issues like this: do what Intel did.

To summarize their response, Intel says that a misconfigured EEPROM caused this issue. The EEPROM is written by the motherboard manufacturer and not Intel. Intel says my motherboard manufacturer did not follow their published guidelines for this process. Based on what I’ve seen, how my issue was fixed, and what I’m learning from the crowdsourced testing process this seems like a perfectly plausible explanation. Once again, thanks Intel!

However, I still don’t believe this issue is completely isolated to this specific instance and one motherboard manufacturer. For one, I have received at least two confirmed reports from people who were able to reproduce this issue - my “packet of death” shutting down 82574L hardware from different motherboard manufacturers. This doesn’t surprise me at all.

One thing we’re reminded of in this situation is just how complex all of these digital systems have become. We’re a long way from configuring ethernet adapters with IRQ jumpers. Intel has designed an incredibly complex ethernet controller - the datasheet is 490 pages long! Of course I’m not faulting them for this - the features available in this controller (or many other controllers) dictate this level of complexity - management/IPMI features, WOL, various offloading mechanisms, interrupt queues, and more. This complexity doesn’t even scratch the surface of the various other systems involved in getting data across the internet and into your eyeballs!

Like any sufficiently advanced product all of these features are driven by a configuration mechanism. The Linux kernel module for the 82574L (e1000e) has various options that can be passed to modify the behavior of the adapter. Makes sense. If I passed some stupid or unknown parameter to this module I would expect it to return with some kind of error informing me of my mistake. I’m only human, mistakes are going to happen.

At a lower level Intel has exposed these EEPROM configuration parameters to control various aspects of the controller. As Intel says these EEPROM values are to be set by the motherboard manufacturer. Here’s where the problem lies - it’s certainly possible this could be done incorrectly. Motherboard manufacturers are human and they make mistakes too.

Unfortunately, as we’ve learned in this case, there isn’t quite the same level of feedback when EEPROM misconfigurations happen. In my previous example if I pass unknown parameters to a kernel module it’s going to come back and say “Hey - I don’t know what that is (dummy)” and exit with an error.

As I’ve shown in some cases (mine) if an EEPROM is misconfigured everything appears normal until some insanely specific packet is received. Then the controller shuts down, for some reason.

Does that behavior make sense to anyone?

I suggest the following:

1) Make future controllers have as much in-hardware sane behavior as possible when unknown conditions are encountered. Error checking, basically. Users can input data on a web form, that’s why there’s error checking. Everyone knows users do stupid things. Clearly some of the people programming Intel EEPROMs for motherboard OEMs do stupid things too. What is sane default behavior? EEPROM error encountered = adapter shutdown and error message. Give the user notification and provide some mechanism for EEPROM validation and management...

2) Put more EEPROM validation in operating system drivers. Intel maintains ethernet drivers for various platforms. Why aren’t these drivers doing more validation of the adapter EEPROM? If my EEPROM was so badly misconfigured, why couldn’t the e1000e module have discovered that and notified me?

3) Produce and support an external tool for EEPROM testing, programming, and updating. In the course of working with Intel last fall I was provided a version of this tool for my testing so I know it exists. While I can understand why you don’t want random users messing with their EEPROM (and causing potential support nightmares) it seems the benefits would clearly outweigh any potential problems (of which there are already plenty).

The reality is Intel has no idea how many systems are affected by this issue or could be affected by issues like it. How could they? They’re expecting motherboard OEMs to follow their instructions (and understandably so). Just look at the combination of variables required to reproduce this issue:

- Intel 82574L
- Various specific misconfigured bytes in the EEPROM
- An insanely specific packet with the right value at just the right byte, received at a specific time

While most people weren’t able to reproduce this issue with their controller and EEPROM combinations it did kick off various discussions of periodic, random, sporadic failures across a wide range of ethernet adapters and general computing weirdness. A quick Google search returns a wide assortment of complaints with these adapters (and others like it) from a whole slew of users. EEPROM corruption. Random adapter resets. Packet drops. Various latency issues. PCI bus scan errors. ASPM problems. The list goes on and on.

Perhaps the “packet of death” for a slightly misconfigured Intel 82579 (for example) is my packet shifted 20 bytes in one direction or the other. Who knows? Please, please, please Intel - lets do everyone a favor and get these EEPROMs under control. End users update firmware all of the time - routers, set-top boxes, sometimes even their cars! Why can’t we have some utility to make sure our ethernet adapters aren’t just waiting to freak out when they receive the wrong packet?

I don’t believe in magic, swamp gas, sun spots, or any of the other “explanations” offered for some of the random strange behavior we often see with these complex devices (ethernet adapters or otherwise). That’s why I spent so long working on this issue to find a root cause (well that and screaming customers). I, like anyone else, encounter bugs and general weirdness in devices and software everyday in my life. Most of the time how do I respond to these bugs? I reboot, shrug my shoulders, say “that was weird”, and move on. Meanwhile I know, deep down, that there is a valid explanation for what just happened. Just like there was with my ethernet controllers.

Even with the explanation offered by Intel we could go much deeper. Why these bytes at that location? Why this packet? What’s up with the “inoculation” effect of some of the values? There are still many unanswered questions.

I’ve enjoyed reading many others report their tales of “extreme debugging” with the digital devices in their lives. It seems I’m not the only one that isn’t always satisfied with saying “that was weird” and moving on.

I've said it before and I'll say it again - I love the internet!

Packets of Death

2013-02-06T11:01:00.000-05:00

When you're done reading this check out my update! Experiencing similar ethernet controller issues but don't know where to start?

UPDATE: See the packets and find out if you're affected here.

UPDATE 2: Yes, I've reproduced this issue regardless of OS, ASPM state/settings, or software firewall settings. Obviously if you have a layer 2/3 firewall in front of an affected interface you'll be ok.

Packets of death. I started calling them that because that’s exactly what they are.

Star2Star has a hardware OEM that has built the last two versions of our on-premise customer appliance. I’ll get more into this appliance and the magic it provides in another post. For now let’s focus on these killer packets.

About a year ago we released a refresh of this on-premise equipment. It started off simple enough, pretty much just standard Moore’s Law stuff. Bigger, better, faster, cheaper. The new hardware was 64-bit capable, had 8X as much RAM, could accommodate additional local storage, and had four Intel (my preferred ethernet controller vendor) gigabit ethernet ports. We had (and have) all kinds of ideas for these four ports. All in all it was pretty exciting.

This new hardware flew through performance and functionality testing. The speed was there and the reliability was there. Perfect. After this extensive testing we slowly rolled the hardware out to a few beta sites. Sure enough, problems started to appear.

All it takes is a quick Google search to see that the Intel 82574L ethernet controller has had at least a few problems. Including, but not necessarily limited to, EEPROM issues, ASPM bugs, MSI-X quirks, etc. We spent several months dealing with each and every one of these. We thought we were done.

We weren’t. It was only going to get worse.

I thought I had the perfect software image (and BIOS) developed and deployed. However, that’s not what the field was telling us. Units kept failing. Sometimes a reboot would bring the unit back, usually it wouldn’t. When the unit was shipped back, however, it would work when tested.

Wow. Things just got weird.

The weirdness continued and I finally got to the point where I had to roll my sleeves up. I was lucky enough to find a very patient and helpful reseller in the field to stay on the phone with me for three hours while I collected data. This customer location, for some reason or another, could predictably bring down the ethernet controller with voice traffic on their network.

Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller. The system and ethernet interfaces would appear fine and then after a random amount of traffic the interface would report a hardware error (lost communication with PHY) and lose link. Literally the link lights on the switch and interface would go out. It was dead.

Nothing but a power cycle would bring it back. Attempting to reload the kernel module or reboot the machine would result in a PCI scan error. The interface was dead until the machine was physically powered down and powered back on. In many cases, for our customers, this meant a truck roll.

While debugging with this very patient reseller I started stopping the packet captures as soon as the interface dropped. Eventually I caught on to a pattern: the last packet out of the interface was always a 100 Trying provisional response, and it was always a specific length. Not only that, I ended up tracing this (Asterisk) response to a specific phone manufacturer’s INVITE.

I got off the phone with the reseller, grabbed some guys and presented my evidence. Even though it was late in the afternoon on a Friday, everyone did their part to scramble and put together a test configuration with our new hardware and phones from this manufacturer.

We sat there, in a conference room, and dialed as fast as our fingers could. Eventually we found that we could duplicate the issue! Not on every call, and not on every device, but every once in a while we could crash the ethernet controller. However, every once in a while we couldn’t at all. After a power cycle we’d try again and hit it. Either way, as anyone who’s tried to diagnose a technical issue knows the first step is duplicating the problem. We were finally there.

Believe me, it took a long time to get here. I know how the OSI stack works. I know how software is segmented. I know that the contents of a SIP packet shouldn’t do anything to an ethernet adapter. It just doesn’t make any sense.

Between packet captures on our device and packet captures from the mirror port on the switch we were finally able to isolate the problem packet. Turns out it was the received INVITE, not the transmitted 100 Trying! The mirror port capture never saw the 100 Trying hit the wire.

Now we needed to look at this INVITE. Maybe the userspace daemon processing the INVITE was the problem? Maybe it was the transmitted 100 Trying? One of my colleagues suggested we shutdown the SIP software and see if the issue persisted. No SIP software running, no transmitted 100 Trying.

First we needed a better way to transmit the problem packet. We isolated the INVITE transmitted from the phone and used tcpreplay to play it back on command. Sure enough it worked. Now, for the first time in months, we could shut down these ports on command with a single packet. This was significant progress and it was time to go home, which really meant it was time to set this up in the lab at home!

Before I go any further I need to give another shout out to an excellent open source piece of software I found. Ostinato turns you into a packet ninja. There’s literally no limit to what you can do with it. Without Ostinato I could have never gotten beyond this point.

With my packet Swiss army knife in hand I started poking and prodding. What I found was shocking.

It all starts with a strange SIP/SDP quirk. Take a look at this SDP:

v=0
o=- 20047 20047 IN IP4 10.41.22.248
s=SDP data
c=IN IP4 10.41.22.248
t=0 0
m=audio 11786 RTP/AVP 18 0 18 9 9 101
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:9 G722/8000
a=rtpmap:9 G722/8000
a=fmtp:101 0-15
a=rtpmap:101 telephone-event/8000
a=ptime:20
a=sendrecv

Wireshark picture:

Yes, I saw it right away too. The audio offer is duplicated and that’s a problem but again, what difference should that make to an Ethernet controller?!? Well, if nothing else it makes the ethernet frame larger...

But wait, there were plenty of successful ethernet frames in these packet captures. Some of them were smaller, some were larger. No problems with them. It was time to dig into the problem packet. After some more Ostinato-fu and plenty of power cycles I was able to isolate the problem pattern (with a problem frame).

Warning: we’re about to get into some hex.

The interface shutdown is triggered by a specific byte value at a specific offset. In this case the specific value was hex 32 at 0x47f. Hex 32 is an ASCII 2. Guess where the 2 was coming from?

a=ptime:20

All of our SDPs were identical (including ptime, obviously). All of the source and destination URIs were identical. The only difference was the Call-IDs, tags, and branches. Problem packets had just the right Call-ID, tags, and branches to cause the “2” in the ptime to line up with 0x47f.

BOOM! With the right Call-IDs, tags, and branches (or any random garbage) a “good packet” could turn into a “killer packet” as long as that ptime line ended up at the right address. Things just got weirder.

While generating packets I experimented with various hex values. As if this problem couldn’t get any weirder, it does. I found out that the behavior of the controller depended completely on the value of this specific address in the first received packet to match that address. It broke down to something like this:

Byte 0x47f = 31 HEX (1 ASCII) - No effect
Byte 0x47f = 32 HEX (2 ASCII) - Interface shutdown
Byte 0x47f = 33 HEX (3 ASCII) - Interface shutdown
Byte 0x47f = 34 HEX (4 ASCII) - Interface inoculation

Bad:

Good:

When I say “no effect” I mean it didn’t kill the interface but it didn’t inoculate the interface either (more on that later). When I say the interface shutdown, well, remember my description of this issue - the interface went down. Hard.

With even more testing I discovered this issue with every version of Linux I could find, FreeBSD, and even when the machine was powered up complaining about missing boot media! It’s in the hardware; the OS has nothing to do with it. Wow.

To make matters worse, using Ostinato I was able to craft various versions of this packet - an HTTP POST, ICMP echo-request, etc. Pretty much whatever I wanted. With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!

I know I’ve been pointing out how weird this whole issue is. The inoculation part is by far the strangest. It turns out that if the first packet received contains any value (that I can find) other than 1, 2, or 3 the interface becomes immune from any death packets (where the value is 2 or 3). Also, valid ptime attributes are defined in ~~powers~~ (edit: multiples) of 10 - 10, 20, 30, 40. Depending on Call-ID, tag, branch, IP, URI, etc (with this buggy SDP) these valid ptime attributes line up perfectly. Really, what are the chances?!?

All of a sudden it’s become clear why this issue was so sporadic. I’m amazed I tracked it down at all. I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. At least I hope I don’t...

I was able to get in touch with two engineers at Intel and send them a demo unit to reproduce the issue. After working with them for a couple of weeks they determined there was an issue with the EEPROM on our 82574L controllers.

They were able to provide new EEPROM and a tool to write it out. Unfortunately we weren’t able to distribute this tool and it required unloading and reloading the e1000e kernel module, so it wouldn’t be preferred in our environment. Fortunately (with a little knowledge of the EEPROM layout) I was able to work up some bash scripting and ethtool magic to save the “fixed” eeprom values and write them out on affected systems. We now have a way to detect and fix these problematic units in the field. We’ve communicated with our vendor to make sure this fix is applied to units before they are shipped to us. What isn’t clear, however, is just how many other affected Intel ethernet controllers are out there.

I guess we’ll just have to see...

FreeSWITCH Stable Branch - Sponsored by Star2Star

2012-09-21T20:18:00.003-04:00

As many of you know I’ve been quite fond of FreeSWITCH for some time now. I’ve been impressed with the functionality, stability, and performance. Did I say impressed? I suppose I meant to say thrilled. Some of my more long-term readers may remember this post from three years ago. I can’t believe it’s been that long. I suppose that makes me (officially) old.

Getting back on track (as I often have to do) I have had one concern with FreeSWITCH over the years - the lack of a stable branch. Traditionally (in software development) once a project or piece of code reaches a certain level of maturity (or use, even) there comes a need to segment the code into an independently maintained entity. In most revision control systems this is typically called a “branch”. Makes sense because lines of code are often referred to as “trees”. I get it!

Since the beginning of the project in 2006 all of FreeSWITCH development has take place in one central code repository (trunk - trees again). At any given point in time a user could pull down trunk and receive the latest and “greatest” code - new features, bug fixes, security patches, etc. Unfortunately these new features can introduce new bugs. They can re-expose old bugs. They can change behavior in unexpected ways.

One of the things that has been most impressive about the FreeSWITCH project, actually, has been the relative stability of this often new and untested code. It’s been extremely rare to find a serious issue in trunk. However, the mere knowledge of this code management practice (or relative lack thereof) and potential for new features/bugs is an issue.

I learned a long time ago that people expect their phones to just work. The same user who finds it perfectly acceptable to reboot their computer after a software crash will jump and scream when a telephone doesn’t work. I guess that’s a testament to the level of experience people have historically expected. Interestingly this expectation seems to be changing in the face of new technology, which is something I have covered before.

Anyway, for the time being people (including myself) expect these things to “just work”. Of course this expectation places a tremendous burden on network and facility operators such as myself (and rightfully so). Meeting this expectation with FreeSWITCH trunk has required a significant amount of testing, testing, re-testing, regression testing, patching, etc. We simply cannot unleash a new version of software in the field without significant testing. When issues are found we have to address them. This takes time and resources.

Meanwhile FreeSWITCH is an open source project. That can’t be expected to provide for every need and whim of large commercial users such as myself. Why should they? They already spend so much of their limited time and resources to provide software that essentially powers my business and (indirectly) provides for my livelihood - for free. With these limited resources alone they can’t be expected to make time for the maintenance of yet another collection of code.

That’s why I’m happy to announce the Star2Star sponsorship of the FreeSWITCH stable branch! For the past several months Star2Star has been providing the financial assistance necessary for the FreeSWITCH project to hire another full-time team member to not only maintain a stable branch (1.2 as of this writing) but improve documentation, packaging, and community interaction. Everyone at Star2Star (including myself) couldn’t be happier to provide this resource for the project and the community. We look forward to working more with the FreeSWITCH team on the stable branch and any other projects that may advance FreeSWITCH and the state of the art in communications!

Performance Testing (Part 2) - Going Commercial

2012-07-26T13:48:00.000-04:00

Some time ago I wrote the first installment of a multi-part series on performance testing. Here we go with part two!

In part one I talked about some of the difficulties surrounding performance testing - functionality, flexibility, high scale, quality metrics, etc.

After looking at a couple commercial products we discovered a few things:

1) Some of these products really do “have it all”.
2) They can be very expensive.

There was some initial sticker shock ($100,000+) but looking back I shouldn’t have been so surprised. My first reaction (of course) was to reach out to a few people in the open source community with a proposal. In a classic “build vs. buy” scenario I wanted to build. This is (roughly) what I needed:

- At least 40,000 RTP streams (20,000 calls)
- At least 100,000 SIP calls and/or registrations
- The ability to emulate multiple user agents (VLAN, MAC address, DHCP client, SIP UA)
- RTP stream analysis on each RTP leg (MOS, R-Factor, PESQ, etc)
- Flexible device emulation - SIP headers, supported SIP features, etc
- Multiple codec support (at least G711, G729, G722, SILK/OPUS with FEC et
all, etc).
- Control of test scenarios - CPS, number of calls, duration of call,
total duration of test, etc
- Ability to save/load tests via web interface (for ease of use,
comparison of results, etc)
- Ability to perform feature testing - generate DTMF during a call to
navigate an IVR, for example.
- A modular system for monitoring the device under test - Linux load,
CPU usage, disk usage, network I/O, etc. Could also monitor Cisco
switches in between devices, Windows hosts, etc. Maybe even
FreeSWITCH or Asterisk if that's what was running on the device under
test.
- Saving and graphing of all relevant performance data - call setup
time, delay, duration, RTP jitter, packet loss, RTP stream stats, etc.
Ability to save data and generate reports from said data.
- Scalable design with master/slave architecture to scale across hosts
with the ability for hardware.

Did I mention this tool needs to be usable by various test engineers, some of which don’t know the difference between SIP and SDP (and rightfully so, they shouldn’t need to)?

With the open source software already available I figured this could be made available for less than the cost of a commercial testing solution.

I gave it away with the title of this post but you can guess what happened next: it was going to cost far more to develop everything I needed. By the way - it would also take six months to build and take 10 years off my life hunting bugs, managing the project, etc.

BUY

For less than what it would cost to build everything above I could buy this. A multi-user chassis with one Xcellon-Ultra NP load module and room for another one. 180,000 emulated SIP endpoints. 96,000 RTP streams. Wire speed 10 gigabit VoIP testing (and 12 gigabit ports).

Of course this isn’t a perfect world... The chassis runs Windows. The client software is only available for Windows and the interface is probably the furthest from what I want. As a guy that eats, lives, and breathes CLI (and has for a decade) multi-pane/dropdown/hidden/shadow GUIs are NOT my thing. I don’t even know what to call or how to describe some of the window/GUI elements present in the IxLoad user interface...

Ixia even has a solution to this problem. They offer TCL scripting with clients for various platforms, including Linux! While we’ll eventually get into that for the time being we went with a much simpler solution: we setup a Windows terminal server. I use CoRD from my Mac, login to the terminal server, and run a test. As you’ll see in part three - IT JUST WORKS.

My Linux Story

2012-07-12T13:10:00.001-04:00

While digging through some boxes the other day I came across a book. Not just any book - a very important book. A book I both cursed and loved more than any other book in my entire life.

To tell the story of the book I need to give a little background. My father was a professor at the University of Illinois at Chicago. When I was old enough (13) I would spend my summers volunteering in his department (Occupational Therapy) doing random IT jobs - making ethernet cables, cleaning up viruses, fixing printers, etc.

One day they took me on a tour of the other departments in his building. They had some medical visualization apps running in one of the departments. Here we were, sitting in a room, with some PhD students working on some of the coolest computers I’d ever seen. They had bright colors and huge monitors. They ran tons of advanced looking applications. Most of the interaction was through a very interesting command line interface. I was intrigued.

Being a rambunctious 13 year old I approached one of the students and complimented him on his computer. I also asked him how I could get one of them for my house. He laughed. At the time I didn’t understand why it was a funny question. Looking back now I know it’s because those “computers” were actually high powered Silicon Graphics Indigo workstations that cost upwards of $40,000 each (in 1997 dollars).

His response was a muffled “Ummm, you can’t”. Not being dissuaded my follow-up question was “How can I get something like it?”. One of the students looked at the other student. They both kind of shrugged until one of them said “umm, Linux?”. Still not being quite sure they both agreed that “Linux” would be my best bet. I asked them how I could get “Linux”. They told me to go down to the university book store and buy a book about Linux. A good Linux book, they said, should include a CD-ROM with a copy of Linux I could install on my computer at home.

That day, before we went to Union Station to catch the Metra, my dad and I went to the university bookstore to find a book about Linux. Sure enough we found a Linux book that included not one but TWO CD-ROMs! I read as much of the book as I could before I got home that night.

Once I was at home I booted the family computer (with a DOS bootdisk) and ran a loadlin (wow, remember that?) based install program from the CD-ROM (no el torito!). During the course of the install I repartitioned and reformatted the (only) family computer - an IBM Aptiva M51. It had a Pentium 100, 1GB hard drive, and 16MB of RAM (upgraded from 8MB). It also came with one of those HORRIBLE MWAVE adapters and some shady Cirrus Logic graphics (IIRC).

Anyway, the install process (which was pretty horrible, actually) left me with what was (at the time) a barely usable computer. How was my sister going to do her homework? How were we going to get on to the internet? Uh-oh, looks like I better learn more about this “Linux” thing...

So that’s how it started. Later that year (for Christmas) my parents realized they weren’t going to get the Aptiva back from me so we bought another computer to run Windows. At that point my little Linux workstation became my computer and the “gateway” for my first home network - a 10BASE2 (coax!) Ethernet network using NE2000 cards from Linksys. Internet access went through my Aptiva using an external modem that, regardless of type, could only negotiate a maximum of 19,200 bps on our crappy phone line. PPP, chat scripts, dial-on-demand, squid for caching, etc. 15 years later I’m still (primarily) using Linux!

After finding “My First Linux Book” I wondered what would happen if I tried to install that version of Linux today. Some people reminisce about their childhood by hearing certain songs, playing a sport, or collecting action figures. I (apparently) do it by installing ancient versions of Linux.

I needed to install this version of Linux but the CD-ROMs in my book were missing. I looked around the internet for a while but could not find an ISO or copy of the distro anywhere. I could barely find references to the book. Where else could I look? Everywhere else I look - Amazon!

Sure enough, Amazon had a used copy of the book (with CD-ROM) for $5 with shipping. Two days later it was here. To my surprise the book (and the CDs) were in excellent condition. Who the hell is keeping a warehouse full of mid-nineties Linux books (3rd edition with Kernel 2.0!)?

I get on my main development machine at home, download a FreeDOS ISO (to install from, remember), and create a VirtualBox virtual machine. What should it look like? I decide to “go big”:

1 CPU
256MB of RAM
8GB Intel PIIX3 hard drive
1 Am79C973 ethernet port

Keep in mind I’m going to be running kernel 2.0 here - this hardware needs to be supported by a 15 year old kernel. I get Unifix Linux 2.0 installed. Moments later I’m logged into my “new” Linux system. Not knowing exactly what to do now, I decide to try to get networking to work.

Long story short I could not get Linux 2.0 to recognize the emulated Am79C973 ethernet controller. I tried changing the device ids and recompiling the kernel (takes less than one minute, btw) but couldn’t get it to work.

Hmmm, what else could I do for connectivity? Maybe I could go really nostalgic and get something running over the serial port?

I configured VirtualBox to emulate a 16550 serial port as COM1. I setup VirtualBox to point the other end of the emulated serial port to a local pipe. I figured that if I could somehow run pppd on both sides of this serial port (host and guest) and configure NAT I could get this thing on the internet.
Here’s how I did it:

1) Launch socat to convert the unix domain socket provided by VirtualBox to a standard Linux tty so pppd can run on it:

(on host)
socat UNIX-CONNECT:[com1] PTY,link=[vmodem0],raw,echo=0,waitslave

Where [com1] is the path to your VirtualBox socket and [vmodem0] is the path to your (new) tty.

2) Launch pppd on the new tty:

(on host)
pppd [vmodem0] 57600 192.168.100.1:192.168.100.2 nodetach local

Once again where [vmodem0] is the path to your new socat tty. Make sure that the IP addresses provided for each end of the PPP link don’t conflict with any local IP addresses.

3) Setup kernel iptables on the host:

(on host)
echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ppp0 -o eth0 -j ACCEPT

4) Connect the virtual machine to the host:

(on guest)
pppd /dev/ttyS0 57600 defaultroute passive

Sure enough, here’s what I saw on the host:

Using interface ppp0
Connect: ppp0 <--> /home/kris/projects/linux_universe/vmodem0
local IP address 192.168.100.1
remote IP address 192.168.100.2

Boom! 1990s Linux, meet the 21st century!

Once I had networking up and running things really took off. I noticed all of the services running by default on my old Linux host (portmap, yp, apache, telnet, echo, chargen, sendmail, wu-ftpd, etc). Remember the 90s when the internet wasn’t such a hostile place!?!

Here’s some fun command output from my “new” host old_linux:

root@old_linux:~ # uname -a
Linux old_linux 2.0.25 Unifix-2.0 i686

root@old_linux:~ # ping -c 5 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=15.1 ms
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=19.9 ms

--- 192.168.100.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 15.1/18.9/19.9 ms
root@old_linux:~ #

20ms of latency on the same physical machine (pppd -> VirtualBox -> socat -> pppd)!

root@old_linux:~ # ping -c 5 www.google.com
PING www.l.google.com (74.125.139.104): 56 data bytes
64 bytes from 74.125.139.104: icmp_seq=0 ttl=45 time=59.5 ms
64 bytes from 74.125.139.104: icmp_seq=1 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=2 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=3 ttl=45 time=50.5 ms
64 bytes from 74.125.139.104: icmp_seq=4 ttl=45 time=69.9 ms

--- www.l.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 49.9/55.9/69.9 ms

root@old_linux:~ # gcc -v
Reading specs from /usr/lib/gcc-lib/i486-unknown-linux/2.7.2.1/specs
gcc version 2.7.2.1

root@old_linux:~ # httpd -v
Server version Apache/1.1.1.

root@old_linux:~ # ssh -v
SSH Version 1.2.14 [i486-unknown-linux], protocol version 1.4.
Standard version. Does not use RSAREF.

Pre-iptables. Pre-ipchains. IPFWADM!

root@old_linux:~ # ipfwadm -h
ipfwadm 2.3.0, 1996/07/30

root@old_linux:/usr/src/linux-2.0.25 # time make

[trimmed output]

The new kernel is in file arch/i386/boot/bzImage; now install it as vmlinuz
A name list of the kernel is in vmlinux.map; you may install it as vmlinux
9.86user 11.74system 0:22.78elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #

23 seconds to compile the kernel!

Ok, but what about the modules?

make[1]: Leaving directory `/usr/src/linux-2.0.25/arch/i386/math-emu'
8.80user 7.42system 0:16.80elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #

16 seconds to compile the modules!

I have to admit I’m fascinated with this distro and the state of Linux as I was first introduced to it. Of course some of the memories have faded over time but more than anything it’s amazing to see how far we’ve come...

1997:

Linux 2.0.25

Original Pentium at 100MHz

16MB RAM

1GB hard drive

1MB VRAM

2012:

Linux 3.2.0 (Ubuntu 12.04)

Quad-core Pentium Xeon at 2.27GHz

12GB RAM

1TB hard drive x2 (RAID 1)

1GB VRAM (dual 1080p displays)

...and this machine is pretty old at this point! Needless to say with 256MB of RAM (double the maximum possible in my Aptiva) and even one emulated CPU Unifix 2.0 barely knows what to do with this new hardware (even if it isn’t real)!

While I was never quite sure why I was doing any of this I can tell you that it was very fun to remember how all of this got started. I remember my next distro being RedHat 5.2...

Everything you wish you didn't need to know about VoIP

2012-06-14T15:57:00.000-04:00

A few years back I was talking with my editor at O'Reilly Media about a book I'd like to write. The book would cover details of the SIP protocol, best practice, interop scenarios, and even a few implementation specifics - FreeSWITCH, Asterisk, OpenSIPS, Kamailio, etc. Basically your typical open source software book only this time it would be SIP protocol inward.

While my editor liked the idea (I think they have to tell you that) he said there wouldn't be much of a market for it. If I remember correctly his exact words were "Kris, that's a great idea but only 100 people in the world would buy it". Clearly you can't print a physical book through a major publisher with editors, technical reviewers, etc if only 100 people are going to buy it. I tabled the idea.

Several years later I find myself still regularly talking about SIP and going into many protocol and implementation specifics. Like my editor once told me it seems there aren't a lot of people in this area with either the interest or experience. I guess he was right. Still, SIP is confusing enough and widespread enough that something has to be done.

Over the past couple of months (off and on - I rarely work uninterrupted these days) I sat down and wrote. Stream of consciousness, without reference, writing. What I ended up with is a (currently) 21 page document I like to call "Everything you wish you didn't need to know about VoIP".

It's still an early draft but as we say in open source "release early, release often". It has typos. It may not always be factually correct. There are no headings, chapters, or spacing. I may not always use apostrophes correctly. Over time I will correct these mistakes and hopefully (with your help) address other topics of concern or questions my readers may have. I may even divide it into logical chapters at some point! Wow, that would be nice.

However, as the philosophers say a 100 mile journey begins with a single step. With that said, blog reader, I present to you "Everything you wish you didn't need to know about VoIP".

Let me know what you think (I have a comments section and easy to find e-mail address).

Sprechen sie deutsch?

2012-06-04T09:41:00.001-04:00

Do you speak German?

I don't. I'm sure this comes as a shock to the many people who have sent me e-mails in German over the years. I suppose my last name may give that impression...

Anyway, longtime AstLinux community member and contributor Michael Keuter has setup an AstLinux focused blog in German. Check it out!

AstLinux Custom Build Engine now available!

2012-03-12T10:23:00.000-04:00

Ever since releasing the AstLinux Development Environment several years ago the AstLinux Developers have spent a significant amount of time supporting new users who are (in many cases) building their first image with only minor customizations - slightly different hardware support, different Asterisk versions, etc.

The trouble is, cross compiling anything is an extremely complicated task. To be honest I'm surprised it works as often as it does. When you step back and really look at what's going on it all starts to seem like magic. Many people working in this space will openly admit to being practitioners of vodoo or one of the other dark arts.

After a brief e-mail exchange last week Lonnie Abelbeck and I decided to do something about this. What if we could host a system with a web interface to build custom AstLinux images for users on demand? What if this system could cache previous image configurations and save them for future users? What if this system could be easily adapted to meet future configuration needs?

Amazingly, barely a week later, Lonnie has provided all of these features and more. Available immediately, the AstLinux Custom Build Engine is online to build custom AstLinux images that meet your needs.

In an effort to keep bots, crawlers, and robots in general out we've added simple username and password authentication. The secret is out and the username is "admin" with a password of "astlinux". AstLinux users will recognize these credentials from the default administrative web interface provided with AstLinux. These users will also recognize the familiar tabbed interface.

Go ahead and give it a try!

These interfaces look alike because they share the same DNA. Lonnie Abelbeck has done a great job creating a build system to serve our users now and in the future. Thanks again Lonnie!

P.S. - Lonnie just found this post from me, dated 8/25/2005, where I talk about something that looks a lot like build.astlinux.org. If only Lonnie were around back then to help me actually create such a beast!

Hyperspecialization and the shakeup of a 100 year old industry

2012-02-13T14:06:00.002-05:00

As someone who often finds themselves "in the trenches" dealing with some extremely nerdy technical nuances it's often easy to miss the larger picture. I guess Mark Spencer was right when he said "Not many people get excited about telephones but the ones who do get REALLY excited about telephones".

As someone who's natural inclination is to get stuck in the details I certainly understand this. Some of you might be right there with me. At this point I've gotten so specialized I'm next to useless on some pretty basic "computer things". Think of the aunt or other relative/friend that lights up when they find out you're a "computer guy". Then they inevitably pull you aside at a wedding to ask for help with their printer or "some box that pops up in Windows". I'm happy to not have to feign ignorance any longer: I truly am ignorant on issues like these.

I primarily use a Mac because it just works - for me. That's not the point I'm trying to make here, though. A Mac works so well for me because I use just two applications: a terminal (Iterm2, to be exact) and Google Chrome. Ok, ok every once in a while I whip up some crazy Wireshark display syntax but that's another post for another day. For the most part when I need to work I take out my Mac, it comes out of sleep, connects to a network, and I start working. It's a tool.

As far as anything with a GUI goes that's the extent of my "expertise". If my aunt wanted to ask me about my bash_profile, screenrc settings, IPv4 address exhaustion, or SIP network architecture I may have something to say. Other than that you'll find me speaking in vague generalities than may lead the more paranoid to suspect I'm secretly a double for the CIA or some international crime syndicate member. I wish I were kidding, this has actually happened before although "international crime syndicate" usually gets loosely translated to "drug dealer". How else does a supposed "computer guy" not understand what's wrong with my printer?!?!

As usual there's a point to all of this. My hyperspecialization, in this case, allows me to forget what is really going on all around me: a shakeup in the 100 year old industry I find myself in and a change in the way we communicate.

The evolution of the telephone is a strange thing. It is a device and service that has remained largely unchanged for 100 years. I'm not kidding. To this day, in some parts of the United States, the only telephone service available could be installed by Alexander Graham Bell himself. Sure there have been many advances since the 1900s but they've been incremental improvements at best - digital services with the same voice bandwidth (dating to 1972), various capacity and engineering changes, and of course - the cell phone.

In the end, however, we're left with a service that isn't much different than what my grandparents had. You still have to phonetically spell various upper-frequency consonants ("S as in Sam, P as in Paul, T as in Tom") because the upper limit of the voice bandwidth on these services is ridiculously low (3.1 kHz). Straining to hear the party at the remote end of a phone has only gotten worse with various digital compression standards in use today - EVRC, AMR, G.729, etc. I love to compare the "pin drop" Sprint commercials of the 80s and 90s to the Verizon Wireless "CAN YOU HEAR ME NOW?" campaign over 20 years later. We still dial by randomly assigned strings of 10 digit numbers. This is supposedly progress?

One thing that has changed - the network has gotten bigger. Much bigger. My grandparents may have not had much use for their party line because they didn't have anyone of interest to talk to on the other end. In this manner the network has exploded - and it has exploded using the same standards that have been in place for these past 100 years. I can directly dial a cell phone on the other side of the world and be connected in seconds.

Meanwhile, there has been another network explosion - IP networks and the internet. The internet, of course, needs no introduction. While I'd love to spend some time talking about IP that's time I don't have at this point. Let's just look at a couple of ways IP has been extremely disruptive for this 100 year old franchise.

Not many people outside of telecom noticed it at the time but back in 2009 AT&T (THE AT&T) petitioned the FCC to decommission the legacy PSTN (copper and pairs and what-not). Just over two years later we're starting to see some results, and AT&T is realizing some ancillary benefits.

As someone who has spent some time (not a lot, thankfully) in these central offices the maze of patch cables, wiring blocks, DC battery banks, etc make you really appreciate the analysis of this report. Normally networks are completely faceless - you go to www.google.com or dial 8005551212 without seeing the equipment that gets you to the other end. The fact that SBC reclaimed as much as 250 MILLION square feet by eliminating this legacy equipment is incredible.

That's all well and good but what has AT&T done for us, the users? The answer is, unfortunately, both good and bad. AT&T like many physical, trench-digging network providers, has realized they are in the business of providing IP connectivity. They don't have much of a product anymore and the product they do have is becoming more and more of a commodity everyday.

Getting out of the way is the smartest thing they could be doing. Speaking of AT&T, remember the Apple iPhone deal? At the time a cell phone was a cell phone - AT&T provided an IP network and got some minutes but Apple built an application platform and changed the way people view the devices they carry with them everywhere they go. Huge.

Watch any sci-fi movie from the past 50 years and one almost ubiquitous "innovation" is the video phone. Did AT&T or some other 100 year old company provide the video phone for baby's first steps to be beamed to Grandma across the country? No - Apple did it with Facetime and a little company from Estonia (Skype) did it over the internet. Thanks to these companies and IP networks we finally have video conferencing (maybe they'll release a 30th anniversary edition of Blade Runner to celebrate).

Unfortunately, there will always be people that cling to technologies of days past - this new network works well for all of these applications that were designed for it. Meanwhile, some technologies are being shoehorned in with disastrous results. Has anyone noticed faxing has actually gotten LESS reliable over the past several years? That's what happens when you try to use decades-old modem designs on a completely different network. You might as well try to burn diesel in your gasoline engine.

The future is the network and the network (regardless of physical access medium) is IP.

And now, for good measure, here are some random links for further reading:

Graves on SOHO Technology - An early advocate of HD Voice, etc.
The Voice Communication Exchange - Wants to push the world to HD Voice by 2018.

Performance Testing (Part 1)

2011-12-20T17:23:00.000-05:00

Over the past few years (like many other people in this business) I’ve needed to do performance testing. Open source software is great but this is one place where you need to do your own leg work. This conundrum first presented itself in the Asterisk community. There are literally thousands of variables that can affect system the performance of Asterisk, FreeSWITCH, or any other software solution. In no particular order:

- Configuration. Which modules do you have loaded? How are they configured? If you’re using Kamailio, do you do hundreds of huge, slow, nasty DB queries for each call setup? How is your logging configured? Maybe you use Asterisk or FreeSWITCH and so several system calls, DB lookups, LUA scripts, etc? Even the slightest misstep in configuration (synchronous syslogging with Kamailio, for example) can reduce your performance by 90%.

- Features in use. Paging groups (unicast) are notorious for destroying performance on standard hardware - every call needs to be setup individually, you need to handle RTP, and some audio mixing is involved. Hardware that can’t do 10 members in a page group using Asterisk or FreeSWITCH may be capable of hundreds of sessions using Kamailio with no media.

- Standard performance metrics. “Thousands of calls” you say? How many calls per second? Are you transcoding? Maybe you’re not handling any media at all? What is the delay in call setup?

- Hardware. This may seem obvious (MORE HERTZ) but even then there are issues... If you’re handling RTP, what are you using for timing? If you have lots of RTP, which network card are you using? Does it and your kernel support MSI or MSI-X for better interrupt handling? Can you load balance IRQs across cores? How efficient (or buggy) is the driver (Realtek I’m looking at you)?!?

- The “guano” effect. As features are added to the underlying toolkit (Asterisk, FreeSWITCH, etc) and to your configuration, how is performance affected over time? Add a feature here, and a feature there - and repeat. Over the months and years (even with faster hardware) you may find that each “little” feature reduced call capacity by 5%. Or maybe your calls per second went down by two each time. Not a big deal overall yet over time this adds up - assuming no other optimizations your call capacity could be down by 50% after ten “minor” changes. It adds up - it really does.

Even when pointing out all of these issues you’d still be surprised how often one is faced with the question “Well yeah but how many calls can I handle on my dual core Dell server?”.

In almost every case the best answer is “Get your hardware, develop your solution, run sipp against it and see what happens”. That’s really about as good as we can do.

SIPP is a great example of a typical, high quality open source tool. In true “Unix philosophy” it does one thing and it does it well: SIP performance testing. SIPP can be configured to initiate (or receive) just about any conceivable SIP scenario - from simple INVITE call handling to full SIMPLE test cases. In these tests SIPP will tell you call setup time, messages received, successful dialogs, etc.

SIPP even goes a step further and includes some support for RTP. SIPP has the ability to echo RTP from the remote end or even replay RTP from a PCAP file you have saved to disk. This is where SIPP starts to show some deficiencies. Again, you can’t blame SIPP because SIPP is a SIP performance testing tool - it does that and it does it well. RTP testing leaves a lot to be desired. First of all, you’re on your own when it comes to manipulating any of the PCAP parameters. Length, content, codec, payload types, etc, etc need to be configured separately. This isn’t a problem, necesarily, as there are various open source tools to help you with some of these tasks. I won’t get into all of them here but they too leave something to be desired.

What about analyzing the quality of the RTP streams? SIPP provides mechanisms to measure various SIP “quality” metrics - SIP response times, SIP retransmits, etc. With RTP you’re on your own. Once again, sure, you could setup tshark on a SPAN port (or something) to do RTP stream analysis on every stream but this would be tedious and (once again) subject you to some of the harsh realities of processing a tremendous amount of small packets in software on commodity hardware.

Let’s face it - for a typical B2BUA handling RTP the numbers add up very quickly - let’s assume 20ms packetization for the following:

Single RTP stream = 50 packets per second (pps)
Bi-directional RTP stream = 100 pps
A-leg bi-directional RTP stream = 100 pps
B-leg bi-directional RTP stream = 100 pps

A leg + B leg = 200 pps PER CALL

What does this look like with 10,000 channels (using g711u)?

952 mbit/s (close to Gigabit wire speed) in each direction
1,000,000 (total) packets per second

Open source software is great - it provides us with the tools to (ultimately) build services and businesses. Many of us choose what to focus on (our core competency). At Star2Star we provide business grade communication services and we spend a lot of time and energy to build these services because it’s what we do. We don’t sell, manufacture, or support testing platforms.

At this point some of you may be getting an idea... Why don’t I build/design an open source testing solution? It’s a good question and while I don’t want to crush your dreams there are some harsh realities:

1) This gets insanely complicated, quickly. Anyone who follows this blog knows SIP itself is complicated enough.
2) Scaling becomes a concern (as noted above).
3) Who would use it?

The last question is probably the most serious - who really needs the ability to initiate 10,000 SIP channels at 100 calls per second while monitoring RTP stream quality, etc? SIP carriers? SIP equipment manufacturers? A few SIP software developers? How large is the market? What kind of investment would be required to even get the project off the ground? What does the competition look like? While I don’t have the answers to most of these questions I can answer the last one.

Commercial SIP testing equipment is available from a few vendors:

Spirent
Empirix
Ixia

...and I’m sure others. We evaluated a few of these solutions and I’ll be talking more about them in a follow-up post in the near future.

Stay tuned because this series is going to be good!

Star2Star Gets Noticed

2011-12-02T09:58:00.001-05:00

Just a quick one today (and some shameless self promotion on my part)... Star2Star has been recognized on a few "lists" this year, check it out:

Inc 500
Forbes 100 "Most Promising"

I'm lucky enough to tell people the same story all of the time - when I was a little kid I played with all of this stuff because I thought it was fun and I loved it. Only later did I realize that one day I'd be getting paid for it. I certainly never thought it could come to this!

Ok, enough of that for now. I'll be getting back to some tech stuff soon...

Building a Startup (the right way)

2011-11-15T14:45:00.001-05:00

(Continued from Building a Startup)

Our way wasn’t working. To put it mildly our “business grade” solution didn’t perform much better than Vonage. We became to exemplify VoIP - jittery calls, dropped calls, one way calls, etc, etc, etc. Most of this was because of the lack of quality ITSPs at that time. Either way our customers didn’t care. It was us. If we went to market with what we had the first time around we were going to loose.

The problem was the other predominant architecture at the time was “hosted”. Someone hosts a PBX for you and ships you some phones. You plug them in behind your router and magically you have a phone system. They weren’t doing much better. Sure, their sales looked good but even then it was becoming obvious customer churn was quite high. People didn’t like hosted either, and for good reason. Typically they have less control over the call than we do.

As I’ve eluded before I thought there was a better way. We needed to host the voice applications where it made the most “sense”. We were primarily using Asterisk and with a little creative provisioning, a kick-ass SIP proxy, and enough Asterisk machines we could build the perfect business PBX - even if that meant virtually none of it existed at the customer premise. Or maybe all of it did. That flexibility was key. After a lot of discussions, whiteboard sessions, and late nights everyone agreed. We needed a do-over.

So we got to work and slowly our new architecture began to take shape. We added a kick-ass SIP proxy (OpenSER). OpenSER would power the core routing between various Asterisk servers each meeting different needs - IVR/Auto Attendant, Conferencing, Voicemail, remote phones (for “hosted” phones/softphones), etc. The beauty was the SIP proxy could route between all of these different systems including the original AstLinux system at the customer premise. Customer needs to call voicemail? No problem - the AstLinux system at the CPE fires an INVITE off to the proxy and the proxy figures out where their voicemail server is. The call is connected and the media goes directly between the two endpoints. Same thing for calls between any two points on the network - AstLinux CPE to AstLinux CPE, PSTN to voicemail, IVR to conference.

This is a good time to take a break and acknowledge what really made this all possible - OpenSER. While it’s difficult to explain the exact history and family tree with any piece of SER software I can tell you one thing - this company would not be possible without it. There is no question in my mind. It’s now 2011 and whether you select Kamailio or OpenSIPS for your SIP project you will not be sorry. Even after five years you will not find a more capable, flexible, scalable piece of SIP server software. It was one of the best decisions we ever made.

Need to add another server to meet demand for IVR? No problem, bring another server online, add the IP to a table and presto - you’re now taking calls on your new IVR. Eventually a new IVR lead to several new IVRs, voicemail servers, conference systems, web portals, mail servers, various monitoring systems, etc.

What about infrastructure? Starting at our small scale, regional footprint, and focus on quality we began to buy our own PRIs and running them on a couple of Cisco AS5350XM gateways. This got us past our initial issues with questionable ITSPs. Bandwidth was becoming another problem... We had an excellent colocation provider that provided blended bandwidth but still we needed more control. Here came BGP, ARIN, AS numbers, a pair of Cisco 7206VXRs w/ G2s, iBGP, multiple upstream providers, etc.

At times I would wonder - whatever happened to spending my time worrying about cross compilers? Looking back I’m not sure which was worse - GNU autoconf cross-compiling hell or SIP interop, BGP, etc. It’s fairly safe to say I’m a sadomasochist either way.

Even with all of the pain, missteps, and work we finally had an architecture to take to market. It would be the architecture that would serve us well for several years. Of course there was more work to be done...

Building a Startup

2011-11-02T16:49:00.000-04:00

(Continued from Starting a Startup)

After several days of meetings in Sarasota we determined:

1) I was moving to Sarasota to start a company with Norm and Joe.
2) We were going to utilize open source software wherever possible (including AstLinux, obviously).
3) The Internet was the only ubiquitous, high quality network to build a nationwide platform.
4) The Internet was only getting more ubiquitous, more reliable, and faster in coming months/years/decades/etc.
5) We were going to take advantage of as much of this as possible.

These were some pretty lofty goals. Remember, this is early 2006. Gmail was still invitation-only beta. Google docs didn’t exist. Amazon EC2 didn’t exist. “Cloud computing” hadn’t come back into fashion yet. The term itself didn’t exist. The Internet was considered (by many) to be “best effort”, “inherently unreliable”, and “unsuitable” for critical communications (such as real time business telephony). There were many naysayers who were confident this would be a miserable failure. As it turns out, they were almost right.

We thought the “secret sauce” to business grade voice over the internet was monitoring and management. If one could monitor and manage the internet connection business grade voice should be possible. Of course this is very ambiguous but it lead to several great hires. We hired

Joe had already deployed several embedded Asterisk systems to various businesses in the Sarasota area. They used an embedded version of Linux he patched together and a third party (unnamed) “carrier” to connect to the PSTN. The first step was upgrading these machines and getting them on AstLinux. Once this was accomplished we felt confident enough to proceed with our plan. This was Star2Star Communications and in the beginning of 2006 it looked something like this:

1) Soekris net4801 machines running AstLinux on the customer premise.
2) Grandstream GXP-2000 phones at each desk.
3) Connectivity to a third party “ITSP”.
4) Management/monitoring systems (check IP connectivity, phone availability, ITSP reliability, local LAN, etc).
5) Central provisioning of AstLinux systems, phones, etc.

This was Star2Star and there was something I really liked about it - it was simple. Anyone who knows me or knows of my projects (AstLinux, for example) has to know I favor simplicity whenever possible. Keep it simple, keep it simple, keep it simple (stupid).

As time went on we started to learn that maybe this was too simple. We didn’t have enough control. Out monitoring wasn’t as mature as it should be. We didn’t pick the right IP phones. These could be easily fixed. However, we soon realized our biggest mistake was architecture (or lack thereof). This wasn’t going to be an easy fix.

We couldn’t find an ITSP that offered a level of quality we considered to be acceptable. Very few ITSPs had any more experience with VoIP, SIP, and the internet than we did. More disturbing, however, was an almost complete lack of focus on quality and reliability. No process.

What we (quickly) discovered is the extremely low barrier to entry for ITSPs, especially back then. Virtually anyone could install Asterisk on a $100/mo box in a colo somewhere, buy dialtone from someone (who knows) and call themselves an ITSP. After going through several of these we discovered we needed to do it ourselves.

Even assuming we could solve the PSTN connectivity problem we discovered yet another issue. All of the monitoring and management in the world cannot make up for a terrible last mile. If the copper in the ground is rotting and the DSL modem can only negotiate 128kbps/128kbps that’s all you’re going to get. To make matters worse in the event of a cut or outage the customer would be down completely. While that may have always happened with the PSTN and an on premise PBX we considered this to be unacceptable.

So then, in the eleventh hour, just before launch I met with the original founders and posed a radical idea - scrap almost everything. There was a better way.
(Continued in Building a Startup (the right way))

Starting a Startup

2011-10-25T14:09:00.000-04:00

I know I’ve apologized for being quiet in the past. This is not one of those times because (as you’ll soon find out) I’ve been hard at work and only now can I finally talk about it.

Six years ago I was spending most of my time working with Asterisk and AstLinux. I spent a lot of time promoting both - working the conference circuit, blogging, magazines, books, etc. Conferences are a great way to network and meet new people. I did just that. With each conference I attended came new business opportunities. Sure, not all of them were a slam dunk and eventually I started to pick and chose which conferences I considered worthy of the time and investment.

For anyone involved with Asterisk Astricon is certainly worthy of your time and energy - the mecca of the Asterisk community. Astricon was always a whirlwind and 2005 was no exception. We were in Anaheim, California and embedded Asterisk was starting to really heat up. I announced my port of AstLinux to Gumstix and announced the “World’s Smallest PBX”, leading to an interview and story in LinuxDevices. I worked a free community booth (thanks Astricon) with Dave Taht and was introduced to Captain Crunch (that’s another post for another day).

It was at Astricon in 2005 that I also met one of my soon to be business partners (although I certainly didn’t know it at the time). While I was promoting embedded Asterisk and AstLinux I met a man from Florida named Joe Rhem. Joe had come up with the idea of using embedded Asterisk systems as the cornerstone of a new way to provide business grade telephone services. Joe and I met for a few minutes and discussed the merits of embedded Asterisk. Unfortunately (and everyone already knows this) I don’t remember meeting with Joe. Like I said Astricon was always a whirlwind and I had these conversations with dozens if not hundreds of people at each show. I made my way through Astricon, made a pit stop in Santa Clara for (the now defunct) ISPCon and then returned home to Lake Geneva, WI with a stack of business cards, a few new stories, and a lot of work to finish (or start, depending on your perspective).

A couple of months later I received an e-mail from Joe Rhem discussing how he’d like to move forward with what we discussed in Anaheim. Joe had recruited another partner to lead the new venture. Norm Worthington was a successful serial entrepreneur and his offer to lead the company was the equivalent of “having General Patton lead your war effort”. After some catch up I was intrigued with Joe’s idea. A few hours on the phone later everyone was pretty comfortable with how this could work.

Now I just needed to fly to Sarasota, FL (where’s that - sounds nice, I thought) to meet with everyone, discuss terms, plan a relocation, and (most importantly) start putting the company, product, and technology together.

A short time later I found myself arriving in Sarasota. It was early January and I coming from Wisconsin I couldn’t believe how nice it was. Looking back on it I’m sure Norm and Joe were very confident I’d be joining them in Sarasota. Working with technology I love “in paradise”, how could I resist?

(Continued in Building a Startup)

Breaking RFC compliance to improve monitoring

2010-10-26T17:39:00.005-04:00

A colleague came to me today and had a troubling issue. He's using sipsak and nagios to monitor some SIP endpoints. Pretty standard so far, right? He noticed that when using UDP and checking on an endpoint that was completely offline sipsak would take over 30 seconds to finally return with an error. Meanwhile Nagios would block and wait for sipsak to return...

Without a simple command line option in sipsak that appeared to change this behavior, we had to enter the semi-complicated world of SIP timers. I feared that to change this behavior we'd have to do some things that might not necessarily be RFC compliant...

What's this? For once I'm actually suggesting you do something against the better advice of an RFC?

That's right, I am.

RFC3261 defines multiple timers and timeouts for messages and transactions. It says things like:

"If there is no final response for the original request in 64*T1 seconds"

"The UAC core considers the INVITE transaction completed 64*T1 seconds after the reception of the first 2xx response."

"The 2xx response is passed to the transport with an interval that starts at T1 seconds and doubles for each retransmission until it reaches T2 seconds"

Without even knowing what "T1" is you can start to see that it's a pretty important timing parameter and (more or less) serves as the father of all timeouts in SIP. Let's look at section 17 to find out what T1 is:

"The default value for T1 is 500 ms. T1 is an estimate of the RTT between the client and server transactions. Elements MAY (though it is NOT RECOMMENDED) use smaller values of T1 within closed, private networks that do not permit general Internet connection. T1 MAY be chosen larger, and this is RECOMMENDED if it is known in advance (such as on high latency access links) that the RTT is larger. Whatever the value of T1, the exponential backoffs on retransmissions described in this section MUST be used."

T1 is essentially a variable for RTT between two endpoints that serves as a multiplier for other timeouts. Unless we know better T1 should default to 500ms, which is quite high. Some implementations (such as Asterisk with the SIP peer qualify option) automatically send OPTIONS requests to endpoints in an effort to better determine RTT instead of using the RFC default of 500ms.

In reading through the sipsak source code it appeared to be RFC compliant for timing, using a default T1 value of 500ms and a transaction timeout value of 64*T1. This is why it was taking over 30 seconds (32 seconds to be exact) for sipsak to finally timeout and return the status code to nagios. This comes directly from the RFC:

"For any transport, the client transaction MUST start timer B with a value of 64*T1 seconds (Timer B controls transaction timeouts)."

This is all well and good but what happens when you don't have a way to dynamically determine T1 and you can't wait T1*64 (32s) for your results like my sipsak/nagios check earlier? Simple: you go renegade, throw out the RFC, and hack the sipsak source yourself!

So I had three options:

1) Change the default value of T1.
2) Change the value of T2 by changing the multiplier or setting a static timeout.
3) Some combination of both.

I decided to go with option #3 (RFC be damned). Why?

1) 500ms is crazy high for most of our endpoints. At a glance 100ms would be fine for ~90% of them. I'll pick 150ms.
2) I don't need that many retransmits. If the latency and/or packet loss is that bad I'm not going to wait (my RTP certainly isn't) and I just want to know about it that much quicker.

So I ended up with a quick easy patch to sipsak:

diff -urN sipsak-0.9.6.orig/sipsak.h sipsak-0.9.6/sipsak.h
--- sipsak-0.9.6.orig/sipsak.h 2006-01-28 16:11:50.000000000 -0500
+++ sipsak-0.9.6/sipsak.h 2010-10-26 18:38:45.000000000 -0400
@@ -102,11 +102,7 @@
# define FQDN_SIZE 100
#endif

-#ifdef HAVE_CONFIG_H
-# define SIP_T1 DEFAULT_TIMEOUT
-#else
-# define SIP_T1 500
-#endif
+#define SIP_T1 150

#define SIP_T2 8*SIP_T1

diff -urN sipsak-0.9.6.orig/transport.c sipsak-0.9.6/transport.c
--- sipsak-0.9.6.orig/transport.c 2006-01-28 16:11:34.000000000 -0500
+++ sipsak-0.9.6/transport.c 2010-10-26 18:38:51.000000000 -0400
@@ -286,7 +286,7 @@
}
}
senddiff = deltaT(&(srt->starttime), &(srt->recvtime));
- if (senddiff > (float)64 * (float)SIP_T1) {
+ if (senddiff > inv_final) {
if (timing == 0) {
if (verbose>0)
printf("*** giving up, no final response after %.3f ms\n", senddiff);

This changes the value of T1 to 150ms (more reasonable for most networks) and allows you to specify the number of retransmits (and thus the total timeout) using -D on the sipsak command line:

kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D1 -v
** timeout after 150 ms**
*** giving up, no final response after 150.334 ms

kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D2 -v
** timeout after 150 ms**
** timeout after 300 ms**
*** giving up, no final response after 460.612 ms

kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D4 -v
** timeout after 150 ms**
** timeout after 300 ms**
** timeout after 600 ms**
*** giving up, no final response after 1071.137 ms

kkmac:sipsak-0.9.6-build kris$

Needless to say our monitoring situation is much improved.

A ClueCon Update

2010-08-05T11:56:00.004-04:00

Cluecon is going very well this year... I spoke the first day and have spent the rest of my time here enjoying the presentations and interacting with the community.

A few highlights:

Perfect wireless provided by Meraki. I've never been to a tech conference where the wifi has kept up with the crowd. Well done.
The Trump Tower. Phenomenal.
FreeSWITCH HA support in Sofia! This is worthy of its own post and it will have one when I get back and play with it. In the meantime my guy Jay Binks has been working to document this exciting new feature.
Chicago. I just LOVE this town.

More later... I've got to get back to the conference!

A ClueCon Preview...

2010-05-21T14:54:00.009-04:00

A while back I saw a preview for the new A-Team movie. While the movie itself looks horrible I was reminded of the original TV series with its many interesting characters and catch phrases. Among my personal favorites?

I love it when a plan comes together.

That's exactly how I feel with one of my "pet projects" from the past couple of months. Much like Hanibel and the A-Team I was up against formidable issues in trying to accomplish my task: implementing a flexible (very flexible), reasonably high performance LCR server that could be added to my existing architecture.

First I needed to select an LCR "engine". Multiple possibilities were considered but I left the final recommendation up to the DB and billing teams I work with. They selected mod_lcr from FreeSWITCH. While I was certain droute from OpenSIPS (or something similar) would have higher performance I accepted their recommendation. After playing with mod_lcr a bit I can also see its potential.

So now the question was: can FreeSWITCH respond with the proper SIP signaling (300 Multiple Choices)? Using the redirect application from mod_dptools it could not. I created a bounty to add multiple Contact/300 Multiple Choices functionality to FreeSWITCH. Tony had it implemented that day.

With the ability to respond properly I now had to get the data. Mod_lcr looked nice but it certainly wasn't designed for this application. All of the default syntax, tables, etc showed it being used with FreeSWITCH for FreeSWITCH. The tables and code used several bridge specific syntax examples. I hacked mod_lcr to return data to mod_dptools/redirect properly. A created a JIRA issue with my patch and a couple of days later Rupa had it committed.

So now FreeSWITCH could be a route server. All I needed to do was make sure OpenSIPS could route from what FreeSWITCH returned. Turns out it could not. RFC 3261 (section 21.3.1) states "...the SIP response MAY contain several Contact fields or a list of addresses in a Contact field." The Sofia stack from FreeSWITCH used multiple Contact headers, each with its own URI. OpenSIPS would only parse the first one returned. Sofia couldn't be changed easily so OpenSIPS would need to be changed (it was non-compliant anyway). Without this change there is no ability to handle multiple contacts and only the first would be used. It could be worse but obviously this wasn't good enough.

I contacted Bogdan from OpenSIPS to see what it would take to update the parser to handle multiple Contact headers. He indicated it would take four hours or so. Once he got back to me I had an OpenSIPS system that would handle multiple contact headers and create new branches from a failure route as desired.

So how did it all turn out? Well, you have two ways to hear the end of this story:

1) Attend ClueCon at the Trump Hotel in Chicago, IL in early August.
2) Wait until mid-August for an update here.

I'll make sure to post all of my materials - conference presentation, sipp scenarios for testing, OpenSIPS configuration, FreeSWITCH configuration, DB tweaks, etc.

Too late to make it to ClueCon this year? Just make sure to register next year, I'm sure I'll be there.

I've said it before but I'll say it again...

2010-05-19T18:47:00.005-04:00

FreeSWITCH rocks!

Earlier today I wanted to play with the possibility of using FreeSWITCH as a route/LCR server for another platform. FreeSWITCH already has mod_lcr and redirect. Using these two features FreeSWITCH could be made to respond with a 302 and a single SIP URI in the Contact field.

I wanted more. I wanted a way to respond with multiple routes.

The standard way to do this (using SIP, of course) is to respond to incoming INVITEs with a 300 Multiple Choices. This response should contain a Contact header (or multiple Contact headers) with a list of SIP URIs (along with optional q values, etc) for the original system to route the call to.

As usual I wrote the FreeSWITCH-Users mailing list to make sure this functionality didn't already exist somewhere. It did not and it was suggested I create a bounty.

Creating a bounty is always tough... I don't deal with the source code of FreeSWITCH all that often. I don't know how much work this is going to take. I don't know how much C programmers make. So I did my best to come up with something that seemed fair: $250.

Less than two hours later the feature was coded, committed to FreeSWITCH, tested by me, and paid for.

Once again, Open Source for the win!

Another SIP gotcha: Cisco

2010-05-12T11:27:00.005-04:00

Another quick and dirty SIP interop post.

A while back I was tasked to interface a FreeSWITCH server and a Cisco Unified Communications Manager system. Once the SIP trunk was configured on the Call Manager/CUCM side they sent an INVITE over. It didn't have an SDP.

It appeared that we needed to enable 3pcc (third party call control) in FreeSWITCH. No problem. I enabled 3pcc and interop continued.

Problems arose, however, when we needed to send the Cisco ringback. Whether it be a 180 or 183 (with or without SDP for either) this was going to be tough because with 3pcc enabled the dialog looked like so:

<-- Cisco
--> FreeSWITCH
INVITE (without SDP) <--
100 Trying -->
200 OK (with SDP) -->
ACK (with SDP) <--

So... There was no opportunity to signal progress as long as we 200 OKd the call almost immediately. Sure I probably could generate some ringback after the 200 but that would just be wrong!

As I like to say, the internet to the rescue. Not having much experience with CUCM I thought I'd ask on VoiceOps. Within a few minutes a very nice gentlemen by the name of Mark Holloway mentioned "Media Termination Point Required" as a CUCM configuration option. These were the magic words. After some research it turned out that was the configuration option I needed*. Thanks Mark!

Once "Media Termination Point Required" was enabled on the Cisco side I disabled 3pcc in FreeSWITCH and all was good. Users even get ringback now!

I also brought the issue up on the FreeSWITCH-Users mailing list and found out this has been bothering people for some time. MC from FreeSWITCH was even nice enough to start a wiki page for me to document all of this there.

Sometimes with SIP it's all about the SIMPLE achievements ;).

* That research also brought up another possibility: enabling PRACK/100rel on the CallManager side instead of "MTP Required". Of course the trouble with PRACK is there are a lot of SIP implementations (Asterisk) that don't support it. FreeSWITCH does but can crash. Many SIP implementations don't support the default CUCM configuration (INVITE w/o SDP). I was looking for the most canonical, compatible configuration possible.

(High Quality) VoIP on the iPhone

2010-02-05T12:01:00.005-05:00

(Regular readers will note that my excessive use of parentheses has now spilled into my titles and first sentences)!

Ahhh Apple... Ahh the iPhone. Regardless of how you feel about this company or their product you can't doubt the market impact they've made over the last couple of years (decades perhaps?). Multitouch (NOT multitasking). App Store. iTunes. There are countless other blogs that discuss these topics so I don't need to. As usual I'm here to talk about VoIP.

For the last ten months or so I've been involved (part time) in another local venture. Voalte (pronounced volt) is a startup here in Sarasota, FL founded by Trey Lauderdale. When the Apple iPhone was announced Trey was working in sales for Emergin, a healthcare IT middleware provider. Trey noticed how incredibly arcane the mobile devices used in healthcare are when compared to this new device from Apple. Once the iPhone SDK was announced Trey knew he had to develop an iPhone application for healthcare. This application became Voalte One.

Voalte One is an iPhone application that provides voice, alarms, and text for healthcare point of care providers (that's nurses to you and I). The complete Voalte One solution is comprised of the following parts:

- iPhone
- Voalte Server (XMPP, LDAP, etc)
- Voalte Voice Server (FreeSWITCH using SIP + event socket)
- An overall excellent customer/user experience (also new to healthcare)

Text messaging and alarm integration are cool but as I've already said, I do VoIP. If you'd like to know more about iPhone development, XMPP, LDAP, etc let me know and I can point you in the right direction.

VoIP in our application is interesting. It's a softphone, technically, but unlike one you've ever seen before. As everyone knows the iPhone cannot run multiple applications. It can't background applications. These are just two of the many challenges introduced when developing a user-friendly, always available, reliable non-GSM phone experience for the iPhone. Simply downloading an off the shelf softphone and installing it on the iPhone is not enough.

We're a startup and we get to do cool things. For example, one of the big differences between VoIP/voice with Voalte One on the iPhone is the voice quality. We use G.722 wideband at 16kHz as our standard voice codec. Why? Because one Saturday (after a long night out) Trey and I were having lunch. I asked him if he thought we should set ourselves apart on something as basic as sample rate. After a little explanation on my part we quickly decided - why not?

As cool as G.722 is it introduces some interesting challenges:

- The iPhone. How are we going to get 16kHz audio from the hardware?
- PJSIP (our SIP stack). Does it support G.722? How does it interface with the audio hardware?
- Hospital PBXs. Voalte One interfaces with the hospital PBX as an ordinary extension. Most of them probably don't support G.722. How/where do we resample to the standard 8kHz used in G.711?

After looking through PJSIP and the available audio drivers for the iPhone we decided we needed to write our own. There were legal and technical reasons and I'm glad we did it. Especially because I didn't have to do most of the work! ;) We also confirmed PJSIP supports G.722.

Voalte has an amazing iPhone developer - Robbie Hanson. Robbie, Ben (Voalte CTO), and I were able to look over the available audio frameworks on the iPhone and pick the best. Not only is it the best overall (it supports echo cancellation, etc) it would provide us the sampling rate of 16kHz we knew we needed.

After working with PJSIP and AudioUnit for a while Robbie was able to write an iPhone audio driver (using AudioUnit, of course) for PJSIP. While working on the audio driver Robbie (along with another contributor) also wrote an Objective C wrapper for PJSIP. These are the raw ingredients of a high quality VoIP experience on the iPhone.

In the months leading up to release we had to deal with a plethora of other issues: push notifications, local ringback, wifi, etc, etc. I won't (and probably can't) describe these issues in detail.

The good news is Voalte has done the right thing and released the core components of this solution as open source.

I'm proud to work with companies that "get it" and are willing to actively participate in the free software ecosystem.

Upcoming Review: Building Telephony Systems with OpenSIPS 1.6

2010-02-02T10:39:00.003-05:00

Packt Publishing has once again asked me to review their latest work in the OpenSIPS series: Building Telephony Systems with OpenSIPS 1.6.

My review of the previous edition goes all the way back to when OpenSIPS was called OpenSER. I have another post discussing that topic...

Anyways, I should be receiving the book this week and I should have a review up by next week.

AstLinux 0.7 released and more!

2010-01-20T11:37:00.003-05:00

The AstLinux team (of which I'm an occasional member) has released AstLinux 0.7. Darrick, Philip, Lonnie, and the rest of the community have done a great job getting this release out there. I couldn't be happier with how my little project has grown up!

In addition to getting this release out, they've also taken the time to focus on documentation and a new website.

Well done guys!

Testing with SIPP

2010-01-20T11:34:00.002-05:00

A quick one, I promise...

I'd been having some issues testing Asterisk with sipp. It turns out there is a fairly well known issue with sipp when using five digit port numbers for RTP. A quick Google search found a solution pretty quickly.

Just in case that link ever goes dead, here's the diff:

diff -urb sipp.svn_orig/call.cpp sipp.svn_fixed/call.cpp
--- sipp.svn_orig/call.cpp 2008-12-19 13:14:51.000000000 +0300
+++ sipp.svn_fixed/call.cpp 2008-12-19 13:16:34.000000000 +0300
@@ -192,7 +192,7 @@
/* m=audio not found */
return 0;
}
- begin += strlen(pattern) - 1;
+ begin += strlen(pattern);
end = strstr(begin, "\r\n");
if (!end)
ERROR("get_remote_port_media: no CRLF found");

More on sipp later!

I don't "do" testimonials

2009-10-09T15:44:00.008-04:00

I was (admittedly) getting a little bored at the office today when I realized that one of my favorite activities to pass time is checking the recent changes page of the FreeSWITCH wiki. Think about that for a minute...

If that doesn't make me a FreeSWITCH fanboy, I don't know what does. I guess I just have to finally admit it:

I am a FreeSWITCH fanboy.

How I got here I'm not really quite sure. What I do know is that I despise fanboys of all sorts (in no particular order):

- Linux
- Apple
- Microsoft (how?)
- Asterisk
- FreeSWITCH (self-loathing and denial are very powerful forces)

I've been using FreeSWITCH for over a year and it's the closest thing to a whirlwind romance I've ever had (nerdom solidified, thank you). While I'm sure the honeymoon period will end eventually at the moment I'm still smitten. I sat down and started to write a testimonial on the FreeSWITCH wiki when I realized that this was the perfect opportunity to return to blogging. I'm back and I want to talk about FreeSWITCH. Here's what I have to say (complete with an obligatory car analogy):

Star2Star has over 12,000 (business) endpoints in the field and we currently use FreeSWITCH for:

- SBC (multiple applications - customer and provider facing)
- Conference services

We're very quickly moving to IVR and (ultimately) voicemail.

What is most impressive about FreeSWITCH are the small details that make working with it an absolute joy:

- Sofia profiles (multiple unique SIP identities Just. Like. That.)
- XML flexibility (static, custom module, mod_xml_curl)
- Dialplan (conditions, PCRE, etc)
- Channel variables (access to remote media IP address and anything else in a CHANNEL VARIABLE!)
- proxy_media
- bypass_media (beautiful, just beautiful)
- RICH SIP/RTP support (SIPS, TCP, UDP, SCTP, SRTP, PAI, RPID). Yeah, I can talk to anything any way I need to.)
- Codec support. Resampling. Repacketization. It's just getting ridiculous.
- Event socket. Inbound/outbound. Async. API commands. Events.

It is still *amazing* to me that I can make FreeSWITCH into the ultimate signaling-only SBC just by adding bypass_media=true or that I can send PAI to a destination with sip_cid_type=pid. Just about every component of FreeSWITCH I've come across is very well thought out, completely logical, and completely predictable. It's like driving a German car - if you take driving seriously everything just works and everything just works exactly as you expect it to.

It's as if Anthony, Brian, Michael, and everyone else on the FreeSWITCH team are constantly sitting around, agonizing every detail of this awesome piece of software just as Hanz, Fritz, and Karl are in Munich agonizing over every detail for the next BMW 7 Series. While there are several differences in this analogy perhaps the biggest difference is that it doesn't cost $100,000 for one of the most well engineered products I've ever come across (obviously I'm not talking about the BMW).

Did I mention that in addition to this rich feature support it actually scales? All the features in the world are completely useless to me in my environment unless they can scale. We've seen again and again - FreeSWITCH scales. With OpenSER/OpenSIPS at our core and FreeSWITCH serving various edge and endpoint roles I can't identify a single point where we couldn't (ultimately) scale to meet growth or demand.

Star2Star is very unique. We have a unique product, unique approach, and unique architecture. While the architecture has generally evolved and scaled over the years with the number of users, the introduction of FreeSWITCH in a given role has always been accompanied by an excitement for what we'll be able to do for our company and our customers with our new favorite piece of software. It's a very exciting time.

The future at Star2Star is powered by our (quickly) growing customer base, FreeSWITCH, and our efforts to relentlessly deploy it where we can.