Let’s say you have a CPE with two ports, and you’re connecting a pair of this CPE to your RCG-4001. Further, let’s say your CPE supports SIP, MGCP, and H.248, and that you support a flavor of MGCP that is “native” all the way back to a softswitch, and another that is gatewayed over a GR-303 link back to a switch (so when we say “GR-303″ we mean this flavor of MGCP). The picture below shows this pair of CPEs with one port set up for each protocol type:

On the RCG, the config file that maps a physical port to the named port you use in a script is called portmap.conf. You can have multiple mappings configured in the file, each with a unique name. And you can choose between which portmap is active when you launch a test suite. Here’s a possible portmap for the RCG shown above:

[DEFAULT]
port_map = {'A': 'port0', 'C': 'port1', 'B': 'port2', 'D': 'port3'}

[sip-mgcp]
port_map = {'A': 'port0', 'B': 'port1', 'C': 'port2', 'D': 'port3'}

[sip-h248]
port_map = {'A': 'port0', 'C': 'port1', 'B': 'port2', 'D': 'port3'}

[sip-gr303]
port_map = {'A': 'port0', 'D': 'port1', 'C': 'port2', 'B': 'port3'}

[mgcp-sip]
port_map = {'B': 'port0', 'A': 'port1', 'C': 'port2', 'D': 'port3'}

You could list all of the possible combinations — this is enough for this example. The scripts that ship with the RCG refer to ports as “A”, “B”, “C”, “D”. The “a_calls_b” script places a call from “port A” to “port B”; whichever physical ports map to A and B are the ones that will be used by the test. You’ll notice that we’ve provided a “DEFAULT” mapping — this is required. We’ve chosen to make the “sip-mgcp” mapping the same so that we have a convenient mnemonic that refers to this mapping.

In our most recent release, the test suites take a “–portmap=” argument that allows you to select the portmap. For example:

bash$ a_calls_b.py --portmap=sip-h248 a_calls_b$

This will place a call from physical port 0 to physical port 2 (the SIP port on CPE1 to the H.248 port on CPE2).

In our upcoming release, you can list multiple portmaps on the command line. For example:

bash$ a_calls_b.py --portmap=sip-mgcp,sip-h248-sip-gr303,mgcp-sip a_calls_b$

This will run the script four times, once each using each listed portmap. You can run a custom test suite with this argument, and the suite will be run in the various mappings. You can kick off an three hour long test suite with ten different portmaps on Friday evening and not have to worry about relaunching the test, touching config files, or changing cables every few hours over the weekend.

We’ve talked internally about how we might automatically generate all of the relevant maps. How would you like to see this work? Leave a comment below or drop us a note.

An RCG test suite is simply a python script that calls “main()” on a ScriptRunner object. Here’s a short example suite that will run two tests:

#!/usr/bin/python

from rcg.test_runner import ScriptRunner
import a_calls_b

TESTS = [
    a_calls_b.a_calls_b,
    a_calls_b.ab_mesh,
    ]

if __name__ == '__main__':
    main()

Let’s break that down for readers less familiar with python:

The first line tells the OS we want to run this program using the python interpreter:
#!/usr/bin/python

The next two lines tells python that we want to use two existing modules, both of which in this case come from the RCG libraries. The first line here makes the ScriptRunner class available, and the second line lets us use the “a_calls_b” module, which contains several test scripts related to two-party calls.

from rcg.test_runner import ScriptRunner
import a_calls_b

Then we create a list of two test scripts from the a_calls_b module, which we store in a variable called TESTS. We prefer to format the code this way so that it is easy to temporarily comment out one or more of the tests in a suite (with a # character at the beginning of the line) if we want to quickly change the suite definition for a test run.

TESTS = [
    a_calls_b.a_calls_b,
    a_calls_b.ab_mesh,
    ]

Then we create a ScriptRunner object using that list of test scripts. The ScriptRunner will run those tests when we call it (see below).

main = ScriptRunner(TESTS)
main.title = 'My Suite'

Finally, this checks the built-in python variable “__name__” to see if this file is being run as a standalone program. If it is, we run the ScriptRunner — the ScriptRunner class defines a special method that allows ScriptRunner objects to be “called” as if they were functions. By including the check to see if the file is run as a standalone program, we allow for the case where this suite is imported by a larger suite. Defining suites in this way allows us to build up collections of test suites hierarchically in a way that makes sense for a particular application. If you look at a_calls_b.py, you will see that it is a suite defined using the same basic template provided here.

if __name__ == '__main__':
    main()

Then we should plan for the inevitable failure and make sure we are going to gather as much information as possible. When you are armed with all the right info, you will be able to find the root cause and develop a fix quickly and easily. Otherwise testers and developers wind up in an endless cycle of “can you retest it and capture X?” Half the battle — usually more than half — is simply reproducing the problem and figuring out what is really broken.

We’ve built a great feature into the RCG-4001. It captures, for every test:

• An audio recording, in a .wav file, of what each POTS port on the RCG “hears” during the test.
• A capture of all of the network traffic seen at the RCG’s network interface in a .pcap (wireshark compatible) file.
• Syslog messages from the DUT — you just have to configure the DUT to send syslog messages to the RCG.
• A log of all of the RCG’s actions — debug information from the test scripts.
• The RCG config files that were in effect for the test — so you can easily reproduce the failure with the exact same configuration.
• The source of the script that was running — so you can easily reproduce the failure with the exact same actions.

All of those pieces get put into a ZIP file and saved on the RCG so you can easily copy it into your bug database when you find a broken call flow.

This feature is awesome. That’s not just coming from me — our feedback so far has been enthusiastic. Once you’ve used it, you’ll wonder why you spent so much time in the dark ages.

Here’s a simple but very usable setup that enables the use of this feature:

In this diagram, the DUT is connected to the softswitch through a managed ethernet switch. An old-school hub would work just as well — the point is that the RCG should be able to sniff the VoIP traffic. If a switch is used, configure it so that it mirrors all of the traffic going to and from the DUT to the RCG’s port. This also provides an ethernet connection between the DUT and the RCG for syslog traffic, if your DUT supports it.

And then when something breaks you get a pretty picture like this:

Whoops! The device is sending *67 without a DN and the switch doesn’t like it. We can give the ZIP file containing this to the development team, and they’ll be able to get a fix without needing to ask for more information.

bash$ a_calls_b.py
a_calls_b
a_calls_bcd
a_calls_b_hangup
a_calls_b_ring_timeout
a_calls_b_anon_cid
ERROR: caller id string match error. Received 'anonymous', looking for 'Anonymous'. cid_mode=cid_anonymous
run_all_tests:17
 inner:45
  a_calls_b_anon_cid:118
   call:292
    place_call:353
     _check_cid:508
a_calls_b_terminator_hangup

Ok, that’s a pretty obvious failure — our expected CID string should have been all lowercase. No need to modify the script, just edit /etc/rimay/callgen.conf. Find this line:


[DEFAULT]
caller_id_anonymous_call_string = Anonymous


and change it to this:


[DEFAULT]
caller_id_anonymous_call_string = anonymous


Rerun the test:


bash$ a_calls_b.py a_calls_b_anon_cid
a_calls_b_anon_cid
bash$


Test passes!

This is in keeping with the overall strategy of keeping switch-specific details and behavior in the config file instead of in the scripts. As we can see here, it makes porting to a new interop scenario much easier.

For example, when we take a phone port off-hook, we sometimes have to be careful to avoid immediately performing operations on that port. It needs a slight delay when we’re on a VoIP system to give the RTP stream a chance to start. Otherwise if we slam it with DTMF right away we’ll lose a digit or two. On a pure POTS system, we can reduce the delay.

Sprinkling these delays throughout the script libraries would be a nightmare. When we port to a new deployment type, we’d have to hunt down all of the places with delays and tweak them individually — and then somehow maintain the two versions.

Instead we maintain the delays in a configuration file on the RCG-4001. The different delays are available as properties on the PhonePort objects used by the scripts in our suite. When we need to port to a new switch, we can create a new config file and tweak the delays in one place. This avoids tedious searching through scripts and keeps all of the tuning centralized.

• physical (e.g. unplugged cable)
• misconfiguration (e.g. feature not enabled)
• software defect (e.g. mishandled scenario)
• protocol exchange (e.g. interpreting a field value differently)
• network / transport (e.g. firewall)

The early stages of troubleshooting should try to eliminate some of these broad categories as causes (or find the cause along the way, whichever happens first).

First, and usually easiest, is to make sure it is not a network connectivity problem. Ping is your friend. If you can’t ping the equipment involved in the call, you may have a physical problem. Are the cables plugged in? Is everything powered on?

Does any voice work between the two endpoints? If you’re running automated tests, is your call generator reporting failures for every test in the suite or just for a particular feature? Quick A->B calls between all involved parties proves a lot. Do other features work, especially similar features and those that exercise the same lines and features? If you can’t make a simple two way voice call between each of the endpoint pairings involved, there’s a more fundamental problem — first troubleshoot the simpler version of the problem before tackling the more complex feature. If basic calls work but not advanced features, ensure the failing features are enabled on the switch and activated for the lines in question.

When you have eliminated network and physical problems as the cause, check the configuration. Does the feature work on any lines? Do logs indicate errors like “permission denied” or “feature unavailable”? Do you get error tones or announcement messages? If the feature works on other lines, compare the config of the working lines to the non-working lines.

Once you’re sure that neither of those categories are the culprit, it’s time for deeper digging. If you have access to the entire transport network, a network traffic capture can be very useful. (Even if you only have visibility into part of the network, captures can still help with troubleshooting but there are more limits to what you can uncover.) If at all feasible, it’s best to capture all traffic, not just the protocol you’re expecting to troubleshoot (e.g. SIP or MGCP). I’ve seen failures caused by a misconfigured router that was sending ICMP redirects all over the place. This won’t show up in a filtered capture.

Armed with a traffic capture, start looking for the unexpected. ARP storms, loads of ICMP redirects, and rapid SIP reregistrations are all things I’ve seen as symptoms that point to the real problem. (These were mostly misconfigs that weren’t caught in the previous troubleshooting step.)

In the absence of anything unusual, look at the traffic that makes up the call trace. Filtering is useful here, as are tools that generate a graphical view of the call flow. First look for error packets. If present, these usually have a specific enough error code or even a text explanation of why the call is failing. If there are no error packets, check that the call is being set up as expected. Is audio fully enabled or is an endpoint simply idled? Check for voice path — is RTP flowing in both directions as expected? Does the RTP payload match what is expected (and do both directions match each other)? (If not RTP, then whatever equivalent protocol is in your trace.)

When signaling and RTP look good, it is often useful to extract the audio streams. In wireshark, you can choose to extract either direction or both directions. I usually start with both directions, and then listen on my PC’s speakers. Problems with audio here will point you to one end of the other. Often I have to backtrack to examining configs or traffic for that end of the connection.

At this point I’ve usually found the cause of the problem. If not, it’s time to ask for help. I’ve gotten help from coworkers, partner/vendor tech support, and occasionally partner/vendor developers. Another set of eyes looking at the problem is helpful. Sometimes just explaining the issue and symptoms to a new person will trigger an “Aha!” moment.

Do you have any troubleshooting techniques I’ve missed? Leave a comment. Thanks!

Let’s outline what the test scripts for speed dial 8 would look like:

Once again we follow the basic strategy for call tests:

1. We will use lines A, B, C, and D.
2. A will be the originator and the line that uses speed dial.
3. The happy path for using speed dial is A dials 2#, where 2 is configured to call B, B rings, and A-B have conversation. The happy path for configuring a number is A dials *74, waits for the prompt, enters 2 to set a number for speed dial 2, waits for the prompt (usually stutter tone, could be an announcement), dials the phone number for B, and (sometimes, but not always required) dials # to indicate the end of the phone number. These will be similar for calls to C and D.
4. The first flow — using speed dial — is as easy as it gets. Just dial “2#” using CallGenerator‘s call() method. The second flow requires waiting for prompts. Be careful here not to embed switch-specific delays into your script. The RCG-4001 uses a global configuration file to collect all of the delays in one place so that you can easily port your scripts to another switch by changing the delays in the config file. Your script will dial “*74″, wait, dial “2#”, wait, dial the number for B, dial # if required by the switch, wait, hang up.
5. Variants would be unconfiguring a speed dial number, configuring 7, 10, 1+10-digit, and international numbers, filling all the speed dial slots, reconfiguring a used slot, dialing using an empty slot, and dialing using each slot when they are all configured.
6. Edge cases include using an invalid speed dial 8 slot (0, 1, 20, etc), speed dialing an unconfigured slot, entering too many digits for the dialed phone number, too few digits, letting a timeout occur at the dialed number or slot prompt, or hanging up at the dialed number or slot prompt.
7. A crazy variant would be to configure a VSC as a speed dial number. For example, *70 (cancel call waiting). Maybe this is supposed to be allowed, maybe it’s not, check your spec. If it’s allowed, can you configure a speed dial that unconfigures itself? Another crazy variant is to use speed dial when dialing the third leg of a three way call. (This should be allowed by the switch.)

But running this test suite regularly means that you’re not finding many bugs any more. This is good news (the product is better) and bad news (there are probably still more bugs to find). The really bad news is that the bugs that are left can be really hard to find, and often require stepping outside your current test framework.

We’ve talked about forcing errors from the call generator, and making sure that the DUT can handle them — repeatedly. This builds on that practice and expands it beyond the call generator and into your test network.

What’s the Worst Thing That Could Happen?

This is the question to ask.

1. What is the worst thing that could happen internally? Perhaps a memory leak that uses up all available memory. Or the internal storage (disk, flash) fills up. What if a fan fails and the heat builds up?
2. What is the worst thing that could happen externally? Loss of connection to the network. Power failure. Network storm.
3. What could a malicious person do? Send a flood. Send specially crafted packets.

Now take the answers to those questions, and make the worst thing happen.

1. If you’re testing a VoIP gateway, disconnect it from the network mid-call.
2. Take that VoIP gateway and run a longevity test. Now run a script that toggles the port on the network switch on-off every 2 minutes. (Don’t have a managed switch? Get a managed power supply and just toggle power to your hub.)
3. Using the same gateway and longevity test, this time use your managed power supply to kill power to the gateway at random intervals.
4. Use a network traffic generator to simulate a network storm.
5. Write scripts to generate some of those “specially crafted packets” that might do bad things.
6. Cheat if you can: examine the code and look for error paths. Do whatever it takes to hit those error paths. (Hint: this is what malicious people do.)

Go do your worst.

Sometimes a feature is broken in a way that the call generator can’t detect right away. For instance, let’s say the device under test (DUT) leaks memory with a given call flow. The call will work hundreds or thousands of times, but at some point the entire DUT may stop working — and the cause won’t be obvious.

You want the DUT to melt in your lab, not on your customer. Let’s look at three types of robustness testing.

Load Testing

Load testing is typically performed by bulk call generators with many ports. They specialize in generating high call volume to place a high load on the DUT and verify that it is robust under heavy load. For lab testing, a call volume beyond the rated capacity should be used to verify that the DUT degrades gracefully (i.e. does not crash) under stress. If you are testing a small CPE like a one or two port ATA, then a small port-count call generator can do the job.

Much of the load testing I’ve seen at various companies consists of simple call flows and does not probe the behavior of the DUT with more “interesting” calls under load. This is a missed opportunity for finding bugs. Load tests should include a variety of call flows.

Longevity Testing

With longevity testing we want to find out if the DUT can survive a large number of calls that occur at a reasonable rate. For general longevity testing you may want to randomize the test suite so that — as described above — “interesting” sets of call flows occur throughout the test.

The point of longevity testing is not to overload the DUT but rather to subject the DUT to a reasonable load over a long period of time. (Where “reasonable” is defined for the DUT according to the device’s performance goals.)

The end goal is to find defects that occur after the DUT has processed a huge number of calls over a period of days or even weeks. Again, too often longevity testing uses a single, simple call flow. It makes sense to perform longevity testing of happy-path calls, but every once in a while it’s good to mix it up and make sure all of the boundary cases are handled well.

Repeated Forced Errors

As someone who has written and read thousands of lines of code I know that the best place to find bugs is in the dark corners of the error handling code. It doesn’t get exercised often (if at all), so it’s likely that it is not quite right.

As mentioned above, these failures may not be apparent right away. It may take large numbers of iterations before they rear their head, and then the symptoms may not point to the defect. You can make it easier to diagnose the failure by repeating a test that forces a particular error to occur.

“Errors” too can include multiple classes of errors, including some that can’t be induced at the FXO interface. You may have to create some clever test scripts.

For example, let’s say the DUT is an ATA. Connect the ATA’s analog line to the RCG-4001, and provision the ATA to use the RCG as a switch. On the RCG, your script is the SIP proxy, but it does not always behave properly. Perhaps it drops contact with the ATA, or maybe it returns a 4xx or 5xx response at some point in the flow. If you run this test repeatedly you may find that your production, commercial-quality stack is not as robust to these errors as it should be.