bash$ a_calls_b.py
a_calls_b
a_calls_bcd
a_calls_b_hangup
a_calls_b_ring_timeout
a_calls_b_anon_cid
ERROR: caller id string match error. Received 'anonymous', looking for 'Anonymous'. cid_mode=cid_anonymous
run_all_tests:17
 inner:45
  a_calls_b_anon_cid:118
   call:292
    place_call:353
     _check_cid:508
a_calls_b_terminator_hangup

Ok, that’s a pretty obvious failure — our expected CID string should have been all lowercase. No need to modify the script, just edit /etc/rimay/callgen.conf. Find this line:


[DEFAULT]
caller_id_anonymous_call_string = Anonymous


and change it to this:


[DEFAULT]
caller_id_anonymous_call_string = anonymous


Rerun the test:


bash$ a_calls_b.py a_calls_b_anon_cid
a_calls_b_anon_cid
bash$


Test passes!

This is in keeping with the overall strategy of keeping switch-specific details and behavior in the config file instead of in the scripts. As we can see here, it makes porting to a new interop scenario much easier.

For example, when we take a phone port off-hook, we sometimes have to be careful to avoid immediately performing operations on that port. It needs a slight delay when we’re on a VoIP system to give the RTP stream a chance to start. Otherwise if we slam it with DTMF right away we’ll lose a digit or two. On a pure POTS system, we can reduce the delay.

Sprinkling these delays throughout the script libraries would be a nightmare. When we port to a new deployment type, we’d have to hunt down all of the places with delays and tweak them individually — and then somehow maintain the two versions.

Instead we maintain the delays in a configuration file on the RCG-4001. The different delays are available as properties on the PhonePort objects used by the scripts in our suite. When we need to port to a new switch, we can create a new config file and tweak the delays in one place. This avoids tedious searching through scripts and keeps all of the tuning centralized.

• physical (e.g. unplugged cable)
• misconfiguration (e.g. feature not enabled)
• software defect (e.g. mishandled scenario)
• protocol exchange (e.g. interpreting a field value differently)
• network / transport (e.g. firewall)

The early stages of troubleshooting should try to eliminate some of these broad categories as causes (or find the cause along the way, whichever happens first).

First, and usually easiest, is to make sure it is not a network connectivity problem. Ping is your friend. If you can’t ping the equipment involved in the call, you may have a physical problem. Are the cables plugged in? Is everything powered on?

Does any voice work between the two endpoints? If you’re running automated tests, is your call generator reporting failures for every test in the suite or just for a particular feature? Quick A->B calls between all involved parties proves a lot. Do other features work, especially similar features and those that exercise the same lines and features? If you can’t make a simple two way voice call between each of the endpoint pairings involved, there’s a more fundamental problem — first troubleshoot the simpler version of the problem before tackling the more complex feature. If basic calls work but not advanced features, ensure the failing features are enabled on the switch and activated for the lines in question.

When you have eliminated network and physical problems as the cause, check the configuration. Does the feature work on any lines? Do logs indicate errors like “permission denied” or “feature unavailable”? Do you get error tones or announcement messages? If the feature works on other lines, compare the config of the working lines to the non-working lines.

Once you’re sure that neither of those categories are the culprit, it’s time for deeper digging. If you have access to the entire transport network, a network traffic capture can be very useful. (Even if you only have visibility into part of the network, captures can still help with troubleshooting but there are more limits to what you can uncover.) If at all feasible, it’s best to capture all traffic, not just the protocol you’re expecting to troubleshoot (e.g. SIP or MGCP). I’ve seen failures caused by a misconfigured router that was sending ICMP redirects all over the place. This won’t show up in a filtered capture.

Armed with a traffic capture, start looking for the unexpected. ARP storms, loads of ICMP redirects, and rapid SIP reregistrations are all things I’ve seen as symptoms that point to the real problem. (These were mostly misconfigs that weren’t caught in the previous troubleshooting step.)

In the absence of anything unusual, look at the traffic that makes up the call trace. Filtering is useful here, as are tools that generate a graphical view of the call flow. First look for error packets. If present, these usually have a specific enough error code or even a text explanation of why the call is failing. If there are no error packets, check that the call is being set up as expected. Is audio fully enabled or is an endpoint simply idled? Check for voice path — is RTP flowing in both directions as expected? Does the RTP payload match what is expected (and do both directions match each other)? (If not RTP, then whatever equivalent protocol is in your trace.)

When signaling and RTP look good, it is often useful to extract the audio streams. In wireshark, you can choose to extract either direction or both directions. I usually start with both directions, and then listen on my PC’s speakers. Problems with audio here will point you to one end of the other. Often I have to backtrack to examining configs or traffic for that end of the connection.

At this point I’ve usually found the cause of the problem. If not, it’s time to ask for help. I’ve gotten help from coworkers, partner/vendor tech support, and occasionally partner/vendor developers. Another set of eyes looking at the problem is helpful. Sometimes just explaining the issue and symptoms to a new person will trigger an “Aha!” moment.

Do you have any troubleshooting techniques I’ve missed? Leave a comment. Thanks!

Let’s outline what the test scripts for speed dial 8 would look like:

Once again we follow the basic strategy for call tests:

1. We will use lines A, B, C, and D.
2. A will be the originator and the line that uses speed dial.
3. The happy path for using speed dial is A dials 2#, where 2 is configured to call B, B rings, and A-B have conversation. The happy path for configuring a number is A dials *74, waits for the prompt, enters 2 to set a number for speed dial 2, waits for the prompt (usually stutter tone, could be an announcement), dials the phone number for B, and (sometimes, but not always required) dials # to indicate the end of the phone number. These will be similar for calls to C and D.
4. The first flow — using speed dial — is as easy as it gets. Just dial “2#” using CallGenerator’s call() method. The second flow requires waiting for prompts. Be careful here not to embed switch-specific delays into your script. The RCG-4001 uses a global configuration file to collect all of the delays in one place so that you can easily port your scripts to another switch by changing the delays in the config file. Your script will dial “*74″, wait, dial “2#”, wait, dial the number for B, dial # if required by the switch, wait, hang up.
5. Variants would be unconfiguring a speed dial number, configuring 7, 10, 1+10-digit, and international numbers, filling all the speed dial slots, reconfiguring a used slot, dialing using an empty slot, and dialing using each slot when they are all configured.
6. Edge cases include using an invalid speed dial 8 slot (0, 1, 20, etc), speed dialing an unconfigured slot, entering too many digits for the dialed phone number, too few digits, letting a timeout occur at the dialed number or slot prompt, or hanging up at the dialed number or slot prompt.
7. A crazy variant would be to configure a VSC as a speed dial number. For example, *70 (cancel call waiting). Maybe this is supposed to be allowed, maybe it’s not, check your spec. If it’s allowed, can you configure a speed dial that unconfigures itself? Another crazy variant is to use speed dial when dialing the third leg of a three way call. (This should be allowed by the switch.)

But running this test suite regularly means that you’re not finding many bugs any more. This is good news (the product is better) and bad news (there are probably still more bugs to find). The really bad news is that the bugs that are left can be really hard to find, and often require stepping outside your current test framework.

We’ve talked about forcing errors from the call generator, and making sure that the DUT can handle them — repeatedly. This builds on that practice and expands it beyond the call generator and into your test network.

What’s the Worst Thing That Could Happen?

This is the question to ask.

1. What is the worst thing that could happen internally? Perhaps a memory leak that uses up all available memory. Or the internal storage (disk, flash) fills up. What if a fan fails and the heat builds up?
2. What is the worst thing that could happen externally? Loss of connection to the network. Power failure. Network storm.
3. What could a malicious person do? Send a flood. Send specially crafted packets.

Now take the answers to those questions, and make the worst thing happen.

1. If you’re testing a VoIP gateway, disconnect it from the network mid-call.
2. Take that VoIP gateway and run a longevity test. Now run a script that toggles the port on the network switch on-off every 2 minutes. (Don’t have a managed switch? Get a managed power supply and just toggle power to your hub.)
3. Using the same gateway and longevity test, this time use your managed power supply to kill power to the gateway at random intervals.
4. Use a network traffic generator to simulate a network storm.
5. Write scripts to generate some of those “specially crafted packets” that might do bad things.
6. Cheat if you can: examine the code and look for error paths. Do whatever it takes to hit those error paths. (Hint: this is what malicious people do.)

Go do your worst.

Sometimes a feature is broken in a way that the call generator can’t detect right away. For instance, let’s say the device under test (DUT) leaks memory with a given call flow. The call will work hundreds or thousands of times, but at some point the entire DUT may stop working — and the cause won’t be obvious.

You want the DUT to melt in your lab, not on your customer. Let’s look at three types of robustness testing.

Load Testing

Load testing is typically performed by bulk call generators with many ports. They specialize in generating high call volume to place a high load on the DUT and verify that it is robust under heavy load. For lab testing, a call volume beyond the rated capacity should be used to verify that the DUT degrades gracefully (i.e. does not crash) under stress. If you are testing a small CPE like a one or two port ATA, then a small port-count call generator can do the job.

Much of the load testing I’ve seen at various companies consists of simple call flows and does not probe the behavior of the DUT with more “interesting” calls under load. This is a missed opportunity for finding bugs. Load tests should include a variety of call flows.

Longevity Testing

With longevity testing we want to find out if the DUT can survive a large number of calls that occur at a reasonable rate. For general longevity testing you may want to randomize the test suite so that — as described above — “interesting” sets of call flows occur throughout the test.

The point of longevity testing is not to overload the DUT but rather to subject the DUT to a reasonable load over a long period of time. (Where “reasonable” is defined for the DUT according to the device’s performance goals.)

The end goal is to find defects that occur after the DUT has processed a huge number of calls over a period of days or even weeks. Again, too often longevity testing uses a single, simple call flow. It makes sense to perform longevity testing of happy-path calls, but every once in a while it’s good to mix it up and make sure all of the boundary cases are handled well.

Repeated Forced Errors

As someone who has written and read thousands of lines of code I know that the best place to find bugs is in the dark corners of the error handling code. It doesn’t get exercised often (if at all), so it’s likely that it is not quite right.

As mentioned above, these failures may not be apparent right away. It may take large numbers of iterations before they rear their head, and then the symptoms may not point to the defect. You can make it easier to diagnose the failure by repeating a test that forces a particular error to occur.

“Errors” too can include multiple classes of errors, including some that can’t be induced at the FXO interface. You may have to create some clever test scripts.

For example, let’s say the DUT is an ATA. Connect the ATA’s analog line to the RCG-4001, and provision the ATA to use the RCG as a switch. On the RCG, your script is the SIP proxy, but it does not always behave properly. Perhaps it drops contact with the ATA, or maybe it returns a 4xx or 5xx response at some point in the flow. If you run this test repeatedly you may find that your production, commercial-quality stack is not as robust to these errors as it should be.

Developers will find it handy to use their call generator to reproduce bugs reported by customers. For easy-to-reproduce bugs (i.e. those you can reproduce by hand), this may seem like overkill. However, when you write a script you benefit in several ways:

• Writing a test script to trigger the error helps build a regression test suite.
• You know when you’ve fixed the bug because the script passes.
• The script is precise documentation of the actions that caused the error.

Force Timing Bugs Out into the Open

When bugs are difficult to reproduce because of timing a test script can be a life saver.

When you test by hand, you have accuracy of maybe half a second. The call generator can force timing to an accuracy of tens of milliseconds.

With a script that makes repeated calls and varies the timing by 50 ms on each call, you can find the exact timing that causes the error. Just having the ability to force the error on every call saves valuable debugging time. Knowing the timing may also allow you to review your design or source code for places that use this timing.

Be the Squeaky Wheel

If you are an Installer or interop tester (e.g. IOT labs), you may be able to get faster response times. Try providing a script that precisely reproduces the next bug you report.

When faced with the choice of fixing an obvious bug or chasing down a hard-to-reproduce issue, developers and their managers are usually going to pick the obvious bug. (Developers can spend weeks trying to solve problems like this without anything to show for their efforts. I speak from experience!) If you can provide the obvious bug report, you can jump to the front of the line.

Bottom Line

If you can reproduce a bug by hand, the call generator can too. And hard to reproduce bugs may become easy with the help of automation.

Look at a simple two-way voice call: just pick up the phone (“A”), dial a number, waiting for ringing, answer (“B”), and make sure you can talk. Easy, right?

The Baseline Test

First we have different possibilities for the identity of A and B:

1. A and B are both “onnet”
2. A is “offnet”, B is onnet.
3. B is offnet, A is onnet.

Then we have a few different possible call flows:

1. B answers after one ring (the “happy path”).
2. B is busy.
3. B does not answer. (A hangs up after a few rings.)
4. B picks up after half a ring (mid-ring).
5. B picks up after two rings.

There are probably more conditions we can come up with, but lets look at these. The first grouping gives three different tests to run, which isn’t too bad. The five call flows in the second grouping make for 15 total test combinations. At about 30 seconds per call, that’s 7.5 minutes to run a suite for this feature.

Mix it Up With Different Gear

It gets worse quickly if you’re an interop tester working with various models of gear.

1. Cisco ATA
2. Netopia ATA

1. Asterisk
2. Avaya
3. 3Com

The 15 tests in the matrix mentioned above become 30 when there are two different adapters to test, and that’s assuming a homogenous network. If the Cisco and Netopia are on the same net, there are four combinations of “A – B” to test, meaning 60 tests to run. Three possible switches would mean 180 tests. Now we’re talking about an hour and a half to run the test suite, longer if you’ve got to change physical connections to your call generator. (VoIP gateway testers have an advantage that the connections are virtual and “changing” them is software controlled.)

Overload!

Whoa! That’s too much to think about. We haven’t even tested anything interesting yet. No feature interactions, caller id/name, call waiting, various flavors of forwarding, etc.

Three-line features build bigger test matrices even faster: a three-way-call test with just five flows and the same set of equipment above gives 840 combinations to test — seven hours of testing for a single feature. And TWC will definitely have more than five flows! A dozen flows would take over a dozen hours to test.

How do we manage all of this? The typical technique I’ve seen is to ignore most of the matrix and focus on the most common flows with the most common combinations of features. This makes a lot of sense from one perspective: if something is wrong with the common case, then there’s a real problem. Also, this probably represents most of your customer base.

Looking at it another way, though, it doesn’t make much sense. If the point of testing is to find problems, then we should define a “successful” test case as one that finds an error. Through this lens, the usual technique looks like a lot of wasted time repeatedly running the same batch of passing (“unsuccessful”) tests. It would be more profitable to instead run the odd, boundary probing, edge cases. Here, we can have a high comfort level that if these are passing the common cases are passing too.

Simplify and Find More Bugs

This implies that we can whittle down the matrix to something reasonable:

1. Happy path, Cisco to Netopia, 2 rings, Asterisk.
2. Happy, offnet to Netopia, 1 ring, Avaya.
3. Happy, Cisco to offnet, half ring, 3Com.
4. Happy, Netopia to Cisco onnet, 2 rings, Asterisk.
5. Happy, offnet to Cisco, 1 ring, Avaya.
6. Happy, Netopia to offnet, half ring, 3Com.
7. Busy, Cisco to Netopia, Asterisk.
8. Busy, Netopia to Cisco, Avaya.
9. Busy, Cisco to offnet, 3Com.
10. Busy, Netopia to offnet, Asterisk.
11. No Answer, Cisco to offnet, Avaya.
12. No Answer, Netopia to Cisco, 3Com

This reduces the set to 1/15 of the original cases. It is not comprehensive but it does touch all the devices and all the flows. When the situation calls for comprehensive testing — maybe for interop against a new switch — you’d want to run the full matrix. I chose the tests for the reduced test suite on an intuitive basis, just picking combinations that touched each matrix dimension. If you know through experience that certain combinations are troublesome, make sure they get a spot in the reduced test suite — possibly bumping another less likely test case.

Lastly, if you’re running tests daily, say on new software builds, you can achieve full matrix coverage over time by running different reduced suites on your call generator every day. Partition the full matrix into N distinct subsets, where N is the number of days to spread out the tests. Each subset should touch all the flows, but it may not use all of the equipment — you can connect new equipment to the call generator each day to achieve the full set of combinations.

For example, consider call transfer. This is what I call the “Hot Potato” test:

1. A calls B, A-B talk.
2. B transfers A to C, A-C talk.
3. C transfers A to D, A-D talk.
4. D transfers A to B, A-B talk.
5. Repeat as many times as desired, without B hanging up during the test.

Chaining tests are useful to run on your call generator overnight or over a weekend as longevity tests, probing to see if we can detect a resource leak or similar failure.

The Hot Potato test is real-world too, as I’m sure you’ve experienced calls like this to certain customer “service” hotlines.

Bonus points for chaining and stacking at the same time. When you have built up a good script library, new combinations of features aren’t hard to mash together in a new script.