Realism and Multi-Play - Part 1 NEMs

By Brett Wolmarans, Spirent On July 21, 2009
No tags assigned.

Find and fix your bugs before your customers find them for you

Remember what Jack Nicholson told Tom Cruise in A Few Good Men? “You can’t handle the truth!”

That’s the kind of thing you don’t want a customer saying about your product. “Sure, it works fine in the lab where everything is well-behaved, but it can’t handle the real world.”

So, what exactly is the real world? In the movie, reality for Nicholson was different from reality for Cruise. My reality is lots of conference calls, business travel, deadlines. Not only can I handle it, I love it. Reality for a soccer mom is very different, and let me tell you right now, I can’t handle that reality, so let’s not do Trading Places.

A while back we talked about test realism which includes:

  • Real user behavior: wide range of stateful user behavior, both benign and malicious
  • Real converged traffic: line-rate, fully-emulated, stateful traffic across hundreds of ports
  • Real network conditions: dynamic, time-varying conditions found on deployed, production networks

But reality for an email server is very different from reality for a set-top box. Which means, if I want to get all the benefits of test realism (like increased test coverage, better product reliability, improved customer quality-of-experience, and fewer customer-found bugs, to name a few) I have to first know what reality is for the system I’m testing. Then I have to be able to faithfully create that reality, fully emulating every user, device and condition in that reality, from end to end, that my system will encounter.

In the multi-play world, reality gets complex. It’s the perfect storm of diverse content and behavior. So, how do I get that complex reality in my lab?

First, I need the scalability to generate a city’s-worth of voice calls, IPTV traffic, and data traffic for thousands or millions of users across hundreds of ports. But the ability to scale to reality in number of ports and volume of traffic, while important, isn’t by itself realism. That’s just packet blasting from a lot of ports.  It’s a brute-force assault that is fairly easy to do but doesn’t tell me anything about how my system will handle reality. Which leaves me at risk for Nicholson-type comments from my customers.

Test realism for multi-play means the traffic is generated by fully-emulated devices that generate stateful voice, video and data traffic, including signaling and routing. Emulated devices that respond to queries, requests, and commands from my system.

Second, I need that traffic to come from real users. An artificial blend of dummy transactions based on a percentage or distribution doesn’t create the conditions in the system under test that is created by real users. I need stateful traffic from users doing things in the order that users do them, whether they be consistent (normal usage), seemingly random (multitasking or short attention span) or malicious (security risks).

Third, I need the ability to create the imperfect world my system will exist in, one where packet delay, loss and other unpleasant things sometimes occur. A pristine lab network is not reality. And I need this at wire rate, and when I say wire-rate I am talking at 10G speeds!

Test realism for multi-play means my system is forced to deal with the dynamic, time-varying conditions caused by likely events such as routing table updates, queuing and buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.

Fourth, I need one system that can emulate all of this realism: users, traffic, and network. I don’t need a solution that requires me to patch together a dozen or half-dozen different systems to achieve an approximation of realism that might work as long as there are no conflicts between the various test systems and how they emulate or measure their niche in the test. I need one integrated system with one GUI and one API and one correlated set of results and the power to support all the elements of test realism.

Fifth, I need trickle-down expertise built right into the test equipment that guides me. I need to know I am buying a solution from a company that knows what to test and how to test it. Because, whether painful or not, I have to know the truth before my customers do. Even if that truth is that the performance of my system isn’t as good as my unit test led me to believe. Especially if it’s not as good as I thought.

The best test solution is like a true friend. Not the phony friend that tells you what you want to hear, but the true friend that will tell you the truth, even if it’s harsh, and then will stick with you to fix it.

So I’m looking for expertise built into the test gear, into the software interfaces, and into the support and services I get from my vendor. It’s got to be a complete package. This expertise comes from (1) participating in standards bodies to create protocol and testing specifications and (2) by successfully supporting a user base of thousands of customers, year after year. I need a vendor who is going to be around tomorrow for the long term to support me.

That’s a lot to ask from a test system. But there’s a lot at stake. And when it comes down to it, you really do need to know the truth. And your solution really does need to be able to outperform the competition in the real world.

The truth is out there. Can you handle it?

comments powered by Disqus
× Spirent.com uses cookies to enhance and streamline your experience. By continuing to browse our site, you are agreeing to the use of cookies.