Test Realism - Find and fix your bugs before your customers find them for you
Expectations. When I turn on the faucet, I expect water. When I flip a switch, I expect the light to come on. When I pick up the phone, I expect to be able to make a call. When I punch the remote, I expect to see my show. All pretty much the same, right?
Almost. Water and electricity are delivered to us in basically the same way they were a century ago. But delivery of phone service is changing, and delivery of television, which didn’t even exist a century ago, is being radically transformed.
Regardless of how the infrastructure has changed, customer expectations haven’t. When I punch the remote, I still expect to see my show. Sure, all the new technology that makes it work is cool, but when my team is down by two with five seconds on the clock, the score I’m focused on is not a MOS-V score.
Only through testing can you meet reliability and quality expectations. Of course. Everybody tests.
So why is it that customers still have reliability and quality-of-experience problems? Because it’s not if you test, but how you test.
You avoid customer-reported problems with realistic testing. Strategies used in the past, such as packet blasting, aren’t effective in predicting how applications will work on a converged network. Throwing traffic at your system, whether on the data plane (short packets at line-rate), control plane (massive route updates) or subscriber signaling (high setup-teardown rates), doesn’t tell us a thing about how the system will handle Black Friday shopping or the final vote on American Idol.
Realistic testing means re-creating the environment that the application lives in, from the provider to the customer, in all its dynamic and daunting complexity. Test realism means:
Real user behavior. Users are as unique as their fingerprints. They vary significantly in how long and in what manner they navigate through an application and how they respond to sluggish performance, picture and voice dropouts, dropped calls, and other problems. They violate usage and security policies in different ways. They find unique ways to break your system.
Realistic testing means the flexibility and sophistication to emulate a wide range of user behavior, both benign and malicious.
Hitting a device with a mix of traffic types (say, mixing HTTP requests/responses with IPTV channel changes and P2P file sharing or gaming traffic) isn’t emulating user behavior. It’s just hitting it with a mix of traffic. Emulating real user behavior means supporting stateful traffic that emulates how a user operates, including think time, click-through, abandon, channel surfing, etc. It means good users and malicious users simultaneously attempting to achieve their good and bad goals.
User-centric traffic tests the performance of the device in a real environment. Queues, buffering, and other mechanisms behave differently depending on the order and the nature of the transactions. An arbitrary (and artificial) static mix of messages, or even a dynamic mix of messages that don’t account for the stateful nature of user connections, will not stress the system the way real users do. As a result, failure points can remain undetected until the real users start using it. Which is exactly the wrong time for that to happen!
Real converged traffic. Single-purpose networks are relics of the past. The consolidation of networks onto Carrier Ethernet and MPLS enables mobile and fixed-line voice and data, residential video, and MPLS-based VPNs. The different types of traffic carried on the converged network have different characteristics and requirements, but they all travel on the same path, dependent on CoS and differentiated traffic rules to keep everyone happy.
Realistic testing means the power to create line-rate, fully-emulated, stateful traffic across hundreds of ports.
Real converged traffic not only means a realistic mix of traffic types, but also realistic traffic encapsulation. For example, if the deployed system tunnels user PPP sessions over MPLS, then testing PPP setup-teardown rate and throughput performance without MPLS is not real converged traffic. It’s a dangerous shortcut that will mask problems your users will discover after deployment. Real converged traffic means emulating the actual deployed topology, regardless of how complex or simple, including all encapsulations.
Real network conditions. The network creates time-varying conditions that are linked to a complex set of conditions, influenced by routing table updates, signaling protocols, queuing algorithms, buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.
Realistic testing means the power and complexity to create the dynamic, time-varying conditions found on deployed, production networks.
Real network conditions can’t be emulated through static rates of delay/loss or distribution-based models of impairment. Real networks don’t introduce impairments at fixed rates or follow neat curves. They behave in seemingly non-deterministic ways due to the number of factors affecting them. Testing under real network conditions means emulating this complexity to discover issues before the real network finds them for you.
Test realism gets you closer to the goal of delivering reliability and quality of experience, which is the ultimate differentiator in any market. Oh, hold it, the game is on. I’ll be right back.