Test Case Automation – The Gift That Keeps Giving

By John McLendon, Spirent On July 16, 2009
No tags assigned.

A while back we talked about test lab automation (see Advanced Test Automation), which is returning sanity to test labs all over the globe. In that post I made this statement:

"If you haven’t automated your tests, you’re missing one of the best ways to boost productivity and impress your boss. (And the money guys, too.)"

It occurs to me that there might be outposts of resistance lurking in the hinterlands that have yet to reap the many rewards available from test case automation.

As I list the many benefits of test automation, you may start to wonder if I’m going to claim that it also trims 20 lbs from your waistline and restores hair loss. Not exactly, although it might keep you from tearing out your hair, so maybe there’s something to that after all.

But seriously, the benefits are there. How about fewer customer-reported bugs, expanded test coverage, increased productivity and lab usage, reduced operating costs, shorter test cycles, and more. Let’s look at some hard numbers:

  • A lab I visited recently was testing their 10Gig MPLS architecture for approximately 20 different test cases. Test automation freed up 3.5 engineers to add more test cases and expand test coverage.
  • Another lab was testing RP switchover convergence. We found that it took 35-45 minutes to run a single test case. With more than 20 iterations for each test, one test could take an entire work day with an engineer babysitting it to keep it going. The test was automated to eliminate the need for user interaction. This change produced a 40% productively gain – nearly half a person’s daily activity recovered by automating a single test suite.

This is huge! Can you think of any one single thing you can do that will increase your productivity by 40%? Personal organizer? No. Time management? No. Multitasking? No. Meditation? No. Cloning? Maybe.
Now you see why I call it the gift that keeps on giving. From now on, every time they run that test, they’re saving a boatload of time and money based on a one-time investment. It gives me a warm fuzzy just thinking about it.

One of the real values of scripting is to increase test realism. Scripting means you can precisely control the test environment during execution to recreate the dynamic behavior of a real network. And of course, the more realistic the test, the more meaningful the result.

Which is the point, after all. You want to know your system can handle the real world. It’s not useful to know that it passes all the tests in an artificially favorable environment built with fixed stimuli and static topologies, and then ship it or deploy it and have your customers find the problems for you!

Now that you’re ready to join the club, let me give you a tour of what’s out there and what to watch out for.

Scripting. You write a program, usually in Tcl, to configure the test equipment and the system under test and then to start the test.

  • Pros: If the API is robust, you have more control over the test than you would running it from the GUI.
  • Cons: Must learn a scripting language.
  • What to look for: A test system that allow changes to the configuration during test execution, so that a realistic test can be created.

GUI-to-script. You configure a test with the test system GUI, then save the configuration as a script which may (good) or may not (bad) be modified to customize the test.

  • Pros: No need to learn scripting, especially if you’re not customizing.
  • Cons: Configuring through the GUI can be time-consuming. Unless you learn scripting, the tests will be of limited use, basically configure and go.
  • What to look for: Support for script-to-GUI, useful for troubleshooting configuration issues after tweaking a script.

GUI-based automation editor. You create automation by dragging and dropping commands into a visual command editor, popping up windows to configure parameters.

  • Pros: Create sophisticated, realistic tests without learning a scripting language.
  • Cons: None.
  • What to look for: Tight integration between GUI commands and API commands, so that automation created in one interface can be used in the other.

So, pop on that lab coat and start automating those test cases. Your boss and your bottom line will love you for it.

comments powered by Disqus
× Spirent.com uses cookies to enhance and streamline your experience. By continuing to browse our site, you are agreeing to the use of cookies.