The Complexity of Test Environment Configuration with Heterogenous Hardware and Software

This is the fourth post in a six-part series on lab management challenges. For more information, check out the original post, “Top 5 Challenges of Lab Management.” 

Technologies like virtualization and cloud computing have revolutionized the way organizations develop software, hardware and IT products. The services and sites we rely on today run across integrated hardware and software platforms that include many different networks, servers, routers, applications and virtual machines.

With this complexity comes an increased opportunity for error. A single bug in a product’s infrastructure can damage its ability to function, erode consumer faith in an established product or brand, and even destroy the market share of a new one.

To prevent this scenario from occurring, testing is done to ensure a product or update will function as it is intended. Test teams must execute rigorous testing through heterogeneous software and hardware to identify bugs early, smoothly launch products and release new features without incident.

All too often, however, organizations fail to integrate automated tests across platforms, and rely instead on manual testing that silos various hardware and software components. This critical oversight results in problems during the testing process, and impacts the organization’s products and bottom line.

How Manual Testing with Heterogeneous Software and Hardware Hurts Your Business

Defective products result in costly revisions, bad press and lost revenue. For these reasons, companies rely on testing to identify potential issues before a product release. Unfortunately, many testers face serious challenges in lab environments that lead to inadequate testing. Specifically, issues resulting from the use of manual testing in conjunction with heterogeneous software include:

Project Delays

When hardware and software must be tested separately and manually, the testing cycle lengthens. Unfortunately, tough deadlines mean that the longer the testing cycle, the more likely corners will be cut— resulting in unacceptable product quality on release day.  For example, the team might decide to skip running a few test cases, or decide that certain conditions will rarely happen, and might not test for those conditions.  They might not do performance or scale testing to the level required (e.g. executing a 24-hour stress test rather than a 7-day stress test.  All of these are “holes” or “gaps” in test coverage and could potentially bite in the form of bugs reported from customers.

Test team communications, test cases and testbeds become more efficient through the use of automated testing, resulting in a more transparent testing process. Manual testing, however, requires far more time. It also leaves test teams in the dark regarding testbed usage.

More Mistakes

Products and platforms are becoming much more complex, but products must go to market quicker to obtain a competitive advantage. Manual testing alone cannot cope with limitations of time, especially with an overabundance of heterogeneous hardware and software components included in each test case.

These heterogeneous products can produce almost limitless combinations, further complicating the process for manual testers. Frequent testbed reconfiguration is also more prone to error when done manually because of its intricate nature. The only way to ensure adequate testing is through consistent repetition — an activity made possible with automated testing.

Higher Costs

A longer, more error-prone testing process impacts the bottom line of any business. With such an abundance of compatible software and the complexity of products being developed, it’s easy for tests to outgrow the time you’ve budgeted for them. As the testing backlog builds up, testing costs increase and the quality of your tests suffers.

The longer the test process, the better chance that mistakes will be made and important tests will be skipped. This has a serious impact after a product goes to market, as sales of a buggy product suffer along with your organization’s future business and reputation.

Inconsistent Reporting and Lack of transparency

Manually configured testbeds with heterogeneous software and hardware present the perfect scenario for lost communications and mistakes. Miscommunication among team members affects the entire value chain, with an aftermath of confusion and duplicated efforts.

An integrated testing solution allows the testers to catalog the commands, actions and responses involved in testing various combinations of hardware and software, improving team-wide communications.

Why Incompatible Software Causes Problems During Testbed Configuration

In the lab, testers deal with so many types of software during the testbed configuration process that the number of software and hardware combinations is virtually limitless. Without an automated testing solution, this fact of lab life means that much time is spent having experts and specialists manually configure test cases and testbeds.

These delays, as well as the mistakes that occur when manual testers cannot test certain software and hardware combinations, lead to costly downtime and impact the overall quality of product tests. However, an automated testing solution can address problems caused by heterogeneous software and hardware, resulting in faster, more accurate tests.

How have heterogeneous software and hardware products impacted your testing cycles? What are the most common problems you’ve experienced? Please share your experiences below.

comments powered by Disqus
× Spirent.com uses cookies to enhance and streamline your experience. By continuing to browse our site, you are agreeing to the use of cookies.