Test Optimization for Today's Converged Infrastructure
Communications infrastructure is the foundation for all applications and network services, even in virtualized and cloud computing environments. Today’s businesses rely on that infrastructure in some way for nearly every business objective they have.
"The only way to be sure these objectives are met is through comprehensive infrastructure testing, and the best way to address that is by using ITO."
To raise infrastructure quality and reduce infrastructure failures like these, IT leaders should adopt Infrastructure Test Optimization (ITO). By deploying ITO best practices, test organizations can also identify opportunities to maximize test efficiency, mitigate risk and meet customer requirements.
In order to achieve the full benefits of ITO, test organizations must have the right tools and technology.
Get the details on ITO for your organization in this white paper from Spirent.
TEST OPTIMIZATION FOR TODAY’S
CONVERGED INFRASTRUCTURE
February 2012
Rev. A 02/12
CONTENTS
Executive Summary 1
Background . 2
The New Perspective on Infrastructure . 3
Infrastructure Testing Challenges 3
Infrastructure Test Optimization 5
Understanding ITO . 6
Spirent Offerings for ITO 8
Spirent iTest . 8
Spirent iTest Team Server 9
Spirent TestCenter 9
Spirent Professional Services . 9
ITO Benefits Delivered by Spirent Solutions . 9
Recommendations . 11
SPIRENT WHITE PAPER • i
Test Optimization for Today’s Converged Infrastructure
1 • SPIRENT WHITE PAPER
EXECUTIVE SUMMARY
Network infrastructure is the foundation for all applications and services, even
in virtualized and cloud computing environments. Today’s businesses rely on
that infrastructure in some way for nearly every business objective they have,
some of which are mission-critical. This creates the need for the entire network
infrastructure to have high levels of performance, availability, scalability
and security. The only way to be sure these objectives are met is through
comprehensive infrastructure testing, and the best way to address that is by
using infrastructure test optimization (ITO).
ITO is the intelligent and sustainable deployment of test resources and systems
to maximize test orchestration, collaboration, productivity and execution.
ITO is thus a set of test practices deployed to ensure infrastructures meet a
defined quality of service and quality of experience required to deliver strategic
business objectives. ITO provides organizations with capabilities to help deliver
quality products to market more cost-effectively and consistently, maximizing
their profitability.
With more companies building entire businesses around infrastructure, there
have been an increasing number of high-profile infrastructure outages. Amazon
Web Services, Sony PlayStation Network and Netflix have all experienced costly
service interruptions. To raise infrastructure quality and reduce infrastructure
failures like these, IT leaders should adopt ITO. By deploying ITO best practices,
test organizations can also identify opportunities to maximize test efficiency,
mitigate risk and meet customer requirements.
In order to achieve the full benefits of ITO, engineering organizations must
have the right tools and technology. Spirent offers a broad, integrated set of
tools that support ITO practice areas including emulation and analysis, test
automation, developer testing and lifecycle virtualization. These tools also
deliver on key ITO benefits including traceability, collaboration, integration,
realism and methodology.
Test Optimization for Today’s Converged Infrastructure
SPIRENT WHITE PAPER • 2
BACKGROUND
Information technology and related infrastructure is a critical element of almost every business.
Yet, for most companies, high quality networks are not the end goal. Business organizations
are ultimately focused on obtaining new customers, increasing profits and outperforming
competitors. To meet these objectives, businesses work toward specific goals that make sense
for their particular industry and their unique situation. These may include a combination of:
faster time to market, higher quality, increased customer satisfaction, lower costs and
greater innovation.
While communications infrastructure is not the ultimate business objective, practically every
business has a critical dependency on it. Faster time to market depends on collaboration
between interconnected teams that are located around the world. Higher quality depends on
automated systems that repeat front office, back office and supply chain activities consistently
every time. Cost reduction depends on just-in-time suppliers that exchange orders in real time.
Without the relevant infrastructure, most businesses cannot function, let alone succeed.
Businesses today require systems that are designed, built and tested to meet the demands of
an always-on, always-connected, always-available world. At the same time, network complexity
has increased dramatically, putting additional strain on all parts of the organization. To have
any chance of success, businesses must have an infrastructure that simultaneously delivers
performance, availability, scalability and security (PASS). After all, every service – insignificant
or essential – depends on the infrastructure.
Even in the age of virtualization and cloud computing, infrastructure has never mattered more.
When these technologies are used, the level of resource sharing by applications and related
services is unprecedented. Applications no longer get their own dedicated servers, storage or
subnets. They share CPUs, I/O channels, storage area networks (SAN), virtual switches, virtual
load balancers, virtual firewalls and more. Infrastructure is becoming a separate managed layer
that is shared by multiple applications, services and sometimes even separate customers.
The most modern and advanced infrastructures can still fail. In April 2011, a major OTT Provider
suffered a multi-day infrastructure outage. A simple router configuration error led to a number
of events—including a cascading series of network transmission storms—which kept services
from fully recovering for several days. Businesses that depend heavily on properly functioning
infrastructure must find ways of testing these real-life scenarios before they actually happen.
Test Optimization for Today’s Converged Infrastructure
3 • SPIRENT WHITE PAPER
THE NEW PERSPECTIVE ON INFRASTRUCTURE
Customers, employees and partners derive value from applications and other services –
not directly from the infrastructure. Most of them don’t know exactly how the underlying
infrastructure works. More importantly, they just don’t care about infrastructure. Large
organizations take advantage of specialists that have deep expertise within narrow technology
domains. Some of them only focus on a single large scale application or database and
depend on other individuals or teams for infrastructure knowledge and support. Infrastructure
management teams have traditionally been separated further into sub-groups that specialize in
a single type of infrastructure such as servers, network or storage.
As virtualization and cloud computing take hold, perspectives on networking infrastructure
are changing. Instead of only organizing specialists around different types of infrastructure,
more infrastructure staff members, and even entire teams, are taking a comprehensive view
of the infrastructure. They don’t really have a choice. Servers, networks and storage can no
longer be managed completely independently. Infrastructures are becoming less about distinct
components and devices, and more about a single integrated and orchestrated layer serving a
huge variety of different applications and services.
Organizational structures and management approaches are not the only things that need to
change to support virtualization and cloud computing. Infrastructure testing must also adapt
to the new realities of these evolved infrastructures. Just as with infrastructure management,
testing of servers, networks and storage can no longer be done completely independently.
As infrastructures become more integrated, so must the process of testing. At the same time,
infrastructure testing has a number of historical challenges that must also be addressed.
INFRASTRUCTURE TESTING CHALLENGES
Infrastructure testing challenges have increased for several different reasons: overall
demand for testing has grown; infrastructure components have become more complex and
interdependent; test methodologies have not always kept pace with advances in infrastructure;
and new technologies such as virtualization and cloud computing have impacted the behaviors
and characteristics of infrastructure. These challenges are not only impacting businesses, they
also suggest the need for fundamentally new approaches to infrastructure testing.
Customers and employees depend more and more on ever increasing amounts of infrastructure.
that enable a positive end user experienceService providers deliver a steady stream of new
services month after month. Application updates are released at a faster rate. All this activity
has increased the overall demand for testing, straining the resources and even the capabilities
of test organizations. The time available for testing – as well as the time for individual test
cycles – is decreasing. Test engineers can’t afford to spend this precious time running manual
tests, searching for the right tests or building new tests that should have been leveraged from
other efforts.
With virtualization and cloud computing, infrastructure components have become far more
complex and interdependent. Networks are converging and becoming more virtualized,
extending well beyond the physical world of switches, routers and cables. Virtual machines are
moving from server to server, crossing subnet and VLAN boundaries. Some test tools can’t keep
up with all the new infrastructure capabilities. Test cases are also becoming more complex, and
test development and maintenance is becoming for difficult and time consuming.
Test Optimization for Today’s Converged Infrastructure
SPIRENT WHITE PAPER • 4
Test methodologies are yet another aspect of testing that have not always kept pace with
advances in infrastructure. Many organizations never test the entire shared infrastructure
in a realistic, integrated and holistic fashion. They assume, too often, that infrastructure
validation is a byproduct of application testing, forgetting the unpredictable demands of
multiple applications competing for shared resources. A focus on testing individual applications
made slightly more sense when applications had their own dedicated hardware. However,
applications running in virtualized and cloud environments have little, if any, dedicated
infrastructure.
New technologies can dramatically impact the behaviors and characteristics of the
infrastructure. Utilization levels aren’t just increasing for servers. They are also growing for
networks and shared storage devices. This makes it harder to ensure realism of test scenarios,
which includes pushing the infrastructure to its limits – and sometimes beyond. At the same
time, the release of new technologies and feature upgrades are now so frequent that test teams
can’t always keep up with revalidation testing.
Taken together, these challenges are producing real business impact. They are raising the costs
of testing, from building and maintaining test libraries to performing individual test cycles.
Unfortunately, the push for rapid releases – for both software and hardware – is forcing quality
versus timeliness decisions. When the wrong decision is made, or when the law of averages
finally catches up, massive infrastructure failures end up hurting the company’s image and
bottom line. Hindsight always suggests that it would have been better to identify the issues
during testing, and fix them prior to release.
When infrastructure failures do happen, they can result in negative publicity, lost customers
and lost revenue. Again in April, 2011, a major gaming network was hacked, leaving the service
unavailable for more than 20 days. Personal details from approximately 77 million accounts
were stolen. Given the extensive impact, failures such as this must be prevented whenever
possible. To reduce the risk of infrastructure failures, testing must address a number of factors,
including performance, availability, security and scalability.
There is still more bad news. Automation is no longer enough to overcome these challenges.
Just as infrastructure management teams are handling infrastructure more holistically,
infrastructure test teams must begin to approach testing holistically as well. This still
requires automation, but it also demands collaboration among test team members. It calls for
optimization of available resources. It begs for more efficient and effective test activities. What
is really needed is an integrated, holistic test strategy.
Test Optimization for Today’s Converged Infrastructure
5 • SPIRENT WHITE PAPER
INFRASTRUCTURE TEST OPTIMIZATION
Leading development and test organizations are using an emerging approach to infrastructure
testing called Infrastructure Test Optimization (ITO). ITO is the intelligent and sustainable
deployment of test resources and systems to maximize test orchestration, collaboration,
productivity and execution. ITO is thus a set of test practices deployed to ensure infrastructures
meet a defined quality of service and quality of experience required to deliver strategic
business objectives. ITO provides organizations with capabilities to help deliver quality
products to market more cost-effectively and consistently, maximizing their profitability.
Please see Figure 1 for an overview of ITO.
As shown in the diagram, ITO encompasses five primary practice areas: emulation and analysis,
test automation, manual & developer testing, quality management and lifecycle virtualization.
The practice areas collaborate through common, shareable and reusable test assets.
Collaborative effort through improved communication between and across practice areas is
supported and encouraged. Finally, practice areas may also be complemented by professional
and expert services. Together, these elements of ITO enable an organization to test their
infrastructures in a holistic and integrated manner and, in the end, help drive business value
and innovation.
Figure 1
Business Value
Innovat ion
Emulation &
Analysis
Test
Automation
Manual &
Developer
Testing
Quality
Management
Lifecycle
Virtualization
Professional
& Expert
Services
CO L L A B O R AT I O N
Reusable and Sharable Test Assets
Test Optimization for Today’s Converged Infrastructure
SPIRENT WHITE PAPER • 6
Understanding ITO
With an overview of ITO in mind, it is helpful to dig deeper to understand each of the ITO
practice areas as well as the key benefits.
ITO Practice Areas
Practice areas are at the core of ITO. Along with guidance through expert services, they help
build a complete, cohesive and collaborative platform to enable optimization of infrastructure
testing.
The five ITO practice areas are:
1. Emulation and analysis: Reproducing real world networks and/or actively simulating real
users on the network to ensure infrastructures meet design requirements including: verification
of conformance to networking standards, meeting performance of functional requirements, and
measuring performance under load at scale.
2. Automation: Using tools to establish test pre-conditions (including management of physical/
virtual/lab resources), efficiently developing and executing tests, comparing actual versus
predicted test outcomes, providing in depth reporting, and handling lights-out regression
testing.
3. Manual & developer testing: Testing of the software by developers to verify functionality of
new capabilities. Includes manual testing and also establishes documentation as a pre-cursor
to QA testing.
4. Quality management: Using tools to link business requirements to test cases, efficiently
managing previously specified tests and resources, and tracking defects. Assets under
management may include requirements, reports, metrics and defects.
5. Lifecycle virtualization: Providing on-demand access to critical resources needed for testing
such as: environments, virtualized test equipment, servers, devices, databases, and defects.
Test Optimization for Today’s Converged Infrastructure
7 • SPIRENT WHITE PAPER
ITO Benefits
Organizations adopting ITO practices are not just looking for new methodologies or tools to
implement them. They want to improve the entire development and test lifecycle, including test
orchestration, collaboration and productivity.
They look to ITO for specific benefits that include:
• Visibility: Provides tools for a detailed drill-down into the quality environment. Helps
provide data for analysis to make informed release-based decisions.
• Traceability: Helps correlate activities across the entire infrastructure test workflow. This
enables full lifecycle versioning and complete traceability from requirements and tests,
to defects and remediation.
• Collaboration - Provides sharing of knowledge, activities and test assets. This enables
teams involved in a common task to achieve their goals more effectively. Collaboration
also includes centralized data management, workflow management and shared
documentation.
• Integration: Enables interoperability with all the discrete elements in the infrastructure.
This allows authoring of highly repeatable, integrated tests, and end-to-end
orchestration through workflow automation leveraging industry standards like NTAF.
• Realism: Ensures that infrastructure test stimulus is based on real world conditions.
This includes accurately emulating network traffic and simulating real users consuming
services.
• Productivity: Offers productivity gains through embedded test expertise, improved
collaboration, integrated test orchestration, optimum test execution, and efficient
utilization of resources.
Test Optimization for Today’s Converged Infrastructure
SPIRENT WHITE PAPER • 8
Testing Standards
ITO embraces standards based initiatives such as the Network Test Automation Forum (NTAF)
which facilitates and promotes the interoperability of commercial testing tools and test
infrastructure for the data communications and telecommunications industry. More information
can be found on the NTAF website at www.ntaforum.org.
SPIRENT OFFERINGS FOR ITO
Spirent has a rich set of products and services for implementing nearly all aspects of ITO. Those
looking for additional capabilities – such as requirements management and analytics – can take
advantage of integrated offerings such as IBM Rational Quality Manager or HP Quality Center/
ALM. Several of the core Spirent offerings, which are designed to deliver on the promise of ITO,
are shown in Figure 2 and described in more detail below.
Spirent iTest
iTest is an award winning integrated development environment (IDE) for automated test
development. It enables rapid authoring of complex, heterogeneous, system tests and
provides a rich set of test authoring, execution, and analysis tools. With iTest you gain a single
solution for functional, performance, load, and stress testing of systems and equipment. And
very importantly a standard test case format so test assets can be easily shared across the
organization and with partners, suppliers and customers. From an ITO perspective, iTest serves
the test automation and developer testing practice areas.
Figure 2
Emulation &
Analysis
Test
Automation
Manual &
Developer
Testing
Quality
Management
Lifecycle
Virtualization
Professional
& Expert
Services
CO L L A B O R AT I O N
Reusable and Sharable Test Assets
TestCenter
Avalanche
Landslide
TestCenter &
Avalanche
Virtual
Test Optimization for Today’s Converged Infrastructure
9 • SPIRENT WHITE PAPER
Spirent iTest Team Server
As a server-based product, iTest Team Server enables the test community to work together
more closely and collaboratively. It includes an analytics dashboard and tracks results and
automation progress using built in metrics. This allows managers to make evidence-based
release decisions, reducing the reliance on opinions and gut feelings. It also includes a solution
for manual testing that allows the results of manual tests performed by the entire team to be
automatically documented and integrated into the system. From an ITO perspective, iTest Team
Server provides value to the collaboration and the manual testing practice areas.
Spirent TestCenter
Spirent TestCenter™ minimizes risk by emulating complex networks and realistic traffic
patterns and issues when they occur. This application of high scale traffic under real world
conditions is used to test network and appliance virtualization as well as cloud, mobile and
application experience. From the ITO perspective Spirent TestCenter serves the emulation
& analysis practice area, while Spirent TestCenter Virtual serves the lifecycle virtualization
practice area.
Spirent Professional Services
Spirent’s Professional Services teams, along with best-of-breed technology partners, develop
and deliver a range of ITO solutions using Spirent’s world-class test tools. Originally offered
under Spirent’s NoCode banner, these solutions address the broad range of problem areas
faced by organizations struggling with the cost and complexity of testing. They also deliver the
greatest opportunity for increased efficiency, improved quality, and testing scale, resulting in
lower CAPEX/OPEX.
Companies of all types, from enterprise organizations to service providers, are taking
advantage of Spirent offerings to overcome the challenges of testing and gain the benefits
of ITO.
ITO BENEFITS DELIVERED BY SPIRENT SOLUTIONS
Together, the Spirent solutions for the ITO market deliver increased test coverage, improved
collaboration, and more efficient use of resources. All this helps drive innovation and deliver
quality products and services to market on time. ITO solutions also help link business
requirements to the early stages of the quality process and trace them throughout the entire
development lifecycle to mitigate risk. There are a number of additional benefits that are worth
exploring further.
Since the overall demand for testing has grown and the pressure for faster test cycles has
increased, Spirent solutions for ITO are continually adding higher levels of automation.
Together, these solutions deliver a platform for productivity by offering automation within and
between each product to address the entire test lifecycle. Test activities from authoring and set
up, to execution and analysis all benefit from increased levels of automation.
Test Optimization for Today’s Converged Infrastructure
SPIRENT WHITE PAPER • 10
With the tightly integrated infrastructures that are prominent in virtualized and cloud computing
environments, organizations must take a more holistic approach to infrastructure design,
deployment and management. The same is true for testing. Spirent ITO solutions enable the
collaboration that is needed to bring together disparate tools, skills and capabilities to validate
the entire IT infrastructure. Allowing the sharing and reuse of resources and coordinating
interdependent activities are other important elements delivered by Spirent solutions for ITO.
Network infrastructures have grown more complex than ever, extending from silicon to
devices and from LANs to WANs. To ensure the appropriate level of validation at each layer of
infrastructure, Spirent ITO solutions provide the traceability required to link each requirement
to any test activity. Spirent solutions also provide visibility to all lifecycle activities, including on
demand analytics for in-depth understanding of failures as well as broader perspectives
on progress.
Infrastructure testing has always demanded realism. Yet, with virtualized and cloud
infrastructures, it has become much more difficult to achieve. Spirent ITO solutions
are designed to emulate real-world scenarios that are appropriate for the entire shared
infrastructure, not just the dedicated hardware that used to be associated within individual
applications. This includes the ability to exercise and validate virtual switches with full control
of virtual machine to virtual machine traffic within the same physical server.
To see how each of the key ITO benefits is delivered from individual Spirent products, please
refer to Figure 3.
Figure 3
VISIBILITY
TRACEABILITY
COLLABORATION
REALISM
INTEGRATION
PRODUCTIVITY
BENEFITS OF ITO Team Server
Team Server
Landslide Avalanche TestCenter
Test Optimization for Today’s Converged Infrastructure
11 • SPIRENT WHITE PAPER
RECOMMENDATIONS
Communications infrastructure is the foundation for all applications and network services, even
in virtualized and cloud computing environments. Today’s businesses rely on that infrastructure
in some way for nearly every business objective they have. This creates the need for the entire
network infrastructure to have high levels of performance, availability, scalability and security.
The only way to be sure these objectives are met is through comprehensive infrastructure
testing, and the best way to address that is by using ITO.
The requirement for a high quality infrastructure is raising the importance of testing
organizations throughout the communications industry. Development leaders must recognize
this shift and give their teams the strategies and tools they need to perform infrastructure
testing at the highest levels. By using ITO, development and test organizations can also identify
opportunities to maximize test efficiency, mitigate risk and meet customer requirements, all
while meeting deadlines.
In order to achieve the full benefits of ITO, test organizations must have the right tools and
technology. Spirent offers a broad, integrated set of tools that support ITO practice areas
including emulation and analysis, test automation, developer testing and lifecycle virtualization.
These tools also deliver on the key ITO foundation benefits, including traceability, collaboration,
integration, realism and methodology.
