There is a trend across the board to accelerate time to revenue. From service providers to equipment manufacturers to enterprises—release schedules are escalating, despite the fact that the solutions are more complex.
The test lab is caught in the middle, trying to find a way to do more in less time. In this kind of environment, the tyranny of the urgent can make it difficult or impossible to dig out from under the mountain of expectations.
Do these problems crop up in your test lab?
- Increasing solution complexity requires increasingly sophisticated tests
- Pressure to hit deadlines leaves no time to optimize test cases or infrastructure
- Managing multiple test beds results in time-consuming configuration issues or mistakes
- Reproducing the circumstances under which an error occurred is very difficult
- Lack of standards or conventions creates problems for collaboration or handoff to another engineer or team
- In the heat of the moment, tests aren’t documented and follow up is lacking and incomplete
- No cross-training on script development leaves you vulnerable if the in-house expert is no longer available
The end-goal is to build a test infrastructure that increases the efficiency of both the test system and the test engineers, while at the same time improving the quality of testing and consequently the quality of your product. But can that realistically be done on top of everything you have to do?
Envision a test infrastructure that accommodates the plethora of tools and interfaces currently in use, that automates repetitive, manual tasks like device configuration and test execution, or even better, that automates the documentation of test execution to provide an audit trail. Imagine a systematic way of developing libraries of test cases that makes collaboration easier, that automates the comparison and archiving of test results, with a decision support system that makes sophisticated test cases possible while minimizing human intervention.
ITO Defined
As important as understanding whether an application or service performs as expected, is understanding whether the network and computing infrastructure performs as expected. This requires targeted testing to determine how well the network and infrastructure copes with change, specifically in relation to PERFORMANCE, AVAILABILITY, SECURITY and SCALABILITY. Infrastructure testing goes beyond typical network testing to include the validation of:
- Physical & virtual environments, operating systems and CPU
- Middleware, applications and storage
- Hardware products and IT/cloud infrastructure
- Network and network devices
Essentially, infrastructure can be defined as a structured platform of networked elements required to deliver services.
ITO is the intelligent and sustainable deployment of test resources and systems to maximize test orchestration, collaboration, productivity and execution. ITO is thus a set of test practices deployed to ensure infrastructures meet a defined quality of service and quality of experience required to deliver strategic business objectives. ITO provides organizations with capabilities to help deliver quality products to market more cost-effectively and consistently, maximizing their profitability.
Practice areas are at the core of ITO. Along with guidance through expert services, they help build a complete, cohesive and collaborative platform to enable optimization of infrastructure testing.
As shown in the diagram, ITO encompasses five primary practice areas: Emulation and analysis, test automation, manual & developer testing, quality management and lifecycle virtualization. The practice areas collaborate through common, shareable and reusable test assets. Collaborative effort through improved communication between and across practice areas is supported and encouraged. Finally, practice areas may also be complemented by professional and expert services. Together, these elements of ITO enable an organization to test their infrastructures in a holistic and integrated manner and, in the end, help drive business value and innovation.
Emulation & Analysis
Reproducing real-world networks and/or actively simulating real users on the network to ensure infrastructures meet design requirements including: Verification of conformance to networking standards, meeting performance of functional requirements, and measuring performance under load at scale.
Test Automation
Using tools to establish test pre-conditions, efficiently developing and executing tests, providing in depth reporting, comparing actual versus predicted test outcomes, providing in depth reporting, and handling lights-out regression testing.
Manual & Developer Testing
Testing of the software by developers to verify functionality of new capabilities. Includes manual testing and also establishes documentation as a pre-cursor to QA testing.
Quality Management
Using tools to link business requirements to test cases, efficiently managing previously specified tests and resources, and tracking defects. Assets under management may include requirements, physical/virtual/lab resources, reports, metrics and defects.
Lifecycle Virtualization
Providing on-demand access to critical resources needed for testing such as: Environments, servers, devices, databases, defects etc.
Organizations adopting ITO practices are not just looking for new methodologies or tools to implement them. They want to improve the entire test lifecycle, including test orchestration, collaboration and productivity. They look to ITO for specific benefits that include:
Visibility
Provides tools for a detailed drill-down into the quality environment. Helps provide data for analysis to make informed release-based decisions.
Traceability
Helps correlate activities across the entire infrastructure test workflow. This enables full lifecycle versioning and complete traceability from requirements and tests, to defects and remediation.
Collaboration
Provides sharing of knowledge, activities and test assets. This enables teams involved in a common task to achieve their goals more effectively. Collaboration also includes centralized data management, workflow management and shared documentation.
Realism
Ensures that infrastructure test stimulus is based on real world conditions. This includes accurately emulating network traffic and simulating real users consuming services.
Integration
Enables interoperability with all the discrete elements in the infrastructure. This allows authoring of highly repeatable, integrated tests, and end-to-end orchestration through workflow automation.
Productivity
Offers productivity gains through embedded test expertise, improved collaboration, integrated test orchestration, optimum test execution, and efficient utilization of resources.
