Eleven Steps to Virtualize Your CI Test Environment

As more organizations take advantage of the flexibility and scalability of a cloud-based, OpenStack, Continuous Integration (CI) environment for development, they naturally look to extend those benefits to the rest of the product lifecycle. But the test lab poses significant challenges for this process, including a rapidly changing code base that must perform on multiple operating systems and cloud management platforms.

Spirent assessed the issues and gathered requirements to provide our developers with on-demand, multi-node Openstack test environments where they could, in a whiteboard-like fashion, configure different network functions and customize them across different distributions.

The main objectives were ease and speed of test lab setup, security, repeatability, and cost. The solution had to integrate with our existing product build process, which spanned various adhoc, DevStack-based OpenStack installations (i.e. Havana, Icehouse, Juno) with differing levels of support and documentation, and provide a seamless handoff to the product test environment, regardless of operating system, OpenStack environment, or cloud management distribution.

The result was a real-world, technical implementation of a highly-parallelized OpenStack CI network testing solution.

The Tools

The test rigs perform both functional and performance tests as well as probing for security weaknesses and vulnerabilities. For performance testing we use a cloud based test lab hosted at Nephoscale that has access to the latest Spirent traffic generation equipment. For functional and regression testing the nested virtualization capabilities provided by Ravello Systems are leveraged to run KVM and ESXi Virtual Machines on AWS.

Ravello offers all the functions of our physical test lab yet only uses virtual machines and networks deployed on a public cloud infrastructure. Using their blueprint technology and API SDK, we have empowered our developers to build their own multi-node OpenStack lab environments in the cloud with different distributions at the push of a button. Machines typically live for a few hours at a time, and we can scale out to hundreds of machines, depending on the combination of operating systems and networks to be tested.

We use Mist.io to provision OpenStack host machines through the NephoScale API to kick start the RDO deployment process and to provide real-time monitoring of the testing procedures. We configure custom metrics on the host machines and the guest VMs. These trigger alerts and automated responses when needed, such as to provision additional compute and network nodes. Mist.io also provides a unified dashboard for managing multiple OpenStack installations (Havana, IceHouse, Juno, Kilo).

Since some of the products under test include sensitive intellectual property, we implement strong security controls. The Openstack test labs running on Nephoscale and AWS through Ravello are completely fenced off from external access and from each other. This allows us to implement parallel test labs for our customers with minimal concern for data leakage.

The test bed includes Spirent’s Velocity suite of iTest products for automation and orchestration of test element provisioning, test execution, report generation, and data analysis.

Function Tool
Source control management (SCM) Git
Apache project management Maven
Continuous integration Jenkins
Code quality enforcement Sonar
OpenStack install automation RDO Packstack
Cloud management Mist.io, SaltStack, Heat templates
Hosted cloud NephoScale
Nested virtualization on AWS platform Ravello
Issue and project tracking JIRA
Network test Spirent TestCenter, Avalanche, Avalanche Next, Landside, MCC, Velocity

The Process

The multi-step process for CI is heavily automated to streamline testing and accelerate the test schedule.

Step 1: Developer checks out source files from the locally-hosted product code base repository and makes changes.
Value: A source control management system protects the integrity of the code base by allowing distributed development, asynchronous updates, and selective roll-back to maintain quality.
Tools: Git

Step 2: Developer performs a Git pull to obtain all the latest commits from other team members.
Value: Every developer uses the most current code.
Tools: Git

Step 3: Developer creates a clean, local-workstation build and performs unit testing.
Value: Automated clean builds accelerate the unit-testing process.
Tools: Maven

Step 4: After successful unit testing, the developer commits the changes to the repository for source code version control, which triggers a series of jobs.
Value: New content isn’t committed to the code base until it’s been tested, and any update can be easily backed out if it introduces quality issues. Automation reduces errors, accelerates the schedule.
Tools: Git Hook, Jenkins

Step 5: A code quality enforcement tool reports on compliance with best practices for the development language to a dashboard for review and drill-down to violation details.
Value: Code quality problems are caught early, maintaining the integrity of the code base.
Tools: Sonar

Step 6: An automated build process creates multiple test OpenStack environments and software builds for each target environment, whether it’s locally-hosted QA environment or a virtual, cloud-based environment.
Value: Automating the build process allows for broader test coverage, including multiple hypervisor flavors and OpenStack distributions.
Tools: OpenStack tools: RDO Packstack
Tools: Automated OpenStack test environments on public cloud AWS and GCE via Ravello Systems
Tools: Cloud management tools: Mist.IO, SaltStack, Heat templates
Tools: Hosted cloud: NephoScale
Tools: OpenStack distributions: SUSE, Ubuntu, RedHat RDO, Mirantis Fuel, and Piston with different Neutron networking configurations (VXLAN, GRE, VLAN)

Step 7: Testing includes unit tests, basic integration tests, and functional tests.
Tools: Spirent TestCenter, Avalanche, Avalanche Next, Landside, MCC, Velocity

Step 8: Test data is collected and analyzed into a data warehouse, including performance and security findings correlated with metrics on infrastructure utilization such as CPU, RAM, PCI Bus, network, storage, and power.
Tools: Spirent Velocity

Step 9: Different OpenStack environments in Ravello are saved as blueprints for future reference.
Value: The ability to quickly run instances of Openstack from existing blueprints and compare those with environments with patches applied.
Tools: SaaS platform: Ravello Systems

Step 10: Defects are recorded and tracked for troubleshooting and resolution.
Value: All issues are documented to reduce the possibility of a defect falling through the cracks
Tools: JIRA

Step 11: The OpenStack test environments automatically shut down and the cycle is complete.
Tools: SaaS platform: Ravello Systems

Continuous Integration-Validating Virtual Appliance Image Builds for OpenStack


Based on OpenStack, we built a workflow to continuously test our products without overbuilding the hardware infrastructure or spending time provisioning multi-node Openstack environments with different flavors for each developer. Each of our developers quickly spin up their own multi-node Openstack labs as needed, run them on AWS, execute automated tests including parallel tests on multiple Openstack setups, and shut them down when they are done. This new workflow process has improved quality and allows us to support multiple distributions used by our customers.

comments powered by Disqus
× Spirent.com uses cookies to enhance and streamline your experience. By continuing to browse our site, you are agreeing to the use of cookies.