Spirent Journal of Benchmark PASS Test Methodologies
To provide a stand way of measuring and evaluating a Device Under Test (DUT), the industry has created a series of benchmarks. These benchmarks establish a uniform procedure for the generation of traffic to and from the DUT with a normalized procedure of analysis and reporting. The goal of the benchmark is to generate metrics in a reproducible and unbiased fashion for comparability. This journal presents a comprehensive set of test cases to measure PASS of the Device Under Test (DUT).
PASS
Spirent Journal of
Benchmark PASS Test
Methodologies
February 2011 Edition
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
1
Introduction
Today’s Devices Under Test (DUT) represent complex, multi-protocol network elements with an emphasis
on Quality of Service (QoS) and Quality of Experience (QoE) that scale to terabits of bandwidth across the
switch fabric. The Spirent Catalogue of Test Methodologies represents an element of the Spirent test
ecosystem that helps answer the most critical Performance, Availability, Security and Scale Tests (PASS)
test cases. The Spirent Test ecosystem and Spirent Catalogue of Test Methodologies are intended to help
development engineers and product verification engineers to rapidly develop and test complex test
scenarios.
How to use this Journal
This provides test engineers with a battery of test cases for the Spirent Test Ecosystem. The journal is
divided into sections by technology. Each test case has a unique Test Case ID (Ex. TC_MBH_001) that is
universally unique across the ecosystem.
Tester Requirements
To determine the true capabilities and limitations of a DUT, the tests in this journal require a test tool that
can measure router performance under realistic Internet conditions. It must be able to simultaneously
generate wire-speed traffic, emulate the requisite protocols, and make real-time comparative
performance measurements. High port density for cost-effective performance and stress testing is
important to fully load switching fabrics and determine device and network scalability limits.
In addition to these features, some tests require more advanced capabilities, such as
Integrated traffic, routing, and MPLS protocols (e.g., BGP, OSPF, IS-IS, RSVP-TE, LDP/CR-LDP) to
advertise route topologies for large simulated networks with LSP tunnels while simultaneously
sending traffic over those tunnels. Further, the tester should emulate the interrelationships
between protocol s through a topology.
Emulation of service protocols (e.g., IGMPv3, PIM-SM, MP-iBGP) with diminution.
Correct single-pass testing with measurement of 41+ metrics per pass of a packet.
Tunneling protocol emulation (L2TP) and protocol stacking.
True stateful layer 2-7 traffic.
Ability to over-subscribe traffic dynamically and observe the effects.
Finally, the tester should provide conformance test suites for ensuring protocol conformance and
interoperability, and automated applications for rapidly executing the test cases in this journal.
Further Resources
Additional resources are available on our website at http://www.spirent.com
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
2
Table of Contents
Testing Industry Standards .....................................................................................................3
BNCH_001 RFC 3918 Multicast Join/Leave Latency Test ............................................................ 4
Appendix A – Telecommunications Definitions .......................................................................8
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
3
Testing Industry Standards
To provide a stand way of measuring and evaluating a Device Under Test (DUT), the industry has created a
series of benchmarks. These benchmarks establish a uniform procedure for the generation of traffic to
and from the DUT with a normalized procedure of analysis and reporting. The goal of the benchmark is to
generate metrics in a reproducible and unbiased fashion for comparability.
Benchmarking standards can come
from any organization for potential
industry wide adoption. Two key
organizations, IEEE and IETF, help set
standards in the industry by
coordinating recommendations.
Though the RFC (Request for
Comment) and WG (Working Group)
system, sub groups like the BMWG
(Bench mark Working Group) help
coordinate and refine
recommendations. Key
recommendations such as RFC-2544,
RFC-2889, and RFC-3918 have come
from this process.
In order to execute benchmarks, test and measurement equipment like Spirent TestCenter, Spirent
Avalanche, and Spirent Landslide help the user systematically generate, analyze, and report based on the
industry derived standards. Spirent TestCenter is especially architected to rapidly test and measure
industry standards to its microkernel-architected and high-port-density design, allowing up to 32 multiple
automated processes to execute in parallel per chassis.
This document describes the methodologies associated with testing key industry standards. These
constructive generic frameworks help users understand and execute key standards and reduce the time
associated with testing benchmarks.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
4
BNCH_001 RFC 3918 Multicast Join/Leave Latency Test
Abstract
This test case determines the time it takes the DUT to start/stop forwarding multicast frames
once it receives a successful IGMP group membership report/leave group message.
Description
This test is part of the RFC 3918 tests, in this case to determine the time it takes a DUT to
start/stop forwarding multicast frames from the time a successful IGMP group membership
report/leave group message is sent.
In this test, Spirent TestCenter ports act as both multicast clients and sources with the DUT in
between. The DUT shouldn’t forward multicast traffic until it receives a request from the client. It
should process multicast Join/Leave requests from the client with the minimum possible latency.
This is critical for presenting the user with a good quality of experience, especially with video
applications.
Target Users
All NEMs and service providers
Target Device Under Test (DUT)
Core Equipment
Reference
RFC 3918
Relevance
This test case show cases the DUT’s capability to handle Join/Leave requests from the Multicast
Clients.
Version
1.0
Test Category
Testing Benchmarking Standards
PASS
[x] Performance [x] Availability [ ] Security [] Scale
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
5
Required Tester Capabilities
The tester must support:
RFC 3918
Multicast protocols support – IGMP/MLD
Results reporting capabilities in a template format with key
Topology Diagram
M u l t i c a s t
C l i e n t s
D U T
M u l t i c a s t
S e r v e r
Test Procedure
1. Launch the RFC 3918 Wizard in the Spirent TestCenter GUI.
2. Select the Multicast Join/Leave Test.
3. Select the ports that will be used in the test.
a. Configure multicast hosts and the group automatically with the wizard or manually.
4. Select the endpoint mapping and multicast source and client ports.
5. Configure the following parameters
a. Multicast Client Version (IGMP Version).
b. Join Group Delay.
c. Leave Group Delay.
d. Multicast Message Tx Rate.
e. Multicast Group Base IP Address, Step and the how much should it Increment in each
step.
f. Layer 4 Header – None, TCP, UDP.
i. If TCP/UDP is selected, give a port number range.
g. TOS or Class of Service.
h. VLAN P-bit, if any.
i. TTL.
j. Latency Type.
i. Selection from LILO, FIFO, FILO or LIFO.
k. The Multicast Group Distribution Mode – Even or Weighed, between Client ports, if
more than 1.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
6
6. Configure the test options.
a. Number of Trials to be run.
b. Duration in seconds.
c. Test Start Delay – for the DUT to be able to ramp up.
d. Frame Size.
i. Option to have Fixed, Random, Step, Custom or iMIX.
e. ARP/Learning parameters.
f. Results Collection Delay – after the test has finished, the analyzer will wait this amount
of time before it calculates the results.
7. Finish the wizard. It automatically creates a sequence of steps in the Command Sequencer.
8. Run the Command Sequencer and allow the Results Reporter to open up when the first
iteration has finished.
9. The Results Reporter tool launches when the first iteration is complete and displays the
results in a pre-defined template with all the necessary information.
10. Create a PDF, HTML or Excel report from the template if desired.
11. Run the test for IPv6/MLD if desired.
12. End of test case.
Control Variables & Relevance
Variable Relevance Default Value
Number of IGMP
Groups
The more the number of IGMP Groups, more
processing required on the DUT
1
IGMP Version This would be normally Version 2 or 3. 2
MLD Version This would be normally Version 2 or 3 2
Number of Multicast
Hosts
This defines the number of Multicast Clients per port –
usually the number of Multicast Clients and Groups
are in 1:1 ratio
1
IGMP/MLD Group
Addresses
The Starting Group Address – should be a Class D
address. There are some Class D addresses, which are
private and should not be used.
225.0.0.1
Multicast Group
Address Step
Which octet to increment if we have more than 1
Groups
/8
TOS Class of Service for the Multicast packets 0
IP TTL Time to leave for the multicast packets 10
Multicast Group
Distribution Mode
How the number of groups are distributed amongst
the ports, if more than one
Even
Latency Type The way latency is calculated FIFO
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
7
Key Measured Metrics
Matric Relevance Metric Unit
Join Latency The time it takes for the first Multicast packet to arrive on
the client port from the moment a join message is sent for
the particular group
Microseconds
Leave Latency Time it takes for the DUT to stop forwarding the Multicast
packets for a particular group after a Leave has been sent
Microseconds
Desired Result
The DUT should be able to process the multicast Join/Leaves as fast as possible.
Analysis
The DUT should be able to process the multicast Join/Leaves as fast as possible for multiple Joins
and Leaves happening together.
Typically, a Join latency will be lower than a Leave latency but there shouldn’t be too much of a
gap.
A high Join/Leave latency indicates a significant issue with the DUT processing engine as well as
the buffers.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
8
Appendix A – Telecommunications
Definitions
APPLICATION LOGIC. The computational aspects of an application, including a list of instructions that tells a
software application how to operate.
APPLICATION SERVICE PROVIDER (ASP). An ASP deploys hosts and manages access to a packaged application by
multiple parties from a centrally managed facility. The applications are delivered over networks on a
subscription basis. This delivery model speeds implementation, minimizes the expenses and risks incurred
across the application life cycle, and overcomes the chronic shortage of qualified technical personnel
available in-house.
APPLICATION MAINTENANCE OUTSOURCING PROVIDER. Manages a proprietary or packaged application from
either the customer's or the provider's site.
ASP INFRASTRUCTURE PROVIDER (AIP). A hosting provider that offers a full set of infrastructure services for
hosting online applications.
ATM. Asynchronous Transport Mode. An information transfer standard for routing high-speed, high-
bandwidth traffic such as real-time voice and video, as well as general data bits.
AVAILABILITY. The portion of time that a system can be used for productive work, expressed as a
percentage.
BACKBONE. A centralized high-speed network that interconnects smaller, independent networks.
BANDWIDTH. The number of bits of information that can move through a communications medium in a
given amount of time; the capacity of a telecommunications circuit/network to carry voice, data, and
video information. Typically measured in Kbps and Mbps. Bandwidth from public networks is typically
available to business and residential end-users in increments from 56 Kbps to 45 Mbps.
BIT ERROR RATE. The number of transmitted bits expected to be corrupted per second when two computers
have been communicating for a given length of time.
BURST INFORMATION RATE (BIR). The rate of information in bits per second that the customer may need over
and above the CIR. A burst is typically a short duration transmission that can relieve momentary
congestion in the LAN or provide additional throughput for interactive data applications.
BUSINESS ASP. Provides prepackaged application services in volume to the general business market,
typically targeting small to medium size enterprises.
BUSINESS-CRITICAL APPLICATION. The vital software needed to run a business, whether custom-written or
commercially packaged, such as accounting/finance, ERP, manufacturing, human resources and sales
databases.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
9
BUSINESS SERVICE PROVIDER. Provides online services aided by brick-and-mortar resources, such as payroll
processing and employee benefits administration, printing, distribution or maintenance services. The
category includes business process outsourcing (BPO) companies.
COMMERCE NETWORK PROVIDER. Commerce networks were traditionally proprietary value-added networks
(VANs) used for electronic data interchange (EDI) between companies. Today the category includes the
new generation of electronic purchasing and trading networks.
COMPETITIVE ACCESS PROVIDER (CAP). A telecommunications company that provides an alternative to a LEC
for local transport and special access telecommunications services.
CAPACITY. The ability for a network to provide sufficient transmitting capabilities among its available
transmission media, and respond to customer demand for communications transport, especially at peak
usage times.
CLIENT/DEVICE. Hardware that retrieves information from a server.
CLUSTERING. A group of independent systems working together as a single system. Clustering technology
allows groups of servers to access a single disk array containing applications and data.
COMPUTING UTILITY PROVIDER (CUP). A provider that delivers computing resources, such as storage, database
or systems management, on a pay-as-you-go basis.
CSU/DSU. Channel Server Unit/Digital Server Unit. A device used to terminate a telephone company
connection and prepare data for a router interface.
DATA MART. A subset of a data warehouse, intended for use by a single department or function.
DATA WAREHOUSE. A database containing copious amounts of information, organized to aid decision-
making in an organization. Data warehouses receive batch updates and are configured for fast online
queries to produce succinct summaries of data.
DEDICATED LINE. A point-to-point, hardwired connection between two service locations.
DEMARCATION LINE. The point at which the local operating company's responsibility for the local loop ends.
Beyond the demarcation point (also known as the network interface), the customer is responsible for
installing and maintaining all equipment and wiring.
DISCARD ELIGIBILITY (DE) BIT. Relevant in situations of high congestion, it indicates that the frame should be
discarded in preference to frames without the DE bit set. The DE bit may be set by the network or by the
user; and once set cannot be reset by the network.
DS-1 OR T-1. A data communication circuit capable of transmitting data at 1.5 Mbps. Currently in
widespread use by medium and large businesses for video, voice, and data applications.
DS-3 OR T-3. A data communications circuit capable of transmitting data at 45 Mbps. The equivalent data
capacity of 28 T-1s. Currently used only by businesses/institutions and carriers for high-end applications.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
10
ELECTRONIC DATA INTERCHANGE (EDI). The electronic communication of business transactions (orders,
confirmations, invoices etc.) of organizations with differing platforms. Third parties provide EDI services
that enable the connection of organizations with incompatible equipment.
ENTERPRISE ASP. An ASP that delivers a select range of high-end business applications, supported by a
significant degree of custom configuration and service.
ENTERPRISE RELATIONSHIP MANAGEMENT (ERM). Solutions that enable the enterprise to share comprehensive,
up-to-date customer, product, competitor and market information to achieve long-term customer
satisfaction, increased revenues, and higher profitability.
ENTERPRISE RESOURCE PLANNING (ERP). An information system or process integrating all manufacturing and
related applications for an entire enterprise. ERP systems permit organizations to manage resources
across the enterprise and completely integrate manufacturing systems.
ETHERNET. A local area network used to connect computers, printers, workstations, and other devices
within the same building. Ethernet operates over twisted wire and coaxial cable.
EXTENDED SUPERFRAME FORMAT. A T1 format that provides a method for easily retrieving diagnostics
information.
FAT CLIENT. A computer that includes an operating system, RAM, ROM, a powerful processor and a wide
range of installed applications that can execute either on the desktop or on the server to which it is
connected. Fat clients can operate in a server-based computing environment or in a stand-alone fashion.
FAULT TOLERANCE. A design method that incorporates redundant system elements to ensure continued
systems operation in the event of the failure of any individual element.
FDDI. Fiber Distributed Data Interface. A standard for transmitting data on optical-fiber cables at a rate of
about 100 Mbps.
FRAME. The basic logical unit in which bit-oriented data is transmitted. The frame consists of the data bits
surrounded by a flag at each end that indicates the beginning and end of the frame. A primary rate can be
thought of as an endless sequence of frames.
FRAME RELAY. A high-speed packet switching protocol popular in networks, including WANs, LANs, and
LAN-to-LAN connections across long distances.
GBPS. Gigabits per second, a measurement of data transmission speed expressed in billions of bits per
second.
HOSTED OUTSOURCING. Complete outsourcing of a company's information technology applications and
associated hardware systems to an ASP.
HOSTING PROVIDER. Provider who operates data center facilities for general-purpose server hosting and
collocation.
INFRASTRUCTURE ISV. And independent software vendor that develops infrastructure software to support
the hosting and online delivery of applications.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
11
INTEGRATED SERVICES DIGITAL NETWORK (ISDN). An information transfer standard for transmitting digital voice
and data over telephone lines at speeds up to 128 Kbps.
INTEGRATION. Equipment, systems, or subsystem integration, assembling equipment or networks with a
specific function or task. Integration is combining equipment/systems with a common objective, easy
monitoring and/or executing commands. It takes three disciplines to execute integration: 1) hardware, 2)
software, and 3) connectivity – transmission media (data link layer), interfacing components. All three
aspects of integration have to be understood to make two or more pieces of equipment or subsystems
support the common objective.
INTER-EXCHANGE CARRIER (IXC). A telecommunications company that provides telecommunication services
between local exchanges on an interstate or intrastate basis.
INTERNET SERVICE PROVIDER (ISP). A company that provides access to the Internet for users and businesses.
INDEPENDENT SOFTWARE VENDOR (ISV). A company that is not a part of a computer systems manufacturer
that develops software applications.
INTERNETWORKING. Sharing data and resources from one network to another.
IT SERVICE PROVIDER. Traditional IT services businesses, including IT outsourcers, systems integrators, IT
consultancies and value added resellers.
KILOBITS PER SECOND (KBPS). A data transmission rate of 1,000 bits per second.
LEASED LINE. A telecommunications line dedicated to a particular customer along predetermined routers.
LOCAL ACCESS TRANSPORT AREA (LATA). One of approximately 164 geographical areas within which local
operating companies connect all local calls and route all long-distance calls to the customer's inter-
exchange carrier.
LOCAL EXCHANGE CARRIER (LEC). A telecommunications company that provides telecommunication services
in a defined geographic area.
LOCAL LOOP. The wires that connect an individual subscriber's telephone or data connection to the
telephone company central office or other local terminating point.
LOCAL/REGIONAL ASP. A company that delivers a range of application services, and often the complete
computing needs, of smaller businesses in their local geographic area.
MEGABITS PER SECOND (MBPS). 1,024 kilobits per second.
METAFRAME. The world's first server-based computing software for Microsoft Windows NT 4.0 Server,
Terminal Server Edition multi-user software (co-developed by Citrix).
MODEM. A device for converting digital signals to analog and vice versa, for data transmission over an
analog telephone line.
MULTIPLEXING. The combining of multiple data channels onto a single transmission medium. Sharing a
circuit - normally dedicated to a single user - between multiple users.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
12
MULTI-USER. The ability for multiple concurrent users to log on and run applications on a single server.
NET-BASED ISV. An ISV whose main business is developing software for Internet-based application services.
This includes vendors who deliver their own applications online, either directly to users or via other
service providers.
NETWORK ACCESS POINT (NAP). A location where ISPs exchange traffic.
NETWORK COMPUTER (NC). A thin-client hardware device that executes applications locally by downloading
them from the network. NCs adhere to a specification jointly developed by Sun, IBM, Oracle, Apple and
Netscape. They typically run Java applets within a Java browser, or Java applications within the Java
Virtual Machine.
NETWORK COMPUTING ARCHITECTURE. A computing architecture in which components are dynamically
downloaded from the network onto the client device for execution by the client. The Java programming
language is at the core of network computing.
ONLINE ANALYTICAL PROCESSING (OLAP). Software that enables decision support via rapid queries to large
databases that store corporate data in multidimensional hierarchies and views.
OPERATIONAL RESOURCE PROVIDER. Operational resources are external business services that an ASP might
use as part of its own infrastructure, such as helpdesk, technical support, financing, or billing and payment
collection.
OUTSOURCING. The transfer of components or large segments of an organization's internal IT infrastructure,
staff, processes or applications to an external resource such as an ASP.
PACKAGED SOFTWARE APPLICATION. A computer program developed for sale to consumers or businesses,
generally designed to appeal to more than a single customer. While some tailoring of the program may be
possible, it is not intended to be custom-designed for each user or organization.
PACKET. A bundle of data organized for transmission, containing control information (destination, length,
origin, etc.), the data itself, and error detection and correction bits.
PACKET SWITCHING. A network in which messages are transmitted as packets over any available route rather
than as sequential messages over circuit-switched or dedicated facilities.
PEERING. The commercial practice under which nationwide ISPs exchange traffic without the payment of
settlement charges.
PERFORMANCE. A major factor in determining the overall productivity of a system, performance is primarily
tied to availability, throughput and response time.
PERMANENT VIRTUAL CIRCUIT (PVC). A PVC connects the customer's port connections, nodes, locations, and
branches. All customer ports can be connected, resembling a mesh, but PVCs usually run between the
host and branch locations.
POINT OF PRESENCE (POP). A telecommunications facility through which the company provides local
connectivity to its customers.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
13
PORTAL. A company whose primary business is operating a Web destination site, hosting content and
applications for access via the Web.
REMOTE ACCESS. Connection of a remote computing device via communications lines such as ordinary
phone lines or wide area networks to access distant network applications and information.
REMOTE PRESENTATION SERVICES PROTOCOL. A set of rules and procedures for exchanging data between
computers on a network, enabling the user interface, keystrokes, and mouse movements to be
transferred between a server and client.
RESELLER/VAR. An intermediary between software and hardware producers and end users. Resellers
frequently add value (thus Value-Added Reseller) by performing consulting, system integration and
product enhancement.
ROUTER. A communications device between networks that determines the best path for optimal
performance. Routers are used in complex networks of networks such as enterprise-wide networks and
the Internet.
SCALABILITY. The ability to expand the number of users or increase the capabilities of a computing solution
without making major changes to the systems or application software.
SERVER. The computer on a local area network that often acts as a data and application repository and that
controls an application's access to workstations, printers and other parts of the network.
SERVER-BASED COMPUTING. A server-based approach to delivering business-critical applications to end-user
devices, whereby an application's logic executes on the server and only the user interface is transmitted
across a network to the client. Benefits include single-point management, universal application access,
bandwidth-independent performance, and improved security for business applications.
SINGLE-POINT CONTROL. One of the benefits of the ASP model, single-point control helps reduce the total
cost of application ownership by enabling widely used applications and data to be deployed, managed
and supported at one location. Single-point control enables application installations, updates and
additions to be made once, on the server, which are then instantly available to users anywhere.
SPECIALIST ASP. Provide applications which serve a specific professional or business activity, such as
customer relationship management, human resources or Web site services.
SYSTEMS MANUFACTURER. Manufacturer of servers, networking and client devices.
TELECOMS PROVIDER. Traditional and new-age telecommunications network providers (telcos).
THIN CLIENT. A low-cost computing device that accesses applications and and/or data from a central server
over a network. Categories of thin clients include Windows-Based Terminals (WBT, which comprise the
largest segment), X-Terminals, and Network Computers (NC).
TOTAL COST OF OWNERSHIP (TCO). Model that helps IT professionals understand and manage the budgeted
(direct) and unbudgeted (indirect) costs incurred for acquiring, maintaining and using an application or a
computing system. TCO normally includes training, upgrades, and administration as well as the purchase
price. Lowering TCO through single-point control is a key benefit of server-based computing.
Spirent Journal of Benchmark Test Methodologies | © Spirent Communications 2011
14
TOTAL SECURITY ARCHITECTURE (TSA). A comprehensive, end-to-end architecture that protects the network.
TRANSMISSION CONTROL PROTOCOL/INTERNET PROTOCOL (TCP/IP). A suite of network protocols that allow
computers with different architectures and operating system software to communicate over the Internet.
USER INTERFACE. The part of an application that the end user sees on the screen and works with to operate
the application, such as menus, forms and buttons.
VERTICAL MARKET ASP. Provides solutions tailored to the needs of a specific industry, such as the healthcare
industry.
VIRTUAL PRIVATE NETWORK (VPN). A secure, encrypted private connection across a cloud network, such as
the Internet.
WEB HOSTING. Placing a consumer's or organization's web page or web site on a server that can be
accessed via the Internet.
WIDE AREA NETWORK. Local area networks linked together across a large geographic area.
WINDOWS-BASED TERMINAL (WBT). Thin clients with the lowest cost of ownership, as there are no local
applications running on the device. Standards are based on Microsoft's WBT specification developed in
conjunction with Wyse Technology, NCD, and other thin client companies.
