Jump to content, skipping navigation

White Paper: Testing 40G/100G Ethernet

Get Ready for the New Era of Voice, Data, and Video Performance

High Speed Ethernet solutions pose a colossal challenge for lab managers everywhere—from moving bits at 100Gbps error free, to scaling the upper-layer engines to perform at four to ten times the current rate.

Testing 40-100G Ethernet Whitepaper Cover

As with any new technology, the key to successful implementation is testing

40/100G Ethernet brings a new set of issues to the development cycle, including challenges at the physical layer and the critical choice of test platform.

Find out how can you overcome these challenges to improve ROI and reduce TCO on your testing solutions.

    * Required Field

    Cancel

    007.000 009.000 011.000 015.000 ll, , , ,, ll, ,, ,,ll, , ,,,ll, ,, , , ll,,,,,ll , , , ,,ll , ,, ,, ll, , , ,, ll , , , , , ll , , ,,, ll, , , , ,l l ll, , , , ,ll,,,, , ll,, , , , ll,, , , , ll , , , , , ll,, , , , ll ,,,,, ll ,,, , , ll ,, , ,, ll ,, , , ,ll ,, ,, , l l TESTING 40/100G ETH E RNE T A white paper from ARE YOU READY? TesTing 40/100g eTherneT: Are You reAdY? A white paper from Spirent Communications ConTenTs Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Market Drivers for 40/100G Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 The Spec at a Glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Testing: 40/100G Ethernet Is a Game Changer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Challenges in the physical layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Limitations in the test system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Budget in the test lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 Delays in the development/deployment cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Selecting a Test Platform for 40/100G Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 p2© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? inTroduCTion The multi-year process is drawing to a close and the release of 40/100G Ethernet solutions is imminent . Obviously there is the challenge of moving bits at 100 Gbps, error free . But equally challenging, if not more so, is scaling the upper-layer engines to deliver services and quality of experience at four to ten times the current rate . Brand credibility and customer confidence depend on solving both problems . As with any new technology, the key to success is testing . But 40/100G Ethernet brings a new set of issues to the develop- ment cycle . First are the challenges at the physical layer . Second is the critical issue of the test platform itself . There’s more to being 40/100G ready than providing a 40G or 100G Ethernet interface on the system . Test platform limitations can affect the budget when equipping the test lab with 40G Ethernet or 100G Ethernet test ports . Finally there is the development and testing cycle to consider . Manual test administration and multiple test runs to gather a full set of test results can push out a release date by days or weeks . This white paper explores these issues and suggests ways in which you can improve ROI and TCO for your test systems, including 40/100G capabilities, while you also reduce development costs and improve time-to-revenue . MArkeT drivers for 40/100g eTherneT As usual, the demand for bandwidth is pushing the limits of the telecommunications infrastructure . In 2012, the traffic on the internet is expected to be six times 2007 levels . • Video: The culprit most often fingered is video . YouTube has experienced phenomenal growth since its incep- tion, with monthly growth rates of 20% to 50% in some time periods . As telcos compete more heavily with cable providers, IPTV and video on demand continue to increase . HDTV (19 Mbps) uses over five times as much bandwidth as SDTV (3 .5 Mbps) . • Wireless: Video applications on wireless devices, along with the many other applications associated with the explosion of wireless that seems impervious to recession, is responsible for pushing the limits of the backhaul network . • Social networking: Consumers with low-cost digital cameras upload countless photos of the latest vacation, graduation, or grandkids, typically at full resolution . • Medical: Remote collaboration, consultation and diagnosis is increasing . An MRI can generate 500 Mbytes of data per hour . Genome mapping generates massive amounts of data • Business applications: CRM and ERP solutions, and design applications used in a variety of fields, from silicon development to construction, are frequently used remotely and generate a significant amount of traffic . Server mirroring and backup to remote locations pushes large amounts of data across the network as more companies implement disaster recovery and business continuity plans . • Storage and data centers: Storage networks continue to grow and data center virtualization represents a signifi- cant increase in inter-server traffic as process are spread across multiple physical machines, or virtual machine instances are moved to another physical machine . • 10G interfaces: Servers and, in some high-speed computing cases, even workstations, are now equipped with 10 Gbps Ethernet interfaces, which will have to be aggregated to a higher bandwidth to accommodate the traf- fic across the backbone . • Equities trading industry: Demands for large volumes of data (billions of transactions and terabytes of informa- tion per day) at extremely low latencies is a matter of survival . A small increase in latency can create a large drop in revenue . Surplus bandwidth is essential . As you can see, there is no shortage of candidates with high-bandwidth demands to drive the move to 40/100G Ethernet . p3© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? The speC AT A glAnCe The IEEE standard specifies a number of physical interfaces for both speeds, with their corresponding distance limitations . The optical implementations use multiple fibers or wavelengths as 10G or 25G lanes between endpoints in a link . The trans- mitting system splits a serial 40G or 100G stream into four or ten parallel lanes . The receiving system re-forms the four or ten lanes into a single 40G or 100G stream . Lanes are utilized at several levels in an interface . The diagram shows some examples of possible lane changes between sublayers in the interface . The numbers in parentheses show the number of lanes entering and exiting a layer . There is no requirement in the specification for a static mapping of virtual lanes to physical lanes . Lane swapping can occur . 100gBAse-r implementations Standard Media Distance 40GBase-SR4 MMF, 4 fibers (OM3) 100 m 40GBase-LR4 SMF, 4 wavelengths 10 km 40GBase-CR4 Copper 10 m 40GBase-KR4 Backplane 1 m 100GBase-SR10 MMF, 10 fibers (OM3) 100 m 100GBase-LR4/ER4 SMF, 4 wavelengths 10 km 100GBase-CR10 Copper 10 m MDI MDI 100GBASE-R PCS PMA (10:4) PMD (X4) MEDIUM PMA (10:20) FEC PMA (20:10) PMA (20:10) CAUI PMD SERVICE INTERFACE MDI 100GBASE-R PCS PMA (10:20) MEDIUM PMA (20:10) PMD (X10) PMA (20:10) 100GBASE-R PCS MEDIUM PMA (10:20) FEC PMA (20:10) PMD (X10) PMA (20:10) 100GBASE-R PCS MEDIUM PMA (10:4) PMD (X4) MDI PMA (20:10) 100GBASE-R PCS PMD (X10) MEDIUM PMA (20:10) CAUI CAUI CAUI CAUI CAUI PMD SERVICE INTERFACE 100GBASE-R PCS MEDIUM MDI MDI PMA (20:10) FEC PMD (X10) CAUI : 100Gb/s Attachment Interface FEC : Forward Error Correction MAC : Media Access Control MDI : Medium Dependent Interface PCS : Physical Coding Layer PMA : Physical Medium Attachment PMD : Physical Medium Dependent p4© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? At the PCS layer, the traffic exists in 20 virtual lanes . As the traffic is processed in the lower layers (PMA and PMD), the number of lanes changes, depending on the implementation . For example, in 100GBASE-R4, the lanes go from 20 in the PCS layer to 10 in the PMA layer to 4 in the PMD layer . As in 10GBase-LX4, in implementations where lanes exist on the physical medium, two issues can arise, swapping of lanes and skew between lanes based on different flight times from one set of optics to the other, introduced by imperfections in the electrical or optical interfaces or media . The IEEE specification addresses skew with alignment blocks . The sending system inserts an alignment block into the 40/100G stream periodically, before it is split, to allow the receiving system to identify bits from each lane that should be arriving simultaneously . The receiving system maintains alignment between the lanes by compensating for any inter-lane skew . TesTing: 40/100g eTherneT is A gAMe ChAnger While it’s true that 40/100G Ethernet is still Ethernet, just faster, there’s a lot packed into that phrase “just faster .” In some ways it’s more of the same, in others it changes everything . At the upper layers, that means each component in a device or step in a process must accomplish the same thing it currently does in 1/10th of the time . Consider a router, for example, which strips lower-layer information from an incoming packet, queues it, performs a route lookup, and sends it to the proper outbound queue to be packetized, while performing filtering, SLA monitoring and policing, and CoS/QoS prioritization . In addition, a router also exchanges per VPN MPLS label informa- tion, builds multicast routing trees, performs routing table updates for multiple protocols, maintains statistics and perfor- mance, alarm, event and failure logs, and performs firewall and security functions, such as key exchanges, attack detection and prevention . A router with 100G interfaces must do all this at ten times current maximum speeds without dropping packets, introducing excessive jitter, compromising VPN boundaries, or reordering packets, which is especially disruptive for storage and high- bandwidth video . Testing 40/100G solutions starts with validating the transport and the ability of the system under test (SUT) to pass line-rate traffic, but also includes testing the functionality, performance, scalability and QoE of the upper-layer engines that deliver services . There are implications for the test lab budget, the development cycle, and possible limitations of the test platform itself . ChAllenges in The phYsiCAl lAYer Validating 40/100G Ethernet at Layer 1 is about the basics –moving bits error free . At different sublayers, that means dif- ferent things . Physical Medium Dependent (PMD) Testing the PMD layer is the domain of lower-level testing instruments such as oscilloscopes . Physical Medium Attachment (PMA) Testing the PMA layer involves sending bit patters, such as pseudo-random bit sequences (PRBS), through the system and detecting errors, such as bit errors or pattern synchro- nization issues . Loopback testing is also important at this level . Physical Coding Sublayer (PCS) Testing the PCS layer involves introducing lane swapping and lane skew . The test system swaps lanes to determine if the SUT will detect and compensate for the swap . To validate skew tolerance and compensation algorithms, the test system introduces skew from the transmit port . On the receive port, the amount of skew present after compen- sation by the system under test is reported . The IEEE speci- fications indicate the amount of skew a system should be able to compensate for . Skew testing compares the ability of the system under test to support the standard, or report the degree to which it doesn’t match the specification . 100GBASE-R PCS PMA (10:20) PMA (10:4) PMD (X4) MEDIUM MDI CAUI CAUI : 100Gb/s Attachment Unit Interface MDI : Medium Dependent Interface PCS : Physical Coding Sub-Layer PMA : Physical Medium Attachment PMD : Physical Medium Dependent p5© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? Media Access Control (MAC) The MAC layer is actually at Layer 2, but has similar testing issues to Layer 1 . Testing the MAC layer involves measuring the ability of the SUT to transmit and receive frames and handle frame check sequence (FCS) errors . The test platform sends traffic at a range of desired rates, such as bytes per second or frame per second, and provides metrics of received traffic to compare to what was transmitted . The test system also introduces FCS errors, as well as counting FCS errors received . LIMItAtIonS In thE tESt SyStEM It might seem that, from a testing perspective, regardless of bit rates, the application or service performance require- ments haven’t changed . The metrics of interest remain the same . Video quality testing means measuring VMOS and application performance means measuring transaction response times . It’s just a question of whether the system under test can keep up with the bit rate . Or is it? Actually, it’s also a question of whether the test system can keep up . To get the metrics of interest, such as packet count, loss, sequence errors, latency and jitter, the test system must be able to deliver them . Here’s where a fairly mundane, but ultimately critical, component comes into play, the system clock on the test platform . A test system counts packets and computes latency and jitter by placing a proprietary signature in the test packets it generates . The signature includes a serial number and a timestamp, critical elements for calculating the metrics of inter- est . For example, to measure latency, the test system records the time the packet is transmitted in the timestamp field . When the packet is received, the time is noted . To calculate latency, how long it took the packet to travel the link, the test system subtracts the transmit timestamp from the receive timestamp . For this calculation to have any meeting, the transmit port and receive port must be synchronized . As with any measurement, the precision is important . It’s the plus-or-minus value that follows a measurement . Precision is the basis of reproducibility or repeatability . If you’re measuring the length of your vacation, assuming you have the time to take one, a calendar is fine . But plus or minus one sunrise isn’t useful when you’re timing boiled eggs . You need an instrument with a greater level of precision, a clock, which can measure durations down to a minute, or even a frac- tion of a minute . For purposes of the timestamp, the resolution of the clock on the test system is the limiting factor in the precision of the result . For a 10G Ethernet system, a clock resolution of 20 nanoseconds is typical . This means any measurement you make is +- 20 ns . In a storage environment, where latency requirements are in the single-digit microsecond range, this works out well . A reported latency of 1 µs means the actual latency is between 0 .98 µs and 1 .02 µs . However, if you are dealing with single-digit nanosecond values, +- 20 ns is not adequate precision . To say a number reported as 30 ns is actually somewhere between 10 ns and 50 ns is not helpful when single digits make a difference . And here is where clock speed can create testing problems for 40/100G Ethernet . To uniquely timestamp every packet, the resolution of the test system clock must be less than the time it takes to trans- mit a minimum-sized Ethernet frame of 64 bytes, a preamble of 8 bytes, and the minimum inter-frame gap of 12 bytes . Regardless of the speed of Ethernet in use, a packet can be transmitted every 672 bit times . There must be at least one timestamp clock tick for every frame . You can’t have two packets fall within a single tick . WHAT TO LOOK FOR A test platform with the flexibility to stress the SUT at Layer 1-2 and analyze bit transmission, lane and framing issues . p6© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? For 10G Ethernet, a 20-ns resolution clock works fine . It takes 67 .2 ns to transmit a 64-byte frame . Every frame spans at least four clock ticks . For 40G Ethernet, problems begin to arise with a 20-ns resolution clock . It takes 16 .8 ns to transmit a 64-byte frame . It doesn’t take long for frames to begin overlapping one tick of the timestamp clock . Latency and jitter measurements will not be accurate in this scenario . For 100G Ethernet, accuracy suffers even more with a 20-ns resolution clock . It takes 6 .72 ns to transmit a 64-byte frame . Every tick of the timestamp clock contains three frames . In this scenario, the test system cannot know the time the packet was transmitted with the required precision . Capturing the metrics of interest mentioned earlier (packet count, loss, sequence errors, latency and jitter) requires a clock resolu- tion smaller than the time required to transmit 64-byte frames on the system under test . 64-byte Ethernet frame 64-byte Ethernet frame 0 20 40 60 80 Time (ns) 20-ns resolution clock RATES 10G 100 120 140 160 67.2 67.2 Multiple clock ticks per frame 64-byte Ethernet frame 64-byte Ethernet frame 64-byte Ethernet frame 64-byte Ethernet frame 64-byte Ethernet frame 64-byte Ethernet frame 64-byte Ethernet frame 0 20 40 60 80 Time (ns) 20-ns resolution clock RATES 40G 100 120 140 160 16.8 16.8 16.8 16.8 16.8 16.8 16.8 Multiple Frames per clock tick 64-byte Ethernet frame Multiple Frames per clock tick 0 20 40 60 80 Time (ns) 20-ns resolution clock RATES 100G 100 120 140 160 6.72 p7© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? In multi-chassis tests, the issue of clock resolution goes beyond generating an internal timestamp to synchronizing clocks across multiple systems . Time-based measurements such as latency and jitter involve subtracting the transmit time from the receive time . If the transmit and receive ports are on different systems, the clocks of those systems must be precisely synchronized for those measurements to have meaning . BUDGEt In thE tESt LAB Keeping releases on schedule involves anticipating equipment requirements, budgeting for necessary equipment, and getting it installed and operational before the test cycle begins . Obviously, testing 40/100G Ethernet necessitates budget- ing for the proper interfaces for the test bed . However, there is more to full application and service testing than install- ing an interface that supports the specified bit rates . In the previous section we explored possible limitations in the test system that can affect test coverage . Due diligence is required at an early stage when establishing 40/100G testing requirements and component or system purchases . When developing the budget for 40/100G testing upgrades, it is important to establish what is involved in supporting higher bit rates on a specific test platform . Not all backplanes or system clocks are created equal . Is the expense limited to purchasing interfaces that support the bit rates, or will a chassis upgrade, or even forklift replacement, be required? Thorough research is essential . A seemingly insignificant detail such as clock resolution can affect the ROI of your test lab investment, as well as TCO for the equipment purchased . The time to ask the hard questions, such as can the sys- tem scale to test routing at high speeds, or stream and analyze real video, is before the purchase is made . DELAyS In thE DEVELoPMEnt/DEPLoyMEnt CyCLE For vendors, the development cycle begins with establishing requirements and coding, continues through release of a build for testing, and ends when testing on a build is complete . In a development process where features are released to testing incrementally, subsequent builds will include new features and fixes for bugs discovered during a previous testing cycle . Ultimately, the iterations of the test cycle end in a product release . For carriers, the deployment cycle is similar . Requirements are established, followed by design, configuration and testing . Multiple iterations may be required before deployment to achieve the desired performance . The test portion of the cycle involves running tests and analyzing the results . Due to the significant increase in band- width associated with the move to 40/100G, there is a corresponding increase in test results to be analyzed . More of everything must be supported – more routes, more VPNs, more tunnels, more queues . In the 40/100G environment, the efficiency (or lack of efficiency) of the test platform is magnified . It becomes even more important to keep the number of test runs to a minimum without sacrificing test coverage and product quality . It is commonly known that extensive manual testing is a source of delay in the test portion of the cycle . Automation is an essential element in reducing test time, particularly with the greater test demands imposed by 40/100G Ethernet . Most test platforms include some level of automation, but the coverage varies significantly between platforms . Test case automation is the most common form of automation supported . A TCL script or saved configuration is used to quickly configure and run a test without intervention from the lab engineer . The gains are obvious – tests can be run without the direct involvement of a highly-trained resource, freeing that person to perform other tasks vital to complet- ing the test cycle . However, when a test fails, manual testing is often required . With a test platform that supports full integration between the API and the GUI, valuable time is saved by avoiding manual (which means error-prone) configuration via the GUI to WHAT TO LOOK FOR A test platform with a timestamp clock resolution small enough to uniquely timestamp each frame and synchronize all chassis with that level of precision . WHAT TO LOOK FOR WHAT TO LOOK FOR A test platform that supports more than simple Layer 1-2 metrics, without upgrades or replacements . Buy new interfaces without having to buy new chassis . A Layer 1-7 test platform that scales to test the routes, VPNs, tunnels and VLANs required to deliver services at 40/100G . p8© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready? reproduce the test environment . As a result, defects are found and resolved faster . Some test platforms support the ability to automate without writing code through a drag-and-drop command sequencer interface . This means engineers with technology knowledge but without scripting skills can still automate tests without the overhead of a scripting language learning curve . It also highlights the importance of integration in the other direc- tion, exporting a test to a script that has been optimized via the GUI . The step beyond test case automation is test lab automation . This can be as simple as a scheduling application that manages and executes suites of test cases, perhaps with results management and the ability to do conditional testing based on the outcome of a test . Or it can go beyond managing test cases and suites to configuration of test topologies, including automated cabling, DUT configuration, and power management . The power of reduCing The developMenT CYCle Reducing the time and effort required to complete the development/deployment cycle delivers a double benefit to your company’s profitability . There are not many places in the organization where increasing efficiency can have such a positive ef- fect . By shortening development cycles, you improve time-to-revenue (increasing top-line sales) and reduce OPEX (increasing bottom-line profits on the already increased top line) . What is more, when you release or deploy products faster while improving quality, you go beyond improving profitability to improving customer confidence and brand credibility . seleCTing A TesT plATforM for 40/100g eTherneT It becomes clear that vetting test platforms to support the effort to deliver services over 40/100G Ethernet is not simply a matter of buying new interfaces . You will want to consider the questions discussed above while making the decision . Spirent Communications has been developing solutions with a view toward support of high bit rates for years . Spirent TestCenter is an integrated Layer 1-7 test platform designed from inception to support testing at bit rates from 10 Mbps to 100 Gbps and beyond . The system clock has a resolution of 2 .5 ns, more than twice as fast as required to support Ethernet at 100 Gbps . All ports in a chassis and all chassis in a test can be synchronized at this level of precision to provide the ac- curacy required to measure the critical metrics used to evaluate the performance and QoE of services delivered over 40/100G Ethernet . Spirent TestCenter Intelligent Results deliver the industry’s most accurate and comprehensive set of real-time results, sup- ported by data correlation and hierarchical drill-down analysis for transmit and receive statistics, all available in a single test run . Spirent NoCodeTM automation provides code-free automation with 100% integration between the GUI and the API . End-to- end automation includes automated control of test cases, test cycles, the physical infrastructure and the test automation environment . Sprient TestCenter should certainly be among the platforms you consider when making this decision . ConClusion Delivering services over 40/100G Ethernet with high customer QoE will not be achieved by business as usual in the test lab . It not only places much greater demand on the routers, switches and other devices with these interfaces, it also places greater demand on the test systems that will validate their performance . Lab managers preparing for 40/100G Ethernet must make an assessment of their current test systems to verify that they are capable of providing the metrics required to test the new speeds . WHAT TO LOOK FOR What to look for: A test platform that supports code-free automation, 100% integration between the GUI and the API, and end-to-end automation, not just test case automation . p9© 2009 Spirent Communications, Inc Testing 40/100G Ethernet: Are You Ready?