Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies

You are here:: Home >; Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies
<Back to Home

Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies

Cloud Infrastructure Enterprise storage is a centralized repository for business information that provides common data management and protection, as well as data-sharing functions, through connections to numerous (and possibly dissimilar) computer systems. Developed as a solution for the enterprise that deals with heavy workloads of business-critical information, enterprise storage systems should be scalable for workloads up to thousands of gigabytes without relying on excessive cabling or the creation of subsystems. Other important aspects of the enterprise storage system are unlimited connectivity and multi-platform support. This journal presents a comprehensive set of test cases to measure PASS of the Device Under Test (DUT).

* Required Field

* First Name

* Last Name

* Company

* Email address

* Job Role

* Country

Phone

Send me updates on downloads, events, and promotions

Cancel

PASS Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies February 2011 Edition Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 1 Introduction Today’s Devices Under Test (DUT) represent complex, multi-protocol network elements with an emphasis on Quality of Service (QoS) and Quality of Experience (QoE) that scale to terabits of bandwidth across the switch fabric. The Spirent Catalogue of Test Methodologies represents an element of the Spirent test ecosystem that helps answer the most critical Performance, Availability, Security and Scale Tests (PASS) test cases. The Spirent Test ecosystem and Spirent Catalogue of Test Methodologies are intended to help development engineers and product verification engineers to rapidly develop and test complex test scenarios. How to use this Journal This provides test engineers with a battery of test cases for the Spirent Test Ecosystem. The journal is divided into sections by technology. Each test case has a unique Test Case ID (Ex. TC_MBH_001) that is universally unique across the ecosystem. Tester Requirements To determine the true capabilities and limitations of a DUT, the tests in this journal require a test tool that can measure router performance under realistic Internet conditions. It must be able to simultaneously generate wire-speed traffic, emulate the requisite protocols, and make real-time comparative performance measurements. High port density for cost-effective performance and stress testing is important to fully load switching fabrics and determine device and network scalability limits. In addition to these features, some tests require more advanced capabilities, such as Integrated traffic, routing, and MPLS protocols (e.g., BGP, OSPF, IS-IS, RSVP-TE, LDP/CR-LDP) to advertise route topologies for large simulated networks with LSP tunnels while simultaneously sending traffic over those tunnels. Further, the tester should emulate the interrelationships between protocol s through a topology. Emulation of service protocols (e.g., IGMPv3, PIM-SM, MP-iBGP) with diminution. Correct single-pass testing with measurement of 41+ metrics per pass of a packet. Tunneling protocol emulation (L2TP) and protocol stacking. True stateful layer 2-7 traffic. Ability to over-subscribe traffic dynamically and observe the effects. Finally, the tester should provide conformance test suites for ensuring protocol conformance and interoperability, and automated applications for rapidly executing the test cases in this journal. Further Resources Additional resources are available on our website at http://www.spirent.com Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 2 Table of Contents Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access ..................................................3 CFA_001 Converged LAN/SAN buffering behavior ............................................................. 4 CFA_002 Converged LAN/SAN buffering with histograms.................................................. 7 CFA_003 Measure converged LAN/SAN pause response time.......................................... 11 CFA_004 Converged LAN/SAN load validation test ......................................................... 14 CFA_005 Converged LAN/SAN unicast Queueput ............................................................ 18 CFA_006 Converged LAN/SAN maximum forwarding rate ............................................... 22 CFA_007 Converged LAN/SAN step test with intentional loss .......................................... 25 CFA_008 Virtual switch availability test ........................................................................... 28 CFA_009 Virtual machine performance test ..................................................................... 31 CFA_010 Virtual machine availability test ........................................................................ 34 CFA_011 FCoE fabric login ................................................................................................ 37 CFA_012 DCBX feature negotiation ................................................................................. 40 CFA_013 Converged LAN/SAN Queueput with multicast traffic ....................................... 43 CFA_014 Multiple redundant path performance test ....................................................... 47 CFA_015 Multiple redundant path availability test .......................................................... 50 CFA_016 Microsoft Exchange workload storage testing over FCoE ................................. 54 CFA_017 Microsoft Exchange workload storage testing over iSCSI ................................. 57 CFA_018 Accelerated vs unaccelerated iSCSI storage testing .......................................... 60 CFA_019 HTTP, FTP, and SIP with storage testing ............................................................ 64 CFA_020 Server transaction processing storage testing over FCoE .................................. 68 CFA_021 Server transaction processing storage testing over iSCSI .................................. 71 CFA_022 Verify congestion notification ............................................................................ 74 CFA_023 Verify enhanced transmission selection............................................................. 79 Appendix A – Telecommunications Definitions ..................................................................... 83 Appendix B – Stateful Playlist by QoS ................................................................................... 90 Appendix C – MPEG 2/4 Video QoE ....................................................................................... 91 Appendix D – Storage Queueput Standard ............................................................................ 92 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 3 Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access Enterprise storage is a centralized repository for business information that provides common data management and protection, as well as data-sharing functions, through connections to numerous (and possibly dissimilar) computer systems. Developed as a solution for the enterprise that deals with heavy workloads of business-critical information, enterprise storage systems should be scalable for workloads up to thousands of gigabytes without relying on excessive cabling or the creation of subsystems. Other important aspects of the enterprise storage system are unlimited connectivity and multi-platform support. Enterprise storage involves the use of a storage area network (SAN) rather than a distributed storage system, and includes benefits such as high availability, disaster recovery, data sharing, and efficient, reliable backup and restoration functions, as well as centralized administration and remote support. Through the SAN, multiple paths are created to all data, so that failure of a server never results in a loss of access to critical information. Data center storage and related protocols push the envelope of testing because of the need for sub- microsecond latency, ultra-fast bandwidth up to and exceeding 100 Gbps per link, sensitivity to real-time packet loss, duplication, reorder, and latency, and QoS requirements. Key testing success factors for data center Ethernet include: ULTRALOW LATENCY. Typical local switch times are rated at 1 microsecond or less. As a result, the precision of the test and measurement equipment becomes critical. Low latency is necessary to prevent data underflow through storage transactions. MULTI-BURSTY TRAFFIC AND JITTER. Storage flows are very sensitive to jitter. Because I/O requests are variable bit rate (VBR), and storage responses are also variable, the DUT must accept multiple, overlapping bursts of high-bandwidth traffic without injecting variability in latency (jitter) as a response. HIGH BANDWIDTH UNDER PROPER SEQUENCE. The data center storage device is expected to process terabits of information correctly. Assessing simple TX and RX frame count is no longer acceptable when evaluating a DCE switch because specific sequencing errors (real-time loss, duplicate, reordered, or late packets) halt the upper-layer storage protocol in different ways. Thus a duplicate packet will affect the overall storage flow differently than a lost packet. PRIORITY FLOW CONTROL (PFC). Priority flow control is a key selective back-off protocol at the link layer. Without PFC, true performance cannot be fully measured. With PFC realism, the DUT responds in ways that do not correlate to production networks. Priority flow control at 40 Gbps is especially critical. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 4 CFA_001 Converged LAN/SAN buffering behavior Abstract In terms of a converged LAN and SAN environment, this test determines the buffering and packet discard behavior of the DUT when receiving PFC frames. By setting different BB credits, the test will cycle thought various frame sizes to determine correct buffering. Without correct buffering, the DUT may incorrectly drop storage flow packets. References: IEEE 802.1Qbb, Test Case CFA_11, Test Case CFA_12. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority based flow control (check exact name). To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud converged LAN and SAN Priority Flow Control (PFC) test is a key requirement for lossless Ethernet in the data center. Determining the size of the DUT buffer is an important test. The tester should iterate over all configured permutations of frame size, burst size, and intended vector for all classification groups. For each load tuple, the tester should iterate over all configured PFC frame pause values. The results of the test indicate the DUT initiation threshold for PFC transmission and the size of the DUT buffer. Relevance The PFC Initiation threshold for the DUT needs to be as close as possible to line rate. DUT buffer size indicates the buffer depth. Version 1.1 Test Category Data Center Bridging. PASS [x] Performance [ ] Availability [ ] Security [ ] Scale Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 5 Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Reserve two test ports. 2. Cable tester port 1 and port 2 to the DUT. 3. Ensure the link and protocol are up via software and via LED confirmation. 4. Set the port MTU to be greater than 2148. 5. Configure DUT for FCoE with PFC on VLAN X. 6. Assign Stream Blocks to a Traffic Group. 7. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. 8. Assign an FCoE Draft Version of FIP. 9. Assign an Addressing Mode of FPMA. 10. Assign a BB Credit of 16. 11. Assign a Max RX Size of 2112. 12. Assign an FC Map of EFC00. 13. Assign a FIP Priority of 100. 14. Assign a Host Type of Both. 15. Ensure the FC payload size (the Max Receive Size) is set to 2112 Bytes. 16. For each pair of ports, set the host type to Initiator on one port and Target on another port 17. Alternatively set all ports to Both Initiator and Target. 18. Create streams with specific frame sizes. The frame sizes used must be specifically selected to be a multiple of single pause quanta, i.e., a multiple of 64 octets in length. This makes the conversion of pause quanta to switch buffer size straightforward. Set the frame size to 2148. 19. Create two traffic groups, one for LAN and one for SAN (FCoE) Traffic. 20. Assign one bi-directional stream to each traffic group. 21. Select Pause Devices as all frames. 22. Select pause on Queue X where X is equal to the VLAN. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 6 23. Assign the weighted traffic load less than 100% by the amount required to accommodate the PFC frames. 100% + PFC frames would be greater than 100%. Therefore 99% traffic is sufficient. Or if using LAN and SAN traffic 49.5% traffic for each. 24. Do not assign any Service Groups. 25. Start the traffic. 26. Observe the results in the Results Reporter. Variables & Relevance Variable Relevance VLAN ID A VLAN for the SAN traffic must share a common 802.1q VLAN ID on both the tester and the DUT. The 802.1Qbb pause information relies on a VLAN header. At the time of this writing untagged traffic is not verified to work with Priority Flow Control. PFC Quanta Quanta in 512bit times. At the time of this writing typically 65535 (equivalent to XON) or 0 (equivalent to XOFF). FCoE Draft Version At the time of this writing, set to FIP. Earlier implementations used FIP Interop and this may be necessary to complete the FLOGI process. Traffic Types LAN and SAN Traffic. It is important to do this FCoE test for pausing of LAN traffic in the presence of SAN traffic. Desired Result No lost frames. DUT Initiation Threshold within acceptable range. DUT Buffer Size as expected. Key Measured Metrics Statistic Relevance Lost Frames Understand if any frames are lost by the DUT. Any lost frames would indicate an incorrect implementation of CEE/DCB. DUT Initiation Threshold The results of the test indicate the DUT initiation threshold for PFC transmission and the size of the DUT buffer. Buffer Size The results of the test indicate the DUT initiation threshold for PFC transmission and the size of the DUT buffer. Analysis The user should not see any lost frames. If the user does see a lost frame this indicates an FCoE problem because the DUT is supposed to provide a lossless Ethernet environment. The DUT initiation threshold should be within specifications for the DUT. The Buffer Size should be within specifications for the DUT. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 7 CFA_002 Converged LAN/SAN buffering with histograms Abstract This test is an important baseline test for cloud computing by determining the Latency Behavior (LB) of the DUT when receiving PFC frames. This is achieved by changing the PFC values and relative frame sizes. Without testing buffering with histograms, the device under test may not achieve line rate performance in real world traffic scenarios. References: IEEE 802.1Qbb. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 10 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. Priority Flow Control (PFC) is a key requirement for lossless Ethernet in the Data Center. This test measures the traffic forwarding behavior of the DUT in the presence of test tool generated PFC frames and generates Latency Histograms based on the result. The tester should iterate over all configured permutations of frame size, burst size, and Intended Vector for all Classification Groups. For each load tuple, the tester should iterate over all configured PFC frame pause values. For each load tuple and PFC pause value, the tester should perform a trial iteration to establish the minimum and maximum latency values and optimize the tester’s result histograms based on the recorded min and max. After results are configured, the tester proceeds with the main test iteration. The results of the test indicate when the DUT buffers or discards frames and the effect of buffering on the forwarding latency of the DUT. Relevance The PFC Initiation threshold for the DUT needs to be as close as possible to line rate. DUT buffer size indicates buffer depth. Version 1.1 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 8 Test Category Data Center Bridging. PASS [x] Performance [x] Availability [ ] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Connect the DUT to the Spirent TestCenter and verify link is up and the correct ports are connected on both the tester and DUT. 2. Cable Tester Port 1 to 4 to DUT ports 1 through 4. 3. Optionally more ports can be used. 4. Reserve Four Test Ports in Spirent TestCenter. 5. Set the port MTU to be greater than 2148. 6. Configure DUT for FCoE on all ports. 7. Configure the DUT to pause on Queue 3 using PFC 802.1Qbb. 8. Create a single Emulated Devices on each port that will represent an FCoE host. a. Connect the DUT to the Spirent TestCenter and verify link is up and the correct ports are connected on both the tester and DUT. b. Cable Tester Port 1 to 4 to DUT ports 1 through 4. c. Optionally more ports can be used. d. Reserve four test ports in Spirent TestCenter. e. Set the port MTU to be greater than 2148. f. Configure DUT for FCoE on all ports. g. Configure the DUT to pause on Queue 3 using PFC 802.1Qbb. h. Create a single emulated device on each port that will represent an FCoE host. 9. Add a traffic group named DCB Group 1. 10. Create StreamBlocks in a full-mesh pattern at 95% of line rate. The overhead will take up the other 5%. Set the encapsulation to FC. 11. Assign the StreamBlocks to DCB Group 1. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 9 12. Set the frame size to 2148. a. This includes the 2112 FC Frame and the Ethernet overhead. 13. Leave the Burst size set to 1. 14. Leave the results view set as Latency-Jitter Mode. 15. Set the device Pause Queue to Pause 3 and set the pauses-per-second rate to 100 pauses per second. 16. Set the Quanta Value to 65535. 17. Set the VLAN Priority to 3. 18. Set all devices to PFC Pause Devices. 19. Assign the weighted traffic load less than 100% by the amount required to accommodate the PFC frames. 100% + PFC frames would be greater than 100%. 20. Start traffic. 21. Measure the results. Variables & Relevance Variable Relevance VLAN ID Pausing is independent of VLAN ID but frames are required to be tagged (Depending on 802.1Qbb). VLAN must match on the tester and DUT. PFC Quanta At the time of this writing, the standard specifies that the PFC Quanta needs to be as per 802.1Qbb however most implementations are using either 0 or 65535 for PFC Quanta. Pause Queue There are 8 Pause Queues. The pause queue needs to match on both the tester and DUT. This methodology recommends no more than a single Pause Queue on any given port. Number of Pauses This is important because even the maximum pause quanta (each quanta is 512 bits) is for a very short time and this may not be enough time for the DUT to clear its ingress buffers. And even if it is, the DUT pauses for a split second but does not slow down the traffic rate. Therefore to slow down the traffic rate, a large number of pause frames should be sent (i.e. between 100 to 500). Desired Result The results of the test indicate when the DUT buffers or discards frames and the effect of buffering on the forwarding latency of the DUT. No lost frames. Latency of forwarded frames within acceptable range for non-blocking store-and-forward 10 GB switches. Key Measured Metrics Statistic Relevance Lost Frames Understand if any frames are lost by the DUT. Any lost frames would indicate an incorrect implementation of CEE/DCB. Latency of Forwarded Frames During these tests, the latency of forwarded frames should be measured as many of these tests are intended to cause internal buffering, which may affect the switch latency. There are no standards for switch latency but latency must be compared against DUT design goals, which ideally is as minimal as possible. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 10 Analysis The user should not see lost frames. If the user does see a lost frame this indicates an FCoE problem because the DUT is supposed to provide a lossless Ethernet environment. The latency should be within specifications for the DUT. Excessive latency greater than that typically used for a market leading 10 GB non-CEE/DCE switch would be regarded as undesirable. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 11 CFA_003 Measure converged LAN/SAN pause response time Abstract Modern cloud computing requires Converged LAN and SAN networks. In terms of critical functionality for converged LAN and SAN environments, this test measures how quickly the DUT pauses traffic in response to a priority-based flow control pause. Without testing PFC, QoS may not be verified across the device under test. References: IEEE 802.1Qbb, Test Case CFA_11, Test Case CFA_12. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 10 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud-converged LAN and SAN environment dictates that for each tuple and PFC pause value, the tester transmits a single pause frame and measures a) the time it takes the DUT to cease transmission after the pause frame is transmitted and b) the actual pause duration of the DUT. The results of the test will show that If XON frames are enabled, then the test also measures c) the time it takes the DUT to respond to an XON frame. Relevance DUT Pause Response time must be measured. Version 1.1 Test Category Data Center Bridging. PASS [ ] Performance [x] Availability [ ] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 12 Topology Test Procedure 1. Reserve two test ports. 2. Connect cabling to the DUT. Cable Tester Port 1 and Port 2 to the DUT. 3. Configure DUT for FCoE with PFC on VLAN X. 4. Create 256 emulated devices on each port that will represent FCoE hosts. a. Create a VLAN header for the frames sent from each device with the VLAN corresponding to the FCoE VLAN on the DUT (VLAN X). b. Create 256 VN_Ports. c. Leave the WWN and MAC Addresses to Default (optional: may change them). 5. Pick your FCoE draft version (choose from FIP, FIP Interop, or FCoE). 6. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. 7. Ensure the FC payload size (the Max Receive Size) is set to 2112 bytes. 8. For each pair of ports, set the host type to Initiator on one port and Target on another port a. Alternatively set all ports to Both Initiator and Target. 9. Create streams with specific frame sizes. The frame sizes used must be specifically selected to be a multiple of a single pause quantum, i.e., a multiple of 64 octets in length. This makes the conversion of pause quanta to switch buffer size straightforward. Set the frame size to 2148. 10. Create two traffic groups, one for LAN and one for SAN (FCoE) Traffic. 11. Assign one bi-directional stream to each traffic group. 12. Assign the weighted traffic load less than 100% by the amount required to accommodate the PFC frames. 100% + PFC frames would be greater than 100%. 13. Select both ports to transmit PFC pause frames. 14. Set the number of trials to 1. 15. Set the learning frequency to one iteration of 1000 fps, 2148 size frame. 16. Start traffic. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 13 Variables & Relevance Variable Relevance VLAN ID ID of the test VLAN. DUT PFC Pause Quanta Value 65535 as equivalent to XOFF in most implementations at the time of this writing. Learning Frequency 1000 FPS. Any faster would possibly overwhelm the learning mechanism. These frames are not counted in the final result. Desired Result No lost frames. Pause Time within acceptable range. Response Time within acceptable range. Key Measured Metrics Statistic Relevance Lost Frames Understand if any frames are lost by the DUT. Lost frames indicate an incorrect implementation of CEE/DCB. Response Time This shows how quickly or slowly the DUT pauses traffic in response to a PFC pause frame. Pause Time How long the DUT remains paused after receiving a pause frame. Analysis The user should not see any lost frames. If the user does see a lost frame this indicates an FCoE problem because the DUT is supposed to provide a lossless Ethernet environment. The Response Time should be within specifications for the DUT. A response time that is longer than the (minimum IFG for 1 0 GB 802.3ae Ethernet (i.e. 96 bit times) x (The number of frames the buffer is capable of holding (the buffer depth) +1) would probably be a good indicator of problems to come. Likewise pause times greater than the (Frame Time x Buffer Depth (in frames)) would be a good indicator of problems with optimization. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 14 CFA_004 Converged LAN/SAN load validation test Abstract Within a converged LAN and SAN environment, this test is designed to verify that the DUT pauses for the specified amount of time when paused under load. This is achieved by varying rate and frame size under pause conditions. Testing PFC is a critical part of verifying multiple priority flows in the network. References: IEEE 802.1Qbb, IEEE 802.1Qaz, Test Case CFA_11, Test Case CFA_12. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 10 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority based flow control (check exact name). To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. Effectively testing the cloud-converged LAN and SAN environment requires that for each tuple and PFC pause value, the tester measures the offered load of each DUT port and compares that value to the expected offered load based on the configured PFC pause rate and quanta. The test reports whether each port observed the correct pause duration or not. Results will be a Pass/Fail based on DUT meeting the criteria. Relevance In DCB/CEE, single pauses only serve to pause the traffic for a single quanta. The maximum quanta value of 65535 may not result in enough pause for the end station to clear their buffers or may be too great. However, many implementations at the time of this writing implement pause as either 0 or 65535. Therefore, under load, it is more expected that in implementations it is more common for many pause frames to be sent i.e. 100 per second each with a large quanta of 65535. This is an alternate or complementary method to the XON/XOFF approach that is being deployed by some vendors in the industry. Version 1.1 Test Category Data Center Bridging. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 15 PASS [x] Performance [x] Availability [ ] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Connect the DUT to the Spirent TestCenter and verify link is up and the correct ports are connected on both the tester and DUT. 2. Cable tester port 1 to 4 to DUT ports 1 through 4. 3. Optionally more ports can be used. 4. Reserve four test ports in Spirent TestCenter. 5. Set the port MTU to be greater than 2148. 6. Configure DUT for FCoE on all Ports. 7. Configure the DUT to pause on Queue 3 using PFC 802.1Qbb. 8. Create a single emulated device on each port that will represent an FCoE host. a. Create a VLAN header for the frames sent from each device with the VLAN corresponding to the FCoE VLAN on the DUT (VLAN X). b. Leave the WWN and MAC Addresses to Default (optional: may change them). c. Set the FCoE Draft Version to FIP. d. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. e. FC payload size will be over-written in next step, so ignore this. f. Set the host type to Both. g. Add a DCB Type 2 TLV. 9. Add a traffic group named DCB Group 1 (SAN Traffic). 10. Add a traffic group named DCB Group 1 (LAN Traffic). 11. Create StreamBlocks in a full-mesh pattern at 95% of line rate. The overhead will take up the other 5%. Set the encapsulation to FC. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 16 12. Assign the StreamBlocks to DCB Group 1 (SAN Traffic). 13. Add another set of StreamBlocks for LAN traffic. a. Full Mesh. b. Regular Ethernet frames (no IP or L3 or ARPing required) encapsulated in a different VLAN. c. Size of 512 is acceptable as this is not a traffic mix test. d. Do not worry about the Load as this will be set in the next part of the Wizard. 14. Assign the StreamBlocks to DCB Group 2 (LAN Traffic). 15. Configure the SAN Traffic for Load Validation Test as follows: a. Set the frame size to 2148. i. This includes the 2112 FC Frame and the Ethernet overhead. b. Manually set the DUT to do PFC for priority 3 (or whatever priority (0 – 7) is needed). c. Manually set the tester to also be priority 3 (or whatever priority (0 – 7) is needed). d. Do not use any other priority queues. e. Set the validation Result Tolerance to 1%. f. Leave the Quanta Value fixed at 65535. g. Use weighted traffic loads. h. Set weighted Traffic Loads to 49% to account for extra overhead from the PFC frames. 16. Configure the LAN Traffic for Load Validation Test as follows: a. Set the frame size to 512. b. Set the validation Result Tolerance to 1%. c. Use weighted traffic loads. d. Set weighted Traffic Loads to 49% to account for extra overhead from the PFC frames. e. Set the VLAN Priority to 3 for the SAN Traffic Group. f. Set the LAN Traffic group to a lower VLAN priority to i.e. 2. 17. Run the test. 18. Examine results. Variables & Relevance Variable Relevance SAN VLAN ID The DUT port on which the LAN and SAN traffic will be mixed is a trucking port. The LAN traffic needs to be on a different VLAN than the SAN Traffic. LAN VLAN ID The DUT port on which the LAN and SAN traffic will be mixed is a trucking port. The LAN traffic needs to be on a different VLAN than the SAN Traffic. PFC Quanta Typical shipping commercial implementations at the time of this writing rely on an XON/XOFF style utilizing either 0 or 65535 however this test is designed to allow a range of quanta. Traffic Load Percentage This refers to the offered load and not the Pausing, however the PFC protocol will absorb space on the wire therefore the total traffic load percentage can never exceed (100% - PFC Frame Rate %) Result Tolerance At the time of this writing, results of the actual pause time under load for implementations may vary. The result tolerance variable should be used to adjust for this variance. Desired Result The DUT pauses for the specified amount of time when paused under load. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 17 Key Measured Metrics Statistic Relevance DUT Pause Time DUT Pause time is the amount of time the DUT pauses when under load, as specified by the tester. The equation to derive DUT Pause time is: Quanta x 512/bitrate @ 10 Gbps for 10 GbE. Analysis DCB/CEE standards and practices are still solidifying at the time of this writing. Current implementations are based on the demand for cloud computing products as vendors are going to market with pre-standard market offerings. Much of this testing is necessary to understand the implications of a simplified XON/XOFF approach versus a more complex Quanta and time approach. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 18 CFA_005 Converged LAN/SAN unicast Queueput Abstract This test determines the performance and scalability of a cloud-based converged LAN and SAN core fabric and virtual server access switch, handling multiple traffic types, including LAN and SAN. Determines the Queueput of the DUT for all traffic classes using a full mesh topology with unicast LAN traffic. Queueput is a primary forwarding performance metric in PFC-enabled networks. References: IETF draft-player-dcb-benchmarking-02, Test Case CFA_011, Test Case CFA_012. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud-converged LAN and SAN Queueput utilizes draft DCB Queueput and determines the throughput per VLAN priority of IEEE enabled Data Center Bridging switches verifying converged LAN and SAN fabrics and switches ability to perform and scale to cloud infrastructure requirements levels. A search algorithm is used to determine the Queueput for each Classification Group. Relevance Queueput is the fundamental best-practice way to measure the ability of the DUT to forward traffic and is the best way to test lossless Ethernet given that an RFC-2544 style test is not possible. At the time of this writing, Converged Enhanced Ethernet ports enable VLAN tags on all traffic types and assign a 3-bit priority to each frame for a total of Eight Priorities. Each priority may be paused independently of the others. A typical use case is to configure the port to limit the bandwidth assigned to the LAN traffic while allowing the SAN (FCoE traffic) traffic unlimited bandwidth and lowest latency. Version 1.1 Test Category Data Center Bridging. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 19 PASS [x] Performance [ ] Availability [ ] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Reserve four test ports. 2. Connect cabling to the DUT. 3. Configure DUT for FCoE with PFC running on VLAN X. 4. Set a pause queue of Y on the DUT. 5. Create 1 emulated device on each port that will represent a Virtual Machine. a. This is used for both LAN and SAN traffic. b. Create a VLAN header for the frames sent from each device with the VLAN corresponding to the FCoE VLAN on the DUT (VLAN X). c. Leave the WWN and MAC Addresses to Default (optional: may change them). 6. Pick the FCoE draft version which corresponds to your DUT version of FCoE (choose from FIP, FIP Interop, or FCoE). 7. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings as well. 8. Ensure the DUT and Tester ports have a MTU of at least 9000. 9. Ensure the FC payload size on the test traffic is set to 2112 bytes. 10. For each pair of ports, set the host type to Initiator on one port and Target on another port. a. Alternatively set all ports to Both Initiator and Target. b. This makes the conversion of pause quanta to switch buffer size straightforward. 11. Configure the test equipment to report latency and jitter results in a single test pass. 12. Create two traffic groups, one for LAN and one for SAN (FCoE) Traffic. 13. For the LAN Traffic. a. Create a full mesh of unicast traffic between all ports. b. Set the VLAN Priority to Match the Pause Queue. c. Configure LAN Traffic to be 50% of the maximum bandwidth on the port. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 20 14. For the SAN Traffic. a. Create a full mesh of traffic between all ports. b. Set the VLAN Priority to Match the Pause Queue. c. Set a Binary Search between 50% and 100% of Wire Rate for this traffic. 15. Configure the Pause Queue mapping in the DUT to Pause the LAN traffic as required. 16. Set a test trial duration of at least 300 seconds. 17. Configure a results collection delay of at least 5 seconds to ensure any late frames are received. 18. Optionally do some learning to populate various forwarding tables on the DUT. 19. Optionally configure DUT forwarding table timeouts to be greater than 300 seconds. 20. Once learning is completed, run the test. 21. At test completion Verify there are no dropped frames for any class of traffic. 22. Measure latency for each step of the binary search. 23. Measure the Queueput for the SAN Traffic in the presence of 50% LAN traffic. Variables & Relevance Variable Relevance Latency Type LILO Last In Last Out. Results Collection Delay Some amount of time usually at least 5 seconds to allow any late frames stuck in buffers to be forwarded. Frame Size Must be a multiple of 64 Bytes to ensure easy mapping of the available remaining bandwidth after application of traffic to Quanta. PFC Quanta Priority Flow Control Quanta. At the time of this writing often implemented in an XON/XOFF method. VLAN ID X Must match on DUT and Tester. Pause Queue Y Must match on DUT and Tester. Pause Queue is independent of VLAN ID, however VLAN CLP bits must match the Pause Queue. Test Traffic Since DCB devices are expected to support multiple traffic Classifications, it is RECOMMENDED to benchmark DCB devices with multiple Classification Groups. Desired Result No traffic loss. Key Measured Metrics Statistic Relevance Queueput for SAN Traffic The maximum forwarding rate (in % of Line Rate) of the SAN traffic prior to Pause Frames being sent by the DUT to the tester port Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 21 Analysis RFC2544 and RFC2889 are benchmarking standards that have served the industry very well in the many years since they were first accepted by the RFC editor and published. Updating these is a challenging task and the curve ball of DCB/CEE networking for cloud computing in a Converged LAN and SAN environment drove the need for the adoption of the Queueput Test. It is highly desirable that the rate of SAN traffic (prior to the first Pause Frame) plus the rate of the LAN traffic is equal to 100% of line rate. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 22 CFA_006 Converged LAN/SAN maximum forwarding rate Abstract This test measures the maximum forwarding rate of the DUT. The load varies between the throughput value derived from the forwarding test and the maximum load. This test determines the peak performance of the DUT. References: IEEE 802.1Qbb for the 2010 Standard of FCoE and Priority Flow Control, IETF RFC 2285 for Forwarding Rate definition. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority based flow control (check exact name). To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz enhanced traffic selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud-converged LAN and SAN Maximum Forwarding Rate test will use purely FCoE SAN Traffic. It is a relatively simple test to determine the DUT maximum forwarding rate. The tester should iterate across all configured permutations of frame size, burst size, and intended vector for all Classification Groups. Relevance DUT Maximum Forwarding rate must be measured and reported. Version 1.1 Test Category Data Center Bridging. PASS [x] Performance [ ] Availability [ ] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 23 Topology Test Procedure 1. Reserve 4 test ports. 2. Connect cabling to the DUT. Cable tester port 1 and port 2 to the DUT, etc. 3. Configure the DUT for FCoE with PFC on VLAN X. 4. Create 256 emulated devices on each port that will represent FCoE hosts. a. Create a VLAN header for the frames sent from each device with the VLAN corresponding to the FCoE VLAN on the DUT (VLAN X). b. Create 256 VN_Ports. c. Leave the WWN and MAC Addresses to Default (optional: may change them). 5. Pick your FCoE draft version (choose from FIP, FIP Interop, or FCoE). 6. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. 7. Ensure the FC payload size (the Max Receive Size) is set to 2112 bytes. 8. For each pair of ports, set the host type to Initiator on one port and Target on another port. a. Alternatively set all ports to Both Initiator and Target. 9. Create a single traffic group for SAN (FCoE) Traffic. 10. Configure frame sizes 64, 72, 68, 80, 84, 88, 92, 108, 112,176, 332, 340, 2148. 11. Create full mesh traffic between all group members. a. Step the rate from 80% to 100% Line Rate in 5% increments. 12. PFC is not required as ports should never be oversubscribed. 13. Set traffic to run for at least 300 seconds. 14. Start traffic. 15. At the completion of traffic observe the forwarding rate. 16. Repeat test with a range of frame sizes from 64 to 2148 stepping in 4-byte increments and observe and measure the forwarding rate. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 24 Variables & Relevance Variable Relevance Frame Size Must be a multiple of 64 bytes to ensure easy mapping of the available remaining bandwidth after application of traffic to Quanta. Typically 2148 bytes for FCoE frames carrying a full SCSI payload but can vary from 64 bytes up to 2148 bytes. PFC Quanta Priority Flow Control Quanta. At the time of this writing often implemented in an XON/XOFF method. VLAN ID X Must match on DUT and Tester Pause Queue Y Must match on DUT and Tester. Pause Queue is independent of VLAN ID, however VLAN CLP bits must match the Pause Queue. Traffic Run Time 300 Seconds Beginning Traffic Rate 80% of Line Rate Desired Result A forwarding rate as close as possible to line rate. Key Measured Metrics Statistic Relevance Maximum Forwarding Rate Expressed either in Frames Per Second or as a percentage of total load on the wire Analysis The Maximum Forwarding Rate needs to be as high as possible. It is expected that for the larger 2148 frames that are used by the SAN traffic in a converged LAN and SAN environment, it is easier for the DUT to reach the bits per second that wire rate expresses. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 25 CFA_007 Converged LAN/SAN step test with intentional loss Abstract Cloud computing requires a converged LAN and SAN environment. In terms of a converged LAN and SAN environment, loss is expected in this test. This determines the percentage of frames that should have been forwarded by a network device under steady state (constant) load that were not forwarded due to lack of resources. References: IEEE 802.1Qbb, Test Case CFA_11, Test Case CFA_12. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud-converged LAN and SAN environment requires that the initial trial SHOULD begin with an Intended Load equal or greater than the Maximum Forwarding Rate of the DUT/SUT. For each subsequent trial, the aggregate load is reduced until the DUT is observed to complete a trial without activating congestion management methods. Relevance Additional measurement to understand implications of resource limitations in the DUT. Version 1.1 Test Category Data Center Bridging. PASS [ ] Performance [x] Availability [ ] Security [x] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 26 Topology Test Procedure 1. Reserve all the ports on the DUT and a corresponding number of tester ports. 2. Connect cabling to the DUT. 3. Configure the DUT for FCoE with PFC on VLAN X. 4. Create Emulated Devices & FCoE Parameters as per CFA_003. 5. Pick your FCoE draft version (choose from FIP, FIP Interop, or FCoE). 6. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. 7. Ensure the FC payload size (the Max Receive Size) is set to 2112 bytes. 8. For each pair of ports, set the host type to Initiator on one port and Target on another port. a. Alternatively set all ports to Both Initiator and Target. 9. Create two traffic groups, one for LAN and one for SAN (FCoE) Traffic. 10. Assign one bi-directional stream to each traffic group. 11. Select both ports to transmit PFC pause frames. 12. Start traffic. Variables & Relevance Variable Relevance Frame Size Must be a multiple of 64 bytes to ensure easy mapping of the available remaining bandwidth after application of traffic to Quanta. Typically 2148 bytes for FCoE frames carrying a full SCSI payload but can vary from 64 bytes up to 2148 bytes. PFC Quanta Priority Flow Control Quanta. At the time of this writing often implemented in an XON/XOFF method. VLAN ID X Must match on DUT and Tester Pause Queue Y Must match on DUT and Tester. Pause Queue is independent of VLAN ID, however VLAN CLP bits must match the Pause Queue. Traffic Run Time 300 Seconds. Desired Result Loss tables of measurements showing for each step backwards from the maximum forwarding rate to the point of no congestion control the ratio of loss. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 27 Key Measured Metrics Statistic Relevance Loss Table A table of the ratio of frame loss for each step backwards from the maximum forwarding rate. Frame Loss Ration Frame Loss observed at each Step Step Rate The percentage of line rate Analysis In a converged LAN and SAN environment, there should be a predictable amount of frame loss. This test is necessary to characterize that frame loss and ensure it follows a predictable pattern. There will be frame loss on the lower-priority traffic. It is important to keep in mind this test expects frame loss. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 28 CFA_008 Virtual switch availability test Abstract The purpose of this test case is to test the availability a virtual group consisting of multiple chassis using aggregated links at 10 Gbps speeds using random MAC addresses. Adding a switch in to the virtual group should immediately cause the switch to learn the MAC address table and unidirectional traffic arriving at this new switch should be able to reach any destination that is already present in the table without requiring additional learning frames. Many switch vendors offer a virtual switch configuration. This feature is designed to improve network performance and reduce administrative costs of adding or subtracting switches to a fabric. Description The test uses a virtual, multi-chassis switch consisting of three physical switches. A virtual group of two physical switches is built and share a single dynamic MAC address table. Learning frames are sent from the tester on any port of the second switch, followed by continuous traffic at slightly less than half line rate into any port on the first switch. This traffic is expected to egress the second switch with no loss. A third switch is added into the virtual switch. A wait time to allow for MAC-address learning, such as thirty seconds, is followed by continuous traffic sent to the third switch at less than half line rate, destined for the egress port on the second switch. This traffic is also expected to exit no loss. Target Users Switch Test Engineers Target Device Under Test (DUT) Layer 2 switches that are 10 Gbps capable and which can be configured in a virtual switch group. Reference RFC 4814 Relevance Many switch vendors offer a virtual switch configuration. Virtual switches are not to be confused with virtualization in the Hypervisor sense, but instead are two or more switches which appear as one switch for the purposes of administration, MAC-address learning and forwarding, and other switching features. Virtual switches improve network performance and reduce the administrative costs of adding or subtracting switches to a fabric. Version 1.0 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 29 Test Category Cloud Infrastructure LAN_SAN Fabric & Virtual Server Access PASS [ ] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to test multiple unique Layer-2 hosts with full RFC 4814 support. Detect traffic loss independent of any particular MAC address configuration. Topology Diagram Test Procedure 1. Create a virtual switch out of two switches. a. Cable up three switches, but keep the third switch powered off. 2. On the target test port, configure 1000 emulated hosts using random MAC Addresses as per RFC 4814. 3. On both of the source test ports, configure 1,000 different emulated hosts using random MAC addresses as per RFC 4814. a. Send 10,000 learning frames into the target switches at 1,000 fps. This will send at least 10 learning frames per emulated host. 4. Let HLR = 50% of line rate of 10 Gbps i.e. 5 Gbps. 5. Start continuous traffic at rate HLR line into the source switch for 300 seconds. 6. Verify the traffic rate received on the target port is HLR and dropped frames are zero. a. Power on the third switch where TPO = time to fully power on and boot up in seconds. Wait TPO Seconds. 7. Wait an additional 30 seconds for MAC table sharing to be completed. 8. From the new switch source test port, start continuous traffic at rate HLR into the new switch for 300 seconds. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 30 9. Verify the traffic rate on the target port is now 100% of line rate i.e. 10 Gbps, FLR) and dropped frames are zero. 10. Power off the first switch and wait 300 seconds. 11. Verify the traffic rate received on the target port is HLR and dropped frames are zero. Control Variables & Relevance Variable Relevance Default Value Learning Frame Rate A low rate is chosen to ensure learning. 1000 fps Number of emulated hosts on each port Emulates at 10 Gbps / 1000 = 100 hosts connected at 1 Gbps and utilizing the full 1 Gbps on average 10% of the time. 1000 HLR Half line rate. Ensure the receive port is not overloaded when the third switch is introduced. 5 Gbps FLR Full line rate. 10 Gbps TPO Time to fully boot the DUT. Derived as part of test K Frames per second. Derived as part of test Key Measured Metrics Statistic Relevance Metric Unit Dropped Frames Zero at all times considering learning has taken place before sending traffic. Ordinal Numeric HLR Measured to ensure receive traffic Gbps FLR Measured to ensure receive traffic from both switches Gbps Desired Result The goal is full line rate received traffic when adding the new switch into the virtual group with no dropped frames. Analysis This is not a convergence test, this is a test of MAC address table sharing. If this sharing is done correctly there will be no dropped frames. Half-line rate is actually easy on buffers that are designed for 10 Gbps. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 31 CFA_009 Virtual machine performance test Abstract The purpose of this test case is to determine the bi-directional Layer-2 performance of a virtual machine blade server enclosure environment using 10 Gbps connectivity to a 10 Gbps switch fabric. Because a virtual machine is software, it does not have hardware-accelerated traffic analyzers or generators, so the traffic rate is expected to be lower than line-rate. Also, many factors in a virtualized blade server enclosure environment, such as the 10 Gbps Ethernet connectors, midplane ASCIS, mezzanine card, and the Hypervisor itself, have an effect. In addition, most hypervisors use a virtual switch that could be a factor affecting performance. Description Perform an RFC-2544 test between the virtual machine from a physical port using LAN traffic through a switch fabric. Speed must be 10 Gbps Ethernet. Target Users Switch Test Engineers Target Device Under Test (DUT) Blade server (10 Gbps interface module in blade enclosure, mezzanine card, if any, midplane, CPU and memory power of blade server) Hypervisor and virtual switch 10 Gbps switch or switch fabric Relevance Many switch vendors offer a virtual switch configuration. Virtual switches are not to be confused with virtualization in the Hypervisor sense, but instead are two or more switches which appear as one switch for the purposes of administration, MAC-address learning and forwarding, and other switching features. Virtual switches improve network performance and reduce the administrative costs of adding or subtracting switches to a fabric. Version 1.0 Test Category Cloud infrastructure LAN/SAN fabric and virtual server access Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 32 PASS [ X] Performance [ ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to run an RFC-2544 test between a VM on a Hypervisor without the need for an operating system between the tester and the Hypervisor and record signature frames. Topology Diagram Test Procedure 1. Build the topology as shown in the diagram using a 10 Gbps switching fabric and 10 Gbps connections to the blade enclosure. 2. Configure a standard RFC-2544 test. a. Start with 1% of line rate (SR). b. Use a standard 50% back off algorithm. c. Use Layer 2 learning. d. Use zero dropped frames as the pass criteria. 3. Run the RFC-2544 test. 4. Measure the throughput (in % of line rate) for the tested frame sizes. 5. Let Z be throughput (also known as the maximum forwarding rate) for use in other test cases. Control Variables & Relevance Variable Relevance Default Value SR Starting rate for the RFC-2544 test. 1% Learning Style Layer 2 or Layer 3. Layer 2 Dropped Frame Allowance Allows some tolerance on dropped frames. Zero Key Measured Metrics Statistic Relevance Metric Unit Dropped Frames Zero at all times considering learning has taken place before sending traffic. Ordinal Numeric Z Throughput (also known as the maximum forwarding rate) Percentage Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 33 Desired Result No lost frames are expected. Throughput can actually approach 10 Gbps speeds on a high- powered server, depending on the configuration of the test. Analysis If lost frames are encountered, the test does not pass. However, there may be some margin to run an asymmetric throughput test, as it is expected that the virtual machine can transmit faster rates than it can analyze. Note that a frame size has not been specified. It is expected that jumbo frames will produce a much higher throughput result. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 34 CFA_010 Virtual machine availability test Abstract The purpose of this test case is to quantify the one-way packet loss during virtual machine migration in a blade server enclosure connected to a switching fabric with 10 Gbps Ethernet. The maximum receive rate Z of the hardware behind the virtual machine (VM) must be obtained by running an RFC-2544 throughput test, the throughput being equal to Z, then migration is initiated and loss measured as detailed in the test steps. Packet loss during migration must be quantified to be understood. Description While transmitting traffic continuously to a VM from a physical port, migrate the VM from one blade to another blade. Record the lost frame count and calculate it as a percentage of total frames. Repeat the process between blades in different enclosures in an enclosure stack. Repeat the process between blades located in different enclosures across a switching fabric. Target Users Switch Test Engineers Target Device Under Test (DUT) Blade Server 10 Gbps interface module in blade server Hypervisor and virtual switch 10 Gbps switch or switch fabric Relevance Many switch vendors offer a virtual switch configuration. Virtual switches are not to be confused with virtualization in the Hypervisor sense, but instead are two or more switches which appear as one switch for the purposes of administration, MAC-address learning and forwarding, and other switching features. Virtual switches improve network performance and reduce the administrative costs of adding or subtracting switches to a fabric. Version 1.0 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 35 Test Category Cloud Infrastructure LANSAN Fabric & Virtual Server Access PASS [ ] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to exist as a VM on a Hypervisor without the need for an operating system between the tester and the Hypervisor and record signature frames. Topology Diagram Test Procedure 1. Build the topology as shown in the diagram using a 10 Gbps switching fabric and 10 Gbps connections to all blade enclosures. 2. Transmit traffic continuously from the physical 10 Gbps test port to the VM at rate Z. 3. Verify that traffic rate on the VM is Z and dropped frames are zero. 4. Migrate the virtual machine to a different blade within the same enclosure. 5. Verify that after migration, traffic rate on the VM drops but returns to Z. 6. Let LF be the count of any lost frames. 7. If LF continues to increase, the test fails and cannot continue. 8. Once LF1 stabilizes (i.e. no longer increments), stop traffic. 9. Record AF1 as all frames. 10. Calculate the loss percentage as (LF1 / AF1) * 100% and record this as LFP1. 11. Return the VM back to the original blade and reset all counters on the test equipment. 12. Repeat the process but this time migrate the VM between different blades in an enclosure stack. Record the lost frames as LF2, all frames as AF2, and the lost frame percentage as LFP2. 13. Repeat the process but this time migrate the VM to a different blade located in a different enclosure across the switching fabric. Record the lost frames as LF3, all frames as AF3, and the lost frame percentage as LFP3. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 36 Control Variables & Relevance Variable Relevance Default Value Z Maximum bits per second at which the VM can receive frames with no packet loss. Calculated via RFC-2544 test. 2.5 Gbps Target Target blade to which the VM is migrated. A blade within the same blade enclosure. Key Measured Metrics Statistic Relevance Metric Unit Dropped Frames Zero at all times considering learning has taken place before sending traffic. Ordinal Numeric LF1 Lost frames during migration between blades in the same enclosure. Ordinal Numeric AF1 All frames received during migration between blades in the same enclosure. Ordinal Numeric LFP1 Percentage of lost frames during migration between blades in the same enclosure. Percentage LF2 Lost frames during migration between blades in different enclosures in an enclosure stack. Ordinal Numeric AF2 All frames received during migration between blades in different enclosures in an enclosure stack. Ordinal Numeric LFP2 Percentage of lost frames during migration between blades in different enclosures in an enclosure stack. Percentage LF3 Lost frames during migration between blades in different enclosures separated by the switch fabric. Ordinal Numeric AF3 All frames received during migration between blades in different enclosures separated by the switch fabric. Ordinal Numeric LFP3 Percentage of lost frames during migration between blades in different enclosures separated by the switch fabric. Percentage Desired Result Some lost frames are expected. Traffic rates should return to the initial rate after migration for all migration targets (local, different blade in a stack, different blade located across the network). Analysis It is expected that the percentage of lost frames will increase slightly as the VM is migrated further. Lost frames on a migration within the same enclosure or across an enclosure stack should be less than the lost frames incurred during migration across the switch fabric. If lost frames are higher across the enclosure stack than across the switch fabric, it indicates issues and bottlenecks with the stacking technology. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 37 CFA_011 FCoE fabric login Abstract In terms of a DCB/CEE converged LAN and SAN environment, this test determines whether the login process completes for login to the fabric for a single port. This is an elemental test that must pass in order for more complex tests to be valid. References: FDISC, FCoE. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority based flow control (check exact name). To a guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. The cloud-converged LAN and SAN Queueput Fabric login (FDISC) is the primary step of FCoE technology and is required. Relevance FCoE Login is required and can be itself a difficult step to accomplish and justifies its own test. Trying to troubleshoot a more advanced test when login is not completing can be an overly time consuming task If the DUT cannot login the test ports, there will be no FCID assigned and therefore no ability to switch traffic. Version 1.1 Test Category Data Center Bridging. PASS [ ] Performance [x] Availability [x] Security [ ] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 38 Topology Test Procedure 1. Reserve a single test port. 2. Cable tester port 1 to the DUT. 3. Configure DUT for FCoE. 4. Configure DCBX on the DUT Port. 5. NOTE: PFC is optional for this test. 6. Create 260 VN Ports on the port that will represent FCoE hosts behind a Virtual Machine CAN. a. Leave the WWN and MAC Addresses to Default (optional: may change them). 7. Pick your FCoE draft version (choose from FIP, FIP Interop, or FCoE). 8. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings. 9. On the Test Port, add DCBX Type II TLVs as follows, with the willing bit set to 1 (On). a. PFC. b. Application. c. Priority Group. 10. Start DCBX on the port. 11. Verify DCBX between the Test Port and the DUT Port is Synced. 12. Verify the Operating Mode for PFC, Application, and Priority Group converge to a value of True (On). 13. Start the FCOE Devices. 14. Verify PLOGI completes on devices 1 through 255 and FLOGI times out on devices 256 through 260. Variables & Relevance Variable Relevance VLAN ID Must match DUT VLAN WWN Must be unique per VN Port Source Mac Address May be shared among all VN Port FCoE Version FIP Interop at the time of this writing Desired Result The latency, throughput, frame loss and forwarding rates of the DUT should be within desired expected ranges for the DUT. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 39 Key Measured Metrics Statistic Relevance Aggregate Sub state This contains the total state for all the VN_Ports. Possible Values are: Solicitation Timed Out FLOGI Rejected PLOGI Successful 255 ports should login successfully and 5 ports should receive the FLOGI rejected message. Analysis If Fabric Login completes but Port Login does not, this indicates a reachability problem within the Fabric. Fabric Login is between the CNA and the DUT, but the full login process does include the login to the Port. Therefore both Logins have to complete in order for this test to be considered successful. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 40 CFA_012 DCBX feature negotiation Abstract DCBX is a fundamental requirement to be tested for any devices targeted towards cloud computing. This test determines whether DCBX is functional for features necessary for converged LAN and SAN testing. This test will measure the correctness of the DUTs DCBX capability. References: DCBX as per the FCIA (Fibre Channel Industry Association), Test Case CFA_011. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 1 0 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. DCBX is considered mandatory for equipment that will be deployed in data centers and cloud computing environments and is a foundation feature that is required by the majority of the ports that will comprise the infrastructure for Data Center Bridging / Converged Enhanced Ethernet topologies. Relevance Data Center Discovery and Exchange Protocol (DCBX) is used by switches and end devices to configure and advertise the PFC and ETS configurations. If DCBX negotiation fails, manual configuration will have to be used, and it is expected that the large numbers of ports in a data center will make manual configuration too time consuming. Version 1.1 Test Category Data Center Bridging. PASS [ ] Performance [x] Availability [ ] Security [ ] Scale Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 41 Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Reserve a single test port. a. Test must be tested on every port individually. b. Once all ports have passed individually, test must be scaled to simultaneous login on all ports. 2. Cable tester port 1 to the DUT. 3. Configure DCBX on the DUT port. 4. On the test port, add DCBX Type II TLVs as follows, with the willing bit set to 1 (On). a. PFC. b. Application. c. Priority Group. 5. Start DCBX on the port. 6. Verify DCBX between the test port and the DUT port is Synced. 7. Verify the Operating Mode for PFC, Application, and Priority Group converge to a value of True (On). Variables & Relevance Variable Relevance Willing Bit Generally set to 1 for each DCBX Feature in the Header on both the DUT and the Tester. DCBX TLV Generally at the time of this writing a Type II TLV is most common in implementations. Desired Result DCBX should be in sync. All features should reach Operational Mode On. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 42 Key Measured Metrics Statistic Relevance DCBX Sync State for each Feature In Sync Operational Mode for PFC Feature On Operational Mode for Priority Group Feature On Operational Mode for Application Feature On Analysis If any of the PFC, Application, or Priority Group features are not synchronizing one of the first things to check is the willing bit, which must be set to On or Accepting. If the willing bit is on and the problem still exists then the versions of DCBX should be carefully checked as at the time of this writing there is a lot of movement in the DCBX implementations. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 43 CFA_013 Converged LAN/SAN Queueput with multicast traffic Abstract This test determines the performance and scalability of a cloud-based converged LAN and SAN core fabric and virtual server access switching handling multiple traffic types including LAN and SAN. Determines the queueput of the DUT for all traffic classes using a full mesh topology with a mix of multicast and unicast LAN traffic. References: IETF draft-player-dcb-benchmarking-02, IEEE 802.1Qbb, Test Case CFA_11, Test Case CFA_12. For product verification and engineering. Description A cloud infrastructure with converged LAN and SAN consists of core fabric and virtual server access top-of-rack switches with Ethernet and Fibre Channel interfaces. Servers are typically connected to top-of-rack switches using 10 GB Ethernet. Server Input/Output (I/O) consists of three main categories of traffic types, LAN, SAN with Fibre Channel (FCoE) and Inter-Processor Communication/Remote Direct Memory Access (IPC/RDMA over Infiniband/RoCE-RDMA over Converged Enhanced Ethernet). To transport SAN and IPC which require a lossless medium Ethernet has been enhanced to support lossless transport per VLAN priority in IEEE Data Center Bridging working group specification 802.1Qbb Priority Flow Control. To guarantee a certain amount of bandwidth per traffic type, 802.1Qaz Enhanced Traffic Selection enables setting bandwidth percentages per VLAN priority on a shared Ethernet link. Cloud-converged LAN and SAN Queueput utilizes draft DCB Queueput and determines the throughput per VLAN priority of IEEE enabled Data Center Bridging switches verifying converged LAN and SAN fabrics and switches ability to perform and scale to cloud infrastructure requirements levels. Using multicast, a search algorithm is used to determine the Queueput for each Classification Group. Relevance Queueput is the fundamental best-practice way to measure the ability of the DUT to forward traffic and is the best way to test lossless Ethernet given that an RFC 2544 style test is not possible. At the time of this writing, Converged Enhanced Ethernet ports enable VLAN tags on all traffic types and assign a 3-bit priority to each frame for a total of eight priorities. Each priority may be paused independently of the others. A typical use case is to configure the port to limit the bandwidth assigned to the LAN traffic while allowing the SAN (FCoE traffic) traffic unlimited bandwidth and lowest latency. Version 1.1 Test Category Data Center Bridging. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 44 PASS [ ] Performance [ ] Availability [ ] Security [x] Scale Required Tester Capabilities The tester should have the ability to generate stateful storage transaction flows across the DUT, including PFC, and measure the effects. The tester must be able to make live content changes to existing content. Topology Test Procedure 1. Reserve five test ports. 2. Connect cabling to the DUT. 3. Configure DUT for FCoE with PFC running on VLAN X. 4. Set a pause queue of Y on the DUT. 5. Create 1 emulated device on each port that will represent a Virtual Machine. a. This will be used for both LAN and SAN traffic. b. Create a VLAN header for the frames sent from each device with the VLAN corresponding to the FCoE VLAN on the DUT (VLAN X). c. Leave the WWN and MAC Addresses to Default (optional: may change them). 6. Pick the FCoE draft version which corresponds to your DUT version of FCoE (choose from FIP, FIP Interop, or FCoE). 7. Set your FIP Priority, FC Map, and BB Credit to match the DUT settings as well. 8. Ensure your DUT and tester ports have a MTU of at least 9000. 9. Ensure the FC payload size on the test traffic is set to 2112 bytes. 10. For each pair of ports, set the host type to Initiator on one port and Target on another port a. Alternatively set all ports to Both Initiator and Target. b. This makes the conversion of pause quanta to switch buffer size straightforward. 11. Configure the test equipment to report latency and Jitter results in a single test pass. 12. Create three traffic groups, one for LAN and one for SAN (FCoE) Traffic and one for Multicast LAN Traffic. 13. For unicast LAN Traffic. a. Create a Partial Mesh of Unicast traffic between all ports. b. Set the VLAN Priority to match the Pause Queue. c. Configure LAN Traffic to be 40% of the maximum bandwidth on the port. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 45 14. For Multicast LAN Traffic. a. Create a one-to-many flow of multicast traffic from one sender to the receiver ports. b. Set the VLAN Priority to Match the Pause Queue. c. Configure LAN Traffic to be 10% of the maximum bandwidth on the port. 15. For the SAN Traffic. a. Create a Partial Mesh of traffic between all ports. b. Set the VLAN Priority to Match the Pause Queue. c. Set a Binary Search between 50% and 100% of Wire Rate for this traffic. 16. Configure the Pause Queue mapping in the DUT to Pause the LAN traffic as required. 17. Set a test trial duration of at least 300 seconds. 18. Configure a results collection delay of at least 5 seconds to ensure any late frames are received. 19. Optionally do some learning to populate various forwarding tables on the DUT. 20. Optionally configure DUT forwarding table timeouts to be greater than 300 seconds. 21. Once learning is completed, run the test. 22. At test completion, verify there are no Dropped Frames for any class of traffic. 23. Measure latency for each step of the Binary Search. 24. Verify that the latency of the multicast LAN traffic is no greater than the latency of the unicast LAN Traffic. 25. Measure the Queueput for the SAN Traffic in the presence of 50% LAN traffic. Variables & Relevance Variable Relevance Latency Type LILO Last In Last Out. Results Collection Delay Some amount of time usually at least 5 seconds to allow any late frames stuck in buffers to be forwarded. Frame Size Must be a multiple of 64 Bytes to ensure easy mapping of the available remaining bandwidth after application of traffic to Quanta PFC Quanta Priority Flow Control Quanta. At the time of this writing often implemented in an XON/XOFF method. VLAN ID X Must match on DUT and Tester Pause Queue Y Must match on DUT and Tester. Pause Queue is independent of VLAN ID, however VLAN CLP bits must match the Pause Queue. Test Traffic Since DCB devices are expected to support multiple traffic Classifications, it is RECOMMENDED to benchmark DCB devices with multiple Classification Groups. Desired Result Line rate performance of multicast traffic. Key Measured Metrics Statistic Relevance Queueput for SAN Traffic The maximum forwarding rate (in % of Line Rate) of the SAN traffic prior to Pause Frames being sent by the DUT to the tester port Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 46 Analysis It is highly desirable that the rate of SAN Traffic prior to the reception of the first Pause Frame for that priority (for short, SANPP) plus the rate of the LAN traffic is equal to 100% of Line Rate. Line Rate = (SANPP + LAN Traffic Rate) in bits per second. It is important not to mix frames per second and bits per second, as it is likely that LAN and SAN traffic will use different frame sizes. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 47 CFA_014 Multiple redundant path performance test Abstract The purpose of this test case is to test the performance and bandwidth achieved by deploying the Transparent Interconnection of Lots of Links (TRILL) standard based on the IETF document draft draft-perlman-rbridge-03.txt. The goal is to verify that all switches forward the traffic with zero frame loss, validating the ability to load-share across all redundant connections. Given a pair of 10 Gbps links, the total bandwidth should be twice the bandwidth per link, expressed mathematically 2 x 10 Gbps = 20 Gbps. This must be tested using traffic generators that can transmit at line rate in a cloud computing architecture. This test case is relevant to the requirement that interswitch rates can achieve 10 Gbps or more without upgrading hardware to 40 Gbps or 100 Gbps technologies. Description This test validates the support for 16 redundant paths between two switches. Both edge switches are configured with 16 EtherChannel interfaces, each with four links. Test ports send traffic between all emulated hosts for five minutes. The goal is to verify that all switches forward the traffic with zero frame loss, validating the ability to load-share across 16 redundant connections Target Users Switch Test Engineers Target Device Under Test (DUT) Layer 2 switches that are 10 Gbps capable and support the advanced switching feature of multiple redundant interswitch links. Reference draft-perlman-rbridge-03.txt. Relevance All major layer 2 switching architectures are trending toward simpler and flatter networks. The IETF TRILL standardization effort points towards a flat future for data center network design to increase rates beyond 10 Gbps and providing redundancy. Interswitch rates need to achieve 10 Gbps or more without upgrading hardware to 40 Gbps or 100 Gbps technologies. Version 1.0 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 48 Test Category Testing Datacenter Ethernet Storage PASS [X] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to test multiple links across many unique hosts while measuring key performance metrics. Topology Diagram Test Procedure Where X is the number of redundant paths between two switches, and Y is the number of physical links in each redundant path, using 2 edge switches: 1. Configure both edge switches with X redundant path groups, each consisting of Y physical links between switches. 2. Create emulated hosts on all test ports connected to the same quantity access ports as the Y physical links on the edge in order to provide the maximum load. 3. Run an RFC-2544 test to determine the maximum traffic rate supported, which will be Z. 4. Run Z rate traffic between all emulated hosts for five minutes. 5. Verify that all switches forward the Z traffic rate with zero frame loss, validating the ability to load-share across X redundant connections and Y physical connections. Control Variables & Relevance Variable Relevance Default Value X The number of redundant path groups. 1 Y The number of physical links per redundant path group. 2 Z The maximum traffic rate supported by the test bed as determined by an RFC-2544 test. 20 Gbps Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 49 Key Measured Metrics Statistic Relevance Metric Unit Z Maximum traffic rate that must be found in order to run the frame loss test. Gbps Lost frames Needs to be zero to validate the test. Ordinal Numeric Desired Result All switches forward the traffic with zero frame loss, validating the ability to load-share across all redundant connections. Analysis If there is frame loss, the frame rate should be lowered to a point where zero frame loss is obtained, and the traffic rate at that point becomes the limiting factor. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 50 CFA_015 Multiple redundant path availability test Abstract The purpose of this test is to determine how fast a switching fabric using multiple redundant paths converges after a worst-case failure, such as a complete power failure to a switch. To perform this test, while sending traffic, kill power completely to one of the switches and derive failover time from the resulting frame loss. For layer 2 switching in cloud computing architectures, the key issue is how quickly the network reroutes traffic around a failed link or switch. Recovering from failure of a switch due to power instability must be tested as it is a key design goal that these algorithms and protocols were designed to mitigate. Description While offering the same maximum forwarding rate and traffic pattern as derived by test case DCE_001 (Y path, X link test), kill power to one of the switches. Do not kill power to more than one switch. This will have the effect of causing an unplanned outage that the switch and switching fabric cannot know of beforehand, and is an accurate worst-case scenario for a single failure. This will cause the failure of all links connected to this switch. This failure must result in the convergence of the switching fabric. During this convergence there will be guaranteed frame loss. Failover time in seconds, or fractions thereof, must be calculated. Given that frames are sent at a constant rate, the failover time can be derived by the frame loss observed by the target emulated node. Repeat the test four times, powering off the spine switches one at a time. The expected results are that the multiple redundant paths converge within less than a second, much faster than spanning tree. Target Users Switch Test Engineers Target Device Under Test (DUT) Layer 2 switches that are 10 Gbps capable and support the advanced switching feature of multiple redundant interswitch links. Reference draft-perlman-rbridge-03.txt 802.1aq (optional) IS-IS (ISO/IEC 10589:2002 and RFC 1142) Relevance All major Layer-2 switching architectures are trending towards simpler and flatter networks. The IETF TRILL standardization effort and IEEE 802.1qa progress both point toward a flat future for Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 51 data center network design to increase rates beyond 10 Gbps and provide redundancy. Interswitch rates must be greater than 10 Gbps without upgrading hardware to 40 Gbps or 100 Gbps technologies. The other value is redundancy, resilience and availability in the presence of power failures to switches. Version 1.0 Test Category Testing datacenter Ethernet storage PASS [X] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to test multiple links across many unique hosts. Detect link failures. Provide enough data to calculate failover time. Topology Diagram Test Procedure Where X is the number of redundant paths between two switches, and Y is the number of physical links in each redundant path, using 2 edge switches: 1. Configure the traffic pattern and physical connections as per DCE_001. 2. Derive the maximum traffic rate Z as per DCE_001. 3. Transmit continuous traffic at the maximum traffic rate Z. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 52 4. Measure the received, no-drop traffic frame K in frames per second (fps) observed on the target test node, recorded as K fps. 5. Without administratively notifying any switches, kill power by removing the power cable to switch N where switch N is the last hop switch on the path to the target emulated test nodes. a. Do not use any other method besides pulling the cable, as it is possible that logic could be attached to the a power switch on the switch to notify the switch power supplies and forwarding engine that power loss is imminent. b. Do not stop the continuous traffic running at rate Z. 6. Using the test equipment, verify via live statistics and graphs that the frame rate falls off and then recovers. This can be done manually or in an automated fashion. a. If the frame rate does not recover, the test has failed and cannot proceed further and the test engineer must diagnose the cause of the failure to converge and recover. 7. While traffic runs continuously at Z, observe that the dropped frame statistic on the test equipment target port becomes stable, no longer increments, and remains unchanged for 60 seconds. a. This can be timed manually or preferably by using a built in timer in the test equipment user software that can automatically perform stopwatch-time events. b. Test equipment must have the capability to measure dropped frames from multiple source ports. c. Dropped frame count D must be the aggregate of all frames dropped from all sources. 8. Record the dropped frame count as D. a. Derive and record the empirical convergence time C in seconds to 6 places of accuracy. C seconds = D / K. 9. Once the test iteration is completed, repeat for all spine switches. Control Variables & Relevance Variable Relevance Default Value X The number of redundant path groups. Arbitrary, at least 1 Y The number of physical links per redundant path group. Arbitrary, at least 2 Z The maximum traffic rate supported by the test bed as determined by an RFC-2544 test. Expected to be 10G line rate for a modern cloud computing architecture N Last hop switch. None K Frames per second. As determined Key Measured Metrics Statistic Relevance Metric Unit C Convergence time. Seconds to 6 places of accuracy D Dropped frame count. Ordinal Numeric Desired Result Convergence performance should exceed that of spanning tree by an order of magnitude at the slowest. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 53 Analysis Convergence times for each of the four iterations are expected to be within 10% of all times. If convergence fails at any of the four iterations, this test is considered incomplete because the convergence time cannot be determined. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 54 CFA_016 Microsoft Exchange workload storage testing over FCoE Abstract The purpose of this test case is to test storage performance in a cloud computing environment using FCOE. This test determines the performance of a Microsoft Exchange Workload. This test determines the maximum transaction processing workload the DUT can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8- bit Byte) Per Second). Description This test determines the maximum transaction processing workload the DUT can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8- bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The Exchange workload is modeled with the following specifications: Application Block Size Randomness Exchange 2003 4K 80% Exchange 2007 8K 80% Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) Fibre Channel (or dedicated FCoE) Storage Target All DCB infrastructure in an end-to-end data center topology, in particular the 10G DCB switching fabric between the hosts and the storage. Reference TPC-C benchmark © Transaction Processing Performance Council International Electrotechnical Commission (for definition of Disk KiloByte as 1000 Bytes) 802.1Qbb 802.1Qaz 802.1Qau Relevance Storage testing to TPC standards has been done for many years, traditionally on local disks and across dedicated Storage Area Networks. With a converged cloud computing LAN and SAN Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 55 network, new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well, such as Solid-State Drives (SSD). Version 1.0 Test Category CRA PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulated many storage SCSI initiators from a simple 10G point of view and also from a virtualized perspective that takes into consideration VN port and configurations, including multiple VNICS in a cloud computing architecture. The tester must have the capability to run as a Virtual Machine on a hypervisor. Topology Diagram Test Procedure 1. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the testers Virtual Machines. Optionally, add physical DCB-capable 10GE tester ports. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e Connectivity, optionally offering DCB features. 2. Connect the host interface to the DCB Switch, optionally through other infrastructure such as DCB-aware switches. 3. Connect the DCB Switch to the Storage Target, optionally through other infrastructure such as core switches. 4. Configure the tester as shown in the Control Variables and Relevance Table. 5. Run the test for 120 seconds. 6. Record the results. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 56 Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the Transfer Block Size, in KiloBytes, a KiloByte defined as 1000 bytes. Exchange 2003: 4KB Exchange 2007: 8KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 80% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. Exchange 2003: 60% Exchange 2007: 55% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. Must be larger than Memory Cache to effect a proper test. Must be large enough so that all volume drives are accessed. Full Users per CPU Mapping of emulated users to CPU One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be greater than one up to 16. 4 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second . Ordinal Numeric MB/s MegaBytes per Second (MegaByte defined as 1000000 Bytes). Ordinal Numeric (MegaBytes) Desired Result No specific pass or fail threshold. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host, to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size, especially given that certain technologies such as Fibre Channel have a maximum transfer size on the wire of 2KB per layer 2 frame. Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 57 CFA_017 Microsoft Exchange workload storage testing over iSCSI Abstract The purpose of this test case is to test storage performance in a cloud computing environment using iSCSI. This test determines the performance of a Microsoft Exchange Workload. This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). Description This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The Exchange workload is modeled with the following specifications: Application Block Size Randomness Read/write Ratio Exchange 2003 4K 80% 60% read (40% write) Exchange 2007 8K 80% 55% read (45% write) Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) iSCSI Storage Target All DCB infrastructure in an end-to-end data center topology, in particular the 10G DCB switching fabric between the hosts and the storage. Reference TPC-C benchmark © Transaction Processing Performance Council International Electrotechnical Commission (for definition of Disk KiloByte as 1000 Bytes) 802.1Qbb 802.1Qaz 802.1Qau Relevance Storage testing to TPC standards has been done for many years, traditionally on local disks, and across dedicated Storage Area Networks. With a converged cloud computing LAN and SAN Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 58 network, the new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well such as Solid-State Drives (SSD). Version 1.0 Test Category CRA PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulated many storage SCSI initiators from a simple 10G point of view and also from a virtualized perspective that takes into consideration VN port and configurations including multiple VNICS in a cloud computing architecture. The tester must have the capability to run as a Virtual Machine on a hypervisor. Topology Diagram Test Procedure 1. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the tester Virtual Machines. Optionally, add physical DCB-capable 10GE tester ports. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e Connectivity, optionally offering DCB features. 2. Connect the host interface to the DCB Switch, optionally through other infrastructure such as DCB-aware switches. 3. Connect the DCB Switch to the iSCSI Target, optionally through other infrastructure such as core switches. 4. Configure the tester as shown in the Control Variables and Relevance Table. 5. Run the test for 120 seconds. 6. Record the results. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 59 Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the Transfer Block Size, in KiloBytes, a KiloByte defined as 1000 Bytes. Exchange 2003: 4KB Exchange 2007: 8KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 80% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. Exchange 2003: 60% Exchange 2007: 55% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. It must be larger than the Memory Cache to effect a proper test. It must be large enough so that all Volume drives are accessed. Full Users per CPU Mapping of emulated users to CPU. One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be greater than one up to 16. 4 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second. Ordinal Numeric MB/s MegaBytes per Second (MegaByte defined as 1000000 Bytes). Ordinal Numeric (MegaBytes) Desired Result No specific pass or fail threshold. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size, especially given that certain technologies such as FC have a maximum transfer size on the wire of 2KB per layer 2 frame. Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 60 CFA_018 Accelerated vs unaccelerated iSCSI storage testing Abstract The purpose of this test case is to test storage performance in a cloud computing environment over iSCSI to determine if lack of acceleration at various junctures affects performance. This test uses the Server Transaction Processing Workload. This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). Description This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The transaction processing workload is modeled with the following specifications: 8KB or 2KB transfer size 100% random I/O 67% read I/O Full disk capacity 1 user per CPU Vary number of outstanding I/Os Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) Accelerated and unaccelerated host interface adaptors such as iSCSI adaptors which offer special-purpose silicon for iSCSI acceleration. iSCSI accelerated interfaces on storage arrays themselves. DCB and non-DCB switching fabric between the hosts and the storage. Reference TPC-C benchmark © Transaction Processing Performance Council 802.1Qbb 802.1Qaz 802.1Qau Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 61 Relevance Storage testing to TPC standards has been done for many years, traditionally on local disks, and across dedicated Storage Area Networks. With a converged cloud computing LAN and SAN network, the new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well, such as Solid-State Drives (SSD’s). Version 1.0 Test Category CRA PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulated many storage SCSI initiators from a simple 10G point of view and also from a virtualized perspective that takes into consideration VN port and configurations including multiple VNICS in a cloud computing architecture. The tester must have the capability to run as a Virtual Machine on a hypervisor. It is also required that the tester be able to generate other traffic types at the same time. Topology Diagram Test Procedure 1. Build a data center DCB topology using: a. A Hypervisor host which has an accelerated 10GE interface presented to the VM’s. Accelerated means enabling acceleration features for iSCSI or perhaps employing multiple physical NICs as one virtual NIC, along with associated technology. The interface must be DCB compliant. b. Use any DCB Data Center switch c. A DCB Data Center switch, optionally configured for acceleration features. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 62 d. An iSCSI Target offering 10Gb/e connectivity, optionally using an accelerated interface, optionally offering DCB features. 2. Connect host interface to DCB Switch, optionally through other infrastructure such as DCB- aware switches. 3. Connect DCB Switch to iSCSI Target, optionally through other infrastructure such as core switches. Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the transfer 100% 8KB ; 100% 2KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 100% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. 67% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. Must be larger than Memory Cache to effect a proper test. Full Users per CPU Mapping of emulated users to CPU. One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be some number greater than one, such as 16. 4 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second is the key metric. Ordinal Numeric MB/s MegaBytes per Second. Ordinal Numeric (MegaBytes) Desired Result It is expected that accelerated interfaces, and acceleration on the switching Fabric and optionally on the Storage Target, will produce a significant performance enhancement over their unaccelerated counterparts. No specific pass or fail threshold is specified in this test. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size, especially given that certain technologies such as Fibre Channel have a maximum transfer size on the wire of 2KB per layer 2 frame. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 63 Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 64 CFA_019 HTTP, FTP, and SIP with storage testing Abstract The purpose of this test case is to test storage performance in a converged LAN and SAN environment where HTTP, FTP, and SIP traffic is sent alongside Storage traffic over the same 10GBE interface. This test uses the Server Transaction Processing Workload. This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second), and will also detect if there are any unsuccessful HTTP, FTP, or SIP transactions. Description This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The transaction processing workload is modeled with the following specifications: 8KB or 2KB transfer size 100% random I/O 67% read I/O Full disk capacity 1 user per CPU HTTP, FTP, and SIP are configured for a relatively low 100 concurrent connections per second each using common object sizes Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) Server processing power and overall performance including the memory and CPU as it relates to the tester Virtual machine. Accelerated and unaccelerated host interface adaptors such as iSCSI adaptors which offer special-purpose silicon for iSCSI acceleration. iSCSI accelerated interfaces on storage arrays themselves. DCB and non-DCB switching fabric between the hosts and the storage. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 65 Reference TPC-C benchmark © Transaction Processing Performance Council 802.1Qbb 802.1Qaz 802.1Qau Relevance Storage testing to TPC standards has been done for many years, traditionally done on local disks and across dedicated Storage Area Networks. With a converged cloud computing LAN and SAN network, the new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well, such as Solid-State Drives (SSD). It is also commonly expected that Layer 4-7 Traffic will be present on the same interface, as many servers are Web servers. As such, HTTP, FTP, and SIP protocols are included in this test. Version 1.0 Test Category Storage Testing. PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulate many storage SCSI initiators from a virtualized perspective while also doing HTTP, FTP, and SIP protocols simultaneously on the same link. Topology Diagram Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 66 Test Procedure 1. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the tester Virtual Machines. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e connectivity, optionally offering DCB features. d. A second Hypervisor host on which to deploy the virtual tester, or a physical tester port for the L4-7 Protocols. 2. Connect host interface to DCB Switch, optionally through other infrastructure such as DCB- aware switches. 3. Connect DCB Switch to iSCSI Target, optionally through other infrastructure such as core switches. 4. Configure HTTP, FTP, and SIP Servers on the Virtual Machine. 5. Configure HTTP, FTP, and SIP Clients on another virtual tester, or on a physical tester port. 6. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the tester Virtual Machines. Optionally, add physical DCB-capable 10GE tester ports. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e Connectivity, optionally offering DCB features. 7. Connect host interface to DCB Switch, optionally through other infrastructure such as DCB- aware switches. 8. Configure the tester as shown in the Control Variables and Relevance Table 9. Run the test for 120 seconds. 10. Record the results. Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the transfer. 100% 8KB ; 100% 2KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 100% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. 67% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. Must be larger than Memory Cache to effect a proper test. Full Users per CPU Mapping of emulated users to CPU. One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be some number greater than one, such as 16. One (1). HTTP Object Size Size of the HTTP object on the page. 1024KB HTTP Version 1.0 or 1.1 1.1 Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 67 Variable Relevance Default Value HTTP Concurrent Connections and test Duration This is not a scale test of Connection establishment as the web server is not under test. However, any web server should handle 100 concurrent connections. 100 and 2 Minute load profile. HTTP Connections per Second Not meant to stress-test the ability of the server to set up connections. 10 FTP Transfer Object Size Size of FTP Transfer for example the size of a file transferred via FTP. 10MB FTP and SIP concurrent connections About half and a quarter than of the HTTP concurrent connections respectively. 50 and 25 SIP Call Duration Duration of SIP call. 2 Minutes SIP Codec CODEC used for encoding of the voice call. G.711 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second is the key metric Ordinal Numeric MB/s MegaBytes per Second Ordinal Numeric (MegaBytes) Successful HTTP Transactions Any unsuccessfuls indicate a bottleneck or configuration problem in the host, or switching fabric Ordinal Numeric Successful FTP Transactions Any unsuccessfuls indicate a bottleneck or configuration problem in the host, or switching fabric Ordinal Numeric Successful SIP Transactions Any unsuccessfuls indicate a bottleneck or configuration problem in the host, or switching fabric Ordinal Numeric Desired Result 100% Successful transactions for HTTP, FTP, and SIP. Any failure for any of these protocols constitutes a failure of this test. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size. Especially given that certain technologies such as Fibre Channel have a maximum transfer size on the wire of 2KB per layer 2 frame. Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 68 CFA_020 Server transaction processing storage testing over FCoE Abstract The purpose of this test case is to test Server Transaction Processing Storage performance in a cloud computing environment using FCOE. This test determines the performance of a Server Transaction Processing Workload. This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). Description This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The transaction processing workload is modeled with the following specifications: 8KB or 2KB transfer size 100% random I/O 67% read I/O Full disk capacity 1 user per CPU Vary number of outstanding I/Os Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) Fibre Channel Storage Target All DCB infrastructure in an end-to-end data center topology, in particular the 10G DCB switching fabric between the hosts and the storage. Reference TPC-C benchmark © Transaction Processing Performance Council International Electrotechnical Commission (for definition of Disk KiloByte as 1000 Bytes) 802.1Qbb 802.1Qaz 802.1Qau Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 69 Relevance Storage testing to TPC standards has been done for many years, traditionally on local disks, and across dedicated Storage Area Networks. The relevance of this test is that with a converged cloud computing LAN and SAN network, the new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well such as Solid-State Drives (SSD). Version 1.0 Test Category CRA PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulated many storage SCSI initiators from a dedicated 10G test port point of view and also from a virtualized perspective that takes into consideration VN port and configurations including multiple VNICS in a cloud computing architecture. The tester must have the capability to run as a Virtual Machine on a hypervisor. Topology Diagram Test Procedure 1. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the tester Virtual Machines. Optionally, add Spirent TestCenter physical DCB-capable 10GE ports. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e Connectivity, optionally offering DCB features. 2. Connect host interface to DCB Switch, optionally through other infrastructure such as DCB- aware switches. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 70 3. Connect DCB Switch to the Storage Target, optionally through other infrastructure such as core switches. 4. Configure the tester as shown in the Control Variables and Relevance Table. 5. Run the test for 120 seconds. 6. Record the results. Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the Transfer Block Size, in KiloBytes, a KiloByte defined as 1000 bytes. 100% 8KB ; 100% 2KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 100% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. 67%% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. Must be larger than Memory Cache to effect a proper test. Must be large enough so that all Volume drives are accessed. Full Users per CPU Mapping of emulated users to CPU One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be greater than one up to 16. 4 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second. Ordinal Numeric MB/s MegaBytes per Second (MegaByte defined as 1000000 Bytes). Ordinal Numeric (MegaBytes) Desired Result No specific pass or fail threshold. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size, especially given that certain technologies such as FC have a maximum transfer size on the wire of 2KB per layer 2 frame. Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 71 CFA_021 Server transaction processing storage testing over iSCSI Abstract The purpose of this test case is to test storage performance in a cloud computing environment using iSCSI. This test determines the performance of a Server Transaction Processing Workload. This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). Description This test case determines the maximum transaction processing workload the device under test can reach and sustain, as measured in IOPS (Input Output Operations Per Second) and in MB/s (Mega Bytes (8-bit Byte) Per Second). The I/O profile for transaction processing applications is variously referred to as a transaction processing workload, database server workload, OLTP workload, or TPC-C workload. Under any of these names, the synthetic workload reflects the I/O profile of a database server accessing its storage subsystem while processing transactions. The transaction processing workload is modeled with the following specifications: 8KB or 2KB transfer size 100% random I/O 67% read I/O Full disk capacity 1 user per CPU Vary number of outstanding I/Os Target Users Cloud Computing Test Engineers Storage Test Engineers Target Device Under Test (DUT) iSCSI Storage Target All DCB infrastructure in an end-to-end data center topology, in particular the 10G DCB switching fabric between the hosts and the storage. Reference TPC-C benchmark © Transaction Processing Performance Council International Electrotechnical Commission (for definition of Disk KiloByte as 1000 Bytes) 802.1Qbb 802.1Qaz 802.1Qau Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 72 Relevance Storage testing to TPC standards has been done for many years. This has traditionally been done on local disks and across dedicated Storage Area Networks. With a converged Cloud Computing LAN and SAN network, the new devices and infrastructure in the path between the SCSI controllers on the server and the SCSI targets are all unknown quantities. Other areas of storage are changing as well such as Solid-State Drives (SSD). Version 1.0 Test Category CRA PASS [ X ] Performance [ X ] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to emulate many storage SCSI initiators from a simple 10G point of view and also from a virtualized perspective that takes into consideration VN port and configurations including multiple VNICS in a cloud computing architecture. The tester must have the capability to run as a Virtual Machine on a hypervisor. Topology Diagram Test Procedure 1. Build a data center DCB topology using: a. One or more Hypervisor hosts which present a DCB 10GE interface to the tester Virtual Machines. Optionally, add physical DCB-capable 10GE tester ports. b. A DCB Data Center TOR (Top of Rack) switch optionally with other switching fabric elements included. c. A Storage Target offering 10Gb/e Connectivity, optionally offering DCB features. 2. Connect the host interface to the DCB Switch, optionally through other infrastructure such as DCB-aware switches. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 73 3. Connect the DCB Switch to the Storage Target, optionally through other infrastructure such as core switches. 4. Configure the tester as shown in the Control Variables and Relevance Table. 5. Run the test for 120 seconds. 6. Record the results. Control Variables & Relevance Variable Relevance Default Value Transfer Size Size of the Transfer Block Size, in KiloBytes, a KiloByte defined as 1000 Bytes. 100% 8KB ; 100% 2KB Random I/O Percent In developing storage workload profiles a certain percentage of randomness is always included. It is often a high percentage. 100% Read I/O Percent Different applications have different ratios of read to write activity. The Write I/O percent is 100% - Read I/O Percent. 67%% Disk Capacity A limit of the amount of bytes written to the disk. Full means no limit. Must be larger than Memory Cache to effect a proper test. Must be large enough so that all volume drives are accessed. Full Users per CPU Mapping of emulated users to CPU. One (1) Quantity of Outstanding I/O Sets the number of simultaneous outstanding I/Os per disk. Multiple requests can be queued by increasing this to be greater than one up to 16. 4 Key Measured Metrics Statistic Relevance Metric Unit IOPS Input Outputs per Second. Ordinal Numeric MB/s MegaBytes per Second (MegaByte defined as 1000000 Bytes). Ordinal Numeric (MegaBytes) Desired Result No specific pass or fail threshold. At the time of this writing, for cloud computing performance around 200K IOPS per 10GE link using a 4KB block size should be expected. Analysis Many things affect performance and this test depends on many factors, such as the memory speed of the host to the drive latency, to the data center infrastructure itself. Given those factors remain constant, the largest factor is affecting performance is block size, especially given that certain technologies such as FC have a maximum transfer size on the wire of 2KB per layer 2 frame. Note that iSCSI should be able to fit an 8KB block into a 9K Jumbo Ethernet frame if the TCP stack is correctly tuned to fit that maximum segment size. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 74 CFA_022 Verify congestion notification Abstract This test case verifies that the device under test correctly implements Congestion Notification (CN) for Data Center Bridging (DCB) under high traffic load. CN provides a means for DCB switches to notify the source of traffic to throttle back the rate of one or more classes of traffic. Throttling back (TB) refers to decreasing the sending rate of traffic. The mechanism by which a switch determines the level of congestion it is experiencing is referred to as Control Point Dynamics (CPD), while the mechanism by which a source (sometimes referred to as an End-Station (ES) throttles back is referred to as Rate Limit Dynamics (RLD). This test methodology describes approaches to test both CPD and RLD. By transmitting traffic loads on multiple ports at 10G line rate with a mix of Ethernet and FCoE traffic frames, the Congestion Notification protocol operation can be thoroughly tested. Description This test uses at least three 10G Ethernet ports on a DCB switch, which is the Device under Test (DUT). Each transmitting port of Spirent TestCenter runs the 802.1Qau protocol and acts as either a switch or an end-station. Spirent TestCenter transmits traffic on the first two 10G Ethernet Ports and receives traffic on the third 10G Ethernet Port, resulting in a two-to-one oversubscription ratio, guaranteeing congestion on the DUT. The receiving test port operates at 100% utilization (10G line rate) and measures lost frames about a rate of 50%. As a switch, the DUT becomes aware of the congestion through CPD and transmits CN frames to the Spirent TestCenter transmitting ports, informing them of the degree of congestion. In turn the Spirent TestCenter ports perform the RPD mechanism and throttle the traffic accordingly. Target Users Development and test teams qualifying DCB switches for market. This also applies to teams testing all devices that include the ETS feature. Target Device Under Test (DUT) Physical Ethernet switch (or any device) supporting ETS and having at least 3 x 10G ports. It is recommended that a fully-loaded switch test be performed. Reference 802.1Qaz Enhanced Transmission Selection Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 75 Relevance DCB switches are rapidly replacing traditional Ethernet-only switches to support converged cloud computing environments. DCB switches normally must include support for ETS, therefore ETS must be tested, and tested at performance levels on multiple ports. Version 1.0 Test Category DCRA PASS [ ] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to both transmit and accurately measure different traffic classes. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 76 Topology Diagram Test Procedure 1. Connect the physical topology as shown: a. Each group of 3 ports is known as a tuple. b. Tuples should be scaled to the maximum ports supported by the switch or whole fraction thereof. For example, if the switch supports 48 ports, then you would have 16 tuples. 2. Configure the DUT for ETS with egress port traffic policies as per Table A, the table of traffic classes. a. The egress port is the third port in every 3-port Tuple. 3. Configure the Spirent TestCenter transmission ports in the tuples with traffic ratios from Table B, the table of traffic rates. 4. You run N x M iterations of traffic within a single test, where N is the quantity of traffic policies from Table A, and M is the quantity of traffic policies from Table B. 5. Transmit traffic for 60 seconds. 6. Verify at the end of traffic transmission that the expected traffic ratios are received with tolerance T. a. Result should match Table C, egress traffic ratios. b. The excess bandwidth is to be shared fairly. This has to be taken into account. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 77 Control Variables & Relevance Test Iteration FCoE Traffic Percentage Ethernet Traffic Percentage 1 50% 50% 2 80% 20% 3 20% 80% 4 40% 60% 5 60% 40% Table A – Traffic Classes for ETS Configuration Test Iteration FCoE Traffic Percentage Ethernet Traffic Percentage 1 50% 50% 2 60% 40% 3 40% 60% 4 20% 80% 5 80% 20% Table B – Traffic Rates for Transmission Test Iteration FCoE Traffic % (~ + excess bw %) Ethernet Traffic % (~ + excess bw %) Excess Bandwidth % 1.1 50% 50% 0% 1.2 50% 40% 10% 1.3 40% 50% 10% 1.4 20% 50% 30% 1.5 50% 20% 30% 2.1 50% 20% 30% 2.2 60% 20% 20% 2.3 40% 20% 40% 2.4 20% 20% 60% 2.5 80% 20% 0% 3.1 20% 50% 30% 3.2 20% 40% 40% 3.3 20% 60% 40% 3.4 20% 80% 0% 3.5 20% 20% 60% 4.1 40% 50% 10% 4.2 40% 40% 20% 4.3 40% 60% 0% 4.4 20% 60% 20% 4.5 40% 20% 40% 5.1 50% 40% 10% 5.2 60% 40% 0% 5.3 40% 40% 20% 5.4 20% 40% 40% 5.5 60% 20% 20% Table C – Expected Egress Traffic Ratios Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 78 Key Measured Metrics See Table C above. Desired Result Correlation with 5% of Table C, above. Analysis ETS is a sophisticated QoS scheme, similar to Ethernet QoS but expanded beyond the Ethernet header into other Layer 2 traffic classes. ETS is a critical component of DCB technology for a cloud computing environment. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 79 CFA_023 Verify enhanced transmission selection Abstract This test case verifies that the device under test correctly implements policies for Enhanced Transmission Selection (ETS) under high performance load. ETS provides a means for network administrators to allocate link bandwidth to different priorities as a percentage of total bandwidth. This test also ensures the DUT properly utilizes the excess bandwidth when it is available. By transmitting traffic loads on multiple ports at 10G line rate with a mix of Ethernet and FCoE traffic frames, the policy is thoroughly tested. Description Spirent TestCenter transmit test traffic and determines whether the configured ETS policy matches the measured traffic profile. This test uses at least three 10G Ethernet ports on a Data Center Bridging (DCB) top-of-rack or end-of-row switch. This test creates QoS traffic policies using ETS to ensure a given traffic class ratio, then sends traffic in an oversubscription scenario to ensure the ETS code is exercised. Target Users Development and test teams qualifying Data Center Bridging switches for market and teams testing all devices that support ETS. Target Device Under Test (DUT) Physical Ethernet switch (or any device) supporting ETS and having at least 3 x 10G ports. It is recommended that a fully-loaded switch test be performed. Reference 802.1Qaz Enhanced Transmission Selection Relevance DCB switches are rapidly replacing traditional Ethernet-only switches to support converged cloud computing environments. DCB switches normally must include support for ETS, therefore ETS must be tested at performance levels on multiple ports. Version 1.0 Test Category DCRA Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 80 PASS [ ] Performance [X] Availability [ ] Security [ ] Scale Required Tester Capabilities Ability to both transmit and accurately measure different traffic classes. Topology Diagram Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 81 Test Procedure 1. Connect the physical topology as shown. a. Each group of 3 ports is referred to as a tuple. b. Tuples should be scaled to the maximum ports supported by the switch or whole fraction thereof. For example, if the switch supports 48 ports, then you would have 16 tuples. 2. Configure the DUT for ETS with egress port traffic policies as per Table A, the table of traffic classes. a. The egress port is the third port in every 3-port tuple. 3. Configure the Spirent TestCenter transmission ports in the tuples with traffic ratios from Table B, the table of traffic rates 4. Run N x M iterations of traffic within a single test, where N is the quantity of traffic policies from Table A, and M is the quantity of traffic policies from Table B. 5. Transmit traffic for 60 seconds. 6. At the end of traffic transmission, verify that the expected traffic ratios are received with tolerance T. a. Result should match Table C, egress traffic ratios. b. The excess bandwidth is to be shared fairly. This has to be taken into account. Control Variables & Relevance Test Iteration FCoE Traffic Percentage Ethernet Traffic Percentage 1 50% 50% 2 80% 20% 3 20% 80% 4 40% 60% 5 60% 40% Table A – Traffic Classes for ETS Configuration Test Iteration FCoE Traffic Percentage Ethernet Traffic Percentage 1 50% 50% 2 60% 40% 3 40% 60% 4 20% 80% 5 80% 20% Table B – Traffic Rates for Transmission Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 82 Test Iteration FCoE Traffic % (~ + excess bw %) Ethernet Traffic % (~ + excess bw %) Excess Bandwidth % 1.1 50% 50% 0% 1.2 50% 40% 10% 1.3 40% 50% 10% 1.4 20% 50% 30% 1.5 50% 20% 30% 2.1 50% 20% 30% 2.2 60% 20% 20% 2.3 40% 20% 40% 2.4 20% 20% 60% 2.5 80% 20% 0% 3.1 20% 50% 30% 3.2 20% 40% 40% 3.3 20% 60% 40% 3.4 20% 80% 0% 3.5 20% 20% 60% 4.1 40% 50% 10% 4.2 40% 40% 20% 4.3 40% 60% 0% 4.4 20% 60% 20% 4.5 40% 20% 40% 5.1 50% 40% 10% 5.2 60% 40% 0% 5.3 40% 40% 20% 5.4 20% 40% 40% 5.5 60% 20% 20% Table C – Expected Egress Traffic Ratios Key Measured Metrics See Table C above. Desired Result Correlation with 5% of Table C, above. Analysis ETS is a sophisticated QoS scheme, similar to Ethernet QoS but expanded beyond the Ethernet header into other Layer 2 traffic classes. ETS is a critical component of DCB technology for a cloud computing environment. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 83 Appendix A – Telecommunications Definitions APPLICATION LOGIC. The computational aspects of an application, including a list of instructions that tells a software application how to operate. APPLICATION SERVICE PROVIDER (ASP). An ASP deploys hosts and manages access to a packaged application by multiple parties from a centrally managed facility. The applications are delivered over networks on a subscription basis. This delivery model speeds implementation, minimizes the expenses and risks incurred across the application life cycle, and overcomes the chronic shortage of qualified technical personnel available in-house. APPLICATION MAINTENANCE OUTSOURCING PROVIDER. Manages a proprietary or packaged application from either the customer's or the provider's site. ASP INFRASTRUCTURE PROVIDER (AIP). A hosting provider that offers a full set of infrastructure services for hosting online applications. ATM. Asynchronous Transport Mode. An information transfer standard for routing high-speed, high- bandwidth traffic such as real-time voice and video, as well as general data bits. AVAILABILITY. The portion of time that a system can be used for productive work, expressed as a percentage. BACKBONE. A centralized high-speed network that interconnects smaller, independent networks. BANDWIDTH. The number of bits of information that can move through a communications medium in a given amount of time; the capacity of a telecommunications circuit/network to carry voice, data, and video information. Typically measured in Kbps and Mbps. Bandwidth from public networks is typically available to business and residential end-users in increments from 56 Kbps to 45 Mbps. BIT ERROR RATE. The number of transmitted bits expected to be corrupted per second when two computers have been communicating for a given length of time. BURST INFORMATION RATE (BIR). The rate of information in bits per second that the customer may need over and above the CIR. A burst is typically a short duration transmission that can relieve momentary congestion in the LAN or provide additional throughput for interactive data applications. BUSINESS ASP. Provides prepackaged application services in volume to the general business market, typically targeting small to medium size enterprises. BUSINESS-CRITICAL APPLICATION. The vital software needed to run a business, whether custom-written or commercially packaged, such as accounting/finance, ERP, manufacturing, human resources and sales databases. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 84 BUSINESS SERVICE PROVIDER. Provides online services aided by brick-and-mortar resources, such as payroll processing and employee benefits administration, printing, distribution or maintenance services. The category includes business process outsourcing (BPO) companies. COMMERCE NETWORK PROVIDER. Commerce networks were traditionally proprietary value-added networks (VANs) used for electronic data interchange (EDI) between companies. Today the category includes the new generation of electronic purchasing and trading networks. COMPETITIVE ACCESS PROVIDER (CAP). A telecommunications company that provides an alternative to a LEC for local transport and special access telecommunications services. CAPACITY. The ability for a network to provide sufficient transmitting capabilities among its available transmission media, and respond to customer demand for communications transport, especially at peak usage times. CLIENT/DEVICE. Hardware that retrieves information from a server. CLUSTERING. A group of independent systems working together as a single system. Clustering technology allows groups of servers to access a single disk array containing applications and data. COMPUTING UTILITY PROVIDER (CUP). A provider that delivers computing resources, such as storage, database or systems management, on a pay-as-you-go basis. CSU/DSU. Channel Server Unit/Digital Server Unit. A device used to terminate a telephone company connection and prepare data for a router interface. DATA MART. A subset of a data warehouse, intended for use by a single department or function. DATA WAREHOUSE. A database containing copious amounts of information, organized to aid decision- making in an organization. Data warehouses receive batch updates and are configured for fast online queries to produce succinct summaries of data. DEDICATED LINE. A point-to-point, hardwired connection between two service locations. DEMARCATION LINE. The point at which the local operating company's responsibility for the local loop ends. Beyond the demarcation point (also known as the network interface), the customer is responsible for installing and maintaining all equipment and wiring. DISCARD ELIGIBILITY (DE) BIT. Relevant in situations of high congestion, it indicates that the frame should be discarded in preference to frames without the DE bit set. The DE bit may be set by the network or by the user; and once set cannot be reset by the network. DS-1 OR T-1. A data communication circuit capable of transmitting data at 1.5 Mbps. Currently in widespread use by medium and large businesses for video, voice, and data applications. DS-3 OR T-3. A data communications circuit capable of transmitting data at 45 Mbps. The equivalent data capacity of 28 T-1s. Currently used only by businesses/institutions and carriers for high-end applications. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 85 ELECTRONIC DATA INTERCHANGE (EDI). The electronic communication of business transactions (orders, confirmations, invoices etc.) of organizations with differing platforms. Third parties provide EDI services that enable the connection of organizations with incompatible equipment. ENTERPRISE ASP. An ASP that delivers a select range of high-end business applications, supported by a significant degree of custom configuration and service. ENTERPRISE RELATIONSHIP MANAGEMENT (ERM). Solutions that enable the enterprise to share comprehensive, up-to-date customer, product, competitor and market information to achieve long-term customer satisfaction, increased revenues, and higher profitability. ENTERPRISE RESOURCE PLANNING (ERP). An information system or process integrating all manufacturing and related applications for an entire enterprise. ERP systems permit organizations to manage resources across the enterprise and completely integrate manufacturing systems. ETHERNET. A local area network used to connect computers, printers, workstations, and other devices within the same building. Ethernet operates over twisted wire and coaxial cable. EXTENDED SUPERFRAME FORMAT. A T1 format that provides a method for easily retrieving diagnostics information. FAT CLIENT. A computer that includes an operating system, RAM, ROM, a powerful processor and a wide range of installed applications that can execute either on the desktop or on the server to which it is connected. Fat clients can operate in a server-based computing environment or in a stand-alone fashion. FAULT TOLERANCE. A design method that incorporates redundant system elements to ensure continued systems operation in the event of the failure of any individual element. FDDI. Fiber Distributed Data Interface. A standard for transmitting data on optical-fiber cables at a rate of about 100 Mbps. FRAME. The basic logical unit in which bit-oriented data is transmitted. The frame consists of the data bits surrounded by a flag at each end that indicates the beginning and end of the frame. A primary rate can be thought of as an endless sequence of frames. FRAME RELAY. A high-speed packet switching protocol popular in networks, including WANs, LANs, and LAN-to-LAN connections across long distances. GBPS. Gigabits per second, a measurement of data transmission speed expressed in billions of bits per second. HOSTED OUTSOURCING. Complete outsourcing of a company's information technology applications and associated hardware systems to an ASP. HOSTING PROVIDER. Provider who operates data center facilities for general-purpose server hosting and collocation. INFRASTRUCTURE ISV. And independent software vendor that develops infrastructure software to support the hosting and online delivery of applications. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 86 INTEGRATED SERVICES DIGITAL NETWORK (ISDN). An information transfer standard for transmitting digital voice and data over telephone lines at speeds up to 128 Kbps. INTEGRATION. Equipment, systems, or subsystem integration, assembling equipment or networks with a specific function or task. Integration is combining equipment/systems with a common objective, easy monitoring and/or executing commands. It takes three disciplines to execute integration: 1) hardware, 2) software, and 3) connectivity – transmission media (data link layer), interfacing components. All three aspects of integration have to be understood to make two or more pieces of equipment or subsystems support the common objective. INTER-EXCHANGE CARRIER (IXC). A telecommunications company that provides telecommunication services between local exchanges on an interstate or intrastate basis. INTERNET SERVICE PROVIDER (ISP). A company that provides access to the Internet for users and businesses. INDEPENDENT SOFTWARE VENDOR (ISV). A company that is not a part of a computer systems manufacturer that develops software applications. INTERNETWORKING. Sharing data and resources from one network to another. IT SERVICE PROVIDER. Traditional IT services businesses, including IT outsourcers, systems integrators, IT consultancies and value added resellers. KILOBITS PER SECOND (KBPS). A data transmission rate of 1,000 bits per second. LEASED LINE. A telecommunications line dedicated to a particular customer along predetermined routers. LOCAL ACCESS TRANSPORT AREA (LATA). One of approximately 164 geographical areas within which local operating companies connect all local calls and route all long-distance calls to the customer's inter- exchange carrier. LOCAL EXCHANGE CARRIER (LEC). A telecommunications company that provides telecommunication services in a defined geographic area. LOCAL LOOP. The wires that connect an individual subscriber's telephone or data connection to the telephone company central office or other local terminating point. LOCAL/REGIONAL ASP. A company that delivers a range of application services, and often the complete computing needs, of smaller businesses in their local geographic area. MEGABITS PER SECOND (MBPS). 1,024 kilobits per second. METAFRAME. The world's first server-based computing software for Microsoft Windows NT 4.0 Server, Terminal Server Edition multi-user software (co-developed by Citrix). MODEM. A device for converting digital signals to analog and vice versa, for data transmission over an analog telephone line. MULTIPLEXING. The combining of multiple data channels onto a single transmission medium. Sharing a circuit - normally dedicated to a single user - between multiple users. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 87 MULTI-USER. The ability for multiple concurrent users to log on and run applications on a single server. NET-BASED ISV. An ISV whose main business is developing software for Internet-based application services. This includes vendors who deliver their own applications online, either directly to users or via other service providers. NETWORK ACCESS POINT (NAP). A location where ISPs exchange traffic. NETWORK COMPUTER (NC). A thin-client hardware device that executes applications locally by downloading them from the network. NCs adhere to a specification jointly developed by Sun, IBM, Oracle, Apple and Netscape. They typically run Java applets within a Java browser, or Java applications within the Java Virtual Machine. NETWORK COMPUTING ARCHITECTURE. A computing architecture in which components are dynamically downloaded from the network onto the client device for execution by the client. The Java programming language is at the core of network computing. ONLINE ANALYTICAL PROCESSING (OLAP). Software that enables decision support via rapid queries to large databases that store corporate data in multidimensional hierarchies and views. OPERATIONAL RESOURCE PROVIDER. Operational resources are external business services that an ASP might use as part of its own infrastructure, such as helpdesk, technical support, financing, or billing and payment collection. OUTSOURCING. The transfer of components or large segments of an organization's internal IT infrastructure, staff, processes or applications to an external resource such as an ASP. PACKAGED SOFTWARE APPLICATION. A computer program developed for sale to consumers or businesses, generally designed to appeal to more than a single customer. While some tailoring of the program may be possible, it is not intended to be custom-designed for each user or organization. PACKET. A bundle of data organized for transmission, containing control information (destination, length, origin, etc.), the data itself, and error detection and correction bits. PACKET SWITCHING. A network in which messages are transmitted as packets over any available route rather than as sequential messages over circuit-switched or dedicated facilities. PEERING. The commercial practice under which nationwide ISPs exchange traffic without the payment of settlement charges. PERFORMANCE. A major factor in determining the overall productivity of a system, performance is primarily tied to availability, throughput and response time. PERMANENT VIRTUAL CIRCUIT (PVC). A PVC connects the customer's port connections, nodes, locations, and branches. All customer ports can be connected, resembling a mesh, but PVCs usually run between the host and branch locations. POINT OF PRESENCE (POP). A telecommunications facility through which the company provides local connectivity to its customers. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 88 PORTAL. A company whose primary business is operating a Web destination site, hosting content and applications for access via the Web. REMOTE ACCESS. Connection of a remote computing device via communications lines such as ordinary phone lines or wide area networks to access distant network applications and information. REMOTE PRESENTATION SERVICES PROTOCOL. A set of rules and procedures for exchanging data between computers on a network, enabling the user interface, keystrokes, and mouse movements to be transferred between a server and client. RESELLER/VAR. An intermediary between software and hardware producers and end users. Resellers frequently add value (thus Value-Added Reseller) by performing consulting, system integration and product enhancement. ROUTER. A communications device between networks that determines the best path for optimal performance. Routers are used in complex networks of networks such as enterprise-wide networks and the Internet. SCALABILITY. The ability to expand the number of users or increase the capabilities of a computing solution without making major changes to the systems or application software. SERVER. The computer on a local area network that often acts as a data and application repository and that controls an application's access to workstations, printers and other parts of the network. SERVER-BASED COMPUTING. A server-based approach to delivering business-critical applications to end-user devices, whereby an application's logic executes on the server and only the user interface is transmitted across a network to the client. Benefits include single-point management, universal application access, bandwidth-independent performance, and improved security for business applications. SINGLE-POINT CONTROL. One of the benefits of the ASP model, single-point control helps reduce the total cost of application ownership by enabling widely used applications and data to be deployed, managed and supported at one location. Single-point control enables application installations, updates and additions to be made once, on the server, which are then instantly available to users anywhere. SPECIALIST ASP. Provide applications which serve a specific professional or business activity, such as customer relationship management, human resources or Web site services. SYSTEMS MANUFACTURER. Manufacturer of servers, networking and client devices. TELECOMS PROVIDER. Traditional and new-age telecommunications network providers (telcos). THIN CLIENT. A low-cost computing device that accesses applications and and/or data from a central server over a network. Categories of thin clients include Windows-Based Terminals (WBT, which comprise the largest segment), X-Terminals, and Network Computers (NC). TOTAL COST OF OWNERSHIP (TCO). Model that helps IT professionals understand and manage the budgeted (direct) and unbudgeted (indirect) costs incurred for acquiring, maintaining and using an application or a computing system. TCO normally includes training, upgrades, and administration as well as the purchase price. Lowering TCO through single-point control is a key benefit of server-based computing. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 89 TOTAL SECURITY ARCHITECTURE (TSA). A comprehensive, end-to-end architecture that protects the network. TRANSMISSION CONTROL PROTOCOL/INTERNET PROTOCOL (TCP/IP). A suite of network protocols that allow computers with different architectures and operating system software to communicate over the Internet. USER INTERFACE. The part of an application that the end user sees on the screen and works with to operate the application, such as menus, forms and buttons. VERTICAL MARKET ASP. Provides solutions tailored to the needs of a specific industry, such as the healthcare industry. VIRTUAL PRIVATE NETWORK (VPN). A secure, encrypted private connection across a cloud network, such as the Internet. WEB HOSTING. Placing a consumer's or organization's web page or web site on a server that can be accessed via the Internet. WIDE AREA NETWORK. Local area networks linked together across a large geographic area. WINDOWS-BASED TERMINAL (WBT). Thin clients with the lowest cost of ownership, as there are no local applications running on the device. Standards are based on Microsoft's WBT specification developed in conjunction with Wyse Technology, NCD, and other thin client companies. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 90 Appendix B – Stateful Playlist by QoS The following patterns represent industry best practices in modeling stateful traffic across a Device Under Test (DUT) by QoS. Name DiffServ EF (Real-Time) DiffServ 0x31 (Critical) DiffServ 0x20 (General) DiffServ 0x00 (Best Effort) Enterprise Campus Apps VoIP 15% (SIP+RTP+G.729A),U nicast Web Conference 2-Way (MPEG2-TS, VBR), SIP 5% Routing 3% OSPF Routing Updates 2%, BGP Updates 1%), Database 17% (Oracle SQLNet Updates), Corporate Web 2% , IMAP4 5% Multicast Video 13% (480i, MPGEG-2, IGMPv2, 5 Multicast Channels), Telnet/SSH (2%), CIFS 10% (1:1:3 Small/Medium/Large Ratio) Internet Web 5% HTTP (1024 Byte index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg), BitTorrent 11% Higher Education Network Administration 2% (SSH) SQL 7% SQLNet SQL Table Updates), HTTPS University Admin 3% (64 Bytes index.html, 5x 1K JPEG Images), Video Conference 5% (MPEG2TS, VBR, 480i), VoIP 5% (G.729A CODEC) FTP 7% (Large Files), HTTPS Student Services, HTTP 3%, POP3/SMTP 9%, CIFS 8% (1:1:3 Small/Medium/Large Objects, bidirectional), Multicast Video 5% (480i) IM 12% (AIM) , BitTorrent 24%, HTTP 3% (1024 Byte index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg), HTTPS 1% (64 Bytes index.html, 5x 1K JPEG Images), Mail 5%, FTP 1% (Large Files), Telnet/SSH 3% Service Providers Telnet/SSH 1% BGP Route Updates 1% N/A 50% P2P (Bit Torrent, 5% Peer to Tracker, 95% Peer- 2-Peer), 30% HTTP (1024 Byte index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg), 5% DNS, Video (MPEG2-TS 5%), SIP (G.729A 3%), Gaming (WoW 5%), 2% RAW TCP 10G Max Bandwidth No Payload, RAW TCP 1G max Bandwidth No Payload, RAW TCP Small/Medium Business Apps POP2/SMTP 15% (5:2:1 Ratio of Small/medium/Big ratio). HTTPS 20% (64 Bytes index.html, 5x 1K JPEG Images, CIFS 30% (1:1:3 Small/Medium/Large Objects, bidirectional, BitTorrent 10%, Internet Web 25% HTTP (1024 Byte index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg) WAN Accelerator Network Control 5% (Windows Domain Controller Updates), Network Logins CIFS 40% (1:1:3 Small/Medium/Large Fields). Exchange 35%(5:2:1 Small/Medium/Large ratio) HTTPS 10% (64 Bytes index.html, 5x 1K JPEG Images) BitTorrent 10% Internet AppMix 2011 50% P2P (Bit Torrent, 5% Peer to Tracker, 95% Peer- 2-Peer), 30% HTTP (1024 Byte index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg), 5% DNS, Video (MPEG2-TS 5%), SIP (G.729A 3%), Gaming (WoW 5%), 2% RAW TCP Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 91 Appendix C – MPEG 2/4 Video QoE The following information is a typical pattern for MPGE2TS based video streams with a normalized MOS- AV schedule. Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies © Spirent Communications 2011 92 Appendix D – Storage Queueput Standard The following is the definition of Queueput: The maximum Offered Load than can be transmitted into a DUT such that every transmitted frame matches a specific classification rule. The DUT does NOT use priority-based flow control mechanisms to manage the ingress traffic rate of the classifications of interest, and all ingress frames are forwarded to the correct egress port. A DUT may have a different Queueput value for each configured classification. This definition is based on a draft RFC located at http://tools.ietf.org/pdf/draft-player-dcb-benchmarking- 03.pdf.

Access Edge

Network Equipment

Cloud/Data Center

Core

Mobility

High Speed Ethernet

Automation

Monitoring

Security

Performance and Acceptance

Location Technologies

Research and Development

Applications

Test Environments

Technologies

Products

Services

Spirent Journal of Cloud Infrastructure LAN/SAN Fabric & Virtual Server Access PASS Test Methodologies

About Spirent

Media Center

Partnerships & Alliances

Contacts

Connect