Spirent Journal of Cloud Application & Security Services PASS Test Methodologies
A cloud service has three distinct characteristics that differentiate it from traditional hosting. It is sold on demand, typically by the minute or the hour. It is elastic—a user can have as much or as little of a service as they want at any given time. The service is fully managed by the provider. The consumer needs nothing but a personal computer and Internet access. Significant innovations in virtualization and distributed computing, improved access to high-speed Internet, and financial pressure have accelerated interest in cloud computing. This journal presents a comprehensive set of test cases to measure PASS of the Device Under Test (DUT).
PASS
Spirent Journal of Cloud
Application and Security
Services PASS Test
Methodologies
February 2011 Edition
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
1
Introduction
Today’s Devices Under Test (DUT) represent complex, multi-protocol network elements with an emphasis
on Quality of Service (QoS) and Quality of Experience (QoE) that scale to terabits of bandwidth across the
switch fabric. The Spirent Catalogue of Test Methodologies represents an element of the Spirent test
ecosystem that helps answer the most critical Performance, Availability, Security and Scale Tests (PASS)
test cases. The Spirent Test ecosystem and Spirent Catalogue of Test Methodologies are intended to help
development engineers and product verification engineers to rapidly develop and test complex test
scenarios.
How to use this Journal
This provides test engineers with a battery of test cases for the Spirent Test Ecosystem. The journal is
divided into sections by technology. Each test case has a unique Test Case ID (Ex. TC_MBH_001) that is
universally unique across the ecosystem.
Tester Requirements
To determine the true capabilities and limitations of a DUT, the tests in this journal require a test tool that
can measure router performance under realistic Internet conditions. It must be able to simultaneously
generate wire-speed traffic, emulate the requisite protocols, and make real-time comparative
performance measurements. High port density for cost-effective performance and stress testing is
important to fully load switching fabrics and determine device and network scalability limits.
In addition to these features, some tests require more advanced capabilities, such as
Integrated traffic, routing, and MPLS protocols (e.g., BGP, OSPF, IS-IS, RSVP-TE, LDP/CR-LDP) to
advertise route topologies for large simulated networks with LSP tunnels while simultaneously
sending traffic over those tunnels. Further, the tester should emulate the interrelationships
between protocol s through a topology.
Emulation of service protocols (e.g., IGMPv3, PIM-SM, MP-iBGP) with diminution.
Correct single-pass testing with measurement of 41+ metrics per pass of a packet.
Tunneling protocol emulation (L2TP) and protocol stacking.
True stateful layer 2-7 traffic.
Ability to over-subscribe traffic dynamically and observe the effects.
Finally, the tester should provide conformance test suites for ensuring protocol conformance and
interoperability, and automated applications for rapidly executing the test cases in this journal.
Further Resources
Additional resources are available on our website at http://www.spirent.com
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
2
Table of Contents
Testing Cloud Application and Security Services ......................................................................3
CAS_001 Determine IP stateful traffic throughput per RFC 3511 ....................................... 4
CAS_002 Stateful traffic concurrent TCP connection capacity per RFC 3511 ..................... 7
CAS_003 Determine the maximum TCP connection establishment rate ......................... 11
CAS_004 Determine the maximum TCP connection tear down rate ................................ 15
CAS_005 Measure the BBT external to external basic throughput ................................... 19
CAS_006 Measure the BBT external to internal (virtual) max throughput ...................... 22
CAS_007 Measure the BBT internal (virtual) to internal (virtual) maximum throughput. 25
CAS_008 Measure performance of HTTP over VPLS over BGP in the cloud ...................... 28
CAS_009 BBT – External to internal (virtual): scalability and application availability ..... 31
CAS_010 BBT – Internal (virtual) to internal (virtual): scalability and application
availability in a virtual environment .................................................................. 34
CAS_011 BBT – Internal (virtual) to internal (virtual): security – attacking the virtual
environment ...................................................................................................... 37
CAS_012 BBT – External to internal (virtual): security – combined traffic and attacks ... 40
CAS_013 Determine the effects of a denial of service attack on firewall performance per
RFC 3511 Section 5.5 .......................................................................................... 43
CAS_014 Determine HTTP transfer rate of the firewall performance per RFC3511 Section
5.6 ...................................................................................................................... 48
CAS_015 Measure HTTP transaction rate of the firewall performance per RFC3511
Section 5.7.......................................................................................................... 52
CAS_016 Determine the capacity of NAT444 in the cloud ................................................ 56
CAS_017 Evaluate DNS A and AAAA record performance under attack .......................... 60
Appendix A – Telecommunications Definitions ..................................................................... 63
Appendix B – Stateful Playlist by QoS ................................................................................... 70
Appendix C – MPEG 2/4 Video QoE ....................................................................................... 71
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
3
Testing Cloud Application and Security
Services
Cloud computing is a general term for anything that involves delivering hosted services over the Internet.
These services are broadly divided into three categories: Infrastructure-as-a-Service (IaaS), Platform-as-a-
Service (PaaS) and Software-as-a-Service (SaaS).
A cloud service has three distinct characteristics that differentiate it from traditional hosting. It is sold on
demand, typically by the minute or the hour. It is elastic—a user can have as much or as little of a service
as they want at any given time. The service is fully managed by the provider. The consumer needs nothing
but a personal computer and Internet access. Significant innovations in virtualization and distributed
computing, improved access to high-speed Internet, and financial pressure have accelerated interest in
cloud computing.
IaaS provides virtual server instances with unique IP addresses and blocks of storage on demand.
Customers use the provider's application program interface (API) to start, stop, access and configure
virtual servers and storage. In the enterprise, cloud computing allows a company to pay for only as much
capacity as is needed and bring more online as soon as required. Because this pay-for-what-you-use
model resembles the way electricity, fuel and water are consumed, it's sometimes referred to as utility
computing.
PaaS in the cloud is defined as a set of software and product development tools hosted on the provider's
infrastructure. Developers create applications on the provider's platform over the Internet. PaaS
providers may use APIs, website portals or gateway software installed on the customer's computer.
Currently there are no standards for interoperability or data portability in the cloud. Some providers will
not allow software created by their customers to be moved off the provider's platform.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
4
CAS_001 Determine IP stateful traffic throughput per RFC 3511
Abstract
This test determines the throughput of network-layer data traversing the firewall (DUT), as
defined in RFC 1242. This test determines the maximum throughput for the DUT per fixed packet
size. Suggested packet sizes are 64, 128, 256, 512, 768, 1024, 1218, 1514. Packets should be UDP
packets. Each test should run for at least 10 minutes. References: RFC 3511, RFC 2889, RFC 1242.
For product verification and engineering.
Description
The tester offers unicast IP packets to the DUT at a constant rate. The test may consist of either
bi-directional or unidirectional traffic. For example, an emulated client may offer a unicast
stream of packets to an emulated server, or the tester may simulate a client/server exchange by
offering bidirectional traffic.
This test employs an iterative search algorithm. For iteration the tester varies the intended load
until the maximum rate, at which no packet loss occurs, is found. Since backpressure
mechanisms may be employed, resulting in a difference between the intended load and offered
load, the test should be performed in either a packet-based or time-based manner. As with RFC
1242, the term packet is used in place of frame.
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely
affect overall network performance. Testing the effect on network performance ensures that
the increased security does not decrease performance beyond the levels acceptable for the
business.
Version
1.0
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
5
Test Category
Testing Stateful Multiplay Benchmark Standards.
PASS
[x] Performance [ ] Availability [ ] Security [ ] Scale
Required Tester Capabilities
The test equipment must have the ability to generate simulated users. Each simulated user must
have the ability to schedule full web pages with the ability to schedule specific HTTP objects
within TCP tunnels in a predictable order, matched with a predictable source IP and MAC address
source.
Topology
Test Procedure
1. Reserve two test ports.
2. Connect tester port 1 as the client side to the configured firewall-capable DUT. Connect port
2 as the server side to the other side of the DUT. Establish the link.
3. Begin Step 1 – Configure the bandwidth performance test.
a. Enter the management IP address of the equipment that is used for this test for the
client port.
b. Enter the management IP address of the equipment that is used for this test for the
server port.
c. Select the performance test for specific equipment that is used in this test.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET Per SimUser.
iii. Maximum connections per server 1.
iv. Maximum transaction per connection 10 or 50.
v. User think time 30 seconds.
b. Server side:
i. HTTP version 1.1.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
6
ii. Maximum requests per connection 64 or 10 (if client is set to more than 10).
iii. Server type Irrelevant (only changes HTTP headers).
iv. Packet size 64b, 128b, 256b, 512b, 768b, 1024b, 1218b, 1514b.
5. End.
Variables & Relevance
Variable Relevance
Packet sizes – 64b, 128b, 256b,
512b, 768b, 1024b, 1218b, 1514b
These packet sizes are used to determine the maximum
throughput for the firewall DUT per fixed packet size
Test Duration Every test duration runs for at least 10 minutes
Desired Result
For throughput, maximum offered load, expressed in either bits per second or packets per
second, at which no packet loss is detected. For forwarding rate, expressed in either bits per
second or packet per second, the device is observed to successfully forward to the correct
destination interface in response to a specified offered load.
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput
Pass/Fail Status Show successful/unsuccessful transactions
Total packets offered Show total packets offered
Total packets forwarded Show total packets forwarded
Intended load Show attempted transactions
Offered load (if applicable) N/A
Forwarding rate Show forwarding rate
Analysis
The user should not see lost packets. The user should see the maximum throughput when
running the test for each different packet size. When testing with different packet sizes, the user
should see different throughput measurements and the firewall DUT configuration must remain
the same.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
7
CAS_002 Stateful traffic concurrent TCP connection capacity per
RFC 3511
Abstract
This test determines the maximum number of concurrent TCP connections supported through or
with the firewall (DUT). This test also finds the maximum number of entries the DUT can store in
its connection table. For this test, HTTP 1.1 must be used on both the client and server side. In
the case of a proxy-based firewall, the clients must request at least two objects – one to measure
the TCP establishment time (Time to SYN/ACK and Time to First Byte), the subsequent objects to
evaluate the quality of the connection (Average URL Response Time). References: RFC 3511. For
product verification and engineering.
Description
This test employs an iterative search algorithm to determine the maximum number of
concurrent TCP connections supported through or with the DUT. For each iteration, the
aggregate number of concurrent TCP connections attempted by the virtual clients is varied. The
destination address is that of the server or that of the NAT proxy. The aggregate rate is defined
by the connection attempt rate, and is attempted in a round-robin fashion.
To validate all connections, the virtual clients request an object using an HTTP 1.1 or higher GET
request. The requests are initiated on each connection after all of the TCP connections have been
established.
When testing proxy-based DUTs, the virtual clients request two objects using HTTP 1.1 or higher
GET requests. The first GET request is required for connection time establishment measurements
as specified. The second request is used for validation as previously mentioned. When comparing
proxy and non-proxy based DUTs, the test is performed in the same manner. Between each
iteration, it is recommended that the tester issue a TCP RST referencing each connection
attempted for the previous iteration, regardless of whether or not the connection attempt was
successful. The tester waits for aging time before continuing to the next iteration.
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
8
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely affect
overall network performance. Testing the effect on network performance ensures that the
increased security does not decrease performance beyond the levels acceptable for the business.
Version
1.0
Test Category
Testing Stateful Multiplay Benchmark Standards.
PASS
[ ] Performance [ ] Availability [ ] Security [x] Scale
Required Tester Capabilities
The test equipment must have the ability to generate simulated users. Each simulated user must
have the ability to schedule full web pages with the ability to schedule specific HTTP objects
within TCP tunnels in a predictable order, matched with a predictable source IP and MAC address
source.
Topology
Test Procedure
1. Reserve two test ports.
2. Connect tester port 1 as the client side to the configured firewall capable DUT. Connect port
2 as the server side to the other side of DUT. Establish the link.
3. Begin Step 1 – Generate the open connections performance test.
a. Enter the management IP address of the equipment that is used for this test for the
client port.
b. Enter the management IP address of the equipment that is used for this test for the
server port.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
9
c. Select the performance test for the specific equipment that is used in this test.
d. The goal is to open a connection, perform a GET, wait 30 seconds (the connection will
remain open unless the DUT has a TCP Timeout lower than this – but typically devices
have at minimum 90 seconds TCP Timeout), perform another GET, wait 30 seconds, and
so on until the ten GETs are complete.
e. If the maximum transactions per connection is set to 10, the connection is closed by a
FIN/ACK. If the maximum transactions per connection is greater than the number of
GETs in the action List, the connections is closed by an RST.
f. If the server-side maximum requests per connection is equal to the number of GETs in
the client action list, and the client profile is set to a maximum request per connection
higher than the number of GETs in the client action list, the connection termination is
initiated server-side.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET Per SimUser is 10.
iii. Maximum connection per server is 1.
iv. Maximum transaction per connection is either 10 or 50.
v. User think time is 30 seconds.
b. Server side:
i. HTTP version 1.1.
ii. Maximum requests per connection are either 64 or 10 (if client is set to more than
10).
iii. Server type is Irrelevant (only changes HTTP headers).
iv. Object size is 64b, 512b, 1,024b, 10,240b.
5. End.
Variables & Relevance
Variable Relevance
Packet sizes – 64, 512, 1024,
10240
These packet sizes are used to determine the maximum
throughput for the firewall DUT per fixed packet size
Test Duration Every test duration runs for at least 10 minutes
Desired Result
For maximum concurrent connections, the total number of TCP connections open for the last
successful iteration performance in the search algorithm.
For minimum connection establishment time, the lowest TCP connection establishment time
measured.
For maximum connection establishment time, the highest TCP connection establishment time
measured.
For average connection establishment time, the mean of all measurements of connections
establishment times.
For aggregate connection establishment time, the total of all measurements of connections
establishment time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
10
Key Measured Metrics
Statistic Relevance
Object size Show size of object for each test iteration
Number of completed requests Show number of completed transactions (client side)
Number of completed responses Show number of completed transactions (server side)
Analysis
The user should not see lost packets. The user should see the maximum concurrent connections,
minimum connection establishment time, maximum connection establishment time, average
connection establishment time, aggregate connection establishment time, number of objects
requested, and number of objects returned for each of the test iteration that uses different
packet sizes. When testing with different packet sizes, the user should see different throughput
measurements and the firewall DUT configuration must remain the same.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
11
CAS_003 Determine the maximum TCP connection establishment
rate
Abstract
This test determines the maximum TCP connection establishment rate through or with the
firewall (DUT). This test also finds the maximum rate at which the DUT can update its connection
table. The test defines the aggregate number of TCP connections that must be established. HTTP
1.1 or higher must be used for this test for both clients and servers. The client and server must
use the same HTTP version. The test also defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an HTTP 1.1 or higher GET
request. References: RFC 3511, RFC 2647. For product verification and engineering.
Description
This test employs an iterative search algorithm to determine the maximum rate at which the DUT
can accept TCP connection requests. For each iteration, the aggregate rate at which TCP
connection requests are attempted by the virtual clients is varied. The destination address is that
of the server or that of the NAT proxy. The aggregate number of connections, defined by number
of connections, is attempted in a round-robin fashion.
The same application-layer object transfers required for validation and establishment time
measurements as described in the concurrent TCP connection capacity test is performed.
Between each iteration, it is recommended that the tester issue a TCP RST referencing each
connection attempted for the previous iteration, regardless of whether or not the connection
attempt was successful. The tester waits for aging time before continuing to the next iteration.
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely affect
overall network performance. Testing the effect on network performance ensures that the
increased security does not decrease performance beyond the levels acceptable for the business.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
12
Version
1.0
Test Category
Testing Stateful Multiplay Benchmark Standards.
PASS
[ ] Performance [ ] Availability [ ] Security [x] Scale
Required Tester Capabilities
Specific Spirent features that enable this test case and emphasize the Spirent differentiators and
branding.
Topology
Test Procedure
1. Reserve two test ports.
2. Connect tester port 1 as the client side to the configured firewall capable DUT. Connect port
2 as the server side to the other side of the DUT. Establish the link.
3. Begin Step 1 – Configure the connections per second performance test.
a. Enter the management IP address of the equipment that is used for this test for the
client port.
b. Enter the management IP address of the equipment that is used for this test for the
server port.
c. Select the performance test for the specific equipment that is used in this test.
d. Make sure the test is set up to use HTTP 1.1 for both client and server.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
13
e. To maximize the performance of the tester, this test sends 10 HTTP level 1 GET requests
from each SimUser.
f. Each TCP connection accepts one HTTP transaction as the maximum transaction,
therefore each of the SimUsers generates 10 TCP connections sequentially.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET per SimUser is 10.
iii. A maximum connection per server is 1.
iv. Maximum transaction per connection is 1.
v. User think time 30 seconds.
b. Server side:
i. HTTP version 1.1.
ii. A maximum request per connection is 1.
iii. Server type is Irrelevant (only changes HTTP headers).
iv. Object size is 64b, 512b, 1,024b, 10,240b.
5. End.
Variables & Relevance
Variable Relevance
Packet sizes – 64b, 512b, 1024b,
10240b
These packet sizes are used to determine the maximum
throughput for the firewall DUT per fixed packet size
Test Duration Every test duration runs for at least 10 minutes
Desired Result
For highest connection rate, in connections per second, the rate for which all connections
successfully opened in the search algorithm.
For minimum connection establishment time, the lowest TCP connection establishment time
measured.
For maximum connection establishment time, the highest TCP connection establishment time
measured.
For average connection establishment time, the mean of all measurements of connection
establishment times.
For aggregate connection establishment time, the total of all measurements of connection
establishment times.
Key Measured Metrics
Statistic Relevance
Number of object requested Show the total number requested objects
Number of object returned Show the total number of returned objects
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
14
Analysis
The user should not see lost packets. The user should see the maximum TCP connections rate,
minimum connection establishment time, maximum connection establishment time, average
connection establishment time, aggregate connection establishment time, number of object
requested, and number of object returned for each of the test iteration that uses different
packet sizes. When testing with different packet sizes, the user should see different throughput
measurements and the firewall DUT configuration must remain the same.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
15
CAS_004 Determine the maximum TCP connection tear down rate
Abstract
This test determines the maximum TCP connection teardown rate through or with the firewall
(DUT). This test defines the number of TCP connections that are attempted to be torn down. It
also defines a method for closing TCP connections. The test must be performed with either a
three-way or four-way handshake. In a four-way handshake, each side sends separate FIN and
ACK messages. In a three-way handshake, one side sends a combined FIN/ACK message upon
receipt of a FIN. The test also defines whether closing of connections is to be initiated from the
client or from the server. References: RFC 3511, RFC 2647. For product verification and
engineering.
Description
This test employs an iterative search algorithm to determine the maximum TCP connection tear
down rate supported by the DUT. The test iterates through different TCP connection tear down
rates with a fixed number of TCP connections.
In the case of proxy based DUTs, the DUT receives the ACK in response to issuing a FIN packet to
close its side of the TCP connection. For validation purposes, the virtual client or server,
whichever is applicable, may verify that the DUT received the final ACK by re-transmitting the
final ACK. A TCP RST should be received in response to the retransmitted ACK.
Between each iteration, it is RECOMMENDED that the virtual clients or servers, whichever is
applicable, issue a TCP RST referencing each connection which was attempted to be torn down,
regardless of whether or not the connection tear down attempt was successful. The test waits
for aging time before continuing to the next iteration.
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
16
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely affect
overall network performance. Testing the effect on network performance ensures that the
increased security does not decrease performance beyond the levels acceptable for the business.
Version
1.0
Test Category
Testing Stateful Multiplay Benchmark Standards.
PASS
[ ] Performance [ ] Availability [ ] Security [x] Scale
Required Tester Capabilities
The test equipment must have the ability to generate simulated users. Each simulated user must
have the ability to schedule full web pages with the ability to schedule specific HTTP objects
within TCP tunnels in a predictable order, matched with a predictable source IP and MAC address
source.
Topology
Test Procedure
1. Reserve two test ports.
2. Connect tester port 1 as the client side to the configured firewall-capable DUT. Connect port
2 as the server side to the other side of the DUT. Establish the link.
3. Begin Step 1 – Generate the connections per second performance test.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
17
a. Enter the management IP address of the equipment that is used for this test for the
client port.
b. Enter the management IP address of the equipment that is used for this test for the
server port.
c. Select the performance test for the specific equipment that is used in this test.
d. Make sure the test is set up to use HTTP 1.1 for both client and server.
e. To maximize the performance of the tester, this test sends 10 HTTP level 1 GET requests
from each SimUser.
f. Each TCP connection accepts one HTTP transaction as the maximum transaction,
therefore each of the SimUser will generate 10 TCP connections sequentially.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET per SimUser is 10.
iii. Maximum connections per server is 1.
iv. Maximum transaction per connection is 1.
v. User think time 30 seconds.
b. Server side:
i. HTTP version 1.1.
ii. Maximum requests per connection is 1.
iii. Server type is Irrelevant (only changes HTTP headers).
iv. Object size is 64b, 512b, 1,024b, 10,240b.
5. End.
Variables & Relevance
Variable Relevance
Packet sizes – 64b, 512b, 1024b,
10240b
These packet sizes are used to determine the maximum
throughput for the firewall DUT per fixed packet size
Test Duration Every test duration runs for at least 10 minutes
Desired Result
For highest connection tear down rate, the highest rate in connections per second, for which
all TCP connections were successfully torn down in the search algorithm.
For minimum connection tear down time, the lowest TCP connection tear down time
measured.
For maximum connection tear down, the highest TCP connection tear down time measured.
For average connection tear down time, the mean of all measurements of connection tear
down time.
For aggregate connection tear down time, the total of all measurements of connection tear
down time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
18
Key Measured Metrics
Statistic Relevance
Number of connections Show the total number of connections
Aging time Show the aging time
Close method Show 3-way or 4-way handshake
Close direction Show close of connections are initiated from either client
or server
Analysis
The user should not see lost packets. The user should see the maximum TCP connections tear
down rate, minimum connection tear down time, maximum connection tear down time, average
connection tear down time, aggregate connection tear down time, number of connections, aging
time, close method, and close direction for each of the test iteration which uses different packet
sizes. When testing with different packet sizes, the user should see different throughput
measurements and the firewall DUT configuration must remain the same.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
19
CAS_005 Measure the BBT external to external basic throughput
Abstract
This test is delivered by Broadband Testing (BBT) and is focused on securing the virtual data
center in a cloud computing environment. This test case determines a baseline performance
including the maximum realistic throughput of the DUT under sustained attack. For product
verification and engineering.
Description
This test determines maximum throughput. The DUT is connected to the tester via a 10 GbE
connection and also connected to the main test network via 1 Gbps connections to the virtual
elements of the test bed.
Relevance
Measures throughput of the external-to-external throughput of a DUT
Measures performance of virtualized applications
Version
1.0
Test Category
Testing Security and Cloud Services.
PASS
[x] Performance [ ] Availability [ ] Security [ ] Scale
Required Tester Capabilities
Ability to import the BBT test file
Support for server transactions profiles for different packet sizes
Real-time and summary reports on packet sizes, test duration, and throughput.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
20
Topology
Test Procedure
1. Reserve two test ports.
2. Connect tester port 1 as the client side to the DUT. Connect port 2 as the server side to the
other side of DUT. Establish the link.
3. Begin Step 1 – Import the test file (spf) from BBT
a. Select the Ext_to_Ext_Final_PlusAttacks_0001 test from SBFinal Project
b. Set the client side port to the network that is currently connected.
c. Set the server side port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET per SimUser.
iii. Maximum connections per server 2.
iv. Maximum requests per connection 50.
v. User thinks time 0 seconds.
vi. User-based load profile for http traffic (SimUsers/sec) and threats traffic
(SimUsers/sec).
b. Server side:
i. HTTP version 1.1.
ii. Maximum requests per connection 64.
iii. Server type irrelevant (only changes HTTP headers).
iv. Packet size 100KB.
5. Begin Step 3 – Run the test
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
21
Variables & Relevance
Variable Relevance
Packet sizes – 100KB This packet size is intended to determine the maximum
throughput for the DUT
Stateful Threats (Attack List) – 16
threats were used
There are 16 stateful threats that are used to determine
whether the DUT is able to block them and still sustain the
throughput.
Test Duration Each test duration runs for at least 12 minutes.
Desired Result
For throughput, the maximum offered load, at which no packet loss is detected.
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput
Pass/Fail Status Show successful/unsuccessful transactions
Intended load Show attempted transactions
Exploitation Results Show attack type, client success exploit ratio
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT
Analysis
The user should not see lost packets. The user should see the maximum throughput when
running the test using 100KB objects and while blocking threat traffic.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
22
CAS_006 Measure the BBT external to internal (virtual) max
throughput
Abstract
This test is delivered by Broadband Testing (BBT) and is focused on securing the virtual data
center in a cloud computing environment. This test case determines the performance from
external (physical) to internal (virtual) environments including the maximum realistic throughput
of the DUT. For product verification and engineering.
Description
This test determines the maximum throughput when generating traffic to move from the
physical to the virtual elements of the test bed.
Relevance
Determines the maximum virtual max throughput.
Version
1.0
Test Category
Testing Security and Cloud Services.
PASS
[ ] Performance [x] Availability [ ] Security [ ] Scale
Required Tester Capabilities
Ability to import the BBT test file
Support for server transactions profiles for different packet sizes
Real-time and summary reports on packet sizes, test duration, and throughput
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
23
Topology
Test Procedure
1. Reserve three test ports (1 client and 2 servers).
2. Connect the physical tester to the DUT. Run the virtual tester in a VM on the DUT. Establish
the link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Ext_to_Int_Bandwidth_1200b_Post test from SBFinal Project.
b. Set the client port to the network that is currently connected.
c. Set the server ports port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET and POST per SimUser.
iii. Maximum connections per server 2.
iv. Maximum requests per connection 50.
v. User think time 0 seconds.
vi. Global load profile for http traffic (SimUsers/sec).
b. Server side:
i. HTTP version 1.1.
ii. Maximum requests per connection 64.
iii. Server type irrelevant (only changes HTTP headers).
iv. Packet size 1200 bytes.
5. Begin Step 3 – Run the test.
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
24
Variables & Relevance
Variable Relevance
Packet sizes for GET – 500, 1024,
1200 bytes
This packet size is used to determine the maximum
throughput for the DUT.
Packet sizes for POST – 500, 1024,
1200 bytes
This packet size is used to determine the maximum
throughput for the DUT.
Desired Result
For throughput, the maximum offered load, at which no packet loss is detected. Sample results
below:
External To Internal Tests GET GET GET POST POST POST
Packet size (bytes) 500 1024 1200 500 1024 1200
Incoming (kbps) 280730 433723 451023 251535 304764 308076
Outgoing (kbps) 77205 69064 62883 294369 333936 333234
Transactions Per Second 49189 43779 39860 47901 32915 28370
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput
Pass/Fail Status Show successful/unsuccessful transactions
Intended load Show attempted transactions
Throughput Results
(Incoming and Outgoing Traffic)
Show throughput level of DUT
Analysis
In this test, examine the packet size for both GET and POST methods. Compare the measured
results to the desired results in the above table. Report the results as a table comparing the
differences.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
25
CAS_007 Measure the BBT internal (virtual) to internal (virtual)
maximum throughput
Abstract
This test is delivered by Broadband Testing (BBT) and is focused on securing the virtual data
center in a cloud computing environment. This test case determines the performance from
internal (virtual) to internal (virtual) environments including the maximum realistic throughput of
the DUT. For product verification and engineering.
Description
This test determines the maximum throughput when generating traffic to move from the internal
(virtual) to the virtual elements of test bed. This test determines whether the virtual controller in
the DUT affects performance, by generating HTTP traffic to achieve the maximum level of
throughput in the virtual environments.
Relevance
Measures the virtual to virtual maximum performance
Version
1.0
Test Category
Testing Security and Cloud Services.
PASS
[x] Performance [ ] Availability [ ] Security [ ] Scale
Required Tester Capabilities
Ability to import the BBT test file
Support for server transactions profiles for different packet sizes
Real-time and summary reports on packet sizes, test duration, and throughput
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
26
Topology
Test Procedure
1. Reserve three test ports (1 client and 2 servers).
2. Connect the physical tester to the DUT. Run the virtual tester in a VM on the DUT. Establish
the link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Int_to_Int test from SBFinal Project.
b. Set the client port to the network that is currently connected.
c. Set the server ports to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET per SimUser.
iii. Maximum connections per server 2.
iv. Maximum requests per connection 50.
v. User think time 0 seconds.
vi. Global load profile for http traffic (SimUsers/sec).
b. Server side:
i. HTTP version 1.1.
ii. Maximum requests per connection 64.
iii. Server type irrelevant (only changes HTTP headers).
iv. Packet size 100KB.
5. Begin Step 3 – Run the test.
a. Run with the virtual controller redirector in the DUT disabled. Traffic will pass straight-
through.
b. Run with the virtual controller redirector in the DUT enabled. The DUT will read every
packet as it passes through.
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
27
Variables & Relevance
Variable Relevance
Packet sizes for GET – 100KB, 500
bytes, 1024 bytes, and 1200 bytes
This packet size is used to determine the maximum
throughput for the DUT.
Test Duration Each test duration runs for at least 12 minutes.
Desired Result
For throughput, the maximum offered load, at which no packet loss is detected. Sample results
below:
External To Internal Tests GET GET GET
Packet size (bytes) 500 1024 1200
Incoming (kbps) 64404 111952 126393
Outgoing (kbps) 17831 17886 17715
Transactions Per Second 11348 11342 11205
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput
Pass/Fail Status Show successful/unsuccessful transactions
Intended load Show attempted transactions
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT
Transactions per second Show how many transactions per second are successfully
for different packet size
Analysis
The user should not see lost packets. The user should see the maximum throughput for each test
iteration. When testing with different packet sizes, the user should see different throughput
measurements and the DUT configuration must remain the same. The user should see
performance improve significantly with a higher performance server.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
28
CAS_008 Measure performance of HTTP over VPLS over BGP in the
cloud
Abstract
HTTP goodput is a critical core metric when analyzing a service provider core network. This test
allows the user to measure HTTP goodput ranges over a discrete number of VPLS tunnels.
Without understanding the range of goodput across the cloud, the service provider will not be
able to properly design and deploy the proper network scale of each device in the network.
References: RFC2616 Hyper Text Markup Languages. For functional and system performance
testing.
Description
HTTP goodput is the measure of delivered bandwidth to the HTTP stack with all network
impairments taken into consideration. Since traffic in the service provider cloud is tunneled and
backhauled through the cloud, impairments in the core can have a direct impact on the overall
user experience of web based traffic, end to end. Impairments such as network induced jitter,
excessive latency distribution, sequencing errors, and errors can all contribute to poor network
traffic.
Relevance
Understanding goodput ranges in a specific network environment is a key attribute of end user
experience.
Version
1.0
Test Category
Performance Test.
PASS
[x] Performance [ ] Availability [ ] Security [ ] Scale
Required Tester Capabilities
The test equipment should have the ability the ability to dynamically setup control plane routing
and BGP and VPLS. Over VPLS, the user should have the ability to make dynamic changes to HTTP
traffic. The tester should have the ability to coordinate and trend responses.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
29
Topology
Test Procedure
1. Reserve tester ports A and B. connect the ports to the DUT and bring up the links.
2. If P router emulation is selected, setup desired number of P routers per port.
a. Establish one EBGP peering session per P router to the DUT.
3. Setup the desired number of PE router per tester port or emulated P router.
a. Emulate 1 CE router per PE router.
b. Setup a host clouds behind each CE router.
4. Advertise though BGP all host cloud subnets.
5. Establish one LDP VPLS tunnel from each CE emulated port on tester A to each emulated CE
router on tester port B. Verify all tunnels establish.
6. On tester port B, set up HTTP Servers with the desired HTTP return object size. Start all HTTP
servers.
7. On the tester port A, setup one HTTP client per host. Each client should be connected to a
unique emulated HTTP server.
8. For each client, set the following load specifications:
a. Ramp from 0 kbps to (desired peak aggregate bandwidth / tester port A HTTP host
count) in 30 seconds.
b. Set steady state for (total test time – 100 seconds).
c. Ramp down to 0 kbps in 30 seconds.
9. Start all HTTP clients.
10. Measure aggregate and per host goodput.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
30
Variables & Relevance
Variable Relevance
Emulate P Routers? Ask user is they wish to Emulate P Routers or just PE
routers. Default=Yes
Number of P Routers Per Port How many P Routers to Emulate per Port. Default=5
PE Routers or PE Routers per
Port?
Defines how many PE Routers to emulate per tester port or
per P Router
Tester Port A Starting BGP AS Starting AS Number of Tester Port A
Tester Port A Starting IP Address Starting IP Address of the First P or PE Router
Tester Port A Netmask Classless Netmask of Tester Port A
Tester Port A Starting AS Starting AS of the P or PE Router
Tester Port A Starting BGP AS Starting AS Number of Tester Port A
Tester Port A DUT IP Address Tester Port B DUT IP Address
Tester Port B Starting IP Address Starting IP Address of the First P or PE Router
Tester Port B Netmask Classless Netmask of Tester Port A
Tester Port B Starting AS Starting AS of the P or PE Router
Tester Port B DUT IP Address Tester Port B DUT IP Address
Hosts per CE Cloud Number of Hosts per CE Cloud. Default=5
HTTP Object Size Object Size to Return. Default=384 bytes
Test Duration Duration of the test. Default=600 Seconds
Peak Aggregate HTTP Bandwidth Peak Aggregate HTTP Bandwidth. Default=150 Mbps
Desired Result
The DUT should be able to deliver the offered bandwidth. The variance in goodput per individual
host should be zero.
Key Measured Metrics
Statistic Relevance
Aggregate HTTP Goodput Total HTTP achieved globally
Per Host HTTP Goodput Per Host HTTP goodput
Analysis
To facilitate analysis, the following tables and charts should be created.
The offered and measured bandwidth and goodput should be displayed as a bar chart
comparison, there should also be a table presenting direct numbers and difference in kbps and
percentage.
The next chart should be offered bandwidth vs. aggregate goodput every 10 seconds.
There should be a chart showing per host goodput with the X-Axis representing VPLS tunnel and
associated transactions behind each tunnel, and the Y-Axis repressing measured goodput as a
percentage of offered bandwidth.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
31
CAS_009 BBT – External to internal (virtual): scalability and
application availability
Abstract
This test is delivered by Broadband Testing (BBT) and focuses on securing the virtual data center.
This test case determines application availability under load from external to internal (virtual)
environments and scalability of server performance.
Description
This test determines application availability when generating traffic to move from external to
virtual elements of the BBT test bed. This test also determines scalability of server performance
and response times while under high load.
Relevance
Performance (outright throughput and application availability/latency)
Scalability
Security Efficiency
Version
1.0
Test Category
Testing Security and Cloud Services
PASS
[x] Performance [x] Availability [x] Security [x] Scale
Required Tester Capabilities
Import BBT test file
Server transactions profile support for different packet sizes
Real-time and summary reports packet sizes, test duration, and server response time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
32
Topology
Test Procedure
1. Reserve three test ports (1 client and 2 servers)
2. Connect cabling to the DUT. Connect an Avalanche 3100 and an Avalanche Virtual (running
on a VM) as the physical and virtual components. Also connect to a management server
(which consists of an IPS appliance, software running on VMs under VMware, vController,
SMS management server, and management software). Establish link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Ext_to_Int_CC and Ext_to_Int_CPS tests from SBFinal Project.
b. From the client/port tab, change to the port to the network that is currently connected.
c. From the server/port tab, change to the port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.0 and HTTP 1.1 persistent.
ii. HTTP GET Per SimUser.
iii. Maximum Connections Per Server 2.
iv. Maximum Requests per Connection 50.
v. User Think Time 0 seconds.
vi. Global load profile for http traffic (SimUsers and cps).
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection 64.
iii. Server Type Irrelevant (only changes HTTP headers).
iv. Packet size 500, 1024, and 1200 bytes.
5. Begin Step 3 – Running the test.
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
33
Variables & Relevance
Variable Relevance
Packet sizes for GET – 500, 1024,
1200 bytes
This packet size determines the maximum throughput for
the DUT.
Packet sizes for POST – 500, 1024,
1200 bytes
This packet size determines the maximum throughput for
the DUT.
Test Duration Every test duration runs for at least 12 minutes
Desired Result
The result of this external-to-internal environment test demonstrates that the firewall is able to
scale with more powerful servers in sub milliseconds using different packet sizes for virtual
machines and while sustaining maximum throughput and application availability under load.
External To Internal Tests GET GET GET POST POST POST
Packet size (bytes) 500 1024 1200 500 1024 1200
Server Response Time (ms) 0.324 0.297 0.265 0.311 0.374 0.475
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput.
Pass/Fail Status Show successful/unsuccessful transactions.
Intended load Show attempted transactions.
Server Response Time (ms) Show server response time.
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT.
Analysis
The user should not see lost packets. The user should see the server response time reported in
sub milliseconds and the maximum throughput for each of the test iterations that uses different
packet sizes. When testing with different packet sizes the user should see different server
response times in sub milliseconds and throughput measurements and the DUT configuration
must remain constant. The user should see the performance improve significantly with more
powerful servers for the virtual end for both server response time and throughput
measurements.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
34
CAS_010 BBT – Internal (virtual) to internal (virtual): scalability
and application availability in a virtual environment
Abstract
This test is delivered by Broadband Testing (BBT) and focuses on securing the virtual data center.
This test case determines application availability under load from internal (virtual) to internal
(virtual) environments and scalability of server performance.
Description
This test determines application availability when generating traffic to move from internal to
virtual elements of the BBT test bed. This test also determines the scalability of server
performance and response time while under high load.
Relevance
Performance (outright throughput and application availability/latency)
Scalability
Security Efficiency
Version
1.0
Test Category
Testing Security and Cloud Services
PASS
[x] Performance [x] Availability [x] Security [x] Scale
Required Tester Capabilities
Import BBT test file
Server transactions profile support for different packet sizes
Real-time and summary reports packet sizes, test duration, and server response time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
35
Topology
Test Procedure
1. Reserve three test ports (1 client and 2 servers)
2. Connect cabling to the DUT. Connect Avalanche virtual (running on a VM) as the virtual
components to management server (which consists of IPS appliance, software running on
VMs under VMware, vController, SMS management server, and management software).
Establish link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Int_to_Int_Bandwidth_1200b_0001, Int_to_Int_Bandwidth_1k_0001, and
Int_to_Int_Bandwidth_500_0001 tests from SBFinal Project
b. From the client/port tab, change to the port to the network that is currently connected.
c. From the server/port tab, change to the port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.0 and HTTP 1.1 persistent.
ii. HTTP GET Per SimUser.
iii. Maximum Connections Per Server 2.
iv. Maximum Requests per Connection 50.
v. User Think Time 0 seconds.
vi. Global load profile for http traffic (SimUsers and cps).
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection 64.
iii. Server Type Irrelevant (only changes HTTP headers).
iv. Packet size 500, 1024, and 1200 bytes.
5. Begin Step 3 – Running the test.
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
36
Variables & Relevance
Variable Relevance
Packet sizes for GET – 500, 1024,
1200 bytes
This packet size determines the maximum throughput for
the DUT.
Test Duration Every test duration runs for at least 12 minutes
Desired Result
The result of this external-to-internal environment test demonstrates that the firewall is able to
scale with more powerful servers in sub milliseconds using different packet sizes for virtual
machines and while sustaining maximum throughput and application availability under load.
External To Internal Tests GET GET GET
Packet size (bytes) 500 1024 1200
Server Response Time (ms) 0.58 0.564 0.811
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput.
Pass/Fail Status Show successful/unsuccessful transactions.
Intended load Show attempted transactions.
Server Response Time (ms) Show server response time.
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT.
Analysis
The user should not see lost packets. The user should see the server response time reported in
sub milliseconds and the maximum throughput for each of the test iterations that uses different
packet sizes. When testing with different packet sizes the user should see different server
response times in sub milliseconds and throughput measurements and the DUT configuration
must remain constant. The user should see the performance improve significantly with more
powerful servers for the virtual end for both server response time and throughput
measurements.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
37
CAS_011 BBT – Internal (virtual) to internal (virtual): security –
attacking the virtual environment
Abstract
This test is delivered by Broadband Testing (BBT) and is focused on securing the virtual data
center. This test case determines performance, including the maximum realistic throughput in
the virtual environment, from internal (virtual) to internal (virtual) while under sustained attack.
Description
This test determines the maximum throughput within the virtual environment while under a
series of different attacks. This test uses only the internal (virtual) environment and is setup using
Avalanche virtual targeting vController directly with a combination of http traffic load generation
and injected threat traffic, first based on CodeRed II followed by several different stateful
attacks.
Relevance
Performance (outright throughput and application availability/latency)
Scalability
Security Efficiency
Version
1.0
Test Category
Testing Security and Cloud Services
PASS
[x] Performance [x] Availability [x] Security [x] Scale
Required Tester Capabilities
Import BBT test file
Server transactions profile support for different packet sizes
Real-time and summary reports packet sizes, test duration, and server response time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
38
Topology
Test Procedure
1. Reserve two test ports.
2. Connect cabling to the DUT. Connect vController and management server and Avalanche
virtual (running on a VM) as the virtual and virtual component (which consists of IPS
appliance, software running on VMs under VMware, vController, SMS management server,
and management software). Establish link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Int_to_Int_Final_PlusAttacks_10G test from SBFinal Project.
b. From the client/port tab, change to the port to the network that is currently connected.
c. From the server/port tab, change to the port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.0 and HTTP 1.1 persistent.
ii. HTTP GET Per SimUser.
iii. Maximum Connections Per Server 2.
iv. Maximum Requests per Connection 50.
v. User Think Time 0 seconds.
vi. User-based load profile for http traffic (SimUsers/sec) and threats traffic
(SimUsers/sec)
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection 64.
iii. Server Type Irrelevant (only changes HTTP headers).
iv. Packet size 100KB.
5. Begin Step 3 – Running the test.
a. Generate HTTP traffic over 90 seconds, then inject 1432 CodeRed II attacks.
b. Repeat the same test, generate HTTP traffic over 90 seconds, then inject multiple
stateful attacks.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
39
c. Repeat the same test, generate HTTP traffic over 10 minutes, then inject multiple
stateful attacks for 80 seconds into the test.
d. Repeat the same test, generate HTTP traffic over 10 minutes, then inject multiple
stateful attacks for 80 seconds into the test. On the IPS DUT, add a rule to the firewall
to block a batch of clients based on IP address.
6. End.
Variables & Relevance
Variable Relevance
Packet sizes – 100KB This packet size determines the maximum throughput for the
DUT
Stateful Threats (Attack List) Stateful threats
Test Duration Every test duration runs for at least 90 seconds or 10 minutes
Desired Result
The first test generates the maximum throughput of HTTP traffic over 90 seconds runs and
attacks are injected, the DUT in the virtual environment successfully blocks the bad traffic with
no reduction in overall maximum throughput.
The second test generates the maximum throughput of HTTP traffic for over 90 seconds and
multiple stateful attacks are injected which were contained within the signature database, the
DUT in the virtual environment successfully block all the bad traffic. Since the attacks attack
across a saturated link – VM to VM – the result of this test should show the vController’s ability
to successfully block with no reduction in overall maximum throughput.
The third test should generate the maximum throughput of HTTP traffic and the DUT should
block all bad traffic.
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput.
Pass/Fail Status Show successful/unsuccessful transactions.
Intended load Show attempted transactions.
Server Response Time (ms) Show server response time.
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT.
Analysis
The user should not see lost packets. The user should see the server response time reported in
sub milliseconds and the maximum throughput for each of the test iterations that uses different
packet sizes. When testing with different packet sizes the user should see different server
response times in sub milliseconds and throughput measurements and the DUT configuration
must remain constant. The user should see the performance improve significantly with more
powerful servers for the virtual end for both server response time and throughput
measurements
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
40
CAS_012 BBT – External to internal (virtual): security – combined
traffic and attacks
Abstract
This test is delivered by Broadband Testing (BBT) and is focused on securing the virtual data
center. This test case combines 10Gbps traffic from Avalanche 3100 (external) to the firewall and
1 Gbps traffic within the virtual environment on the vController to determine a baseline
performance including the maximum realistic throughput of the DUT under sustained attack.
Description
This test determines the maximum throughput from an external to a virtual environment. The
DUT is connected to a Spirent 3100 via a 10GbE connection and is also connected to the main
test network via 1 Gbps connections to the virtual elements of the test bed with multiple stateful
threats on the vController. The test bed is designed to run at 1 GbE and 10GbE (in the physical
space) test options to emulate a physical or virtual data center environment.
Relevance
Performance (outright throughput and application availability/latency)
Scalability
Security Efficiency
Version
1.0
Test Category
Testing Security and Cloud Services
PASS
[x] Performance [x] Availability [x] Security [x] Scale
Required Tester Capabilities
Import BBT test file
Server transactions profile support for different packet sizes
Real-time and summary reports packet sizes, test duration, and server response time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
41
Topology
Test Procedure
1. Reserve two test ports.
2. Connect cabling to the DUT. Connect Avalanche 3100 10GigE directly to the Firewall DUT and
Avalanche virtual 1GigE (running on a VM) as the physical and virtual components, also
connect to management server (which consists of IPS appliance, software running on VMs
under VMware, vController, SMS management server, and management software). Establish
link.
3. Begin Step 1 – Import the test file (spf) from BBT.
a. Select the Threats_test test from SBFinal Project.
b. From the client/port tab, change to the port to the network that is currently connected.
c. From the server/port tab, change to the port to the network that is currently connected.
4. Begin Step 2 – Test the effect of load, packet size, and duration.
a. Client side:
i. HTTP version 1.1
ii. HTTP GET Per SimUser.
iii. Maximum Connections Per Server 2.
iv. Maximum Requests per Connection 50.
v. User Think Time 0 seconds.
vi. User-based load profile for http traffic (SimUsers/sec) and threats traffic
(SimUsers/sec)
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection 64.
iii. Server Type Irrelevant (only changes HTTP headers).
iv. Packet size 1024 bytes.
5. Begin Step 3 – Running the test.
a. Generate HTTP traffic between Avalanche 3100 10G to IPS DUT then inject multiple
stateful attacks directly at the vController.
6. End.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
42
Variables & Relevance
Variable Relevance
Packet sizes – 1024 bytes This packet size determines the maximum throughput for the
DUT
Stateful Threats (Attack List) –
16 threats
There are 16 stateful threats to determine whether the DUT
is able to block them and still sustain the throughput
Test Duration Every test duration runs for at least 10 minutes
Desired Result
For throughput, maximum offered load, at which no packet loss is detected.
The external and virtual environment generates around 4.5 Gbps of maximum throughput
between the Avalanche 3100 10 GigE and the firewall DUT and multiple stateful threats are
injected at the vController. The vController should be able to sustain maximum throughput while
blocking threat traffic in addition to http traffic sending 1024 bytes of GET requests.
Key Measured Metrics
Statistic Relevance
Step Iteration Show how many steps are added in the load profile to
achieve maximum throughput.
Pass/Fail Status Show successful/unsuccessful transactions.
Intended load Show attempted transactions.
Server Response Time (ms) Show server response time.
Throughput Results (Incoming and
Outgoing Traffic)
Show throughput level of DUT.
Analysis
The user should not see lost packets. The user should see the server response time reported in
sub milliseconds and the maximum throughput for each of the test iterations that uses different
packet sizes. When testing with different packet sizes the user should see different server
response times in sub milliseconds and throughput measurements and the DUT configuration
must remain constant. The user should see the performance improve significantly with more
powerful servers for the virtual end for both server response time and throughput
measurements.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
43
CAS_013 Determine the effects of a denial of service attack on firewall
performance per RFC 3511 Section 5.5
Abstract
This test determines the effect of a denial of service attack on a firewall (DUT) TCP connection
establishment and/or HTTP transfer rate. The denial of service handling test must be run after
obtaining baseline measurements. The TCP SYN flood attack exploits TCP’s three-way handshake
mechanism by having an attacking source host generate TCP SYN packets with random source
addresses towards a victim host, thereby consuming that host’s resources.
Without determining the performance of the firewall under attack, the user will not understand
the corner case traffic conditions of the DUT.
Description
Use the same procedure as defined in the Maximum TCP Connection Establishment Rate test
case depending on whether testing against the baseline TCP connection establishment rate. In
addition, the test instrument generates TCP SYN packets targeting the server‘s IP address or NAT
proxy address at a rate defined by the SYN attack rate.
The test instrument originating the TCP SYN attack must be attached to the unprotected
network. In addition, the test instrument must not respond to the SYN/ACK packets sent by the
target server or NAT proxy in response to the SYN packet.
Target Users
Product verification, Engineering
Target Device Under Test (DUT)
Firewall
Reference
RFC 3511
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
44
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely
affect overall network performance. Testing the effect on network performance ensures that
the increased security does not decrease performance beyond the levels acceptable for the
business.
Version
1.0
Test Category
Testing Firewalls and VPN
PASS
[X] Performance [X] Availability [X] Security [X] Scale
Required Tester Capabilities
Generate performance test for connections per second.
Load profiles for different step iterations and test duration.
Server transactions profile support for different object sizes.
Real-time and summary reports on highest connections tear down rate, minimum connection
tear down time, maximum connection tear down time, average connection tear down time,
aggregate connection tear down time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
45
Topology Diagram
Test Procedure
1. Reserve two test ports.
2. Connect cabling to the DUT. Cable tester port 1 as the client side to the configured firewall-
capable DUT. Connect port 2 as the server side to the other side of DUT. Establish the link.
3. Begin Step 1 – Generate the connections per second performance test.
a. Enter the management IP address of the client port.
b. Enter the management IP address of the server port.
c. Select the performance test for the specific equipment used in this test.
d. Configure HTTP 1.1 for both client and server.
e. To maximize the performance of the tester, send 10 HTTP level 1 GET requests from
each SimUser.
f. Each TCP connection accepts one HTTP transaction as maximum transaction, therefore
each of the SimUsers generate 10 TCP connections sequentially.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET Per SimUser is 10.
iii. Maximum Connections Per Server is 1.
iv. Maximum Transaction per Connection is 1.
v. User Think Time 30 seconds.
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection is 1.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
46
iii. Server Type is Irrelevant (only changes HTTP headers).
iv. Object size is 64b, 512b, 1024b, 10240b.
5. Begin Step 3 – Run test.
a. Generate HTTP traffic over 90 seconds then inject CodeRedII attacks.
b. Repeat the same test, generating HTTP traffic over 90 seconds then inject multiple
stateful attacks.
c. Repeat the same test, generating HTTP traffic over 10 minutes then inject multiple
stateful attacks for 80 seconds into the test.
d. Repeat the same test, generating HTTP traffic over 10 minutes then inject multiple
stateful attacks for 80 seconds into the test. On the IPS DUT, add a rule to the firewall to
block a batch of clients based on IP address.
6. End test.
Control Variables & Relevance
Variable Relevance Default Value
Packet sizes – 64b,
512b, 1024b, 10240b
Determine the maximum throughput for the firewall
DUT per fixed packet size.
64 bytes
Test Duration At least 10 minutes. 10 minutes
Key Measured Metrics
Matric Relevance Metric Unit
Step Iteration Indicates how many steps are added in the load profile
to achieve maximum throughput.
Pass/Fail Status Indicates successful/unsuccessful transactions.
Intended load Indicates attempted transactions.
Exploitation Results Indicates attack type, client success exploit ratio.
Throughput Results
(Incoming and
Outgoing Traffic)
Indicates throughput level of DUT.
Desired Result
In the first test the DUT in the virtual environment should successfully block the bad traffic with
no reduction in overall maximum throughput.
In the second test, the DUT should successfully block all the bad traffic. Since the attacks
attacking across a saturated link, this test should show the ability of DUT to successfully block
with no reduction in overall maximum throughput.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
47
In the third test, the DUT should able to block all bad traffic.
Exploitation Results Attack Success/Failure @ Client/Server
Attack Name Attack Type Client Success Client Failure Server Success
antvilleXSS.xml Stateful
coderedII.xml Stateful
core_mailslot_dos.xml Stateful
ms05-051.xml Stateful
WebDAV_rs.xml Stateful
Analysis
No packets should be lost. Maximum throughput should be achieved when running the test using
512Kb, 1024Kb, 1Mb objects and while blocking threat traffic.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
48
CAS_014 Determine HTTP transfer rate of the firewall performance per
RFC3511 Section 5.6
Abstract
This test determines the transfer rate of HTTP requested objects traversing the firewall. It defines
the aggregate number of connections attempted. The number should be a multiple of the
number of virtual clients participating in the test. It also defines the method of closing TCP
connections. This test must be performed with either a three-way or four-way handshake. In a
four-way handshake, each side sends separate FIN and ACK messages. In a three-way handshake,
one side sends a combined FIN/ACK message upon receipt of a FIN. Finally, it defines whether
closing of connections is to be initiated from the client or from the server.
With the test, the user can determine the peak HTTP rate of the DUT.
Description
Each HTTP 1.1 or higher virtual client requests one or more objects from an HTTP 1.1 or higher
server using one or more HTTP GET requests over each connection. The aggregate number of
connections attempted, defined by number of connections, must be evenly divided among all the
participating virtual clients.
If the virtual clients make multiple HTTP GET requests per connection, it must request the same
object size for each GET request. Multiple iterations of this test may be run with objects of
different sizes.
Target Users
Product verification, Engineering
Target Device Under Test (DUT)
Firewall
Reference
RFC 3511
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
49
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely
affect overall network performance. Testing the effect on network performance ensures that
the increased security does not decrease performance beyond the levels acceptable for the
business.
Version
1.0
Test Category
Testing Firewalls and VPN
PASS
[X] Performance [X] Availability [X] Security [X] Scale
Required Tester Capabilities
Generate performance test for connections per second.
Load profiles for different step iterations and test duration.
Server transactions profile support for different object sizes.
Real-time and summary reports on highest connections tear down rate, minimum connection
tear down time, maximum connection tear down time, average connection tear down time,
aggregate connection tear down time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
50
Topology Diagram
Test Procedure
1. Reserve two test ports.
2. Connect cabling to the DUT. Cable tester port 1 as the client side to the configured firewall-
capable DUT. Connect port 2 as the server side to the other side of DUT. Establish the link.
3. Begin Step 1 – Generate the HTTP 1.1 transactions per second performance test:
a. Enter the management IP address of the client port.
b. Enter the management IP address of the server port.
c. Select the performance test for the specific equipment used in this test.
d. Configure HTTP 1.1 for both client and server.
e. To maximize the performance of the tester, send 10 HTTP level 1 GET requests from
each SimUser.
f. Each TCP connection accepts one HTTP transaction as maximum transaction, therefore
each of the SimUsers generate 1 TCP connection sequentially.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET Per SimUser is 50.
iii. Maximum Connections Per Server is 1.
iv. Maximum Transaction per Connection is 50.
v. User Think Time 0 seconds.
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection is 10.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
51
iii. Connection Termination with FIN.
iv. Object/Page size is 100Kb, 500Kb, 1Mb.
5. Load Reference.
a. Load (Transactions/second) 1150 with 100Kb object.
b. Load (Transactions/second) 230 with 500Kb object.
c. Load (Transactions/second) 110 with 1Mb object.
6. End test.
Control Variables & Relevance
Variable Relevance Default Value
Packet sizes – 100Kb,
500Kb, 1Mb
Determine the maximum throughput for the firewall
DUT per fixed packet size.
980 Mbps
throughput
Test Duration At least 10 minutes. 10 minutes
Key Measured Metrics
Matric Relevance Metric Unit
Number of bytes
transferred
Total payload bytes transferred
Number of Timeouts Total number of timeout events
Retransmitted bytes Total number of retransmitted bytes
Desired Result
The transfer rate results are reported for each of the object sizes tested, including the number
and percentage of completed requests, number and percentage of completed responses, and the
resultant transfer rate for each iteration of the test. The transfer rate results also include the
total bytes transferred, total timeouts, total retransmitted bytes and resultant goodput.
Analysis
There should be no packet loss. Metrics of interest include maximum TCP connections tear down
rate, minimum connection tear down time, maximum connection tear down time, average
connection tear down time, aggregate connection tear down time, number of connections, aging
time, close method, and close direction. Different packet sizes should produce different
throughput measurements with the same firewall DUT configuration.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
52
CAS_015 Measure HTTP transaction rate of the firewall performance
per RFC3511 Section 5.7
Abstract
This test determines the maximum transaction rate the firewall can sustain. It defines method for
closing TCP connections. The test must be performed with either a three-way or four-way
handshake. In a four-way handshake, each side sends separate FIN and ACK messages. In a three-
way handshake, one side sends a combined FIN/ACK message upon receipt of a FIN. It defines
whether closing of connections is to be initiated from the client or from the server.
This test ensures the users understand the upper transaction rate limit of the firewall to prevent
anomalous dropped packets and sessions.
Description
For each iteration of the test, HTTP 1.1 or higher simulated clients vary the aggregate GET
request rate offered to HTTP 1.1 or higher servers. The simulated clients maintain the offered
request rate for the defined test duration.
If a simulated client makes multiple HTTP GET requests per connection, it must request the same
object size for each GET request. Multiple tests may be performed with different object sizes.
Target Users
Product verification, Engineering
Target Device Under Test (DUT)
Firewall
Reference
RFC 3511
Relevance
For the networks that a firewall secures, often the value of the data and systems that they must
help protect cost many times that of the firewall and the testing. Deploying a security
infrastructure without understanding its performance and security runs the risk of doing little to
protect the network, especially if it also introduces apathy from a false sense of security. Among
the reasons that show the criticality of testing firewalls:
Security at load: Often, security flaws do not appear until the network encounters a large load.
Attacks can hide more easily within large amounts of traffic, potentially causing problems right
when network downtime is most harmful.
Firewall behavior at load and at failure: Firewalls often exhibit different behaviors as they
encounter increasing loads. Eventually, with enough traffic, the firewall will fail, providing
valuable insights. First, the conditions and behavior leading up to failure are now known, giving
the security administrator things to look for in production and providing a useful pre-failure
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
53
warning. Second, the failure state of the firewall is known—firewalls that fail closed will stop
all traffic from passing (essentially a successful denial of service, or DoS, attack), and firewalls
that fail open permit all traffic, which is a security failure.
Pre-deployment capacity planning: Deployment of a security infrastructure will most likely
affect overall network performance. Testing the effect on network performance ensures that
the increased security does not decrease performance beyond the levels acceptable for the
business.
Version
1.0
Test Category
Testing Firewalls and VPN
PASS
[X] Performance [X] Availability [X] Security [X] Scale
Required Tester Capabilities
Generate performance test for connections per second.
Load profiles for different step iterations and test duration.
Server transactions profile support for different object sizes.
Real-time and summary reports on highest connections tear down rate, minimum connection
tear down time, maximum connection tear down time, average connection tear down time,
aggregate connection tear down time.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
54
Topology Diagram
Test Procedure
1. Reserve two test ports.
2. Connect cabling to the DUT. Cable tester port 1 as the client side to the configured firewall-
capable DUT. Connect port 2 as the server side to the other side of DUT. Establish the link.
3. Begin Step 1 – Generate the HTTP 1.1 transactions per second performance test:
a. Enter the management IP address of the client port.
b. Enter the management IP address of the server port.
c. Select the performance test for the specific equipment used in this test.
d. Configure HTTP 1.1 for both client and server.
e. To maximize the performance of the tester, send 10 HTTP level 1 GET requests from
each SimUser.
f. Each TCP connection accepts one HTTP transaction as maximum transaction, therefore
each of the SimUsers generate 10 TCP connections sequentially.
4. Begin Step 2 – Test the effect of load, object size, and duration.
a. Client side:
i. HTTP version 1.1.
ii. HTTP GET Per SimUser is 50.
iii. Maximum Connections Per Server is 1.
iv. Maximum Transaction per Connection is 50.
v. User Think Time 0 seconds.
b. Server side:
i. HTTP version 1.1.
ii. Maximum Requests per Connection is 10.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
55
iii. Connection Termination with FIN.
iv. Object/Page size is 64b, 512b, 1024b, 10240b.
5. End test.
Control Variables & Relevance
Variable Relevance Default Value
Packet sizes – 64b,
512b, 1024b, 10240b
Determine the maximum throughput for the firewall
DUT per fixed packet size.
980 Mbps
throughput
Test Duration At least 10 minutes. 10 minutes
Key Measured Metrics
Matric Relevance Metric Unit
Maximum
transaction rate
Maximum rate for all transactions, that is all
requests/responses are completed
tps
Max transaction
time
Transaction time starts when client issues GET request
and end when client receives the last bit of request object
ms
Min transaction
time
Transaction time starts when client issues GET request
and end when client receives the last bit of request object
ms
Average
transaction time
Transaction time starts when client issues GET request
and end when client receives the last bit of request object
ms
Desired Result
The test results include GET request attempt rate, number of requested attempted, number and
percentage of requests completed, number of responses attempted, number and percentage of
responses completed minimum transaction time, average transaction time, and maximum
transaction time.
The test results also include the number of connections attempted, number and percentage of
connections completed, number and percentage of connection teardowns completed.
Analysis
There should be no packet loss. Metrics of interest include maximum TCP connections tear down
rate, minimum connection tear down time, maximum connection tear down time, average
connection tear down time, aggregate connection tear down time, number of connections, aging
time, close method, and close direction. Different packet sizes should produce different
throughput measurements with the same firewall DUT configuration.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
56
CAS_016 Determine the capacity of NAT444 in the cloud
Abstract
NAT444, or Carrier Grade NAT, provides a mechanism to extend the IPv4 address space within a
carrier core. This test case determines the peak capacity of a Carrier Grade NAT device.
Frequently NAT444 is used in a transition strategy from IPv4 to mixed IPv4 and IPv6 networks.
Determining the upper scale of the NAT Device under Test (DUT) allows for proper capacity
planning in the network.
Description
As IPv4 addresses allocated to carriers by IANA run out, carriers need ways to better utilize their
existing allocation of IPv4 addresses. One technique is NAT444, or industrial grade NAT.
Effectively, NAT444 is a double network address translation. The subscriber’s cloud is translated
by the default gateway router. (This is sometimes called NAT44.) The upstream translated traffic
is further translated by an industrial grade NAT router. This technique has the potential for
problems. Specifically, the technique breaks the OSI model, adds single points of failure, and can
cause performance and protocol problems for subscribers. This test measures the peak capacity
of a NAT444 industrial grade router. The following QoS table details the pass/fail targets:
Codepoint Max
Jitter
(uSec)
Max
Latency
(uSec)
Max
Loss
(Frames)
Max
Duplicate
(Frames)
Max
Reordered
(Frames)
Max Late
(Frames)
EF 0 >=1 0 0 0 0
AF31 0 2 0 0 0 0
AF21 2 5 0 1 1 1
AF11 3 5 1 1 1 1
BE ANY ANY ANY ANY ANY ANY
Target Users
Product Verification, Engineering, End users
Target Device Under Test (DUT)
Type Device Under Test (DUT) is an industrial grade NAT444 Router
Reference
draft-shirasaki-nat444-isp-shared-addr-05
RFC 4787
RFC 5382
RFC 5508
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
57
Relevance
Measuring the peak performance of a NAT444 router is critical to the deployment capacity of
NAT in the network. With the potential for so many performance issues, NAT444 must be
carefully tuned and measured before deployment. Because significant modification to the
subscriber traffic occurs, there is a large potential for jitter, sequencing, latency, and corruption
issues. Special consideration for loss, duplication, re-order, out of order, and late packets must
be taken into account. In addition, a NAT44 device can be very susceptible to microburst
conditions and real-time change.
Version
1.0
Test Category
CAS
PASS
[x] Performance [ ] Availability [ ] Security [ ] Scale
Required Tester Capabilities
To test NAT444 fully, the tester must be able to measure latency, jitter, loss, re-order,
duplication, late, and error is one pass of the packet. In addition the tester must have the ability
to make real-time changes to content, add and subtract content, and produce microbursts.
Topology Diagram
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
58
Test Procedure
1. Begin
a. Setup a subscriber pool of dual-homed IPv4 and IPv6 subscribers. Set 255 hosts per
client subscriber. Randomize the IPv4 destination. Set the IPv4 source address to the
address of the Public NAT address. Encapsulate IPv6 using 6over4 rules.
b. On the IPv6 hosts, setup HTTP, SIP, and Video.
c. Transmit traffic through the NAT444 DUT.
d. Setup the number of subscribers as measured in the performance test.
2. Loop until a QoS or QoE failure is detected.
a. Continuously measure the CPU of the DUT using Spirent iTest (1 Sample every 1
Second).
b. Take down 50% of the subscribers, wait 30 seconds.
c. Measure the QoE of the remaining subscribers while bringing the downed 50% of
subscribers back up.
d. Take down a random number of subscribers for 30 seconds.
e. Measure the QoE of the remaining subscribers while bringing the downed random
number of subscribers back up.
f. Examine the variance of the CPU utilization sample. The variance should be no more
than +/- 5%.
a. if no failures are measured, add 10% more subscribers, otherwise stop.
3. End Loop
Control Variables & Relevance
Variable Relevance Default Value
Duration Time per Iteration 120 seconds
Number of Subscribers Pressure on the NAT444 DUT 1000
Random Subscribers Number of Subscribers to Down/Up 50%
Key Measured Metrics
Metric Relevance Metric Unit
HTTP Goodput Ratio QoE Metric ratio for Web traffic (Sent : Measured) >0.99
SIP MOS Measured SIP Quality >4.5
Subscriber capacity Measured number of Peak Subscribers Subscribers
Video MOS-AV Measure Video Quality >4.5
Desired Result
As the number of subscribers increase, decrease and change, the QOS/QoE metrics of traffic
should be both consistent and high. There should be no measured packet errors.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
59
Analysis
When examining QoE metrics, pay special attention to when traffic is changing in the network.
Specifically, when subscriber traffic is being added in microburst, you should see almost no
change to the QoS and QoE metrics during the time of bursts and change in general. In addition,
pay special attention to QoS foundation variables like packet reorder and duplication. These are
signs that you are approaching the upper limit of the NAT444 DUT.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
60
CAS_017 Evaluate DNS A and AAAA record performance under
attack
Abstract
DNS protocols are a key determining factor of overall user experience. This test measures the
ability of DNS servers to respond in a predictable manner under extreme load. This test allows
the user to determine the scale to which DNS may be provisioned in the network.
Description
Domain Name System (or Service or Server) is an Internet service that translates domain names
into IP addresses. Because domain names are alphabetic, they're easier to remember than IP
addresses. The Internet however, is really based on IP addresses. Every time you use a domain
name, therefore, a DNS service must translate the name into the corresponding IP address. For
example, the domain name www.example.com might translate to 198.105.232.4. The DNS
system is, in fact, its own network. If one DNS server doesn't know how to translate a particular
domain name, it asks another one, and so on, until the correct IP address is returned.
This test measures the capacity of DNS A Record and DNSv6 AAAA Record requests against a live
set of the top internet sites.
Target Users
Enterprise, PV, Engineering
Target Device Under Test (DUT)
Primary and Secondary DNS Servers
Reference
RFC 1034
RFC 3901
Relevance
DNS quality of experience and stability is a critical component of a user’s perception of network
performance. The predictability of DNS address resolution has a direct correlation to perceived
performance. In addition, it is assumed that DNS must perform even while under attack.
Version
1.0
Test Category
CAS
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
61
PASS
[x] Performance [ ] Availability [x] Security [ ] Scale
Required Tester Capabilities
The tester must have the ability to generate predictable A and AAAA record rates while
simultaneously attacking the primary and secondary DNS servers
Topology Diagram
Test Procedure
1. Reserve the test port and bring up link.
2. Determine the desired rates of a list for A Record and AAA record look ups.
3. Program the list of the top 500 Internet sites into the tester.
4. Determine the baseline by testing the top 120 sites at 1 request per second. Determine the
maximum resolve time, as defined by the latency between the sent request and the
reception of the resolve. Measure the maximum resolve time for both A Record and AAAA
Record.
5. Sync the Attack database with spirent.com. Program all DNS attacks in the database at 100
attacks per second, targeting both the primary and secondary DNS servers.
6. Start the attacks.
7. For each of the desired A and AAAA Record Rates:
a. Set the rate for both A and AAAA requests.
b. Round Robin the list of DNS sites to be resolved
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
62
c. Set the desired duration. Set a mix of 90% DNS requests to the primary DNS and 10% to
the secondary DNS.
d. Transmit DNS requests.
e. Measure the maximum resolve time and the number of failed resolves.
f. Repeat for the desired number of trials.
8. Stop the attacks.
9. End.
Control Variables & Relevance
Variable Relevance Default Value
Primary DNS Server DNS Server NA
Secondary DNS Server Backup DNS Server NA
Top 500 Sites List of top 500 DNS Resolves List
A Record Rate List List of DNS Resolve Rates per Second 1,1000, 5000, 10000
Iteration Duration Time per Iteration 120 Seconds
Trials Trails per Rate 3
Key Measured Metrics
Metric Relevance Metric Unit
A Record Resolve Latency Maximum time to resolve A Record mSec
AAAA Record Resolve Latency Maximum time to resolve A Record mSec
Desired Result
The baseline of 1 resolve per second for A Record and AAAA Record should be 2 mSec or faster.
The variance for additional rate and trials should be no slower than 1% of the baseline.
Analysis
Report the maximum time per rate per trial in the form of a table and chart, one table and chart
for A Record resolves and one for AAAA Record resolves.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
63
Appendix A – Telecommunications
Definitions
APPLICATION LOGIC. The computational aspects of an application, including a list of instructions that tells a
software application how to operate.
APPLICATION SERVICE PROVIDER (ASP). An ASP deploys hosts and manages access to a packaged application by
multiple parties from a centrally managed facility. The applications are delivered over networks on a
subscription basis. This delivery model speeds implementation, minimizes the expenses and risks incurred
across the application life cycle, and overcomes the chronic shortage of qualified technical personnel
available in-house.
APPLICATION MAINTENANCE OUTSOURCING PROVIDER. Manages a proprietary or packaged application from
either the customer's or the provider's site.
ASP INFRASTRUCTURE PROVIDER (AIP). A hosting provider that offers a full set of infrastructure services for
hosting online applications.
ATM. Asynchronous Transport Mode. An information transfer standard for routing high-speed, high-
bandwidth traffic such as real-time voice and video, as well as general data bits.
AVAILABILITY. The portion of time that a system can be used for productive work, expressed as a
percentage.
BACKBONE. A centralized high-speed network that interconnects smaller, independent networks.
BANDWIDTH. The number of bits of information that can move through a communications medium in a
given amount of time; the capacity of a telecommunications circuit/network to carry voice, data, and
video information. Typically measured in Kbps and Mbps. Bandwidth from public networks is typically
available to business and residential end-users in increments from 56 Kbps to 45 Mbps.
BIT ERROR RATE. The number of transmitted bits expected to be corrupted per second when two computers
have been communicating for a given length of time.
BURST INFORMATION RATE (BIR). The rate of information in bits per second that the customer may need over
and above the CIR. A burst is typically a short duration transmission that can relieve momentary
congestion in the LAN or provide additional throughput for interactive data applications.
BUSINESS ASP. Provides prepackaged application services in volume to the general business market,
typically targeting small to medium size enterprises.
BUSINESS-CRITICAL APPLICATION. The vital software needed to run a business, whether custom-written or
commercially packaged, such as accounting/finance, ERP, manufacturing, human resources and sales
databases.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
64
BUSINESS SERVICE PROVIDER. Provides online services aided by brick-and-mortar resources, such as payroll
processing and employee benefits administration, printing, distribution or maintenance services. The
category includes business process outsourcing (BPO) companies.
COMMERCE NETWORK PROVIDER. Commerce networks were traditionally proprietary value-added networks
(VANs) used for electronic data interchange (EDI) between companies. Today the category includes the
new generation of electronic purchasing and trading networks.
COMPETITIVE ACCESS PROVIDER (CAP). A telecommunications company that provides an alternative to a LEC
for local transport and special access telecommunications services.
CAPACITY. The ability for a network to provide sufficient transmitting capabilities among its available
transmission media, and respond to customer demand for communications transport, especially at peak
usage times.
CLIENT/DEVICE. Hardware that retrieves information from a server.
CLUSTERING. A group of independent systems working together as a single system. Clustering technology
allows groups of servers to access a single disk array containing applications and data.
COMPUTING UTILITY PROVIDER (CUP). A provider that delivers computing resources, such as storage, database
or systems management, on a pay-as-you-go basis.
CSU/DSU. Channel Server Unit/Digital Server Unit. A device used to terminate a telephone company
connection and prepare data for a router interface.
DATA MART. A subset of a data warehouse, intended for use by a single department or function.
DATA WAREHOUSE. A database containing copious amounts of information, organized to aid decision-
making in an organization. Data warehouses receive batch updates and are configured for fast online
queries to produce succinct summaries of data.
DEDICATED LINE. A point-to-point, hardwired connection between two service locations.
DEMARCATION LINE. The point at which the local operating company's responsibility for the local loop ends.
Beyond the demarcation point (also known as the network interface), the customer is responsible for
installing and maintaining all equipment and wiring.
DISCARD ELIGIBILITY (DE) BIT. Relevant in situations of high congestion, it indicates that the frame should be
discarded in preference to frames without the DE bit set. The DE bit may be set by the network or by the
user; and once set cannot be reset by the network.
DS-1 OR T-1. A data communication circuit capable of transmitting data at 1.5 Mbps. Currently in
widespread use by medium and large businesses for video, voice, and data applications.
DS-3 OR T-3. A data communications circuit capable of transmitting data at 45 Mbps. The equivalent data
capacity of 28 T-1s. Currently used only by businesses/institutions and carriers for high-end applications.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
65
ELECTRONIC DATA INTERCHANGE (EDI). The electronic communication of business transactions (orders,
confirmations, invoices etc.) of organizations with differing platforms. Third parties provide EDI services
that enable the connection of organizations with incompatible equipment.
ENTERPRISE ASP. An ASP that delivers a select range of high-end business applications, supported by a
significant degree of custom configuration and service.
ENTERPRISE RELATIONSHIP MANAGEMENT (ERM). Solutions that enable the enterprise to share comprehensive,
up-to-date customer, product, competitor and market information to achieve long-term customer
satisfaction, increased revenues, and higher profitability.
ENTERPRISE RESOURCE PLANNING (ERP). An information system or process integrating all manufacturing and
related applications for an entire enterprise. ERP systems permit organizations to manage resources
across the enterprise and completely integrate manufacturing systems.
ETHERNET. A local area network used to connect computers, printers, workstations, and other devices
within the same building. Ethernet operates over twisted wire and coaxial cable.
EXTENDED SUPERFRAME FORMAT. A T1 format that provides a method for easily retrieving diagnostics
information.
FAT CLIENT. A computer that includes an operating system, RAM, ROM, a powerful processor and a wide
range of installed applications that can execute either on the desktop or on the server to which it is
connected. Fat clients can operate in a server-based computing environment or in a stand-alone fashion.
FAULT TOLERANCE. A design method that incorporates redundant system elements to ensure continued
systems operation in the event of the failure of any individual element.
FDDI. Fiber Distributed Data Interface. A standard for transmitting data on optical-fiber cables at a rate of
about 100 Mbps.
FRAME. The basic logical unit in which bit-oriented data is transmitted. The frame consists of the data bits
surrounded by a flag at each end that indicates the beginning and end of the frame. A primary rate can be
thought of as an endless sequence of frames.
FRAME RELAY. A high-speed packet switching protocol popular in networks, including WANs, LANs, and
LAN-to-LAN connections across long distances.
GBPS. Gigabits per second, a measurement of data transmission speed expressed in billions of bits per
second.
HOSTED OUTSOURCING. Complete outsourcing of a company's information technology applications and
associated hardware systems to an ASP.
HOSTING PROVIDER. Provider who operates data center facilities for general-purpose server hosting and
collocation.
INFRASTRUCTURE ISV. And independent software vendor that develops infrastructure software to support
the hosting and online delivery of applications.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
66
INTEGRATED SERVICES DIGITAL NETWORK (ISDN). An information transfer standard for transmitting digital voice
and data over telephone lines at speeds up to 128 Kbps.
INTEGRATION. Equipment, systems, or subsystem integration, assembling equipment or networks with a
specific function or task. Integration is combining equipment/systems with a common objective, easy
monitoring and/or executing commands. It takes three disciplines to execute integration: 1) hardware, 2)
software, and 3) connectivity – transmission media (data link layer), interfacing components. All three
aspects of integration have to be understood to make two or more pieces of equipment or subsystems
support the common objective.
INTER-EXCHANGE CARRIER (IXC). A telecommunications company that provides telecommunication services
between local exchanges on an interstate or intrastate basis.
INTERNET SERVICE PROVIDER (ISP). A company that provides access to the Internet for users and businesses.
INDEPENDENT SOFTWARE VENDOR (ISV). A company that is not a part of a computer systems manufacturer
that develops software applications.
INTERNETWORKING. Sharing data and resources from one network to another.
IT SERVICE PROVIDER. Traditional IT services businesses, including IT outsourcers, systems integrators, IT
consultancies and value added resellers.
KILOBITS PER SECOND (KBPS). A data transmission rate of 1,000 bits per second.
LEASED LINE. A telecommunications line dedicated to a particular customer along predetermined routers.
LOCAL ACCESS TRANSPORT AREA (LATA). One of approximately 164 geographical areas within which local
operating companies connect all local calls and route all long-distance calls to the customer's inter-
exchange carrier.
LOCAL EXCHANGE CARRIER (LEC). A telecommunications company that provides telecommunication services
in a defined geographic area.
LOCAL LOOP. The wires that connect an individual subscriber's telephone or data connection to the
telephone company central office or other local terminating point.
LOCAL/REGIONAL ASP. A company that delivers a range of application services, and often the complete
computing needs, of smaller businesses in their local geographic area.
MEGABITS PER SECOND (MBPS). 1,024 kilobits per second.
METAFRAME. The world's first server-based computing software for Microsoft Windows NT 4.0 Server,
Terminal Server Edition multi-user software (co-developed by Citrix).
MODEM. A device for converting digital signals to analog and vice versa, for data transmission over an
analog telephone line.
MULTIPLEXING. The combining of multiple data channels onto a single transmission medium. Sharing a
circuit - normally dedicated to a single user - between multiple users.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
67
MULTI-USER. The ability for multiple concurrent users to log on and run applications on a single server.
NET-BASED ISV. An ISV whose main business is developing software for Internet-based application services.
This includes vendors who deliver their own applications online, either directly to users or via other
service providers.
NETWORK ACCESS POINT (NAP). A location where ISPs exchange traffic.
NETWORK COMPUTER (NC). A thin-client hardware device that executes applications locally by downloading
them from the network. NCs adhere to a specification jointly developed by Sun, IBM, Oracle, Apple and
Netscape. They typically run Java applets within a Java browser, or Java applications within the Java
Virtual Machine.
NETWORK COMPUTING ARCHITECTURE. A computing architecture in which components are dynamically
downloaded from the network onto the client device for execution by the client. The Java programming
language is at the core of network computing.
ONLINE ANALYTICAL PROCESSING (OLAP). Software that enables decision support via rapid queries to large
databases that store corporate data in multidimensional hierarchies and views.
OPERATIONAL RESOURCE PROVIDER. Operational resources are external business services that an ASP might
use as part of its own infrastructure, such as helpdesk, technical support, financing, or billing and payment
collection.
OUTSOURCING. The transfer of components or large segments of an organization's internal IT infrastructure,
staff, processes or applications to an external resource such as an ASP.
PACKAGED SOFTWARE APPLICATION. A computer program developed for sale to consumers or businesses,
generally designed to appeal to more than a single customer. While some tailoring of the program may be
possible, it is not intended to be custom-designed for each user or organization.
PACKET. A bundle of data organized for transmission, containing control information (destination, length,
origin, etc.), the data itself, and error detection and correction bits.
PACKET SWITCHING. A network in which messages are transmitted as packets over any available route rather
than as sequential messages over circuit-switched or dedicated facilities.
PEERING. The commercial practice under which nationwide ISPs exchange traffic without the payment of
settlement charges.
PERFORMANCE. A major factor in determining the overall productivity of a system, performance is primarily
tied to availability, throughput and response time.
PERMANENT VIRTUAL CIRCUIT (PVC). A PVC connects the customer's port connections, nodes, locations, and
branches. All customer ports can be connected, resembling a mesh, but PVCs usually run between the
host and branch locations.
POINT OF PRESENCE (POP). A telecommunications facility through which the company provides local
connectivity to its customers.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
68
PORTAL. A company whose primary business is operating a Web destination site, hosting content and
applications for access via the Web.
REMOTE ACCESS. Connection of a remote computing device via communications lines such as ordinary
phone lines or wide area networks to access distant network applications and information.
REMOTE PRESENTATION SERVICES PROTOCOL. A set of rules and procedures for exchanging data between
computers on a network, enabling the user interface, keystrokes, and mouse movements to be
transferred between a server and client.
RESELLER/VAR. An intermediary between software and hardware producers and end users. Resellers
frequently add value (thus Value-Added Reseller) by performing consulting, system integration and
product enhancement.
ROUTER. A communications device between networks that determines the best path for optimal
performance. Routers are used in complex networks of networks such as enterprise-wide networks and
the Internet.
SCALABILITY. The ability to expand the number of users or increase the capabilities of a computing solution
without making major changes to the systems or application software.
SERVER. The computer on a local area network that often acts as a data and application repository and that
controls an application's access to workstations, printers and other parts of the network.
SERVER-BASED COMPUTING. A server-based approach to delivering business-critical applications to end-user
devices, whereby an application's logic executes on the server and only the user interface is transmitted
across a network to the client. Benefits include single-point management, universal application access,
bandwidth-independent performance, and improved security for business applications.
SINGLE-POINT CONTROL. One of the benefits of the ASP model, single-point control helps reduce the total
cost of application ownership by enabling widely used applications and data to be deployed, managed
and supported at one location. Single-point control enables application installations, updates and
additions to be made once, on the server, which are then instantly available to users anywhere.
SPECIALIST ASP. Provide applications which serve a specific professional or business activity, such as
customer relationship management, human resources or Web site services.
SYSTEMS MANUFACTURER. Manufacturer of servers, networking and client devices.
TELECOMS PROVIDER. Traditional and new-age telecommunications network providers (telcos).
THIN CLIENT. A low-cost computing device that accesses applications and and/or data from a central server
over a network. Categories of thin clients include Windows-Based Terminals (WBT, which comprise the
largest segment), X-Terminals, and Network Computers (NC).
TOTAL COST OF OWNERSHIP (TCO). Model that helps IT professionals understand and manage the budgeted
(direct) and unbudgeted (indirect) costs incurred for acquiring, maintaining and using an application or a
computing system. TCO normally includes training, upgrades, and administration as well as the purchase
price. Lowering TCO through single-point control is a key benefit of server-based computing.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
69
TOTAL SECURITY ARCHITECTURE (TSA). A comprehensive, end-to-end architecture that protects the network.
TRANSMISSION CONTROL PROTOCOL/INTERNET PROTOCOL (TCP/IP). A suite of network protocols that allow
computers with different architectures and operating system software to communicate over the Internet.
USER INTERFACE. The part of an application that the end user sees on the screen and works with to operate
the application, such as menus, forms and buttons.
VERTICAL MARKET ASP. Provides solutions tailored to the needs of a specific industry, such as the healthcare
industry.
VIRTUAL PRIVATE NETWORK (VPN). A secure, encrypted private connection across a cloud network, such as
the Internet.
WEB HOSTING. Placing a consumer's or organization's web page or web site on a server that can be
accessed via the Internet.
WIDE AREA NETWORK. Local area networks linked together across a large geographic area.
WINDOWS-BASED TERMINAL (WBT). Thin clients with the lowest cost of ownership, as there are no local
applications running on the device. Standards are based on Microsoft's WBT specification developed in
conjunction with Wyse Technology, NCD, and other thin client companies.
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
70
Appendix B – Stateful Playlist by QoS
The following patterns represent industry best practices in modeling stateful traffic across a Device Under
Test (DUT) by QoS.
Name DiffServ EF
(Real-Time)
DiffServ 0x31 (Critical) DiffServ 0x20
(General)
DiffServ 0x00 (Best Effort)
Enterprise Campus
Apps
VoIP 15%
(SIP+RTP+G.729A),U
nicast Web
Conference 2-Way
(MPEG2-TS, VBR),
SIP 5%
Routing 3% OSPF Routing
Updates 2%, BGP Updates 1%),
Database 17% (Oracle SQLNet
Updates), Corporate Web 2% ,
IMAP4 5%
Multicast Video 13% (480i,
MPGEG-2, IGMPv2, 5
Multicast Channels),
Telnet/SSH (2%), CIFS 10%
(1:1:3 Small/Medium/Large
Ratio)
Internet Web 5% HTTP (1024 Byte index.html, 30
500 Byte JPEG, 5 1K JPEG, 1x 100k jpeg), BitTorrent
11%
Higher Education Network
Administration 2%
(SSH)
SQL 7% SQLNet SQL Table
Updates), HTTPS University
Admin 3% (64 Bytes index.html,
5x 1K JPEG Images), Video
Conference 5% (MPEG2TS, VBR,
480i), VoIP 5% (G.729A CODEC)
FTP 7% (Large Files), HTTPS
Student Services, HTTP 3%,
POP3/SMTP 9%, CIFS 8%
(1:1:3 Small/Medium/Large
Objects, bidirectional),
Multicast Video 5% (480i)
IM 12% (AIM) , BitTorrent 24%, HTTP 3% (1024 Byte
index.html, 30 500 Byte JPEG, 5 1K JPEG, 1x 100k
jpeg), HTTPS 1% (64 Bytes index.html, 5x 1K JPEG
Images), Mail 5%, FTP 1% (Large Files), Telnet/SSH
3%
Service Providers Telnet/SSH 1% BGP Route Updates 1% N/A 50% P2P (Bit Torrent, 5% Peer to Tracker, 95% Peer-
2-Peer), 30% HTTP (1024 Byte index.html, 30 500
Byte JPEG, 5 1K JPEG, 1x 100k jpeg), 5% DNS, Video
(MPEG2-TS 5%), SIP (G.729A 3%), Gaming (WoW
5%), 2% RAW TCP
10G Max Bandwidth No Payload, RAW TCP
1G max Bandwidth No Payload, RAW TCP
Small/Medium Business Apps POP2/SMTP 15% (5:2:1 Ratio of Small/medium/Big
ratio). HTTPS 20% (64 Bytes index.html, 5x 1K JPEG
Images, CIFS 30% (1:1:3 Small/Medium/Large
Objects, bidirectional, BitTorrent 10%, Internet
Web 25% HTTP (1024 Byte index.html, 30 500 Byte
JPEG, 5 1K JPEG, 1x 100k jpeg)
WAN Accelerator Network Control 5%
(Windows Domain
Controller Updates),
Network Logins
CIFS 40% (1:1:3
Small/Medium/Large Fields).
Exchange 35%(5:2:1
Small/Medium/Large ratio)
HTTPS 10% (64 Bytes
index.html, 5x 1K JPEG
Images)
BitTorrent 10%
Internet AppMix 2011 50% P2P (Bit Torrent, 5% Peer to Tracker, 95% Peer-
2-Peer), 30% HTTP (1024 Byte index.html, 30 500
Byte JPEG, 5 1K JPEG, 1x 100k jpeg), 5% DNS, Video
(MPEG2-TS 5%), SIP (G.729A 3%), Gaming (WoW
5%), 2% RAW TCP
Spirent Journal of Cloud Application and Security Services PASS Test Methodologies
© Spirent Communications 2011
71
Appendix C – MPEG 2/4 Video QoE
The following information is a typical pattern for MPGE2TS based video streams with a normalized MOS-
AV schedule.
