Spirent circle logo

Structuring a Real World Test for High Performance Firewalls


Spirent and Crossbeam were recently part of a major collaboration to define and document a test methodology that could accurately assess the performance of a firewall on the demanding Gi(3G) or SGi(4G-LTE) interface of a mobile operator's network.

“Structuring a real world test”

Spirent and Crossbeam were recently part of a major collaboration to define and document a test methodology that could accurately assess the performance of a firewall on the demanding Gi(3G) or SGi(4G-LTE) interface of a mobile operator's network.  Working to design this test were EANTC (the European Advanced Networking Test Center, internationally recognized for its test expertise), Heavy Reading (a provider of deep analysis of telecom trends), Spirent (a leading test equipment vendor whose systems simulate user behavior), and Crossbeam (a network security provider, whose solutions are used extensively on mobile operators' Gi/SGi interfaces).  For the test results to be predictive of real world performance, the test needed to:

  1. Simulate mobile subscriber behavior and measure all metrics simultaneously

  2. Measure the quality of experience of those subscribers

  3. Emulate the scale and topology of a mobile operator’s network

Traditional testing methods lacking

Firewalls have been traditionally validated by generating HTTP traffic of a fixed object size (or more recently a series of sizes) and then measuring the open connections, bandwidth, and connections/second. This methodology (based on IETF RFC 3511 from nearly a decade ago) has typically resulted in publication of a single large bandwidth metric, which the test had been designed to optimize. Although the results of such tests can be valuable for generating marketing headlines, they are of limited use in predicting or even assessing network performance. So it shouldn’t come as a surprise that deployments relying on test results of this type have in some cases yielded only a fraction of the performance that was expected.

A new approach

The goals of this most recent collaboration were to design a test that would credibly evaluate network security performance on the Gi/SGi interface and that would measure all performance metrics simultaneously. Since the Gi/SGi interface is a pure IP interface (where IP traffic generated by UMTS or LTE leaves the mobile core), typical traffic on this interface was examined. The traffic from a mobile subscriber comes from a predictable set of activities: get on the network, browse to a web page, move from page to page and tab to tab, send and receive emails, and download apps and OS updates. The test simulated subscribers carrying out these activities, using the SimUser functionality on four Spirent Avalanche 3100Bs, to emulate the traffic that would actually cross the Gi/SGi interface. As in actual mobile networks, users were "born" (i.e. activated their connection) and then "died" once their activities were completed.

Layer 7 (not layer 4) full-stack traffic  was simulated and users were loaded in a realistic manner using the Spirent Avalanche LoadSpec to find the actual upper limit of the device under test.  From there, the test was scaled to test the impact of millions of users, representing the anticipated subscriber growth that mobile service providers will face over the next 5 years.  After all, just a year and a half ago EANTC, together with Light Reading, published results for Cisco’s flagship packet gateway test showing 20Gbit/s of packet throughput. According to Heavy Reading’s Gabriel Brown, the "throughput [in this test] is in excess of what is typically deployed in mobile networks today and should provide headroom for operators to securely scale mobile internet services in the LTE era."

Measurement of the User Experience

As part of the test, explicit minimum user thresholds were set and user experience measured by page render times, time to traverse the web application, errors detected, mail access time, and OS update download time for each user in the test. Pages composed of HTML and real-world sized graphics were expected to render end–to-end in the network with no TCP retransmissions and no layer 2-7 errors. These thresholds gated the test - simulated users were added, with their resulting bandwidth, only until the device under test exceeded one of those established thresholds. Over 12 million users were spawned during the 5-minute steady state period of the test on the Given recent studies that correlate subscriber retention with consistency of reasonable page load times, this was considered essential to measure user experience.

Defining & Documenting an Applicable Methodology & Topology

This test defined a methodology and topology which was as faithful as possible to a mobile operator’s actual network, including simulation of subscriber behavior and simultaneous measurement of all metrics. Reports of the test topology and methodology and its applicability to mobile operators are available for review, enabling contrasts to be drawn to other environments and methodologies.  With its wealth of documentation, the real world relevance of this test is far more significant than publication of a single maximum number without context, which has been typical of some firewall tests in the past.

Gail Ferreira is the Product Marketing Manager – Mobile at Crossbeam

Like our content?

Subscribe to our blogs here.

Blog Newsletter Subscription