Spirent circle logo

How Intelligent Testing Strategies Facilitate Early AI Deployments


There are many kinds of AI systems being integrated into network elements like routers, switches, next-gen firewalls, consumer devices, and much more. Just as networks are tested for scalability, predictability, reproducibility, performance, and reliability, so must the AI software that analyzes network data. Learn more about the critical factors in developing any AI testing strategy.

Is there anything AI can’t do? If you believe all the hype, promise and hysteria, the answer is a resounding no.

Of course, it’s not quite that simple.

Artificial intelligence (AI) is real. Its capabilities may one day know no bounds. But today, it’s still finding its footing. Still, AI has been deemed ready enough to begin shouldering the burden of increasingly complex tasks. Among them, helping to manage the telecom networks of tomorrow.

How do we know AI is up for the challenge? We don’t. Not really. That’s why testing it extensively and continuously, both before and during deployments (in controlled environments like a digital twin) will be critical to assuring it doesn’t go off the rails.

There are many kinds of AI systems, each with varying types of inputs and goals. We’re seeing these systems show up in consumer devices like cameras and home appliances. Increasingly, they’re being integrated into network elements like routers, switches, and next-gen firewalls.

Regardless of the application, like any other software, AI expert systems must be tested. Just as networks are tested for scalability, predictability, reproducibility, performance, and reliability, so must the AI software that analyzes network data.


Regardless of the application, like any other software, AI expert systems must be tested.

Let’s explore the early role of AI in telecom networks and key considerations for testing that will lead to successful outcomes.

Important considerations for AI testing

In telecom, the nature of the AI system deployed and the underlying network it will support dictate what and how to test.

In the lab, we need to make the test environment as close to reality as possible by subjecting the system under test (SUT) to traffic that emulates the real world. Emulation enables consistent testing since it can reproduce traffic at will. It can determine, for example, how quickly the expert system learns, whether it learns (or trains) correctly and how it performs during peak data rates.

Topology of an Active Assurance AI Testing

AI testing should consider factors that include:

  • Scalability. Input and output data from sourcing nodes flows across an AI system, which then parses the data, creates models and draws conclusions. Can the system scale from handling a few nodes of data now to tens of thousands or millions of flows and users that may be needed by the end of the anticipated product lifetime? By emulating large networks (1M+ data points/hour), determination of how well the AI system works under scale can be achieved.

  • Training efficiency and prediction stability. How quickly can AI learn from the data and draw the correct conclusion with five-nines (99.999%) predictability? Is it resilient to a variety of training sets? Regression techniques are used to determine how many cycles it takes for the system to properly get the correct input answer. Having an independent metric like the number of cycles is critical for determining whether code upgrades impact training efficiency.

  • Predictability over time. An AI system’s performance must not degrade over time. This is particularly important for the AI in security devices. Since AI traffic emulation can speed up the clock and emulate scale, it can see over a long period of time and whether the system maintains its stability and performance.

  • Geographical variations. Different continents, countries, regions, cities, and small towns will have different source data densities. The expert system needs to be able to handle and model, for example, a city’s random internet data bursts. Geographic-specific emulations assess whether the system can handle all the data patterns to which it may be exposed.

  • Background data. In real AI networks, source data will be mixed with random background data that may or may not be relevant to the AI training. The AI system needs to be able to parse all data and pinpoint the data that actually matters. Network emulations must include background data from the input and output nodes as well as other relevant nodes. If a third-party element is used to remove the background noise, it needs to become part of the test domain and subject to the same tests as the AI system.

  • Attack data. In an AI-level attack, a nefarious entity attempts to mislead the AI by intentionally inserting bad data, hoping to create false negatives or positive conclusions. For example, a facial recognition hack may make it look like a person is in ten places at once. The AI system must determine what makes sense in context. If it sees ten people with the same face at the same time, it can look back to identify when it first saw the face and the location of the camera where it appeared. The emulation tests should subject the expert system to these kinds of bad data inputs to determine how well it can detect and mitigate them.

AI testing emulation means reproducible, unbiased results

Useful AI testing must ultimately provide reproducible, unbiased comparisons of the performance of the many AI products coming to market. Instead of reviewing vendor data sheets and their test results, a proof of concept independently tests the vendor’s product using emulation techniques as described in this blog.

Spirent is blazing the trail for AI testing on behalf of enterprises and vendors. We work with our customers to set up the test environment—typically in the cloud—based on specific source data and AI goals.

The capabilities of AI will be vast. Assuring it can deliver early results means making testing with the right strategy from the earliest development phases a core component of any successful deployment.

Learn more about Spirent’s Cloud and Virtualization Test solutions.

Like our content?

Subscribe to our blogs here.

Blog Newsletter Subscription

Chris Chapman

Senior Methodologist, Spirent

With over 20 years in Telecommunications and 11+ years of network performance theory, Chris has extensive knowledge in testing and deployment of L1-7 network systems. His expertise includes performance analysis of QoS, QoE, TCP, IP (v4 and v6), UDP, QoE, HTTP(S), FTP, WAN acceleration, BGP, OSPF, IS-IS. MPLS, LDP, RSVP, VPLS, firewalls and load balancers. His specialties are centered on testing L1-7.