Spirent Broadband Blog

Creating an Industry-First Performance Test: Validating a “super-class” platform

Fri, 20 Jan 2012 18:55:00 GMT

By Gail Ferreira, Product Marketing Manager at Crossbeam, and Chris Chapman, Technical Marketing Engineer at Spirent

Crossbeam and Spirent recently partnered with EANTC, an internationally recognized test center, and Heavy Reading analyst Gabriel Brown to define and emulate the most realistic real-world conditions within mobile networks, and evaluate the performance of Crossbeam X80-S as a network security device under these demanding conditions.

We agreed on one key requirement — we wanted to emulate, as closely as possible, what happens on a mobile network when subscribers are actually using their smartphones and tablets, not just a limited best-case viewpoint. Throughout the planning process, some of the questions we wanted to examine included:

How do you evaluate a "super-class" platform like the Crossbeam X80-S?
What would be meaningful to address the needs of a mobile operator/service cloud provider?
How consistent will the subscriber Quality of Experience and happiness remain when running over a hundred of gigabits per second of throughput?
Will it take 30 seconds to load a web page?

We set up the test to stress the Crossbeam platform with a true worst-case scenario, and evaluate all vectors simultaneously. We knew that the performance and quality levels created in this scenario would represent the most demanding conditions required to actually deliver service, which would enable mobile operators to accurately predict the peak performance in their networks and the devices they choose.

Simulating Real-World Mobile Users

To be realistic we wanted to fully emulate user behavior on a mobile device using the SimUser functionality on the Spirent Avalanche. For example, a mobile user will not judge the quality of the service by a single transaction (a GET). Instead, a user is more concerned with how fast the page renders on their mobile device. In our test, we wanted to form a mobile page by combining an index.html document with sub-objects, and then measure the stack (“New Reno” TCP) from bandwidth through page load time. Because we are directly measuring the render time of the mobile browser, we decided to add additional realism by emulating a tabbed mobile browser, viewing three pages with eight seconds of “browsing time by the user” per page.

We used 8 Avalanche 3100, with 4 generating traffic, and 4 as clients receiving the traffic, as follows:

Simulating Mobile Traffic

We wanted to emulate this mobile traffic as a series of transactions. For instance, depending on the actions each subscriber took, the relationship of Simulated Users (SimUser) to TCP Connections to Layer 7 Transactions varied (as would a real subscriber in a production network). So we used a distribution of 95% HTTP, 2% POP3, 1% SMTP, 1% DNS, and 1% large bulk transfer (over HTTP). Each of these subscribers generated mobile data traffic as the test ramped according to the loading curve defined in the Spirent Avalanche, increased up for a sustained period, and then ramped down. Thus each Subscriber performed the following:

Traffic	Type	Traffic Generated per SimUser (1 Million concurrently active)
Traffic	Type	# Open TCP Connections	Transaction Count	Notes/Ending sequences
95%	HTTP	12	15 (Layer 7)	Closed all open TCP connections with a RST, the Server responded with a RST, ACK and resources deallocated
2%	POP3	1 – creating a 64k byte message to 2 users	1 - sent to mail server
1%	SMTP	Emulated POP3 server 1 or 2 64k messages	1 - each subscriber logs into the POP3 mailbox	Checked and retrieved messages.
1%	DNS	1 – subscriber issuing 1 DNS record lookup to an emulated DNS Zone with an A Record resolving to a URL	1
1%	Bulk/OS Update	1 – the subscriber requesting a 2Mb object	1

Results

The Avalanche system measured subscriber QoE bandwidth, TCP connection count and rate, transaction count and rate, errors, HTTP status codes, and more, all time trended. Over 700 metrics were read by the system every four seconds. We tried various combinations on the Crosbeam X80-S, including: baseline testing, firewall with IPS enabled, firewall with NAT enabled, and finally firewall with NAT+IPS simultaneously enabled. Across these we impressively saw 106-108 Gbps of throughput, while simultaneously maintaining 270,000 concurrent open connections per second and 675,000 transactions per second, with only tens of nanoseconds of impact on the Web browser page render time. Check out Crossbeam’s press release for details of the results as well as this whitepaper from Heavy Reading that examines the EANTC test results.

Metrics for Network Performance

Tue, 20 Dec 2011 18:36:00 GMT

Joint post by Gail Ferreira, Chris Chapman & Ankur Chadda

What do you look at to size a network solution, to make sure that it has adequate capacity for your network? What measurements should you focus on when evaluating the capacity of a component in a network?

1) Gbps of throughput? But how much traffic is being dropped in the process of reaching peak Gbps of throughput?
2) Connections per second? Will there be enough CPS to cover peak loads?
3) Or maybe total number of connections that enable all users to be supported?
4) Or is CPU utilization under load the most important indicator so that you can cover peak loads?
5) Or maybe it’s 5 9’s of availability and a failover pair is more important than capacity?

There is an entire constellation of metrics, but which one is the most important?

Do you measure and evaluate each specification (e.g. throughput, connections per second, max concurrent, etc.) separately and exclusive of each other? If so, how do you then apply these technical metrics back to a business metric directly, such as the number of subscribers a mobile operator needs to support on their networks?

It’s not easy to evaluate current capacity and meet anticipated future capacity needs, especially as security threats continue to increase. The pitfalls include:

Focusing just on a single metric
One might anticipate that throughput is the single most important metric, but then find that the traffic profile associated with the throughput of the current or anticipated environment isn’t comparable to the throughput claims on the devices chosen.
Selecting a few elements to measure
Maybe throughput and connections per second are key, but focusing only on them could limit the ability to scale to the total connections needed.
Focus across several capacity measures
But lack an accurate characterization of current and projected traffic to which to contrast alternative devices.
Fail to realize the impact on the user experience
A device under load may not provide a satisfactory user experience even if it can pass all traffic.

How essential is it to look at so many variables? Is it really necessary to have a “wide-angle” perspective of this constellation of metrics? If that were the case, it could mean that there is no substitute for actually measuring the anticipated workload. Actually measuring and comparing the real-world capabilities of devices eliminates the guesswork in finding the best solution for your network’s needs. That avoids the problem of trying to adapt to inadequate devices on your networks. For instance, reducing security to obtain the desired performance, or worse, turning off security altogether, as discussed in the recent survey results.

To learn more about how you can help test these areas and others, Spirent’s PASS methodology journal is covers in-depth, real-world scenarios that you can use when testing. Click here to access the test plans.

Gail Ferreira is the Product Marketing Manager – Mobile at Crossbeam
Chris Chapman is Technical Marketing Engineer at Spirent Communications
Ankur Chadda is Product Marketing Manager at Spirent Communications

Top Five Test Issues of the Mobile Internet Revolution

Fri, 09 Dec 2011 01:19:00 GMT

I don’t know if you’ve noticed, but there’s a mobile internet revolution going on. The signs are all there: a rapid increase in the number of mobile subscribers, the number of devices those subscribers use, and the type and quantity of content and services those subscribers want.

Consider one household on my block: two people, fifteen IP-addressable devices. Seriously. They have three laptops, one desktop, three smartphones, two iPads, one PlayStation 3, one DVR, one network printer and one file server. And they’re about to add an internet-enabled TV for the guest bedroom.

The mobile internet revolution builds on the evolution of several technologies, and beyond upgrades to the wireless access network, core and backhaul, it is also driving a wholesale upgrade – to IPv6 and 100GE. Subscribers’ expectations are set by the fixed broadband experience. To meet their expectations for mobile broadband services, providers must deliver on performance, availability, scalability and security (PASS).

Mobile internet test methodologies focus on five key areas:

Raw performance (L2/L3)
Before a device or network can deliver services, it must reliably deliver bits, frames, and packets at line rate. Use PRBS fill patterns and standards-based forwarding and throughput tests before moving on to other features which depend on this essential capability.

QoS
Real-time communications and entertainment are major drivers for the mobile internet revolution, and Quality of Service (QoS) is the key enabler for these high-bandwidth, latency-sensitive services. Use standards-based test methodologies to validate the QoS functionality of the solution.

Application performance (L4/L7)
To evaluate true application performance, the solution must be tested in a realistic environment that reflects the conditions under which it will be deployed. Use a test platform that goes beyond emulation to simulate real user behavior, real stateful, converged traffic, and real network conditions.

QoE
Quality of Experience (QoE) is the new metric for assessing and delivering end-to-end services and content. Delivering QoE goes beyond assessing the queuing and prioritization capabilities of a device or system to measuring the quality of the actual content delivered to the subscriber. Use a test platform capable of simulating the constantly-changing nature of the mobile internet environment and assessing end-user quality scores.

IPv4/IPv6 performance
IPv4 and IPv6 will coexist for many years. Use standards-based test methodologies to evaluate dual-stack performance, 6to4 tunneling performance, prefix-length performance, and other key aspects of performance.

To find out how your testing can support test realism, scalability (whether in ports, subscribers, sessions, VPNs, routes, tunnels, or VLANs), automation and intelligent results, download the white paper from Spirent, 100G Ethernet: The Speed of Now.

For more Ethernet Testing information, visit Spirent.com/Ethernet.

Avalanche 3.80 Features

Tue, 08 Nov 2011 12:30:00 GMT

The latest General Availability build of Avalanche (and Spirent TestCenter) has been released. I won't cover Spirent TestCenter, but for Avalanche, it's a pretty big release.

Major features

Action List Generator: You can now use a built-in tool to generate action list. For each action, you pick a protocol. The mandatory parameters are directly shown, and you can add the optional ones by the click of a button. I'll cover this in more detail in a future post.
Easy Test: The old "Quick Test" is gone and replaced by the Easy Test. You can quickly create a single- or multi-protocol mix from there, mesh the traffic the way you like and define your load (CPS, Throughput, etc...). If you need more options you can simply convert it to an Advanced Test. So this could be seen as a way to do complex operations in a simpler GUI, or just as a nice introduction to Avalanche.
Adaptive Streaming (Adaptive Bit Rate): This release moves ABR from Alpha to GA. Microsoft Smooth, Apple HLS and Adobe Zeri are supported, for both Video on Demand (VoD) and Live. You can fine-tune the behaviour of the clients, too. For instance, you can choose if the clients should start from the Maximum, Minimum or Median bitrate. You can also disable the up/downshifting (“Constant”), manually configure it (“Normal”), or have the clients behave exactly like real clients (“Smart”).
Run-time Stats Refactor: This is a big one. The Runtime Stats—the statistics you get while the test is running—have been rebuilt from scratch. You can choose which stats you want to see, zoom in, zoom out, save to PNG or clipboard, and filter by port (or even per core). You can even create your own custom charts to meet your specific needs. Last but not least, you can save “Views” which are templates of graphs. For example, you could have one view for IPSec testing (show active SA, Phase 1 and Phase 2 tunnels, etc…), one for VoIP (with MOS scores and goodput for instance), and so on. This allows one to load the view for all future tests without needing to build it from scratch every single time.

Some Other features

RTSP Bitband client now supports Retransmission
Cisco SSL-VPN (AnyConnect) support
Fragmentation is now supported for IPv6 traffic
It’s now possible to manually specify the OID for SNMP polling—no more MIB required
IPv6 PCAP’s can now be imported in SAPEE (Capture & Replay Engine)
It’s now possible to use variables for the file names in FTP
Under the hood—we moved the RAW attacks from the Vulnerability Assessment module to use the FPGA on the 3100B appliance and STC-based modules. This will drastically improve the DDoS capabilities of this module with no impact on the performance.

Note: This post covers major features and is not an all-inclusive list of features in this release.

Experienced a network outage? I feel your pain.

Thu, 20 Oct 2011 15:37:00 GMT

Sort of.

By now, all of us in networking have read about the four-day Blackberry outage or experienced our own personal Crackberry withdrawal. We also know it was a core switch failover event and backup-switch-gone-bad that is shouldering the blame for this. (According to RIM co-CEO Mike Lazaridis, the outage occurred “when a dual-redundant, dual capacity core switch failed and its backup switch failed to activate.”)

I feel for those who were working in the networking department at RIM. I can’t think of a worse position to be in. Wait, actually I can think of a worse position to be in: It would be worse to be the support engineer on the phone for the last 24hours from whichever core switch vendor they used. For that person or persons, it’s probably going to be another long night tonight including a sleeping bag next to the cubicle. Live production networking is a tough gig, too tough for me, which is why I moved out of it and into network testing. In fact, I don’t think I could handle live production networks anymore, especially not with all the outages lately.

In case you missed it with all the Blackberry excitement, Target also had an outage a few weeks ago (see “Target’s New Missoni Collection Brings Down Site”). Last week, Apple, AT&T and Sprint had an issue with a massive wave of iPhone 4S orders. I won’t even get into the Bank of America outage that spanned four days right at the first of the month when paychecks and mortgage payments were due. So for those keeping count in the last two weeks we’ve had the following outages ( some major ) that are threatening to tear apart our faith in the fixed & mobile internet fabric that keeps all our business and sometimes personal lives interconnected:

Research in Motion
Target
Apple, AT&T, Sprint
Bank of America

And that’s just the last two weeks.

Unlike some of the others (big bank, I’m looking at you) which never gave details for their outages, RIM has at least taken the high road from a PR standpoint and offered both a sincere apology and a YouTube video from the CEO.

So to all network engineers that have been experiencing outages: I feel for you. I really do. We can’t undo what happened so my only suggestion is, if you are reading this, please consider doing more stress testing. That may seem a trite statement at this stressful time because I’m sure you did test, but consider more testing with line-rate load generators such as the ones Spirent makes, to truly create stress and cause outages to understand the full impact before they happen for real. Move it up your supply chains. Urge your vendors to recreate this scenario in their labs, and suggest they use Spirent tools (Spirent TestCenter, Spirent Landslide, and Spirent Avalanche) to generate the loads you experienced when the Failover event occurred. We have other tools, network impairment emulators, for instance (Spirent GEM) to help create those environments.

This should henceforth be a mandatory test case that every major core switch vendor should run their equipment through. I’m sure we don’t want to see these kinds of things happening again, and neither do any of the millions of customers. And of course, neither do the poor support engineers that have to deal with this.

Good luck folks, I hope you get it under control soon.

Forced to Choose Between Network Protection and Business Productivity? Think Again.

Wed, 31 Aug 2011 20:19:00 GMT

Enterprise networks are being attacked on multiple fronts—hacking, data breaches, and denial of service attacks—sometimes simultaneously. Are your IT departments adequately prepared to address these multi-dimension security challenges without impacting business productivity? To effectively stop attacks on the network, many IT departments increase their network security, sometimes at the cost of reducing access to legitimate traffic and end users. The key to successful data center and cloud infrastructure security deployment is maximizing network security without compromising performance. However, without the right testing tool, enterprise data protection can come at a cost of lost productivity and business.

Traditional physical security devices are designed to handle the most common attacks with little impact to performance. However, virtual appliances present a new layer of complexity as resources needed by virtual servers are now shared with virtual appliances requiring a balance of security and network performance. Testing in today’s physical-virtual world of cloud computing and virtualized data centers requires a holistic approach that addresses Performance, Availability, Security and Scale across the entire network.

With security testing tools designed specifically for cloud and virtualized environments, enterprises can take the guesswork out of security in a virtual environment. Security test solutions must meet the following criteria:

Provide IT managers the ability measure the impact of increasing security in both physical and virtual environments
Have the ability to emulate thousands to millions of transactions per second so that even the most robust systems can be tested
The test solution needs to be combined with a database of thousands of threats that are updated weekly and can assess impact on the network while under attack and more importantly, determine the impact to the end user

Testing in a virtual environment is similar to a physical one. You must find the capacity threshold for a level of performance you expect and then test it under realistic conditions of an attack to see the performance impact.
If you are at VM World 2011 be sure to stop by Spirent’s booth (#561) to learn more about how the right testing solution can ensure both protection and productivity. Also, check out our latest White Paper: Cyber Security for Virtual and Cloud Environments.

Learn more about how to prepare your network against a cyber-attack.

What do Lady Gaga and top Defense Contractors have in common?

Tue, 23 Aug 2011 18:14:00 GMT

Well, as you might have guessed, they both need to do more network testing!

Two recent news stories highlight this:

First, instead of picking on Amazon (see my earlier blog post on cyber security this year), maybe Lady Gaga's entourage needs to buy some network testing equipment like, yesterday, and start doing some P.A.S.S. ( Performance, Availability, Scalability and Security) testing. Read how technical issues impeded the high-profile promotion of Lady Gaga's latest album here.

Secondly, like so many security bloggers, I am now following up on an earlier post on RSA having their security key information stolen by pointing out that in the last week, both Lockheed Martin and L3 Communications, both top Defense Contractors have been confirmed to be attacked by hackers using the very same stolen RSA key information. However I am glad for two things:

1. Their intrusion detection systems have detected it and
2. They admitted it right away instead of letting the media leak it and make the PR situation even worse.

Now, simply doing testing is not going to solve your performance and security problems. You need to buy the best products, solutions and services for that. I can't stress that enough; it's getting scary out there.

But even once it is all deployed, your confidence in a system will only be as good as your testing. The ONLY way to know you are secure enough or scalable enough is to test your systems for whatever gets thrown at them.

That's where Spirent can help. Our P.A.S.S. test methodologies provide some great test reference architectures to help you get started.

So, people, please reach out and find ways to do better testing. I'd prefer you test with Spirent, but frankly I'd be happy as long as you increase your test coverage in any way, using any approach.

And Lady Gaga: call me.

Spirent at VMworld 2011, Las Vegas

Mon, 22 Aug 2011 21:25:00 GMT

You can’t go to an IT conference these days without hearing the buzzwords du jour, “cloud computing” and “virtualization.”

Despite all the hype and promises of increased productivity and lower costs, implementing and operating virtual environments can expose your organization to risks or even failures, some of which not even the largest organizations in the country are immune.

The ONLY way to know that your system is secure enough or scalable enough is to test your system for anything and everything that might get thrown at it. These days, a new web threat is detected every 4.5 seconds. Now, it’s not a matter of what if you get attacked. Rather, when you get attacked, you’d want to know

How secure is my company network and customer information?
What is the performance of my virtual appliances?
Will the quality of user experience be the same, better or worse?

Come talk to us at VMworld to learn more about the importance of validating your VMware infrastructure.

While you’re at our booth (#561), be sure to enter to win an iPad 2.

VMworld 2011 Social Media Information & Resources

VMworld's Official Twitter Account (@VMworld)
Official Hashtag: #VMworld
VMworld Blog Contributors
VMworld Twitter Contributors

And for real human interaction, see VMworld 2011 (Las Vegas) Gatherings, Tweetups & Parties.

Sophos Mid-Year Threat Report

Mon, 08 Aug 2011 15:37:00 GMT

Sophos, a UK-based security company, released their Mid-Year Top Threats report a few days ago. I found it to be very informative and focused. If you are not subscribed to their Naked Security blog yet, I suggest you do.

The report, as the title implies, focuses on the major IT security threats that their researchers have found. The numbers they give are frightening---although not surprising for anyone keeping an eye on the state of security---Sophos has seen 150,000 malware samples every day and an increase of the web as a vector for threats, like malicious URLs. In fact, a new web threat is detected every 4.5 seconds.

The interesting part is that a huge majority (80%) of these websites sending malicious payloads belong to legitimate companies. How come? Well, they simply have been hacked. If your motivations are not hacktivism (the report has a special section for LulzSec), but instead money, you might want to keep your hacking quiet, so that you can infect even more people. And the hacked servers are not in third-world countries, either. Most of them are in the U.S.:

Now if 19,000 new websites everyday are hacked and modified to infect PCs, what does your company do? Here’s to hoping you not only have inbound anti-virus scanning---Unified Threat Management platforms are now reaching the 10 Gbps mark, and some even more, so you should be covered---but also outbound. You don’t want to be hacking the computers and phones of your visitors, now do you?

Threats from Social Networking

Another interesting part of the report is the section about Social Networking. I try to keep an eye on the threats that you can catch on those, and sometimes it’s bafflingly simple, like an application requesting permanent access to your personal data, and you’ll just allow it because you don’t really read what Facebook tells you. Or, worse, crafted status updates containing Cross-Site Scripting (XSS) code redirecting you to a page that’ll inject a payload in your browser---using a 0-day in the very worst case, hoping you didn’t upgrade your browser in a slightly-less-worst case---and redirect you then again to the original page. Here, too, Facebook seems to be a big playing field for those kinds of attacks. It’s not easy to be the leader.

Email-based Attacks

The next section moves onto email-based attacks. Interestingly, email usage is decreasing at a huge speed – almost 60% less this year than the one before for the younger age period (12-17 years old). Attacks using this vector don’t use the good-old infected attachment so much anymore. It’s best to use HTML here again (allowing XSS or Cross Site Request Forgery, CSRF). That doesn’t mean you should suddenly trust every attachment, mind you. The chart below illustrates the families of e-mail based attacks. Make sure you’re testing your IPS, UTM or AV appliance against those, using the Avalanche Vulnerability Assessment or something similar.

Conclusion

The report contains much more information, and you can never be too careful when it comes to hackers and threats. Some will go after you just for the kicks or to make you look bad. Some will go after you with the intention to steal confidential data. Some will even go after you just to let you know you have vulnerabilities. The point is: you need to be sure you’ve done the best you can to protect your own as well as your customer’s data. It’s very hard to put a price on your reputation, and even harder to recover from a bad reputation. Ask Sony.

I should conclude by saying that all of this being true, there’s an extra layer to take into consideration: Virtualization, or Cloud Computing. I love this technology; I think it’s a major evolution. But it can also potentially lead to disasters. When you put all your eggs in the same basket, you have to make sure there’s no rotten egg in it. Now that there is no more physical separation between servers, it’s even more important to protect them and pick the right tool. Fortunately, the Avalanche Vulnerability Assessment is also available in a Virtual Edition, so you will be able to test your infrastructure, using the same test cases for Virtual and non-Virtual scenarios, therefore testing end-to-end with a unique tool.

The Video Tsunami

Thu, 14 Jul 2011 19:38:00 GMT

I attended Cisco Live yesterday morning in Simi Valley. I was able to watch live on my 63” Samsung Plasma the live broadcast from Las Vegas. It’s better than being there live (no pun intended). A very short commute upstairs, best seat in the house and unlimited supply of coffee. Unfortunately there was no golden ticket under my seat for the Cisco party.

For some time, I have been thinking about the tsunami of data that is going to hit the Internet as video usage continues its logarithmic growth. It was interesting to see that amongst all of the talk concerning Cloud Computing, video was able to be highlighted as one of the hottest applications. As John Chambers said in his keynote presentation,“Video is the new voice.”

The highlight so far was the keynote presentation from Padmasree Warrior, Cisco’s Chief Technology Officer. She gave a great presentation on the cloud and how Cisco’s network is evolving to support these new applications.

Padmasree stressed how important video is going to be as an ever increasing component of network traffic. By 2015 she stated that 70% of network traffic will be video; within the Cisco network, 25% of traffic is already video-based today. In other Cisco presentations, they pointed out that 51% of Internet traffic during 2010 was video and that by 2014 it is expected 91% of Internet traffic will be video.

Using Spirent TestCenter as a powerful network loader, Cisco was keen to demonstrate high performance capabilities with their new platforms. They were able to differentiate between video and other data streams to ensure reliable end to end video quality, while they used the Spirent TestCenter to effectively hammer the network.

Cisco also demonstrated their ability to provide rapid self-healing in the event of network failure. Although Cisco demonstrated that their technology works, this capability is not deployed everywhere.

Moving forward, I think enterprise networks and the Internet will face interesting challenges in maintaining good reliable video quality. This type of application is very different than the short bursty types of traffic that dominate the networks today. What is going to happen as relatively large video packets begin to increasingly dominate network traffic? This is also going to have a serious effect on mobile networks.

Cisco predicts that by 2015, two-thirds of data traffic on mobile networks will be video. A combination of Internet, Video & Cell technology should be interesting! Without doubt, testing network architectures prior to deployment will continue to be the best approach. Video will be a literally in your face problem, and if you want to maintain sticky customers, Quality of Experience will continue to be a priority. Testing and careful network turn up will be a must.

Participate in our poll: What type of QoS will be required to support video in Enterprise networks?

Are Your IPv6 Defenses Ready for World IPv6 Day?

Wed, 08 Jun 2011 15:14:00 GMT

World IPv6 day is nearly upon us and many networks are not ready to repel borders (hackers). As the world becomes more aware of IPv6, more people will try to exploit potential weaknesses in our IPv6 defenses.

My recommendations, even if your network is not connected directly to an IPv6 connection are:

Turn off Teredo (Shipworm support on your PC’s). This is literally one of the IPv6 support protocols of last resort, and can leave your network vulnerable to attack. This worm (Teredo is Latin for Shipworm) and can easily burrow its way into your network.
In your firewalls set them to block protocol 41 (IPv6 encapsulated with IPv4).
Disable ping support on your Firewall.
If possible disable IPv6 on your corporate PC’s, even if you are not actively supporting IPv6. By simply having IPv6 enabled (Windows 7 defaults to enable IPv6) your network is more susceptible to attack. One of the most publicized attacks is the “SLAAC ATTACK”. For more details check this link: http://resources.infosecinstitute.com/slaac-attack/#comment-18675
Remember IPv6 can be tunneled into your network via IPv4. Your firewalls may not be capable of doing a deep packet inspection into the tunnel.

Note: Full name for Shipworm in Latin is: “Teredo Navalis”

Finding Avalanche Versions From Project Files

Thu, 28 Apr 2011 12:00:00 GMT

If you’re a big Avalanche user, you probably one day or another had an old test that you needed to import in a recent version of the Avalanche Commander. To optimize the data structure and therefore improve the overall performance of the tool, sometimes the XML structure gets updated. This doesn’t happen between every version but when it happens there is a chance that these are not backwards compatible.

There is one caveat: any given version will only know about its own XML data structure (obviously), but also the previous XML data structure. So if you are more than one XML structure behind in the project you are attempting to import, the Commander will complain that it can’t import it – because it doesn’t understand the structure. Installing a previous version usually fixes this, and our Support Services have always been helpful in converting older tests to current XML structure.

But did you know you could find out, on your own, the version the project was created with? It’s a little bit of an advanced topic, but here goes.

You need to get the project in its “uncompressed” form (i.e., not in a SPF file). Note that SPF files are ZIP in a ZIP. So simply unzip the SPF twice, and you’ll see the usual XML files. If it’s not in a SPF, the default location for these files is:
Windows 7: C:\Users\\AppData\Roaming\Avalanche\\
Windows XP: C:\Document and Settings\\Application Data\Avalanche\\
Now browse to the “tests” directory. This directory contains the list of tests of the project. Simply open one of them with a XML-aware text editor such as Notepad++. You should see this:

In the first node (“TestSpec”) you will see a key named “Version”. The value of that key (20 in the example above) is the information you are looking for.

“But 20 isn’t an Avalanche version!” you say. And that would be correct. You need a table to convert that internal version number to match it to a General Availability (GA) release. And here’s such a table:

AV4_00_XML_VERSION = "21";
AV3_60_XML_VERSION = "20";
AV3_50_XML_VERSION = "19";
AV3_40_XML_VERSION = "18";
AV3_30_XML_VERSION = "17";
AV3_10_XML_VERSION = "16";
AV2_31_XML_VERSION = "15";
AV2_30_XML_VERSION = "14";
AV2_20_XML_VERSION = "13";
AV2_16_XML_VERSION = "12";
AV7_8_XML_VERSION = "11";
P2_7_7_XML_VERSION = "10";
AV7_5_XML_VERSION = "9";
AV7_0_XML_VERSION = "8";

With this, you can now browse to any project you have and figure out which version was used to create the test you’re looking at.

Cyber Security: It’s Not Just for Breakfast Anymore

Wed, 20 Apr 2011 19:02:00 GMT

Several weeks ago, I received two disturbing emails, one from my airline points program (and no it wasn’t an announcement that I’ve made my 1K for the year yet!) and the other from my bank. You may know the emails I am talking about, you may have received them yourself.

They informed me my information had been stolen from a 3rd party marketing company. Looking at the news the following day I realized this was no hoax – this was real.

Well, as a marketing guy myself, the natural response would be to blog about it! And that was the plan, until I got busy with my “day job” and put it off…and the weeks flew by.

Then out of the blue all sorts of security breaches started making headlines such as:

Wordpress had a DDOS attack take down all three of their data centers
Ventura County (where I live) had their online payment system hacked from the Phillipines!
Skype for Android (which I use!) exposes user files through sloppy programming
Several of the top firewall vendors accused of having security holes in their products
Breaches all over the world including Australia
RSA ( possibly the most trusted name in security) breached by a tightly focused attack, leaving SecureID users (like me!) vulnerable

All this, from just March and April. Thinking what May might bring is a scary thought.

So I wondered to myself: What on earth is going on with network security these days? Or was it just a bad month?

We’ve seen record attendance at Black Hat conferences, our first-ever national Cyber Security Czar appointed, and at Spirent, we’ve done our part by participating in the latest advanced security tests.

C’mon, it’s 2011 people. Are CIO’s simply not aware of the importance of network security? Or are they not getting budget? Or perhaps ( a more darker thought ) are the hackers simply #Winning?

Now, I am not singling out the companies above. All of us, no matter how big or small, weather we manage a data center or just our own personal smartphone, have to accept the Threat Level Orange environment we are operating in, take a deep breath and re-re-re-review our security policies and practices, yet again.

So here are a few tips for consumers and businesses:

If you have a modern smartphone, that has all your files on it, download an AntiVirus for it. These are available for BlackBerry, iPhone, and Android phones. And make sure to put a lock code on it. And if it does get stolen, change your corporate and personal passwords immediately.
If you run a Data Center, make sure all your employees get a refresher on identifying Phishing and other Social Engineering attacks
If you design firewalls, make sure you understand they will be tested!
And, if you work in marketing, scrutinize the security policies of those who handle your mailing lists! You don’t want to be the next Epsilon. Yes, I’m still mad at them.

And that reminds me, I have to get back to work.

Until next time: stay vigilant.

While service providers fret about the end of IPv4 addresses, how should enterprises prepare for IPv6?

Thu, 31 Mar 2011 18:17:00 GMT

IPv6 is suddenly hot news. Running out of Internet addresses has a suitably Armageddon sound to boost news headlines, and we are all sufficiently hooked on the Internet to shiver at the thought of losing it.

You would think that IPv6 had just leapt out of the unknown like some marauding beast, but that’s far from true. We’ve been helping organisations test their support for the new [IPv6] protocol and dual stack functionality since 2004.

I was helping some organisations, including the US Department of Defence, get ready for IPv6 over five years back. The problem was understood, but did not look like an imminent threat so people got a bit bored with it. Now suddenly IPv6 is back in the news and we’re being asked to help again. In the meantime, it seems, economic difficulties put IPv6 on the back burner and it was forgotten, until the recent scare about ”the end of IPv4”.

So it is time to take another cool look at the issue from the enterprise user angle. How serious is it? What is the real threat for the enterprise? What is the best strategy to adopt and why?

Read the entire article, "While service providers fret about the end of IPv4 addresses, how should enterprises prepare for IPv6?" at Business Computing World.

Using Avalanche to test MAC Address Capacity of Stateful Network Devices

Fri, 11 Mar 2011 16:00:00 GMT

Using Avalanche to test MAC address capacity of stateful network devices

Using a lot of MAC addresses in an Avalanche test is something we see on a regular basis. This can be used for Network Access Control, RADIUS, 802.1x, or simply to stress the MAC table of a switch. Avalanche can do that; in two different ways, even. One way is simple but gives you less flexibility than the second, more complex solution that gives you more control.

TL; DR: Here's a 3.60 SPF file that contains two tests, one for each method.

First method

Let's take a look at the simple way first. Using this method, the first two bytes of each SimUser's MAC address will increment with each new SimUser born - and then cycle back to the start. The remainder of the MAC will come from the SimUser's IP address. To set this, simply go to Client(Server)/Subnet and check the "MAC" box at the top of the screen:

This will enable the MAC options (the "MAC" checkbox and the two "Byte" fields). Checking the box will make the two fields editable. By default the fields will be filled with "12" and "22", but you are of course free to change that to whatever you need this to be (remember, MAC addresses use hexadecimal values). It goes without saying that Multicast MAC addresses will not be accepted as this is only for unicast testing.

If you want to make sure that, during the test, no two MAC are the same, you can check the "Randomize" box. What this does is that each time a SimUser is born, instead of taking the next IP address in the range a random one will be picked from the pool. But since the MAC address bytes increment in a linear fashion, this creates a randomness that in most cases ensures that no two MAC addresses are the same in the test. And of course, the larger the subnet, the lower the chances to use the same MAC twice.

The picture below illustrates that kind of large, random subnet (that would probably kill most switches, too!):

One last important note about this feature: if you uncheck the "MAC" checkbox sitting on top of the list of subnets, it does not disable the feature. This checkbox merely hides the settings but, I'll say it again, does not disable them.

Second method

The second method is slightly more complex - depending on how you look at it - but allows you to specify the whole MAC (instead of only setting the last four bytes from the SimUser's IP address). This can be useful in test cases where specific MAC must be used (e.g.: only MAC 00:00:00:00:00:00 can access the subnet 192.168.43.0/24).

To test in this way, the work doesn't take place in the Subnets but in the Action List. Advanced users will immediately know what this mean: you cannot influence the servers' MAC using this method (but you can use the first one for the server-side while using the second method for the client side).

The example below pulls the MAC addresses from a Form Database (more on this below) named "myMacs", will apply it to the current SimUser, which will then do a HTTP GET, then die:

Action List Commands:

ASSIGN VARIABLE <myMacVar myMacs.$1>
ETHSETTING MACADDR = <apply myMacVar>
1 GET http://192.168.0.10/

The only thing left to do is create the previously mentioned Forms Database. Forms DBs are very handy comma-separated flat files you can pull data from. There are not set limits to the number of columns or rows. In our example, however, we'll just keep it simple with one column and multiple rows:

You will notice three checkboxes. "Auto Increment" does just what it says: for each new SimUser, the next row will be used. When the last row is reached, Avalanche cycles back to the first row.

"Skip First Row" can be pretty handy. FormsDB are pretty much CSV files. And in many cases, the first row in a CSV file is used to describe the name of that column ("email,firstName,lastName,areaCode", etc..). This feature exists to allow user to keep these column names.

Last but not least, the "URL Encode Form" means that Avalanche will URL Encode the content of the FormsDB for you. This is usually needed when testing against HTTP servers or HTTP-aware devices, and is therefore not needed for dynamic MAC addressing.

Conclusion

There are multiple ways of doing same thing, the question is do you want it to be fast and easy or do you want high control. That's about it regarding these two separate, yet similar, features. If you have any more questions, feel free to contact me at arnaud.castaner@spirent.com.

Can We Trust the Cloud?

Fri, 04 Mar 2011 18:57:00 GMT

Can we trust the cloud?

Cloud security is receiving a lot of attention nowadays, because fears about data security in the cloud remain the biggest hurdle to mass acceptance. Cloud computing removes many of the familiar physical control points for protecting sensitive data – from personnel screening to PIN and smartcard access to the datacentre. Instead of visible, tangible security systems, we must put our trust in specialist IT skills, and the people with those skills, to safeguard data in the cloud. For the non-specialist, this is scary.

A natural first approach to cloud security is to model those familiar perimeter defence systems, to focus on Internet traffic to and from the cloud – if the bad guys can’t get in, and then the cloud must be safe. That’s a very necessary security component, but it is not enough. At worst, it can lead to a false sense of security, because it does not address attacks that originate within the cloud itself – the Enemy Within.

Confidence on the rise

Confidence is infectious. If the authorities in 1886 had known how many people would die on the roads over the next century, they might have taken steps to silence Karl Benz and his patent for a ”vehicle with gas engine propulsion”. Yet we now drive out on the killing fields with little thought for the risks.

You can argue that fears about cloud security are ’irrational’ and you can equally argue that they are very rational. And when the tide turns, and cloud computing gains mass acceptance, some will say it is because the herd mind-set has shifted, and others will say the cloud has been ’proven’ safe.

Gartner conducted a survey between September and December last year that showed growing confidence amongst two thousand CIOs across fifty nations. Among its findings were:

Cloud computing and virtualization are their two highest priorities for 2011
They expect these services to allow up to 50% of current budgets to be shifted from operations towards new business strategies
Only 3% currently run the majority of their operations over cloud or SaaS, but this should rise to 43% over the next four years.
That last result indicates the extent CIOs do not yet trust the cloud but are prepared to bet on it being made sufficiently secure in the near future. Is their confidence justified?

First, the bad news

For the end user, security in the cloud simply means ”is my data safe?”. For the service provider there is the question of data security but also one of maintaining SLAs.

It is no good guaranteeing rock solid data protection if the security measures are so cumbersome that they degrade the service provided. And if the response to cyber attack slows down the system so much that the service is unusable, damage has been done – however well the data was protected.

So all security measures must stand or fall on the results of stringent testing under realistic operating conditions. The tests must confirm not only that data is protected, and not altered, corrupted or leaked, but also that service performance is maintained under all sorts of operating and attack conditions – for ultimately user perception is the thing that counts.

But there is a fundamental problem in testing any virtual system, in that it is not tied to specific hardware. The processing for a virtual switch or virtual server is likely to be allocated dynamically to make optimal use of available resources. Test it now and it may pass every test, but test it again and the same virtual device may be running on different hardware and there could be a different response to unexpected stress conditions. Malicious traffic could even be forcing the system to dynamically assign all CPU power to the attack process.

This is what worries the customer – is it really possible to apply definitive testing to something as formless as a virtual system? And knowing what we do about the determination and skill of cyber-criminals, how can we secure a system as amorphous, dynamic and complex as the cloud?

This is why cloud security has focused on securing access to the virtual system. The concept of a secure perimeter is easier to grasp, because it is analogous to physical security keeping the wrong people out of a building, and it lends itself to traditional solutions such as firewalls and intruder prevention systems (IPS) monitoring access points to the virtual system.

In the case of a physical datacentre, people also understand that access security alone is not enough. Because an authorised operator can enter the building with good intentions but still be subject to human error – maybe connecting their laptop without realising it has been infected with a virus. So you need to inspect and clean traffic within the network as well as what comes into the network from outside.

When it comes to the cloud, even if every precaution is taken to stop such human errors, there are other risks. Take the proliferation of Internet ’app stores’, where independent software developers are given space in the cloud to set up shop and sell their software. How can we be sure that even a trusted developer might not upload malicious software into the cloud?

In a physical datacentre, the solution is to disinfect all traffic within the system as well as what comes in and out. Hence the need for IPS monitoring and cleaning traffic at strategic points within the datacentre, in addition to firewall protection at the perimeter. In a virtual system this is a far greater problem, because the system is never static. So where do you install the IPS and how can you monitor network traffic that is constantly being re-routed?

Bear in mind also that virtualization adds another level of vulnerability to the data centre – intra-server vulnerability. Traditional data centres have inter-server and infrastructure vulnerabilities, such as the possibility of performance and security weaknesses internally between servers, externally at the gateway, and in the end-to-end network. Virtualization not only intensifies these potential threats, it adds the risk of intra-server threats, between virtual machines inside a single physical server.

What is needed is some way of monitoring and disinfecting traffic in a virtual network as reliably as a physical IPS in a physical network. And unless there is some way of testing the protection in the virtual environment, we can never be sure how safe it is.

Now for the good news

The question is: is it really possible to apply definitive testing to something as formless as a virtual system? The answer to this question came in a report published last year by Broadband Testing, Secure Virtual Data Centre Testing (September 2010).

Broadband Testing set out to determine whether it is possible to secure a virtual environment, knowing that their first problem was to create a rigorous and repeatable test process. Using cloud computing testing solutions with performance, availability, security and scalability (PASS) methodology, Broadband Testing were able to monitor and test internal and external-to-internal traffic under normal operating and extreme conditions plus a wide range of attack scenarios.

All the threats in the HP TippingPoint signature base were successfully blocked, and the only ones that passed were those that had not yet been added to the then-current database.

Such was the success of their test process, that Steve Broadhead, founder and director, Broadband Testing was able to say in conclusion: “Can we trust the cloud? The answer now is ‘yes’. Virtual Security works in theory but, until there was a way to test it thoroughly under realistic conditions, solution vendors have had a hard time convincing their customers. With the use of combined physical and virtual test machines, the testing proved not only highly rigorous, but also quite simple to operate.”

Test criteria for the cloud

There are two aspects to testing applications in a virtual environment. Firstly functional testing, to make sure the installed application works and delivers the service it was designed to provide, and then volume testing under load.

The first relates closely to the design of the virtual system – although more complex, the virtual server is designed to model a hardware server and any failures in the design should become apparent early on. Later functional testing of new deployments is just a wise precaution in that case.

Load testing is an altogether different matter, because it concerns the impact of unpredictable traffic conditions on a known system. In a virtual system, and even more so in the cloud, there can be unusual surges of traffic leading to unexpected consequences.

Applications that perform faultlessly for ten or a hundred users may not work so well for a hundred thousand users – quite apart from other outside factors and attacks that can heavily impact Internet and overall performance. So the service provider cannot offer any realistic service level agreement to the clients without testing each application under volume loading and simulated realistic traffic conditions.

Large data centres and the cloud pose particular problems because of their sheer scale. The Spirent test platforms used by Broadband Testing allow for this in rack systems supporting large numbers of test cards scalable to several terabits per rack.

These modular devices can be adapted to any number of test scenarios, and the one used by Broadband Testing had the option of a software module that specifically addresses the challenge of testing the performance, availability, security and scalability of virtualized network appliances as well as cloud-based applications across public, private and hybrid cloud environments.

This combination of a physical test device plus virtual test software was shown to provide exceptional visibility into the entire data centre infrastructure, where as many as 64 virtual servers, including a virtual switch with as many virtual ports, may reside on a single physical server and switch access port. With this combination, it is not only possible to test application performance holistically under realistic loads and stress conditions, but also to determine precisely what component – virtual or physical – is impacting performance.

To create realistic test conditions, the virtual software was used in conjunction with devices designed to generate massive volumes of realistic simulated traffic. The simulation replicates real world traffic conditions including fault conditions and realistic user behaviour, while maintaining over one million open connections from distinct IP addresses.

Also under test was the quality of user experience at all times. Here latency becomes a critical parameter. Even minute levels of latency can become an issue across a virtual server, while the very presence of monitoring devices produces delays that must be compensated for. So it was important that Broadband Testing chose a test platform that provided automatic latency compensation, adjusting according to the interface technology and speed.

The test scenario described so far covers the security testing of the infrastructure and the inter server security. Mentioned earlier was a further security issue in today’s datacentres – the intra server security between virtual machines running on the same server. Such traffic stays within that server and never hits the actual wire, so testing that part of the secured cloud requires tests run between two virtualized testers. Spirent Avalanche allows not only the testing between physical and virtual machines but also testing virtual to virtual on the same server in the cloud.

Testing the virtual system proved not only possible, but surprisingly straightforward, because the testing methodology is no different from that used for regular security testing, and the same test cases can be used.

Conclusion

To make a virtual environment secure, we need to address what is happening inside it, not just what is coming in or out. Two key conclusions emerged from the BroadBand Testing report. First it is reassuring to know that it is indeed possible to secure the virtual environment using the TippingPoint SVF solution.

More significantly in the long run, the fact that rigorous and consistent system tests can be carried out in a virtual environment means that from now on we can confidently test the security of virtual networks and the cloud. With Spirent equipment and methodology we can test the cloud and all aspects of the infrastructure, inter-server and intra-server security.

The factors that cause people to doubt the cloud – its dynamism, shapelessness and lack of tangible boundaries – have been put to rigorous test and have been shown to work. So – yes, we can trust the cloud.

Author profile: Marc Meulensteen

As EMEA Business Development Manager for Layer 4-7 Application and Security testing, Marc Meulensteen is responsible for new business for Spirent L4-7 Test Solution in the European and Middle East and African Markets. Marc joined Spirent in 2001 as System Engineer for the BeNeLux market. In this role Marc was responsible for technical Sales support with large accounts like Alcatel-Lucent, Juniper and Cisco in Belgium and the Netherlands. In this time Marc gained his expertise in testing and more specific in the newer technologies. In 2006 he changed to the current role of Business Development Manager. Marc is involved of introduction of new testing methodologies and new testing areas like video streaming and video quality testing and more recently in deploying there test technologies in the DataCenter and Cloud.

Preparing for a Virtual Demo

Wed, 02 Mar 2011 23:00:00 GMT

I just spent a week getting ready for this advanced testing seminar. It’s been a whirlwind of activity, mostly involved in getting my lab ready for the demo. Let me tackle each of these in turn.

First, I upgraded to VmWare 4.1. For anyone who has been through this, you know what I am talking about here. Suffice it to say, quite a bit of effort and a bit of trial by fire for me, but nothing I couldn’t handle.

Second, I learned how to install the Cisco Nexus 1000v. Cisco did do a great job on posting lots of YouTube videos and help documents on their website, but unfortunately there were actually too many and it was a bit confusing to find one set of directions that actually covered the whole install.

Third, I set up FanFare iTest (FanFare, as you know, was recently acquired by Spirent) in my lab, and that at least was relatively easy. We bought iTest for test automation and it delivers with a nice graphical interface and even though the last time I used it was 4 years ago, it was easy to pick up again. I was able to setup an automated test that moves a test case from the physical world to the virtual world and back again. Now that my test case is in iTest, I can run it again and again with just a click. Definitely worth the effort.

I'm now looking forward to the Cisco Innovating Test show tomorrow. I think we’ll have a great demo to display that really drives home the value of virtualization as a way to speedily share test cases without having to duplicate hardware test racks. Definitely a great value proposition, something I’ll be very happy to show off.

Learn more about Spirent TestCenter Virtual and the iTest Community.

The end of the Internet?

Wed, 26 Jan 2011 12:00:00 GMT

Were the Mayans correct in predicting the end of the world in 2012? Or did they get some mixed signals on the state of the Internet and the availability of IPv4 addresses?

For many years the pundits have been preaching imminent disaster as we near the exhaustion of the IPv4 address pool, but somehow we kept dodging the bullet. This year however there is no doubt, the end is nigh.

The Internet Assigned Numbers Authority (IANA) is down to less than 2% of the available IPv4 address pool (at the time of writing according to Latif Ladid, president of the IPv6 Forum, the IANA ran out of addresses the week of Jan 17th). It is true that the Regional Internet Registries (RIRs) may still have enough to keep them going until 2012, but it is going to get increasingly difficult to get large contiguous blocks of addresses. It has also been reported that the RIRs could run out as early as November 2011! At the last Reseaux IP Europeens Network Coordination Center (RIPE NCC) meeting held in November of 2010, they announced that this could well be the last meeting before they run out of free IPv4 addresses.

So what are we going to do? We could continue with the use of Network Address Translation (NAT), but this is not a very elegant solution even though it has been used for many years. Basically it interferes with the data between the source and destination. It can cause problems for IPSec and make it more difficult for lawful intercept, as well as creating speed bumps for the Internet superhighway, degrading performance and QoE (Quality of Experience) for the users.

IDC reported in 2009 that they expect more then 15,000,000,000—yes, that is 15 BILLION—devices to be connected to the Internet. A new term used to describe this phenomenon is the “embedded Internet.” In other words, this growth of connected devices will not be driven so much by computers, but by the new generation of cell phones that use native IP. New consumer and entertainment devices, such as tablets and Internet enabled TVs, will also help drive this new connectivity, along with more industrial and medical automation.

Network Address Translation (NAT) or Carrier Grade Nat (CGN) is not a real solution; it is only a band aide. Ultimately, IPv6 will surpass IPv4, even though this will take many years. In the meantime, the network will need to evolve. New equipment should be deployed with IPv6, while current systems need to be updated to dual stack. With dual stack, we can maintain the ability to connect with IPv4 and still connect to new devices being deployed with IPv6.

Techniques such as Dual Stack Lite (DS-Lite), although not perfect, are for now possibly our best interim hope. DS-Lite enables an ISP to provide an IPv6 service, this allows for the transport of IPv4 over IPv6 tunnels. I have not seen any large scale deployment, and there may be some large scaling issues, but in my opinion, this could be one of the best comprises for now.

Learn more about DS-Lite and watch the following video.

PASS: A Test Methodology for Cloud Services

Wed, 28 Apr 2010 12:30:00 GMT

While the terminology surrounding cloud services may be a bit cloudy itself, the goal is clear—to create the dynamic data center and infinite scalability, which can reduce the labor-intensive, tactical aspects of providing IT services, freeing managers to pursue innovation and strategic planning. This makes cloud computing testing an imperative.

An IDC survey shows that the top concerns of IT enterprise organizations are performance, security and availability. Cloud services address performance and availability and are offered at different tiers of service:

Infrastructure as a service (IaaS): Hardware (data center space, network equipment and servers), on which the customer installs its own operating systems, applications and data.
Platform as a service (PaaS): Infrastructure with operating system, protocol stacks and development tools installed, on which the customer develops software or hosts servers.
Software as a service (SaaS): Platform with application software installed, on which the customer loads data.

Gartner sums up cloud computing as delivering scalability as a service. Real-time infrastructure (RTI) is a basic building block of the dynamic data center, the fundamental level of IT element automation. It maps the demand for shared services and resources to the supply, making possible the dynamic provisioning and scaling of resources for services to meet service level agreements efficiently and economically. At the higher levels of platform and software services, virtualization can be used to deliver scalability and elasticity by dynamically launching and terminating virtual machine (VM) instances based on varying demand.

Delivering cloud services involves a complex and possibly confounding amalgam of processes, technologies, platforms and devices, many of which are in the early stages of the Hype Cycle. Of course, all systems must be assessed and validated, even those built on mature technologies, and even more so systems that incorporate technology at the early stages of adoption.

Existing technologies have reached maturity and achieved the Plateau of Productivity through the development of test methodologies that assessed and validated specific implementations. The evolving elements of cloud services will do the same only through rigorous testing with new methodologies designed specifically to reveal their weaknesses, and their strengths.

PASS is a test methodology for cloud computing testing that addresses the Performance, Availability, Security and Scalability of cloud services.

Performance: Optimize cloud services and infrastructure to maximize user experience
Availability: Ensure high availability in daily operation and under disaster conditions
Security: Eliminate vulnerability and exposure between tenants in the cloud
Scalability: Validate responsiveness as demand varies according to tenant needs

PASS incorporates a wide range of use cases to assess the application components and network elements of a private, public or hybrid cloud. PASS achieves unprecedented granularity through use cases that subject the cloud service to inspection at the infrastructure level (end to end), the inter-server level (applications, services, VMs on different physical servers), and intra-server level (between VMs within a physical server).

For example, at the service level, PASS assesses user quality of experience under realistic and peak subscriber loads, scaling to multi-tenant loads, under the stress of failover scenarios, and while subjected to threats mixed with encrypted traffic.

At the infrastructure level, PASS assesses quality of service performance for the network elements (high-density 10/40/100 Gb Ethernet, converged FC and Ethernet), the application elements (firewalls, intrusion prevention systems, WAN accelerators, proxy servers, and others), and the virtual elements (virtual switches and virtual appliances), end-to-end.

The development of PASS is a milestone in the journey from innovation to adoption, and a significant enabler for those looking to establish world-class cloud services.

Testing the Cloud From the Inside Out

Wed, 28 Apr 2010 12:30:00 GMT

Cloud computing offers business benefits without the risk of a degraded user experience. The challenge is determining whether or not virtualized data center environments deliver on that promise.

A virtual machine (VM) performs at the most fundamental level of content generation, which poses a serious challenge for those who need to understand real performance numbers, which means getting a per-VM view of performance versus utilization.

Traditional methods of validating inter-server performance aren’t granular enough for testing virtualization. When working with a dedicated server, there is a one-to-one relationship between port and machine. It’s a fairly straightforward process to characterize performance and isolate factors that affect it.

Cloud computing testing and in particular testing a virtualized environment is not as straightforward. Now there is a one-to-many relationship between port and machine – there are multiple VMs behind a single port, making it impossible to isolate and measure the performance of each VM.

In addition, there is no way to measure performance between applications running on separate VMs within a single physical server. This is intra-server traffic, which never leaves the server or passes through the port, so the test system will never see it.

To get definitive per-VM detailed results and composite per-server results, we need visibility into the performance of each VM. We need a virtual test system inside the virtual server.

A virtual server is implemented in a virtual machine, but to the user it looks and behaves exactly as if it were a dedicated, physical server. In the same way, a virtual test system is a software-based test system implemented in a virtual machine, but to the network devices under test, and to the test engineer, it looks and behaves exactly as if it were a hardware test system.

A virtual tester for cloud computing testing makes it possible to test virtualization at all the levels it has impact: intra-server, inter-server and infrastructure.

Intra-server: When multiple applications are running on a single physical server, an application in one VM may send a request to a server in another VM on the same physical server. For example, a CRM application might request data from an integrated ERP system running on a different VM on the same server. A virtual tester in a VM can emulate a request from the CRM application and measure the intra-server response time from the ERP application.

Inter-server: In a typical tiered architecture, an application-layer server sends requests to a data-layer repository. For example, to process the request from the CRM application, the ERP application makes a call to a different physical server supporting the data layer. A virtual tester on the ERP server can emulate a request to the data tier and measure the inter-server response time from the data repository.

Infrastructure: With virtualization and dozens of VMs per physical server, the amount of traffic one box can generate increases dramatically, easily filling a 10 Gigabit Ethernet link. A standard rack holds many physical servers and dozens of blade servers. Some data centers house hundreds or thousands of racks. With the number of server instances per rack possibly increasing by an order of magnitude, not to mention the impact of hypervisor tools, the increase in network traffic can be significant. The implications for the data center infrastructure are serious.

Is Your Network Ready for Carrier Ethernet Interconnects

Mon, 12 Apr 2010 13:00:00 GMT

If you don’t already know, network convergence is central to what’s driving complexity in the communication networks of today. And despite technology, development and deployment of devices and networks, there are significant challenges facing service providers.

How can service offerings be expanded to support enterprise customers? Is the network ready to deliver secure and reliable enterprise applications? Are they prepared for challenges of interconnecting to other large and complex multi-play networks?

The exponential growth in mobile devices and applications both within the consumer and business segments is opening up a large and lucrative market for local and regional service providers who are looking to expand their footprint. However, to be able to offer effective and manageable services, interconnectivity between operators is critical.

While interconnectivity concerns may weigh heavy on the minds of many, there are coordinated efforts underway to ensure the industry realizes the full potential and the opportunities associated with delivering the widespread availability of Ethernet services.

Fortunately, by implementing interconnect standards such as those specified by the Metro Ethernet Forum (MEF), operators are better prepared to manage and deliver wholesale end-to-end Carrier Ethernet service while maintaining global service level agreements (SLAs). Through these standards, the needs of tomorrow can start to be deployed today.

So what is the next step? In one word “Test”. System vendors and service providers will be working to implement and deploy these standards but the only way you are sure it works is to test it. Companies cannot afford to learn about issues from their customers when there is stiff competition.

So is Carrier Ethernet interconnectivity ready for prime time? EANTC is conducting a series of public tests to validate Carrier Ethernet service interoperability across eight European service providers. The demo service provider networks are interconnected and then tested to check ENNI and CoS conformance. This helps to demonstrate that these service providers are taking that important next step to be successful.

Carsten Rossenhövel, managing director of EANTC AG, stresses that this Global Interconnect Demonstration is fundamental in advancing the successful implementation of carrier Ethernet interconnects in service provider networks. The path to achieving interconnection goals includes validating the performance of carrier Ethernet services delivery by utilizing solutions designed specifically for testing MEF specifications.

If you want to learn more about what you need to do to leverage the MEF ENNI into your network interconnect strategy register for our webinar titled “Globally Interconnected Carrier Ethernet Services - Are We There Yet?”. This one hour webinar will discuss the test methodology used to validate the performance of Carrier Ethernet Interconnects in your networks and explain what was done at the demonstration during Light Reading Ethernet Expo 2010.

Leveraging Your Automation Infrastructure

Tue, 16 Mar 2010 13:00:00 GMT

Maybe it’s possible to maintain a competitive feature set and release schedule without test automation, but I don’t see how. These days, test case automation is table stakes for survival and full test environment automation streamlines development to give you a competitive edge.

Here are a few suggestions for those looking to take advantage of the CapEx and OpEx savings that full automation can give you.

What: If your test engineers aren’t already Tcl experts, look for a system that allows automation without coding.
Why: Save on startup time, save on test-case development, easy to debug.

What: At the basic level of test case automation, look for an API that is integrated with the test application GUI.
Why: The flexibility of GUI-to-script and script-to-GUI removes the artificial barrier between coders and GUI users. The same test can be run interactively or automated.

What: Look for a system that helps you manage test the cycle through test suites, test scheduling and a results database.
Why: Get 24/7 testing without working nights and weekends. Spend your time analyzing results instead of running tests.

What: Look for a system that can automate test lab configuration.
Why: Manual re-cabling for a new test configuration is time-consuming, error-prone, and requires a body on-site. Imagine the convenience and time-savings you would enjoy if it could be done remotely, from your cube or from your off-shore site.

How to avoid painting yourself into a corner

In discussions about test automation, the issue of portability is a big issue for engineers and managers. Every test equipment vendor has its own API. Many labs have test gear from multiple vendors, which means learning multiple APIs. Others are concerned that an investment in creating an automation infrastructure will lock them into a specific vendor simply because of the effort and cost required to migrate the scripts to another vendor’s API.

The best way to address these concerns is to look for a vendor that is a member of the Network Test Automation Forum (NTAF). The NTAF was recently created by a group of test equipment vendors, network equipment manufacturers and service providers to promote the interoperability of commercial testing tools and test infrastructure for the data communications and telecommunications industry.

This means you can get the advantages of automation without having to worry about painting yourself into a corner with any one vendor’s API.

There’s really no reason to avoid automating your lab, and every reason to start now if you haven’t already. Your budget, your management and your customers will thank you.

The Death of Best-Effort QoS Testing

Tue, 02 Mar 2010 13:00:00 GMT

Well, it’s official. Testing exclusively with best-effort traffic is dead. Today, even the simplest network implements Quality of Service (QoS). Testing with exclusively best-effort traffic used to be acceptable for numerous reasons: Simplification of testing, the experience of the end user, and the premise that more bandwidth solves all problems.

As with all industries, both the people who make the products (network equipment manufacturers) and the people who consume those products (service providers and enterprise) are becoming more sophisticated. As they have correctly deduced, bandwidth cannot solve all problems. In the end game, the winner is the one that can roll out network services first with consistently high Quality of Experience (QoE). In other words, what will prevent the end customer from calling tech support?

Actually, this level of customer satisfaction is not possible with only best-effort testing. In the modern network, congestion at each hop is a real phenomenon. Just as security exploits and new ultra-low latency and bursty protocols such as P2P, Storage and HD Video over IP, each of these will require a very specific network ecosystem to succeed, which is only achievable with QoS. In a nutshell, every test must be performed with QoS considerations in mind.

Testing Video over Carrier LDP/RSVP-TE MPLS

Tue, 09 Feb 2010 14:27:00 GMT

One of the great features of Spirent TestCenter is its ability to directly test services over structured Ethernet. This is critical to carriers because their business model is becoming increasingly dependent upon revenue from IP video.

With Spirent TestCenter, you have the ability to completely isolate the carrier Ethernet DUT, regardless of its role. Say the device under test is an MPLS P router. With Spirent TestCenter topology emulation technology, you can model routers behind routers as true objects.

Once you build out the IGP, you can model MPLS, adding depth and width to the MPLS tunnels. Then, within the LSPs, you can source Unicast and multicast MPEG2-TS video streams over unicast, or optionally over multicast, with Spirent TestCenter stateful PIM-SM and SSM support.

How is this unique to the industry? Because of the pure object-oriented network-module features of Spirent TestCenter, structures are layered upon each other as opposed to a relational model, where entities are next to each other in tables. When you test a lower-layer feature, such as a BGP route flap event, the event instantaneously traverses up and down the relationship tree to highlight the impact.

With relational models, you would have to reconfigure the IGP and then manually propagate the changes to each table in the tester. Many events align within microseconds and thus become untestable in relational models. In the case of video, the BGP convergence loss may or may not affect user experience.

Say a flap event loses 10 datagrams. Depending upon the encoding of the video (SD or HD, MPEG2-TS with more or fewer I-Frames) the event may or may not affect the MPEG GPOB. If I-Frames are lost within the data stream, you will see MPEG artifacts on screen (blockiness, blur, etc). If P-Frames are lost, this is fairly recoverable and you may not see any effect.

The point is that unless the test tool has the ability to relate objects with real-time processing, you will not be able to test many real-world errors with video. Properly testing IPTV is critical to the success of the business.

The 21st Century Workplace

Tue, 26 Jan 2010 12:00:00 GMT

Is your network ready for telecommuting?

Institutional resistance to working remotely (aka telecommuting, telework) continues to crumble. While the H1N1 virus may force the issue, it will merely be the final drop that breaches the dam. There are many drivers and corresponding benefits for the organization that adopts telecommuting.

Energy savings. Going green saves resources (and dollars) for business and personal budgets. The company can light, heat and cool fewer square feet and power fewer computers and monitors. The workers spend less on gas for the commute. They might even be able to downsize their personal fleet, saving not only on the cost of the extra car but the associated insurance, taxes and maintenance costs.
Minimizing disruptions to business. The long list of things that can disrupt your business by interfering with a commute range from the infrequent (natural disaster, pandemic, terrorist attack) to the regular occurrence (winter weather, traffic). Such disruptions are now completely avoidable. There is no longer any reason why the inability to commute means the inability to work.
Increased productivity. No more extended water-cooler discussions or co-workers camping out in a neighboring cube to chat. Commute time can be replaced by work time, or time to take care of personal business that would otherwise mean time out of the office. Studies indicate that many employees tend to work more, even after hours, when the office is a few feet away instead of several miles away.
Stealth pay raise. Supporting remote work effectively gives employees a pay raise (by eliminating significant personal expenses) without the cost of increasing salaries. In addition, the quality-of-life dividend can result in greater job satisfaction and company loyalty.

However, issues must be addressed when making the changes necessary to support the workplace of the 21st century.

Infrastructure. With a few decades of development behind it, the technology may be the easiest part of the puzzle. However, verifying that it is configured properly and will work reliably (not only during normal business but also under peak usage such as during severe weather) is another matter. Proper testing by a third-party is the best way to assure that your system for maintaining productivity during a disruption doesn’t itself become disrupted (see H1N1).
Management. Perhaps more challenging than the technology, and the reason many organizations are slow to adopt, are the human resource issues associated with managing a remote workforce. However, the reality is that you don’t guarantee productivity simply by forcing people to drive to a central location and sit in close proximity while working on disparate tasks. There are many publications and websites, such as www.telework.gov, that provide guidance for managing this aspect of the transition.

Organizations that make the strategic decision to include telework in their culture are positioned to thrive in the 21st-century marketplace. Organizations with no telecommuting infrastructure or a system that is either unproven or not designed to scale will increasingly find themselves at a competitive disadvantage.

Best practices indicate that the time to finalize the organization, communication, and implementation of the technical and management infrastructure for telework is before a disruptive event that will stress the performance. As always, thorough testing is the key to success (see H1N1).

Testing 10G Data Center Switches: Scaling 10 Times Higher

Mon, 18 Jan 2010 17:33:00 GMT

If there’s one overarching conclusion I’ve drawn from three months of testing 10-gigabit top-of-rack data center switches, it’s that “switch” and “data center switch” are very different beasts.

Understanding the latter means testing new features like virtualization support and storage/data network convergence, while also driving unicast and multicast scalability benchmarking to new heights.

In a project recently published in Network World, we compared switches from six vendors, each with at least 24 10-gigabit Ethernet ports. We compared products in 10 areas: features; usability; power consumption; MAC address capacity; forward pressure; multicast group capacity; multicast group join/leave delay; link aggregation hashing fairness; and, of course, basic unicast and multicast performance.

Performance testing remains as important as ever, if not more so, when it comes to data center switching. That’s an important point: Buyers are interested in these new features, to be sure, but only in addition to switches’ long-time role as fast packet pushers.

In other words, the same industry-standard methods of benchmarking switching and routing performance (as defined in RFCs 2544, 2889 and 3918) remain vitally important in the context of data center switching. In fact, line-rate performance and low latency and jitter are even more important for many data center applications than for general-purpose enterprise networking.

Data center switches tend to have much longer features lists than their wiring-closet counterparts. We paid special attention to three areas. First is redundancy protocols. Some data center switches offer new methods to connect multiple servers and/or switches. Some methods even eliminate slower redundancy protocols, such as spanning tree. Others offer “active/active” connectivity across multiple links until a failure occurs, boosting bandwidth in a way that “active/standby” protocols such as spanning tree cannot. While new protocols are intriguing, it’s a good idea to test their resiliency and benchmark failover times before deploying them in your network.

Second, some switches allow the convergence of previously separate storage and data networks using technologies such as Fibre Channel or Fibre Channel over Ethernet (FCoE) on the same switch. The IEEE has defined several new protocols to accommodate the FCoE’s stringent delay and loss requirements. We didn’t test these protocols this time, since only one switch tested supported all these new mechanisms, but look forward to comparing data/storage performance in upcoming tests.

Finally, data center switches support virtualization in a variety of ways. Since virtual machines (VMs) often move between physical hosts in the data center, some switches offer the ability to have the VMs’ access control policies move with them. Other switches can carve up physical interfaces to appear as multiple logical links to different sets of VMs. And some support end-to-end management of physical and virtual switches, offering the same set of capabilities for both.

I’m looking forward to future tests comparing Fibre Channel and FCoE performance as well as even larger-scale tests of large modular data center systems. We’re still in the early days of data center switch testing, and things will only get bigger from here.

Newman is president of Network Test, an independent test lab and engineering services consultancy. He can be reached at dnewman@networktest.com.

Has Your Network Had Its Flu Shot?

Thu, 17 Dec 2009 17:43:00 GMT

As of this writing, the World Health Organization H1N1 virus pandemic alert is still at phase 6, the highest level, and has been for six months. The WHO defines the top two phases as follows:

Phase 5 is characterized by human-to-human spread of the virus into at least two countries in one WHO region. While most countries will not be affected at this stage, the declaration of Phase 5 is a strong signal that a pandemic is imminent and that the time to finalize the organization, communication, and implementation of the planned mitigation measures is short.

Phase 6, the pandemic phase, is characterized by community level outbreaks in at least one other country in a different WHO region in addition to the criteria defined in Phase 5. Designation of this phase will indicate that a global pandemic is under way.

The US Center for Disease Control (CDC) reports widespread activity (the highest level) in 46 states and regional activity (second highest level) in the four remaining states for the week ending 11/07/2009.

The vaccine continues to remain in short supply, with specific groups targeted for the early doses, such as pregnant women, children, and healthcare and emergency personnel.

The CDC estimates that up to 40% of employees could be absent from work during a severe pandemic, either sick themselves or caring for sick family members, and recommends social distancing strategies, such as telework, to mitigate the spread.

This raises the question of whether an organization’s communications infrastructure can support 40% or more of the workforce working remotely. In June 2009, the Senate asked that question of the Government Accountability Office (GAO) and got a disturbing answer. Of the 24 CFO Act agencies, only one, the National Science Foundation, had done extensive testing of its IT infrastructure. They reported assessing their telework system formally several times each year and each day through various means. Five more agencies reported testing their IT systems little or none.

Unfortunately, many businesses are in the same position, with no telecommuting infrastructure or a system that is either unproven or not designed to scale to the usage a severe pandemic would impose. As the WHO Phase 5 recommendations and best practices in the industry indicate, the time to finalize the organization, communication, and implementation of the planned mitigation measures is before it hits. Once you have high levels of absenteeism, it will be much more difficult to get a system into place and lost productivity and revenue are no longer theoretical.

There are many reasons beyond H1N1 to implement telework in an organization, including the flexibility that makes for more satisfied and productive employees, reducing Opex costs for office space, going green (reducing energy consumption related to commuting and the heating, cooling and lighting of office space) and preparedness for other crises, such as severe weather, blackouts or other public infrastructure failure, but H1N1 may force many organizations to make the investment sooner rather than later.

Here are a few tips for those looking to validate a new or existing telework infrastructure. Your organization is likely already stretched to the limits of individual productivity with day-to-day operations and ongoing projects. Diverting staff to take on a large project will affect schedules. Instead, look to a third-party to handle the project, especially one with telecommunications testing expertise. While you’re in the middle of an absenteeism/telework crisis is not time to discover a gotcha that was overlooked while validating the system due to lack of experience.

Another tip – when it comes to the test system, lease, don’t buy. Assessing a telework program is an annual effort, not a day-to-day activity. There is no need to expend Capex funds for a test platform that will scale sufficiently to test your system but then will sit idle for the rest of the year. Either lease the system or select a testing service that includes use of a test system.

The bonus is that preparing for the flu can deliver benefits beyond a crisis event by enabling the advantages of telecommuting for your organization.

End Of Life Questions: Automation and Transition Plans

Thu, 05 Nov 2009 17:20:00 GMT

An end-of-life (EOL) announcement on a product that is an integral part of your test strategy is disruptive to schedules, and possibly revenue, as gap analysis is done and transition plans are formed. On the other hand, it forces you to evaluate your lab infrastructure against available solutions, an important exercise that is often pushed to the back burner in the urgency of day-to-day operations and task oversubscription.

Automation

In the test and measurement world, investment in test automation is one of the most significant contributors to resistance to change, and also one of the most significant enablers for cost-savings and faster time-to-market. In the last few years, innovations in automation have expanded the industry beyond test case automation to offer dramatic schedule and productivity gains through automating test cycles, the test environment and the physical infrastructure.

A product that is in EOL may not offer this level of automation support, and if not, then it likely won’t in the future. Eventually it will also be inadequate for your testing requirements, probably well before the EOL date. It’s best to deal with that sooner rather than later, and when you do, automation should be a major factor in your decision.

There may be promises of porting the API of the EOL product to another platform, but porting an API is a high-risk proposition potentially fraught with compatibility and interoperability issues. Test tools developed by different vendors, or even two different tools developed by a single vendor, don’t have a one-to-one correspondence between features and configuration parameters. Tests on the EOL platform may not be supported on the target platform, or may not be configurable to the same detail.

The reality is that porting an API is not likely to offer efficiencies over transition to any other new platform. It’s simply one transition option among several.

Transition Plans

There are many considerations when building a transition plan, such as budget, disruption to existing development schedules, and how quickly test requirements outstrip waning development on the legacy platform. It is a mistake to allow a five-year EOL plan to reduce your sense of urgency. From all perspectives, an aggressive strategy for evaluating alternatives and navigating the transition is best.
In an EOL situation, there are several key factors to evaluate when selecting a target platform for your test strategy:

Coverage for existing requirements (protocols and test cases)
Roadmap for future development
Productivity tools, such as test wizards, to simplify configuration of complex and realistic test environments
A comprehensive automation environment
Professional services support

A transition plan is greatly enhanced when the target platform has a single API that supports GUI-based and script-based automation. Such a “configure once, run anywhere” capability can dramatically increase productivity and reduce configuration errors.

An EOL announcement is usually unwelcome, but it can also be a wake-up call to find the best-of-breed solution on a platform with a clear future and a commitment to supporting technology at the bleeding edge. Not only will your productivity and reliability improve, you’ll avoid facing another EOL situation in the future.

End Of Life Questions: Alternatives and Sunk Cost

Tue, 03 Nov 2009 15:00:00 GMT

Alternatives

Every buying decision is an opportunity to evaluate the current state of all the alternatives on the market to indentify the best solution for your requirements. Unfortunately, the reality is that collapsed schedules and budgets can result in taking the path of least resistance, which leads to considering only the familiar.

Buying more of what you bought before seems reasonable. After all, you evaluated the options at the time, found the best fit for your application, and bought it. But the market is not static. Your products don’t look the same as they did a year ago, and neither do the solutions for your test lab. An EOL product, on the other hand, was likely in decline for some time before the announcement, meaning the product probably already lags the market now, and will increasingly fall behind in features and performance as the market continues to move forward.

An EOL announcement often provides the inertia to do what should be done every time you invest in your lab – investigate all the options to make sure you have the best solution for your application.

Sunk Cost

Sometimes the motivation behind buying more of what you have isn’t hectic schedules or inertia, but sunk cost – the money already spent. But sunk cost is as much an emotional barrier as a financial consideration. The desire to leverage sunk cost creates resistance to change, which can override the opportunity to take advantage of productivity and performance gains available from other platforms.

In an EOL situation, change is inevitable. You can’t leverage sunk cost for future testing because the EOL platform won’t support future technologies. A transition is imperative and it won’t be any cheaper or simpler a year in the future. In fact, if you face this decision now, in a year you could be in a much better position, with productivity gains, a higher-performance test bed and a roadmap that supports your current and on-going development plans.

End Of Life Questions: Development and Support

Thu, 29 Oct 2009 19:51:00 GMT

Support

The key issue in creating an EOL response strategy is support. It sets the ultimate deadline for the completion of your transition plan. However, in most cases the best strategy is to begin the transition sooner rather than later.

An EOL decision on the part of a vendor signals that the business case for the product is no longer viable. As such, diminishing resources will be devoted to support until it hits the EOL date, at which time support is no longer available. Issues that require bug fixes or development are likely to be a low priority, or even a no priority, for the vendor, leading to frustrating support calls with an unsatisfying resolution when problems arise during testing. The impact to development schedules and delivery/deployment dates can be significant.

Development

The problem of diminishing support is exacerbated by diminishing, or non-existent, development for new technologies. A vendor is unlikely to add new protocol coverage, or even enhance coverage of an existing protocol, on a platform that has been officially identified as lacking a viable business case.

As we all know, the telecommunications industry is a fast-paced environment where working groups and committees constantly issue new drafts of recommendations and standards for cutting-edge technologies to address the challenges of delivering faster and more diverse content to consumers with endlessly expanding expectations. An EOL target of five years in the future isn’t the gating factor in your decision for when to start the transition. It’s the date when the schedule says you have to test a must-have feature that can only be delivered by supporting that new draft.

Going beyond protocol support, there are the issues of features and performance. A product under development introduces new features to enhance productivity and ease-of-use. A product in EOL leaves well enough alone, meaning you won’t be able to take advantage of productivity gains provided by a platform in active development.

In addition, new technologies place increasing demands on the solutions you develop, requiring greater power and performance from the device or network you deliver. This places greater demands on your test infrastructure to deliver corresponding performance increases. If the test platform can’t keep up, your budget or your schedule, or both, will suffer. A product in EOL will not have the benefit of on-going focus on the performance improvements your test lab requires. So, once again, the gating factor on your transition decision is not the EOL date, but your testing requirements.

End Of Life or Beginning of Productivity?

Tue, 27 Oct 2009 14:00:00 GMT

The announcement that Agilent is taking the N2X platform to end of life (EOL) in 2015 raises questions about testing strategy for many customers. An EOL announcement is inevitably disruptive to schedules and possibly revenue, but often it causes people to refocus on issues that been unaddressed too long.

Support. The first issue that comes to mind when you hear that a platform you rely on daily is being discontinued is support. What happens when something goes wrong? How long will they fix it? Placing the EOL five years in the future gives you the feeling that you don’t have to worry about it right now. But the reality is that regardless of how far in the future support is dropped, it’s going to be a problem when it happens. The sooner you transition to a platform with a future, the less risk to your development schedules and delivery/deployment dates.
Alternatives. Every buying decision should include looking for the best solution, but frequently the path of least resistance leads to considering only the familiar. An EOL announcement often provides the inertia to do what should be done every time you invest in your lab – investigate all the options to make sure you have the best solution for your application.
New Technologies. A product going EOL in five years may support your requirements for now, but what about that standards committee draft that will be finalized in six months? Is there a plan to support it? Is there an aggressive development schedule to improve performance and support new technologies for the remaining lifespan?
Sunk Cost. In addition to the real expense of transitioning to a new platform, there’s the emotional barrier of sunk cost – the money already spent. The desire to leverage sunk costs creates resistance to change, but in an EOL situation, change is inevitable. The transition won’t be any cheaper or simpler a year in the future. In fact, if you face this decision now, in a year you could be in a much better position.
Automation. In the test and measurement world, investment in test automation is one of the most significant contributors to resistance to change, and also one of the most significant enablers for cost-savings and faster time-to-market. There’s no escaping the fact that the EOL product will eventually be inadequate for your requirements, probably well before the EOL date, and you’ll be forced to change. Whether it’s trying to port an API to an another platform (a risky proposition fraught with compatibility and interoperability problems) or transition to a new platform, it’s best to deal with that sooner rather than later.
Transition Plans. There are many considerations when building a transition plan, such as budget, disruption to existing development schedules, and how quickly test requirements outstrip waning development on the legacy platform. It is a mistake to allow a five-year EOL plan to reduce the sense of urgency. From all perspectives, an aggressive strategy for evaluating alternatives and navigating the transition is best.

Ethernet - Is faster fast enough?

Thu, 22 Oct 2009 15:09:00 GMT

Modern wisdom says you can never be too rich, too thin, or have too much bandwidth. There may be some debate about the first two, but it appears that the third is right on the money.

In the discussion about the need for faster Ethernet we hear of drivers like HD video, mobile backhaul, and point-to-point applications. Social media doesn’t immediately come to mind, but it is a significant factor. (How often have you submitted a tweet only to get the “Twitter is busy” screen?) According to Facebook engineer Donn Lee, many companies need 100G Ethernet in the data center today and could use 400G or 1T Ethernet by the end of 2010.

At the Technology Exploration Forum in Santa Clara, CA in September, Lee said Facebook data centers will require a 64 Tbps data center aggregation layer by the end of 2010. Sixty-four terabits per second. They can implement that with 64 x 1T ports (manageable) or 640 x 100G ports (not so manageable).

The box he needs is a 16Tb switch with 800 x 10G ports from the server clusters aggregated into 80 x 100G (or 8 x 1T) ports to the data center aggregation layer. And he needs eight of them. Lee says that the top 25 websites are probably in the same situation.

How could they need so much so quickly? Phenomenal growth in the user base. Facebook has over 300 million users. It took three years to get the first 100 million users, but only six months to get the last 100 million.

Users (Million)	Time to Acquire
100	36 months
50	12 months
100	9 months
100	6 months

Regarding the arrival of 100G Ethernet, instead of being just in time, it might be, ”It’s about time. When can I get more?” Which points to the urgency of time to market for 100G solutions. Lee may find 640 x 100G ports to be cumbersome to manage, but he’ll probably be happy to get them instead of being forced to use 6,400 10G ports.

Lee also alluded to the challenge of implementing Layer 3 at such high speeds. The issues surrounding upper-layer services at high-speeds, whether 10G, 100G or 1000G, are complex and will require breakthroughs in technology and creativity. Exciting times are ahead.

Donn Lee’s presentation via EETimes.com: http://link.brightcove.com/services/player/bcpid1701276884?bclid=1622640422&bctid=40363249001

Testing Virtualization

Sat, 19 Sep 2009 20:03:00 GMT

It’s no secret what’s driving the move to virtualization in data centers. The demand for new and expanded software systems is growing, but the geographic and carbon footprint required for scaling underutilized dedicated servers is too costly on many levels.

This issue has led to the maturation of the virtual server, where increased reliability and stability means reduced risk, making virtualization of the data center a viable solution. Virtualization makes it possible to replace physical servers running at 10% capacity with fewer, more powerful servers running at 60% capacity or more.

When working with a dedicated server, it’s a fairly straightforward process to characterize performance and isolate factors that affect it. You use a test system to emulate realistic users, traffic and network conditions and measure the response times of the application with the test system.

When working with dozens of virtual servers in a single physical server, it’s not as straightforward. A connection from the test system to the physical server won’t provide the granularity of testing required for meaningful results. The single physical interface handles traffic for many VM instances, making it difficult to isolate and measure the performance of each VM instance or the performance of the virtual switch.

We need visibility into the performance of each VM, but how can we get it? By creating the capability to capture results and generate traffic from within the virtual server instance. We need a virtual test system inside the virtual server.

Read the rest at the Virtualization Journal

The 40/100G Ethernet Testing: Business As Usual?

Sat, 19 Sep 2009 19:47:00 GMT

Everybody’s talking about 40/100G Ethernet. It will replace SONET in a decade. It’s a revenue playground for those with the vision and expertise to deliver in a timely fashion with quality and reliability. This group is doing a trial. That group has released a product.

And, from a market perspective, the time for 40/100G is here. Due to the abundant proliferation of mobile applications and the explosion of IP video, not to mention a host of other applications dumping new traffic on the network, the fiber overbuild of the past decade has gone from feast to famine, as far as bandwidth is concerned. For those who want to expand the revenue stream to accommodate these services, 40/100G Ethernet is clearly the path forward.

When it comes to trials and implementation, everyone talks about lane timing and skew to solve the problem of moving bits at 100 Gbps, error free. But equally challenging, if not more so, is scaling the upper-layer engines to deliver services and quality of experience at four to ten times the current rate. Brand credibility and customer confidence depend on solving both problems.

This of course means testing. The sheer speed required to send data and to support services at 40 Gbps and 100 Gbps might not just break network devices, it might even break the testers used to validate the performance of services on those devices.

Because a trial is only as good as the test platform. If the tester can’t accurately count packets and measure latency and jitter at 100 Gbps, it could pass systems that won’t stand up to real-world traffic in a real network.

To learn more about the issues associated with testing 40/100G Ethernet, take a look at this whitepaper. [Testing 10/100G Ethernet] It explores the challenges in the physical layer, limitations in the test system, the budget in the test lab, and delays in the development/deployment cycle.

Because passing a trial but failing in the network can ultimately cost a vendor or provider a lot more than a place in the 40/100G Ethernet revenue playground.

Going Green in the Data Center

Thu, 27 Aug 2009 20:34:00 GMT

Maybe it’s not easy being green, as the infamous flannel frog pointed out decades ago, but being un-green is no picnic, either. Many data center managers rate their top three concerns as security, availability and energy.

Data center costs, including energy, have climbed eight-times since 1996. When high-density blade servers can push power requirements past 30 kilowatts per rack, the transition to more efficient systems can have a huge impact not only on energy usage, but operating expenses. Especially considering that a watt saved in power is another watt saved in cooling. It’s like a save-one-get-one-free sale on watts!

The more efficient power supplies now available mean less power wasted in conversion and distribution. However, many of the energy savings are only available for those who enable the power management for their servers, something many data centers don’t do. The reliability of the power management tools concern some managers, as uptime and performance are key issues for any data center.

When it comes to servers, the metric of interest is the correlation between utilization and energy. Legacy servers tend to use at least 30% of peak power even while idle. Obviously, for maximum efficiency the server should run at greater than 30% utilization, but 6% to 15% is more typical. The move to virtualization partially addresses this concern with more powerful blade servers running many virtual machines at higher utilization, allowing administrators to find the sweet spot between performance and power.

Virtualization, multiplay, and network and service convergence, all result in a greater demand being placed on the network infrastructure. Devices may be decreasing in power as it relates to energy usage, but they are increasing in another kind of power - performance, features, protocol depth, and number of queues, routes, sessions, or tunnels supported. And this increase in the density of functionality and performance per rack places a greater burden on test systems.

In the greening of the infrastructure, test systems are often overlooked. But test systems are not and should not be exempt. They should also deliver more power (functionality and performance) while consuming less power (energy).

You should ask some pointed questions when it comes to your test systems:

Are power requirements (per port and per rack) decreasing?
Are functionality and performance (per port and per rack) increasing?

As you simultaneously reduce your carbon footprint and your operating expenses, make sure your test systems are working with you.

Why Test Realism Matters Or When traffic goes up and websites go down

Tue, 25 Aug 2009 12:00:00 GMT

Maybe it only seemed like the world stopped for the Michael Jackson memorial on July 7, but there’s one thing we know for sure that did stop.

The City of Los Angeles set up a website to accept donations to offset the $1.4 million required to support the memorial with services like security, traffic control and sanitation. According to CNN, the website crashed repeatedly Tuesday and Wednesday, sometimes for periods as long as 12 hours. Clearly they didn’t test the system to verify it could handle high volumes of traffic. That’s understandable, given the unexpected nature of the event.

During the same time period, a spike in traffic caused by a fare sale at Southwest Airlines resulted in outages and poor performance on their website from 10am to 3pm. The headline read, “Southwest Air Won’t Extend $30 Fare Sale After Web Site Lockup.” Unlike the City of Los Angeles situation, this event was not unexpected. No estimates of the cost of lost business were provided, but five hours of problems during the middle of the business day undoubtedly took a big bite out of revenues, not to mention customer goodwill and corporate reputation.

And that is why test realism matters. Most corporations subject their ecommerce infrastructure to extensive testing before going live. Most likely Southwest did. But it’s very easy to test a system and never find the failure points lurking within. It usually goes like this: set up a test network, hook up some packet blasters and bombard your system with traffic. The results show good performance from the canned traffic and it’s all thumbs up until the system hits the real-world. Then the wheels come off.

Test realism avoids this unfortunate fate by hitting the system with the real-world before deployment. As mentioned in an earlier post [Test Realism], test realism involves:

Real user behavior. The flexibility and sophistication to emulate a wide range of user behavior, both benign and malicious.
Real converged traffic. The power to create line-rate, fully-emulated, stateful traffic across hundreds of ports.
Real network conditions. The power and complexity to create the dynamic, time-varying conditions found on deployed, production networks.

In addition, test realism includes a test methodology based on decades of experience with the technology and application. Of course, you need a test system with the power and scalability to create a real-world environment. But it’s possible to have that test system and still create unrealistic tests. That’s why the test system must have expertise built-in, in the form of test wizards that follow best-practices test methodologies, and also available via professional services for more customized testing.

Unfortunately, in the telecommunications world the lack of real-world testing can result in high-visibility, non-career-enhancing failures. It’s nice when your company is making headlines, but this is one headline you don’t want to be in.

Test realism. Don’t leave the lab without it.

UPDATE: And the hits just keep on coming. At end of July, the website for the “Cash for Clunkers” federal rebate program crashed 15 minutes after it went online due to overwhelming demand. It was down for 5 hours.

IPTV - What to Test

Thu, 13 Aug 2009 16:00:00 GMT

IPTV is on the rise, more quickly in some places than in others. While some areas have already hit double-digit market penetration, others, like the US, are still in single digits. Growth rates vary by region from 20% to over 100%. More than half of the video rights holders and distributors surveyed said they are distributing video online today, and only 10 percent said they had no plans to do so¹.

As the new kid on the pedestal, IPTV must deal with viewer expectations set by decades of broadcast and cable television. It has to prove itself, offering similar or better quality of experience (QoE) while enhancing the value proposition with more features. That’s why testing has been such an important part of the development and rollout of IPTV offerings, like the test EANTC did of Cisco IP Video Infrastructure solutions.

The true litmus test of an IPTV service is QoE, but there are a lot of components that contribute to this metric, and there can be no weak links in this chain.

When it comes to testing IPTV, throughput performance is the most obvious place to start. If the system can’t support the throughput required without issues such as congestion and link or node failure, there’s no point in testing further.

Cable and broadcast TV have the luxury of dedicated wavelengths to guarantee throughput. IPTV, on the other hand, must deal with packet forwarding and contention, making it more difficult to achieve the level of throughput required to deliver standard definition TV, much less high definition. Packet latency, packet loss and packet delay variation all conspire to sabotage service quality. For video, packet loss must typically be zero for an acceptable service. RFC-based throughput testing is the key to verifying that a system or service can deliver the throughput required for high QoE.

IPTV is typically deployed in a multiplay environment. Television video traffic will coexist with VoIP and internet traffic. Some of the traffic may come from high bandwidth applications such as online gaming, file transfers, and P2P sharing. To assure that the solution delivers high video QoE even in the presence of congestion, packet classification and prioritization mechanisms are tested, independently and together. The metrics of interest here are packet latency and packet loss.

Resiliency indicates the ability to maintain service even in the face of a link (the connection between two points of presence) failure or node (entire device) failure. For such failures, it’s not a question of if, but rather a question of when. Redundancy is a method of enforcing resiliency. Regardless of the method of redundancy implemented, for example, redundant headends or redundant video streams, testing is required to verify the response time of the method and its effect on QoE.

The metric of interest for testing a resiliency mechanism is the out-of-service time (OoST). One method of calculating OoST is to count the number of lost frames during a failure event. A more accurate method is to continuously sample the received frame rate for very short intervals and report the duration between falling below an acceptable frame rate and returning to that rate.

That’s an overview of what to test and why when it comes to IPTV. To do the testing you will need a test system that can:

Generate and track stateless and stateful traffic at line rate
Integrate unicast and multicast traffic streams within a single test
Perform video analysis on live traffic is imperative
Collect key metrics such as packet delay, packet delay variations and packet loss in a single test run
Create realistic multi-protocol stacking scenarios, such as HTTP over PPPoE over MPLS

¹ Source: Lightreading.com

When Data Centers Collide - A Modern Fable Of A Lost Lunch

Wed, 29 Jul 2009 19:00:00 GMT

Last week I almost did lunch with a friend, an IT guy. We picked a new deli close to his office and, after a long wait, I finally had a nice Reuben all alone. I texted him and he texted back with apologies and tales of application performance issues.

Being a nice guy, I got a sandwich and chips to go and stopped by his office. I knew I would be welcome. While it might be prudent to beware Greeks bearing gifts most people are glad to see geeks bearing food. Especially when they had to work through lunch because a VP is breathing down their neck about their CRM slowing to a crawl.

My buddy took a short break and told me his sad tale of woe between bites. “We’ve been having issues with our customer relationship management system ever since we installed it. We thought we had it straightened out and picked up on the data center consolidation project.” He paused to wolf down some more sandwich and continued. “Last night we did the cutover from our various corporate data centers to a central facility in Denver. This morning the phones lit up like a radio talk show switchboard. Response times went through the roof. People going to get coffee between screens. I’m not talking down to the break room. I’m talking the Starbucks on the corner.”

I nodded in sympathy and offered him another potato chip.

His situation is not as unusual as it should be. Companies are developing and deploying distributed applications all the time without testing them against realistic WAN conditions. I’ve seen lots of reasons why this happens.

The test plan overlooks the problems that can be caused by a real network. Or they run a few test trials over their production network, usually after hours to prevent business disruptions, and have a false sense of confidence when they don’t see any problems. Or they don’t realize that actual network conditions can be recreated in the test lab.

It’s called network emulation, one of the essential elements of Test Realism.

Test realism lets you do to your application what the WAN is going to do to it, before you have hundreds of users like Kim complaining about your system. The alternative can be ugly. Productivity loss. Finger pointing between the network group and the applications group. Acrimonious meetings. Loss of confidence in your department. Morale issues among your staff.

It’s a shame, really, when, like karaoke, it’s completely preventable.

It is possible to know in advance how your system works on a real network. To test during the relatively calm and rational time of development and testing instead of the high-visibility, company-wide exposure of a post-deployment crisis. To troubleshoot and identify performance and response-time issues early, when it is less expensive to find solutions and implement them.

In fact, it’s not only possible, it’s pretty much essential.

Using test realism in the form of real-world network conditions makes everyday life better for hundreds of thousands, if not millions of people. Kind of like buying a sandwich for everyone.

Realism and Multi-Play - Part 1 NEMs

Tue, 21 Jul 2009 12:00:00 GMT

Find and fix your bugs before your customers find them for you

Remember what Jack Nicholson told Tom Cruise in A Few Good Men? “You can’t handle the truth!”

That’s the kind of thing you don’t want a customer saying about your product. “Sure, it works fine in the lab where everything is well-behaved, but it can’t handle the real world.”

So, what exactly is the real world? In the movie, reality for Nicholson was different from reality for Cruise. My reality is lots of conference calls, business travel, deadlines. Not only can I handle it, I love it. Reality for a soccer mom is very different, and let me tell you right now, I can’t handle that reality, so let’s not do Trading Places.

A while back we talked about test realism which includes:

Real user behavior: wide range of stateful user behavior, both benign and malicious
Real converged traffic: line-rate, fully-emulated, stateful traffic across hundreds of ports
Real network conditions: dynamic, time-varying conditions found on deployed, production networks

But reality for an email server is very different from reality for a set-top box. Which means, if I want to get all the benefits of test realism (like increased test coverage, better product reliability, improved customer quality-of-experience, and fewer customer-found bugs, to name a few) I have to first know what reality is for the system I’m testing. Then I have to be able to faithfully create that reality, fully emulating every user, device and condition in that reality, from end to end, that my system will encounter.

In the multi-play world, reality gets complex. It’s the perfect storm of diverse content and behavior. So, how do I get that complex reality in my lab?

First, I need the scalability to generate a city’s-worth of voice calls, IPTV traffic, and data traffic for thousands or millions of users across hundreds of ports. But the ability to scale to reality in number of ports and volume of traffic, while important, isn’t by itself realism. That’s just packet blasting from a lot of ports. It’s a brute-force assault that is fairly easy to do but doesn’t tell me anything about how my system will handle reality. Which leaves me at risk for Nicholson-type comments from my customers.

Test realism for multi-play means the traffic is generated by fully-emulated devices that generate stateful voice, video and data traffic, including signaling and routing. Emulated devices that respond to queries, requests, and commands from my system.

Second, I need that traffic to come from real users. An artificial blend of dummy transactions based on a percentage or distribution doesn’t create the conditions in the system under test that is created by real users. I need stateful traffic from users doing things in the order that users do them, whether they be consistent (normal usage), seemingly random (multitasking or short attention span) or malicious (security risks).

Third, I need the ability to create the imperfect world my system will exist in, one where packet delay, loss and other unpleasant things sometimes occur. A pristine lab network is not reality. And I need this at wire rate, and when I say wire-rate I am talking at 10G speeds!

Test realism for multi-play means my system is forced to deal with the dynamic, time-varying conditions caused by likely events such as routing table updates, queuing and buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.

Fourth, I need one system that can emulate all of this realism: users, traffic, and network. I don’t need a solution that requires me to patch together a dozen or half-dozen different systems to achieve an approximation of realism that might work as long as there are no conflicts between the various test systems and how they emulate or measure their niche in the test. I need one integrated system with one GUI and one API and one correlated set of results and the power to support all the elements of test realism.

Fifth, I need trickle-down expertise built right into the test equipment that guides me. I need to know I am buying a solution from a company that knows what to test and how to test it. Because, whether painful or not, I have to know the truth before my customers do. Even if that truth is that the performance of my system isn’t as good as my unit test led me to believe. Especially if it’s not as good as I thought.

The best test solution is like a true friend. Not the phony friend that tells you what you want to hear, but the true friend that will tell you the truth, even if it’s harsh, and then will stick with you to fix it.

So I’m looking for expertise built into the test gear, into the software interfaces, and into the support and services I get from my vendor. It’s got to be a complete package. This expertise comes from (1) participating in standards bodies to create protocol and testing specifications and (2) by successfully supporting a user base of thousands of customers, year after year. I need a vendor who is going to be around tomorrow for the long term to support me.

That’s a lot to ask from a test system. But there’s a lot at stake. And when it comes down to it, you really do need to know the truth. And your solution really does need to be able to outperform the competition in the real world.

The truth is out there. Can you handle it?

Test Case Automation – The Gift That Keeps Giving

Thu, 16 Jul 2009 11:00:00 GMT

A while back we talked about test lab automation (see Advanced Test Automation), which is returning sanity to test labs all over the globe. In that post I made this statement:

"If you haven’t automated your tests, you’re missing one of the best ways to boost productivity and impress your boss. (And the money guys, too.)"

It occurs to me that there might be outposts of resistance lurking in the hinterlands that have yet to reap the many rewards available from test case automation.

As I list the many benefits of test automation, you may start to wonder if I’m going to claim that it also trims 20 lbs from your waistline and restores hair loss. Not exactly, although it might keep you from tearing out your hair, so maybe there’s something to that after all.

But seriously, the benefits are there. How about fewer customer-reported bugs, expanded test coverage, increased productivity and lab usage, reduced operating costs, shorter test cycles, and more. Let’s look at some hard numbers:

A lab I visited recently was testing their 10Gig MPLS architecture for approximately 20 different test cases. Test automation freed up 3.5 engineers to add more test cases and expand test coverage.
Another lab was testing RP switchover convergence. We found that it took 35-45 minutes to run a single test case. With more than 20 iterations for each test, one test could take an entire work day with an engineer babysitting it to keep it going. The test was automated to eliminate the need for user interaction. This change produced a 40% productively gain – nearly half a person’s daily activity recovered by automating a single test suite.

This is huge! Can you think of any one single thing you can do that will increase your productivity by 40%? Personal organizer? No. Time management? No. Multitasking? No. Meditation? No. Cloning? Maybe.
Now you see why I call it the gift that keeps on giving. From now on, every time they run that test, they’re saving a boatload of time and money based on a one-time investment. It gives me a warm fuzzy just thinking about it.

One of the real values of scripting is to increase test realism. Scripting means you can precisely control the test environment during execution to recreate the dynamic behavior of a real network. And of course, the more realistic the test, the more meaningful the result.

Which is the point, after all. You want to know your system can handle the real world. It’s not useful to know that it passes all the tests in an artificially favorable environment built with fixed stimuli and static topologies, and then ship it or deploy it and have your customers find the problems for you!

Now that you’re ready to join the club, let me give you a tour of what’s out there and what to watch out for.

Scripting. You write a program, usually in Tcl, to configure the test equipment and the system under test and then to start the test.

Pros: If the API is robust, you have more control over the test than you would running it from the GUI.
Cons: Must learn a scripting language.
What to look for: A test system that allow changes to the configuration during test execution, so that a realistic test can be created.

GUI-to-script. You configure a test with the test system GUI, then save the configuration as a script which may (good) or may not (bad) be modified to customize the test.

Pros: No need to learn scripting, especially if you’re not customizing.
Cons: Configuring through the GUI can be time-consuming. Unless you learn scripting, the tests will be of limited use, basically configure and go.
What to look for: Support for script-to-GUI, useful for troubleshooting configuration issues after tweaking a script.

GUI-based automation editor. You create automation by dragging and dropping commands into a visual command editor, popping up windows to configure parameters.

Pros: Create sophisticated, realistic tests without learning a scripting language.
Cons: None.
What to look for: Tight integration between GUI commands and API commands, so that automation created in one interface can be used in the other.

So, pop on that lab coat and start automating those test cases. Your boss and your bottom line will love you for it.

40/100 GbE Ready

Thu, 09 Jul 2009 11:00:00 GMT

Only 176 shopping days until Christmas. And, before you know it, you’ll be testing 40 GbE or 100 GbE solutions. These things tend to creep up on you.

You may not be the type to think about Christmas shopping in July. I know I’m not. But when it comes to investing in a test bed, it pays to plan ahead. And the next generation of high-speed Ethernet poses significant challenges for legacy test platforms.

Synchronization and timestamps are two of the challenges posed by high-speed testing.

Synchronization. A test is all about results, but results are meaningless if they aren’t synchronized. At high speeds, synchronization becomes even more important. In a 100 GbE environment that can send one frame every 6.72 nanoseconds, nanosecond resolution is required. And synchronization has to be maintained across all test ports, whether they are in one chassis or spread across multiple chassis. Without synchronization, results from different ports, blades or chassis cannot be correlated to isolate anomalies during troubleshooting, bug detection and isolation.

Timestamps. Another critical capability of network test equipment is the ability to accurately timestamp packets or frames. Timestamps are used to measure latency and jitter. Every frame must have a unique timestamp. At higher speeds, legacy test platforms may not be able to uniquely mark packets because the internal timestamp clock is too slow. If two or more packets fit into one timestamp clock tick, then those packets are marked with the same time, which renders latency and jitter measurements meaningless.

Many legacy test systems don’t have a clock with the speed and resolution to test 40 and 100 GbE.

So, when you’re putting together your shopping list, make sure you get on the right side of the naughty/nice list by select a system that will still be useful when you’re ready to test 40 GbE and 100 GbE. Look for a system with a timestamp clock with the granularity to support the speeds you’ll be testing soon. Also, it won’t hurt to make sure that it automatically syncs and adjusts to provide latency measurements with nanosecond accuracy between ports, modules and chassis, regardless of operating temperature change and cable lengths.

Test Realism - Find and fix your bugs before your customers find them for you

Tue, 07 Jul 2009 07:49:00 GMT

Expectations. When I turn on the faucet, I expect water. When I flip a switch, I expect the light to come on. When I pick up the phone, I expect to be able to make a call. When I punch the remote, I expect to see my show. All pretty much the same, right?

Almost. Water and electricity are delivered to us in basically the same way they were a century ago. But delivery of phone service is changing, and delivery of television, which didn’t even exist a century ago, is being radically transformed.

Regardless of how the infrastructure has changed, customer expectations haven’t. When I punch the remote, I still expect to see my show. Sure, all the new technology that makes it work is cool, but when my team is down by two with five seconds on the clock, the score I’m focused on is not a MOS-V score.

Only through testing can you meet reliability and quality expectations. Of course. Everybody tests.

So why is it that customers still have reliability and quality-of-experience problems? Because it’s not if you test, but how you test.

You avoid customer-reported problems with realistic testing. Strategies used in the past, such as packet blasting, aren’t effective in predicting how applications will work on a converged network. Throwing traffic at your system, whether on the data plane (short packets at line-rate), control plane (massive route updates) or subscriber signaling (high setup-teardown rates), doesn’t tell us a thing about how the system will handle Black Friday shopping or the final vote on American Idol.

Realistic testing means re-creating the environment that the application lives in, from the provider to the customer, in all its dynamic and daunting complexity. Test realism means:

Real user behavior. Users are as unique as their fingerprints. They vary significantly in how long and in what manner they navigate through an application and how they respond to sluggish performance, picture and voice dropouts, dropped calls, and other problems. They violate usage and security policies in different ways. They find unique ways to break your system.

Realistic testing means the flexibility and sophistication to emulate a wide range of user behavior, both benign and malicious.

Hitting a device with a mix of traffic types (say, mixing HTTP requests/responses with IPTV channel changes and P2P file sharing or gaming traffic) isn’t emulating user behavior. It’s just hitting it with a mix of traffic. Emulating real user behavior means supporting stateful traffic that emulates how a user operates, including think time, click-through, abandon, channel surfing, etc. It means good users and malicious users simultaneously attempting to achieve their good and bad goals.

User-centric traffic tests the performance of the device in a real environment. Queues, buffering, and other mechanisms behave differently depending on the order and the nature of the transactions. An arbitrary (and artificial) static mix of messages, or even a dynamic mix of messages that don’t account for the stateful nature of user connections, will not stress the system the way real users do. As a result, failure points can remain undetected until the real users start using it. Which is exactly the wrong time for that to happen!

Real converged traffic. Single-purpose networks are relics of the past. The consolidation of networks onto Carrier Ethernet and MPLS enables mobile and fixed-line voice and data, residential video, and MPLS-based VPNs. The different types of traffic carried on the converged network have different characteristics and requirements, but they all travel on the same path, dependent on CoS and differentiated traffic rules to keep everyone happy.

Realistic testing means the power to create line-rate, fully-emulated, stateful traffic across hundreds of ports.

Real converged traffic not only means a realistic mix of traffic types, but also realistic traffic encapsulation. For example, if the deployed system tunnels user PPP sessions over MPLS, then testing PPP setup-teardown rate and throughput performance without MPLS is not real converged traffic. It’s a dangerous shortcut that will mask problems your users will discover after deployment. Real converged traffic means emulating the actual deployed topology, regardless of how complex or simple, including all encapsulations.

Real network conditions. The network creates time-varying conditions that are linked to a complex set of conditions, influenced by routing table updates, signaling protocols, queuing algorithms, buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.

Realistic testing means the power and complexity to create the dynamic, time-varying conditions found on deployed, production networks.

Real network conditions can’t be emulated through static rates of delay/loss or distribution-based models of impairment. Real networks don’t introduce impairments at fixed rates or follow neat curves. They behave in seemingly non-deterministic ways due to the number of factors affecting them. Testing under real network conditions means emulating this complexity to discover issues before the real network finds them for you.

Test realism gets you closer to the goal of delivering reliability and quality of experience, which is the ultimate differentiator in any market. Oh, hold it, the game is on. I’ll be right back.

Advanced Test Automation: Making the Case for Test Lab Automation

Thu, 02 Jul 2009 17:20:00 GMT

Hold the phone! Stop the presses! Suspend the blogs! Abate the tweets!

Some breaking news: Testing is essential for achieving product quality, improving customer satisfaction, reducing costs and increasing revenue.

OK. So that’s not news. It’s pretty much common knowledge.

How about this: Test lab automation is an incredibly effective (but often overlooked) strategy to significantly reduce operating costs while at the same time improving test lab utilization, test coverage and product quality.

What? You say you already use automation to script all your tests? Good, because that’s a huge step to increase efficiency and reduce cost. If you haven’t automated your tests, you’re missing one of the best ways to boost productivity and impress your boss. (And the money guys, too.)

But I’m not talking about test automation. I’m talking about test lab automation. Aha!

See if this sounds familiar. A test lab with many systems to be tested and a range of test equipment. They are connected either directly or through a patch panel. Every time a test configuration changes, somebody has to go in and rewire the physical connections, hopefully not making a simple mistake that will eat up an hour or two in debugging time later on. Then the systems under test must be configured to support the particular test, or series of tests, for this session. Once again, hopefully with no time-wasting errors. Add to this scenario increased demand for access to the test lab from groups that may be anywhere in the world and may require on-site assistance with the physical setup and configuration, or with power-cycling a system at some odd hour if something goes wrong.

It’s not a pretty picture.

This kind of setup, while common, is inefficient and makes it difficult to share, track and manage test lab resources. And it will continue to get worse as the number and types of interfaces and technologies to test increase, along with the demands for access from remote-site teams, telecommuters and off-shore developers.

And don’t forget collapsed development cycles, increased time-to-market pressure, lab consolidation and whatever else hits you tomorrow.

Which is why lots of organizations are turning to test lab automation to stop the madness and get some sanity back in the test lab.

Instead of what you’re doing now, consider this: A test lab where the systems under test and the test equipment are wired into physical layer switches that serve as programmable patch panels. An application on a PC (anywhere on the network, local or around the world) is used to make test topology changes, power cycle devices, search for and substitute alternate devices or ports if problems arise, configure DUTs and test devices, and schedule tests.

Now that’s a picture you could learn to like. Am I right?

Some pioneers have created what we might call the first generation of lab automation: home grown, ad hoc solutions. They’re a start, but suffer the shortcomings of all in-house efforts: usability issues, lack of scalability, limited or non-existent user support, and no path forward for enhancements and expansion.

The next generation comes from vendors who have been at the game for some time and have worked through the various customer challenges associated with lab automation. Second generation lab automation is characterized by holistic and product-agnostic solutions that simplify your life and accommodate the wide range of brands and technologies you already have in your lab.

Now that might be a stop-the-presses moment after all!

Unified Threat Management

Wed, 24 Jun 2009 03:21:00 GMT

You don’t need me to tell you that realistically testing a Unified Threat Manager (UTM) can be a nightmare. The convergence that is happening on so many fronts offers a utopian future of lower OPEX and streamlined services, all great for the bottom line. But under the hood, the dirty little secret is a more-than-compensating increase in complexity, and security solutions are not exempt.

A UTM squeezes multiple security functions, including content filtering, spam filtering, intrusion detection, anti-virus protection, security and policy management, into a single box. But whether it takes up a whole rack or one box, somebody has to make sure it actually (and efficiently) does what it’s supposed to do. And that somebody is you.

You need a tool that tells you if your system keeps the bad guys out while letting the good guys get their work done. You want to know if it will pass through traffic with minimum delay and loss while stopping threats and enforcing class of service and usage rules. And, if possible, you’d like to find that out without going crazy trying to recreate a realistic environment or going blind trying to correlate the results to see what they mean. Talk about a nightmare.

The ideal UTM test system would:

1. Save time by generating threats along with safe traffic like VOIP, P2P, C IFS, and real MPEG4 video (from the same port at the same time)

2. Deliver the horsepower to scale at all protocol layers in all the dimensions that matter, like threats per second, sessions per second, SSL sessions per second, and concurrent connections. But these sessions have to be real – no fake TCP stacks, please! You know what I’m talking about here ;-)

3. Be controllable from a single GUI for fast and accurate test setup, and if I want to get scriptable I want at a minimum automatic script generation, and ideally, integration with leading test frameworks

Guess what? Spirent has the only UTM test solution that brings all that to the table in a way I’d like. Of course you expected me to say that, but seriously, integrated threats and protocols on both IPv6 and IPv4, with a single GUI, correlated results in a single blade on a flexible chassis-based architecture hasn't existed until now.

The combination delivers unprecedented test realism with the scalability and performance required for the job.

What’s that mean for you?

Reduced time-to-test. Less setup, less waiting, more combined results in a single test run.
More realistic testing with real TCP stacks and user behavior that catches real problems without reporting time-wasting false positives
Integrated results for better interpretation of what is going on the network
A more reliable solution

That’s right. The nightmare is over.

Testing Data Center Virtualization: Are you asking the right questions?

Mon, 15 Jun 2009 11:28:00 GMT

Data centers today are larger, faster and more complex than ever. New technologies such as virtualization, Fibre Channel over Ethernet and 40/100 Gigabit Ethernet aim to help organizations move multiple traffic types – data, storage, video, and voice – onto a single, converged core. Validation of all these new technologies through testing is a crucial part of the product development and deployment process.

Today, let’s look at testing virtualization. Testing virtual network devices and servers poses several interesting new questions:

How can a test instrument attach to a virtual network device?
Often one physical interface may handle traffic for dozens of Virtual Machine (VM) instances, making it difficult to isolate and measure the performance of each VM instance. What’s needed is to virtualize the capabilities of the test instrument. A virtual test instrument resides in software, and thus runs inside the physical machine hosting virtual network and server instances. From the standpoint of the virtual network device, a test port looks exactly the same as it would in the physical world.

Can test instruments on virtual machines be trusted?
One strategy to ensure measurements can be trusted is to implement the entire test instrument – including software-based emulation of hardware components – in software. This approach requires a far more rigorous approach to system design than does software-only tool design. But the benefit is clear: by emulating the entire test instrument in software, the instrument’s measurements are far less dependent on extraneous factors. Such an instrument will produce more meaningful measurements than software-only tools.

Do virtual and physical switches offer comparable performance?
Line-rate throughput and low latency and jitter have long been the hallmarks of physical Ethernet switches, but their virtual counterparts may not compare. Even if virtual switches will never handle loads as heavy as physical switches (a dubious assumption in these early days of virtual networking), it is still important to conduct stress tests to describe the limits of system performance. Industry-standard methodologies for assessing unicast and multicast performance in switches and routers still apply in the virtual world.

Does a virtual switch support the same protocols and functions as a physical switch?
A network manager can reasonably expect any modern Ethernet switch to support features such as Virtual LANs (VLANs), Access Control Lists (ACLs), and Internet Group Management Protocol (IGMP) for forwarding multicast traffic. These protocols (and often many others) are often included as part of physical switch performance testing and should also be included as well when testing virtual switches.

For more information on virtualization and data center testing download our latest whitepaper titled “Data Center Testing: A Holistic Approach.”

Welcome to The Tested With Spirent Blog

Thu, 11 Jun 2009 11:29:00 GMT

Testing is a strategic imperative as service providers deploy highly complex converged networks delivering differentiated voice, video, data and applications based services. Testing today’s, as well as tomorrow’s solutions and networks is driven by the requirement to achieve uncompromised performance and scalability while delivering a superior quality of service and experience across the telecommunication infrastructure.

Today, our commitment to the test and measurement community is extending beyond just products and services to offer a forum dedicated to discussing emerging technologies and the related challenges surrounding testing, improved quality and faster time-to-market..

The Tested with Spirent blog is designed to provide test engineers with insights on methodologies, approaches, challenges and the latest trends affecting new and emerging technologies. Industry experts such as Alan Way, David Newman, and others will deliver commentary that will help you make informed decisions regarding your product and service test methodologies. The insight shared within this blog is intended to accelerate the development and deployment of next-generation converged networks and technologies.

Our role as leaders in test and measurement is to enable you to develop and validate the most successful products and services, not just from a technological perspective but also to ensure the greatest return on your investment. We hope this blog provides a unique perspective on testing enabling you to be more successful while providing a unique forum for you to provide feedback and share your opinions.

Data Center Testing: When It Comes to a Data Center Rollout, Kicking the Tires and then Driving It Off the Lot is Not Enough

Thu, 11 Jun 2009 08:20:00 GMT

It seems like there are a dozen ways to succeed and a thousand ways to fail. And it is disconcerting how easily a project that is on track can end up in the ditch. I’ve noticed that often success comes down to discerning between the merely urgent and the truly important. And when the tyranny of the urgent overshadows more important issues, you can end up with a wrecked project.

Nowhere is this more true than in the data center. I know how it is. Feature creep and schedule delays can edge performance and scalability testing out of the schedule. But that leaves you vulnerable to SLA violations and service outages. And that is when deferring the important for the urgent can really cost you.

Estimated Outage Cost-per-minute

Application	Cost
Supply chain management	$11,000
e-commerce	$10,000
Customer service	$3,700
ATM/POS/EFT	$3,500
Financial management	$1,500
Source: Alinean

What it comes down to is this: you can’t wait until the system is deployed to find out how it will perform. You have to know before deployment, when fixes don’t cost five digits per minute. Failure to budget for testing (typically three to five percent of the data center budget) is like building a new house with top-of-the-line plumbing but not turning on a single faucet to check any leakages in the installation until it is finished, decorated and moved into. Something you hope your home-builder didn’t do.

And you shouldn’t do it, either. Testing is a strategic imperative, a plan that avoids the almost certain loss of revenue and customers when things go wrong - performance issues, SLA violations or service outages. I’ve seen organizations gain a competitive advantage through testing, because it enables them to do performance optimization and offer a level of reliability that separates them from the crowd.

One recent OnDemand project involved a major financial trading site flipped the switch on a new data center with zero problems in the first 48 hours, a first for them. They were one of only two trading sites in the US that were able to keep up with the pre-April 15th trading peak, avoiding the delays and outages experienced by other sites.

All because of robust performance testing during the project lifecycle with best-practices test methodology expertise available with OnDemand. It is no surprise that those who come to us for testing keep coming back. The ROI is clear.

Some organizations rely on the team that designed and implemented the system to do the testing. However, the project team, whether an in-house team or outside consultants, will not see their own blind spots and therefore won’t test for them. Testing, like accounting, requires the objectivity and accuracy that comes with separation of duties. And, like a good auditor, Spirent follows best practices, in fact develops best practices as new technologies and protocols emerge, through participation in standards bodies and partnering with manufacturers as implementations are developed. That is simply a level of expertise not likely to be duplicated in-house or with a consultant.

It is that kind of expertise that enables us to develop test methodologies that anticipate future needs for capacity, services and traffic types.

For the sake of your project, your revenue and your customers, take a break from the deceptively urgent to consider the value of the truly important for your organization and avoid ending up in a ditch.