Spirent Broadband Blog

The Death of Best-Effort QoS Testing

Leslie.Gollehon@spirent.com (Leslie Gollehon) — Tue, 02 Mar 2010 13:00:00 GMT

Well, it’s official. Testing exclusively with best-effort traffic is dead. Today, even the simplest network implements Quality of Service (QoS). Testing with exclusively best-effort traffic used to be acceptable for numerous reasons: Simplification of testing, the experience of the end user, and the premise that more bandwidth solves all problems.

As with all industries, both the people who make the products (network equipment manufacturers) and the people who consume those products (service providers and enterprise) are becoming more sophisticated. As they have correctly deduced, bandwidth cannot solve all problems. In the end game, the winner is the one that can roll out network services first with consistently high Quality of Experience (QoE). In other words, what will prevent the end customer from calling tech support?

Actually, this level of customer satisfaction is not possible with only best-effort testing. In the modern network, congestion at each hop is a real phenomenon. Just as security exploits and new ultra-low latency and bursty protocols such as P2P, Storage and HD Video over IP, each of these will require a very specific network ecosystem to succeed, which is only achievable with QoS. In a nutshell, every test must be performed with QoS considerations in mind.

Testing Video over Carrier LDP/RSVP-TE MPLS

Leslie.Gollehon@spirent.com (Leslie Gollehon) — Tue, 09 Feb 2010 14:27:00 GMT

One of the great features of Spirent TestCenter is its ability to directly test services over structured Ethernet. This is critical to carriers because their business model is becoming increasingly dependent upon revenue from IP video.

With Spirent TestCenter, you have the ability to completely isolate the carrier Ethernet DUT, regardless of its role. Say the device under test is an MPLS P router. With Spirent TestCenter topology emulation technology, you can model routers behind routers as true objects.

Once you build out the IGP, you can model MPLS, adding depth and width to the MPLS tunnels. Then, within the LSPs, you can source Unicast and multicast MPEG2-TS video streams over unicast, or optionally over multicast, with Spirent TestCenter stateful PIM-SM and SSM support.

How is this unique to the industry? Because of the pure object-oriented network-module features of Spirent TestCenter, structures are layered upon each other as opposed to a relational model, where entities are next to each other in tables. When you test a lower-layer feature, such as a BGP route flap event, the event instantaneously traverses up and down the relationship tree to highlight the impact.

With relational models, you would have to reconfigure the IGP and then manually propagate the changes to each table in the tester. Many events align within microseconds and thus become untestable in relational models. In the case of video, the BGP convergence loss may or may not affect user experience.

Say a flap event loses 10 datagrams. Depending upon the encoding of the video (SD or HD, MPEG2-TS with more or fewer I-Frames) the event may or may not affect the MPEG GPOB. If I-Frames are lost within the data stream, you will see MPEG artifacts on screen (blockiness, blur, etc). If P-Frames are lost, this is fairly recoverable and you may not see any effect.

The point is that unless the test tool has the ability to relate objects with real-time processing, you will not be able to test many real-world errors with video. Properly testing IPTV is critical to the success of the business.

The 21st Century Workplace

Leslie.Gollehon@spirent.com (Leslie Gollehon) — Tue, 26 Jan 2010 12:00:00 GMT

Is your network ready for telecommuting?

Institutional resistance to working remotely (aka telecommuting, telework) continues to crumble. While the H1N1 virus may force the issue, it will merely be the final drop that breaches the dam. There are many drivers and corresponding benefits for the organization that adopts telecommuting.

Energy savings. Going green saves resources (and dollars) for business and personal budgets. The company can light, heat and cool fewer square feet and power fewer computers and monitors. The workers spend less on gas for the commute. They might even be able to downsize their personal fleet, saving not only on the cost of the extra car but the associated insurance, taxes and maintenance costs.
Minimizing disruptions to business. The long list of things that can disrupt your business by interfering with a commute range from the infrequent (natural disaster, pandemic, terrorist attack) to the regular occurrence (winter weather, traffic). Such disruptions are now completely avoidable. There is no longer any reason why the inability to commute means the inability to work.
Increased productivity. No more extended water-cooler discussions or co-workers camping out in a neighboring cube to chat. Commute time can be replaced by work time, or time to take care of personal business that would otherwise mean time out of the office. Studies indicate that many employees tend to work more, even after hours, when the office is a few feet away instead of several miles away.
Stealth pay raise. Supporting remote work effectively gives employees a pay raise (by eliminating significant personal expenses) without the cost of increasing salaries. In addition, the quality-of-life dividend can result in greater job satisfaction and company loyalty.

However, issues must be addressed when making the changes necessary to support the workplace of the 21st century.

Infrastructure. With a few decades of development behind it, the technology may be the easiest part of the puzzle. However, verifying that it is configured properly and will work reliably (not only during normal business but also under peak usage such as during severe weather) is another matter. Proper testing by a third-party is the best way to assure that your system for maintaining productivity during a disruption doesn’t itself become disrupted (see H1N1).
Management. Perhaps more challenging than the technology, and the reason many organizations are slow to adopt, are the human resource issues associated with managing a remote workforce. However, the reality is that you don’t guarantee productivity simply by forcing people to drive to a central location and sit in close proximity while working on disparate tasks. There are many publications and websites, such as www.telework.gov, that provide guidance for managing this aspect of the transition.

Organizations that make the strategic decision to include telework in their culture are positioned to thrive in the 21st-century marketplace. Organizations with no telecommuting infrastructure or a system that is either unproven or not designed to scale will increasingly find themselves at a competitive disadvantage.

Best practices indicate that the time to finalize the organization, communication, and implementation of the technical and management infrastructure for telework is before a disruptive event that will stress the performance. As always, thorough testing is the key to success (see H1N1).

Testing 10G Data Center Switches: Scaling 10 Times Higher

Leslie.Gollehon@spirent.com (Leslie Gollehon) — Mon, 18 Jan 2010 17:33:00 GMT

If there’s one overarching conclusion I’ve drawn from three months of testing 10-gigabit top-of-rack data center switches, it’s that “switch” and “data center switch” are very different beasts.

Understanding the latter means testing new features like virtualization support and storage/data network convergence, while also driving unicast and multicast scalability benchmarking to new heights.

In a project recently published in Network World, we compared switches from six vendors, each with at least 24 10-gigabit Ethernet ports. We compared products in 10 areas: features; usability; power consumption; MAC address capacity; forward pressure; multicast group capacity; multicast group join/leave delay; link aggregation hashing fairness; and, of course, basic unicast and multicast performance.

Performance testing remains as important as ever, if not more so, when it comes to data center switching. That’s an important point: Buyers are interested in these new features, to be sure, but only in addition to switches’ long-time role as fast packet pushers.

In other words, the same industry-standard methods of benchmarking switching and routing performance (as defined in RFCs 2544, 2889 and 3918) remain vitally important in the context of data center switching. In fact, line-rate performance and low latency and jitter are even more important for many data center applications than for general-purpose enterprise networking.

Data center switches tend to have much longer features lists than their wiring-closet counterparts. We paid special attention to three areas. First is redundancy protocols. Some data center switches offer new methods to connect multiple servers and/or switches. Some methods even eliminate slower redundancy protocols, such as spanning tree. Others offer “active/active” connectivity across multiple links until a failure occurs, boosting bandwidth in a way that “active/standby” protocols such as spanning tree cannot. While new protocols are intriguing, it’s a good idea to test their resiliency and benchmark failover times before deploying them in your network.

Second, some switches allow the convergence of previously separate storage and data networks using technologies such as Fibre Channel or Fibre Channel over Ethernet (FCoE) on the same switch. The IEEE has defined several new protocols to accommodate the FCoE’s stringent delay and loss requirements. We didn’t test these protocols this time, since only one switch tested supported all these new mechanisms, but look forward to comparing data/storage performance in upcoming tests.

Finally, data center switches support virtualization in a variety of ways. Since virtual machines (VMs) often move between physical hosts in the data center, some switches offer the ability to have the VMs’ access control policies move with them. Other switches can carve up physical interfaces to appear as multiple logical links to different sets of VMs. And some support end-to-end management of physical and virtual switches, offering the same set of capabilities for both.

I’m looking forward to future tests comparing Fibre Channel and FCoE performance as well as even larger-scale tests of large modular data center systems. We’re still in the early days of data center switch testing, and things will only get bigger from here.

Newman is president of Network Test, an independent test lab and engineering services consultancy. He can be reached at dnewman@networktest.com.

Has Your Network Had Its Flu Shot?

Leslie.Gollehon@spirent.com (Leslie Gollehon) — Thu, 17 Dec 2009 17:43:00 GMT

As of this writing, the World Health Organization H1N1 virus pandemic alert is still at phase 6, the highest level, and has been for six months. The WHO defines the top two phases as follows:

Phase 5 is characterized by human-to-human spread of the virus into at least two countries in one WHO region. While most countries will not be affected at this stage, the declaration of Phase 5 is a strong signal that a pandemic is imminent and that the time to finalize the organization, communication, and implementation of the planned mitigation measures is short.

Phase 6, the pandemic phase, is characterized by community level outbreaks in at least one other country in a different WHO region in addition to the criteria defined in Phase 5. Designation of this phase will indicate that a global pandemic is under way.

The US Center for Disease Control (CDC) reports widespread activity (the highest level) in 46 states and regional activity (second highest level) in the four remaining states for the week ending 11/07/2009.

The vaccine continues to remain in short supply, with specific groups targeted for the early doses, such as pregnant women, children, and healthcare and emergency personnel.

The CDC estimates that up to 40% of employees could be absent from work during a severe pandemic, either sick themselves or caring for sick family members, and recommends social distancing strategies, such as telework, to mitigate the spread.

This raises the question of whether an organization’s communications infrastructure can support 40% or more of the workforce working remotely. In June 2009, the Senate asked that question of the Government Accountability Office (GAO) and got a disturbing answer. Of the 24 CFO Act agencies, only one, the National Science Foundation, had done extensive testing of its IT infrastructure. They reported assessing their telework system formally several times each year and each day through various means. Five more agencies reported testing their IT systems little or none.

Unfortunately, many businesses are in the same position, with no telecommuting infrastructure or a system that is either unproven or not designed to scale to the usage a severe pandemic would impose. As the WHO Phase 5 recommendations and best practices in the industry indicate, the time to finalize the organization, communication, and implementation of the planned mitigation measures is before it hits. Once you have high levels of absenteeism, it will be much more difficult to get a system into place and lost productivity and revenue are no longer theoretical.

There are many reasons beyond H1N1 to implement telework in an organization, including the flexibility that makes for more satisfied and productive employees, reducing Opex costs for office space, going green (reducing energy consumption related to commuting and the heating, cooling and lighting of office space) and preparedness for other crises, such as severe weather, blackouts or other public infrastructure failure, but H1N1 may force many organizations to make the investment sooner rather than later.

Here are a few tips for those looking to validate a new or existing telework infrastructure. Your organization is likely already stretched to the limits of individual productivity with day-to-day operations and ongoing projects. Diverting staff to take on a large project will affect schedules. Instead, look to a third-party to handle the project, especially one with telecommunications testing expertise. While you’re in the middle of an absenteeism/telework crisis is not time to discover a gotcha that was overlooked while validating the system due to lack of experience.

Another tip – when it comes to the test system, lease, don’t buy. Assessing a telework program is an annual effort, not a day-to-day activity. There is no need to expend Capex funds for a test platform that will scale sufficiently to test your system but then will sit idle for the rest of the year. Either lease the system or select a testing service that includes use of a test system.

The bonus is that preparing for the flu can deliver benefits beyond a crisis event by enabling the advantages of telecommuting for your organization.

End Of Life Questions: Automation and Transition Plans

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 05 Nov 2009 17:20:00 GMT

An end-of-life (EOL) announcement on a product that is an integral part of your test strategy is disruptive to schedules, and possibly revenue, as gap analysis is done and transition plans are formed. On the other hand, it forces you to evaluate your lab infrastructure against available solutions, an important exercise that is often pushed to the back burner in the urgency of day-to-day operations and task oversubscription.

Automation

In the test and measurement world, investment in test automation is one of the most significant contributors to resistance to change, and also one of the most significant enablers for cost-savings and faster time-to-market. In the last few years, innovations in automation have expanded the industry beyond test case automation to offer dramatic schedule and productivity gains through automating test cycles, the test environment and the physical infrastructure.

A product that is in EOL may not offer this level of automation support, and if not, then it likely won’t in the future. Eventually it will also be inadequate for your testing requirements, probably well before the EOL date. It’s best to deal with that sooner rather than later, and when you do, automation should be a major factor in your decision.

There may be promises of porting the API of the EOL product to another platform, but porting an API is a high-risk proposition potentially fraught with compatibility and interoperability issues. Test tools developed by different vendors, or even two different tools developed by a single vendor, don’t have a one-to-one correspondence between features and configuration parameters. Tests on the EOL platform may not be supported on the target platform, or may not be configurable to the same detail.

The reality is that porting an API is not likely to offer efficiencies over transition to any other new platform. It’s simply one transition option among several.

Transition Plans

There are many considerations when building a transition plan, such as budget, disruption to existing development schedules, and how quickly test requirements outstrip waning development on the legacy platform. It is a mistake to allow a five-year EOL plan to reduce your sense of urgency. From all perspectives, an aggressive strategy for evaluating alternatives and navigating the transition is best.
In an EOL situation, there are several key factors to evaluate when selecting a target platform for your test strategy:

Coverage for existing requirements (protocols and test cases)
Roadmap for future development
Productivity tools, such as test wizards, to simplify configuration of complex and realistic test environments
A comprehensive automation environment
Professional services support

A transition plan is greatly enhanced when the target platform has a single API that supports GUI-based and script-based automation. Such a “configure once, run anywhere” capability can dramatically increase productivity and reduce configuration errors.

An EOL announcement is usually unwelcome, but it can also be a wake-up call to find the best-of-breed solution on a platform with a clear future and a commitment to supporting technology at the bleeding edge. Not only will your productivity and reliability improve, you’ll avoid facing another EOL situation in the future.

End Of Life Questions: Alternatives and Sunk Cost

Michael.Lynge@spirent.com (Michael Lynge) — Tue, 03 Nov 2009 15:00:00 GMT

Alternatives

Every buying decision is an opportunity to evaluate the current state of all the alternatives on the market to indentify the best solution for your requirements. Unfortunately, the reality is that collapsed schedules and budgets can result in taking the path of least resistance, which leads to considering only the familiar.

Buying more of what you bought before seems reasonable. After all, you evaluated the options at the time, found the best fit for your application, and bought it. But the market is not static. Your products don’t look the same as they did a year ago, and neither do the solutions for your test lab. An EOL product, on the other hand, was likely in decline for some time before the announcement, meaning the product probably already lags the market now, and will increasingly fall behind in features and performance as the market continues to move forward.

An EOL announcement often provides the inertia to do what should be done every time you invest in your lab – investigate all the options to make sure you have the best solution for your application.

Sunk Cost

Sometimes the motivation behind buying more of what you have isn’t hectic schedules or inertia, but sunk cost – the money already spent. But sunk cost is as much an emotional barrier as a financial consideration. The desire to leverage sunk cost creates resistance to change, which can override the opportunity to take advantage of productivity and performance gains available from other platforms.

In an EOL situation, change is inevitable. You can’t leverage sunk cost for future testing because the EOL platform won’t support future technologies. A transition is imperative and it won’t be any cheaper or simpler a year in the future. In fact, if you face this decision now, in a year you could be in a much better position, with productivity gains, a higher-performance test bed and a roadmap that supports your current and on-going development plans.

End Of Life Questions: Development and Support

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 29 Oct 2009 19:51:00 GMT

Support

The key issue in creating an EOL response strategy is support. It sets the ultimate deadline for the completion of your transition plan. However, in most cases the best strategy is to begin the transition sooner rather than later.

An EOL decision on the part of a vendor signals that the business case for the product is no longer viable. As such, diminishing resources will be devoted to support until it hits the EOL date, at which time support is no longer available. Issues that require bug fixes or development are likely to be a low priority, or even a no priority, for the vendor, leading to frustrating support calls with an unsatisfying resolution when problems arise during testing. The impact to development schedules and delivery/deployment dates can be significant.

Development

The problem of diminishing support is exacerbated by diminishing, or non-existent, development for new technologies. A vendor is unlikely to add new protocol coverage, or even enhance coverage of an existing protocol, on a platform that has been officially identified as lacking a viable business case.

As we all know, the telecommunications industry is a fast-paced environment where working groups and committees constantly issue new drafts of recommendations and standards for cutting-edge technologies to address the challenges of delivering faster and more diverse content to consumers with endlessly expanding expectations. An EOL target of five years in the future isn’t the gating factor in your decision for when to start the transition. It’s the date when the schedule says you have to test a must-have feature that can only be delivered by supporting that new draft.

Going beyond protocol support, there are the issues of features and performance. A product under development introduces new features to enhance productivity and ease-of-use. A product in EOL leaves well enough alone, meaning you won’t be able to take advantage of productivity gains provided by a platform in active development.

In addition, new technologies place increasing demands on the solutions you develop, requiring greater power and performance from the device or network you deliver. This places greater demands on your test infrastructure to deliver corresponding performance increases. If the test platform can’t keep up, your budget or your schedule, or both, will suffer. A product in EOL will not have the benefit of on-going focus on the performance improvements your test lab requires. So, once again, the gating factor on your transition decision is not the EOL date, but your testing requirements.

End Of Life or Beginning of Productivity?

Michael.Lynge@spirent.com (Michael Lynge) — Tue, 27 Oct 2009 14:00:00 GMT

The announcement that Agilent is taking the N2X platform to end of life (EOL) in 2015 raises questions about testing strategy for many customers. An EOL announcement is inevitably disruptive to schedules and possibly revenue, but often it causes people to refocus on issues that been unaddressed too long.

Support. The first issue that comes to mind when you hear that a platform you rely on daily is being discontinued is support. What happens when something goes wrong? How long will they fix it? Placing the EOL five years in the future gives you the feeling that you don’t have to worry about it right now. But the reality is that regardless of how far in the future support is dropped, it’s going to be a problem when it happens. The sooner you transition to a platform with a future, the less risk to your development schedules and delivery/deployment dates.
Alternatives. Every buying decision should include looking for the best solution, but frequently the path of least resistance leads to considering only the familiar. An EOL announcement often provides the inertia to do what should be done every time you invest in your lab – investigate all the options to make sure you have the best solution for your application.
New Technologies. A product going EOL in five years may support your requirements for now, but what about that standards committee draft that will be finalized in six months? Is there a plan to support it? Is there an aggressive development schedule to improve performance and support new technologies for the remaining lifespan?
Sunk Cost. In addition to the real expense of transitioning to a new platform, there’s the emotional barrier of sunk cost – the money already spent. The desire to leverage sunk costs creates resistance to change, but in an EOL situation, change is inevitable. The transition won’t be any cheaper or simpler a year in the future. In fact, if you face this decision now, in a year you could be in a much better position.
Automation. In the test and measurement world, investment in test automation is one of the most significant contributors to resistance to change, and also one of the most significant enablers for cost-savings and faster time-to-market. There’s no escaping the fact that the EOL product will eventually be inadequate for your requirements, probably well before the EOL date, and you’ll be forced to change. Whether it’s trying to port an API to an another platform (a risky proposition fraught with compatibility and interoperability problems) or transition to a new platform, it’s best to deal with that sooner rather than later.
Transition Plans. There are many considerations when building a transition plan, such as budget, disruption to existing development schedules, and how quickly test requirements outstrip waning development on the legacy platform. It is a mistake to allow a five-year EOL plan to reduce the sense of urgency. From all perspectives, an aggressive strategy for evaluating alternatives and navigating the transition is best.

Ethernet - Is faster fast enough?

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 22 Oct 2009 15:09:00 GMT

Modern wisdom says you can never be too rich, too thin, or have too much bandwidth. There may be some debate about the first two, but it appears that the third is right on the money.

In the discussion about the need for faster Ethernet we hear of drivers like HD video, mobile backhaul, and point-to-point applications. Social media doesn’t immediately come to mind, but it is a significant factor. (How often have you submitted a tweet only to get the “Twitter is busy” screen?) According to Facebook engineer Donn Lee, many companies need 100G Ethernet in the data center today and could use 400G or 1T Ethernet by the end of 2010.

At the Technology Exploration Forum in Santa Clara, CA in September, Lee said Facebook data centers will require a 64 Tbps data center aggregation layer by the end of 2010. Sixty-four terabits per second. They can implement that with 64 x 1T ports (manageable) or 640 x 100G ports (not so manageable).

The box he needs is a 16Tb switch with 800 x 10G ports from the server clusters aggregated into 80 x 100G (or 8 x 1T) ports to the data center aggregation layer. And he needs eight of them. Lee says that the top 25 websites are probably in the same situation.

How could they need so much so quickly? Phenomenal growth in the user base. Facebook has over 300 million users. It took three years to get the first 100 million users, but only six months to get the last 100 million.

Users (Million)	Time to Acquire
100	36 months
50	12 months
100	9 months
100	6 months

Regarding the arrival of 100G Ethernet, instead of being just in time, it might be, ”It’s about time. When can I get more?” Which points to the urgency of time to market for 100G solutions. Lee may find 640 x 100G ports to be cumbersome to manage, but he’ll probably be happy to get them instead of being forced to use 6,400 10G ports.

Lee also alluded to the challenge of implementing Layer 3 at such high speeds. The issues surrounding upper-layer services at high-speeds, whether 10G, 100G or 1000G, are complex and will require breakthroughs in technology and creativity. Exciting times are ahead.

Donn Lee’s presentation via EETimes.com: http://link.brightcove.com/services/player/bcpid1701276884?bclid=1622640422&bctid=40363249001

Testing Virtualization

Michael.Lynge@spirent.com (Michael Lynge) — Sat, 19 Sep 2009 20:03:00 GMT

It’s no secret what’s driving the move to virtualization in data centers. The demand for new and expanded software systems is growing, but the geographic and carbon footprint required for scaling underutilized dedicated servers is too costly on many levels.

This issue has led to the maturation of the virtual server, where increased reliability and stability means reduced risk, making virtualization of the data center a viable solution. Virtualization makes it possible to replace physical servers running at 10% capacity with fewer, more powerful servers running at 60% capacity or more.

When working with a dedicated server, it’s a fairly straightforward process to characterize performance and isolate factors that affect it. You use a test system to emulate realistic users, traffic and network conditions and measure the response times of the application with the test system.

When working with dozens of virtual servers in a single physical server, it’s not as straightforward. A connection from the test system to the physical server won’t provide the granularity of testing required for meaningful results. The single physical interface handles traffic for many VM instances, making it difficult to isolate and measure the performance of each VM instance or the performance of the virtual switch.

We need visibility into the performance of each VM, but how can we get it? By creating the capability to capture results and generate traffic from within the virtual server instance. We need a virtual test system inside the virtual server.

Read the rest at the Virtualization Journal

The 40/100G Ethernet Testing: Business As Usual?

Michael.Lynge@spirent.com (Michael Lynge) — Sat, 19 Sep 2009 19:47:00 GMT

Everybody’s talking about 40/100G Ethernet. It will replace SONET in a decade. It’s a revenue playground for those with the vision and expertise to deliver in a timely fashion with quality and reliability. This group is doing a trial. That group has released a product.

And, from a market perspective, the time for 40/100G is here. Due to the abundant proliferation of mobile applications and the explosion of IP video, not to mention a host of other applications dumping new traffic on the network, the fiber overbuild of the past decade has gone from feast to famine, as far as bandwidth is concerned. For those who want to expand the revenue stream to accommodate these services, 40/100G Ethernet is clearly the path forward.

When it comes to trials and implementation, everyone talks about lane timing and skew to solve the problem of moving bits at 100 Gbps, error free. But equally challenging, if not more so, is scaling the upper-layer engines to deliver services and quality of experience at four to ten times the current rate. Brand credibility and customer confidence depend on solving both problems.

This of course means testing. The sheer speed required to send data and to support services at 40 Gbps and 100 Gbps might not just break network devices, it might even break the testers used to validate the performance of services on those devices.

Because a trial is only as good as the test platform. If the tester can’t accurately count packets and measure latency and jitter at 100 Gbps, it could pass systems that won’t stand up to real-world traffic in a real network.

To learn more about the issues associated with testing 40/100G Ethernet, take a look at this whitepaper. [Testing 10/100G Ethernet] It explores the challenges in the physical layer, limitations in the test system, the budget in the test lab, and delays in the development/deployment cycle.

Because passing a trial but failing in the network can ultimately cost a vendor or provider a lot more than a place in the 40/100G Ethernet revenue playground.

Going Green in the Data Center

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 27 Aug 2009 20:34:00 GMT

Maybe it’s not easy being green, as the infamous flannel frog pointed out decades ago, but being un-green is no picnic, either. Many data center managers rate their top three concerns as security, availability and energy.

Data center costs, including energy, have climbed eight-times since 1996. When high-density blade servers can push power requirements past 30 kilowatts per rack, the transition to more efficient systems can have a huge impact not only on energy usage, but operating expenses. Especially considering that a watt saved in power is another watt saved in cooling. It’s like a save-one-get-one-free sale on watts!

The more efficient power supplies now available mean less power wasted in conversion and distribution. However, many of the energy savings are only available for those who enable the power management for their servers, something many data centers don’t do. The reliability of the power management tools concern some managers, as uptime and performance are key issues for any data center.

When it comes to servers, the metric of interest is the correlation between utilization and energy. Legacy servers tend to use at least 30% of peak power even while idle. Obviously, for maximum efficiency the server should run at greater than 30% utilization, but 6% to 15% is more typical. The move to virtualization partially addresses this concern with more powerful blade servers running many virtual machines at higher utilization, allowing administrators to find the sweet spot between performance and power.

Virtualization, multiplay, and network and service convergence, all result in a greater demand being placed on the network infrastructure. Devices may be decreasing in power as it relates to energy usage, but they are increasing in another kind of power - performance, features, protocol depth, and number of queues, routes, sessions, or tunnels supported. And this increase in the density of functionality and performance per rack places a greater burden on test systems.

In the greening of the infrastructure, test systems are often overlooked. But test systems are not and should not be exempt. They should also deliver more power (functionality and performance) while consuming less power (energy).

You should ask some pointed questions when it comes to your test systems:

Are power requirements (per port and per rack) decreasing?
Are functionality and performance (per port and per rack) increasing?

As you simultaneously reduce your carbon footprint and your operating expenses, make sure your test systems are working with you.

Why Test Realism Matters Or When traffic goes up and websites go down

Michael.Lynge@spirent.com (Michael Lynge) — Tue, 25 Aug 2009 12:00:00 GMT

Maybe it only seemed like the world stopped for the Michael Jackson memorial on July 7, but there’s one thing we know for sure that did stop.

The City of Los Angeles set up a website to accept donations to offset the $1.4 million required to support the memorial with services like security, traffic control and sanitation. According to CNN, the website crashed repeatedly Tuesday and Wednesday, sometimes for periods as long as 12 hours. Clearly they didn’t test the system to verify it could handle high volumes of traffic. That’s understandable, given the unexpected nature of the event.

During the same time period, a spike in traffic caused by a fare sale at Southwest Airlines resulted in outages and poor performance on their website from 10am to 3pm. The headline read, “Southwest Air Won’t Extend $30 Fare Sale After Web Site Lockup.” Unlike the City of Los Angeles situation, this event was not unexpected. No estimates of the cost of lost business were provided, but five hours of problems during the middle of the business day undoubtedly took a big bite out of revenues, not to mention customer goodwill and corporate reputation.

And that is why test realism matters. Most corporations subject their ecommerce infrastructure to extensive testing before going live. Most likely Southwest did. But it’s very easy to test a system and never find the failure points lurking within. It usually goes like this: set up a test network, hook up some packet blasters and bombard your system with traffic. The results show good performance from the canned traffic and it’s all thumbs up until the system hits the real-world. Then the wheels come off.

Test realism avoids this unfortunate fate by hitting the system with the real-world before deployment. As mentioned in an earlier post [Test Realism], test realism involves:

Real user behavior. The flexibility and sophistication to emulate a wide range of user behavior, both benign and malicious.
Real converged traffic. The power to create line-rate, fully-emulated, stateful traffic across hundreds of ports.
Real network conditions. The power and complexity to create the dynamic, time-varying conditions found on deployed, production networks.

In addition, test realism includes a test methodology based on decades of experience with the technology and application. Of course, you need a test system with the power and scalability to create a real-world environment. But it’s possible to have that test system and still create unrealistic tests. That’s why the test system must have expertise built-in, in the form of test wizards that follow best-practices test methodologies, and also available via professional services for more customized testing.

Unfortunately, in the telecommunications world the lack of real-world testing can result in high-visibility, non-career-enhancing failures. It’s nice when your company is making headlines, but this is one headline you don’t want to be in.

Test realism. Don’t leave the lab without it.

UPDATE: And the hits just keep on coming. At end of July, the website for the “Cash for Clunkers” federal rebate program crashed 15 minutes after it went online due to overwhelming demand. It was down for 5 hours.

IPTV - What to Test

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 13 Aug 2009 16:00:00 GMT

IPTV is on the rise, more quickly in some places than in others. While some areas have already hit double-digit market penetration, others, like the US, are still in single digits. Growth rates vary by region from 20% to over 100%. More than half of the video rights holders and distributors surveyed said they are distributing video online today, and only 10 percent said they had no plans to do so¹.

As the new kid on the pedestal, IPTV must deal with viewer expectations set by decades of broadcast and cable television. It has to prove itself, offering similar or better quality of experience (QoE) while enhancing the value proposition with more features. That’s why testing has been such an important part of the development and rollout of IPTV offerings, like the test EANTC did of Cisco IP Video Infrastructure solutions.

The true litmus test of an IPTV service is QoE, but there are a lot of components that contribute to this metric, and there can be no weak links in this chain.

When it comes to testing IPTV, throughput performance is the most obvious place to start. If the system can’t support the throughput required without issues such as congestion and link or node failure, there’s no point in testing further.

Cable and broadcast TV have the luxury of dedicated wavelengths to guarantee throughput. IPTV, on the other hand, must deal with packet forwarding and contention, making it more difficult to achieve the level of throughput required to deliver standard definition TV, much less high definition. Packet latency, packet loss and packet delay variation all conspire to sabotage service quality. For video, packet loss must typically be zero for an acceptable service. RFC-based throughput testing is the key to verifying that a system or service can deliver the throughput required for high QoE.

IPTV is typically deployed in a multiplay environment. Television video traffic will coexist with VoIP and internet traffic. Some of the traffic may come from high bandwidth applications such as online gaming, file transfers, and P2P sharing. To assure that the solution delivers high video QoE even in the presence of congestion, packet classification and prioritization mechanisms are tested, independently and together. The metrics of interest here are packet latency and packet loss.

Resiliency indicates the ability to maintain service even in the face of a link (the connection between two points of presence) failure or node (entire device) failure. For such failures, it’s not a question of if, but rather a question of when. Redundancy is a method of enforcing resiliency. Regardless of the method of redundancy implemented, for example, redundant headends or redundant video streams, testing is required to verify the response time of the method and its effect on QoE.

The metric of interest for testing a resiliency mechanism is the out-of-service time (OoST). One method of calculating OoST is to count the number of lost frames during a failure event. A more accurate method is to continuously sample the received frame rate for very short intervals and report the duration between falling below an acceptable frame rate and returning to that rate.

That’s an overview of what to test and why when it comes to IPTV. To do the testing you will need a test system that can:

Generate and track stateless and stateful traffic at line rate
Integrate unicast and multicast traffic streams within a single test
Perform video analysis on live traffic is imperative
Collect key metrics such as packet delay, packet delay variations and packet loss in a single test run
Create realistic multi-protocol stacking scenarios, such as HTTP over PPPoE over MPLS

¹ Source: Lightreading.com

When Data Centers Collide - A Modern Fable Of A Lost Lunch

Michael.Lynge@spirent.com (Michael Lynge) — Wed, 29 Jul 2009 19:00:00 GMT

Last week I almost did lunch with a friend, an IT guy. We picked a new deli close to his office and, after a long wait, I finally had a nice Reuben all alone. I texted him and he texted back with apologies and tales of application performance issues.

Being a nice guy, I got a sandwich and chips to go and stopped by his office. I knew I would be welcome. While it might be prudent to beware Greeks bearing gifts most people are glad to see geeks bearing food. Especially when they had to work through lunch because a VP is breathing down their neck about their CRM slowing to a crawl.

My buddy took a short break and told me his sad tale of woe between bites. “We’ve been having issues with our customer relationship management system ever since we installed it. We thought we had it straightened out and picked up on the data center consolidation project.” He paused to wolf down some more sandwich and continued. “Last night we did the cutover from our various corporate data centers to a central facility in Denver. This morning the phones lit up like a radio talk show switchboard. Response times went through the roof. People going to get coffee between screens. I’m not talking down to the break room. I’m talking the Starbucks on the corner.”

I nodded in sympathy and offered him another potato chip.

His situation is not as unusual as it should be. Companies are developing and deploying distributed applications all the time without testing them against realistic WAN conditions. I’ve seen lots of reasons why this happens.

The test plan overlooks the problems that can be caused by a real network. Or they run a few test trials over their production network, usually after hours to prevent business disruptions, and have a false sense of confidence when they don’t see any problems. Or they don’t realize that actual network conditions can be recreated in the test lab.

It’s called network emulation, one of the essential elements of Test Realism.

Test realism lets you do to your application what the WAN is going to do to it, before you have hundreds of users like Kim complaining about your system. The alternative can be ugly. Productivity loss. Finger pointing between the network group and the applications group. Acrimonious meetings. Loss of confidence in your department. Morale issues among your staff.

It’s a shame, really, when, like karaoke, it’s completely preventable.

It is possible to know in advance how your system works on a real network. To test during the relatively calm and rational time of development and testing instead of the high-visibility, company-wide exposure of a post-deployment crisis. To troubleshoot and identify performance and response-time issues early, when it is less expensive to find solutions and implement them.

In fact, it’s not only possible, it’s pretty much essential.

Using test realism in the form of real-world network conditions makes everyday life better for hundreds of thousands, if not millions of people. Kind of like buying a sandwich for everyone.

Realism and Multi-Play - Part 1 NEMs

Michael.Lynge@spirent.com (Michael Lynge) — Tue, 21 Jul 2009 12:00:00 GMT

Find and fix your bugs before your customers find them for you

Remember what Jack Nicholson told Tom Cruise in A Few Good Men? “You can’t handle the truth!”

That’s the kind of thing you don’t want a customer saying about your product. “Sure, it works fine in the lab where everything is well-behaved, but it can’t handle the real world.”

So, what exactly is the real world? In the movie, reality for Nicholson was different from reality for Cruise. My reality is lots of conference calls, business travel, deadlines. Not only can I handle it, I love it. Reality for a soccer mom is very different, and let me tell you right now, I can’t handle that reality, so let’s not do Trading Places.

A while back we talked about test realism which includes:

Real user behavior: wide range of stateful user behavior, both benign and malicious
Real converged traffic: line-rate, fully-emulated, stateful traffic across hundreds of ports
Real network conditions: dynamic, time-varying conditions found on deployed, production networks

But reality for an email server is very different from reality for a set-top box. Which means, if I want to get all the benefits of test realism (like increased test coverage, better product reliability, improved customer quality-of-experience, and fewer customer-found bugs, to name a few) I have to first know what reality is for the system I’m testing. Then I have to be able to faithfully create that reality, fully emulating every user, device and condition in that reality, from end to end, that my system will encounter.

In the multi-play world, reality gets complex. It’s the perfect storm of diverse content and behavior. So, how do I get that complex reality in my lab?

First, I need the scalability to generate a city’s-worth of voice calls, IPTV traffic, and data traffic for thousands or millions of users across hundreds of ports. But the ability to scale to reality in number of ports and volume of traffic, while important, isn’t by itself realism. That’s just packet blasting from a lot of ports. It’s a brute-force assault that is fairly easy to do but doesn’t tell me anything about how my system will handle reality. Which leaves me at risk for Nicholson-type comments from my customers.

Test realism for multi-play means the traffic is generated by fully-emulated devices that generate stateful voice, video and data traffic, including signaling and routing. Emulated devices that respond to queries, requests, and commands from my system.

Second, I need that traffic to come from real users. An artificial blend of dummy transactions based on a percentage or distribution doesn’t create the conditions in the system under test that is created by real users. I need stateful traffic from users doing things in the order that users do them, whether they be consistent (normal usage), seemingly random (multitasking or short attention span) or malicious (security risks).

Third, I need the ability to create the imperfect world my system will exist in, one where packet delay, loss and other unpleasant things sometimes occur. A pristine lab network is not reality. And I need this at wire rate, and when I say wire-rate I am talking at 10G speeds!

Test realism for multi-play means my system is forced to deal with the dynamic, time-varying conditions caused by likely events such as routing table updates, queuing and buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.

Fourth, I need one system that can emulate all of this realism: users, traffic, and network. I don’t need a solution that requires me to patch together a dozen or half-dozen different systems to achieve an approximation of realism that might work as long as there are no conflicts between the various test systems and how they emulate or measure their niche in the test. I need one integrated system with one GUI and one API and one correlated set of results and the power to support all the elements of test realism.

Fifth, I need trickle-down expertise built right into the test equipment that guides me. I need to know I am buying a solution from a company that knows what to test and how to test it. Because, whether painful or not, I have to know the truth before my customers do. Even if that truth is that the performance of my system isn’t as good as my unit test led me to believe. Especially if it’s not as good as I thought.

The best test solution is like a true friend. Not the phony friend that tells you what you want to hear, but the true friend that will tell you the truth, even if it’s harsh, and then will stick with you to fix it.

So I’m looking for expertise built into the test gear, into the software interfaces, and into the support and services I get from my vendor. It’s got to be a complete package. This expertise comes from (1) participating in standards bodies to create protocol and testing specifications and (2) by successfully supporting a user base of thousands of customers, year after year. I need a vendor who is going to be around tomorrow for the long term to support me.

That’s a lot to ask from a test system. But there’s a lot at stake. And when it comes down to it, you really do need to know the truth. And your solution really does need to be able to outperform the competition in the real world.

The truth is out there. Can you handle it?

Test Case Automation – The Gift That Keeps Giving

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 16 Jul 2009 11:00:00 GMT

A while back we talked about test lab automation (see Advanced Test Automation), which is returning sanity to test labs all over the globe. In that post I made this statement:

"If you haven’t automated your tests, you’re missing one of the best ways to boost productivity and impress your boss. (And the money guys, too.)"

It occurs to me that there might be outposts of resistance lurking in the hinterlands that have yet to reap the many rewards available from test case automation.

As I list the many benefits of test automation, you may start to wonder if I’m going to claim that it also trims 20 lbs from your waistline and restores hair loss. Not exactly, although it might keep you from tearing out your hair, so maybe there’s something to that after all.

But seriously, the benefits are there. How about fewer customer-reported bugs, expanded test coverage, increased productivity and lab usage, reduced operating costs, shorter test cycles, and more. Let’s look at some hard numbers:

A lab I visited recently was testing their 10Gig MPLS architecture for approximately 20 different test cases. Test automation freed up 3.5 engineers to add more test cases and expand test coverage.
Another lab was testing RP switchover convergence. We found that it took 35-45 minutes to run a single test case. With more than 20 iterations for each test, one test could take an entire work day with an engineer babysitting it to keep it going. The test was automated to eliminate the need for user interaction. This change produced a 40% productively gain – nearly half a person’s daily activity recovered by automating a single test suite.

This is huge! Can you think of any one single thing you can do that will increase your productivity by 40%? Personal organizer? No. Time management? No. Multitasking? No. Meditation? No. Cloning? Maybe.
Now you see why I call it the gift that keeps on giving. From now on, every time they run that test, they’re saving a boatload of time and money based on a one-time investment. It gives me a warm fuzzy just thinking about it.

One of the real values of scripting is to increase test realism. Scripting means you can precisely control the test environment during execution to recreate the dynamic behavior of a real network. And of course, the more realistic the test, the more meaningful the result.

Which is the point, after all. You want to know your system can handle the real world. It’s not useful to know that it passes all the tests in an artificially favorable environment built with fixed stimuli and static topologies, and then ship it or deploy it and have your customers find the problems for you!

Now that you’re ready to join the club, let me give you a tour of what’s out there and what to watch out for.

Scripting. You write a program, usually in Tcl, to configure the test equipment and the system under test and then to start the test.

Pros: If the API is robust, you have more control over the test than you would running it from the GUI.
Cons: Must learn a scripting language.
What to look for: A test system that allow changes to the configuration during test execution, so that a realistic test can be created.

GUI-to-script. You configure a test with the test system GUI, then save the configuration as a script which may (good) or may not (bad) be modified to customize the test.

Pros: No need to learn scripting, especially if you’re not customizing.
Cons: Configuring through the GUI can be time-consuming. Unless you learn scripting, the tests will be of limited use, basically configure and go.
What to look for: Support for script-to-GUI, useful for troubleshooting configuration issues after tweaking a script.

GUI-based automation editor. You create automation by dragging and dropping commands into a visual command editor, popping up windows to configure parameters.

Pros: Create sophisticated, realistic tests without learning a scripting language.
Cons: None.
What to look for: Tight integration between GUI commands and API commands, so that automation created in one interface can be used in the other.

So, pop on that lab coat and start automating those test cases. Your boss and your bottom line will love you for it.

40/100 GbE Ready

Michael.Lynge@spirent.com (Michael Lynge) — Thu, 09 Jul 2009 11:00:00 GMT

Only 176 shopping days until Christmas. And, before you know it, you’ll be testing 40 GbE or 100 GbE solutions. These things tend to creep up on you.

You may not be the type to think about Christmas shopping in July. I know I’m not. But when it comes to investing in a test bed, it pays to plan ahead. And the next generation of high-speed Ethernet poses significant challenges for legacy test platforms.

Synchronization and timestamps are two of the challenges posed by high-speed testing.

Synchronization. A test is all about results, but results are meaningless if they aren’t synchronized. At high speeds, synchronization becomes even more important. In a 100 GbE environment that can send one frame every 6.72 nanoseconds, nanosecond resolution is required. And synchronization has to be maintained across all test ports, whether they are in one chassis or spread across multiple chassis. Without synchronization, results from different ports, blades or chassis cannot be correlated to isolate anomalies during troubleshooting, bug detection and isolation.

Timestamps. Another critical capability of network test equipment is the ability to accurately timestamp packets or frames. Timestamps are used to measure latency and jitter. Every frame must have a unique timestamp. At higher speeds, legacy test platforms may not be able to uniquely mark packets because the internal timestamp clock is too slow. If two or more packets fit into one timestamp clock tick, then those packets are marked with the same time, which renders latency and jitter measurements meaningless.

Many legacy test systems don’t have a clock with the speed and resolution to test 40 and 100 GbE.

So, when you’re putting together your shopping list, make sure you get on the right side of the naughty/nice list by select a system that will still be useful when you’re ready to test 40 GbE and 100 GbE. Look for a system with a timestamp clock with the granularity to support the speeds you’ll be testing soon. Also, it won’t hurt to make sure that it automatically syncs and adjusts to provide latency measurements with nanosecond accuracy between ports, modules and chassis, regardless of operating temperature change and cable lengths.

Test Realism - Find and fix your bugs before your customers find them for you

Michael.Lynge@spirent.com (Michael Lynge) — Tue, 07 Jul 2009 07:49:00 GMT

Expectations. When I turn on the faucet, I expect water. When I flip a switch, I expect the light to come on. When I pick up the phone, I expect to be able to make a call. When I punch the remote, I expect to see my show. All pretty much the same, right?

Almost. Water and electricity are delivered to us in basically the same way they were a century ago. But delivery of phone service is changing, and delivery of television, which didn’t even exist a century ago, is being radically transformed.

Regardless of how the infrastructure has changed, customer expectations haven’t. When I punch the remote, I still expect to see my show. Sure, all the new technology that makes it work is cool, but when my team is down by two with five seconds on the clock, the score I’m focused on is not a MOS-V score.

Only through testing can you meet reliability and quality expectations. Of course. Everybody tests.

So why is it that customers still have reliability and quality-of-experience problems? Because it’s not if you test, but how you test.

You avoid customer-reported problems with realistic testing. Strategies used in the past, such as packet blasting, aren’t effective in predicting how applications will work on a converged network. Throwing traffic at your system, whether on the data plane (short packets at line-rate), control plane (massive route updates) or subscriber signaling (high setup-teardown rates), doesn’t tell us a thing about how the system will handle Black Friday shopping or the final vote on American Idol.

Realistic testing means re-creating the environment that the application lives in, from the provider to the customer, in all its dynamic and daunting complexity. Test realism means:

Real user behavior. Users are as unique as their fingerprints. They vary significantly in how long and in what manner they navigate through an application and how they respond to sluggish performance, picture and voice dropouts, dropped calls, and other problems. They violate usage and security policies in different ways. They find unique ways to break your system.

Realistic testing means the flexibility and sophistication to emulate a wide range of user behavior, both benign and malicious.

Hitting a device with a mix of traffic types (say, mixing HTTP requests/responses with IPTV channel changes and P2P file sharing or gaming traffic) isn’t emulating user behavior. It’s just hitting it with a mix of traffic. Emulating real user behavior means supporting stateful traffic that emulates how a user operates, including think time, click-through, abandon, channel surfing, etc. It means good users and malicious users simultaneously attempting to achieve their good and bad goals.

User-centric traffic tests the performance of the device in a real environment. Queues, buffering, and other mechanisms behave differently depending on the order and the nature of the transactions. An arbitrary (and artificial) static mix of messages, or even a dynamic mix of messages that don’t account for the stateful nature of user connections, will not stress the system the way real users do. As a result, failure points can remain undetected until the real users start using it. Which is exactly the wrong time for that to happen!

Real converged traffic. Single-purpose networks are relics of the past. The consolidation of networks onto Carrier Ethernet and MPLS enables mobile and fixed-line voice and data, residential video, and MPLS-based VPNs. The different types of traffic carried on the converged network have different characteristics and requirements, but they all travel on the same path, dependent on CoS and differentiated traffic rules to keep everyone happy.

Realistic testing means the power to create line-rate, fully-emulated, stateful traffic across hundreds of ports.

Real converged traffic not only means a realistic mix of traffic types, but also realistic traffic encapsulation. For example, if the deployed system tunnels user PPP sessions over MPLS, then testing PPP setup-teardown rate and throughput performance without MPLS is not real converged traffic. It’s a dangerous shortcut that will mask problems your users will discover after deployment. Real converged traffic means emulating the actual deployed topology, regardless of how complex or simple, including all encapsulations.

Real network conditions. The network creates time-varying conditions that are linked to a complex set of conditions, influenced by routing table updates, signaling protocols, queuing algorithms, buffering, traffic management and policing policies, malicious attacks, EMI and other environmental factors.

Realistic testing means the power and complexity to create the dynamic, time-varying conditions found on deployed, production networks.

Real network conditions can’t be emulated through static rates of delay/loss or distribution-based models of impairment. Real networks don’t introduce impairments at fixed rates or follow neat curves. They behave in seemingly non-deterministic ways due to the number of factors affecting them. Testing under real network conditions means emulating this complexity to discover issues before the real network finds them for you.

Test realism gets you closer to the goal of delivering reliability and quality of experience, which is the ultimate differentiator in any market. Oh, hold it, the game is on. I’ll be right back.

Advanced Test Automation: Making the Case for Test Lab Automation

Adam.Reese@spirent.com (Adam Reese) — Thu, 02 Jul 2009 17:20:00 GMT

Hold the phone! Stop the presses! Suspend the blogs! Abate the tweets!

Some breaking news: Testing is essential for achieving product quality, improving customer satisfaction, reducing costs and increasing revenue.

OK. So that’s not news. It’s pretty much common knowledge.

How about this: Test lab automation is an incredibly effective (but often overlooked) strategy to significantly reduce operating costs while at the same time improving test lab utilization, test coverage and product quality.

What? You say you already use automation to script all your tests? Good, because that’s a huge step to increase efficiency and reduce cost. If you haven’t automated your tests, you’re missing one of the best ways to boost productivity and impress your boss. (And the money guys, too.)

But I’m not talking about test automation. I’m talking about test lab automation. Aha!

See if this sounds familiar. A test lab with many systems to be tested and a range of test equipment. They are connected either directly or through a patch panel. Every time a test configuration changes, somebody has to go in and rewire the physical connections, hopefully not making a simple mistake that will eat up an hour or two in debugging time later on. Then the systems under test must be configured to support the particular test, or series of tests, for this session. Once again, hopefully with no time-wasting errors. Add to this scenario increased demand for access to the test lab from groups that may be anywhere in the world and may require on-site assistance with the physical setup and configuration, or with power-cycling a system at some odd hour if something goes wrong.

It’s not a pretty picture.

This kind of setup, while common, is inefficient and makes it difficult to share, track and manage test lab resources. And it will continue to get worse as the number and types of interfaces and technologies to test increase, along with the demands for access from remote-site teams, telecommuters and off-shore developers.

And don’t forget collapsed development cycles, increased time-to-market pressure, lab consolidation and whatever else hits you tomorrow.

Which is why lots of organizations are turning to test lab automation to stop the madness and get some sanity back in the test lab.

Instead of what you’re doing now, consider this: A test lab where the systems under test and the test equipment are wired into physical layer switches that serve as programmable patch panels. An application on a PC (anywhere on the network, local or around the world) is used to make test topology changes, power cycle devices, search for and substitute alternate devices or ports if problems arise, configure DUTs and test devices, and schedule tests.

Now that’s a picture you could learn to like. Am I right?

Some pioneers have created what we might call the first generation of lab automation: home grown, ad hoc solutions. They’re a start, but suffer the shortcomings of all in-house efforts: usability issues, lack of scalability, limited or non-existent user support, and no path forward for enhancements and expansion.

The next generation comes from vendors who have been at the game for some time and have worked through the various customer challenges associated with lab automation. Second generation lab automation is characterized by holistic and product-agnostic solutions that simplify your life and accommodate the wide range of brands and technologies you already have in your lab.

Now that might be a stop-the-presses moment after all!

Unified Threat Management

Sailaja.Tennati@spirent.com (Sailaja Tennati) — Wed, 24 Jun 2009 03:21:00 GMT

You don’t need me to tell you that realistically testing a Unified Threat Manager (UTM) can be a nightmare. The convergence that is happening on so many fronts offers a utopian future of lower OPEX and streamlined services, all great for the bottom line. But under the hood, the dirty little secret is a more-than-compensating increase in complexity, and security solutions are not exempt.

A UTM squeezes multiple security functions, including content filtering, spam filtering, intrusion detection, anti-virus protection, security and policy management, into a single box. But whether it takes up a whole rack or one box, somebody has to make sure it actually (and efficiently) does what it’s supposed to do. And that somebody is you.

You need a tool that tells you if your system keeps the bad guys out while letting the good guys get their work done. You want to know if it will pass through traffic with minimum delay and loss while stopping threats and enforcing class of service and usage rules. And, if possible, you’d like to find that out without going crazy trying to recreate a realistic environment or going blind trying to correlate the results to see what they mean. Talk about a nightmare.

The ideal UTM test system would:

1. Save time by generating threats along with safe traffic like VOIP, P2P, C IFS, and real MPEG4 video (from the same port at the same time)

2. Deliver the horsepower to scale at all protocol layers in all the dimensions that matter, like threats per second, sessions per second, SSL sessions per second, and concurrent connections. But these sessions have to be real – no fake TCP stacks, please! You know what I’m talking about here ;-)

3. Be controllable from a single GUI for fast and accurate test setup, and if I want to get scriptable I want at a minimum automatic script generation, and ideally, integration with leading test frameworks

Guess what? Spirent has the only UTM test solution that brings all that to the table in a way I’d like. Of course you expected me to say that, but seriously, integrated threats and protocols on both IPv6 and IPv4, with a single GUI, correlated results in a single blade on a flexible chassis-based architecture hasn't existed until now.

The combination delivers unprecedented test realism with the scalability and performance required for the job.

What’s that mean for you?

Reduced time-to-test. Less setup, less waiting, more combined results in a single test run.
More realistic testing with real TCP stacks and user behavior that catches real problems without reporting time-wasting false positives
Integrated results for better interpretation of what is going on the network
A more reliable solution

That’s right. The nightmare is over.

Testing Data Center Virtualization: Are you asking the right questions?

martin.vickery@spirent.com (Martin Vickery) — Mon, 15 Jun 2009 11:28:00 GMT

Data centers today are larger, faster and more complex than ever. New technologies such as virtualization, Fibre Channel over Ethernet and 40/100 Gigabit Ethernet aim to help organizations move multiple traffic types – data, storage, video, and voice – onto a single, converged core. Validation of all these new technologies through testing is a crucial part of the product development and deployment process.

Today, let’s look at testing virtualization. Testing virtual network devices and servers poses several interesting new questions:

How can a test instrument attach to a virtual network device?
Often one physical interface may handle traffic for dozens of Virtual Machine (VM) instances, making it difficult to isolate and measure the performance of each VM instance. What’s needed is to virtualize the capabilities of the test instrument. A virtual test instrument resides in software, and thus runs inside the physical machine hosting virtual network and server instances. From the standpoint of the virtual network device, a test port looks exactly the same as it would in the physical world.

Can test instruments on virtual machines be trusted?
One strategy to ensure measurements can be trusted is to implement the entire test instrument – including software-based emulation of hardware components – in software. This approach requires a far more rigorous approach to system design than does software-only tool design. But the benefit is clear: by emulating the entire test instrument in software, the instrument’s measurements are far less dependent on extraneous factors. Such an instrument will produce more meaningful measurements than software-only tools.

Do virtual and physical switches offer comparable performance?
Line-rate throughput and low latency and jitter have long been the hallmarks of physical Ethernet switches, but their virtual counterparts may not compare. Even if virtual switches will never handle loads as heavy as physical switches (a dubious assumption in these early days of virtual networking), it is still important to conduct stress tests to describe the limits of system performance. Industry-standard methodologies for assessing unicast and multicast performance in switches and routers still apply in the virtual world.

Does a virtual switch support the same protocols and functions as a physical switch?
A network manager can reasonably expect any modern Ethernet switch to support features such as Virtual LANs (VLANs), Access Control Lists (ACLs), and Internet Group Management Protocol (IGMP) for forwarding multicast traffic. These protocols (and often many others) are often included as part of physical switch performance testing and should also be included as well when testing virtual switches.

For more information on virtualization and data center testing download our latest whitepaper titled “Data Center Testing: A Holistic Approach.”

Welcome to The Tested With Spirent Blog

Sailaja.Tennati@spirent.com (Sailaja Tennati) — Thu, 11 Jun 2009 11:29:00 GMT

Testing is a strategic imperative as service providers deploy highly complex converged networks delivering differentiated voice, video, data and applications based services. Testing today’s, as well as tomorrow’s solutions and networks is driven by the requirement to achieve uncompromised performance and scalability while delivering a superior quality of service and experience across the telecommunication infrastructure.

Today, our commitment to the test and measurement community is extending beyond just products and services to offer a forum dedicated to discussing emerging technologies and the related challenges surrounding testing, improved quality and faster time-to-market..

The Tested with Spirent blog is designed to provide test engineers with insights on methodologies, approaches, challenges and the latest trends affecting new and emerging technologies. Industry experts such as Alan Way, David Newman, and others will deliver commentary that will help you make informed decisions regarding your product and service test methodologies. The insight shared within this blog is intended to accelerate the development and deployment of next-generation converged networks and technologies.

Our role as leaders in test and measurement is to enable you to develop and validate the most successful products and services, not just from a technological perspective but also to ensure the greatest return on your investment. We hope this blog provides a unique perspective on testing enabling you to be more successful while providing a unique forum for you to provide feedback and share your opinions.

Data Center Testing: When It Comes to a Data Center Rollout, Kicking the Tires and then Driving It Off the Lot is Not Enough

martin.vickery@spirent.com (Martin Vickery) — Thu, 11 Jun 2009 08:20:00 GMT

It seems like there are a dozen ways to succeed and a thousand ways to fail. And it is disconcerting how easily a project that is on track can end up in the ditch. I’ve noticed that often success comes down to discerning between the merely urgent and the truly important. And when the tyranny of the urgent overshadows more important issues, you can end up with a wrecked project.

Nowhere is this more true than in the data center. I know how it is. Feature creep and schedule delays can edge performance and scalability testing out of the schedule. But that leaves you vulnerable to SLA violations and service outages. And that is when deferring the important for the urgent can really cost you.

Estimated Outage Cost-per-minute

Application	Cost
Supply chain management	$11,000
e-commerce	$10,000
Customer service	$3,700
ATM/POS/EFT	$3,500
Financial management	$1,500
Source: Alinean

What it comes down to is this: you can’t wait until the system is deployed to find out how it will perform. You have to know before deployment, when fixes don’t cost five digits per minute. Failure to budget for testing (typically three to five percent of the data center budget) is like building a new house with top-of-the-line plumbing but not turning on a single faucet to check any leakages in the installation until it is finished, decorated and moved into. Something you hope your home-builder didn’t do.

And you shouldn’t do it, either. Testing is a strategic imperative, a plan that avoids the almost certain loss of revenue and customers when things go wrong - performance issues, SLA violations or service outages. I’ve seen organizations gain a competitive advantage through testing, because it enables them to do performance optimization and offer a level of reliability that separates them from the crowd.

One recent OnDemand project involved a major financial trading site flipped the switch on a new data center with zero problems in the first 48 hours, a first for them. They were one of only two trading sites in the US that were able to keep up with the pre-April 15th trading peak, avoiding the delays and outages experienced by other sites.

All because of robust performance testing during the project lifecycle with best-practices test methodology expertise available with OnDemand. It is no surprise that those who come to us for testing keep coming back. The ROI is clear.

Some organizations rely on the team that designed and implemented the system to do the testing. However, the project team, whether an in-house team or outside consultants, will not see their own blind spots and therefore won’t test for them. Testing, like accounting, requires the objectivity and accuracy that comes with separation of duties. And, like a good auditor, Spirent follows best practices, in fact develops best practices as new technologies and protocols emerge, through participation in standards bodies and partnering with manufacturers as implementations are developed. That is simply a level of expertise not likely to be duplicated in-house or with a consultant.

It is that kind of expertise that enables us to develop test methodologies that anticipate future needs for capacity, services and traffic types.

For the sake of your project, your revenue and your customers, take a break from the deceptively urgent to consider the value of the truly important for your organization and avoid ending up in a ditch.