Jump to content, skipping navigation

The Importance of Testing Software in Realistic Network Conditions

The Importance of Testing Software in Realistic Network ConditionsIncreasingly, applications are expected to work over Wide Area Networks (WANs), wireless LANs, GPRS, 3G or satellite networks. But software performance testing is still frequently only conducted over the fast & reliable Local Area Networks (LAN) of the test lab. Satisfactory application performance in these conditions is no guarantee of acceptable performance in non-LAN networks.

This white paper is aimed at individuals with a limited knowledge of networking and aims to provide a basic introduction to the impact of deploying software over a WAN. It is not intended to be a complete discussion of non-LAN application issues.

It will explain some of the reasons why applications can behave very differently in the WAN/ Wireless networks and why testing in live network environments, even in so-called “off-peak times”, is not really an option. It goes on to explore the methods that are available to software testers to carry out safe but realistic performance testing in the WAN environment.

Read more about Spirent INE products.

Resources

Download a trial version of Spirent INE Windows

    * Required Field

    Cancel

    THE IMPORTANCE OF TESTING SOFTWARE IN REALISTIC NETWORK CONDITIONS September 2011 Rev. A 09/11 SPIRENT 1325 Borregas Avenue Sunnyvale, CA 94089 USA Email: sales@spirent.com Web: www.spirent.com AMERICAS 1-800-SPIRENT • +1-818-676-2683 • sales@spirent.com EUROPE AND THE MIDDLE EAST +44 (0) 1293 767979 • emeainfo@spirent.com ASIA AND THE PACIFIC +86-10-8518-2539 • salesasia@spirent.com © 2011 Spirent. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name “Spirent” and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the document. A White Paper from iTrinegy discussing why applications can behave very differently in the WAN/Wireless networks and why it is important to be able to carry out safe but realistic software performance testing in the WAN environment. The Importance of Testing Software in Realistic Network Conditions CONTENTS Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Welcome to the Topsy-Turvy World of Networking . . . . . . . . . . . . . . . . . . . . . . . 2 Can I Carry On? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 TCP Windowing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Competing Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Real-World Impact of the WAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 The Greater the Distance, the Longer the Delay . . . . . . . . . . . . . . . . . . . . . . . . . 4 Throw More Bandwidth at the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 So How Do You Test Application Performance Over the WAN . . . . . . . . . . . . . . . 6 WAN Emulation – The Safe Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 WAN Simulation vs WAN Emulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 SPIRENT WHITE PAPER • i The Importance of Testing Software in Realistic Network Conditions 1 • SPIRENT WHITE PAPER EXECUTIVE SUMMARY Increasingly, applications are expected to work over Wide Area Networks (WANs), wireless LANs, GPRS, 3G or satellite networks. But software performance testing is still frequently only conducted over the fast & reliable Local Area Networks (LAN) of the test lab. Satisfactory application performance in these conditions is no guarantee of acceptable performance in non- LAN networks. This white paper is aimed at individuals with a limited knowledge of networking and aims to provide a basic introduction to the impact of deploying software over a WAN. It is not intended to be a complete discussion of non-LAN application issues. It will explain some of the reasons why applications can behave very differently in the WAN/ Wireless networks and why testing in live network environments, even in so-called “off-peak times”, is not really an option. It goes on to explore the methods that are available to software testers to carry out safe but realistic performance testing in the WAN environment. INTRODUCTION These days we are creating applications where the servers on which they reside are situated in one place and the clients requiring access to these applications are remote from them. Some of the reasons for this are: • The process of “server-consolidation”. Increasingly, companies are concentrating all their servers into a smaller number of IT centers as the cost of managing servers in a disbursed environment has proved too expensive • Provision of applications to remote users/offices, previously only available at head office • Homeworking • More and more self-service applications: insurance quotes, tax forms… • Making applications available to mobile users via GPRS or 3G networks – serving the “Road Warrior” As a result, more and more applications are being delivered over Wide Area Networks (WANs), wireless LANs, GPRS or 3G networks. We’ll refer to them all as WANs for simplicity from here on. Unfortunately, when it comes to software, performance testing is frequently conducted over the simple Local Area Networks (LAN) of the test lab. These LANs are normally very fast and reliable. The problem is that the applications will ultimately need to run over the WAN and there is a big difference between these two network environments. The Importance of Testing Software in Realistic Network Conditions SPIRENT WHITE PAPER • 2 WELCOME TO THE TOPSY-TURVY WORLD OF NETWORKING If the conventional road network (local roads / motorway) behaved in the same way as a data network then all local roads would be 3-lanes wide and all inter-city routes would be single carriage ways (except if you had paid a small fortune to have it widened as some larger companies are be prepared to do). So, if you were traveling within your town (equivalent to a LAN), you could get to your destination extremely quickly on the 3-lane roads available to you. Even if there were other cars on the road, it is very unlikely that there will be any congestion due to the network capacity and the number of routes available. However, things wouldn’t be so great when you wished to travel from one city to another (equivalent to a WAN). Now you would find yourself having to share a single lane with all the other cars on a relatively long road. The result would be much longer journey times and this is exactly the same experience that applications encounter when they pass from the LAN to the WAN. CAN I CARRY ON? But it gets worse because of the way TCP/IP networks work. TCP/IP networks break data down into packets (similar to cars traveling down the motorway). However, it doesn’t like to send lots of packets before getting an acknowledgement back from the destination point that they have arrived successfully (how many you may ask? - see TCP Windowing side bar). This is because in the world of networking, all kinds of things can happen to packets (such as packet loss, errors & reordering) as they travel over the network. The LAN The WAN TCP/IP networks require acknowledgement before sending the next batch of packets The Importance of Testing Software in Realistic Network Conditions 3 • SPIRENT WHITE PAPER If our road networks operated like TCP/IP networks do, then whenever we sent a few cars on a journey, we would wait for one car to drive back and the driver to tell us that the all the other cars had arrived safely before sending some more vehicles to the same destination. So, if it took 3.5 hours to drive from London to Leeds, we would wait another 3.5 hours for our “confirming car” to make the return creating a total journey time of 7 hours before starting the process all over again. COMPETING TRAFFIC Of course, data traffic travels very much faster than road traffic with the same London to Leeds journey being achieved in around 10ms (20ms round trip). It is a small but significant delay and you can see that to transfer quite a lot of data, as we are typically doing, we have to send it in a series of packets, then wait for the acknowledgements and as we wait for these acknowledgements so the delays build up. But this waiting means the road (or network) can be empty for a lot of the time (see TCP Windowing side bar). However, there are always other cars (applications) waiting to fill the road. In the data network world this other traffic could be VoIP, streaming media (watching Reality TV such as Big Brother or the World Cup) and other business applications – all competing with your application for space on the network. Sometimes it just doesn’t all fit and this is the reason that some packets never reach their destinations. TCP Windowing TCP windowing allows a set amount of data to be ‘in flight’ at any time before receiving an acknowledgement. Using our road analogy this would allow for a set number of cars to be on their way at any time. However, when one or more cars arrive, a return car can immediately be sent back to say that they got there. When it arrives, it has the effect of reducing the balance of “in flight” cars and so more cars can make the outward journey (up to the TCP Window limit). Assuming that the TCP window size is sufficiently big to allow a car to complete the return journey whilst not yet having achieved the quota of cars allowed on the road (i.e. 5 cars are allowed on the road but only 4 have so far been dispatched), then the data flow will never stop (because once a car has been acknowledged as having arrived at its destination, another car is allowed to go). This is unfortunately not the case for higher latency links and the sender will have to wait for an acknowledgement before more data can be sent. The Importance of Testing Software in Realistic Network Conditions SPIRENT WHITE PAPER • 4 THE REAL-WORLD IMPACT OF THE WAN So what does that mean in practice – let’s consider something that you do everyday, a file transfer. You are probably connected to a Windows file server in your company’s network. Imagine that you are opening a 20mb Word file which takes about 2.5 seconds to load in a LAN environment. How long do you think it will take to perform the same download if you were based in London and wished to retrieve this file from the server in Leeds? Assuming the same high bandwidth is available, which is unlikely due to cost, then typically it would take 35 seconds. So something that was previously reasonable to download over a LAN is starting to look unacceptably slow (in this case 14 times slower) when retrieved over a WAN. This is exactly the same thing that is going to happen to any application that you are developing and testing that operates over a WAN – and it gets worse because some applications need to be delivered over very long distances around the world. THE GREATER THE DISTANCE, THE LONGER THE DELAY The network delay times – sometimes described as latency- from the UK to New York are in the order of 90ms (round trip) and your application will probably be competing with lots of other traffic. But regardless of how big the network pipe (bandwidth) is and how much other traffic there is, by common laws of physics, the fastest journey time we can achieve is going to be 35 ms. However, due to the indirect route and the various network devices encountered on this journey, which all add their own delay, in the end it takes 90ms to complete a round trip. The table above provides some Internet delays we observed. Your network may be better (or worse) – check. TABLE 1: ROUND TRIP DELAYS From To Via Latency London California Internet 150ms London Texas Internet 130ms London New Zealand Internet 310ms London New York Internet 90ms London North Carolina Internet 100ms London Manchester Internet 150ms London Leeds Internet 200ms London Reading Internet 100ms London Bristol Internet 200ms London Cardiff Internet 280ms UK UK 3G 280ms* *Land latency using internety from same location 43ms (3G=384kbps. Wired 1M, Target for both 256kbps) The Importance of Testing Software in Realistic Network Conditions 5 • SPIRENT WHITE PAPER Take a particular look at the transfer rate we experienced for a 3G card test – the delay time is close to that of a wired connection to New Zealand. Therefore, with mobile workforce applications you need to be especially careful. All radio networks are generally subject to greater packet loss. Download times can also be affected by the physical location of the laptop in a room because the signal can be weakened as it travels through walls etc. THROW MORE BANDWIDTH AT THE PROBLEM We all know about bandwidth – at home you probably started with dial-up connections to access the internet and when you found the download times were proving too slow, moved to broadband (A)DSL or cable. Problem solved. So why not buy more bandwidth? Beyond a certain point it does not matter how much bandwidth you have as the transfer is limited by the need to get an acknowledgement back from the receiving end every time you send a certain amount of data. This table shows the effect of copying a file from a Windows file server (share) with different latencies (delays). Notice that although we didn’t restrict the bandwidth at all (our connection is 100Mbps) by the time we reach 150ms (London to California type round trips) the maximum bandwidth used is just 1.5Megabits per second. This is a case where buying bandwidth can’t speed this transfer up. If you think we are overstating the case, we know of several projects, including a document retrieval project (our Word document example is a simple document retrieval process) that failed utterly in the WAN although it had successfully passed several LAN-based tests. It simply proved so slow as to be unusable. TABLE 2 Roundtrip Delay (ms) Time (seconds) Peak Transfer Rate (Mbps) 0.5 Lan 8 Internet 10 55 Internet 20 105 Internet 50 210 Internet 100 470 Internet 150 680 Internet 200 890 Internet Table 2: Test of Delta Transfer for a 62mb File Using NetBIOS In All Cases Using a 100 Megabit Connection The Importance of Testing Software in Realistic Network Conditions SPIRENT WHITE PAPER • 6 SO HOW DO YOU TEST APPLICATION PERFORMANCE OVER THE WAN? Clearly, testing in the LAN is an essential first step. After all, if an application will not work over the internal network, it’s very unlikely to perform over the WAN. However, as discussed, it does not follow that an application that performs in the LAN will necessarily perform well or in extremis even work at all in non-LAN environments. OK, so now you’re (hopefully!) convinced of the importance of testing in the WAN. However, a request to conduct testing of a new and untried application in your live WAN environment carrying business-critical data is likely to be declined. A possible alternative is to confine your testing over the live WAN to out-of-office hours. The problem is that most live networks are actually busy (or even busier) at night as the company performs network back-ups etc. It is also going to be impractical to get the right people in place at the required locations in order to conduct the initial test and subsequent retesting. A third way is to create a duplicate of the live network but when you consider the cost of doing this (including national or international circuits), the chances are that such a request will be turned down. Furthermore, a duplicate network will not have the competing traffic to create the necessary realism. WAN EMULATION – THE SAFE OPTION The tables in this article, showing the relationship between delay times and file transfer times were not produced using an actual WAN but with network emulation technology, namely the iTrinegy Network Emulator-Enterprise. This is a technology that behaves like a real WAN environment but which can be deployed in the same room (a test lab, if we’re being formal) as your normal test rig. It allows you to recreate a wide variety of different WAN conditions and enables you to inexpensively ensure that software can be tested during prototype and pre-deployment testing. Typical network impairments that an emulator should be able to produce include: • Bandwidth Restrictions • Delay (Latency & Jitter) • Packet Re-ordering, Packet Error & Packet Loss • Traffic Shaping and Traffic Prioritization (QoS) A good network emulator will also be able to recreate the following types of network: • High Latency WANs (National, International and Satellite) • Wireless Networks (e.g. Wi-Fi, WiMAX and 3G) • Jittery networks – such as cause VoIP deployments a problem • Networks that lose and/or damage traffic • QoS type networks, including MPLS, ATM and VLANs It should also be possible to apply different impairments to different traffic as would happen in a real WAN. The Importance of Testing Software in Realistic Network Conditions 7 • SPIRENT WHITE PAPER SUMMARY • In the topsy-turvy world of networks, local (LAN) routes are fast and inter-office routes (WAN) are slow. • Even relatively modest amounts of delay (latency, loss etc) can have a big impact on application performance • Satisfactory performance in LAN environments is no guarantee of acceptable performance in the WAN • Testing in the live production WAN is rarely an option • WAN Emulation is the safe alternative • The ability to test in WAN conditions will enhance the value of the software tester Spirent INE Network Emulators are available in rackmount and software-only formats WAN Simulation vs WAN Emulation WAN simulation is a technique where a program mathematically models simulated entities such as hosts and routers in order to determine their behavior under various conditions. This is sometimes mistakenly known as network emulation. The advantage of WAN emulation is it allows you to apply real changes to network characteristics which, in turn affect the actual performance of the application i.e. you get to test the application “for real”.