Weighing the Options - Three Approaches for Testing Real-World A-GPS Performance - Part 2
Our last post [Stuck Between a Rock and a Hard Place] presented the problems associated with typical approaches for predicting how the A-GPS functionality in mobile devices will perform in the real world. This post will consider some potential solutions in more detail.
To give a clear understanding of the difficulties encountered in testing real-world A-GPS performance of mobile devices, we first need a brief review of the most common testing techniques used today. These are Drive Testing, Record and Replay and (the approach discussed in this series of postings) Simulation and Modeling.
(1) Drive Testing
Drive testing, at its simplest, first requires identification of a location with the desired geographical characteristics; for example, a particular route through a city environment or a suburban highway might be chosen, due to some known difficulty encountered on that route. The intent is to present the device with common characteristics of a typical end-user environment. In addition to mapping out the drive route, a specific time and day is often chosen to take into account satellite orbital patterns – i.e., the almanac. A test plan is devised, and key criteria are evaluated such as percentage of successful position fixes, accuracy of position fix, time to first fix, etc.
The main advantage of such an approach is obvious: the results provide the most complete view of network and device performance. Drive testing and methods derived from it are necessary to completely characterize performance, and their value cannot be overstated. However, depending on drive testing alone has several drawbacks.
Firstly, when a route is mapped out, the actual environmental characteristics at the time of the drive test are not under the control of the tester. Precisely repeating the test is impossible, since the environment is constantly changing. GPS satellites will not be in the same position in the sky, and weather patterns may have changed which can affect GPS signals in unpredictable ways. At ground level, unexpected geographical changes to objects such as buildings, trees, cars, etc result in unpredictable obscuration and multipath effects.
Secondly, the drive route needs to be repeated multiple times in order to obtain a comprehensive idea of device performance, which adds time and cost. Another consideration is the drive testing is usually carried out in a range of different environments, which can be a very time consuming affair.
So while some drive testing is essential, it is highly desirable to reduce the amount of time spent out in the field. If the bulk of the testing can be done in the lab using a simulation that provides comparable results, then significant savings in time and cost are possible. This leads to consideration of the next approach – record and replay.
(2) Record and Replay
Record and replay, as the name suggests, involves recording and then recreating the signals at a particular point or route. Record and replay can be achieved in two different ways:
- Recording of I/Q signaling: a reliable scanner is used to capture I/Q signals at a particular location (or throughout a drive route.) The signals are then played back in the entirety of the receiver bandwidth. In some cases, the GPS signal alone can be played back.
- Recording of NMEA data: NMEA-formatted data contains key parameters of the GPS measurement made by the receiver, including position fix information, velocity, satellite PRNs and HDOP. This data is collected for the duration of the drive test (or at a stationary position). The data is then used by a GPS simulator to ‘play back’ the conditions experienced by the GPS receiver.
A record and replay approach has its limitations. Firstly, the data can only be as good as the quality of the receiver or scanner used in its collection. Secondly, the incident signals as viewed by the receiver have been affected by a synthesis of all environmental and geographical features at a particular location. As a result, it is impossible to separately quantify the effects of key factors such as reflection patterns and obscuration.
Record and playback does however offer advantages over pure field testing, since it allows a single run of the test (a single drive-through) to be simulated multiple times in the lab, thereby avoiding multiple runs and ensuring repeatability. Costs (both in terms of money and time) can be reduced because the scope of testing in the field is smaller. Lastly, the same GPS ‘scenario’ can be executed for a variety of handsets; this allows for a direct comparison of handset performance, since GPS test conditions remain constant.
In comparison to pure field testing, record and playback can offer significant time and cost savings, as well as convenience. It has, however, one significant disadvantage: due to the nature of the data collection, it is impossible to break down, or observe, the effects of individual geographical features. For example, the effects of any multipath reflections are indeterminate. Not only are they unknown, it is impossible to determine how changing geographical elements affect the results. For example, is device performance poor at a specific point in a particular test due to bad multipath, poor HDOP, or high obscuration?
GPS conditions are also affected by the positions of the satellites in the sky, which are directly tied to a specific time and date. Data collected during a drive test is only representative of the particular interval the device was exposed to GPS signals and is therefore limited. Clearly, greater control over the GPS environment is highly desirable. This leads us to modeling, the technique of identifying and simulating key parameters of the environment.
(3) Simulation and Modeling
Modeling offers a powerful solution to some of the difficulties encountered in pure field testing or record and playback. Firstly, each environmental characteristic is clearly identified and is completely under the control of the test creator, so modifications in geographical elements can be correlated to changes in device performance. Secondly, any location can be modeled, potentially without needing multiple visits. Lastly, the simulation itself can be executed for long periods of time; there are no artificial limits such as those imposed by some record and playback techniques.
While modeling can never perfectly recreate real-world conditions, key parameters can be very closely simulated. Once models are constructed, they are very easy to modify to explore how changes impact device performance. The flexibility and control gained by a modeling approach makes it a valuable addition to pure field testing and record and playback.
So what type of modeling and simulation is needed to adequately test A-GPS performance? Stay tuned to this blog for the next post in this series.