Back in June, I wrote about my experience in the lab with Mike Thelander of Signals Research Group (SRG) working on the industry’s first independent performance analysis comparing the Enhanced Voice Services (EVS) and AMR codecs. This time, we took our equipment in the field to examine the comparative performance of the two using T-Mobile’s live network, which is currently the only North American cellular provider that supports EVS.
EVS codec test prep and methodology
The stage was set as follows:
Testing was done in two locations: California and Minnesota
Devices: Four identical smartphones were purchased over-the-counter at various T-Mobile stores to ensure only commercially available software and firmware was used; one other non-EVS commercial brand device was purchased to be used on both T-Mobile and AT&T for some tests
Devices were controlled to use EVS settings of 13.2 and 9.6 predominately along with AMR-WB
The smartphones were locked to LTE to prevent handovers to the 3G network; a limited number of tests were conducted over Wi-Fi and 3G
Calls were made in both stationary and mobility scenarios
Based on earlier discoveries from the lab testing study, we used both clean speech audio files as well as those that included realistic background crowd noise; the latter had shown to provide more pronounced improvements for the EVS codec in the lab and we were interested in seeing if there would be similar results in the live network
Spirent’s Nomad User Experience Analytics System was used to play the speech files and analyze the call quality via Mean Opinion Scores (MOS) using the POLQA algorithm; this was the same instrument used in the earlier EVS lab study
And the codec winner is…
Similar to what we found behind closed doors the first time, the EVS codec prevailed again. At the very least with clean speech, EVS voice quality was comparable with AMR-WB. And, as expected, when background noise was introduced or lower signal strengths were used, EVS came out on top with moderate to significant improvements over the AMR-WB performance. In most tests, the more challenging the conditions, the better the relative performance of EVS. Also as expected, EVS generally required fewer network resources on both the downlink and the uplink for voice calls with identical audio activity. Lastly, there were no measurable differences in power consumption for either codec.
Performing a test such as this in the field really highlights the value of lab testing to control the environment. We often struggled to get devices to use the codec/bitrate we wanted because network conditions or SDP negotiation changed our desired settings. Fortunately, we had the benefit of benchmarking the devices in the lab and could recognize when improper codec changes or other anomalies in the field occurred. Utilizing the same audio quality test setup in the lab and field proved invaluable in detecting these anomalies.
For full details on the complete study entitled Behind the VoLTE Curtain, Part 6: You Can’t Ride in my Little Red Wagon … because it is Possessed by the Powers of EVS, visit SRG’s website.
For more information on the intricacies of testing the EVS codec, reference Spirent’s white paper:
Testing the EVS Codec: Addressing Complex Challenges to Ensure Superior Audio Quality Performance