I recently had the pleasure of being locked up in a lab with Michael Thelander of Signals Research Group (SRG) completing the industry’s first independent performance analysis comparing EVS and AMR codecs under a wide range of channel conditions. This study is the 5th in a collaborative series entitled “Behind the VoLTE Curtain,” where varying aspects of the VoLTE user experience have been tested in both live and lab environments. The goal for this study was to determine if the EVS codec can deliver on its promise to provide superior voice quality over AMR codecs under ideal or impaired conditions on multiple pre-commercial wireless devices.
EVS codec testing methodology
Three devices, the Samsung Galaxy S7 Edge, the LG G5, and a Qualcomm Mobile Test Platform (MTP), were tested across multiple AMR codec bit rates (NB and WB) and EVS bit rates (NB, WB, and SWB) ranging from 5.9kbps through 24.4kbps, depending on the specific codec range and capabilities of the particular device. Even though operators will likely limit themselves to just a few options, the EVS codec supports a broad set of bit rates (up to 128kbps) and we wanted to experiment with a wide range to determine how changes in the bit rate influenced voice quality. The devices were configured to work on the T-Mobile USA network.
Our audio files were sourced from 3GPP specification 26.444 and contained numerous languages and dialects:
Baseline tests used a full band clean speech file
One impairment scenario interpolated this clean speech file with low-level background noise
Other scenarios introduced IP impairments to the clean speech file: random packet delay of 50ms, 100ms and 150ms mixed with packet loss of 6%, 10% and 30%
Jitter was an impairment that resulted from the delay; the actual jitter in each scenario is derived as 50% of the configured random delay value. For example, a random delay of 50ms would have an average delay of 25ms and thus 25ms of jitter.
For greater details on these types of impairments and how the EVS codec compensates for them, reference Spirent’s blog post Packet Loss, Jitter, Delay and the New EVS Audio Codec [link: http://www.spirent.com/Blogs/Wireless/2016/January/Packet-Loss-Jitter-Delay-New-EVS-Codec]
Key findings for EVS codec performance
Bottom line, we found that the EVS codec generally outperformed the AMR codec for superior voice quality at similar bit rates and was more impervious to IP impairments. However, based on initial test results, the improved performance was not as pronounced as expected, and highly reliant on the audio capabilities of the device. The full study with all pertinent data is available on the SRG website.
Later this summer, we intend to go out into the field and conduct extensive benchmark testing using commercial devices and Nomad in a live LTE network as a companion to this study. In our past field studies, I’ve been subjected to kangaroo dodging in Australia, walking around outside in sub-zero temperatures balancing a laptop and several devices in Stockholm, and introduced to cracker-covered pan-fried fish in Minneapolis (we drove a long way for it). I wonder what’s in store for me this time…
Spirent’s EVS Codec Testing Solution
Spirent’s Elevate provided the integrated solution to perform the analysis, comprised of the Wireless Test Station, ProLab IMS Test Server, and Nomad User Experience Analytics System.
The Wireless Test Station emulated the T-Mobile LTE network core and enabled device-to-agent testing via its internal integrated ProLab client.
ProLab provided the IMS server, codec-specific agent, call setup at all stack levels, and simulated IP impairments.
Nomad injected the audio files and measured the call quality via POLQA MOS. Nomad has been used repeatedly in our past SRG field studies and continues to be our de facto standard for audio quality performance analysis.
Elevate Solution Architecture for Testing EVS Audio Quality