Assuring QoE with over-the-top Video
Realistic Adaptive Bit Rate Testing at Production Scale
Traditional approaches to measuring QoE fail to produce realistic results, however, because they don’t test at production scale with the actual number of expected users. They also don’t account for realistic levels of background traffic. When QoE is not done with realism it results in surprises and finger pointing in production.
This paper provides guidelines for realistic measurement of end-user QoE for video OTT services using adaptive bit rate streaming. “Realistic” means gaining a true picture of end-user QoE by testing at actual scale while using background traffic comparable to that expected in deployment. Realistic measurement is the only way to spot the bottlenecks in server capacity and network bandwidth that too easily lead to failure for OTT providers.
ASSURING QUALITY OF EXPERIENCE WITH OVER-THE-TOP VIDEO
Realistic Adaptive Bit Rate Testing at Production Scale
June 2012
Rev. A 06/12
SPIRENT
1325 Borregas Avenue
Sunnyvale, CA 94089 USA
Email: sales@spirent.com
Web: www.spirent.com
AMERICAS 1-800-SPIRENT • +1-818-676-2683 • sales@spirent.com
EUROPE AND THE MIDDLE EAST +44 (0) 1293 767979 • emeainfo@spirent.com
ASIA AND THE PACIFIC +86-10-8518-2539 • salesasia@spirent.com
© 2012 Spirent. All Rights Reserved.
All of the company names and/or brand names and/or product names referred to in this document, in particular,
the name “Spirent” and its logo device, are either registered trademarks or trademarks of Spirent plc and its
subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or
trademarks are the property of their respective owners.
The information contained in this document is subject to change without notice and does not represent a
commitment on the part of Spirent. The information in this document is believed to be accurate and reliable;
however, Spirent assumes no responsibility or liability for any errors or inaccuracies that may appear in the
document.
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
CONTENTS
SPIRENT WHITE PAPER • i
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
Realistic Adaptive Bit Rate Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
Simulate the Viewer Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
Simulate the Functionality of the Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Support all Major ABR Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Adopt a Meaningful Benchmark of Measured QoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Implement a Comprehensive Strategy to Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
1 • SPIRENT WHITE PAPER
EXECUTIVE SUMMARY
The Internet and mobile devices have already changed the faces of computing,
communications and commerce . Now they are changing the face of broadcasting .
Consumers are choosing where, when and what they want to watch over the Internet, rather
than simply relying on the scheduled programming of multi-channel TV . Some are “cutting
the cord”—discontinuing traditional services from cable and satellite network providers
in favor of Netflix, Hulu and similar services which offer video over the top of the Internet
(OTT) . And there’s strong evidence that younger viewers are bypassing multi-channel
TV altogether in favor of video OTT . In response to this trend, media companies, cable
operators and other broadcasters are stepping up with their own OTT offerings .
Adaptive bit rate streaming lets video providers serve a variety of end-user devices and
network configurations . But like e-commerce providers before them, they’re learning
consumers have little tolerance for spinning hour glasses, circling arrows or other symbols
that tell them to wait . The user’s quality of experience (QoE) is determined by many factors
including server capacity, network bandwidth and background network traffic . But user QoE
is central to the success of any OTT service, so measuring, assessing and understanding
QoE is a critical success factor for OTT video and network providers .
Traditional approaches to measuring QoE fail to produce realistic results, however, because
they don’t test at production scale with the actual number of expected users . They also
don’t account for realistic levels of background traffic . When QoE is not done with realism it
results in surprises and finger pointing in production .
This paper provides guidelines for realistic measurement of end-user QoE for video OTT
services using adaptive bit rate streaming . “Realistic” means gaining a true picture of
end-user QoE by testing at actual scale while using background traffic comparable to that
expected in deployment . Realistic measurement is the only way to spot the bottlenecks in
server capacity and network bandwidth that too easily lead to failure for OTT providers .
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
SPIRENT WHITE PAPER • 2
BACKGROUND
Delivery of video over IP networks is not new . Since the mid 1990s IPTV efforts have
accompanied the growth of broadband network access in homes . Service providers—usually
telecom carriers or cable operators—provide their own network infrastructure, including
a set-top box in the user’s living room . But IPTV falls short of the promise of the mobile
Internet . It enables video on demand—“what they want, when they want it”—but it cannot
address user demand for video wherever they happen to be and via whatever device they
prefer . And since these deployments require expensive network build out and specialized
infrastructure, providers usually structure services under a subscription model with multi-
year contracts or as add-on functionality to conventional multi-channel TV services .
Video OTT is a different approach . It uses the existing Internet to the user’s location as
provided by whatever carriers happen to provide it . That means the video service can be
delivered anywhere there’s an Internet connection—no set-top box required . And since
video providers don’t have to invest in their own network infrastructure, service fees are
significantly lower than typical multi-channel TV services . For example, a basic Netflix
subscription is currently $7 .99 per month in the U .S .
That handles the “where they want it” demands of mobile Internet users, but it introduces
several complications . Since OTT providers do not control the network, they are not assured
of the bandwidth quality available for video transmission all the way to the user . Further,
available bandwidth is reduced by background traffic . Insufficient bandwidth results in
users experiencing pauses in video display—usually accompanied by the dreaded spinning
hourglass or circling arrows .
In addition, providers must enable the user to view the video content on their device of
choice . This may be an Internet-connected set-top box like a Roku streaming player or Wii
console, or more commonly a laptop computer, tablet or smartphone . Depending on the
device in use, the user will have different expectations for video definition . He or she will
expect 720p high-definition video when viewed on the 55 inch flat-screen TV in the living
room, but may accept standard-definition output when viewed on a smartphone . Like
available network bandwidth, the capacity of a device can vary if its CPU becomes busy
processing other tasks . That too can result in circling arrows . Figure 1 shows an overview of
over-the-top video distribution .
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
3 • SPIRENT WHITE PAPER
All of these issues—unpredictable bandwidth to the user, the need to accommodate
multiple types of viewing devices and the need to adapt to device and network capacity—
are addressed by a single technology—adaptive bit rate (ABR) streaming . ABR is typically
implemented over the hypertext transfer protocol (HTTP), so it is a suitable technology to
use with video over the Internet .
Using ABR, video servers maintain multiple copies of the video—typically up to five—
encoded at different bit rates corresponding to different levels of definition . Each copy is
actually a series of short fragments of a few seconds each . The server makes information
about the encoded copies and fragments available to smart ABR clients via a manifest/
playlist file . The client assesses the resources available to it and requests the server
stream video at a bit rate (and corresponding definition) suitable to each situation . If
those characteristics change during the transmission, the client can request the server to
deliver video fragments at a different bit rate . This is possible without restarting the whole
transmission because the video content is organized into small fragments . The server can
simply select the next fragment from a copy with higher or lower bit-rate encoding . The
overriding goal is that viewing be continuous—no pauses, no circling arrows .
Video Source
File
Package
Distribution
Server
Video Asset
Live Video
Audio Asset
Live Audio
Protect
Fragment
Manifest
Live
Package
Protect
Fragment
Manifest
Media
Fragment
Manifest
Files
Video
Ingress
Server
HTTP
Origin
Module
HTTP
Cache
Module
Ethernet
Delivery
NetworksOrigin Storage
Network
Equipment
Encoders
Media Clients
(Netflix, Web
browsers,
Internet-
Connected TV’s,
Blu-Ray players,
STBs, etc.)
ENCODING AND FRAGMENTATION
DISTRIBUTION CONSUMPTION
Over-the-top Video Distribution
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
SPIRENT WHITE PAPER • 4
THE PROBLEM
Since OTT providers do not provide or control the network, managing user quality of
experience (QoE) of ABR transmissions is challenging . Video display pauses experienced by
the user can be caused by:
• Insufficient server capacity at the video provider to handle the number of users active
• Insufficient network bandwidth to the user
• Competing background traffic on the network
• Changing loads on the user’s viewing device
From the user’s perspective, it’s generally not possible to tell which facility is at fault . User
complaints can result in finger pointing between video providers and network carriers .
And without realistic QoE data, both lack the information needed to plan infrastructure
expansion or reconfiguration . This can result in poor service, over provisioning and
dissatisfied customers for both .
The solution for assuring acceptable QoE in deployment is testing at scale with realism prior
to deployment . Testing can also measure and characterize server and bandwidth utilization .
It helps OTT providers plan server capacity, and it enables network providers to anticipate
bandwidth needs (or demonstrate suitable bandwidth exists) . Most of all, it improves
satisfaction of their mutual customers .
As we have seen, server and network utilization vary with load—the number of users
actually viewing video from the server and the bit rates delivered to them based on network
capacity and client devices . Laboratory testing rarely simulates more than a few users;
therefore, results rarely provide a realistic view of real-world performance over the real-
world network . The problem is further complicated by the fact that there are multiple ABR
streaming technologies in use: Microsoft IIS Smooth Streaming, Apple HTTP Live Streaming
(HLS), Adobe HTTP Dynamic Streaming (Project Zeri) and Dynamic Adaptive Streaming over
HTTP (DASH) .
In a 2011 trial, BBC broadcasted the Wimbledon Tennis Championships over the Internet
using ABR streaming . They embedded a visible bit rate indicator in the video and
encouraged viewers to log onto the BBC web site and report their experience . This was an
effort to get a realistic view of what their viewers experienced . But even that could only
test a specific case—the number of viewers who watched and the devices and network
connections they used . What if twice as many users access the broadcast next year?
Ten times as many? How many servers would be needed? What network bottlenecks
might exist?
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
5 • SPIRENT WHITE PAPER
What network and OTT providers need is a testing approach that provides a realistic view
of any scenario—number of users, available bandwidth, background traffic and client type .
The testing should collect and report comprehensive data and provide a useable metric for
characterizing QoE . It should support all the major ABR streaming technologies and client
functions . It would also be best if the testing didn’t require viewers to serve as guinea pigs .
The ABR testing approach described below does just that .
REALISTIC ADAPTIVE BIT RATE TESTING
BBC had the right basic idea . The test that matters most is the test that shows what users
actually experience . Tests must be performed at the network edge, where users connect .
This provides the most realistic view of end-to-end performance—from the servers, through
any caching or content delivery systems, to the end users’ viewing devices . Tests must also
be performed at the actual scale anticipated in production—the number of users, kinds
of devices, amount of background traffic . Rather than simply collecting technical metrics,
service parameters should be tested . An automated mechanism to collect and analyze test
data is also needed .
The next few sections present the core elements needed to perform realistic ABR testing at
production scale .
Simulate the Viewer Experience
User requirements for OTT video reception are straightforward: display video at the
maximum resolution available on their device and network connection, and maintain
continuous streaming and display without pauses . The test approach uses software
to simulate the users’ devices . The test system is connected at the network edge and
configured to simulate:
• Number of users
• Connection bandwidth
• Client device type
• Background network traffic
Each simulated user connects to the video servers and consumes the video stream like
a typical device . As the test runs, the test solution measures bit rates achieved, bit rate
shifts, the number and duration of pauses (buffering), protocol events and a number of
other metrics .
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
SPIRENT WHITE PAPER • 6
Testing as described above provides a direct indication of video server performance under a
variety of user loads and client types, and it measures user QoE under specified loads and
network conditions . This provides the video OTT or network provider the ability to project
QoE under anticipated—or even unanticipated—real-world conditions . The test solution
also verifies conformance of the manifest file and video stream to specifications to spot
problems with encoding or streaming facilities .
Simulate the Functionality of the Client
ABR clients implement certain common functions like play, seek, pause, etc . They may also
embody configuration parameters that control their operation . The testing approach should
enable testers to simulate client functionality and establish settings for parameters such as
starting bit rate, bit rate shift algorithm and others .
Support all Major ABR Technologies
As mentioned earlier, there are multiple ABR streaming technologies available in the
market . Because of the dominance of Apple iOS and relatively widespread use of ABR
technologies from Adobe and Microsoft, OTT providers must support the major variants of
this technology, including:
• Microsoft Smooth Streaming—a service extension of Microsoft Internet Information
Server (IIS) that streams video over HTTP to Microsoft Silverlight and compatible clients
• Adobe HTTP Dynamic Streaming—an element of Adobe’s Flash Media Server providing
ABR streaming for Adobe Flash Player 10 .1
• Apple HTTP Live Streaming—a streaming protocol developed by Apple in support of
QuickTime and iPhone/iPad
• Dynamic Adaptive Streaming over HTTP (DASH)—a proposed industry standard and
codec-agnostic ABR streaming technology growing from the work of the Moving Picture
Experts Group (MPEG)
Video OTT providers must accommodate different choices among viewers . Therefore, testing
must address the most common ABR streaming technologies .
Adopt a Meaningful Benchmark of Measured QoE
Proper video OTT testing enables testers to simulate thousands of users . This produces
millions of data points that must be collected and analyzed . Tests should collect data on
protocol events like buffering wait times, HTTP transactions, manifest file transactions,
video fragment downloads, video download requests and responses, total bandwidth
used and other metrics . When analyzing test results, testers should be able to drill down
to any of those details to investigate specific problems and characterize specific attributes
of the transmission . A method to transform the data into a realistic and understandable
characterization of user experience is also needed .
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
7 • SPIRENT WHITE PAPER
The Spirent Adaptive Streaming (AS) score provides an effective method to quantify
user experience . The AS score is an index which indicates how many users in the test
environment received the highest quality video stream with continuous display . It is a
realistic single indicator of user QoE for the duration of the test, and it is the best starting
point for analyzing test results and comparing one test run to another .
Implement a Comprehensive Strategy to Testing
Performance testing is critical, but it’s not enough . Comprehensive testing enables a
broader view of the user experience and can spot vulnerabilities in the end-to-end system
that could impact service in production . A comprehensive test methodology should test and
verify performance, availability, security and scalability (PASS) as follows:
• Performance—the speed at which new video sessions are set up, the number of clients
watching and the measured performance-oriented results for all the users
• Availability—verification that sessions are set up and torn down properly, the response
delay to session setup and seamless delivery of video (no pauses)
• Security—verification that video can be delivered through firewalls and other security
equipment
• Scalability—determination of the maximum number if simultaneous users successfully
supported
Comprehensive PASS testing not only projects real-world user QoE, it verifies operation of
the OTT and network facilities under real-world conditions and provides data to support the
need for server expansion, caching and network build out .
Assuring Quality of Experience with Over-the-Top Video
Realistic Adaptive Bit Rate Testing at Production Scale
SPIRENT WHITE PAPER • 8
SUMMARY
Video OTT is enabling new, profitable business models and providing an essential element
of the mobile Internet . But since OTT providers don’t own or control the network all the way
to their customers, the risk of performance issues—and customer dissatisfaction—is high .
Network providers share the risk . They are asked to assure service to the end user, but they
lack the information needed to project the impact on the network . Both OTT and network
providers need a way of testing planned OTT services to the scale anticipated in actual
production .
This paper has presented guidelines for realistic testing of video OTT . It enables testers
to simulate loads representing thousands of users using a variety of viewing devices . It
accounts for the network background traffic experienced by users in the real world . The
test solution supports the major ABR streaming protocols and client functions . It not only
provides detailed test results but also characterizes results in a single, meaningful Adaptive
Streaming score . And it implements a comprehensive and established test methodology
that verifies performance, availability, scalability and security (PASS) .
Implementation of this testing supplies OTT and network providers with critical data to
facilitate planning of infrastructure expansion and configuration to meet anticipated needs .
It also helps them maintain customer satisfaction for their mutual customers by ensuring
viewers see what they “tuned in” to see—not circling arrows .
Spirent Communications works behind the scenes to help the world communicate and
collaborate faster . To learn more about Spirent and solutions for adaptive bit rate testing,
please visit: www .spirent .com .
