Testing Audio/Video Quality in the Home Network
testing audio/video Quality
in the home network
August 2009
Rev. A 08/09
Application Note
sPirent
1325 Borregas Avenue
Sunnyvale, CA 94089 USA
Email: sales@spirent.com
Web: http://www.spirent.com
americas 1-800-SPIRENT • +1-818-676-2683 • sales@spirent.com
euroPe and the middle east +44 (0) 1293 767979 • emeainfo@spirent.com
asia and the Pacific +86-10-8518-2539 • salesasia@spirent.com
© 2009 Spirent. All Rights Reserved.
All of the company names and/or brand names and/or product names referred to in this
document, in particular, the name “Spirent” and its logo device, are either registered
trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in
accordance with relevant national laws. All other registered trademarks or trademarks are the
property of their respective owners.
The information contained in this document is subject to change without notice and does not
represent a commitment on the part of Spirent. The information in this document is believed
to be accurate and reliable; however, Spirent assumes no responsibility or liability for any
errors or inaccuracies that may appear in the document.
testing audio/video Quality in the home network
CoNTENTS
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Audio/Video Technologies overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Video Encoding and Compression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Audio/Video Stream Encapsulation and Transport . . . . . . . . . . . . . . . . . . . 3
IP Transport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Factors Affecting Audio/Video Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Video Codec and Codec Bit Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Packet Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Packet Delay Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Audio/Video Quality Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Quality of Experience Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Expert Analysis Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Video Stream Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Transport Quality Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Testing Audio/Video Quality Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
SPIRENT APPLICATIoN NoTE • i
Testing Audio/VideoQuality in the Home Network
1 • SPIRENT APPLICATIoN NoTE
1. INTRoDUCTIoN
over the past few years, advances in the technologies used to create, store and
deliver audio/video content have enabled a substantial increase in the number
of audio/video programming choices and services available to consumers. The
efficiencies realized through sophisticated audio/video compression techniques,
storage solutions and packet-based transports have been leveraged by audio/
video service providers to expand program lineups and offer consumers an array
of compelling new services—high-definition television, video-on-demand, multi-
room digital video recorders and more. The same technological advancements
also have contributed to a competitive shift among audio/video service
providers, including the entrance of many more diverse organizations into the
industry. Today, traditional audio/video service providers such as cable and
satellite television operators compete vigorously with non-traditional service
providers such as telcos, mobile network operators and “over-the-top” Internet
video providers to attract new customers and retain their current customer
bases.
The ability to assure the quality of the viewing experience will be one of the
key factors that will allow audio/video service providers to attain an advantage
in this highly competitive environment. This should include investments in
service assurance equipment which covers the core, edge and access networks,
and most importantly, the customer’s home network. Increasingly, the home
network is becoming the responsibility of the audio/video service provider and
is often the portion of the service delivery network most susceptible to quality
issues.
This application note describes how portable test tools may be used in the
home to support the delivery of audio/video service and to guarantee a
satisfactory quality of experience. First, a brief overview of the technologies
used to create and deliver audio/video content is provided, including a
description of the factors that may contribute to quality degradation in the
home. Next, the application note discusses the key performance metrics
provided by Spirent’s Tech-X Flex field test set to assess the quality of the
viewing experience. Finally, the methodology for using the Tech-X Flex to
measure audio/video quality, identify the root causes of quality degradation and
resolve issues that adversely affect the customer viewing experience is offered.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 2
2. AUDIo/VIDEo TECHNoLogIES oVERVIEW
2.1. video encoding and compression
A digital video is a series of pictures or frames with a common resolution which
are displayed sequentially at a specific rate. Typically, digital video includes
a significant quantity of redundant information which may be compressed to
reduce the amount of capacity that is needed to transfer and store the images.
Most video codecs use two types of compression—spatial compression and
temporal compression. Spatial compression operates on a single frame and
leverages the fact that the color and brightness of the objects included in a
frame may be similar over a given area. For example, an outdoor scene may
have images in which a significant portion of the blue sky has adjacent pixels
that are very similar. Rather than encoding each pixel individually, spatial
compression encodes only the differences between neighboring pixels. This
technique reduces the amount of information needed to display the frame.
By contrast, temporal compression works over multiple frames. Because a
series of digital video frames typically has a large amount of redundant data
(for example, background objects that are not moving), temporal compression
techniques are used to encode only the differences between successive frames.
This significantly reduces the quantity of data needed to display the frames,
allowing the realization of substantial network and/or storage capacity savings.
Common video codecs organize a series of successive digital video frames
encoded with spatial and temporal compression to form a group of pictures
(goP). The spatially-compressed frames in a goP are called “intra-coded”
frames or “I-frames,” which contain all of the information needed to
independently render a full digital video frame. The frames included in the goP
encoded with temporal compression are called “inter-coded” which reflects their
dependency upon other frames in the goP to be reconstructed. Most video
compression techniques use two types of inter-coded frames – “P-frames” and
“B-frames”. A P-frame (or “predictive coded frame”) is coded using the changes
from the most recent I-frame or P-frame in the goP. A B-frame (or “bidirectional
predictive coded frame”) is coded using either the changes from the previous
I-frame or P-frame in the goP, the following I-frame or P-frame, or a combination
of both the previous and following I-frames or P-frames. Note that a typical goP
includes one I-frame followed by a variable number of P-frames and B-frames.
The following diagram shows a sample goP:
Testing Audio/VideoQuality in the Home Network
3 • SPIRENT APPLICATIoN NoTE
2.2. audio / video stream encapsulation and transport
The methods used to encapsulate and transport compressed audio and
video follow a layered model much like the oSI protocol stack. Initially, the
compressed audio or video becomes a unit called an elementary stream. The
elementary stream includes header information which allows the audio/video
content to be organized and decoded properly. Elementary streams then are
divided into smaller units called packetized elementary streams. These streams
contain additional information such as the timestamps needed to decode and
present the audio/video content correctly. Then, multiple packetized elementary
streams are multiplexed into a unit called a transport stream which includes
one or more audio/video programs. Typically, the transport stream is divided
into 188-byte packets for transmission and additional headers are appended
to allow the transport stream to be reconstructed by the decoder. The most
important transport stream header fields include the program identifier (PID)
which associates a transport stream packet with a specific program and the
program clock reference (PCR) which provides the decoder with the time
reference needed to present and synchronize a program’s audio/video streams.
The following diagram illustrates how elementary streams are organized into
programs and transport streams:
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 4
A decoder needs to understand how to de-multiplex a transport stream to
present a specific program to the viewer. This is accomplished using a number
of tables which are included in the transport stream by the encoder. Each
table forms an elementary stream which is assigned a PID. Collectively, these
tables are called program specific information. The primary table is called the
program association table (PAT) which identified using PID 0. Essentially, the
PAT includes a list of other tables called program map tables (PMTs) included
in the transport stream with their corresponding PIDs. one PMT is included in
the transport stream for each audio/video program and contains a list of the
elementary streams that form the respective program, along with the associated
PIDs. The following tables illustrate the program specific information included
in the transport stream.
2.3. iP transport
Commonly, two methods are available for encapsulating and delivering audio/
video services to residential customers over IP networks. one method is
placing 188-byte transport stream packets directly into UDP segments which
are then encapsulated using IP. A UDP segment may carry between one and
seven transport stream packets, with seven packets per segment most typically
used to maximize network utilization. A second method is defined by IETF RFC
2250 which describes how transport stream packets may be encapsulated
using the real-time transport protocol (RTP). Usually, transport stream packets
encapsulated using RTP are transported using UDP / IP. Using RTP to encapsulate
transport stream packets provides a number of operational advantages. This
includes using RTP timestamps and sequence numbers for error recovery and
retransmission; and, RTP also provides performance metrics that may be useful
for service providers. A tradeoff exists, however, between the benefits realized
from the information included in RTP headers and the additional network capacity
needed to transport the headers. The following diagrams show how transport
stream packets are typically encapsulated for IP transport.
Testing Audio/VideoQuality in the Home Network
5 • SPIRENT APPLICATIoN NoTE
3. FACToRS AFFECTINg AUDIo/VIDEo QUALITy
Audio/video streams compressed using common codecs and delivered to
consumers using IP packet-based transports are quite susceptible to a number
of factors that may reduce quality. generally, the factors contributing to quality
degradation may be grouped into three broad categories—video codec selection
and codec bit rate, packet loss, and packet delay variation.
3.1. video codec and codec Bit rate
A relationship exists between the amount of compression applied to a
video stream and the quality of the video, where more compression reduces
quality and less compression increases quality. This relationship has a few
dependencies such as the type of video codec selected, how the codec bit rate
is controlled (either a constant or variable bit rate) and the resolution of the
compressed images. The degradation caused by compression may include
blockiness, blurring, jerkiness, pixelization or other impairments visible to the
viewer. Since codec selection and bit rate typically are attributes controlled
by systems engineers at the head end, degradation from codec may be
differentiated from degradation caused by network transport such as packet
loss or packet delay variation.
3.2. Packet loss
Packet loss which affects quality may occur due to a number of factors including
network congestion, reduced network capacity, and physical and logical link
errors. often, packet loss occurs in “bursts” or periods during which significant
numbers of packets are dropped. Techniques like forward error correction
(FEC), interleaving, packet loss concealment and error recovery/retransmission
commonly are applied to protect audio/video streams from packet loss. yet,
occasionally packet loss is sufficient enough to exceed the capabilities of these
techniques and severe quality degradation can occur. Even minimal amounts
of uncorrected packet loss will affect a video stream and may be noticed by
the viewer. Typically, the degradation from packet loss observed by the viewer
will appear to be distorted blocks or slices of the video stream, black screens,
screen “freezes”, or gaps in the audio stream.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 6
Understanding how packet loss affects a video stream requires special
consideration. Recall that the video codecs used by most service providers
combine a sequence of successive frames into goPs, each of which includes
one intra-coded I-frame and a series of inter-coded P-frames and B-frames
that represent the motion vectors from prior frames. Packet loss affecting the
I-frame or a P-frame in the group of pictures will propagate through all of the
following P-frames and B-frames and significant degradation can occur. Packet
loss affecting a B-frame, however, will not affect the other frames included in
the goP and therefore degradation may be limited or even not noticed by the
viewer. Thus, the effects of packet loss on the quality of a video stream will vary
depending on where the packet loss occurs within the stream. The following
figure illustrates this concept.
3.3. Packet delay variation
Packet delay variation often is caused by network congestion, limited network
capacity or the grouping of many audio/video streams (typically multicast) into
the same queues, buffers and schedulers of the network elements that audio/
video streams transit from their source towards their destinations. Packet delay
variation may cause audio/video streams to overflow or underflow the de-jitter
buffers of decoders which are designed to allow the audio/video streams to be
presented to the viewer at a constant rate. Decoders will discard audio/video
packets which arrive at the buffer too early (overflow) or too late (underflow) and
quality degradation will subsequently occur. The effects of quality degradation
observed by the viewer are similar to those caused by packet loss, such as
distorted blocks or slices, black screens and/or “freezes.” Audio distortion or
gaps in the audio may also be present.
Testing Audio/VideoQuality in the Home Network
7 • SPIRENT APPLICATIoN NoTE
4. AUDIo/VIDEo QUALITy METRICS
The following sections describe the metrics reported by Spirent Tech-X Flex
which may be utilized to assess the quality of the audio/video streams delivered
to the customer’s home.
4.1. Quality of experience metrics
The Tech-X Flex uses a mean opinion score (MoS) to describe the overall
quality of the audio/video streams under test. The MoS is a number on a scale
between 1.0 (worst quality) and 5.0 (best quality) that represents how a person
viewing the program would rate the experience. Typically, a MoS greater than 4
indicates a stream that will provide a satisfactory experience. A MoS between
3.0 and 4.0 describes stream that has suffered some quality degradation
and will offer a sub-optimal viewing experience. A stream rated below 3.0
has significant quality degradation and a viewer would likely consider the
viewing experience unsatisfactory. The following table includes more detailed
descriptions of the MoS values reported by the Tech-X Flex.
metric Description
mos-v A MoS that describes the overall quality of the video stream included
in the program under test. This metric considers the video codec bit
rate, goP type and length and packet loss distribution.
mos-a A MoS that describes the overall quality of the audio stream included
in the program under test. The metric considers audio codec bit rate,
audio sampling rate and packet loss distribution..
mos-av A MoS that describes the overall quality of the audio and video
streams present in the program under test. This metric considers
audio and video quality as well as the synchronization between the
audio and video streams.
The audio/video program’s MoS values may be used by technicians to identify
quickly whether the quality of the viewing experience will be satisfactory and to
assess whether performance issues may need to be addressed.
4.2. expert analysis metrics
The Tech-X Flex offers an expert analysis of the measurements performed during
audio/video quality testing. The expert analysis is designed to isolate the
cause(s) of the stream degradation that may be affecting the viewing experience.
The expert analysis provides three metrics which are defined in the table below.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 8
metric Description
degradation from loss The percentage of overall quality degradation caused by packet loss.
degradation from Jitter The percentage of overall quality degradation that may be attributed
to de-jitter buffer packet discards (overflows or underflows).
degradation from codec The percentage of overall quality degradation caused by stream com-
pression. Some percentage of degradation from the codec is always
displayed by the Tech-X Flex since the analysis considers the quality
of the original uncompressed audio/video streams. The amount of
degradation caused by codec will depend upon the codec type as
well as the bit rate of the compressed streams.
The information offered by the Tech-X Flex expert analysis can assist with
differentiating between the causes of degradation and allow technicians to
isolate issues more effectively. For example, degradation from jitter commonly
is caused by network congestion. Significant congestion will introduce sufficient
delay between audio/video packets and will force the decoder’s de-jitter buffers
to either overflow or underflow and therefore discard packets. often, congestion
occurs upstream from a customer’s residence, at a customer’s residential gateway
or perhaps within the residential network due to competition with other services
and applications for network capacity. Quality degradation from packet loss may
occur due to network congestion as well; however, other sources of packet loss
may be present including link errors and other transmission or physical media
issues. Determining the cause of audio/video degradation from packet loss will
require a more thorough investigation that addresses all of these factors.
4.3. video stream metrics
The Tech-X Flex reports several metrics that describe the specific attributes of the
video stream under test, the video stream’s bit rate and the correlation between
transport stream packet loss / discards and the video frames affected. The
following table provides descriptions of these metrics.
Testing Audio/VideoQuality in the Home Network
9 • SPIRENT APPLICATIoN NoTE
metric Description
image size The horizontal x vertical resolution of the video stream under test
expressed using pixels.
image type A short description of the video frame resolution. This may be SDTV
(standard-definition television) or HDTV (high-definition television).
codec type The codec used to compress the video stream under test. Examples
include MPEg-2 and H.264.
goP type A description of the types of frames included in the goP and the
way the frames are organized, for example: IBBPBB…
goP length The total number of I, P and B frames included in a goP by the
video stream’s encoder.
frames received The total number of frames received during the measurement,
organized by type – I-Frames, P-Frames and B-Frames.
Packets lost The total number of MPEg-2 transport stream packets lost during
the measurement by frame type.
Packets discarded The total number of MPEg-2 transport stream packets discarded
during the measurement by frame type.
frames impaired The total number of frames by type impaired during the measure-
ment due to transport stream packet loss and/or discards.
receive rate The average rate at which the video stream is received by the test
set expressed using kilobits per second.
Peak receive rate The peak rate at which the video stream is received by the test set
expressed using kilobits per second.
Note that the Tech-X Flex video stream image, codec, goP and frame metrics
provided may be used to optimize codec selection and the goP type and length
selected for the video stream.
4.4. transport Quality metrics
The Tech-X Flex provides several metrics that allow technicians to assess the
quality of the network transporting the stream under test. generally, these
metrics characterize the packet loss and packet delay variation added to the
stream while transiting from the source to the measurement point. The following
table provides further description of these metrics.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 10
metric Description
Packets lost The total number of transport stream packets lost during the mea-
surement period.
Packets discarded The total number of transport stream packets discarded by the de-
jitter buffer emulator during the measurement period due to over-
flows and/or underflows.
Packets out-of-sequence The total number of transport stream packets received out-of-se-
quence during the measurement period.
gap Period loss rate The average packet loss rate measured during gap loss periods.
gap Period loss length The average packet loss length measured during gap loss periods
expressed in milliseconds.
Burst Period loss rate The average packet loss rate measured during burst loss periods.
Burst Period loss length The average packet loss length measured during burst loss peri-
ods expressed in milliseconds.
maPdv The mean absolute packet delay variation. This is the delay varia-
tion experienced by stream packets relative to the short term
running average of the delay calculated for prior packets.
PPdv The packet-to-packet delay variation. This is the statistical vari-
ance of the delay experienced by consecutive stream packets
over the measurement duration.
The packets lost/discarded metrics, as well as the gap/burst loss metrics, allow
technicians to more accurately qualify the “burstiness” of packet loss which
directly corresponds with the viewer’s quality experience. The reported packet
delay variation metrics may be compared to the de-jitter buffer sizes supported
by decoders to permit technicians to estimate whether or not overflows or
underflows will occur. Additionally, the packet loss and delay variation metrics
may be correlated with service level specifications established for the customer to
determine whether or not satisfactory quality of service is provided.
The Tech-X Flex also reports the media delivery index (MDI) which is a set of
metrics that describe the quality of the network used to transport the audio/video
streams. MDI includes two measurements—media loss rate (MLR) and delay
factor (DF). The MLR is the number of transport stream packets lost per second.
The DF describes the variation of the bit rate of the transport stream under test
relative to a constant bit rate and calculates the amount of de-jitter buffer (or
time) that will be needed to accommodate the bit rate variation and avoid buffer
overflow and underflow. Essentially, the DF estimates the size of the de-jitter
buffer required to assure satisfactory audio/video quality. The following table
provides descriptions of the MDI metrics provided by the Tech-X Flex.
Testing Audio/VideoQuality in the Home Network
11 • SPIRENT APPLICATIoN NoTE
metric Description
mlr The number of transport stream packets lost per second.
average df The average delay factor calculated over the measurement duration.
minimum df The minimum delay factor calculated over the measurement
duration.
maximum df The maximum delay factor calculated over the measurement
duration.
receive rate The actual rate of the transport stream under test expressed us-
ing kilobits/second.
5. TESTINg AUDIo/VIDEo QUALITy METHoDoLogy
Testing the quality of a residential audio/video service commonly occurs
throughout the service lifecycle, including pre-qualification to determine whether
or not the service may be offered, following equipment installation and service
provisioning, after equipment and/or service upgrades and while responding to
troubles which affect the service quality. The following information describes
the methodology for using the Tech-X Flex to measure and verify the health of an
audio/video service and generally applies to each of these situations.
The Tech-X Flex operates using two modes—host mode and passive mode. Host
mode allows the Tech-X Flex to emulate the customer’s set top box (STB) to
validate audio/video quality. Typically, host mode testing is used to assess audio/
video streams transported using IP multicast. The following diagram illustrates
how the Tech-X Flex is connected to the residential network to conduct testing
using host mode.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 12
Testing using passive mode uses two Tech-X Flex interfaces which allows
the Tech-X Flex to be connected “in-line” with the audio/video stream on the
residential network. This configuration allows the stream to be tested while
simultaneously being viewed on and controlled by customer equipment. often,
passive mode testing is used to measure streams that are transported using
IP unicast such as video-on-demand (VoD) or multi-room DVR programs. The
following diagram shows how the Tech-X Flex is attached to the home network
using passive mode.
After determining whether host mode or passive mode should be used, multiple
measurement points targeting each segment of the home network may be
tested to qualify the service and equipment or locate specific troubles. The
following diagram illustrates different possible measurement points.
Testing Audio/VideoQuality in the Home Network
13 • SPIRENT APPLICATIoN NoTE
Typically, the first measurement point should be “upstream” from the residential
gateway to evaluate the quality of the access network link (e.g. DSL, MoCA) and
to determine audio/video quality before the stream transits the home network.
The MoS values (MoS-V, MoS-A, MoS-AV) reported by the Tech-X Flex may
be used to quickly verify quality. Remember that MoS values greater than 4
typically mean that quality is satisfactory at the measurement point, otherwise
quality issues may exist. The following image illustrates how MoS values are
reported by the Tech-X Flex.
Multiple quality measurement reports with MoS values below 4 demonstrate
that the quality of the streams and/or network upstream from the home will
not provide satisfactory service for the customer. In this case, a technician may
refer the issue to the appropriate operations center. otherwise, a technician
may proceed to test directly downstream from the residential gateway, shown
as measurement point 2 in the previous diagram. The same testing procedure
may be followed using the same MoS criteria, with unsatisfactory MoS values
leading to the investigation of the residential gateway and related interfaces and
links. otherwise, if the residential gateway is performing properly, a technician
may continue to validate quality throughout the home network by testing at
STBs, shown in the previous diagram as measurement point 3. This allows the
technician to verify the links between the residential gateway and the STBs
and to differentiate between issues occurring on the residential network and
problems associated with the customer’s equipment.
Testing Audio/Video Quality in the Home Network
SPIRENT APPLICATIoN NoTE • 14
After audio/video quality measurements which report unsatisfactory MoS at
any measurement point within the home network, a technician may leverage the
additional metrics available with the Tech-X Flex to further identify and resolve
service issues. The technician should start this process by determining the
factor(s) that are causing the quality degradation such as video codec, packet
loss and/or packet delay variation. The Tech-X Flex expert analysis, displayed in
the following image, summarizes this information.
The technician then may pursue the root cause of the issue, perhaps network
congestion or reduced network capacity causing degradation from jitter (packet
delay variation), or possibly physical/link layer issues causing degradation from
loss. The additional packet delay variation and packet loss metrics provided by
the Tech-X Flex may be used to fully understand the characteristics of the quality
degradation and troubleshoot the issue more completely.
Testing Audio/VideoQuality in the Home Network
15 • SPIRENT APPLICATIoN NoTE
6. SUMMARy
Competition between audio/video service providers has escalated significantly
over the past few years and the ability to assess the home viewing experience
can be a key factor when attracting and retaining customers. This effort
requires advanced test tools that accurately assess the quality of the audio/
video at the portion of the service delivery network most at risk—the customer’s
home. Spirent’s Tech-X Flex in-home field tester has a complete audio/video
analysis solution which may be used throughout the lifecycle of the audio/video
service to guarantee the quality of experience and allow providers to install
equipment, provision services and troubleshoot and resolve quality issues more
efficiently. Additional information about Spirent’s Tech-X Flex is available at
www.spirent.com or by contacting your Spirent account manager
