The Emerging 100GE Network Era
A Heavy Reading White Paper
100 Gigabit Ethernet (100GE) has been described by senior service provider experts as one of the most innovative and promising technologies to hit the market during the past 18 months. Network operators worldwide are excited about the potential of 100GE to help them (1) handle explosive traffic demand that is pushing existing 10GE networks to their limit, (2) simplify their networks, (3) deal more efficiently with new traffic patterns; and (4) affordably scale with customer requirements by leveraging Ethernet’s proven ability to deliver remarkable performance/price advances over time.
This paper will provide an update on major 100GE ecosystem developments that should give confidence to the service provider community that we are approaching the 100GE era at a quickening pace.
White Paper
The Emerging 100GE Network Era
Prepared by
Stan Hubbard
Senior Analyst, Heavy Reading
www.heavyreading.com
Part of the Light Reading/Heavy Reading 100GE Initiative
on behalf of
December 2011
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 2
Executive Summary
100 Gbit/s Ethernet (100GE) has been described by senior service provider experts
as one of the most innovative and promising technologies to hit the market during
the past 18 months. Network operators worldwide are excited about the potential
of 100GE to help them (1) handle explosive traffic demand that is pushing existing
10GE networks to their limit, (2) simplify their networks, (3) deal more efficiently with
new traffic patterns; and (4) affordably scale with customer requirements by
leveraging Ethernet’s proven ability to deliver remarkable performance/price
advances over time.
A key question operators have is: ―When will 100GE network solutions become
affordable enough so that they can be deployed beyond limited applications
and be used in strategic ways to control network costs and scale with revenue
opportunities?‖ The answer to this question will vary from operator to operator –
depending on their particular network and customer requirements. Some compa-
nies already have begun to deploy 100GE strategically for data center intercon-
nection, research, core, and metro applications, while others are testing the
waters and waiting for more affordable 2nd generation platforms to become
available before they engage in significant deployments.
An important goal of this paper is to provide an update on major 100GE ecosys-
tem developments that should give confidence to the service provider communi-
ty that we are approaching the 100GE era at a quickening pace. 18 months after
the IEEE ratified the first 100GE standards, work has progressed to the point that the
industry now has a full set of specifications for transmitting 100GE up to 2 km, 10
km, and 40 km. Multiple component suppliers are shipping 100GE optical modules
that can reach up to 2 km or 10 km. A number of component companies are now
shipping fixed-function Ethernet ASICs and programmable network processors for
100GE line cards. At least a half-dozen network equipment vendors are now
shipping platforms with 100GE cards. And test suites are available and being used
to cover Layer 1 through 7 performance and scalability on these systems. Several
dozen network operators and research organizations have engaged in 100GE
tests, demonstrations, and trials. Finally, we estimate that at least two dozen
companies and organizations have begun deploying 100GE technology as of the
end of 2011.
Going forward, we expect a combination of the (1) growing volume of 100GE-
related component and systems shipments and (2) introduction of systems based
on 2nd-generation components will help drive down systems cost and accelerate
deployments. Heavy Reading expects to see an increase in both limited and
strategic network deployments of 100GE in 2012 and believes the market is likely to
hit its stride before the end of 2013.
This white paper is part of Light Reading/Heavy Reading's 100GE Industry Initiative
Program that includes ongoing editorial coverage of 100GE-related topics, a
100GE webinar and LRTV video on the 100GE topic. The 100GE Briefing Center on
Light Reading includes this and other important material.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 3
Introduction
In December 2010, Heavy Reading published our first 100GE white paper in which
we explored the emergence of the 100GE-service enabling architecture. We
highlighted strong operator interest in the technology, spotlighted ecosystem
developments, and identified challenges that remained to be overcome before
we could see widespread adoption of 100GE.
In this white paper, we seek to bring readers up to speed on important 100GE
ecosystem developments that have taken place over the past year. Heavy
Reading’s analysis of these developments has given us increased confidence that
the pieces are falling into place that will enable the market to grow at an accel-
erating pace in the coming quarters.
This first section of this paper (1) briefly summarizes primary carrier drivers for the
deployment of 100GE technology on switches and routers, (2) discusses key
developments within the 100GE ecosystem, and (3) shares webinar survey and
other industry feedback that sheds light on expectations for deployment of 100GE
technology. The ecosystem section reviews progress related to: (a) standards and
specifications, (b) optical modules, (c) packet processing & packet management;
(d) network equipment; (e) test solutions; (f) equipment & component tests and
demonstrations, (g) network operator demonstrations and trials, and (h) network
deployment activity and plans.
The second section of this paper includes two-page perspectives on 100GE
technology provided by Alcatel-Lucent, Brocade, and Spirent.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 4
100GE Market Drivers
This section provides a brief summary of the primary drivers for carrier interest in
100GE technology. A more extensive discussion of key drivers can be found in
Heavy Reading’s The Emerging 100GE Service-Enabling Architecture white paper
published in late 2010.
There are at least five main drivers of network operator interest in deploying 100GE
technology.
Bandwidth demand continues to grow at a rapid pace. Service providers
worldwide continue to report year-over-year double-digit traffic growth relat-
ed to next-gen business, residential, and mobile services and applications.
They particularly are concerned about explosive growth in mobile broadband
data and video; high-capacity Ethernet and optical business, wholesale, and
mobile backhaul services; and cloud services and cloud-based applications.
Operators need options as traffic pushes 10GE networks to their limits. 10GE
and wavelength pipes are carrying ever-increasing amounts of digital fuel to
keep up with the growing and unpredictable bandwidth needs of social net-
working, entertainment, mobile, cloud, and other applications in the con-
nected culture. The challenge here is that many of the carrier networks built
with 10GE interfaces on routers and switches are going to be pushed toward
their limits over the next couple of years. Already, many are using n x 10GE
links to interconnect their major sites. AT&T, NTT, BT, Verizon, Level 3, Interoute,
tw telecom, AboveNet, XO, China Telecom, and other large operators have
expressed strong interest in shifting to 100GE because of the future-proof
headroom offered by its order of magnitude (tenfold) bandwidth gain.
Industry feedback indicates there is strong and growing demand for using
100GE for data center interconnection, submarine, core, and metro applica-
tions. We also have heard about emerging interest in 100GE for access use.
Operators want to simplify their networks. Among other things, service provid-
ers are seeking to (1) operate fewer, higher-capacity links, (2) overcome link
aggregation limitations, (3) simplify routing, and (4) simplify MPLS in their net-
works.
Operators need more efficient ways to handle new traffic patterns. In both the
edge and the core, network operators are looking to take better advantage
of statistical multiplexing to handle unpredictable traffic patterns from new
applications, "migrating" content (video distribution and clouds), and new
connectivity paradigms (i.e., mobility).
Operators expect to ride down the Ethernet cost curve. Over the past 15 years,
Ethernet and optical DWDM technologies have delivered the greatest per-
formance/price improvements of any wireline technology. Nothing else
comes close. One of the major reasons for Ethernet's success is the fact that it
benefits from an extraordinary level of strategic investment, industry focus,
collaboration, and high production volumes. Despite today’s concerns about
the initial cost of 100GE vs. 10 x 10GE solutions, many industry players expect
100GE to become increasingly competitive and ultimately be a choice tech-
nology for building high-capacity networks.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 5
100GE Ecosystem Is Getting Stronger
The 100GE ecosystem has developed significantly during the 18 months since the
IEEE ratified the 802.3ba standard. Figure 1 summarizes key developments that are
moving the industry forward, creating opportunities for 100GE deployments by
early adopters in 2011 and 2012, and likely paving the way for widespread use of
100GE interfaces on carrier Ethernet switches and routers in 2013. We will elabo-
rate in more detail on the ecosystem developments in the rest of this section.
Figure 1: 100GE Ecosystem Developments
ECOSYSTEM
ELEMENTS IMPORTANT DEVELOPMENTS
Standards &
Specifications
In June 2010, the IEEE ratified the P802.3ba standard that defined
100GE and 40GE MAC and physical layer (PHY) standards on network
devices like routers and switches. Also in 2010, the ITU-T defined a 112
Gbit/s OTU4 signal that can encapsulate 100GE.
In 2011, the 10x10 MSA industry group approved technical specifica-
tions to create compatible sources of lower-cost, lower-power,
pluggable optical modules based on 10G signaling specifications not
covered by P802.3ba. This includes new 2 km,10 km, and 40 km specs.
The OIF, meanwhile, has been working on initiatives designed to
enable lower-cost interfaces and higher-density 100GE line cards.
Optical Modules Suppliers have shipped 100GE optical modules for 2 km and 10 km. One vendor also has formally introduced a 40 km module.
Packet
Processing &
Packet
Management
Multiple companies are shipping fixed-function Ethernet ASICs and
programmable network processors for 100GE/OTU-4, including support
for traffic management with hierarchical per-service and per-user
packet buffering and shaping.
Network
Equipment
Multiple equipment suppliers are now shipping platforms with 100GE
line cards.
Test &
Measurement
100GE and OTU4 test suites are available that cover Layer 1 through 7
performance and scalability. Tests have been conducted for IPTV,
VoIP, VOD, HDTV, gaming, and multimedia streaming over 100GE.
Network
Equipment
Tests & Demos
Multiple equipment vendors tested and demonstrated 100GE inter-
faces on core routers, Ethernet service edge (ESE) routers, and carrier
Ethernet switch/routers (CESR) in 2010, and this activity ramped in
2011. This coincided with similar activity related to 100G transceivers
on DWDM and packet-optical transport systems.
Network
Operator Demos
& Trials
The past year also has been characterized by ramping 100GE demon-
stration and trial activity that reportedly has involved dozens of
network operators, research organizations, and universities worldwide.
Network
Deployments
Industry feedback suggests at least two dozen service providers and
research organizations on multiple continents have begun deploying
100GE technology, as of the end of 2011.
Sources: IEEE, ITU-T, service providers, network equipment providers, research institutions,
component suppliers, and test & measurement companies
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 6
100GE Standards & Specifications
While initial 100GE-related standards were finalized in 2010 by the IEEE and ITU-T,
leading industry players have continued developing and approving new specifi-
cations in an effort to fill in perceived standards gaps and to drive
price/performance improvements that will help accelerate technology adoption.
The industry now has a full set of specifications for transmitting 100GE up to 2 km,
10 km, and 40 km. Important activity is listed below.
IEEE
In June 2010, the IEEE approved the P802.3ba-2010 100GE & 40GE standard. The
fiber-related 100GE PHY standards covered include the following:
100GBASE-SR10. 10 Gbit/s x 10 lanes (fiber pairs) over OM3 multi-mode fiber
(MMF) for a distance of at least 100 meters and OM4 MMF for a distance of at
least 150 meters.
100GBASE-LR4. 25 Gbit/s x 4 WDM lanes over single-mode fiber (SMF) for a
distance of at least 10 km.
100GBASE-ER4. 25 Gbit/s x 4 WDM lanes over SMF for a range of at least 40 km.
ITU-T
In June 2010, ITU-T Study Group 15 and the IEEE 802.3ba Task Force agreed on a
common mapping between 802.3ba and ITU-T G.709/Optical Transport Network
(OTN) standards to ensure that 100GE is transportable over optical networks. The
OTU4 signal has a data rate of 112 Gbit/s and thus is capable of carrying 100GE.
10x10 Multi-Source Agreement Group
In December 2010, Google, Brocade, JDSU, and Santur (recently acquired by
NeoPhotonics) created the 10x10 MSA group to deliver the lowest-cost 100GE
solution over SMF. An additional 21+ companies subsequently joined them,
including Facebook, AMS-IX, Hitachi Cable America, MRV, and others.
10x10 up to 2 km. In March 2011, the 10x10 MSA group announced the
ratification of a new 100 Gbit/s module specification that directly maps 10
electrical lanes at 10 Gbit/s onto 10 lasers and supports transmission over SMF
up to 2 km. Google was one of the key proponents because (1) 100GBASE-
SR10 does not provide enough reach to address many of its data center re-
quirements and (2) the longer-reach 100GBASE-LR4 was viewed as too costly
and energy inefficient for shorter distance applications.
10x10 up to 10 km. In August 2011, the 10x10 MSA group ratified a new 10x10
specification that extends reach to 10 km but also is backward compatible
with the 2 km specs. This approach has been described by supporters as more
cost effective than the first generation of 100GBASE-LR4 implementations be-
cause it does not require any 25 Gbit/s electronics, such as gearbox integrat-
ed circuits used to convert 10 Gbit/s streams to 25 Gbit/s lanes.
10x10 up to 40 km. Also in August 2011, the 10x10 MSA group created a
specification for extending reach up to 40 km. This builds upon existing 10
Gbit/s DWDM specifications to allow existing 10 Gbit/s metro DWDM links to be
upgraded to 100 Gbit/s over SMF without regeneration or amplification.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 7
OIF
The OIF has been engaged in multiple projects to steer components development
in a common direction and to accelerate deployment of 100G transmission
technology in long-haul DWDM networks. One of the most important areas of
development is the organization’s work on 28 Gbit/s electrical signaling that is
designed to permit lower cost interfaces and higher density 100GE line cards.
CEI-25G-LR & CEI-28-SR. In September 2011, the OIF approved a Common
Electrical Interface (CEI) agreement that includes CEI-25G-LR and CEI-28G-SR
interfaces. The 25 Gbit/s LR interface is used in the backplane to connect line
cards, while the 28 Gbit/s SR interface is used to connect chips within a line
card.
CEI-28G-VSR. The OIF also is defining a lower power, very short reach (VSR) 28
Gbit/s interface (CEI-28G-VSR) that will be used to connect a chip on a line
card with the plugged-in 100GE media module. The CEI-28-VSR enables mod-
ule vendors to use 25 Gbit/s electrical signaling within the CFP and thus elimi-
nates the need to have a 10:4 gearbox convert 10 Gbit/s electrical streams to
25 Gbit/s optical lanes. This promises to reduce complexity, cost, space, and
power requirements. The CEI-28G-VSR implementation standard is targeted to
be completed in early 2012.
Optical Modules
Over the past year, the industry has made significant progress in increasing 100GE
CFP availability and developing common specifications for the lower-cost short
reach (2 km) option, but customers have continued to express concern that the
cost of the longer distance modules remains a near-term obstacle to widespread
adoption. The availability of 10x10 10 km CFPs is a step toward improved afforda-
bility in the near-term, but many experts believe we will not see major perfor-
mance/price advances until after the second generation of long-reach 4x25 CFPs
begins generally shipping sometime after mid 2012.
Figure 2: Example Suppliers of 100GE CFP Optical Modules
SUPPLIER
802.3BA STANDARD 10X10 MSA
100GBASE-LR4
(10 KM)
100GBASE-ER4
(40 KM) 2 KM 10 KM 40 KM
Finisar
Fujitsu Optical Components
NeoPhotonics (Santur)
Opnext Design Demo
Reflex Photonics
Sumitomo Electric
Sources: Module suppliers.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 8
1st Generation CFPs – Ramping Shipments But Cost Remains A Big Concern
Optical module suppliers currently are shipping first-generation CFPs based on the
100GBASE-LR4, 100GBASE-ER4, 10x10 10 km, and 10x10 2 km specifications. These
CFPs have been used in a variety of 100GE-related tests, demonstrations, trials,
and early deployments, some of which will be discussed later in this paper.
In December 2011, the CEO of Finisar stated that business for the company’s
100GBASE-LR4 CFP is ―doing very, very well‖ and that revenue ―continues to
grow quickly.‖ Finisar reportedly is shipping every product that it can make
and expects to be capacity-limited through Q1 2012 due to strong demand.
In November 2011, the CEO of Opnext said that his company is seeing solid
demand from multiple customers for its 100GBASE-LR4 module. In March 2011,
Opnext demonstrated design feasibility of an early version of a 100GBASE-ER4,
but it has not announced commercial availability of such a module yet.
In July 2011, Santur (NeoPhotonics) announced the beginning of commercial
shipments of the 10x10 2 km modules. This complements the company’s pio-
neering 10x10 10 km modules that have been used in customer line cards in
the past couple of years.
In 2Q 2011, Fujitsu Optical Components began shipping a 100GBASE-LR4
module as well as the industry’s first available 100GBASE-ER4 module.
Despite this progress, currently available 10 km CFPs frequently are still being
described by senior service provider technology experts and other industry players
as too expensive to enable broad adoption of 100GE technology today.
During Ethernet Expo in November 2011, Level 3’s SVP of Transport and
Infrastructure Services described the 100GE standardization work as one of the
best examples of technology innovation in the market over the past 18
months and noted that Level 3 has several 100 Gigabit deployments under-
way. But, he said, ―The cost points around 100 Gig technology and interfaces
are still at the point where it is still not affordable for us to use as our main
transport methodology.‖
The CTO of a large U.S. operator told Heavy Reading in late October 2011 that
the 100GBASE-LR4 modules, in particular, are ―very expensive across all ven-
dors.‖ According to the CTO, this high expense has been ―spoiling the party.
People are kind of accepting it, but it constrains the rate of acceptance.‖
2nd Generation CFPs – Addressing Cost Concerns
Optical module suppliers have gotten the message and are working on 2nd
generation CFPs that promise to deliver more attractive cost points and help drive
the type of volume shipments that will enable economies of scale. Among other
things, newer modules will have a smaller form factor that enables greater line
card density and will be designed to consume significantly less power than 1st
generation solutions. An important difference will be the introduction of 4x25
Gbit/s electrical signaling enabled by the CEI-28-VSR interface, which will remove
current electrical conversion complexities inside the CFP.
Estimates from component and systems vendors suggest we will see CFP2s
introduced by mid 2012, with meaningful market growth and impact in 2013.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 9
Packet Processing & Packet Management
Looking inside the line card, multiple companies are now shipping fixed-function
Ethernet ASICs and programmable network processors for 100GE/OTU-4, including
support for traffic management with hierarchical per-service and per-user packet
buffering and shaping. As a senior expert at Xelerated recently noted, suppliers of
100GE line-rate packet processors fundamentally have been using the same type
of approach for scaling from 10GE to 100GE that they did when they scaled from
1GE to 10GE. There has not been a disruptive shift in terms of basic technologies
and architectures, so the major cost challenges seen on the optical side have not
really been present in developing 100GE packet processors.
Solutions that support 100GE packet processing with services on a single chip are
now available from merchant NPU suppliers as well as in-house development
teams within systems vendors.
NPU suppliers currently are focused on integrating packet processor, forwarding
engine, traffic manager, and buffer manager functions into fewer chips to help
accelerate commercial deployments by driving out cost and reducing power
consumption.
In mid 2011, Alcatel-Lucent introduced the FP3 network processor that is designed
to process 400 Gbit/s at line rate while offering more features with 50% less power
per bit than the FP2. The vendor has demonstrated the FP3 chip in labs, and line
cards with the new chip are scheduled to ship commercially by mid 2012.
Figure 3: 1st & 2nd Generation 100GE CFPs
Source: Brocade, IEEE, 10x10 MSA, OIF
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 10
Network Equipment
Multiple network equipment vendors – including Alcatel-Lucent, Brocade, and
Cisco, and several others – have shipped switching and routing platforms with
100GE line cards for tests, demonstrations, trials, and early deployments. 100GE is
now available on core, service edge, and aggregation products. In addition, one
vendor, Canoga Perkins, told the Ethernet Expo audience that it expects to begin
commercial shipments of a 100GE access platform by mid 2012.
Test & Measurement
Over the past three years, Spirent and other test and measurement suppliers have
introduced 100GE testing solutions and demonstrated successful transmission and
reception of traffic through their testing modules and CFP optical modules. Several
of these tests not only have taken place over carrier network links, but multiple
network operators reportedly are now using equipment to test deployment
scenarios.
Test and measurement companies have designed solutions to be standards-
compliant and also to provide testing capabilities that go beyond the basic
requirements. Spirent, for example, tests Layer 1-7 performance and scalability,
with generation of multi-play traffic and video streaming with quality of experi-
ence (QoE) measurements. Spirent also supports large-scale multi-protocol testing
and real-time routing and traffic updates.
As illustrated in Figure 4, 100GE testing life cycle can be broken down into three
main stages, which are associated with the different layers of the OSI model.
Layers 1-2. Much of the testing that has taken up until now has focused on
Layers 1 to 2 ―bit blasting‖ with 1st generation technology. This includes bit error
rate, packet integrity, throughput, stream integrity, and forwarding capacity
testing. According to Spirent, the testing reveals that the ―optics are very solid
now.‖ As we enter 2012, we will see Layer 1-2 testing with 2nd gen hardware.
Figure 4: Current Status of 100GE Testing
Source: Spirent
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 11
Layers 2-3. The next stage of testing that already has started or is currently
planned for multiple equipment vendors focuses on control plane & data
plane functionality. This includes QoS, routing performance, MPLS, and IPv6
testing.
Layers 4-7. The final stage of testing deals with application performance. This
includes quality of experience (QoE), security, and the performance of mobili-
ty, video, and other applications.
One of the most interesting observations made when comparing 100GE and 10GE
testing is that the industry is experiencing accelerated testing of 100GE functionali-
ty and compression of the testing cycle. The Spirent Perspective in the second half
of this paper discusses this trend in more detail. Service providers essentially expect
all of the same services and capabilities supported on 10GE ports to be available
on 100GE ports. This has put pressure on software engineers to start testing higher
layer features even before they have the hardware ready.
Network Equipment & Component Tests/Demonstrations
Many 100GE network equipment & component tests/demos have taken place
since mid 2010. Below are some highlights from recent months.
Figure 5: 100GE Network Equipment & Component Tests/Demonstrations
DATE TESTS & DEMONSTRATIONS
Oct 2011
For MPLS 2011, Brocade and a test supplier demonstrated that traffic can be forwarded for
LDP-based VPLS services and that 802.1ag Ethernet connectivity fault management can be
supported on 100GE interfaces.
Sep 2011
In a showcase hosted by EANTC, the Brocade MLXe-4, Spirent TestCenter, and another test
supplier configured IP/MPLS over 100GE interfaces, and the test equipment suppliers emulat-
ed multiple PE devices connected to a VPLS service built with the MLXe-4. Baseline perfor-
mance monitoring tests were conducted over the MLXe-4 interface.
Sep 2011 Alcatel-Lucent used Spirent’s TestCenter to validate the performance and scale of its FP3-based 100GE router modules under real-world service and traffic scenarios.
Jul 2011
As part of Light Reading’s 100GE test program, EANTC and a test supplier validated 100GE
capabilities on Alcatel-Lucent’s 7750 SR-12, with a 10x10 10 km CFP used at that time in the
100GE line card (the vendor also supports 100GBASE-LR4 CFPs). Among other things, the 7750
(1) demonstrated line rate performance for IPv4, IPv6, and a mix of both, (2) passed QoS
verifications, and (3) demonstrated multicast service scalability with almost no issues.
Apr 2011
The Ethernet Alliance organized its second 100GE interoperability test, which took place at
the University of New Hampshire interoperability lab. Fourteen members – including Brocade,
Spirent, Broadcom, and others – verified interoperability of switches, routers, test equipment,
transceivers, and NICs. Basic Level 3 traffic was used to verify error-free operation.
Source: Light Reading, EANTC, Alcatel-Lucent, Brocade, Spirent, and Ethernet Alliance.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 12
Demonstrations & Trials Involving Network Operators
The past 15 months has been characterized by ramping 100GE demonstration and
trial activity that reportedly has involved dozens of network operators, research
organizations, and universities worldwide. Examples of announced trials and
deployments are listed below.
Figure 6: 100GE Demonstrations / Trials Involving Network Operators
DATE OPERATORS DEMONSTRATION / TRIAL
Nov
2011 ESnet, Internet2
At SC11, as part of SCinet, ESnet used multiple Alcatel-Lucent 7750 SR platforms
for a series of demonstrations including ―Visualizing the Universe at 100 Gbps‖,
presented by ESnet, LBNL and NERSC and ―Scaling the Earth System Grid to 100
Gbps Networks‖.
Nov
2011
National Lambda
Rail
At SC11, Cisco teamed with National Lambda Rail to demonstrate the ability to
deliver 100GE connectivity for the National Oceanic and Atmospheric Administra-
tion. Cisco ran 100GE between interfaces on the ASR 9000 and the CRS3.
Nov
2011
NASA Goddard’s
HECN
At SC11, NASA’s High End Computer Networking team partnered with Alcatel-
Lucent, Brocade, and others to support a live, real-time demonstration of a 100GE
file transfer across a wide area network.
Nov
2011
Indiana University,
Internet2
At SC11, Indiana University conducted a high performance computing (HPC)
cloud demonstration using data-intensive scientific applications and OpenFlow
over 100GE. Internet2 contributed a 100GE circuit between Indianapolis and
Chicago. The demo included Brocade MLXe routers.
Nov
2011
BCNET, CANARIE,
University of
Victoria, Caltech
At SC11, British Columbia’s advanced network organization (BCNET), Canada’s
Advanced Research & Innovation Network (CANARIE), the University of Victoria,
and Caltech demonstrated the ability to transmit particle physics data for 24
hours at a sustained rate of 95 Gbit/s, using 100GE blades on Brocade MLXe
routers as well as 100G optical equipment from another vendor.
Sep
2011
Amsterdam
Internet Ex-
change (AMS-IX),
CERN, SURFnet
AMS-IX, SURFnet (the National Research and Education Network organization in
the Netherlands), and CERN (the European Organization for Nuclear Research)
trialed 100GE connectivity between Amsterdam and Geneva, using Brocade
routers and DWDM equipment from another vendor. The trial transferred a
constant stream of 100GE over a long distance for hours without any loss.
Jun
2011
AMS-IX & Lime-
light Networks
Brocade announced that Limelight Networks was the first service provider to field-
test a live 100GE Internet peering service provided by AMS-IX over MLXe routers.
Jun
2011
Internet Multifeed
(MF), Internet
Initiative Japan
(IIJ),NTT Commu-
nications
Japan’s largest Internet exchange provider, Internet Multifeed (MF), validated the
readiness of the country’s two largest ISPs to support 100GE with an Ethernet IXP
interoperability test on its JNAP exchange. The ISPs interconnected routers and
switches from Brocade, Cisco, and third vendor and used the Spirent TestCenter
to confirm that 100GE is ready to be deployed in an exchange environment in
terms of both interoperability and performance. See Figure 8.
Source: Select operators and network equipment suppliers.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 13
Figure 8: Internet Multifeed (MF) Internet Exchange Network Testbed
Source: Internet Multifeed (MF) in Japan.
Figure 7: 100GE Demonstrations / Trials Involving Network Operators (Continued)
DATE OPERATORS DEMONSTRATION / TRIAL
Dec
2010 True
Alcatel-Lucent announced that True conducted a successful 100GE field trial in
Thailand that demonstrated the ability to deliver 3G and LTE services, residential
broadband, subscriber management, business Ethernet and VPN services over its
existing network that includes 7750 and 7450 platforms.
Nov
2010 Portugal Telecom
PT demonstrated a 100GE connection between Lisbon and Oporto using 100GE
interfaces on Alcatel-Lucent’s 7750 and 100G optical transport equipment.
Sep
2010
Nextgen
Networks
Nextgen Networks conducted a 100GE demonstration using Alcatel-Lucent 7750
and optical equipment deployed in the operator’s backbone network.
Source: Select operators and network equipment suppliers.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 14
Early Deployment Activity & Plans
100GE was deployed on a limited basis in a handful of carrier and research
networks in 2010, but we saw a notable increase in activity in 2011 as more
equipment became available and a number of customers completed early trials.
Industry feedback suggests that at least a couple of dozen network operators and
research organizations have begun deploying 100GE. In recent months, multiple
service providers worldwide reportedly have purchased 100GE test equipment for
the type of testing that takes place in preparation for ramping deployments.
Key deployment scenarios thus far have involved data center interconnect,
research network, core, and metro applications. Around the middle of 2012, we
expect to start hearing about limited use in access applications as well.
Figure 9: 100GE Early Deployment Activity & Plans
DATE OPERATOR DEPLOYMENT ACTIVITY & PLANS
Nov
2011
Energy
Science
Network
(ESnet)
The U.S. Department of Energy’s ESnet formally launched its 100GE-based Advanced
Networking Initiative (ANI), creating the world’s fastest science network. Based on
Alcatel-Lucent’s 7750, the live prototype 100GE network currently connects three
supercomputing centers. The DOE plans to transition this network to a nationwide
production network that integrates all DOE laboratories in 2012.
Nov
2011
Howard
Hughes
Medical Inst.
HHMI’s Janelia Farm Research campus has installed 56+ ports of 100GE on Brocade
MLXe-32 and MLXe-16 routers in one of the largest single-site deployments thus far.
HHMI is using 100GE to facilitate the delivery of 10GE directly to researchers’ systems.
Sep
2011
P&T
Luxembourg
P&T Luxembourg turned up a live 100GE circuit between Luxembourg and Frankfurt
using Alcatel-Lucent 7750s. The operator’s Teralink pan-European network connects 16
data centers in 6 countries and is now fully 100GE-ready.
2H11 U.S. Federal Government Brocade MLXe routers with 100GE are being deployed to support U.S. federal govern-ment applications.
Jul
2011
China
Telecom
Alcatel-Lucent announced that China Telecom intends to use 100GE interfaces on the
7750 following a successful live demonstration of the technology in September 2010.
Jun
2011
Amsterdam
Internet
Exchange
AMS-IX installed Brocade MLXe routers in 2010 and began upgrading them with 100GE
interface cards in 2H11 to support its exchange network that interconnects 9 data
centers in the Amsterdam metro area.
Nov
2010 Internet2
Internet2 announced that it is beginning deployment of a new nationwide 100GE
network that it expects to complete in 2013.
Nov
2010 Verizon
Verizon announced it was deploying the industry's first standards-based, multivendor
100GE link between Paris and Frankfurt.
Jun
2010 Verizon
Verizon publicly stated it plans to deploy 100GE on its metro Switched Ethernet Services
(SES) network, following a successful trial with Alcatel-Lucent 7450s.
Jun
2010 T-Systems
T-Systems and Alcatel-Lucent announced that the operator deployed the first
commercial 100GE link spanning both optical and router platforms. The 60 km link
connected data centers of two research universities in Germany using the Alcatel-
Lucent 7750 Service Routing and 1830 PSS DWDM optical equipment.
Source: Select operators and network equipment suppliers.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 15
Figure 10: Energy Science Network (ESnet) – Advanced Networking Initiative
Source: ESnet
Source: …
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 16
100GE Deployment Expectations
This section provides insight regarding (1) industry expectations about the timing of
100GE deployments, (2) the type of optical modules that network operators are
likely to use in initial deployments, and (3) service provider attitudes about what
needs to be overcome to encourage broad adoption of 100GE.
100GE Webinar Audience Feedback
During a December 2011 webinar on 100GE, Heavy Reading asked attendees
when they expect to see significant, strategic adoption of 100GE in multiple carrier
networks. We received 127 survey responses from a mix of service provider, vendor
and other professionals. It is clear that many had the view that we are in the very
early stages of the 100GE market. Only 8% of respondents in the December
webinar poll said that they expect to see significant, strategic adoption of 100GE
in multiple carrier networks by mid 2012, while close to one-third indicated they
expect strategic adoption by the end of 2012. The vast majority of respondents
(71%) stated that they believe the market will hit its stride before the end of 2013.
Figure 12: Network Operator Feedback – Which CFP Modules Are You Likely To Use On Your First 100GE
Links?
Source: Light Reading 100GE Webinar, December 2011, 79 respondents
Figure 11: Industry Feedback – When Are We Likely To See Significant, Strategic Adoption of 100GE?
Source: Light Reading 100GE Webinar, December 2011, 127 respondents
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 17
During the December webinar, Heavy Reading asked network operators what
type of CFP modules they expect to use on their first 100GE links. We received 79
responses that suggested there will be demand for a mix of modules in the near
term. Slightly more than one-fourth said that they were likely to deploy 10x10 2 km
modules, while 24% indicated they would likely go with 100GBASE-LR4 10 km
modules.
Overcoming The Cost Obstacle To Widespread 100GE Deployment
We continue to hear a consistent message from senior service provider experts
that there is very strong interest in 100GE technology, but the costs compared to
10GE need to come down further to unleash the market horses. Below are
examples of recent comments we have heard.
CTO, XO: ―I would vote for 100 Gigabit Ethernet [as the most significant
technology innovation in the carrier Ethernet market over the past 18 months].
That is an important forward-moving technology. It hasn’t seen widespread
deployment yet. I have no doubt that it will.‖
SVP, Transport & Infrastructure Services, Level 3. ―100 Gig is going to make a
big impact on our industry,‖ but ―from Level 3’s perspective it still has a long
way to go from a cost-competitiveness standpoint.‖
CTO, AMS-IX. ―I think we’ll see a big growth in deployments,‖ but ―cost is still a
major issue. It’s still not competitive vs. 10GE, although it’s changing rapidly.‖
Many industry players expect the availability of 2nd generation 100GE solutions to
have a notable impact on pricing, but numerous network operators clearly have
pressing needs that are compelling them to move ahead in the meantime with
early deployments using available 1st generation technology.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 18
Alcatel-Lucent Perspective
100G technology represents the next major inflection point in the scaling and
evolution of service provider infrastructure. Bit-streams are the oxygen of multi-
media communications, and handling them most efficiently defines the health of
a service provider’s network. The leap to 100G links in the infrastructure gives
operators a tenfold increase in capacity to deliver their services and drives
operational efficiency in their networks.
The move to 100G is about delivering speed without compromising services.
It is about making the transport network as efficient, future-proof, flexible and cost-
effective as possible, such that services can be delivered profitably and the
quality of experience across the range of services can be maintained. This applies
throughout the transport infrastructure, spanning both the IP and optical domains
within a network. 100G technology plays a key role in each, and Alcatel-Lucent
leads the way in both domains.
With experience gained through close partnership with over 25 service providers
who have deployed or field-trialed 100GE service routing on our 7750 SR and
7450 ESS platforms over the past 18 months, we believe more so than ever that
unleashing the full power of 100GE means being able to deliver speed without
compromising the broad range of services, in a way that makes it deployable
throughout the network wherever it is needed, and in a way that minimizes cost,
complexity and operational risk.
Breadth of 100GE Deployment Scenarios
Service providers’ expectations remain very consistent, but the range of early
applications for 100GE within their networks varies broadly depending on their
service mix, geographical footprint and operational realities.
Every service provider strives to make their networks as capable and efficient as
possible, to scale their networks cost-effectively, and to streamline their operations.
Without exception they push to get the most out of their infrastructure assets so
they can offer more to their customers, and do so profitably. The move to 100GE is
appealing against this backdrop, as it represents a 10x increase over the 10GE
links that are the de facto option on today’s IP networks. Managing 100GE links is
far more operationally efficient than bundles of aggregated nx10GE links, proving
cost effective and less prone to provisioning and configuration errors that can
negatively impact SLAs. This is particularly true if 100GE links can be enabled on
routers that are already operationalized and deployed in the network, minimizing
operational complexity, risk, and overall costs.
When it comes to avenues for deployment of 100GE, however, a whole spectrum
of possibilities unfold. They range from driving 100GE into the metro to increase
efficiency and capacity, to the service edge where the whole range of business,
residential and mobile services hit the network, to datacenters and Internet
peering points where more and more capacity is constantly needed. Ever since
being the first in the industry to deliver 100GE service routing modules in mid-2010,
we have worked closely with service providers as they test, trial and deploy 100GE
interfaces on their existing service routers within metro, service edge and core
networks. Innovative service providers continually push the limits, and we’ve
already seen 100GE LAG (with up to four links) running live in a 7750 SR network.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 19
In addition to service providers, application content providers and datacenter
operators, there has been significant activity within Research and Education
networks (RENs) over the past year. As a great example, the U.S. Department of
Energy’s (DOE) Energy Sciences Network (ESnet) is running 100GE live between
three leading supercomputing facilities today, as part of a nationwide upgrade. A
powerful demo by ESnet at the recent SC11 showed a visualization of the evolu-
tion of the universe using 100GE links, comparing it side-by-side to visualization that
is possible using 10 Gbit/s technology that has been the norm to date. The ability
to observe the richness of the data that was simply not practical or possible before
clearly demonstrates the value of 100GE technology in unleashing collaboration in
groundbreaking research.
Please refer to the links below or alcatel-lucent.com/100ge for a selection of
public announcements and activities:
ESnet and LGS Innovations Deploying 100G IP Network Capabilities Nation-
wide
Alcatel-Lucent and P&T Luxembourg launch one of Europe’s fastest data
networks
China Telecom to address rising broadband demand in China by selecting
Alcatel-Lucent to upgrade network performance
Verizon completes industry-leading 100G Ethernet field trial
T-Systems and Alcatel-Lucent deploy industry-first commercial 100 Gbit/s link
spanning optical and IP routing technologies
The Alcatel-Lucent 100GE Service Routing Solution
The Alcatel-Lucent Service Router portfolio leverages three successive generations
of groundbreaking network processor (NPU) silicon innovation to deliver 100GE
service routing that is deployable throughout the service providers’ network on
existing systems they have already operationalized. The stage for 100GE service
routing was set way back in 2008 when we delivered FP2, the industry’s first 100
Gbps NPU chipset. In 2011, we raised the bar yet again by introducing FP3, the
industry’s first 400 Gbps NPU chipset. As 100GE becomes the new currency in the
network infrastructure and 100GE proliferates throughout service provider net-
works, FP3-based service routing modules will pave the way. Investment in home-
grown silicon innovation that drives the highest level of integration enables
Alcatel-Lucent to control our own destiny in meeting service providers’ highest
expectations for density, services capability and power efficiency.
Alcatel-Lucent continues to innovate and push the limits of technology: to ensure
that our systems, IP routers and DWDM optical transport equipment alike, are at
the leading edge of what is commercially possible. We do so with the belief that
innovation and attention to operational deployability drive superior economics for
network scaling at speeds of 100G and beyond. In doing so, we work closely with
service providers as well as a broad community of component suppliers, technol-
ogy and testing partners.
Our goal is to accelerate 100GE adoption, allowing service providers to get the
most from their infrastructure investment: to profitably deliver more services more
efficiently, in a smaller footprint, using less power, and with superior operational
economics.
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 20
Brocade Perspective
Networks today are at the forefront of a deluge of traffic straining their existing
infrastructure – with demands for ubiquitous high bandwidth delivering on-
demand personalized content at anytime and anywhere. Leading-edge services
such as high-definition video streaming, mobile broadband, and cloud services
have significantly altered network traffic behavior. Instead of localized flows with
occasional bursts, traffic flows are more collaborative over geographical distanc-
es and last longer. These new traffic patterns not only consume enormous
amounts of network capacity, but also add a greater degree of complexity to
network operations. As a result, today’s network planners are seeking solutions
that provide the right mix of scalability, performance, and operational simplicity.
High-density 100 gigabit Ethernet (GbE) is the solution to meet this rising commodi-
ty traffic demands while maximizing the revenue-cost gap.
Since the 40 GbE and 100 GbE standards were adopted by the Institute of
Electrical and Electronics Engineers (IEEE) 802.3 Working Group in June 2010,
shipments of first generation media modules, test equipment, router interfaces and
optical transport solutions have begun and deployments – both active and in trials
– have picked up rapidly. First generation 100 GbE is now a mature technology
with broad vendor support, several successful interoperability tests, and variety of
deployments in production environments today.
100 GbE Applications
100 Gigabit Ethernet applications are targeted for service provider core and
aggregation networks, metro core, and large campus core networks. Driven by
Figure 13: 100 GbE Applications
Source: Brocade
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 21
wired and mobile needs for business and consumer access to high-bandwidth
personalized content, in particular video, these applications have the highest
bandwidth and redundancy requirements for ultra-high-capacity Ethernet
networks, and support distances up to 40 km over SMF. For data center core and
aggregation applications, short-reach 100 Gigabit Ethernet technologies over
copper cable and up to 150 m MMF are defined.
100 GbE In The Service Provider Core
Service provider core networks today are seeing traffic increases at the rate of
40%-60% on an annualized basis, and this is where the demand for 100 GbE links is
the greatest. Currently, most service providers use 10 GbE link aggregation groups
(LAGs) to combat the heavy load of traffic, but there are severe limitations to this
method. In the service provider network core, traffic flows are very heavy, and 10
GbE LAGs have scaling challenges when the traffic flow is large relative to
individual constituent links. In this case, 100 GbE becomes more efficient by
delivering terabit capacity. Not only is 100 GbE a better performance option in
the core, but it also offers considerable economic benefits compared to 10 GbE
LAG solutions. Through simplified management of links, installation costs and
operational expenses can be reduced, while preparing for anticipated band-
width growth. Networks that want to scale beyond 100 GbE can utilize the
industry’s only multi-terabit trunks—a single logical connection formed by aggre-
gating multiple 100 GbE ports—to achieve superior scalability and performance.
The Brocade 100 GbE Solution
The Brocade MLX Series of advanced routers delivers unprecedented scale and
performance, high reliability, and cost-saving operational efficiency for the world’s
most demanding service provider and enterprise networks. The industry’s first 2-
port 100 GbE module provides unmatched performance and scalability of up to
32 wire-speed 100 GbE ports in a single Brocade MLXe-32 router. This full-slot
module complies with the IEEE 802.3ba and 10x10 MSA standards and supports
Figure 14: Terabit Networking
Source: Brocade
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 22
CFP-based short-reach and long-reach optics. Each two port 100 GbE module
delivers 400 Gbps of throughput per module without compromising the perfor-
mance of features such as IPv4, IPv6, and MPLS. The ports-on-demand capability
is designed for the business challenges of today and tomorrow, providing flexibility
that protects prior investments, and enables the ability to migrate and evolve the
network as demand for bandwidth grows. For more information on Brocade’s high
density 100 GbE solutions, please visit the Brocade MLX Series product page.
The Brocade MLXe 100 gigabit Ethernet (GbE) solution has participated in many
exciting events and garnered impressive accolades in 2011. See below for a list of
some of those activities, interoperability tests, and deployments with links to the
press releases.
Interop Tokyo 2011 Best of Show Awards
AMS-IX and Limelight Networks Joint Field-Test Brocade 100 Gigabit Ethernet
Technology
Success of Industry's First High Speed 100 Gigabit Ethernet Joint Interoperability
Test at Internet Exchange Point
Live Public Demonstration of 100 Gigabit Ethernet at Interop Las Vegas 2011
Ethernet Alliance Completes Interoperability Event for 40 and 100 Gigabit
Ethernet
Isocore Completes Leading-Edge Code Testing of Key Internet Technologies
Brocade Selected as the Core for World's Largest Single Site Deployment of
100 GbE in a Research Institute
Supercomputer network blasts torrent of data
IU Team Demonstrated 100GbE Network Technology at SC11
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 23
Spirent Perspective: Testing 100GE
After the ratification of the IEEE 100GE standard in mid 2010, the release and
deployment of 100GE solutions took on significant momentum in 2011. From
interoperability tests among leading Japanese ISPs in June and demonstrations at
the Amsterdam Carrier Ethernet World Congress in October, to validating ALU’s
400G silicon, the development of 100GE implementations is accelerating.
This acceleration of the technology life cycle is a trend in the history of Ethernet.
For each successive generation, the curve of innovation to maturity over time gets
steeper as demand increases and schedule expectations are collapsed.
For 10GE, the evolution from achieving 10GE transport to 10GE multiservice routing
took seven years. For 100GE, the expectation is 100GE multiservice routing in two
years.
This increasing momentum is driven by another cycle, the self-perpetuating cycle
whereby demand for content drives the evolution of technology, which supports
new services that, once delivered, spur more demand for content.
The latest incarnation is the convergence of business VPN, fixed broadband and
the mobile internet onto a single network, which presents a new challenge for
delivering high-speed Ethernet – the evolution of metrics for evaluating, delivering,
and assuring its ability to support these new services.
Figure 15: Understanding the Ethernet Test Life-Cycle
Source: Spirent
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 24
Quality of service (QoS) has been supplanted by quality of experience (QoE) as
the measure that best represents how a system will behave once deployed. This,
again, is due to the expectations of the subscribers, as can be seen in the evolu-
tion of how carriers advertise their offerings. The ―can you hear me now‖ guy has
given way to spokespersons making data throughput claims and demonstrating
video quality.
The QoE challenge has implications for the world of test and measurement at any
speed, but the implications are intensified at 100G. Deterministic legacy test
methodologies deliver valuable information about individual components or
functions, but good QoS numbers alone are no guarantee of QoE. To evaluate
QoE, 100GE test solution providers have gone beyond validation of discrete
components and functions to create test methodologies that recreate the
complete environment in which new services must perform, a concept known as
test realism.
Realistic testing re-creates the environment that the application lives in, from the
provider to the subscriber, in all its dynamic and daunting complexity. Test realism
is achieved by moving beyond simulation to emulation. Simulation attempts to
predict the behavior of a system by creating an approximate mathematical
model based on a set of assumptions. Emulation replaces a part of the system and
performs in exactly the same way as the element it replaces. It adjusts dynamically
to a changing environment and responds to actual stimuli from the system it
interacts with.
There are three essential elements of test realism.
Real user behavior: the flexibility and sophistication to emulate a wide range
of user behavior, both benign and malicious.
Figure 16: 100G Ethernet Means Accelerated Testing
Source: Spirent
HEAVY READING | DECEMBER 2011 | WHITE PAPER | THE EMERGING 100GE NETWORKING ERA 25
Real converged traffic: the power to create line-rate, fully-emulated multi-
layered network topologies.
Real network conditions: the power and complexity to create the dynamic,
time-varying conditions found on deployed, production networks like users
and traffic burstiness, routing updates and failover of tunnels and physical
links.
Test and measurement vendors facilitate network equipment manufacturers and
service providers as they adapt to the accelerated Ethernet life cycle by providing
the ability to perform more powerful testing in less time through test realism. In
practical terms, this involves test modules with multicore processors, which provide
the flexibility required to support advanced network topology emulation, and with
high port density to meet the massive traffic requirements of 100GE.
