400 Gigabit Ethernet for the ages Jul

400 Gigabit Ethernet for the ages


July 10, 2014

By JOHN D'AMBROSIA

The process to create the specifications for 400 Gigabit Ethernet has begun. But there are many factors, technical and otherwise, that will come into play.

The first meeting of the IEEE P802.3bs 400 Gigabit Ethernet (400GbE) Task Force took place in May. This group is chartered with developing the next rate of Ethernet. Such endeavors are unlike other Ethernet projects, since the entire "personality" and characteristics of the new rate of Ethernet are up for consideration. Furthermore, industry pressures driving the development of the next rate of Ethernet must be balanced with Ethernet's past and future needs and requirements.

The development of 400GbE illustrates the various pressures that the development of a new rate of Ethernet faces from a multitude of perspectives. Like previous projects targeting a rate jump, the development of an underlying architecture capable of supporting follow-on projects for that new rate has been a key priority. Similarly, at this point in the project, various technologies, both existing and emerging, are being introduced to the task force for consideration. Essentially, the book is open, and it's up to the task force to write "the story" that will become 400GbE. However, the definition of 400GbE is facing other pressures that make this speed jump feel different from prior jumps.

The simultaneous development of 40 Gigabit Ethernet (40GbE) and 100 Gigabit Ethernet (100GbE) was driven by the need to address the different bandwidth growth rates of servers and networking. The subsequent deployment by the market is somewhat different than envisioned back in 2007. The mass market is still making the transition to 10 Gigabit Ethernet (10GbE)-based servers. Data centers are deploying 40GbE in their cores, while service-provider networks are seeing growth in 100GbE connections. But as forecasted by the IEEE 802.3 Ethernet Bandwidth ad hoc, bandwidth is expected on average to experience a 58% compounded annual growth rate (CAGR), with the actual range for studied applications growing from 32% to 95% CAGR.

It's important to note that while bandwidth demand is growing exponentially, the actual growth rate is highly application-dependent. Cloud-scale data centers, for example, are looking to move from 10GbE/40GbE architectures to 100GbE architectures and beyond. 100GbE is still undergoing a lot of development effort in the IEEE 802.3 Working Group as well as the industry. Thus, some perceive that the development of 100GbE and 400GbE are ultimately intertwined in moving the industry to higher speeds.

Let's take a look at the project - and understand this interesting dichotomy for the industry, which must balance meeting nearer term bandwidth requirements with solutions that will meet industry ROI requirements - to develop a long term scalable 400GbE family.

Project objectives

The stated objectives for the 400GbE project are the following:

• Support a MAC data rate of 400 Gbps.
• Support a BER of better than or equal to 10-13 at the MAC/PLS service interface (or the frame loss ratio equivalent).
• Support full-duplex operation only.
• Preserve the Ethernet frame format using the Ethernet MAC.
• Preserve minimum and maximum frame size of the current Ethernet standard.
• Provide appropriate support for Optical Transport Network (OTN).
• Specify optional Energy Efficient Ethernet (EEE) capability for 400-Gbps physical layer entities (PHYs).
• Support optional 400-Gbps attachment unit interfaces for chip-to-chip and chip-to-module applications.
• Provide PHY specifications which support link distances of at least 100 m over multimode fiber (MMF), at least 500 m over singlemode fiber (SMF), at least 2 km over SMF, and at least 10 km over SMF.

Given that the mass market for servers is currently migrating to 10GbE and the existence of 40GbE and 100GbE, it's probably no surprise that all noted objectives for PHY specifications target networking requirements. Additionally, electrical interfaces will be developed for chip-to-chip and chip-to module applications. Other important items to note include the support for OTN (transport of Ethernet across optical transport networks), optional support for EEE, and the improvement in the target BER of better than or equal to 10-13 or frame loss ratio equivalent, which addresses the potential use of forward error correction (FEC).


Typical Ethernet architecture.
Typical Ethernet architecture.

Architecture

The Figure shows the typical basic architecture that has been the basis for previous generations of Ethernet. There will be no changes to the 802.3 MAC other than increasing the data rate; the frame format and minimum and maximum frame size will be preserved, and only full-duplex operation will be supported. The Reconciliation Sublayer (RS) maps the serial bitstream of the MAC into the Media Independent Interface (MII). The 400G MII (CDGMII) provides an interconnection between the MAC and the 400GbE PHY, which comprises the Physical Coding Sublayer (PCS), Physical Medium Attachment (PMA) sublayer, and Physical Medium Dependent (PMD) sublayer. The PCS encodes the data from the MII, which is then serialized by the PMA sublayer. The PMD sublayer is responsible for the signaling through the Medium Dependent Interface (MDI) and across the medium that the PMD is intended to support.

A key point to note is that there's an expectation that some sort of FEC will be necessary for 400GbE. It's not being shown as a sublayer at this time, as is typically done, because the FEC may actually get embedded in the PCS.

PHY specifications

As mentioned, there are four PHY objectives, one addressing 100 m over MMF and three addressing different reaches (500 m, 2 km, and 10 km) over SMF. From a high-level perspective, a single 400-Gbps optical stream on a single wavelength is unlikely. Therefore, it's highly probable that the ultimate solution addressing any of these objectives will be based on some form of parallelization that may be achieved either by use of multiple optical lambdas or through multiple fibers. Again, it's also likely that FEC will be employed.

It's anticipated the MMF approach will employ a parallel fiber design that might require up to 16 fibers in each direction. The 2-km and 10-km singlemode objectives likely will employ a multi-lambda full-duplex fiber approach. The two approaches for these objectives garnering the most interest are 4×100G and 8×50G. During straw polls at the IEEE 802.3 May interim meeting, the IEEE P802.3bs Task Force participants expressed the most interest in the 4×100G effective optical lane rate approach for the 2-km and 10-km objectives.

On the other hand, the 500-m specification may actually employ parallel SMFs with multiple lambdas per fiber. This approach may be a nearer term, lower cost solution for this reach and may enable a breakout approach to multiple ports of a lower speed such as 100GbE.

In addition to some form of parallelization, increasing the effective lane rate through some form of higher order modulation such as discrete multitone (DMT) or pulse amplitude modulation (PAM) is also under consideration. This topic was explored by the Ethernet Alliance and OIDA in a workshop in June that focused on single lambda for 100GbE.

Electrical interfaces

While it's true that all of the PHY specifications for this project are optical, efforts furthering the development of copper also exist in this project. As noted, electrical interfaces targeting chip-to-chip and chip-to-module will be defined. Developing a 16×25-Gbps interface may be desirable given the current investment in 25-Gbps signaling technology for 100GbE. It may be necessary to support a 16-fiber MMF approach.

There's also concern, however, that a 16-lane-wide electrical interface will be short-lived given the associated cost and power, as well as the port-density limitations it causes. Some favor an 8×50-Gbps approach. But as the speed of the interface increases, the effective reach may be lowered. Another option might employ some sort of FEC.

And beyond 50 Gbps? The development of a 4×100-Gbps electrical interface might sound appealing to many. However, the power limitations, technology cost, or use of FEC must be considered before deciding the final solution's appeal. These concerns are totally reasonable, but the industry should consider how many times it has written off copper, only to see a copper-based solution emerge. Copper could be considered the true phoenix of the industry.

Project challenges

The IEEE P802.3bs 400 Gigabit Ethernet project clearly has a number of challenges, both technical and nontechnical. Developing a forward-looking architecture that will last for the ages needs to be a priority. Part of the challenge here, however, is not knowing what will be required of the architecture in the future. Therefore, it's important that the task force do its homework and consider as many possible implementations as it can imagine.

Further, moving to higher signaling rates that will challenge the laws of physics should not be underestimated. The frequent mention of FEC also creates the potential for solutions that might not meet industry latency needs or, worse yet, creates an atmosphere where interoperability is not optimized. That's clearly not an acceptable outcome for an Ethernet project, and many are already diligently considering multigeneration interoperability among implementations based on different technology points.

The largest challenge, however, is not the technical aspect. As always, it's consensus building. While many point to the IEEE for either the success or failure of a project, the industry participants who attend the IEEE 802 meetings make the technical decisions. IEEE 802 merely provides a venue for the meetings and a set of rules for making such decisions.

Focus on what's next

Focusing on the next rate of Ethernet by the industry will help provide the consensus that'll make 400GbE a reality, as well as have potential implications for the future of 100GbE. The project timeline, adopted by the IEEE P802.3bs Task Force, ultimately calls for new proposals to be submitted this year for task force consideration. With this November being the last opportunity for proposals, the industry will be quite busy.

Per this timeline, it's anticipated that the ultimate ratification of the standard will be in the first quarter of 2017. Thus, the path forward for 400 Gigabit Ethernet will be quite exciting.


JOHN D'AMBROSIA is chairman of the Ethernet Alliance, chief Ethernet evangelist at Dell, and chair of the IEEE P802.3bs 400 Gigabit Ethernet Task Force.



http://www.lightwaveonline.com/articles/print...-ages.html