The data center’s biggest industry conference centered around high-speed interconnects, the Optical Fiber Conference, was in San Diego, CA March 13 to 15 with attendance of about 15,000 people.
In the Mellanox booth, we showed live demonstrations of our 200G and 400G Direct Attached Copper (DAC) interconnect product line.
In this section of the booth we showed our 200G/400G DAC product line which is based on 50G PAM4 signaling and consists of:
- 200G QSFP56 -to- QSFP56 in 4x50G PAM4
- 200G QSFP56 -to- Dual 100G QSFP56 in 4x50G PAM4 and 2x50G PAM4
- 400G QSFP-DD -to- QSFP-DD in 8x50G PAM4
- 400G QSFP-DD -to- Dual 200G QSFP56 in 8x50G PAM4 and 4x50G PAM4
The LinkX DAC portfolio offers just about every combination of 100G, 200G and 400G copper DACs using the new modulation scheme of 50G PAM4 and new form-factors of QSFP56 and QSFP-DD. 200G is likely to be priced between 100G and 400G and provide a more gradual step to faster networking better matching customer needs as the jump from 100G to 400G is pretty big and entails a lot of signaling and system changes.
The 100G rate is likely to be around for a long time starting out a 4x25G NRZ, transitioning to 2x50G PAM4 and eventually ending up at 100G PAM4 per lane and 4x100G (400G) and 8x100G (800G) PAM4-based links. The QSFP56 100G ends will enable 100G to the server in a traditional 4-channel QSFP form-factor which is slightly shorter than the QSFP-DD and considerably smaller than the OSFP form-factors. Additionally, QSFP56 switch and network adapter ports are be backwards compatible with QSFP28s.
200G QSFP56 & 400G QSFP-DD Straight and Y-Splitter Static DAC Display
200G/400G DAC Live Demo Using Ixia System
Dual 26 AWG
The 400G DACs uses dual 26 AWG copper wires to enable easy bending of the cable. This removed all doubts about 3 meters and 26AWG cables can be used in a QSFP-DD form-factor which several opponents said could not be done and would require shorter reach and thinner copper cables or a new form-factor such as the OSFP. QSFP-DD ports are backwards compatible with QSFP28 which is very important to provide a smooth upgrade path and links with older systems. Also, the dual cable makes it very easy to split 200G into dual 100G and 400G into dual 200G cables.
Dual 26AWG Cables Each with 4-Channels Enables Easy Bending
Power, Latency and Bit Errors
DAC cables form a directly attached electrical connection between switches and network adapters. Since there is no optics or electronics in the data path, there is no power consumption or latency delay in converting electrical signals to optics and back and astoundingly low bit error ratios can be achieved since there is no optics or electronics to add bit errors.
Near Zero Power Consumption
Since there is no optics or electronics in the data path, there is no power consumption. Jumping to 400G AOCs or transceivers will require 7-15 Watts of power consumption per port! People forget that data center equipment operates 24×7 for 3-5 years, so even a few Watts of power consumption really adds up – for every cable and thousands of cables may be deployed.
“DAC cables consume zero power.
Upgrade to optics at 7-15 Watts per port times 32 ports and boom,
you just turned equivalent of a hair dryer on low!”
DAC cables also offer near zero latency delays which is important to enable the data to get through the link as fast as possible. Low-latency is especially important in server-switch-memory links as this not only the highest traffic area but also the highest value traffic as it is where all the computation occurs. Mellanox is the leader in InfiniBand networking where the ultimate in low-latency links is demanded. We have transferred this design and manufacturing technology knowledge to the Ethernet space to build near zero latency DAC cables.
At 400G, Latency is Getting Much Worse
At 100G (4x25G NRZ) latency delay from the host RS-FEC is about 120ns to compute and correct bit errors. Ethernet-based, 400G systems use the KP4 FEC standard which can induce up to 250ns of latency delay to compute and correct errors. This occurs in each direction. So, a single round trip has built in 500ns of latency delay. By using very high-quality 100G DAC cables from Mellanox, for <3-meter lengths, FEC can be turned off in the host saving considerable power and latency in each port.
“Half a microsecond savings on a
zillion server-to-memory I/O transactions in
thousands of DAC cables can really add up
and zero power consumption is very hard to beat!”
Very Low Bit Error Ratio (BER) = High Quality
In our OFC 2018 booth, Mellanox demonstrated 400G dual 26 AWG DAC cables operating at a BER of 1E-10 continuously for four days! This is an incredibly low BER for a 3-meter cable operating with 8-channels of 50G PAM4. With the host FEC turned off, will save considerable power and latency from the FEC circuitry. With FEC turned on, bit errors might be seen measured in days instead of seconds as with competitive offerings. The IEEE now requires FEC on for 400G links but may relax the standard in future years as they did recently with 25/100G very short lengths.
BER is the measure of the number of bit errors compared to the number of bits transferred. It is also a measure of the quality of the cable’s design and construction.
At 4x25G NRZ (100G), competitive DAC cables BER typically starts out around 5E-5 and relies on the host FEC to clean up the cable errors to meet the IEEE minimum standard of 1E-12. When the number of errors exceeds the ability of the FEC to correct, the data has to be retransmitted. This is when cheap cables become very expensive.
Mellanox’s low BER is achieved by using high-speed design techniques, special materials and robotic assembly. Mellanox is the leader in low-latency InfiniBand switching systems. These systems require FEC to be off as it induces too much latency, so DACs have to be tested to BER rating of 1E-15 assuming FEC off vs IEEE Ethernet standard of 1E-12 with FEC on. Mellanox applies these design and manufacturing techniques to address minimizing BER at 50G PAM4 for both 200G and 400G.
The combination of very low BER with no FEC enabled means data center operators can run the server-switch-memory links at 3 meters or more without the use of FEC. This saves considerable power in the switches and network adapters in computing the FEC and 250ns of latency delay in each direction. While this approach now works with 25G/100G signaling as an industry standard, it may be a while before the IEEE standardizes it for 400G.
There is a considerable difference in DAC cables that all claim meeting low BER ratings. Some “barely” make the IEEE spec requirement and may fail under stresses or even normal cable bending. High bit errors mean forced retransmits losing considerable time. Mellanox’s cables are designed to operate well within required parameters offering considerable signal integrity margins and protecting high-value-added data.
The Bottom Line
DAC cable have the simplest design, fewest components to fail (high-reliability), the lowest acquisition cost (Capex) coupled with zero power consumption operating at 24×7 for 3-5 years (Opex) with a side of zero latency thrown in. After these three, the important features are seeking the lowest bit error ratio (BER) and possibly deploying DAC without the use of FEC to save more power and latency delays. At enormous data rates of 100, 200 and 400G, low quality DAC cables can become extremely expensive when they fail or worse – fail intermittently!
25/50/100G clearly has hit mainstream and Mellanox offers a full line of DAC, AOC and transceiver products for any data center application. 200G is powering up for 2018 and 400G beyond. The new PAM4 modulation scheme, 8-channel architectures and new form-factors will bring a lot of new capabilities and changes to the systems infrastructure. Stay tuned for more LinkX cables and transceivers product developments. 2018 promises to show several new product announcements and capabilities.
- Learn more: LinkX Cables and Transceivers
- Sign up for free ebook download here: LinkX Cables and Transceivers ebook
- Learn more about: Mellanox’s complete 100GbE Switches and Adapter solutions
- Follow Mellanox on: Twitter, Facebook, Google+, LinkedIn, and YouTube