Spectrum Ethernet Storage Switches Unleash Revolutionary Storage Performance


Panama: The canal that unlocked the world a century ago is doing it again. After nine years of toil, and a whopping $5.8B in investments, the wider canal opened in early 2017. Before 2017, large ships carrying natural gas from U.S. East coast had to travel all the way around the tip of South America to reach China, Korea and Japan. Expanding the canal created more efficient routes to East Asia and Australia from U.S. East coast as well as Europe. With more efficient routes to their destinations, many cargo businesses in the U.S. are expecting to grow at double the usual rate. Clearly, a better interconnected planet promotes efficiency and adds business value.

Just like a wider canal was needed to enable efficient cargo transportation and unlock latent business potential, a high bandwidth and fast Ethernet Storage Fabric is needed to unleash revolutionary storage performance. In the rest of this blog, we will explore Spectrum-based Ethernet Storage Switches that improve overall system performance, and do so much better than any typical deep buffer switch.

We are witnessing the widespread adoption of high performance, direct-attached storage devices; namely solid state drives (SSDs). These storage devices have tremendous performance but are physically stranded within individual servers. The solution often uses a high performance Ethernet Storage Fabric to share these high performance resources across multiple servers. The performance of an Ethernet Storage Fabric is heavily influenced by the switches used in the fabric. When you dig into the details of the silicon, there are two types of switches in the market that target storage applications:

  1. Switches with on-chip buffers
  2. Switches with off-chip buffers (aka ultra-deep buffered switches)

To summarize, a recent blog covering the topic of on-chip buffers, Spectrum has better burst absorption capability due to its on-chip monolithic packet buffer architecture. Legacy switches have oversubscribed and fragmented on-chip buffers that lead to sub-optimal burst absorption capability. Spectrum offers line rate non-blocking performance with zero packet loss as its buffers support line rate bandwidth.

In the second part of that blog series, we discussed why switches with off-chip buffers are not suitable for high performance applications, as they are implemented with slow but huge DRAM modules. Since they are much slower than on-chip buffers, multiple DRAM instances have to be used in each switch to provide a reasonable packet buffer bandwidth. This results in switches with gigantic but oversubscribed/blocking buffers—also known as a deep buffer switch. Because these ultra-deep buffered switches are blocking in nature, they introduce jitter and thus are not a good fit for high performance storage interconnect applications in the datacenter.

Now, we will cover a few example storage scenarios to contrast Spectrum attributes and its performance vis-à-vis legacy solutions.

  1. Why should you care about oversubscription and packet loss?

Storage applications demand high bandwidth. For example, Intel’s Ruler SSDs can pack a 1U server platform with one petabyte of storage. In the case of a distributed storage, this petabyte of data should be replicated at least two additional times across the network to provide redundancy. A non-oversubscribed, line rate, zero loss and high bandwidth network is therefore needed to move petabytes of data around. Mellanox Spectrum provides just that (See results here).

Packet loss and unpredictable interconnect performance complicate distributed storage systems. For example, if an initiator does not get a response back for a write request, it does not know whether the write request itself was dropped, the response from the target was dropped on the way back (See Figure 1), or the network is just being slow. More handshakes and communication would need to happen in order to ensure that the write request is committed into the storage target.

Figure 1: Mellanox Spectrum Zero packet loss, high performance storage fabric is perfect for an Ethernet storage switch.


  1. Why should you care about fairness?

Storage backup, data replication and shading of related traffic are typically bursty in nature and run in the background. These heavy duty flows, which run in the background, should not block other interactive sessions that run over the same fabric. It is important that the storage fabric functions fairly, without much interference between unrelated ports. Mellanox Spectrum supports non-blocking, line-rate packet switching, which helps ensure that each port gets the same line-rate performance and all connections are treated fairly. Common other switches in the market today including the,“ultra-deep” buffered switches are oversubscribed and blocking, so they cannot maintain line rate on all ports. As a result, some workloads get better performance than others, depending which ports they are using.

  1. Why should you care about jitter?

Mellanox Spectrum provides consistent low cut-through latency across all packet sizes. There are switches in the market today with 4GB packet buffer and a 4GB buffer can introduce a whopping 3.2 second delay on a congested 10GbE port. Multiple hops through deep buffered switches can introduce significant delay and variation in latency. (Variation in latency is known as jitter and is very undesirable.) This unpredictable performance with ultra-deep buffers will upset storage performance. Figure 2 shows gives an example on how the storage performance can be impacted by a delayed response packet.

Figure 2: Mellanox Spectrum provides consistent low latency across all packet sizes.



Faster storage needs faster networks. Mellanox Spectrum, with its high bandwidth, zero packet loss and consistent low latency is the ideal switch that can be used for high performance applications such as storage inside the datacenter. Spectrum includes a monolithic, high-performance on-chip buffer to ensure non-blocking performance, good burst absorption, low jitter, and fairness. Legacy switches with on-chip buffers do not have enough burst absorption capability to support Ethernet Storage Fabrics. Legacy switches with off-chip (deep) buffers are blocking in nature, introduce jitter and are not suitable for Ethernet Storage Fabrics either. In other words, a deep buffer switch is a poor choice for an Ethernet storage switch. Given the clear advantages of Spectrum’s design and performance it’s no wonder HPE Storage recently picked Mellanox Spectrum technology to power their StoreFabric M-series switches.

Supporting Resources:

About Karthik Mandakolathur

Karthik is a Senior Director of Product Marketing at Mellanox. Karthik has been in the networking industry for over 15 years. Before joining Mellanox, he held product management and engineering positions at Cisco, Broadcom and Brocade. He holds multiple U.S. patents in the area of high performance switching architectures. He earned an MBA from The Wharton School, MSEE from Stanford and BSEE from Indian Institute of Technology.

Comments are closed.