InfiniBand Performance

The need for complex simulations and research in high performance computing and the virtualized enterprise data center mandates the usage of a high throughput and low latency I/O solution.  InfiniBand is the only industry-standard interconnect that provides the required bandwidth, latency, low power consumption and CPU utilization characteristics for maximizing compute systems productivity, efficiency and scalability.

Mellanox delivers industry-leading, cost-effective, high-performance interconnect solutions for high performance computing, mainstream grids, and enterprise data centers. Mellanox InfiniBand 20Gb/s adapters and switch silicon provides the required I/O capabilities for current and future multi-core compute systems, and the ability to scale to any required cluster size.

Maximizing Productivity
Mellanox InfiniBand technology reduces the high CPU processing overhead dictated by other InfiniBand and high-performance interconnect solutions. Mellanox technology allows performance-hungry applications and virtualized environments to fully utilize CPU resources and overlaps I/O communication with CPU computation cycles. Moreover, it increases overall compute cluster efficiency, and delivers the same world-class performance regardless of the amount of the server CPU cores.

Basic Numbers
The preferred method for selecting the best interconnect is testing the target applications, and for that, Mellanox provide a cluster environment for application testing and benchmarking (click for more details).

The most common performance parameters used to compare between interconnects are latency and bandwidth. Mellanox HCAs are design to provide the best performance and price/performance connectivity solution for clusters and storage. 

	InfiniBand		Proprietary		GigE	10GigE
	Mellanox ConnectX  IB 20Gb/s PCIe x8	QLogic InfiniPath IB 20Gb/s PCIe x8	Myrinet   10G PCIe x8	Quadrics QSNetII		Chelsio T210-CX PCIe x8
Application Latency (µs)	1	1.3	2.2	1.5	30-100	8.9
Peak Unidirectional Bandwidth (MB/s) for PCIe Gen1	1500	1400	1200	910	125	860
Peak Unidirectional Bandwidth (MB/s) for PCIe Gen2	1900	N/A	N/A	N/A	N/A	N/A

*Mellanox Technology testing; Ohio State University; QLogic, Myricom, Quadrics, and Chelsio websites

For IP-based application (TCP or UDP) over InfiniBand one can utilized the upper layer protocol called IP over IB (IPoIB). IPoIB running over high bandwidth InfiniBand adapters provides an instant performance boost to any IP-based applications.

	Mellanox ConnectX InfiniBand
	IB 10Gb/s PCIe Gen1	IB 20Gb/s PCIe Gen1	IB 20Gb/s PCIe Gen2
IPoIB Bandwidth	939MB/s	1340MB/s	1860MB/s

 

Multi-core latency
Multi-core CPUs have become the basic building block in every compute system. In such environments, the interconnect must be able to handle multiple IO streams, while maintaining high efficiency. Typically the interconnect latency is being measured between a single core on one server and a single core on a second system, while the rest of the CPU cores are idle. For multi-core environments, it is more important to benchmark real multi-core latency, where all CPUs core generate traffic between the server nodes. While other interconnect solutions demonstrate increase in latency when multi cores execute IO traffic simultaneously , Mellanox ConnectX keeps the same low latency regardless of the number of cores and guaranties linear scalability for the applications.

Mellanox's InfiniBand technology provides high performance on any platform and the performance is not tied to the server architecture or to the amount of CPU cores. Moreover, it liberates the CPU from network related tasks and increases the compute power dedicated for the applications. (View Case Studies)