Category Archives: InfiniBand

Missed Mellanox at Interop?

Just in case you missed us at Interop 2009, below are just a few of the presentations that took place in our booth.

Mellanox 10 Gigabit Ethernet and 40Gb/s InfiniBand adapters, switches and gateways are key to making your data center F.U.E.L. Efficient

 

Mellanox Product Manager, Satish Kikkeri, provides additional details on Low-Latency Ethernet

 

Mellanox Product Manager, TA Ramanujam, provides insight on how data centers can achieve true unified I/O today

 

Fusion-io’s CTO, David Flynn, presents “Moving Storage to Microsecond Time-Scales”

 

We look forward to seeing you at our next event or tradeshow.

Brian Sparks
brian@mellanox.com

Mellanox Key to Fusion-io’s Demo at Interop

I’m still pondering my take on Interop this year. It’s been a while since I’ve seen so many abandoned spaces on the show floor. Mind you most were 10×10 or 10×20 spots, but you could tell there were others who really went light on their presence. I saw one booth which had a 40×40 booth and just filled it with banner stands. Yikes! So nothing was really grabbing at me until I went to Fusion-io’s booth and saw the wall of monitors with a 1,000 videos playing on it at once. Fusion-io Booth

FINALLY SOMETHING IMPRESSIVE!

Even more amazing, the videos were all being driven by a single PCIe card which had 1.2TB of SSD RAM on it. This one “ioSAN” card from Fusion-io completely saturated 16 servers (126 cpu cores)…and they were able to achieve this through the bandwidth performance and ultra low-latency of 20Gb/s InfiniBand via Mellanox’s ConnectX adapters. In fact, they told me the 20Gb/s InfiniBand connection would allow them to saturate even more servers, yet they only brought 16.

iodrive_duo_flat-24

The video below, featuring Fusion-io’s CTO David Flynn, tells the complete story:

The ioSAN can be used as networked, server-attached storage or integrated into networked storage infrastructure, making fundamental changes to the enterprise storage area. This is a great example of how Mellanox InfiniBand is the enabling technology for next generation storage.

Talk with you again soon,

Brian Sparks
brian@mellanox.com

The Automotive Makers Require Better Compute Simulations Capabilities

This week I presented in the LS-DYNA user conference. LS-DYNA is one of the most used applications for automotive related computer simulations – simulations that are being used throughout the vehicle design process and decreases the need to build expensive physical prototypes. Computer simulation usage has decreased the vehicle design cycle from years to month, and is responsible for cost reduction throughout the process. Almost every part in the vehicle is designed with computer aided simulations. From crash/safety simulation to engine and gasoline flow, from air condition to water pumps, almost every part of the vehicle is simulated.

Today challenges in vehicle simulations are around the motivation to build more economical and ecological designs, how to do design lighter vehicles (less material to be used) while meeting the increased safety regulation demands. For example, national and international standardizations have been put in place, which provide structural crashworthiness requirements for railway vehicle bodies.

In order to be able to meet all of those requirements and demands, higher compute simulation capability is required. It is not a surprise that LS-DYNA is being mostly used in high-performance clustering environments as they provide the needed flexibility, scalability and efficiency for such simulations. Increasing high-performance clustering productivity and the capability to handle more complex simulations is the most important factor for the automotive makers today. It requires using balanced clustering design (hardware – CPU, memory, interconnect, GPU; and software), enhanced messaging techniques and the knowledge on how to increase the productivity from a given design.

For LS-DYNA, InfiniBand interconnect-based solutions have been proven to provide the highest productivity compared to Ethernet (GigE, 10GigE, iWARP). With InfiniBand, LS-DYNA demonstrated high parallelism and scalability, which enabled it to take full advantage of multi-core high-performance computing clusters. In the case of Ethernet, the better choice between GigE, 10GigE and iWARP is 10GigE. While iWARP aim to provide better performance, typical high-performance applications are using send-receive semantics which iWARP does not provide any added value with, and even worse, it just increase the complexity and the CPU overhead/power consumption.

If you want to get a copy of a paper that present the capabilities to increase simulations productivity while decrease power consumption, don’t hesitate to send me a note (hpc@mellanox.com).

Gilad Shainer
shainer@mellanox.com

Web Retailer Uses InfiniBand to Improve Response Time to Its Customers

Recently while talking with an IT operations manager for a major Web retailer, I was enlightened on the importance of reducing latency in web-based applications. He explained that they were challenged to find a way to reduce the response time to their web customers. They investigated this for quite some time before discovering that the major issue seemed to be the time it takes to initiate a TCP transaction between their app servers and database servers. Subsequently their search focused on finding the best interconnect fabric to minimize this time.

Well, they found it in InfiniBand. With its 1 microsecond latency between servers, this web retailer saw tremendous opportunity to improve response time to its customers. In their subsequent proof of concept testing, they found that indeed they could reduce latency between their app servers and database servers. Resulting improvement to their customers is over 30%. This is a huge advantage in their highly competitive market. I would tell you who they are but they would probably shoot me.

More and more enterprise data centers are finding that low latency, high-performance interconnects, like InfiniBand, can improve their customer-facing systems and their resulting web business.

If you want to hear more, or try it for yourself, send me an email.

Thanks,

Wayne Augsburger
Vice President of Business Development
wayne@mellanox.com

I/O Virtualization

I/O virtualization is a complimentary solution for server and storage virtualization, which aims to reduce the management complexity of physical connections in and out of virtual hosts. Virtualized data center clusters will have multiple networking connections to LAN and SAN, and virtualizing the network avoids the extra complexity associated with it. While I/O virtualization reduces the management complexity, in order to maintain high productivity and scalability one should pay attention to other characteristics of the network being virtualized.

Offloading the network virtualization from the VMM (virtual machine manager, e.g. Hypervisor) to a smart networking adapter, not only reduces the CPU overhead associated with the virtualization management, but also increases the performance capability of the virtual machines (or guest OSs) and can provide the native performance capabilities to them.

The PCISIG has standards in place to help simplify I/O virtualization. The most interesting solution is Single Root I/O virtualization (SR-IOV). SR-IOV allows a smart adapter to create multiple virtual adapters (virtual functions) for a given physical server. The virtual adapters can be assigned directly to a virtual machine (VM) instead of relying on the VMM to manage everything.

SR-IOV provides a standard mechanism for devices to advertise their ability to be simultaneously shared among multiple virtual machines. SR-IOV allows the partitioning of PCI functions into many virtual interfaces for the purpose of sharing the resources of a PCI device in a virtual environment.

Mellanox interconnect solutions provide full SR-IOV support while adding the required scalability and high throughput capabilities to effectively support multiple virtual machines on a single physical server. With Mellanox 10GigE or 40Gb/s InfiniBand solutions, each of the virtual machines can get the needed bandwidth allocation to ensure highest productivity and performance, just as if it was a physical server. 

Gilad Shainer
Director of Technical Marketing
gilad@mellanox.com

SSD over InfiniBand

Last week I was at Storage Networking World in Orlando, Florida.  The sessions were a lot better organized with focus on all the popular topics like Cloud Computing, Storage Virtualization and Solid State Storage (SSD).  In our booth, we demonstrated our Layer 2 agnostic storage supporting iSCSI, FCoE (Fibre Channel over Ethernet) and SRP (SCSI RDMA Protocol) all coexisting in a single network. We partnered with Rorke Data who demonstrated a 40Gb/s InfiniBand-based storage array and Texas Memory System’s ‘World’s Fastest Storage’ in our booth demonstrating sustained rates of 3Gb/s and over 400K I/Os using Solid State Drives. 

I attended few of the sessions on the SSD and Cloud Computing stream. SSD was my favorite topic primarily because InfiniBand and SSD together will provide the highest storage performance and has the potential to carve out a niche in the data center OLTP applications market. Clod Barrera, IBM’s Chief Technical Storage Strategist’s presentation on SSD was very good. He had a chart which talked about how HDD I/O rates per GByte had dropped so low and currently staying constant at around 150 to 200 I/Os per drive. On the contrary SSD’s have capability to produce 50K I/Os on Read and 17K I/Os on Write.  Significant synergy can be achieved by combining SSD with InfiniBand technology. InfiniBand delivers the lowest latency of sub 1us and the highest bandwidth of 40Gb/s.  The combination of these technologies will provide significant value in the datacenter and has the potential to change the database and OLTP storage infrastructure.

SSD over InfiniBand delivers:

-  Ultra-fast, lowest latency infrastructure for transaction processing applications

-  Delivering a more compelling Green per GB 

-   Faster recovery time for business continuity applications

-   Disruptive scaling

I see lot of opportunity for InfiniBand technology in the storage infrastructure as SSD provides the much needed discontinuity to the rotary media. 

TA Ramanujam (TAR)
tar@mellanox.com

Unleashing Performance, Scalability and Productivity with Intel Xeon 5500 Processors “Nehalem”

The industry has been talking about it for a long time, but on March 30th, it was officially announced. The new Xeon 5500 “Nehalem” platform from Intel has introduced a totally new concept of server architecture for Intel-based platforms. The memory has moved from being connected to the chipset to be connected directly to the CPU, and the memory speed has increased. More importantly, PCI-Express (PCIe) Gen2 can now be fully utilized to unleash new performance and efficiency levels from Intel-based platforms. PCIe Gen2 is the interface between the CPU and memory to the networking that connects servers together to form compute clusters. With PCIe Gen2 now being integrated in compute platforms from the majority of OEMs, more data can be sent and received in a single server or blade. This means that applications can exchange data faster and complete simulations much faster, bringing a competitive advantage to end-users. In order to feed the PCIe Gen2, one needs to have a big pipe for his networking solutions, and this is what InfiniBand 40Gb/s brings to the table. No surprise that multiple server OEMs have announced the availability of 40Gb/s InfiniBand in conjunction with Intel announcement (for example HP and Dell).

 

I have been testing several applications to compare the performance benefits of Intel Xeon 5500 processors and Mellanox end-to-end 40Gb/s networking solutions. One of those applications was the Weather Research and Forecasting (WRF) application, widely used around the world. With Intel Xeon-5500-based servers and Mellanox 40Gb/s ConnectX InfiniBand adapters and MTS3600 36-port 40Gb/s InfiniBand switch system, we witnessed a 100% increase in performance and productivity over previous Intel platforms.

With a digital media rendering application – Direct Transport Compositor, we have seen a 100% increases in frames per second delivery, while increasing the screen anti-aliasing at the same time. Other applications have shown similar level of performance and productivity boost as well.

 

The reasons for the new performance levels are the decrease in the latency (1usec) and the huge increase in throughput (more than 3.2GB/s throughput uni-directional on more than 6.5GB/s bi-directional on a single InfiniBand port). With the increase in the number of CPU cores, and new server architecture, bigger pipes in and out from the servers are required in order to keep the system balanced and to avoid creating artificial bottlenecks. Another advantage for InfiniBand is its ability to use RDMA and transfer data directly to and from the CPU memory, without the involvement of the CPU in the data transfer activity. This mean one thing only – more CPU cycles can be dedicated to the applications!

 

Gilad Shainer

Director, HPC Marketing

Moore’s Law’s Data Center Disruption

Change happens, and when you talk to anyone involved in the enterprise data center, change has been accelerating and is making their life more and more complicated. The most recent issue is the growing list of network protocols which the network engineer has to choose from.

 

Previously, the decision on what network protocol was very simple. For IP traffic, you used Ethernet, and for storage, Fibre Channel. Speeds were pretty simple to choose from also. You used 1 Gb Ethernet for the IP and 2 or 4 Gb Fibre Channel. The only challenge was choosing the vendor to purchase the equipment from.

 

Now what has happened is Moore’s Law has made the legacy data center network obsolete. Moore’s Law was originally conceived by one of the founders of Intel, Gordon Moore. He noticed that every generation of microprocessor that Intel made tracked a straight line when transistor count was plotted against time. What was more profound, he noticed that most all semiconductor companies tracked this line. He determined that transistor density of the microprocessors doubled every 18 months. His world famous graphical plot is still used today and now used to describe the steady march of technology.

 

Moore’s Law has caused an issue in the data center. Here is what has happened. For any data center to work properly, its major building blocks (storage, servers and network) should be in balance. Meaning, for them to work most efficiently, they should be matched. Also, you could say these three components of the data center have their functionality primarily dependent on semiconductor manufacturing processes i.e. the advance of Moore’s Law. Historically, storage and servers have tracked Moore’s Law very nicely. But when you look at the network you find a big discrepancy. Ethernet and Fibre Channel have not been tracking Moore’s Law. What has happened recently is that the efficiencies of server processing power and storage bandwidth have progressed so far ahead of the network, that the network has become a bottleneck.

 

Looking at present day data center networks, you can see that not only is the performance sub-par to the I/O needs of the server and storage, but also its functionality and features are woefully behind too. Why is this? If you look at Ethernet and Fibre Channel, you discover these protocols don’t track Moore’s Law. Go ahead and plot the advance in bandwidth over time with both Ethernet and Fibre Channel. Then overlay that onto server CPU density and storage bandwidth (aggregated) and you discover that the legacy network (Ethernet and Fibre Channel) have fallen way behind. Even their future roadmaps don’t track Moore’s Law. We are beginning to see the bottlenecks happening. While Ethernet is very popular, it was never designed for the data center. (Try pumping lots of data from tens-to-hundreds of servers and watch the congestion)! Fibre Channel is really too slow. Even 8 Gb is too slow. This lack of matching the technological advance of the servers and storage has made traditional approaches to data center network topology a dead-end. To get back in balance, the network needs to be matched using newer ways of deploying data enter networks.

 

Getting back to my original point; the network administrator of a large data center is probably noticing network problems and is pretty fed up with having to run 8 to 10 network cables to every server. Also, he can move servers anywhere from his desktop but when it comes to the network, he has to physically go into the data center and add NICs and HBAs plus cables. Throwing adapters and more cables at the problem is counterintuitive and not productive. These activities drive CapEx and OpEx through the roof.

 

There are many new network technologies which are available to the data center network administrator that offer compelling solutions to the Moore’s Law problem. 10Gb Ethernet, Low Latency Ethernet, Data Center Ethernet and InfiniBand all offer a wide range of features and solutions for the enterprise data center and cloud computing. The issue is, can people let go of the legacy way and embrace a new way to think about their network? It’s not about the protocol anymore. There are too many choices for that. The new way is to leverage what makes the most sense for the application. By leveraging the newer protocols and their powerful features

 

The change in the enterprise data center which is causing the network problems is actually a good thing. It is forcing people to think about how they deploy their networks in a new light. By adapting an open viewpoint rather than stubbornly holding onto legacy ways, the network engineer in the enterprise data center can leverage powerful alternatives which makes choice a good thing.


Tony Rea
tony@mellanox.com

Gain A Competitive Advantage

BridgeX received an excellent response from all the analysts that we briefed over the last few weeks. 

 One article talked about how BridgeX reminded the author of the early days of networking when networking companies delivered bridges for Ethernet, Token Ring and Banyan Vines.  The other one talked about the mish-mosh of protocols in the data center as a familiar story.   

 In my opinion, when data centers moved from Fast Ethernet to Gigabit Ethernet it was an easy decision because of the 10x performance improvements that were necessitated by the growth in Internet applications. The same 10x performance is now available with 10 Gigabit Ethernet but the data centers have not jumped into deploying the technology yet. Why?  The killer-app for 10 Gigabit Ethernet is I/O consolidation but the Ethernet protocol itself is still being enhanced in order for it to be deployed as an I/O consolidation fabric. Enhancements to the Ethernet protocol are being made within the IEEE Data Center Bridging Workgroup.  These enhancements will deliver new functionalities to Ethernet, yet the timeline for products is still a BIG question mark. Normally, in a growth economy, products will roll out within 12 to 18 months of spec finalization, whereas in the current economic condition it might taker a longer time and the spec is at least 18 months away for finalization. Till then,10 Gigabit Ethernet deployments will happen in data centers in smaller, niche applications and will not be deployed for I/O consolidation. So, if data centers want to save energy costs, reduce floor space and lower TCO today, then deploying a proven I/O consolidation fabric is critical. 

Just some of the enhancements currently being made to the Ethernet protocol in the IEEE:

  1. Lossless fabric
  2. Creating Virtual Lanes and providing granular QoS
  3. Enabling Fat-Tree
  4. Congestion management

These are already part of the InfiniBand fabric which has been shipping for almost 9 years now, and has been successfully deployed in several data centers and high-performance commercial applications.

Oracle Exadata is a great product that drives InfiniBand to the forefront of data centers for database applications. Exadata brings in new thinking and new strategy for delivering higher I/Os and lowering energy costs. Exadata certainly delivers a competitive advantage. 

Similarly, BridgeX coupled with ConnectX adapters and InfiniScale switching platforms provides competitive advantages by delivering a cost-optimized,I/O consolidation fabric. Data centers can consolidate their I/O using InfiniBand as the physical fabric and the virtual fabric will continue to be Ethernet or Fibre Channel. This means that the applications that need an Ethernet transport or a Fibre Channel transport will run un-modified in the InfiniBand cloud.   

I think it is time for the data centers to take a new look at their infrastructure and re-strategize the investments to gain an even greater competitive advantage. When the economy turns around, those who have infrastructure that can leapfrog their competition will eventually win.

TA Ramanujam (TAR)
tar@mellanox.com

Mellanox at VMworld Europe

Yesterday, myself along with Motti Beck and Ali Ayoub (our main VMware software developer at Mellanox) diligently put together a very compelling demo that highlights the convergence capabilities of our BridgeX BX 4000 gateway that we announced last week.

We unpacked everything and got it all up and running in less than an hour (this after we sorted out the usual power and logistical issues that always comes with having a booth).

 

 
The slide below illustrates the topology of the demo. Essentially, we have two ConnectX adapters cards in one of the Dell server running two different interconnect fabrics. One adapter is running 40Gb/s InfiniBand, while the other adapter is running 10 Gigabit Ethernet.

1. The 40Gb/s InfiniBand adapter is connected to our MTS3600 40Gb/s InfiniBand switch which then passes through the BridgeX BX4020 where we convert the packets to Ethernet. The packets then run through the Arista 10GigE Switch and then into the LeftHand Appliance Virtual Machine which resides on the Dell Server (which is running ESX 3.5 and our certified 10GigE driver over our ConnectX EN 10GigE SFP+ adapter). We are showing a movie from the iSCSI storage on the IB end-point (the Dell Linux Server).

2. The 10 Gigabit Ethernet Adapter connects directly to the BridgeX BX4020 where it converts the traffic to FC (effectively FCoE). The traffic then moves to the Brocade Fibre Channel switch and then directly to the NetApp storage. We are showing a movie from the FC NetApp storage on the 10GigE end-point (the Dell Linux Server).

If you are coming to VMorld Europe (or already here) come and see us in Booth #100 and we will be happy to walk you through the demo.

Brian Sparks
Director, Marketing Communications
brian@mellanox.com