Monthly Archives: March 2009

Unleashing Performance, Scalability and Productivity with Intel Xeon 5500 Processors “Nehalem”

The industry has been talking about it for a long time, but on March 30th, it was officially announced. The new Xeon 5500 “Nehalem” platform from Intel has introduced a totally new concept of server architecture for Intel-based platforms. The memory has moved from being connected to the chipset to be connected directly to the CPU, and the memory speed has increased. More importantly, PCI-Express (PCIe) Gen2 can now be fully utilized to unleash new performance and efficiency levels from Intel-based platforms. PCIe Gen2 is the interface between the CPU and memory to the networking that connects servers together to form compute clusters. With PCIe Gen2 now being integrated in compute platforms from the majority of OEMs, more data can be sent and received in a single server or blade. This means that applications can exchange data faster and complete simulations much faster, bringing a competitive advantage to end-users. In order to feed the PCIe Gen2, one needs to have a big pipe for his networking solutions, and this is what InfiniBand 40Gb/s brings to the table. No surprise that multiple server OEMs have announced the availability of 40Gb/s InfiniBand in conjunction with Intel announcement (for example HP and Dell).

 

I have been testing several applications to compare the performance benefits of Intel Xeon 5500 processors and Mellanox end-to-end 40Gb/s networking solutions. One of those applications was the Weather Research and Forecasting (WRF) application, widely used around the world. With Intel Xeon-5500-based servers and Mellanox 40Gb/s ConnectX InfiniBand adapters and MTS3600 36-port 40Gb/s InfiniBand switch system, we witnessed a 100% increase in performance and productivity over previous Intel platforms.

With a digital media rendering application – Direct Transport Compositor, we have seen a 100% increases in frames per second delivery, while increasing the screen anti-aliasing at the same time. Other applications have shown similar level of performance and productivity boost as well.

 

The reasons for the new performance levels are the decrease in the latency (1usec) and the huge increase in throughput (more than 3.2GB/s throughput uni-directional on more than 6.5GB/s bi-directional on a single InfiniBand port). With the increase in the number of CPU cores, and new server architecture, bigger pipes in and out from the servers are required in order to keep the system balanced and to avoid creating artificial bottlenecks. Another advantage for InfiniBand is its ability to use RDMA and transfer data directly to and from the CPU memory, without the involvement of the CPU in the data transfer activity. This mean one thing only – more CPU cycles can be dedicated to the applications!

 

Gilad Shainer

Director, HPC Marketing

Giving Back to Our Community

One of Mellanox’s strongest values as a company is a commitment to give back to the community. Continuing our tradition of giving back to the community, we announced today the donation of $160,000. This amount has been divided among fourteen charities and education programs within Israel and the United States: Arazim Elementary School, Baldwin Elementary School, Foothill High School, Harvest Park Middle School, Highlands Elementary School, Leukemia & Lymphoma Society, Oakland’s Children’s Hospital & Research Center, Ort Israel, Simonds Elementary School, Susan G. Komen for the Cure, Twelve Bridges Elementary School, and Williams Wins Foundation.

 

These organizations, handpicked by our employees, provide an excellent opportunity to create awareness and raise funds for the advancement of invaluable health and education programs for the community. We are proud to be supporters of those efforts, especially during these times.

New Territories Explored: Distributed File System Hadoop

It took me a while but I’m back – hope you’re all been waiting to hear from me .

With that, I’ve decided to go into un-charted territories…HADOOP

 

Hadoop is an Apache project. It is a framework, written in Java, for running applications on large clusters built with commodity hardware (distributed computing). Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster.

 

Hadoop provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. The 2 key functions in Hadoop are map and reduce. I would like to briefly touch on what they mean.

 

The map function processes a key/value pair to generate a set of intermediate key/value pairs, while the reduce function merges all intermediate values associated with the same intermediate key. A map-reduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner, the framework sorts the outputs of the maps, which then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system Master/Slave architecture.

 

The  test we’ve conducted was the DFSIO benchmark, a map-reduce job where each map task opens a file and writes/reads to/from it, closes it, and measures the I/I time. There is only single reduce task which aggregates individual times and sizes. We’ve limited the test for 10 to 50 files measurements with 360 MB that we found reasonable compared with the ratio of number of nodes used and number of files. We then compared that to a public publish from Yahoo  which used 14k files over 4k nodes. This boils down to 3.5 files per node where we are using 50 files over 12 nodes, which equates to over 4 files per node.

 

Given the above configuration and the test described above, here is a snap shot of the results we’ve seen:

 

 



It can clearly be seen from the above, as well as through other results we’ve been given, that InfiniBand and 10GigE (via our ConnectX adapters) is half the time in execution time and over triple in bandwidth…these are very conclusive results by any matrix. 

 

A very interesting point to review is that the tests which were executed using DFS located on a hard disk showed significant better performance, but when testing with RamDisk, the gap increased even more. e.g. latency became from half to one-third… it seems like a clear way to unleash the potential.

 

In my next blog post I’ll plan to either review a new application or anther aspect of this application.

 

 

Nimrod Gindi

Director of Corporate Strategy

nimrodg@mellanox.com

Mellanox ConnectX Ethernet – I/O Consolidation for the Data Center

Today’s data center requires a low-cost, low-power I/O solution with network flexibility to provide I/O consolidation on a single adapter. Network administrators want the best performance, scalability, latency while solving all their LAN, SAN and IPC (Clustering) needs packed into one adapter card in a virtualized or data center environment.

ConnectX® is a single chip solution from Mellanox that provides these features for the Data Center I/O unification with its hardware and software capabilities.  ConnectX® EN 10 Gigabit Ethernet drivers provide seamless connectivity by providing optimized 10 Gigabit Ethernet I/O services that easily scale with multi-core CPUs and virtualized servers and storage architectures.

Mellanox entry into the 10GigE landscape was rather late if you consider 10GigE started showing up in 2001 on servers as PCI-X followed by PCIe adapters. With Mellanox’s extensive experience in high-performance computing and broad range of industry relationships, it has forged ahead with this technology and was the first company to offer 10GigE with PCIe 2.0. Along the way, our products have matured to become the market-leader for performance, latency, as well as consolidating all data center networking onto a single adapter.

In a span of less than 2 years, Mellanox has introduced a broad range of products supporting various media interconnects and cabling options including UTP, CX4 for copper and SR, LR and LRM for fiber optics.

Technology leadership in networking requires that a company not only have the best hardware solution, but compliment this with the best software solution to make a winning combination.

In my experience, working at other early startups, as well as network technology bellwether 10Gigabit Ethernet companies, the Gen1/Gen2 10GigE products introduced lacked the vision of what the end-customer requirements were. The products were a “mish-mosh” of features addressing 10GigE for LAN, clustering (iWARP), TCP acceleration (aka TOE) and iSCSI acceleration. They missed the mark by not solving the pain-points of a data center, be it blazing performance, low-latency, CPU utilization or true I/O consolidation.

Mellanox took the holistic approach to data center networking with a deep understanding from its InfiniBand leadership and knowledgebase, server and system configuration, virtualization requirements and benefits, driver software requirements, and most importantly, understanding customer requirements for each vertical segment.

Today, ConnectX® EN 10 Gigabit Ethernet drivers support a broad array of major operating systems, including Windows, Linux, VMware Infrastructure, Citrix XenServer and FreeBSD.

The ConnectX® EN 10 Gigabit Ethernet drivers provide:

- All Stateless offload features
- Virtualized accelerations
- Data Center Ethernet (DCE) support
- FCoE with full hardware offload for SAN consolidation on 10GigE
- Lowest 10GigE (TCP/IP) latency comparable to expensive iWARP solutions
- Single Root – IO Virtualization (SR-IOV) for superior virtualization performance
- Linux Kernel and Linux Distribution support
- WHQL certified drivers for Windows Server 2003 and 2008
- VMware Ready certification for VMware Virtual Infrastructure (ESX 3.5)
- XenServer 4.1 inbox support
- Line-rate performance with very low CPU utilization
- Replace multiple GigE NICs with a single ConnectX Ethernet adapter

To complete the last piece of the puzzle, i.e. IPC (clustering) for the Data Center, I’ll soon post in my blog on Industry’s Low Latency Ethernet (LLE) initiative and its advantages compared to current available clustering solutions on 10GigE.

Regards,
Satish Kikkeri
satish@mellanox.com

Moore’s Law’s Data Center Disruption

Change happens, and when you talk to anyone involved in the enterprise data center, change has been accelerating and is making their life more and more complicated. The most recent issue is the growing list of network protocols which the network engineer has to choose from.

 

Previously, the decision on what network protocol was very simple. For IP traffic, you used Ethernet, and for storage, Fibre Channel. Speeds were pretty simple to choose from also. You used 1 Gb Ethernet for the IP and 2 or 4 Gb Fibre Channel. The only challenge was choosing the vendor to purchase the equipment from.

 

Now what has happened is Moore’s Law has made the legacy data center network obsolete. Moore’s Law was originally conceived by one of the founders of Intel, Gordon Moore. He noticed that every generation of microprocessor that Intel made tracked a straight line when transistor count was plotted against time. What was more profound, he noticed that most all semiconductor companies tracked this line. He determined that transistor density of the microprocessors doubled every 18 months. His world famous graphical plot is still used today and now used to describe the steady march of technology.

 

Moore’s Law has caused an issue in the data center. Here is what has happened. For any data center to work properly, its major building blocks (storage, servers and network) should be in balance. Meaning, for them to work most efficiently, they should be matched. Also, you could say these three components of the data center have their functionality primarily dependent on semiconductor manufacturing processes i.e. the advance of Moore’s Law. Historically, storage and servers have tracked Moore’s Law very nicely. But when you look at the network you find a big discrepancy. Ethernet and Fibre Channel have not been tracking Moore’s Law. What has happened recently is that the efficiencies of server processing power and storage bandwidth have progressed so far ahead of the network, that the network has become a bottleneck.

 

Looking at present day data center networks, you can see that not only is the performance sub-par to the I/O needs of the server and storage, but also its functionality and features are woefully behind too. Why is this? If you look at Ethernet and Fibre Channel, you discover these protocols don’t track Moore’s Law. Go ahead and plot the advance in bandwidth over time with both Ethernet and Fibre Channel. Then overlay that onto server CPU density and storage bandwidth (aggregated) and you discover that the legacy network (Ethernet and Fibre Channel) have fallen way behind. Even their future roadmaps don’t track Moore’s Law. We are beginning to see the bottlenecks happening. While Ethernet is very popular, it was never designed for the data center. (Try pumping lots of data from tens-to-hundreds of servers and watch the congestion)! Fibre Channel is really too slow. Even 8 Gb is too slow. This lack of matching the technological advance of the servers and storage has made traditional approaches to data center network topology a dead-end. To get back in balance, the network needs to be matched using newer ways of deploying data enter networks.

 

Getting back to my original point; the network administrator of a large data center is probably noticing network problems and is pretty fed up with having to run 8 to 10 network cables to every server. Also, he can move servers anywhere from his desktop but when it comes to the network, he has to physically go into the data center and add NICs and HBAs plus cables. Throwing adapters and more cables at the problem is counterintuitive and not productive. These activities drive CapEx and OpEx through the roof.

 

There are many new network technologies which are available to the data center network administrator that offer compelling solutions to the Moore’s Law problem. 10Gb Ethernet, Low Latency Ethernet, Data Center Ethernet and InfiniBand all offer a wide range of features and solutions for the enterprise data center and cloud computing. The issue is, can people let go of the legacy way and embrace a new way to think about their network? It’s not about the protocol anymore. There are too many choices for that. The new way is to leverage what makes the most sense for the application. By leveraging the newer protocols and their powerful features

 

The change in the enterprise data center which is causing the network problems is actually a good thing. It is forcing people to think about how they deploy their networks in a new light. By adapting an open viewpoint rather than stubbornly holding onto legacy ways, the network engineer in the enterprise data center can leverage powerful alternatives which makes choice a good thing.


Tony Rea
tony@mellanox.com