Advantages of RDMA for Big Data Applications

Hadoop MapReduce is the leading Big Data analytics framework. This framework enables data scientists to process data volumes and variety never processed before. The result from this data processing is new business creation and operation efficiency.  

As MapReduce and Hadoop advance, more organizations try to use the frameworks in near real-time capabilities. Leveraging RDMA (Remote Direct Memory Access) capabilities for faster Hadoop MapReduce capabilities has proven to be a successful method.

In our presentation at Oracle Open World 2013, we show the advantages RDMA brings to enterprises deploying Hadoop and other Big Data applications:

-        Double analytics performance, accelerating MapReduce framework

-        Double Hadoop file system ingress capabilities

-        Reducing NoSQL Databases’ latencies by 30%

On the analytics side, UDA (Unstructured Data Accelerator), doubles the computation power by offloading networking and buffer copying from the server’s CPU to the network controller. In addition, a novel shuffle and merge approach helped to achieve the needed performance acceleration. The UDA package is and open source package available here (https://code.google.com/p/uda-plugin/).  The HDFS (Hadoop Distributed File System) layer is also getting its share of performance boost.

While the community continues to improve the feature, work conducted at Ohio State University brings the RDMA capabilities to the data ingress process of HDFS. Initial testing shows over 80% improvement in the data write path to the HDFS repository. The RDMA HDFS acceleration research and downloadable package is available from the  Ohio State University website at: http://hadoop-rdma.cse.ohio-state.edu/

We are expecting more RDMA acceleration enablement to different Big Data frameworks in the future.  If you have a good use case, we will be glad to discuss the need and help with the implementation.

Contact us through the comments section below or at bigdata@mellanox.com

 

eyal gutkind
Author: Eyal Gutkind is a Senior Manager, Enterprise Market Development at Mellanox Technologies focusing on Web 2.0 and Big Data applications. Eyal held several engineering and management roles at Mellanox Technologies over the last 11 years. Eyal Gutkind holds a BSc. degree in Electrical Engineering from Ben Gurion University in Israel and MBA from Fuqua School of Business at Duke University, North Carolina.

About Cecelia Taylor

Cecelia has served as the Sr. Social Media Manager for Mellanox since 2013. She previously worked at Cisco & ZipRealty managing social media marketing, publishing and metrics. Prior to social media, she worked in B2B publishing in the area of audience development. She has a BA from Mills College and resides in the SF East Bay. Follow her on Twitter: @CeceliaTaylor