Pushing the frontiers of science and technology will require extreme-scale computing with machines that are 500-to-1,000 times more capable than today’s supercomputers. As researchers continuously refine their models, the demand for more parallel computation and advanced networking capabilities is paramount.
As a result of the ubiquitous data explosion and the ascendance of big data, today’s systems need to move enormous amounts of data and perform more sophisticated analysis; the interconnect truly becomes the critical element of enabling the use of data.
This week at the IBM Enterprise2014 Executive Summit in Las Vegas, IBM unveiled new Power8 based infrastructure for cloud, data, web 2.0, and mobile engagement. Mellanox is being showcased as a key partner enabling critical platforms for IBM’s Big Data analytics, cloud, and software defined “Elastic Storage” solutions. The new Power8 platform incorporates Mellanox 40Gb Ethernet networking gear and a fully integrated Turbo LAMP (Linux, Apache, MySQL, PHP) software stack.
This Turbo LAMP stack came about through a development partnership between IBM, Mellanox and several software vendors:
- Canonical (Ubuntu Linux & Apache Web Server)
- SkySQL (MariaDB/MySQL Database)
- Zend (PHP)
The Turbo LAMP integration is important as it is the foundation for the most common ecommerce, content management systems, and Big Data analytics platforms. This integration allows customers to deliver optimized mobile and web applications while offering critical performance, scale and secure access that businesses need.
On Thursday October 9, our very own Matthew Sheard will be on stage at IBM Enterprise Conference providing details on the solution as outlined in this presentation.
Guest post by Alex Henthorn-Iwane, QualiSystems
Big data is for real, but its places heavy demands on IT teams, who have to pull together and provision cloud infrastructure, then offer big data application deployments with validated performance to meet pressing business decision timelines. QualiSystems is partnering with Mellanox to simplify big data deployments over any cloud infrastructure, enabling IT teams to meet line of business needs while reducing operational costs.
Companies today are finding that the size and growth of stored data is becoming overwhelming. As the databases grow, the challenge is to create value by discovering insights and connections in the big databases in as close to real time as possible. In the recently published whitepaper, “Achieving Real-Time Business Solutions Using Graph Database Technology and High Performance Networks“ we describe a combination of high performance networking and graph base and analytics technologies which offers a solution to this need.
Each of the examples in the paper is based on an element of a typical analysis solution. In the first example, involving Vertex Ingest Rate shows the value of using high performance equipment to enhance real-time data availability. Vertex objects represent nodes in a graph, such as Customers, so this test is representative of the most basic operation: loading new customer data into the graph. In the second example, Vertex Query Rate highlights the improvement in the time needed to receive results, such as finding a particular customer record or a group of customers.
The third example, Distributed graph navigation processing starts at a Vertex and explores its connections to other Vertices. This is representative of traversing social networks, finding optimal transportation or communications routes and similar problems. The final example, Task Ingest Rate shows the performance improvement when loading the data connecting each of the vertices. This is similar to entering orders for products, transit times over a communications path and so on.
Each of these elements is an important part of a Big Data analysis solution. Taken together, they show that InfiniteGraph can be made significantly more effective when combined with Mellanox interconnect technology.
Resources: Mellanox Web 2.0 Solutions
Big Data solutions such as Hadoop and NoSQL applications are no longer a sole game for Internet moguls. Today’s retail, transportation and entertainment corporations use Big Data practices such as Hadoop for data storage and data analytics.
IBM BigInsights makes Big Data deployments an easier task for the system architect. BigInsights with IBM’s GPFS-FPO file system support provides enterprise level Big Data solution, eliminating Single Point of Failure structures and increasing ingress and analytics performance.
The inherent RDMA support in IBM’s GPFS takes the performance aspect a notch higher. The testing conducted at Mellanox Big Data Lab with IBM BigInsights 2.1, GPFS-FPO and FDR 56Gbps InfiniBand showed an increased performance for write and read of 35% and 50 %, respectively, comparing to a vanilla HDFS deployment. On the analytics benchmarks, the system provided 35% throughput gain by enabling the RDMA feature.
Hadoop MapReduce is the leading Big Data analytics framework. This framework enables data scientists to process data volumes and variety never processed before. The result from this data processing is new business creation and operation efficiency.
As MapReduce and Hadoop advance, more organizations try to use the frameworks in near real-time capabilities. Leveraging RDMA (Remote Direct Memory Access) capabilities for faster Hadoop MapReduce capabilities has proven to be a successful method.
In our presentation at Oracle Open World 2013, we show the advantages RDMA brings to enterprises deploying Hadoop and other Big Data applications:
- Double analytics performance, accelerating MapReduce framework
- Double Hadoop file system ingress capabilities
- Reducing NoSQL Databases’ latencies by 30%
On the analytics side, UDA (Unstructured Data Accelerator), doubles the computation power by offloading networking and buffer copying from the server’s CPU to the network controller. In addition, a novel shuffle and merge approach helped to achieve the needed performance acceleration. The UDA package is and open source package available here (https://code.google.com/p/uda-plugin/). The HDFS (Hadoop Distributed File System) layer is also getting its share of performance boost.
While the community continues to improve the feature, work conducted at Ohio State University brings the RDMA capabilities to the data ingress process of HDFS. Initial testing shows over 80% improvement in the data write path to the HDFS repository. The RDMA HDFS acceleration research and downloadable package is available from the Ohio State University website at: http://hadoop-rdma.cse.ohio-state.edu/
We are expecting more RDMA acceleration enablement to different Big Data frameworks in the future. If you have a good use case, we will be glad to discuss the need and help with the implementation.
Contact us through the comments section below or at firstname.lastname@example.org
: Eyal Gutkind is a Senior Manager, Enterprise Market Development at Mellanox Technologies focusing on Web 2.0 and Big Data applications. Eyal held several engineering and management roles at Mellanox Technologies over the last 11 years. Eyal Gutkind holds a BSc. degree in Electrical Engineering from Ben Gurion University in Israel and MBA from Fuqua School of Business at Duke University, North Carolina.