Big Data solutions such as Hadoop and NoSQL applications are no longer a sole game for Internet moguls. Today’s retail, transportation and entertainment corporations use Big Data practices such as Hadoop for data storage and data analytics.
IBM BigInsights makes Big Data deployments an easier task for the system architect. BigInsights with IBM’s GPFS-FPO file system support provides enterprise level Big Data solution, eliminating Single Point of Failure structures and increasing ingress and analytics performance.
The inherent RDMA support in IBM’s GPFS takes the performance aspect a notch higher. The testing conducted at Mellanox Big Data Lab with IBM BigInsights 2.1, GPFS-FPO and FDR 56Gbps InfiniBand showed an increased performance for write and read of 35% and 50 %, respectively, comparing to a vanilla HDFS deployment. On the analytics benchmarks, the system provided 35% throughput gain by enabling the RDMA feature.
This week is EMC World, a huge event with tens of thousands of customers, partners, resellers and EMC employees talking about cloud, storage, and virtualization. EMC sells many storage solutions but most of the excitement and recent growth (per the latest EMC earnings announcement) are about scale-out storage, including EMC’s Isilon, XtremIO, and ScaleIO solutions.
As mentioned in my blog on the four big changes in storage, traditional scale-out storage connects many storage controllers together, while the new scale-out server storage links the storage on many servers. In both designs the disk or flash on all the nodes in each node is viewed and managed as one large pool of storage. Instead of having to manually partition and assign workloads to different storage systems, workloads can be either shifted seamlessly from node to node (no downtime) or distributed across the nodes.
Clients connect to (scale-out storage) or run on (scale-out server storage) different nodes but must be able to access storage on other nodes as if it were local. If I’m connecting to node A, I need rapid access to the storage on node A, B, C, D, and all the other nodes in the cluster. The system may also migrate data from one node to another, and rapidly exchange metadata or control traffic to keep track of who has which data.
In 1967, Gene Amdahl developed a formula that calculates the overall efficiency of a computer system by analyzing how much of the processing can be parallelized and the amount of parallelization that can be applied in the specific system.
At that time, deeper performance analysis had to take into consideration the efficiency of three main hardware resources that are needed for the computation job: the compute, memory and storage.
On the compute side, efficiency has to be measured by how many threads can run in parallel (which depends on the number of cores). The memory size affects the percentage of IO operation that needs to access the storage, which slows significantly the execution time and the overall system efficiency.
Those three hardware resources worked very well until the beginning of 2000. At that time, the computer industry started to use a grid-computing or as it known today, scale-out systems. The benefits of the scale-out architecture are clear. It enables building systems with higher performance, easy to scale with built-in high availability at a lower cost. However, the efficiency of those systems heavily depend on the performance and the resiliency of the interconnect solution.
The importance of the Interconnect became even bigger in the virtualized data center, where the amount of east west traffic continues to grow (as more parallel work is being done). So, if we want to use Amdahl’s law to analyze the efficiency of the scale-out system, in addition to the three traditional items (compute, memory & storage) the fourth item, which is the Interconnect, has to be considered as well.
Las Vegas, Nevada is not only the home of games, art, shows and fun, also serves as home to one of the largest Hadoop clusters in the world!
During the upcoming 2014 EMC World show, we invite you to join us for an informative tour of SuperNAP, The World’s leader in Data Center EcoSystem Development and home of the 1000-node Hadoop cluster. In this tour, we will show how a Hadoop cluster is deployed in a co-location data center, maintained and provide analytics tools for a large community of businesses and academic institutes. It will be a great opportunity to learn about actual working cluster workloads, design considerations and available tools for next generation businesses opportunities in Big Data.
Building a large scale data center is not an easy task and one that includes considerable cost. The larger the cluster is, the larger the core switching element needs to be to carry traffic between servers and storage elements of the data center.
Multiple redundancy and distribution mechanisms are needed to avoid network outages, make implementations resilient and reduce the business impact of failed network elements.
The Virtual Modular Switch (VMS) solution provides a distributed core element to the data center. The VMS is logically placed where you would traditionally place a chassis. Its benefit is targeted for increased resiliency by offering built-in redundancy and distribution of the networking load between multiple elements.
Mellanox recently announced a collaboration with IBM to produce a tightly integrated server and storage solutions that incorporate our end-to-end FDR 56Gb/s InfiniBand and 10/40 Gigabit Ethernet interconnect solutions with IBM POWER CPUs. By combining IBM POWER CPUs with the world’s highest-performance interconnect solution will drive data at optimal rates, maximizing performance and efficiency for all types of applications and workloads, as well as enable dynamic storage solutions to allow multiple applications to efficiently share data repositories.
Advances in high-performance applications are enabling analysts, researchers, scientists and engineers to run more complex and detailed simulations and analyses in a bid to gather game-changing insights and deliver new products to market. This is placing greater demand on existing IT infrastructures, driving a need for instant access to resources – compute, storage, and network.
Companies are looking for faster and more efficient ways to drive business value from their applications and data. The combination of IBM processor technologies and Mellanox high-speed interconnect solutions can provide clients with an advanced and efficient foundation to achieve their goals.
This week, Las Vegas hosts the National Association of Broadcasters conference, or NAB Show. A big focus is the technology needed to deliver movies and TV shows using 4K video.
Standard DVD video resolution is 720×480. Blue-ray resolution is 1920×1080. But, thanks to digital projection in movie theatres and huge flat-screen TVs at home, more video today is being shot in 4K (4096×2160) resolutions. The video is stored compressed but must be streamed uncompressed for many editing, rendering, and other post-production workflows. Each frame has over 8 million pixels and requires 24x greater bandwidth than DVD (4x greater bandwidth than Blue-ray).
The world of information technologies is chaotic. Technologists are expected to learn, keep pace and implement ever-evolving technologies. As a technology manager, you will face challenges related to bridging the knowledge gap on your team. Your education and training solution should address these needs:
Always available: 24 hours a day, 7 days a week, 365 days a year, On Demand availability
A variety of education options including online, face to face instructor- led and remote training
Self-paced learning systems to allow learners to work around their own schedules
With these challenges in mind, Mellanox has customized its Global Education Services to provide more resources for your team. All training includes:
Learners’ support channels, such as the “Ask the Expert” feature
In our day to day tech lives, it is our goal to improve our team’s performance using the right combination of resources and budget, while showing the best return on investment (ROI). This can come at the cost of skipping technical training.
It’s a fact. We are all stressed. We face unreasonable deadlines, prioritization, immediate to-do tasks and budget shortfalls. While trying to survive in this chaotic environment, I hear some customers saying: “I don’t have the time to allow my tech team to get technical training. Even if I have the time, I just can’t afford it.”
Let’s think again.Market standards show that the average number of days a technology employee works each year is 207. The average number of days dedicated to maintenance of ongoing projects is 170 (82.5%). The rest (37 days) of the time is dedicated to “first-time operations and exploration”.
Studies have shown that every day spent in training reduces the ongoing maintenance time by 4% (per participant) on average. This means that from 170 working days, you could save at least 6 days per year per employee if you just provide them just 1 training day. Each training day reduces first-time operations by 20% on average. From 37 working days, you could save at least 7 more days.
Shout out to anyone who happens to attend the GPU Technology Conference 2014! This conference is touted as the world’s biggest and most important GPU developer conference. Follow all the social conversation around the event using the hashtag #GTC2014. The conference will be held next week, March 24-27,2014 at the San Jose McEnery Convention Center in San Jose, CA.
This is the fourth year I am attending this event and I will be hanging out at the “Ask the Expert Table” at the GTC. Feel free to swing by and chat about any of your burning questions you may have on GPUDirect RDMA with Mellanox InfiniBand!