I had the pleasure to be little bit involved in the creation of the fastest supercomputer in Asia, and the second fastest supercomputer in the world – the Dawning “Nebulae” Petaflop Supercomputer at SIAT. If we look on the peak flops capacity of the system – nearly 3 Petaflops, it is the largest supercomputer in the world. I visited the supercomputer site in April and saw how fast it was assembled. It took around 3 weeks to get it up and running – amazing, well, this is one of the benefits of using cluster architecture instead of the expensive proprietary systems. The first picture by the way was taken during the system setup in Shenzhen.
The system includes 5200 Dawning TC3600 Blades, each with NVIDIA Fermi GPU to provide 120K cores, all connected with Mellanox ConnectX InfiniBand QDR adapters, IS5000 switches and the fabric management. It is the 3rd system in the world to provide more than sustained Petaflop performance (after Roadrunner and Jaguar). Unlike Jaguar (from Cray) that requires 20K nodes to reach the required performance, Nebulae does it with only 5.2K nodes – reducing the needed real-estate etc, making is much more cost effective. It is yet another prove that commodity-based supercomputers can deliver better performance, cost/performance and other x/performance metrics compared to the proprietary systems. As GPUs gain higher popularity, we also witness the effort that is being done to create and port the needed applications to GPU-based environments, which will bring a new era of GPU computing. It is clear that GPUs will drive the next phase of supercomputers, and of course the new speeds and feeds of the interconnect solutions (such as the IBTA’s new specifications for the FDR/EDR InfiniBand speeds).
The second picture was taken at the ISC’10 conference, after the Top500 award ceremony. You can see the Top500 certificates…
Started in 1993, the TOP500 lists the fastest computers used today, ranked according to Linpack benchmark results. Published twice a year, the TOP500 list provides an important tool for tracking usage trends in high-performance computing. The 33rd TOP500 List was released in Hamburg, Germany, during the ISC’09 conference.
This year’s list revealed that Mellanox InfiniBand demonstrated up to 94 percent system utilization, only 6 percent under the theoretical limit, providing users with the best return on investment for their high-performance computing server and storage infrastructure. This year’s TOP500 list reveals that InfiniBand is the only growing industry-standard interconnect solution, increasing 25 percent to 152 systems, representing more than 30 percent of the TOP500. Mellanox ConnectX® InfiniBand adapters and switch systems based on its InfiniScale® III and IV switch silicon provide the scalable, low-latency, and power-efficient interconnect for the world’s fastest supercomputer and the majority of the top 100 systems. Mellanox end-to-end 40Gb/s InfiniBand solutions deliver the leading performance and highest Top10 system efficiency in the 10th ranked Jülich cluster.
Highlights of InfiniBand usage on the June 2009 TOP500 list include:
- Mellanox InfiniBand interconnect products connect the world’s fastest supercomputer, 4 of the top 10 most prestigious positions, and 9 of the top 20 systems
- Mellanox InfiniBand provides the highest system utilization, up to 94 percent, which is 50 percent higher than the best GigE-based system
- Mellanox 40Gb/s end-to-end solutions provide the highest system utilization on the top 10, 15% higher than the average top 20 efficiency
- All InfiniBand-based clusters (152 total supercomputers) use Mellanox solutions
- InfiniBand is the most used interconnect among the top 100 supercomputers with 59 systems, nearly 12 times the number of Gigabit Ethernet-based clusters and the number of proprietary high speed cluster interconnects
- The total number of InfiniBand-connected CPU cores on the list has grown from 606,000 in June 2008 to 1,040,000 in 2009 (72 percent yearly growth)
- InfiniBand is the only growing industry-standard clustered interconnect in the TOP500 with a 25 percent growth rate compared to June 2008
- Mellanox InfiniBand interconnect products present in the TOP500 are used by a diverse list of applications, from large-scale, high-performance computing to commercial technical computing and enterprise data centers
- The entry level for the TOP500 list is 17TFlops, 91% percent higher than the 8.9TFlops necessary to be on the June 2008 list
Full analysis of the TOP500 can be found HERE.