Mellanox Academy

Connect with us:  Follow the Mellanox Community  Connect with Mellanox on Facebook  Follow Mellanox on Twitter  Connect with Mellanox on Google +  Watch Mellanox on YouTube  Network with Mellanox on LinkedIn

Mellanox Academy Course

InfiniBand Fabrics – Advanced Training

Part Number: MTR-IB-OST-A

Duration: 2 Days

This course is focusing on multicast, along with fabric design guidelines and what obstacles you should avoid when designing a fabric.
We will also focus on how to select and configure the most suitable routing algorithm for a chosen topology.
In addition this course provides the necessary skills and knowledge to effectively diagnose and troubleshoot InfiniBand fabrics.  
You will also receive and overview of the required tools to effectively compare InfiniBand fabrics

This course is a continuation of the Fundamentals of InfiniBand Fabrics Course (MTR-IB­OST-B, MTR-IB-OPEC-3). 
It is designed to provide more advanced materials to support InfiniBand Level 1 and Level 2 technologists by focusing on fabric design, tuning and troubleshooting, multicast, and SNMP principles. 

What's in it for me?

Do you find yourself needing to understand InfiniBand with multicast or needing to know how to properly design fabrics? Maybe configure SNMP monitoring with InfiniBand? How about using virtualization with InfiniBand?
We have created this InfiniBand advanced course for just that reason.

Much of the Financial Services Industry (FSI) has adopted InfiniBand to accelerate their multicast applications. This advanced course is going to focus on multicast, along with fabric design and which tools to use when comparing InfiniBand fabrics. What obstacles you should look out for and avoid when designing your fabrics. Do you need to set up SNMP with InfiniBand? Or basic RDMA with InfiniBand? We have a section for that as well, plus much more.

This course is a continuation of the InfiniBand Foundations Basic Course (MTR-IBOST- B). It is designed to provide more advanced foundation materials to support InfiniBand Level 1 and Level 2 technologists by focusing on fabric design, fabric tuning, multicast, and SNMP principles.

Course Objectives

Upon completion of this course, the student will understand the following:
  • RDMA and ULPs
  • InfiniBand Routing Algorithms
  • OpenSM Configuration
  • InfiniBand Quality of Service (QoS)
  • Performance Parameters & Test Process
  • InfiniBand ULPs, SRP and IPoIB
  • InfiniBand Multicast
  • InfiniBand Fabric Congestion
  • Cluster Bring-up
  • InfiniBand with VMware Introduction

Target Audience

  • Network & system administrators
  • Network & system engineers
  • Architects
  • InfiniBand pre-sales staff


Data communications, Linux knowledge, InfiniBand Fundamentals course or equivalent

Day 1 Agenda

InfiniBand Protocol Layers

  • InfiniBand Header Structure
  • Physical Layer L1
  • Link Layer L2
  • Transport Layer & InfiniBand Queue Pairs
  • Transport Modes
  • Checking Kernel Support for IPoIB

InfiniBand Supported Protocols

  • Describe the Primary Upper Layer Protocols
  • Which Protocols can Utilize the RDMA Capabilities? (Verbs)
  • Who Uses What?

IP over InfiniBand (IPoIB)

  • Architecture Overview
  • IPoIB Packet Frame
  • Addressing
  • Using DHCP with IPoIB
  • Port Bonding
  • CM/UD Modes
  • Performance and Tuning

InfiniBand Usage of RDMA Principles

  • RDMA
  • RDMA Benefits
  • RDMA Interconnects
  • Transport Layer Work Request
  • Transport Layer Send Operation
  • Transport Layer RDMA Write

SCSI RDMA Protocol (SRP)

  • SRP Overview
  • Discovery Mechanism
  • SRP Login
  • I/O Operations Data Flow
  • Indirect and Partial Entry Table
  • SRP Initiator Installations
  • SRP Demon Overview

Routing Protocols (Algorithms)

  • Supported by OpenSM
  • What is Dead Lock / Credit Loop?
  • Introducing the Minimal Hop Algorithms
  • Introducing the Up Down Algorithms
    • Up Down Ranking
    • Up Down Connect Roots
  • Introducing Fat Tree Algorithms
  • Fat Tree Design Rules
  • Routing Optimization
  • Multicast Routing Optimization
  • UFM® Tara Optimization Overview
  • How to Check for Fabric Credit Loops

Day 2 Agenda

Performance Parameters

  • Server Tuning
  • HCA Performance Tuning
  • Linux Tuning Parameters
  • How to Preserve Tuning Settings
  • Improving IPV6 Performance
  • Power Management Tuning
  • Affinity Tuning
  • Tuning for Multiple Adapters
  • Tuning Multi-Threading IP Forwarding
  • Collecting Debug Information

Performance Test Process

  • Performance Comparison
  • Introducing mlnxperftuner Utility
  • Introducing Performance Utilities

InfiniBand Cluster Topology Validation

  • How to Review Action Flows
  • Comparison Workflow Main Stages
  • Cluster Topology Validation
  • Building a Topology File

Fabric Troubleshooting

  • Troubleshooting Congestion

Cluster Bring Up

  • Cluster Planning
  • Physical Installation
  • Topology Matching
  • Fabric Cleanup
  • SM Optimization

InfiniBand and VMware

  • Introduction
  • ESX Server Architecture
  • VMware & Client Infrastructure
  • ESX Installation
  • Configuring the VM
  • IPoIB and SRP Identification on VIC
  • Using Snapshots with IPoIB and SRP

Q&A Wrap Up

Get assistance from
Mellanox Academy

Call us: +1-408-419-0461
Email us:

Contact Form
* Please note all fields required.

* First Name:
* Last Name:
* Email:
* Comment:
* What is 2+5?