Mellanox Technologies =============================================================================== Mellanox OFED for Linux Version MLNX_OFED-1.5.3-1.0.0 Last Modified on July 13, 2011 =============================================================================== =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Contents of Mellanox OFED for Linux 3. Supported Platforms and Operating Systems 4. Supported HCAs 5. Switch and Gateway Systems Used for Testing 6. Compatibility 7. Change log 8. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of Mellanox OFED for Linux, version 1.5.3. This software package is based upon the Open Fabrics Enterprise Distribution (OFED) version 1.5.3.1 (see http://www.openfabrics.org) and supports all addapters supported by OFED (HCAs and 10GigE NICs). This release was tested with only Mellanox Adapters (HCAs and 10GigE NICs). Mellanox OFED for Linux supports VPI (Virtual Protocol IO): IB, Eth, RoCE and FCoE. Mellanox OFED for Linux is presented as ISO images, one for each supported operating system. See Section 6 below. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. =============================================================================== 2. Contents of Mellanox OFED for Linux =============================================================================== Mellanox OFED for Linux software contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, qib) - iWARP RNIC driver (cxgb3, nes) - 10GigE NIC driver (mlx4_en) - core with RoCE support - Upper Layer Protocols: IPoIB, SDP, SRP Initiator, SRP Target, RDS Note: qib, cxgb3, nes and mthca were not tested in MLNX_OFED_LINUX-1.5.3 - Additional kernel modules: - EoIB - FCoE - Socket Acceleration (mlx4_accl) - OpenFabrics utilities: - ib-bonding: Bonding driver for IPoIB interface - OpenSM: IB Subnet Manager with Mellanox proprietary Adaptive Routing - Diagnostic tools - Performance tests - MPI: - OSU MPI (MVAPICH 1.2.0) stack supporting the InfiniBand interface and RoCE - Open MPI stack 1.4.1 supporting the InfiniBand interface and RoCE - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) - Extra packages: - ibutils2 - ibdump - OpenFC: - mlx4_fc - libhbalinux - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) except for OpenSM, ibutils2, and ibdump - Documentation Notes: - See release notes for each package in the docs/release_notes directory. - SDP in OFED is at beta level - EoIB is at beta quality (We recommend using the GA version, MLNX_OFED_LINUX-1.5.2-2.1.0). - FCoE is at beta quality. - FCoE is not supported on PPC64 platforms. =============================================================================== 3. Supported Platforms and Operating Systems =============================================================================== Summary of changes in the list of supported OSes: Added support for: - RHEL6.1 - Citrix XenServer Host 5.6.100-39215p Removed support for: - RHEL5.4 - SLES10 SP3 1. RHEL5.5 (MLNX_OFED_LINUX-1.5.3-rhel5.5.iso) i686: 2.6.18-194.el5 2.6.18-194.el5PAE x86_64: 2.6.18-194.el5 ppc64: 2.6.18-194.el5 2. RHEL5.6 (MLNX_OFED_LINUX-1.5.3-rhel5.6.iso) i686: 2.6.18-238.el5 2.6.18-238.el5PAE x86_64: 2.6.18-238.el5 ppc64: 2.6.18-238.el5 3. RHEL6.0 (MLNX_OFED_LINUX-1.5.3-rhel6.iso) i686: 2.6.32-71.el6.i686 x86_64: 2.6.32-71.el6.x86_64 ppc64: 2.6.32-71.el6.ppc64 4. RHEL6.1 (MLNX_OFED_LINUX-1.5.3-rhel6.1.iso): i686: 2.6.32-131.0.15.el6.i686 x86_64: 2.6.32-131.0.15.el6.x86_64 5. SLES10 SP4 (MLNX_OFED_LINUX-1.5.3-sles10sp4.iso): i686: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-bigsmp 2.6.16.60-0.85.1-default x86_64: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-default ppc64: 2.6.16.60-0.85.1-ppc64 6. SLES11 (MLNX_OFED_LINUX-1.5.3-sles11.iso): i686: 2.6.27.19-5-default 2.6.27.19-5-pae x86_64: 2.6.27.19-5-default ppc64: 2.6.27.19-5-default 7. SLES11 SP1 (MLNX_OFED_LINUX-1.5.3-sles11sp1.iso): i686: 2.6.32.12-0.7-default 2.6.32.12-0.7-pae x86_64: 2.6.32.12-0.7-default 2.6.32.36-0.5-default ppc64: 2.6.32.12-0.7-ppc64 8. Citrix XenServer Host 5.6.100-39215p i686: 2.6.32.12-0.7.1.xs5.6.100.307.170586xen NOTE: If you wish to install OFED on a different kernel, you need to create a new ISO image, using mlnx_add_kernel_support.sh script. See the MLNX_OFED User Guide for instructions. =============================================================================== 4. Supported HCAs =============================================================================== Mellanox supports the following adapters with MLNX_OFED_LINUX 1.5.3: - Mellanox Technologies HCAs (SDR and DDR Modes are Supported): - ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000) - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000) - ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1000) - ConnectX-3 Ready Note: To receive ConnectX-3 firmware, please contact your Mellanox representative. For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download =============================================================================== 5. Switch and Gateway Systems Used for Testing =============================================================================== This release was tested with switches and gateways provided by the following companies: - Arastra - Cisco - Dell - Flextronics - Fujitsu - HP - IBM - Juniper - Mellanox Technologies - Oracle - Extreme networks - Brocade =============================================================================== 6. Compatibility =============================================================================== - Linux vs. VMware Tested IPoIB connectivity (ping, Arp, Iperf) - Red Hat Enterprise Linux Server release 6.0 x64, w/w ConnectX(R) IB QDR, PCIe 2.0 5GT/s - VMWare Esx4.0 w/w ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s - SRP targer with VMWare initiator - Linux vs. Windows Tested IPoIB, Verbs and Eth (ping,netprf, ib_read/write*, opensm handovers) - Red Hat Enterprise Linux Server release 6.0 x64 w/w ConnectX-2 (Tavor compatibility mode) and ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE as Eth - Windows 2003 x64 NT runing WinOF 2.0, MLNX_VPI 2.1.1 RC4 w/w ConnectX VPI PCIe 2.0 5GT; IB QDR / 10GigE ConnectX EN 10GigE, PCIe 2.0 5GT/s - Linux vs. FabricIT(TM) EFM Tested IPoIB, Verbs and OpenSM pririty handover - SLES 11 x64 w/w ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE, ConnectX VPI - 10GigE / IB QDR - IS5030 (Shark GT) w/w FabricIT EFM version 1.1.2500 - Linux vs. FabricIT(TM) BXM - MBX5020 (BrigeX) w/w FabricIT BXM version 2.0.0400 - VMA 4.5 / 5.0Beta - FCA 2.0 - UFM 3.3 / 3.4 - VSA 2.0 =============================================================================== 7. Change log =============================================================================== MLNX_OFED_LINUX-1.5.3-1.0.0 Main Changes (from v1.5.2-2.1.0): ------------------------------------------------------------------------------- - MLNX_OFED_LINUX-1.5.3-1.0.0 is based on OFED-1.5.3.1 - Changes in /etc/infiniband/openib.conf. See MLNX_OFED_configuration_files.txt - Improved RAW Ethernet QP support (requires ConnectX/ConnectX2 FW >= 2.8.0600) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - RAW QP API changes - Added mlx4_accl Socket Acceleration package (relevant for mlx4_en and IPoIB) - mlx4: add support for 64 byte CQEs (CX3 only) - mlx4: enable raw eth QPs to work with inline and blueflame - mlx4_en: new steering model - mlx4_en: using GRO for 2.6.32 - mlx4_en: Ethernet NCSI support - ipoib: Add acceleration support - ipoib: enable LRO by default - ipoib: bug fixes (see ipoib_release_notes.txt) - sdp: ipv6 support - mlx4_vnic EoIB is at beta level. We recommend using the GA version, MLNX_OFED_LINUX-1.5.2-2.1.0 MLNX_OFED_LINUX-1.5.2-2.1.0. - mlx4_vnic: Add PV support (Beta) ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1 ---------------------------------------------------- (http://openfabrics.org/downloads/OFED/ofed-1.5.3/OFED-1.5.3.1.tgz) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - Packages included in MLNX_OFED-1.5.3: - ibdump-1.0.5.4 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.7.0-20 (proprietary) - Firmware Tools - ar_mgr (proprietary) - Adaptive Routing Manager - cc_mgr (proprietary) - Congestion Control Manager - dump_pr (proprietary) - Dump PathRecord Plugin - mlx4_accl (proprietary) - Socket Acceleration - Core-Direct libraries - libmverbs - libmqe - FC support: - mlx4_fc (FCoE) - libhbalinux-1.0.7 - VNIC support: - mlx4_vnic (EoIB) - Modules not included in MLNX_OFED-1.5.3 - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.1.0 Main Changes (from v1.5.2-2.0.0): ------------------------------------------------------------- - ConnectX/2 Firmware v2.8.0600 - RHEL5.6 support - Fix openibd service priority on RHEL5.x ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.0.0 Main Changes (from v1.5.1): ------------------------------------------------------- - RoCE is at GA level in MLNX_OFED 1.5.2 - Added support for EoIB for Mellanox BridgeX - Added OpenSM Congestion control - Added 3-D Torus (Beta quality level) - Improved SDP zCopy stability - MLNX_OFED 1.5.2 uses ConnectX/2 Firmware v2.8.0000 - Added SLES11 SP1 support - Added RHEL6.0 support ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.0.0 Main Changes (from v1.5.2-1.0.6): -------------------------------------------------------------- - Updated packages - ibdump-1.0.5-4 - mft-2.6.2-8 - kernel-mft-2.6.2-8 - libsdp-1.1.106-0.5.gd48f6f9 - mstflint-1.4-0.8.gc569a81 - opensm-3.3.7_MLNX - perftest-1.3.0-0.37.gc269d2d - sdpnetstat-1.60-0.1.g4e0a557 - New features and Bug fixes - MLX4_IB/MLX4_EN Changed blue flame to use new kenrel API Fixed RoCE link state report Workaround firmware bug: wrong number reporting of blue flame registers. mlx4_en: Added Blue Flame support mlx4_en: Fixed handling VLAN operations when working under bond interfaces - SDP Added Inline and blueflame support Stability issues. For further information, see sdp_release_notes.txt. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-1.0.6 Main Changes (from v1.5.2-1.0.0): ------------------------------------------------------------- - New packages - dump_pr-1.0-0.5.g5ad6d46 SM plugin for dumping pass record information for 3D Torus support in Open MPI - Updated packages - ibutils-1.5.6 - ibutils2-2.0-0.12.g1d281ad - kernel-mft-2.6.2-6 - infinipath-psm-1.13-1 - libibmad-1.3.6 - libibumad-1.3.6 - libsdp-1.1.105-0.4.g1b9b996 - openmpi-1.4.2-1.MLNX - opensm-3.3.7_MLNX-0.1.gb85eaf6 - perftest-1.3.0-0.36.g5e5630c - Bug fixes - mlx4: Fixed port speed in IBoE - mlx4: Added kernel blue flame support - mlx4: Fixed IBoE link state - mlx4_vnic: Changed tx/rx rings num description to #cpus (previously #"cores") - mlx4_vnic_info: Added capability to check vid when reload flag is used - mlx4_fc: Added support for SLES11 SP1 - cxgb3: Added 7.11 firmware - IPoIB: Fixed IPoIB to correspond to ethtool definitions - sdp: Added support to accept AF_INET_SDP in address supplied to connect() - sdp: Added ability to set a maximum memory usage for the entire module - sdp: Fixed compilation problems on 32 bit hosts - sdp: Do not compile in debug mode when not asked - sdp: Improved recovery from errors - sdp: Enhanced statistics in /proc/sdpstats - sdp: added debugfs: - sdpprf was moved from /proc to debugfs/sdp - debugfs/ - Socket history - openibd: Fixed security issues ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-1.0.0 Main Changes (from v1.5.1): -------------------------------------------------------- General Changes: - /etc/modprobe.conf deprecated - ib_ipoib use /etc/modprobe.d/ib_ipoib.conf - ib_sdp use /etc/modprobe.d/ib_sdp.conf - mlx4_en use /etc/modprobe.d/mlx4_en.conf - Added RAW Ethernet QP support in libibverbs - Added multicast support in perftest - Added new package: ibacm. See ibacm_release_notes.txt - Updated packages: - libibverbs to rev 1.1.4-0.14.gb6c138b - libehca to rev 1.2.2-0.1.g69e1a88 - libmlx4 to rev 1.0-0.13.g4e5c43f - libnes to rev 1.1.0 - librdmacm to rev 1.0.13 - dapl to rev 2.0.30 - compat-depl to rev 1.2.19 - libsdp to rev 1.1.103 - mstflint to rev 1.4-0.6.g63179f4 - perftest to rev 1.3.0-0.27 - rds-tools to rev 2.0.4 - Management: - opensm to rev 3.3.7 - libibmad to rev 1.3.6 - libibumad to rev 1.3.6 - infiniband-diags to rev 1.5.7 - MPI: - openmpi to rev 1.4.2-1 - mpitests to rev 3.2-923 - SDP: - SDP is at Beta level for MLNX_OFED 1.5.2. - Improved copy stability - Bug fixes - NFS-RDMA: - Removed from MLNX_OFED 1.5.2 ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.2-2.0.0 and OFED 1.5.2 (http://openfabrics.org/downloads/OFED/ofed-1.5.2/OFED-1.5.2.tgz) ----------------------------------------------------------------- - Packages included in MLNX_OFED-1.5.2-2.0.0: - ibdump-1.0.5-4 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.6.2 (proprietary) - cc_mgr (proprietary) - dump_pr (proprietary) - FC support: - mlx4_fc (FCoE) - dcbx-1.0.0 - libhbalinux-1.0.7 - VNIC support: - mlx4_vnic (EoIB) - Packages that are different: - OpenSM - Includes Adaptive Routing (proprietary code) - Modules not included in MLNX_OFED-1.5.2 - iSER - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.2-1.0.6 and OFED 1.5.2 (http://openfabrics.org/downloads/OFED/ofed-1.5.2/OFED-1.5.2.tgz) ------------------------------------------------------------------ - Packages included in MLNX_OFED-1.5.2: - ibdump-1.0.3 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.6.2 (proprietary) - cc_mgr (proprietary) - dump_pr (proprietary) - FC support: - mlx4_fc (FCoE) - dcbx-1.0.0 - libhbalinux-1.0.7 - VNIC support: - mlx4_vnic (EoIB) - Packages that are different: - OpenSM - Includes Adaptive Routing (proprietary code) - Modules not included in MLNX_OFED-1.5.2 - iSER - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.1 Main Changes (from v1.4) ----------------------------------------------- - General Changes - Kernel code is based on 2.6.30 - Added RDMA over Converged Ethernet (RoCE) support. - Added enhanced atomic operations to ConnectX (kernel only) - Fixed IPv6 support and IPv4 routing corner cases for RDMA CM - libraries location - all userspace libraries can be downloaded from http://www.openfabrics.org/downloads/ See BUILD_ID for exact location - Added a new (RO) file called link_layer to sysfs. The file is located under /sys/class/infiniband//ports// - SDP - SDP is at Beta level for MLNX_OFED 1.5.1. - Improved copy stability - PPC support - Bug fixes - uDAPL - Updated DAPL to rev 2.0.27 - New UCM provider (ofa-v2-mlx4_0-1) with IB UD based CM per process. More scalable than rdma_cm (cma) and socket cm (scm). - Bug fixes - librdmacm - Updated librdmacm to rev 1.0.11 - librdmacm/mckey: reports multicast events to the user - librdmacm/cmatose: supports IPv6 - librdmacm/mckey: enforces local binding for unmapped multicast address - Updated documentation on using unmapped MGIDs - perftest - Renamed tests: - ib_rdma_bw -> rdma_bw - ib_rdma_lat -> rdma_lat - perftest: Added Multicast support in ib_send_bw and ib_send_lat - Management - OpenSM - Support for Mesh Analysis for LASH routing algorithm - Reloadable OpenSM configuration (preliminary implementation) - Routing paths sorted balancing (for UpDown and MinHops) - Weighted LID matrices calculation (for UpDown, MinHop and DOR) - I/O nodes connectivity (for FatTree) - New Diagnostic Tools - ibdump - dumps InfiniBand traffic that flows to/from Mellanox Technologies ConnectX/ConnectX-2 (adapters) InfiniBand ports - ibutils2 - a re-implementation of the ibutils package with performance and scalability improvements - MPI - OSU MVAPICH 1.2.0 - Open MPI 1.4.1 - MPI tests 3.2 - NFS-RDMA - Support for RHEL5.4, SLES11 - NFS-RDMA is at beta level =============================================================================== 8. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release. - When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version. - When upgrading from the previous OFED installation to this release, you must reboot after driver installation since the kernel module ipoib_helper does not unload. - "openibd stop" can sometime fail with the error: Unloading ib_cm [FAILED] ERROR: Module ib_cm is in use by ib_ipoib Workaround: run "openibd stop" again. - Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. - The Intel compiler on RH5.4/5.5/5.6 requires that the compat-libstdc++-33-3.2.3-61 RPM be installed on your host. - When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even if IPoIB is set to connected mode). This is because there is a bug in older kernels which causes a kernel panic. - OpenMPI and Mvapich with the PGI compiler are not supported on SLES11 32-bit, SLES11 SP1 32-bit, RHEL6.0 and RHEL6.1. - On SLES11, and in case uninstall is failing, check the error log and remove the remaining RPMs manually using 'rpm -e '. - On SLES11, set the allow_unsupported_modules parameter to 1 in the file /etc/modprobe.d/unsupported-modules. Without this setting, the modules will not load. - The SRP Target included in the Mellanox OFED package is supported on the following distributions only: - RHEL 5.4, 5.5, 5.6 - SLES10 sp4 - When working under bonding, unloading the mlx4_en driver may cause unexpected behavior by the bonding driver. - Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring. (https://bugs.openfabrics.org/show_bug.cgi?id=1990) - MLNX_OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox proprietary packages. For further detail please see section 4. Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1. - The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in kernel or userspace. - mlx4_en with bonding on SLES11 SP1: Do not unload mlx4_en (modprobe -r mlx4_en) while interfaces are enslaved. Unloading mlx4_en kernel module while interfaces are enslaved can lead to unexpected behavior. - Configuring Bonding over IPoIB interfaces while QIB driver is loaded can lead to kernel panic - SDP does not work with EoIB interfaces. - When creating port configuration files using YAST on SLES systems, not all (or no) mlx4_en network interfaces are recognized by the tool. Workaround: The interface configuration needs to be added as a new interface and the MAC address needs to be set manually. - EoIB (mlx4_vnic) is not supported on SLES10 SPx - cxgb3i kernel module fails to be loaded after MLNX_OFED installation: # service iscsid start Starting iSCSI daemon: FATAL: Error inserting cxgb3i (/lib/modules/2.6.18-238.el5/kernel/drivers/scsi/cxgb3i/cxgb3i.ko): Unknown symbol in module, or unknown parameter (see dmesg) cxgb3i: disagrees about version of symbol cxgb3_register_client cxgb3i: Unknown symbol cxgb3_register_client cxgb3i: disagrees about version of symbol cxgb3_alloc_atid cxgb3i: Unknown symbol cxgb3_alloc_atid ... - On Citrix XenServer Host 5.6.100-39215p mlx4_core driver may failed to be loaded with the following error: mlx4_core 0000:08:00.0: NOP command failed to generate MSI-X interrupt IRQ 1217). mlx4_core 0000:08:00.0: Trying again without MSI-X. mlx4_core 0000:08:00.0: SW2HW_EQ failed (-110) mlx4_core 0000:08:00.0: Failed to initialize event queue table, aborting. The fix is expected in MLNX_OFED_LINUX-1.5.3-2.0.0 - To forward TCP traffic LRO should be disabled (Relevant for IPoIB, EoIB and MLX4_EN) To disable LRO EoIB: modprobe mlx4_vnic lro_num=0 IPoIB: Set "options ib_ipoib lro=0" in /etc/modprobe.d/ib_ipoib.conf MLX4_EN: Use ethtool: ethtool --offload lro off - mlx4_vnic kernel module (EoIB) is not loaded by default. Workaround: Set MLX4_VNIC_LOAD=yes in /etc/infiniband/openib.conf to load mlx4_vnic upon boot. - Broadcast and UDP traffic may be lost when running over EoIB RoCE ONLY: - PowerPC and ia64 architecture currently are not supported. - x32 architectures were not tested. - SRP is currently not supported. - UD QPs that send traffic with VLAN tags (e.g., 802.1q tagged frames) currently are not supported. Note: See the release notes of each component for additional issues.