Mellanox Technologies =============================================================================== Mellanox OFED for Linux Version MLNX_OFED-1.5.3-3.0.0 Last Modified on December 8, 2011 =============================================================================== =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Contents of Mellanox OFED for Linux 3. Supported Platforms and Operating Systems 4. Supported HCAs 5. Switch and Gateway Systems Used for Testing 6. Compatibility 7. Change log 8. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of Mellanox OFED for Linux, version 1.5.3. This software package is based upon the Open Fabrics Enterprise Distribution (OFED) version 1.5.3.2 (see http://www.openfabrics.org) and supports all adapters supported by OFED (HCAs and 10GigE NICs). This release was tested with only Mellanox Adapters (HCAs and 10GigE NICs). Mellanox OFED for Linux supports VPI (Virtual Protocol IO): IB, Eth and RoCE Mellanox OFED for Linux is presented as ISO images, one for each supported operating system. See Section 6 below. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. =============================================================================== 2. Contents of Mellanox OFED for Linux =============================================================================== Mellanox OFED for Linux software contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, qib) - iWARP RNIC driver (cxgb3, nes) - 10/40GigE NIC driver (mlx4_en) - core with RoCE support - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and RDS Note: qib, cxgb3, nes and mthca were not tested in MLNX_OFED_LINUX-1.5.3 - Additional kernel modules: - EoIB - OpenFabrics utilities: - ib-bonding: Bonding driver for IPoIB interface - OpenSM: IB Subnet Manager with Mellanox proprietary Adaptive Routing - Diagnostic tools - Performance tests - MPI: - OSU MPI (MVAPICH 1.2.0) stack supporting the InfiniBand interface and RoCE - Open MPI stack 1.4.1 supporting the InfiniBand interface and RoCE - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) - Extra packages: - ibutils2 - ibdump - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) except for OpenSM, ibutils2, and ibdump - Documentation Notes: - See release notes for each package in the docs/release_notes directory. - SDP is at beta level =============================================================================== 3. Supported Platforms and Operating Systems =============================================================================== Summary of changes in the list of supported OSes: Added support for: - RHEL6.1 - Citrix XenServer Host 5.6.100-46766p Removed support for: - RHEL5.4 - SLES10 SP3 RHEL5.5 (MLNX_OFED_LINUX-1.5.3-rhel5.5.iso) i686: 2.6.18-194.el5 2.6.18-194.el5PAE x86_64: 2.6.18-194.el5 ppc64: 2.6.18-194.el5 RHEL5.6 (MLNX_OFED_LINUX-1.5.3-rhel5.6.iso) i686: 2.6.18-238.el5 2.6.18-238.el5PAE x86_64: 2.6.18-238.el5 ppc64: 2.6.18-238.el5 RHEL5.7 (MLNX_OFED_LINUX-1.5.3-rhel5.7.iso) i686: 2.6.18-274.el5 2.6.18-274.el5PAE x86_64: 2.6.18-274.el5 ppc64: 2.6.18-274.el5 RHEL5.8 (MLNX_OFED_LINUX-1.5.3-rhel5.8.iso) i686: 2.6.18-308.el5 2.6.18-308.el5PAE x86_64: 2.6.18-308.el5 RHEL6.0 (MLNX_OFED_LINUX-1.5.3-rhel6.iso) i686: 2.6.32-71.el6.i686 x86_64: 2.6.32-71.el6.x86_64 ppc64: 2.6.32-71.el6.ppc64 RHEL6.1 (MLNX_OFED_LINUX-1.5.3-rhel6.1.iso): i686: 2.6.32-131.0.15.el6.i686 x86_64: 2.6.32-131.0.15.el6.x86_64 RHEL6.2 (MLNX_OFED_LINUX-1.5.3-rhel6.2.iso): i686: 2.6.32-220.el6.i686 x86_64: 2.6.32-220.el6.x86_64 SLES10 SP4 (MLNX_OFED_LINUX-1.5.3-sles10sp4.iso): i686: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-bigsmp 2.6.16.60-0.85.1-default x86_64: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-default ppc64: 2.6.16.60-0.85.1-ppc64 SLES11 (MLNX_OFED_LINUX-1.5.3-sles11.iso): i686: 2.6.27.19-5-default 2.6.27.19-5-pae x86_64: 2.6.27.19-5-default ppc64: 2.6.27.19-5-default SLES11 SP1 (MLNX_OFED_LINUX-1.5.3-sles11sp1.iso): i686: 2.6.32.12-0.7-default 2.6.32.12-0.7-pae x86_64: 2.6.32.12-0.7-default 2.6.32.36-0.5-default ppc64: 2.6.32.12-0.7-ppc64 SLES11 SP2 (MLNX_OFED_LINUX-1.5.3-sles11sp2.iso): i686: 3.0.13-0.27-default 3.0.13-0.27-pae 3.0.13-0.27-xen x86_64: 3.0.13-0.27-default 3.0.13-0.27-xen OEL 6.1 (MLNX_OFED_LINUX-1.5.3-oel6.1.iso): x86_64: 2.6.32-100.34.1.el6uek.x86_64 Citrix XenServer Host 5.6.100-46766p i686: 2.6.32.12-0.7.1.xs5.6.100.323.170596xen Citrix XenServer Host 6.0.0-50762p i686: 2.6.32.12-0.7.1.xs6.0.0.529.170661xen NOTE: Citrix XenServer is at alpha level. To receive the XenServer driver, please contact Mellanox support (support@mellanox.com). NOTE: If you wish to install OFED on a different kernel, you need to create a new ISO image, using mlnx_add_kernel_support.sh script. See the MLNX_OFED User Guide for instructions. =============================================================================== 4. Supported HCAs =============================================================================== Mellanox supports the following adapters with MLNX_OFED_LINUX 1.5.3: - Mellanox Technologies HCAs (SDR and DDR Modes are Supported): - ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000) - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000) - ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1000) - ConnectX-3 (fw-4099 Rev 2.10.0) Note: To receive ConnectX-3 firmware, please contact your Mellanox representative. For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download =============================================================================== 5. Switch and Gateway Systems Used for Testing =============================================================================== This release was tested with switches and gateways provided by the following companies: - Arastra - Cisco - Dell - Flextronics - Fujitsu - HP - IBM - Juniper - Mellanox Technologies - Oracle - Extreme networks - Brocade =============================================================================== 6. Compatibility =============================================================================== - Linux vs. VMware Tested IPoIB connectivity (ping, Arp, Iperf) - Red Hat Enterprise Linux Server release 6.0 x64, w/w ConnectX(R) IB QDR, PCIe 2.0 5GT/s - VMWare Esx4.0 w/w ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s - SRP target with VMWare initiator - Linux vs. Windows Tested IPoIB, Verbs and Eth (ping,netprf, ib_read/write*, opensm handovers) - Red Hat Enterprise Linux Server release 6.0 x64 w/w ConnectX-2 (Tavor compatibility mode) and ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE as Eth - Windows 2008 R2 x64 runing WinOF 2.0, MLNX_VPI 2.3.0 w/w ConnectX VPI PCIe 2.0 5GT; IB QDR / 10GigE ConnectX EN 10GigE, PCIe 2.0 5GT/s - Linux vs. FabricIT(TM) EFM Tested IPoIB, Verbs and OpenSM priority handover - SLES 11 x64 w/w ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE, ConnectX VPI - 10GigE / IB QDR - IS5030 (Shark GT) w/w FabricIT EFM version 1.1.2500 - Linux vs. FabricIT(TM) BXM - MBX5020 (BrigeX) w/w FabricIT BXM version 2.1.1000 - VMA 4.5.12 / 5.5.10 - FCA 2.1 - UFM 3.5 - VSA 2.0 =============================================================================== 7. Change log =============================================================================== MLNX_OFED_LINUX-1.5.3-3.0.0 Main Changes (from v1.5.3-1.0.0): ------------------------------------------------------------------------------- - Added extended link capabilities support (FDR/FDR10) for ConnectX-3 - Added SLES11SP2 support - Added RHEL5.8 support - Added RHEL6.2 support - Added RHEL5.7 support - Added OEL6.1 support - Added support for kernel 2.6.3[7-9] and 3.0 - Added kernel module package (KMP) support - Added iser support for RHEL 5.7 - EoIB new features: o Para Virtualization support o The following features are enabled only while working with FabricIT-BXM v2.1.1000 or newer: o Bridge-level LAG support o PKEY discovery support o ALL VLAN support - ibdump: Added ConnectX®-3 support - mstflint: Added ConnectX®-3 support - Removed Fiber Channel support - Removed mlx4_accl module Changelog: ---------- core: - Changed link_layer string to match mainline kernel - Fix NULL pointer dereference - Added extended link capabilities support (FDR/FDR10) - Added fixes for masked atomic in kernel level RDMA/cm: - Destroy idr as part of the cleanup in the error flow mlx4: - Added warnings in case of command timeouts - Reduced the waiting period on software reset to 500 mili seconds - Updateed version to mlnx_ofed-1.5.3 - Fixed setting of port mask in multifunction mode - Added sync_qp to mlx4_dev capabilities - Added sanity check to prevent posting of empty SR - Fixed the attribute sq_sig_type value in query qp - Fixed cqe 64 support - Enabled A0 steering mode - Limited the min profile numbers - Fixed an error message in /var/log/messages (bug #119232) mlx4_ib: - Increased ABI version for 64 byte CQEs - RoCE: Fixed gid table population when using bonding over vlan mlx4_en: - Updating version to 1.5.7 - Rename mac_steer modparam to high_rate_steer - Unregister mac after releasing rss_steer - Fix a bug with QP release - Fix a race when setting a multicast list. - Fix endianness with blue flame support - Fix WOL handlers always look at port2 capability bit - Added 16 TX queues patches to 2.6.39 and 3.0 - Added hardware features flags in kernels 2.6.39 and up - Added MLX4_CMD_QUERY_IF_STAT_wrapper for flex10 - Added MLX4_CMD_SET_NODE_wrapper as NOOP for flex10 - Added support for fast packet drop - Enabled sys tune only if APM is configured in kernels 3.0 and up - Fixed bad flow for illegal mac - Fixed cq moderation timer - Fixed XenServer backport - Fixed HP mezz initialization error - Stopped calculating fcs in skb lenght - Removed flag allocation from memtrack - Set default MTU to dev->mtu - Enabled the use of GRO for PPC - Fixed linux-3.0 compilation issue with smp_lock.h - Modified swapping XAUI and XFI 10G speed report according to the PRM - Canceled bad patches disabling LRO - Added 40GigE speed report in ethtool - Fixed the crash that occured on initial port configuration (bug #3216) - Removed acceleration support - Disabled default mac steer param - Fixed the vlans that did not work on one port (bug #1694) - Fixed vlan 0 panic - Fixed uninitialized ip_summed at rx - Cleared ioremap allocation at en_main.c - Fixed UDP traffic issue related to ConnectX-3 - Fixed broken iboe counters addition to mlx4_en's mlx4_vnic: - Fixed printf exception due to wrong formatting - Added support in all vlan feature - Added missing ib_dma_unmap_single - Added missing spinlock to vnic_mcast_reattach() - Fixed dma unmap in case of an error in vnic_dma_map_tx() - Fixed error handling in vnic_unmap_and_replace_rx - Fixed IPv6 TCP traffic for kernels newer than 2.6.24 (FM #121735) - Fixed TCP/UDP header calculation in get skb/frag hdr - Fixed the error behavior of vnic_alloc_frag() - Fixed the mgid construction on ppc machines (RM #3450) - Removed unused field static_alloc from struct fip_ring - Removed lower case converting of vnic name in get_vnic() - Replaced all the calls to "dma_sync_single_range" with "ib_dma_sync_single" - Replaced the use of dma map in ib_dma map - Set ip_hdr in vnic_get_frag_header() (FM #123775, RM #3441) - Increased unicast solicit at FIP_GW_MCAST_RCVD - iLAG - Updated the saved gateway address when getting adv packet - Fixed ctrl iport address update (FM #120905) - Addeded support for iLAG feature - Added support for power cycle id - Decreased stack usage in fip_discover_rx_packet_bh() - Fixed LAG functions to work with child vNics (FM #120576) - Bug fixes mthca: - Fixed the attribute sq_sig_type value in query qp ipoib: - Don't release resources if the add_one failed - Adaptive mode, no assumption according the TCP traffic. - Set default TX/RX buffer size to 512 - Disabled lro whenever 4k MTU is enabled - Enabled by default lro and sync between ethtool and module param - Fine-tuned adaptive parameters - Added support for ethtool in -g option. That option shows the ring size of Rx and Tx. - Enabled the usage of the exact value for rx_coalesced_frames in ethtool. The value matches the value received by the user via the ethtool. - Added adaptive moderation algorithm for better latency ib/iser: - Work-around for XenServer junk dma issue srp: - Don't release resources if the add_one failed - libmlx4: - Avoid bad packets with RoCE/UD/VLAN - Added Core Direct support for ConnecX-3 HCAs - Added sanity check for posting a SR without any s/g entries to a RAW QP New packages: - Added MXM: mxm-1.0.601-1 - Added FCA: fca-2.1.12028-1 - Added KNEM: knem-0.9.7-0 Updated packages: - libibverbs: 1.1.4-1.26.ga73201a - libmlx4: 1.0.1-1.23.g49e4182 - mstflint: 1.4-1.18.g1adcfbf (Fixed port status report) - MFT: mft-2.7.1-7 - opensm: 3.3.9.MLNX_20111006_e52d5fc - perftest: 1.3.0-0.56.gd6f6093 - dapl: 2.0.34 - dump_pr: 1.0-0.1.g79715a9 - ibdump: 1.0.6-1 - ibutils: 1.5.7-0.1.g05a9d1a - ibutils2: 2.0-0.34.g9d3133a - infiniband-diags: 1.5.8.MLNX_20110906 - librdmacm: 1.0.15 - libibmad: 1.3.7.MLNX_20110814 - libibumad: 1.3.7.MLNX_20110814 - libmqe: 0.3.0-4.9.g973226d - libmverbs: 0.1.0-3.10.g3e83f31 MLNX_OFED_LINUX-1.5.3-1.0.0 Main Changes (from v1.5.2-2.1.0): ------------------------------------------------------------------------------- - MLNX_OFED_LINUX-1.5.3-1.0.0 is based on OFED-1.5.3.1 - Added RHEL6.1 support - Changes in /etc/infiniband/openib.conf. See MLNX_OFED_configuration_files.txt - Improved RAW Ethernet QP support (requires ConnectX/ConnectX2 FW >= 2.8.0600) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - KMP support by ofa_kernel package - RAW QP API changes - Added mlx4_accl Socket Acceleration package (relevant for mlx4_en and IPoIB) - mlx4: add support for 64 byte CQEs (ConnectX-3 only) - mlx4: enable raw eth QPs to work with inline and blueflame - mlx4_en: new steering model - mlx4_en: using GRO for 2.6.32 - mlx4_en: Ethernet NCSI support - ipoib: Add acceleration support - ipoib: enable LRO by default - ipoib: bug fixes (see ipoib_release_notes.txt) - sdp: ipv6 support - mlx4_vnic: Add Para Virtualization (PV) support ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1 ---------------------------------------------------- (http://openfabrics.org/downloads/OFED/ofed-1.5.3/OFED-1.5.3.1.tgz) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - Packages included in MLNX_OFED-1.5.3: - ibdump-1.0.5.4 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.7.0-20 (proprietary) - Firmware Tools - ar_mgr (proprietary) - Adaptive Routing Manager - cc_mgr (proprietary) - Congestion Control Manager - dump_pr (proprietary) - Dump PathRecord Plugin - mlx4_accl (proprietary) - Socket Acceleration - Core-Direct libraries - libmverbs - libmqe - VNIC support: - mlx4_vnic (EoIB) - Modules not included in MLNX_OFED-1.5.3 - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.1.0 Main Changes (from v1.5.2-2.0.0): ------------------------------------------------------------- - ConnectX/2 Firmware v2.8.0600 - RHEL5.6 support - Fix openibd service priority on RHEL5.x ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.0.0 Main Changes (from v1.5.1): ------------------------------------------------------- - RoCE is at GA level in MLNX_OFED 1.5.2 - Added support for EoIB for Mellanox BridgeX - Added OpenSM Congestion control - Added 3-D Torus (Beta quality level) - Improved SDP zCopy stability - MLNX_OFED 1.5.2 uses ConnectX / ConnectX-2 Firmware v2.8.0000 - Added SLES11 SP1 support - Added RHEL6.0 support ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-2.0.0 Main Changes (from v1.5.2-1.0.6): -------------------------------------------------------------- - Updated packages - ibdump-1.0.5-4 - mft-2.6.2-8 - kernel-mft-2.6.2-8 - libsdp-1.1.106-0.5.gd48f6f9 - mstflint-1.4-0.8.gc569a81 - opensm-3.3.7_MLNX - perftest-1.3.0-0.37.gc269d2d - sdpnetstat-1.60-0.1.g4e0a557 - New features and Bug fixes - MLX4_IB/MLX4_EN Changed blue flame to use new kenrel API Fixed RoCE link state report Workaround firmware bug: wrong number reporting of blue flame registers. mlx4_en: Added Blue Flame support mlx4_en: Fixed handling VLAN operations when working under bond interfaces - SDP Added Inline and blueflame support Stability issues. For further information, see sdp_release_notes.txt. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-1.0.6 Main Changes (from v1.5.2-1.0.0): ------------------------------------------------------------- - New packages - dump_pr-1.0-0.5.g5ad6d46 SM plugin for dumping pass record information for 3D Torus support in Open MPI - Updated packages - ibutils-1.5.6 - ibutils2-2.0-0.12.g1d281ad - kernel-mft-2.6.2-6 - infinipath-psm-1.13-1 - libibmad-1.3.6 - libibumad-1.3.6 - libsdp-1.1.105-0.4.g1b9b996 - openmpi-1.4.2-1.MLNX - opensm-3.3.7_MLNX-0.1.gb85eaf6 - perftest-1.3.0-0.36.g5e5630c - Bug fixes - mlx4: Fixed port speed in IBoE - mlx4: Added kernel blue flame support - mlx4: Fixed IBoE link state - mlx4_vnic: Changed tx/rx rings num description to #cpus (previously #"cores") - mlx4_vnic_info: Added capability to check vid when reload flag is used - mlx4_fc: Added support for SLES11 SP1 - cxgb3: Added 7.11 firmware - IPoIB: Fixed IPoIB to correspond to ethtool definitions - sdp: Added support to accept AF_INET_SDP in address supplied to connect() - sdp: Added ability to set a maximum memory usage for the entire module - sdp: Fixed compilation problems on 32 bit hosts - sdp: Do not compile in debug mode when not asked - sdp: Improved recovery from errors - sdp: Enhanced statistics in /proc/sdpstats - sdp: added debugfs: - sdpprf was moved from /proc to debugfs/sdp - debugfs/ - Socket history - openibd: Fixed security issues ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.2-1.0.0 Main Changes (from v1.5.1): -------------------------------------------------------- General Changes: - /etc/modprobe.conf deprecated - ib_ipoib use /etc/modprobe.d/ib_ipoib.conf - ib_sdp use /etc/modprobe.d/ib_sdp.conf - mlx4_en use /etc/modprobe.d/mlx4_en.conf - Added RAW Ethernet QP support in libibverbs - Added multicast support in perftest - Added new package: ibacm. See ibacm_release_notes.txt - Updated packages: - libibverbs to rev 1.1.4-0.14.gb6c138b - libehca to rev 1.2.2-0.1.g69e1a88 - libmlx4 to rev 1.0-0.13.g4e5c43f - libnes to rev 1.1.0 - librdmacm to rev 1.0.13 - dapl to rev 2.0.30 - compat-depl to rev 1.2.19 - libsdp to rev 1.1.103 - mstflint to rev 1.4-0.6.g63179f4 - perftest to rev 1.3.0-0.27 - rds-tools to rev 2.0.4 - Management: - opensm to rev 3.3.7 - libibmad to rev 1.3.6 - libibumad to rev 1.3.6 - infiniband-diags to rev 1.5.7 - MPI: - openmpi to rev 1.4.2-1 - mpitests to rev 3.2-923 - SDP: - SDP is at Beta level for MLNX_OFED 1.5.2. - Improved copy stability - Bug fixes - NFS-RDMA: - Removed from MLNX_OFED 1.5.2 ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.2-2.0.0 and OFED 1.5.2 (http://openfabrics.org/downloads/OFED/ofed-1.5.2/OFED-1.5.2.tgz) ----------------------------------------------------------------- - Packages included in MLNX_OFED-1.5.2-2.0.0: - ibdump-1.0.5-4 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.6.2 (proprietary) - cc_mgr (proprietary) - dump_pr (proprietary) - FC support: - mlx4_fc (FCoE) - dcbx-1.0.0 - libhbalinux-1.0.7 - VNIC support: - mlx4_vnic (EoIB) - Packages that are different: - OpenSM - Includes Adaptive Routing (proprietary code) - Modules not included in MLNX_OFED-1.5.2 - iSER - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.2-1.0.6 and OFED 1.5.2 (http://openfabrics.org/downloads/OFED/ofed-1.5.2/OFED-1.5.2.tgz) ------------------------------------------------------------------ - Packages included in MLNX_OFED-1.5.2: - ibdump-1.0.3 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.6.2 (proprietary) - cc_mgr (proprietary) - dump_pr (proprietary) - FC support: - mlx4_fc (FCoE) - dcbx-1.0.0 - libhbalinux-1.0.7 - VNIC support: - mlx4_vnic (EoIB) - Packages that are different: - OpenSM - Includes Adaptive Routing (proprietary code) - Modules not included in MLNX_OFED-1.5.2 - iSER - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. ------------------------------------------------------------------------------- MLNX_OFED_LINUX-1.5.1 Main Changes (from v1.4) ----------------------------------------------- - General Changes - Kernel code is based on 2.6.30 - Added RDMA over Converged Ethernet (RoCE) support. - Added enhanced atomic operations to ConnectX (kernel only) - Fixed IPv6 support and IPv4 routing corner cases for RDMA CM - libraries location - all userspace libraries can be downloaded from http://www.openfabrics.org/downloads/ See BUILD_ID for exact location - Added a new (RO) file called link_layer to sysfs. The file is located under /sys/class/infiniband//ports// - SDP - SDP is at Beta level for MLNX_OFED 1.5.1. - Improved copy stability - PPC support - Bug fixes - uDAPL - Updated DAPL to rev 2.0.27 - New UCM provider (ofa-v2-mlx4_0-1) with IB UD based CM per process. More scalable than rdma_cm (cma) and socket cm (scm). - Bug fixes - librdmacm - Updated librdmacm to rev 1.0.11 - librdmacm/mckey: reports multicast events to the user - librdmacm/cmatose: supports IPv6 - librdmacm/mckey: enforces local binding for unmapped multicast address - Updated documentation on using unmapped MGIDs - perftest - Renamed tests: - ib_rdma_bw -> rdma_bw - ib_rdma_lat -> rdma_lat - perftest: Added Multicast support in ib_send_bw and ib_send_lat - Management - OpenSM - Support for Mesh Analysis for LASH routing algorithm - Reloadable OpenSM configuration (preliminary implementation) - Routing paths sorted balancing (for UpDown and MinHops) - Weighted LID matrices calculation (for UpDown, MinHop and DOR) - I/O nodes connectivity (for FatTree) - New Diagnostic Tools - ibdump - dumps InfiniBand traffic that flows to/from Mellanox Technologies ConnectX/ConnectX-2 (adapters) InfiniBand ports - ibutils2 - a re-implementation of the ibutils package with performance and scalability improvements - MPI - OSU MVAPICH 1.2.0 - Open MPI 1.4.1 - MPI tests 3.2 - NFS-RDMA - Support for RHEL5.4, SLES11 - NFS-RDMA is at beta level =============================================================================== 8. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release. - When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version. - When upgrading from the previous OFED installation to this release, you must reboot after driver installation since the kernel module ipoib_helper does not unload. - "openibd stop" can sometime fail with the error: Unloading ib_cm [FAILED] ERROR: Module ib_cm is in use by ib_ipoib Workaround: run "openibd stop" again. - Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. - The Intel compiler on RH5.4/5.5/5.6 requires that the compat-libstdc++-33-3.2.3-61 RPM be installed on your host. - When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even if IPoIB is set to connected mode). This is because there is a bug in older kernels which causes a kernel panic. - OpenMPI and Mvapich with the PGI compiler are not supported on SLES11 32-bit, SLES11 SP1 32-bit, RHEL6.0 and RHEL6.1. - On SLES11, and in case uninstall is failing, check the error log and remove the remaining RPMs manually using 'rpm -e '. - On SLES11, set the allow_unsupported_modules parameter to 1 in the file /etc/modprobe.d/unsupported-modules. Without this setting, the modules will not load. - When working under bonding, unloading the mlx4_en driver may cause unexpected behavior by the bonding driver. - Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring. (https://bugs.openfabrics.org/show_bug.cgi?id=1990) - MLNX_OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox proprietary packages. For further detail please see section 4. Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1. - The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in kernel or userspace. - mlx4_en with bonding on SLES11 SP1: Do not unload mlx4_en (modprobe -r mlx4_en) while interfaces are enslaved. Unloading mlx4_en kernel module while interfaces are enslaved can lead to unexpected behavior. - Configuring Bonding over IPoIB interfaces while QIB driver is loaded can lead to kernel panic - SDP does not work with EoIB interfaces. - When creating port configuration files using YAST on SLES systems, not all (or no) mlx4_en network interfaces are recognized by the tool. Workaround: The interface configuration needs to be added as a new interface and the MAC address needs to be set manually. - cxgb3i kernel module fails to be loaded after MLNX_OFED installation: # service iscsid start Starting iSCSI daemon: FATAL: Error inserting cxgb3i (/lib/modules/2.6.18-238.el5/kernel/drivers/scsi/cxgb3i/cxgb3i.ko): Unknown symbol in module, or unknown parameter (see dmesg) cxgb3i: disagrees about version of symbol cxgb3_register_client cxgb3i: Unknown symbol cxgb3_register_client cxgb3i: disagrees about version of symbol cxgb3_alloc_atid cxgb3i: Unknown symbol cxgb3_alloc_atid ... - On Citrix XenServer Host 5.6.100-39215p mlx4_core driver may failed to be loaded with the following error: mlx4_core 0000:08:00.0: NOP command failed to generate MSI-X interrupt IRQ 1217). mlx4_core 0000:08:00.0: Trying again without MSI-X. mlx4_core 0000:08:00.0: SW2HW_EQ failed (-110) mlx4_core 0000:08:00.0: Failed to initialize event queue table, aborting. The problem isn't seen on Xen Server 5.6 sp2 (exact version 5.6.100-46766p kernel 2.6.32.12-0.7.1.xs5.6.100.323.170596xen). - On Citrix XenServer 5.6 the iser initiator may get from the dom0 kernel "bad" dma addressees for the iscsi login request/response buffer which if used result in data-corruption during login. Limiting the buffer used for login to 4k seems to provide work around for the problem. If the driver would observe such "bad" dma address, it will intentionally fail the login attempt, such that a similar to the following message will be printed to the system log: iser: iser_create_ib_conn_res:login buf cc5c0000 size 16460 kvirt/dma - req cc5c0000/77d7000 \ resp cc5c2000/77d9000 dma err 0/0 bad 1/1 iser: iser_create_ib_conn_res:unable to alloc mem or create resource, err -12 iser: iscsi_iser_ep_poll:ib conn c9f864bc rc = -1 -- note that the size is 16460 where with the workaround applied it will be 4096 when things are okay, the following should be seen iser_create_ib_conn_res:login buf deb3b000 size 4096 kvirt/dma - req deb3b000/617878000 \ resp deb3b800/617878800 dma err 0/0 bad 0/0 - EoIB (mlx4_vnic) is not supported on SLES10 SP4 - To forward TCP traffic LRO should be disabled (Relevant for IPoIB, EoIB and MLX4_EN) To disable LRO EoIB: modprobe mlx4_vnic lro_num=0 IPoIB: Set "options ib_ipoib lro=0" in /etc/modprobe.d/ib_ipoib.conf MLX4_EN: Use ethtool: ethtool --offload lro off - mlx4_vnic kernel module (EoIB) is not loaded by default. Workaround: Set MLX4_VNIC_LOAD=yes in /etc/infiniband/openib.conf to load mlx4_vnic upon boot. - EoIB: to support macvtap use docs/errata/mlx4_vnicd instead of /sbin/mlx4_vnicd - Running more than 2 instances of openSM in the same IB subnet has not been qualified. - An IPoIB cannot be a full member of a Multicast Group (MC) if the "umcast" flag (/sys/class/net/$dev/umcast) is set to 1, or if that IPoIB has previously joined the MC group as a "send only" member - The introduction of 64 bit CQE support in ConnectX3 devices has increased the ABI version of the kernel. Thus upgrading the included libmlx4 library accordingly. Hence, old libraries/executables that were built using the older headers will fail to load. Workaround: Rebuild the binaries using the new headers. - Ethernet and RoCE cannot work on the same port of ConnectX-2 device. Note: To receive ConnectX-2 firmware that fix this issue, please contact your Mellanox representative. RoCE ONLY: - PowerPC and ia64 architecture currently are not supported. - x32 architectures were not tested. - SRP is currently not supported. - When running the ibdiagnet check nodes_info on the fabric, a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed. Workaround: Run ibdiagnet --skip nodes_info Note: See the release notes of each component for additional issues.