Mellanox Technologies =============================================================================== Mellanox OFED for Linux Version MLNX_OFED-1.5.3-3.1.0 Last Modified on January 10, 2013 =============================================================================== =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Changes in This Release 3. Contents of Mellanox OFED for Linux 4. Supported Platforms and Operating Systems 5. Software Requirements 6. Supported HCAs 7. Switch and Gateway Systems Used for Testing 8. Compatibility 9. Change Log History 10. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of Mellanox OFED for Linux, version 1.5.3. This software package is based upon the Open Fabrics Enterprise Distribution (OFED) version 1.5.3.2 (see http://www.openfabrics.org) and supports all adapters supported by OFED (HCAs and 10GigE NICs). This release was tested with only Mellanox Adapters (HCAs and 10GigE NICs). Mellanox OFED for Linux supports VPI (Virtual Protocol IO): IB, Eth and RoCE Mellanox OFED for Linux is presented as ISO images, one for each supported operating system. See Section 4 below. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. =============================================================================== 2. Changes in This Release =============================================================================== MLNX_OFED v1.5.3-3.1.0 provides the following changes and new features: MLNX_OFED_LINUX-1.5.3-3.1.0 Main Changes (from v1.5.3-3.0.0): ------------------------------------------------------------- mlx4_en: - Updated version to 1.5.8.3 - Added queue stop/start counters per ring - Enabled atomic counter usage to decide when queue is full - Removed keep alive support - Updated set_irq_affinity script - Enabled neighbour core selection for TX interrupts - Added effective MTU calculation at device initialization - Canceled TX interrupts shifting - Fixed QinQ parameters definition - Added the disable sys_tune backport option - Ran sysctl_perf_tuning on mlx4_en load - Added affinity scripts to package - Added parameter for default NUMA node - Prevented TX and RX interrupts for the same index to fall on the same IRQ - Enabled NAPI usage to handle TX completions - Changed the default TX moderation - Added Ethtool support for TX moderation - Moved to interrupts for TX completions - Added a check to look at firmware capability before reporting no QinQ - Added suppressing page allocation failure warnings - Added the option to consider headers when setting MTU - Enabled NUMA node to adapter node setting - Fixed mcast attach issues - Changed defaults for TX interrupt moderation - Moderated mcast attachments - Fixed RSS UDP base QP configuration - Disabled systune for XEN RH5.2-RH5.7 - Fixed ethtool displayed counters - Added rxhash support - Added recording RX queue for GRO packets - Added a fix for vlan id 0 and UP 0 - Fixed UDP RSS related settings libmlx4-1.0.2mlnx3: Fixed PPC machines support libmlx4: - Added an option to stall poll cq - SandyBridge issue - Added a check test for max_inline_data when creating an IBV_QPT_RAW_PACKET QP mlx4_core: - Added the module parameter log_num_mgm_entry_size - Updated bitmap allocator implementation to match upstream - Changed bitmap allocator to work in round-robin fashion - Added QinQ support - Enabled MLX4_CMD_TIME_CLASS_A usage for CLOSE_PORT timeout - Using correct index when setting the port types - Added support for suspend/resume events rds - using __rds_rdma_send_complete instead of __rds_send_complete - Fixed rds_send_drop_to parameter - De-pessimized rds_page_copy_user ofed_scripts/mlx4_vnicd: - Added support to CentOS family ibutils2-2.1.1-0.7.g07c744d perftest-1.4.0-0.66.g9074687 (removed rdma_lat and rdma_bw) backports/iser: - Fixed pick fixes from upstream 3.5 on top of RHEL6.2 - Fixed pick fixes from upstream (2.6.36..3.4) on top of RHEL6.1 - Fixed pick fixes from upstream (2.6.36..3.4) on top of RHEL6.2 - Added backports to 2.6.34 General: - Added yum install support - Added affinity scripts - Added ethtool backports to define ETHTOOL_FWVERS_LEN - Updated perftest to v1.3.0-0.58.g8f82435 mlnx_ofed/sysctl: - Added net.ipv4.tcp_low_latency mellanox-mlnx-en.spec: - Removed unused defines IB/core: - Fixed generate_unmatched_response logic for unhandled MADs. IB/mad: - Return unsupported for MADs as appropriate - Added MAD error codes per IBTA spec mlx4: - Fixed bad scheduling of NOP command - Increased Command Timeout for NOP operation - Starting keep_alive task only after all iitialization completed - Added a fallback to single completion vector option after NOP failure - Added keep alive support - Increased IB ULP EQ limit mlx4_ib: - Fixed hugetlb support for user mr - Set link type for the primary path - Added hugetlb support for user mr with unalign virt_addr - Fixed a bug in mlx4_ib_netdev_event, consider VLAN device as well - Removed support for hugetlb for user_mr with unalign virt_addr ofed_scripts/connectx_port_config: - Do not use port_trigger if not supported ofa_kernel.spec: - Loaded UCM kernel module by default libibverbs: - Added support to ibv_devinfo for displaying extended speeds - Enabled huge pages handling in ibv_fork_init() and madvise tracking - Updated man page with the support of the huge page with fork ofa_kernel.spec: - Added weak updates support for RedHat mlx4_vnic: - Added support in EQs per port fix - Added macvtap support iser: - Added back-port for iser to align with the GA RHEL 6.2 iscsi stack =============================================================================== 3. Contents of Mellanox OFED for Linux =============================================================================== Mellanox OFED for Linux software contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, qib) - iWARP RNIC driver (cxgb3, nes) - 10/40GigE NIC driver (mlx4_en) - core with RoCE support - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and RDS Note: qib, cxgb3, nes and mthca were not tested in MLNX_OFED_LINUX-1.5.3 - Additional kernel modules: - EoIB - OpenFabrics utilities: - ib-bonding: Bonding driver for IPoIB interface - OpenSM: IB Subnet Manager with Mellanox proprietary Adaptive Routing - Diagnostic tools - Performance tests - MPI: - OSU MPI (MVAPICH 1.2.0) stack supporting the InfiniBand interface and RoCE - Open MPI stack 1.4.1 supporting the InfiniBand interface and RoCE - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) - Extra packages: - ibutils2 - ibdump - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) except for OpenSM, ibutils2, and ibdump - Documentation Notes: - See release notes for each package in the docs/release_notes directory. - SDP is at beta level =============================================================================== 4. Supported Platforms and Operating Systems =============================================================================== Summary of changes in the list of supported OSes: Added support for: - RHEL6.3 RHEL5.5 (MLNX_OFED_LINUX-1.5.3-rhel5.5.iso) i686: 2.6.18-194.el5 2.6.18-194.el5PAE x86_64: 2.6.18-194.el5 ppc64: 2.6.18-194.el5 RHEL5.6 (MLNX_OFED_LINUX-1.5.3-rhel5.6.iso) i686: 2.6.18-238.el5 2.6.18-238.el5PAE x86_64: 2.6.18-238.el5 ppc64: 2.6.18-238.el5 RHEL5.7 (MLNX_OFED_LINUX-1.5.3-rhel5.7.iso) i686: 2.6.18-274.el5 2.6.18-274.el5PAE x86_64: 2.6.18-274.el5 ppc64: 2.6.18-274.el5 RHEL5.8 (MLNX_OFED_LINUX-1.5.3-rhel5.8.iso) i686: 2.6.18-308.el5 2.6.18-308.el5PAE x86_64: 2.6.18-308.el5 RHEL6.0 (MLNX_OFED_LINUX-1.5.3-rhel6.iso) i686: 2.6.32-71.el6.i686 x86_64: 2.6.32-71.el6.x86_64 ppc64: 2.6.32-71.el6.ppc64 RHEL6.1 (MLNX_OFED_LINUX-1.5.3-rhel6.1.iso): i686: 2.6.32-131.0.15.el6.i686 x86_64: 2.6.32-131.0.15.el6.x86_64 ppc64: 2.6.32-131.0.15.el6.ppc64 RHEL6.2 (MLNX_OFED_LINUX-1.5.3-rhel6.2.iso): i686: 2.6.32-220.el6.i686 x86_64: 2.6.32-220.el6.x86_64 ppc64: 2.6.32-220.el6.ppc64 RHEL6.3 (MLNX_OFED_LINUX-1.5.3-rhel6.3.iso): i686: 2.6.32-279.el6.i686 x86_64: 2.6.32-279.el6.x86_64 SLES10 SP4 (MLNX_OFED_LINUX-1.5.3-sles10sp4.iso): i686: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-bigsmp 2.6.16.60-0.85.1-default x86_64: 2.6.16.60-0.85.1-smp 2.6.16.60-0.85.1-default ppc64: 2.6.16.60-0.85.1-ppc64 SLES11 (MLNX_OFED_LINUX-1.5.3-sles11.iso): i686: 2.6.27.19-5-default 2.6.27.19-5-pae x86_64: 2.6.27.19-5-default ppc64: 2.6.27.19-5-default SLES11 SP1 (MLNX_OFED_LINUX-1.5.3-sles11sp1.iso): i686: 2.6.32.12-0.7-default 2.6.32.12-0.7-pae x86_64: 2.6.32.12-0.7-default 2.6.32.36-0.5-default ppc64: 2.6.32.12-0.7-ppc64 SLES11 SP2 (MLNX_OFED_LINUX-1.5.3-sles11sp2.iso): i686: 3.0.13-0.27-default 3.0.13-0.27-pae 3.0.13-0.27-xen x86_64: 3.0.13-0.27-default 3.0.13-0.27-xen OEL 6.1 (MLNX_OFED_LINUX-1.5.3-oel6.1.iso): x86_64: 2.6.32-100.34.1.el6uek.x86_64 Citrix XenServer Host 5.6.100-46766p i686: 2.6.32.12-0.7.1.xs5.6.100.323.170596xen Citrix XenServer Host 6.0.0-50762p i686: 2.6.32.12-0.7.1.xs6.0.0.529.170661xen 2.6.32.12-0.7.1.xs6.0.0.531.170662xen NOTE: Citrix XenServer is at alpha level. To receive the XenServer driver, please contact Mellanox support (support@mellanox.com). NOTE: If you wish to install OFED on a different kernel, you need to create a new ISO image, using mlnx_add_kernel_support.sh script. See the MLNX_OFED User Guide for instructions. ============================================================================== 5. Software Requirements ============================================================================== 1) Linux operating system (see Supported Platforms and Operating Systems above) 2) Administrator privileges on your machine(s) 3) Disk Space: 500MB 4) For the OFED Distribution to compile on your machine, some software packages of your operating system (OS) distribution are required. These are listed here. OS Distribution Required Packages --------------- ---------------------------------- General: o Common to all gcc, glib, glib-devel, glibc, glibc-devel, glibc-devel-32bit (to build 32-bit libraries on x86_64 and ppc64), zlib-devel, libstdc++-devel o RedHat, Fedora kernel-devel, rpm-build, redhat-rpm-config o SLES kernel-source, kernel-syms, rpm Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit glibc-devel should be installed. Specific Component Requirements: o Mvapich a Fortran Compiler (such as gcc-g77) o Mvapich2 libsysfs-devel o Open MPI libsysfs-devel o ibutils tcl, tcl-devel, tk, libstdc++-devel o mstflint libstdc++-devel (32-bit on ppc64), gcc-c++ Note: The installer will warn you if you attempt to compile any of the above packages and do not have the prerequisites installed. On SLES, some of required RPMs can be found on SLES SDK DVD. E.g.: libgfortran43, kernel-source, ... =============================================================================== 6. Supported HCAs =============================================================================== Mellanox supports the following adapters with MLNX_OFED_LINUX 1.5.3: - Mellanox Technologies HCAs (SDR and DDR Modes are Supported): - ConnectX(R) and ConnectX EN (fw-25408 Rev 2.9.1000) - ConnectX-2 (fw-ConnectX2 Rev 2.9.1000) - ConnectX-2 EN (fw-ConnectX2 Rev 2.9.1000) - ConnectX-3 (fw-4099 Rev 2.10.0) Note: To receive ConnectX-3 firmware, please contact your Mellanox representative. For official firmware versions please see: http://www.mellanox.com/content/pages.php?pg=firmware_download =============================================================================== 7. Switch and Gateway Systems Used for Testing =============================================================================== This release was tested with switches and gateways provided by the following companies: - Arastra - Cisco - Dell - Flextronics - Fujitsu - HP - IBM - Juniper - Mellanox Technologies - Oracle - Extreme networks - Brocade =============================================================================== 8. Compatibility =============================================================================== - Linux vs. VMware Tested IPoIB connectivity (ping, Arp, Iperf) - Red Hat Enterprise Linux Server release 6.0 x64, w/w ConnectX(R) IB QDR, PCIe 2.0 5GT/s - VMWare Esx4.0 w/w ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s - SRP target with VMWare initiator - Linux vs. Windows Tested IPoIB, Verbs and Eth (ping,netprf, ib_read/write*, opensm handovers) - Red Hat Enterprise Linux Server release 6.0 x64 w/w ConnectX-2 (Tavor compatibility mode) and ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE as Eth - Windows 2008 R2 x64 runing WinOF 2.0, MLNX_VPI 2.3.0 w/w ConnectX VPI PCIe 2.0 5GT; IB QDR / 10GigE ConnectX EN 10GigE, PCIe 2.0 5GT/s - Linux vs. FabricIT(TM) EFM Tested IPoIB, Verbs and OpenSM priority handover - SLES 11 x64 w/w ConnectX VPI PCIe 2.0 5GT/s - IB QSFP QDR / 10GigE, ConnectX VPI - 10GigE / IB QDR - IS5030 (Shark GT) w/w FabricIT EFM version 1.1.2500 - Linux vs. FabricIT(TM) BXM - MBX5020 (BrigeX) w/w FabricIT BXM version 2.1.1000 - VMA 4.5.12 / 5.5.10 - FCA 2.1 - UFM 3.5 - VSA 2.0 =============================================================================== 9. Change Log History =============================================================================== MLNX_OFED_LINUX-1.5.3-3.0.0 Main Changes (from v1.5.3-1.0.0): ------------------------------------------------------------------------------- - Added extended link capabilities support (FDR/FDR10) for ConnectX-3 - Added RHEL6.2 support - Added RHEL5.7 support - Added OEL6.1 support - Added support for kernel 2.6.3[7-9] and 3.0 - Added kernel module package (KMP) support - Added iser support for RHEL 5.7 - EoIB new features: o Para Virtualization support o The following features are enabled only while working with FabricIT-BXM v2.1.1000 or newer: o Bridge-level LAG support o PKEY discovery support o ALL VLAN support - ibdump: Added ConnectX®-3 support - mstflint: Added ConnectX®-3 support - Removed Fiber Channel support - Removed mlx4_accl module Changelog: ---------- core: - Changed link_layer string to match mainline kernel - Fix NULL pointer dereference - Added extended link capabilities support (FDR/FDR10) - Added fixes for masked atomic in kernel level RDMA/cm: - Destroy idr as part of the cleanup in the error flow mlx4: - Added warnings in case of command timeouts - Reduced the waiting period on software reset to 500 mili seconds - Updateed version to mlnx_ofed-1.5.3 - Fixed setting of port mask in multifunction mode - Added sync_qp to mlx4_dev capabilities - Added sanity check to prevent posting of empty SR - Fixed the attribute sq_sig_type value in query qp - Fixed cqe 64 support - Enabled A0 steering mode - Limited the min profile numbers - Fixed an error message in /var/log/messages (bug #119232) mlx4_ib: - Increased ABI version for 64 byte CQEs - RoCE: Fixed gid table population when using bonding over vlan mlx4_en: - Updating version to 1.5.7 - Rename mac_steer modparam to high_rate_steer - Unregister mac after releasing rss_steer - Fix a bug with QP release - Fix a race when setting a multicast list. - Fix endianness with blue flame support - Fix WOL handlers always look at port2 capability bit - Added 16 TX queues patches to 2.6.39 and 3.0 - Added hardware features flags in kernels 2.6.39 and up - Added MLX4_CMD_QUERY_IF_STAT_wrapper for flex10 - Added MLX4_CMD_SET_NODE_wrapper as NOOP for flex10 - Added support for fast packet drop - Enabled sys tune only if APM is configured in kernels 3.0 and up - Fixed bad flow for illegal mac - Fixed cq moderation timer - Fixed XenServer backport - Fixed HP mezz initialization error - Stopped calculating fcs in skb lenght - Removed flag allocation from memtrack - Set default MTU to dev->mtu - Enabled the use of GRO for PPC - Fixed linux-3.0 compilation issue with smp_lock.h - Modified swapping XAUI and XFI 10G speed report according to the PRM - Canceled bad patches disabling LRO - Added 40GigE speed report in ethtool - Fixed the crash that occured on initial port configuration (bug #3216) - Removed acceleration support - Disabled default mac steer param - Fixed the vlans that did not work on one port (bug #1694) - Fixed vlan 0 panic - Fixed uninitialized ip_summed at rx - Cleared ioremap allocation at en_main.c - Fixed UDP traffic issue related to ConnectX-3 - Fixed broken iboe counters addition to mlx4_en's mlx4_vnic: - Fixed printf exception due to wrong formatting - Added support in all vlan feature - Added missing ib_dma_unmap_single - Added missing spinlock to vnic_mcast_reattach() - Fixed dma unmap in case of an error in vnic_dma_map_tx() - Fixed error handling in vnic_unmap_and_replace_rx - Fixed IPv6 TCP traffic for kernels newer than 2.6.24 (FM #121735) - Fixed TCP/UDP header calculation in get skb/frag hdr - Fixed the error behavior of vnic_alloc_frag() - Fixed the mgid construction on ppc machines (RM #3450) - Removed unused field static_alloc from struct fip_ring - Removed lower case converting of vnic name in get_vnic() - Replaced all the calls to "dma_sync_single_range" with "ib_dma_sync_single" - Replaced the use of dma map in ib_dma map - Set ip_hdr in vnic_get_frag_header() (FM #123775, RM #3441) - Increased unicast solicit at FIP_GW_MCAST_RCVD - iLAG - Updated the saved gateway address when getting adv packet - Fixed ctrl iport address update (FM #120905) - Addeded support for iLAG feature - Added support for power cycle id - Decreased stack usage in fip_discover_rx_packet_bh() - Fixed LAG functions to work with child vNics (FM #120576) - Bug fixes mthca: - Fixed the attribute sq_sig_type value in query qp ipoib: - Don't release resources if the add_one failed - Adaptive mode, no assumption according the TCP traffic. - Set default TX/RX buffer size to 512 - Disabled lro whenever 4k MTU is enabled - Enabled by default lro and sync between ethtool and module param - Fine-tuned adaptive parameters - Added support for ethtool in -g option. That option shows the ring size of Rx and Tx. - Enabled the usage of the exact value for rx_coalesced_frames in ethtool. The value matches the value received by the user via the ethtool. - Added adaptive moderation algorithm for better latency ib/iser: - Work-around for XenServer junk dma issue srp: - Don't release resources if the add_one failed - libmlx4: - Avoid bad packets with RoCE/UD/VLAN - Added Core Direct support for ConnecX-3 HCAs - Added sanity check for posting a SR without any s/g entries to a RAW QP New packages: - Added MXM: mxm-1.0.601-1 - Added FCA: fca-2.1.12028-1 - Added KNEM: knem-0.9.7-0 Updated packages: - libibverbs: 1.1.4-1.26.ga73201a - libmlx4: 1.0.1-1.23.g49e4182 - mstflint: 1.4-1.18.g1adcfbf (Fixed port status report) - MFT: mft-2.7.1-7 - opensm: 3.3.9.MLNX_20111006_e52d5fc - perftest: 1.3.0-0.56.gd6f6093 - dapl: 2.0.34 - dump_pr: 1.0-0.1.g79715a9 - ibdump: 1.0.6-1 - ibutils: 1.5.7-0.1.g05a9d1a - ibutils2: 2.0-0.34.g9d3133a - infiniband-diags: 1.5.8.MLNX_20110906 - librdmacm: 1.0.15 - libibmad: 1.3.7.MLNX_20110814 - libibumad: 1.3.7.MLNX_20110814 - libmqe: 0.3.0-4.9.g973226d - libmverbs: 0.1.0-3.10.g3e83f31 MLNX_OFED_LINUX-1.5.3-1.0.0 Main Changes (from v1.5.2-2.1.0): ------------------------------------------------------------------------------- - MLNX_OFED_LINUX-1.5.3-1.0.0 is based on OFED-1.5.3.1 - Added RHEL6.1 support - Changes in /etc/infiniband/openib.conf. See MLNX_OFED_configuration_files.txt - Improved RAW Ethernet QP support (requires ConnectX/ConnectX2 FW >= 2.8.0600) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - KMP support by ofa_kernel package - RAW QP API changes - Added mlx4_accl Socket Acceleration package (relevant for mlx4_en and IPoIB) - mlx4: add support for 64 byte CQEs (ConnectX-3 only) - mlx4: enable raw eth QPs to work with inline and blueflame - mlx4_en: new steering model - mlx4_en: using GRO for 2.6.32 - mlx4_en: Ethernet NCSI support - ipoib: Add acceleration support - ipoib: enable LRO by default - ipoib: bug fixes (see ipoib_release_notes.txt) - sdp: ipv6 support - mlx4_vnic: Add Para Virtualization (PV) support ------------------------------------------------------------------------------- Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1 ---------------------------------------------------- (http://openfabrics.org/downloads/OFED/ofed-1.5.3/OFED-1.5.3.1.tgz) - Added iSER support on RHEL5.5, RHEL5.6, RHEL6.0, RHEL6.1, SLES11 SP1 and XenServer 5.6.100 - Packages included in MLNX_OFED-1.5.3: - ibdump-1.0.5.4 (proprietary) - ibutils2-2.0 (proprietary) - mft-2.7.0-20 (proprietary) - Firmware Tools - ar_mgr (proprietary) - Adaptive Routing Manager - cc_mgr (proprietary) - Congestion Control Manager - dump_pr (proprietary) - Dump PathRecord Plugin - mlx4_accl (proprietary) - Socket Acceleration - Core-Direct libraries - libmverbs - libmqe - VNIC support: - mlx4_vnic (EoIB) - Modules not included in MLNX_OFED-1.5.3 - NFS-RDMA - MPI: Mvapich2 NOTE: For other kernels based on the operating systems above, use the mlnx_add_kernel_support.sh script NOTE: For other operating systems (e.g., Fedora, Ubuntu), use OFED that is available through www.openfabrics.org. =============================================================================== 10. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release. - When traversing an FDR switch spine using an MTU smaller than 4K, the throughput may differ up to -~300MB/s. - When upgrading from an earlier Mellanox OFED version, the installation script does not stop the earlier version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to this new version. - When upgrading from the previous OFED installation to this release, you must reboot after driver installation since the kernel module ipoib_helper does not unload. - "openibd stop" can sometime fail with the error: Unloading ib_cm [FAILED] ERROR: Module ib_cm is in use by ib_ipoib Workaround: run "openibd stop" again. - Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. - The Intel compiler on RH5.4/5.5/5.6 requires that the compat-libstdc++-33-3.2.3-61 RPM be installed on your host. - When working with ISCSI over IPoIB or mlx4_en, you must disable LRO (even if IPoIB is set to connected mode). This is because there is a bug in older kernels which causes a kernel panic. - OpenMPI and Mvapich with the PGI compiler are not supported on SLES11 32-bit, SLES11 SP1 32-bit, RHEL6.0 and RHEL6.1. - On SLES11, and in case uninstall is failing, check the error log and remove the remaining RPMs manually using 'rpm -e '. - On SLES11, set the allow_unsupported_modules parameter to 1 in the file /etc/modprobe.d/unsupported-modules. Without this setting, the modules will not load. - When working under bonding, unloading the mlx4_en driver may cause unexpected behavior by the bonding driver. - Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring. (https://bugs.openfabrics.org/show_bug.cgi?id=1990) - MLNX_OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox proprietary packages. For further detail please see section 4. Differences between MLNX_OFED-1.5.3 and OFED 1.5.3.1. - The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in kernel or userspace. - mlx4_en with bonding on SLES11 SP1: Do not unload mlx4_en (modprobe -r mlx4_en) while interfaces are enslaved. Unloading mlx4_en kernel module while interfaces are enslaved can lead to unexpected behavior. - Configuring Bonding over IPoIB interfaces while QIB driver is loaded can lead to kernel panic - SDP does not work with EoIB interfaces. - When creating port configuration files using YAST on SLES systems, not all (or no) mlx4_en network interfaces are recognized by the tool. Workaround: The interface configuration needs to be added as a new interface and the MAC address needs to be set manually. - cxgb3i kernel module fails to be loaded after MLNX_OFED installation: # service iscsid start Starting iSCSI daemon: FATAL: Error inserting cxgb3i (/lib/modules/2.6.18-238.el5/kernel/drivers/scsi/cxgb3i/cxgb3i.ko): Unknown symbol in module, or unknown parameter (see dmesg) cxgb3i: disagrees about version of symbol cxgb3_register_client cxgb3i: Unknown symbol cxgb3_register_client cxgb3i: disagrees about version of symbol cxgb3_alloc_atid cxgb3i: Unknown symbol cxgb3_alloc_atid ... - On Citrix XenServer Host 5.6.100-39215p mlx4_core driver may failed to be loaded with the following error: mlx4_core 0000:08:00.0: NOP command failed to generate MSI-X interrupt IRQ 1217). mlx4_core 0000:08:00.0: Trying again without MSI-X. mlx4_core 0000:08:00.0: SW2HW_EQ failed (-110) mlx4_core 0000:08:00.0: Failed to initialize event queue table, aborting. The problem isn't seen on Xen Server 5.6 sp2 (exact version 5.6.100-46766p kernel 2.6.32.12-0.7.1.xs5.6.100.323.170596xen). - On Citrix XenServer 5.6 the iser initiator may get from the dom0 kernel "bad" dma addressees for the iscsi login request/response buffer which if used result in data-corruption during login. Limiting the buffer used for login to 4k seems to provide work around for the problem. If the driver would observe such "bad" dma address, it will intentionally fail the login attempt, such that a similar to the following message will be printed to the system log: iser: iser_create_ib_conn_res:login buf cc5c0000 size 16460 kvirt/dma - req cc5c0000/77d7000 \ resp cc5c2000/77d9000 dma err 0/0 bad 1/1 iser: iser_create_ib_conn_res:unable to alloc mem or create resource, err -12 iser: iscsi_iser_ep_poll:ib conn c9f864bc rc = -1 -- note that the size is 16460 where with the workaround applied it will be 4096 when things are okay, the following should be seen iser_create_ib_conn_res:login buf deb3b000 size 4096 kvirt/dma - req deb3b000/617878000 \ resp deb3b800/617878800 dma err 0/0 bad 0/0 - EoIB (mlx4_vnic) is not supported on SLES10 SP4 - To forward TCP traffic LRO should be disabled (Relevant for IPoIB, EoIB and MLX4_EN) To disable LRO EoIB: modprobe mlx4_vnic lro_num=0 IPoIB: Set "options ib_ipoib lro=0" in /etc/modprobe.d/ib_ipoib.conf MLX4_EN: Use ethtool: ethtool --offload lro off - mlx4_vnic kernel module (EoIB) is not loaded by default. Workaround: Set MLX4_VNIC_LOAD=yes in /etc/infiniband/openib.conf to load mlx4_vnic upon boot. - Running more than 2 instances of openSM in the same IB subnet has not been qualified. - An IPoIB cannot be a full member of a Multicast Group (MC) if the "umcast" flag (/sys/class/net/$dev/umcast) is set to 1, or if that IPoIB has previously joined the MC group as a "send only" member - The introduction of 64 bit CQE support in ConnectX3 devices has increased the ABI version of the kernel. Thus upgrading the included libmlx4 library accordingly. Hence, old libraries/executables that were built using the older headers will fail to load. Workaround: Rebuild the binaries using the new headers. - Ethernet and RoCE cannot work on the same port of ConnectX-2 device. Note: To receive ConnectX-2 firmware that fix this issue, please contact your Mellanox representative. RoCE ONLY: - PowerPC and ia64 architecture currently are not supported. - x32 architectures were not tested. - SRP is currently not supported. - When running the ibdiagnet check nodes_info on the fabric, a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed. Workaround: Run ibdiagnet --skip nodes_info - On RHEL6.3, mlnxofedinstall gives the following warning: Installing kmod-mlnx-ofa_kernel RPM Preparing... ################################################## kmod-mlnx-ofa_kernel ################################################## WARNING: /lib/modules/2.6.32-279.el6.x86_64/kernel/drivers/infiniband/hw/ipath/ib_ipath.ko needs unknown symbol ib_wq WARNING: /lib/modules/2.6.32-279.el6.x86_64/kernel/drivers/infiniband/hw/ipath/ib_ipath.ko needs unknown symbol ib_wq This message can be safely ignored. ib_ipath driver was replaced by ib_qib driver. - When the HCA temperature reaches its threshold, (default, 105 Celsius degrees), the following message is logged in the kernel logs (the device number and the index may be different): mlx4_core 0000:08:00.0: Unhandled event 1b(00) on EQ 11 at index 3986 If this message appears in the logs, please check the server's cooling system. - The parameters below must be added to the IPoIB interfaces' configuration files (ifcfg-ibX) on RHEL6.3 otherwise, different network interfaces may get the same IP address. NM_CONTROLLED="yes" TYPE="InfiniBand" - perftest: Removed rdma_bw and rdma_lat tests from the perftest package. All of their features and functionality are contained in other tests. They originally performed SEND verb with/o rdma_cm, this feature is already supported in all other tests with "-R" flag. When using this feature ("-R"), you must supply the IPoIB interface IP in order to connect them. - Unidirectional RDMA Write bandwidth can be low when remote CPU is idle. On some Intel systems when the CPU is idle, the bandwidth is lower for messages larger than 4MB. This can occur even if the power management is disabled. In order to increase the bandwidth the CPU need to be active. Example: # ib_write_bw -a : Remote CPU is idle, 4MB messages bandwidth is not the maximum. # ib_write_bw -a -z : Remote CPU is occupied by rdma_cm qp polling on CQ, 4MB messages bandwidth is maximum. # ib_send_bw -a : Remote CPU is occupied by receiving qp polling on CQ, 4MB messages bandwidth is maximum. - Bidirectional bandwidth of messages larger than 4MB could be lower when not using huge pages. In order to increase the bandwidth for these messages, configure huge pages and use the appropiate memory allocation functions when allocating buffers for these nmessages. Example for huge pages configuration: # echo 400 > /proc/sys/vm/nr_hugepages This will allocate 400 pages of 2MB each Please note that the application is responsible for freeing the allocated huge page memory and not the OS. Note: See the release notes of each component for additional issues.