Mellanox Technologies ********************* InfiniBand OFED Driver for VMware(R) Infrastructure 5.0/5.1 Release Notes Version ESXi 1.8.1 February 2013 =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Changes in This Release 3. Contents of Mellanox OFED ESXi 4. Supported Platforms and Operating Systems 5. Supported HCAs 6. Switch and Gateway Systems and Storage 7. Change Log History 8. Known Issues 9. Bug Fixes =============================================================================== 1. Overview =============================================================================== These are the release notes of "InfiniBand OFED Driver for VMware(R) vSphere 5.x". This document provides instructions on IB drivers for Mellanox Technologies ConnectX(R) based adapter cards with VMware ESXi Server environment. This release notes file and the user's manual are available from http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers =============================================================================== 2. Changes in This Release =============================================================================== MLNX-OFED-ESX-1.8.1 provides the following changes and new features: - SRP support - Bug Fixes (See section 9. Bug Fixes) =============================================================================== 3. Contents of Mellanox OFED ESXi =============================================================================== 3.1 Package Contents -------------------- MLNX-OFED-ESX package contains: o MLNX-OFED-ESX-1.8.1.zip - Hypervisor bundle which contains the following kernel modules: - mlx4_core (ConnectX family low-level PCI driver) - mlx4_ib (ConnectX family InfiniBand driver) - ib_core - ib_sa - ib_mad - ib_umad - ib_ipoib - ib_cm - ib_srp The driver package is distributed as an offline bundle (.zip file). =============================================================================== 4. Supported Platforms and Operating Systems =============================================================================== o CPU architectures: - x86_64 o ESX Hypervisor: - ESXi5.0 with BUILD ID: 469512 - ESXi5.0u1 with BUILD ID: 623860 - ESXi5.1 with BUILD ID: 799733 Note: For the supported hardware compatibility list (HCL) and guest operating systems, please refer to VMware support documentation at http://www.vmware.com/support/pubs and follow the "Compatibility Guides" link =============================================================================== 5. Supported HCAs =============================================================================== This release supports Mellanox Technologies InfiniBand (IB) HCAs: - ConnectX-2: 2.9.1000 - ConnectX-3: 2.11.0500 Please note that older FW versions were not tested with this release. For the latest firmware versions, visit: http://www.mellanox.com/page/vmware_matrix or http://www.mellanox.com/supportdownloader/ =============================================================================== 6. Switch and Gateway Systems and Storage =============================================================================== o All production InfiniBand switches and gateways are supported. o Tested platforms: - This release was tested with switches provided by Mellanox: IS5025, IS5035 SX6036, SX6518, SX6512, Grid Director 4036 and 4036E - Storage: o NFS over IPoIB o GPFS over IPoIB o SRP =============================================================================== 7. Change Log History =============================================================================== MLNX-OFED-ESX-1.8.0-672 Main Changes (from v1.8.0) --------------------------------------------------- - ESXi 5.1 support - GPFS support - InfiniBand resource cleanup tuning - Removed LID based MAC generation - Debug file system (PROCFS) is currently at beta level - SR-IOV is not supported in the current release MLNX-OFED-ESX-1.8.0 Main Changes and New Features --------------------------------------------------- o This is the initial GA release for ESXi 5.x driver, containing the following main features: - IPoIB – Para Virtualization - ESXi 5.0 support - InfiniBand support only - FDR support - 4k MTU support =============================================================================== 8. Known Issues =============================================================================== o Number of MAC addresses per host is limited to 400 o ibportstate utility (MLNX_OFED) usage for setting port down/up is not supported o IPv6 is not supported in the current release. o The minimum supported MTU value is 1500 o On ESXi5.0/ESXi5.0u1, the "esxcli network nic list" command (a.k.a esxcfg-nics -l) displays the wrong link state. We recommend running the ethtool instead. o Using vSwitch nic teaming feature requires setting the network failure detection to “Link status Only” o Packets might get dropped upon initial communication between interfaces o The "lspci" command displays an incorrect interface name o Poor TCP performance may occur in Linux virtual machines with LRO enabled. To solve the issue, please refer to the VMware website: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1027511 o Unloading the driver is not supported by VMware o The "esxcli network nic down -n " and the "esxcli network nic up -n " commands are only supported in ESXi 5.1 and above. 8.1 Migration/VMotion Known Issues ------------------------------------------------------------------------------- o Frequent migration/VLAN changes may cause connectivity loss up to 1 minute o When using VGT, connectivity may be lost for up to 40 seconds after vmotion. To restore connectivity, send ping from the migrated VM. Migration/VMotion Workaround: o To solve the issues above flash the ARP tables in all sources and targets 8.2 SRP Known Issues ------------------------------------------------------------------------------- o Unloading mlx4 low-level drivers under heavy SRP traffic may cause system PSOD. Workaround: Unload the SRP module before mlx4 drivers. Note that running "/opt/mellanox/bin/openib.sh restart" does not make sure that SRP module is successfully unloaded before continuing to unload mlx4 drivers, and therefore may cause PSOD under heavy traffic. o Restart subnet manager with re-assign lid or perform subnet manager failover under heavy traffic from VMs and/or multiple concurrent storage migrations conditions may cause increasing in latency, or lose connectivity to storages, or datastor corruption. o "/opt/mellanox/bin/mlnx-srp-config start/stop" does not save and revert VAAI features settting if VAAI's /DataMover/HardwareAcceleratedMove, /DataMover/HardwareAcceleratedInit, and /VMFS3/HardwareAcceleratedLocking options are set to "1" by default before executing the script. o Multiple ESXi SRP hosts/initiators may flood embedded subnet manager running in InfiniBand switch with PortInfoRecord queries and get "multiple matching records" back from switch. This behavior has no effect on the system. o Unloading SRP driver without traffic or storage connected may take up to 60 seconds. =============================================================================== 9. Bug Fixes =============================================================================== The following are ESXi 5.x driver bug fixes: o A multicast group remained opened after being closed when working with PV VM o A VM listening to a multicast group does not receive packets for that multicast group after vmotion. The mcast group must be created manually.