Mellanox Technologies ===================== =============================================================================== MLNX_EN for Linux Release Notes Driver Kit for Mellanox ConnectX EN/ConnectX-2 EN/ConnectX-3 EN Adapter Cards with 10GigE and 40GigE Support Rev 1.5.10, February 2013 =============================================================================== ------------------------------------------------------------- NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (PRODUCT(S)) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------- © Copyright 2013. Mellanox Technologies. All rights reserved. Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd. Connect-IB™, FabricIT™, MLNX-OS™, MetroX™, MetroDX™, ScalableHPC™, Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. Contents: ========= 1. Introduction 2. Supported Platforms, Operating Systems and Firmware 3. Changes and New Features 4. Bug Fixes 5. Known Issues 6. Network Tuning 1. Introduction =============== These are the release notes for Mellanox Technologies' MLNX_EN for Linux version 1.5.10 driver kit for Mellanox adapter cards with 10GigE and 40GigE support. 2. Supported Platforms, Operating Systems and Firmware ====================================================== o CPU architectures: - x86_64 - x86 - power-pc o Linux Operating Systems: - RedHat EL5.8 - RedHat EL5.9 - RedHat EL6.2 - RedHat EL6.3 - OEL6.2 + 2.6.32-279.19.1 - OEL6.3 + 2.6.32-279.19.1 - SLES11 SP1 - SLES11 SP2 o Supported Firmware - ConnectX/ConnectX-2 firmware fw-25408 v2.9.1200 or higher - ConnectX-3 firmware fw-4099 v2.11.0500 or higher 3. Changes and New Features ====================================================== 3.1 Changes in Rev 1.5.10 From Rev 1.5.9 ------------------------------------------- 1. Bug fixes 3.2 Changes in Rev 1.5.9 From Rev 1.5.8.3 ------------------------------------------- 1. Added support for kernel.org 3.5 2. Improved latency by optimizing RX repost mechanism 3.3 Changes in Rev 1.5.8.3 From Rev 1.5.8.2 ------------------------------------------- 1. Added support for RHEL6.3 3.4 Changes in Rev 1.5.8.2 From Rev 1.5.7.2 ------------------------------------------- 1. Added support for new kernels: 3.1, 3.2, 3.3. 2. Moved to interrupt mode to handle TX completions. 3. Added support for Ethtool. 4. Optimized Numa aware memory allocations. 5. Optimized interrupt usage for TX/RX completions. 6. Added KMP compliant installation process. 7. Added IRQ affinity control scripts (please see README file for more details). 3.5 Changes in Rev 1.5.7.2 From Rev 1.5.7 ------------------------------------------- 1. Added support for new OS's: RHEL6.2, RHEL5.8, SLES11SP2 2. Added recording RX queue for GRO packets. 3. Added the usage of Toeplitz hash function for RSS calculation. 4. Enabled RXHASH report on supported systems 3.6 Changes in Rev 1.5.7 From Rev 1.5.6 ----------------------------------------- 1. Added support for new OS's: RHEL6.1, RHEL5.5, RHEL5.7, kernel.org (2.6.37, 2.6.38, 2.6.39, 3.0). 2. Added support for RHEL6.1 KVM 3. Improved performance on PPC systems (Using GRO where LRO is not efficient) 4. Added IPv6 support to LRO. 5. Incremented number of TX and RX queues 6. Enabled NAPI usage at any given time 7. Enabled TX completions spread among multiple MSI-X vectors. 8. Improved small packets packet rate. 9. Added 40GigE support (including Ethtool report). 10. Added NUMA support. 11. Added general performance improvements. 3.7 Changes in Rev 1.5.6 From Rev 1.5.1.3 ----------------------------------------- 1. Added support for new OS's: RHEL6.0, RHEL5.6, SLES11SP1, kernel.org (2.6.35, 2.6.36). 2. Added blue flame support for kernels > 2.6.28 (improves TX latency by 0.4 usec) 3. Added RX acceleration feature that supports recvmsg and recvmmsg system calls See MLNX_EN_Linux_README for further details. 4. Added exact match multicast filtering 5. Added option to use interrupts for TX completion (polling is the default) 6. Added option to disable NAPI (enabled by default) 7. Added support for control number of RX rings from module parameter. 8. Added interrupt vector per each RX ring. See /proc/interrupts. 9. Link is brought up upon driver load. 10. Adaptive moderation improvements. 11. Added system tuning option to achieve better performance (idle loop polling). 11. Added hardware revision report via Ethtool. 3.8 Changes in Rev 1.5.1.3 From Rev 1.4.2 ----------------------------------------- 1. Support for new Operating Systems: RedHat EL5 Update5; Kernel.org: 2.6.16 up to 2.6.32 (partial testing) 2. Added UDP RSS support (on ConnectX-2 HW only). 3. Improved VLAN tagging performance. 4. Ethtool -e support. 4. Bug Fixes =============== 4.1 Bug Fixes in Rev 1.5.10 ----------------------------- 1. Fixed driver unloading after installation. Currently, the driver is reloaded after installation 2. Fixed an issue related to the configuration files. Configuration files are no longer modified. 3. Fixed MTU size calculation 4. Fixed race in join/leave multicast group flow 5. Restored RX buffer pointer in case of failure 6. Fixed IPMI 7. Fixed BlueFlame race 8. Added sanity check for module parameters 9. Fixed lost of connectivity bug for iperf long run 4.2 Bug Fixes in Rev 1.5.9 ----------------------------- 1. Fixed bad TX completion notification request logic, could lead to TX timeout 2. Fixed PowerPC connectivity loss 4.3 Bug Fixes in Rev 1.5.8.3 ----------------------------- 1. Fixed possible transmit timeout under heavy traffic load. 4.4 Bug Fixes in Rev 1.5.8.2 ----------------------------- 1. Fixed error flows in case of MSI-X assignment failures. 2. Fixed loss of multicast traffic during addition of multicast addresses. 3. Enabled legal MTU configuration prior to interface bring-up. 4.5 Bug Fixes in Rev 1.5.7.2 ------------------------------ 1. Fixed WoL configuration behavior for single ported devices. 2. Fixed the assignment of valid flow control parameters during interface initialization 3. Fixed qdisc behavior when transmitting packets with VLAN ID 0. 4. Fixed badly placed counters on ethool -S report. 4.6 Bug Fixes in Rev 1.5.7 ---------------------------- 1. Fixed the crash occurred during initialization if an invalid mac address was burned on the card. 2. Fixed a bug where interface link state was not updated correctly in certain scenarios. 3. Fixed a crash occurred when LRO enabled via ethtool but disabled in module parameter. 4. Fixed a warning when using netconsole. 5. Fixed a crash occurred during driver initialization due to multicast list update. 4.7 Bug Fixes in Rev 1.5.6 ---------------------------- 1. Fixed a bug that caused the driver not to work with MSI-X on RedHat EL4 systems. 2. Fixed the driver functionality in case the device is enslaved to bond with VLANs over it. 3. Fixed the crash occurred when sending malformed TCP packets. 4.8 Bug Fixes in Rev 1.5.1.3 ------------------------------ 1. If the two ports of a ConnectX/ConnectX-2 device are bonded together and one port is closed, then the other port will cease carrying traffic. Fixed in this release. 2. Multiple error messages when working under bonding on some operating systems. 4.9 Bug Fixes in Rev 1.4.2 ---------------------------- 1. Fixed a bug avoiding the driver to work with MSI-X on RedHat EL4 systems. 2. Enabled the driver to load on systems with 32 cores and higher. 3. The driver is being stuck if the HW/FW stops responding, reset is done instead. 4. Fixed recovery flows from memory allocation failures. 5. Corrected checksum reports on SLES11 systems. 6. When the system is low on memory, the driver now allocates smaller RX rings. 7. The driver now retries to obtain MSI-X vectors if the initial request is rejected by the OS. 5. Known Issues ================ 1. On RH6.* systems IQR-balancer should be disabled to avoid lost interrupt issue: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=539671 The outcome of not disabling it could be connectivity loss. 2. The driver is not loaded by default during boot on SLES10SPx systems. Workaround: Edit the /etc/sysconfig/kernel file. Set: MODULES_LOADED_ON_BOOT="mlx4_en" 3. When an iSCSI target is working over the mlnx_en driver, LRO should be disabled by adding the following line to /etc/modprobe.conf: "options mlx4_en num_lro=0" 4. When creating port configuration files using YAST on SLES systems, not all (or no) mlx4_en network interfaces are recognized by the tool. The interface configuration needs to be added as a new interface. Also the MAC address needs to be set manually. 5. VLAN double-tagging is not supported in this release. 6. When an interface is released from bonding in RH5.*, a crash may occur due to a bug in the bonding module. For more info see https://bugzilla.redhat.com/show_bug.cgi?id=659594. 7. Module unload while interfaces are under bond is not supported. 8. If you encounter any problem when loading the mlx4_core/mlx4_en module with customized parameters, please check dmesg. 6. Network Tuning ================== MLNX_EN installation changes several default network parameters in order to achieve optimized networking performance in terms of high throughput, low latency and low CPU utilization. These parameters are changed through the 'sysctl' interface. In order to restore these parameters to their default values, download the "mlnx_en_reverse_settings.conf" file from the MLNX_EN for Linux Web page http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=27&menu_section=35 and run: > /sbin/sysctl -p mlnx_en_reverse_settings.conf