Mellanox Technologies ===================== =============================================================================== Driver Kit for Mellanox Adapter Cards with 10GigE Support README Version 1.0, July 2008 Document No. 2950 =============================================================================== Contents: ========= 1. Overview 1.1 Package Contents 2. Supported Platforms, Operating Systems and Firmware 3. Software Dependencies 4. Fibre Channel over Ethernet (FCoE) Driver 4.1 Overview 4.2 Configuration and Usage 5. Driver Installation 5.1 Installing the Driver 5.2 Loading the Driver 5.3 Unloading the Driver 6. Ethernet Driver Usage and Configuration 7. Firmware Programming 7.1 Installing Firmware Tools 7.2 Downloading the Firmware Image of the Adapter Card 7.3 Updating Adapter Card Firmware 8. Known Issues 9. Revision History 1. Overview =========== This document provides information on the ConnectX EN Linux driver and instructions for installing the driver on Mellanox adapter cards supporting 10Gb/s Ethernet. The driver is intended for adapter cards that identify on the PCI bus as with one of the following PCI Device IDs (decimal): 25408, 25418, 25448, 26418, and 26448. The MLNX_EN driver release exposes the following capabilities: - Single/Dual port - Fibre Channel over Ethernet (FCoE) - Up to 16 Tx and Rx queues per port - 2 Rx steering modes: Receive Core Affinity (RCA, default) or VLAN user-priority - 2 Tx arbitration modes: per-core (default) or VLAN user-priority - MSI-X or INTx - Adaptive interrupt moderation - HW Tx/Rx checksum calculation - Large Send Offload (i.e., TCP Segmentation Offload) - Large Receive Offload - Multi-core NAPI support - Header separation - VLAN Tx/Rx acceleration (HW VLAN stripping/insertion) - HW VLAN filtering - HW multicast filtering - ifconfig up/down + mtu changes (up to 10K) - Ethtool support - Net device statistics - CX4 connectors (XAUI) or XFP 1.1 Package Contents -------------------- This driver kit contains the following: - Kernel modules * mlx4 driver mlx4 is the low level driver implementation for the ConnectX adapters designed by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter, as an Ethernet NIC, or as a Fibre Channel (FC) HBA. To accommodate the three flavors, the driver is split into modules: mlx4_core, mlx4_en, mlx4_fc and mlx4_ib. Note: mlx4_ib is not part of this release - mlx4_core Handles low-level functions like device initialization and firmware commands processing. Also controls resource allocation so that the InfiniBand, Ethernet and FC functions can share a device without interfering with each other. - mlx4_en Handles Ethernet specific functions and plugs into the netdev mid-layer. - mlx4_fc Handles the FCoE functions using ConnectX Fibre Channel hardware offloads. Another kernel module, mlx4_ofc, is attached to it to enable plugging mlx4_fc into the openfc module (see below) in the same way the original "fcoe" module is plugged into the Open-FCoE project. * openfc Handles FC logic such as fabric login and logout, remote port login and logout, and transforms SCSI requests to FC exchanges. It plugs into the SCSI mid layer using the kernel provided scsi_transport_fc module. openfc handles the FCoE using scsi_transport_fc. - mstflint An application to burn a firmware binary image - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) - Documentation 2. Supported Platforms, Operating Systems and Firmware ====================================================== o CPU architectures: - x86_64 - x86 o Linux Operating Systems: - RedHat EL4 up4: 2.6.9-42.ELsmp - RedHat EL4 up5: 2.6.9-55.ELsmp - RedHat EL4 up6: 2.6.9-67.ELsmp - RedHat EL5: 2.6.18-8.el5 - RedHat EL5 up1: 2.6.18-53.el5 - RedHat EL5 up2: 2.6.18-92.el5 - SLES10: 2.6.16.21-0.8-smp - SLES10 SP1: 2.6.16.46-0.12-smp - SLES10 SP2: 2.6.16.60-0.21-smp - kernel.org: 2.6.24, 2.6.25.9 (or later) Note: FCoE supports only the following OSes and kernels: RedHat EL5:2.6.18-8.el5, kernel.org:2.6.24 o Supported Firmware - ConnectX firmware fw-25408 rev 2.5.000 3. Software Dependencies ======================== - To run the driver software, kernel sources must be installed on the machine. - The mlx4_en module uses a Linux implementation for Large Receive Offload (LRO) in kernel 2.6.24 and later. These kernels require installing the "inet_lro" module. 4. Fibre Channel over Ethernet (FCoE) ===================================== 4.1 Overview ------------ The FCoE feature provided by this driver allows connecting to FC targets on an FC fabric using an FCoE-capable switch or gateway. Key features include: - Complete hardware offload of FC exchanges (SCSI operations) - Hardware offload of FC-CRC calculations - Zero copy FC stack The FCoE feature is based on and interacts with the Open-FCoE project. The mlx4_fc module is designed to replace the original "fcoe" module and to allow using the ConnectX hardware offloads. See http://www.open-fcoe.org/ for further information on Open-FCoE project. 4.2 FCoE Driver Usage and Configuration --------------------------------------- After the driver is loaded, it immediately starts an FLOGI process on the Ethernet ports that are up. For all responding targets, LUNs will be scanned and mapped to SCSI disk devices (/dev/sdX, /dev/sdY, etc). The provided installation defaults are good to start running. The following configuration parameters can be modified if needed: MTU: If MTU of the Ethernet device was changed from the default (1500), edit /etc/mlxfc/mlxfc.conf and change the ETH_MTU variable to reflect the new MTU. Then restart the FC service. Max number of MAC addresses: This is controlled by the "num_mac" parameter of the mlx4_core module. FC requires at least 3 MAC addresses; therefore, this parameter should NOT be set to a number less than 3. 5. Driver Installation ====================== 5.1 Installing the Driver ------------------------- Step 1: Download Driver Package Please download the current driver package from http://www.mellanox.com/products/software.php. Step 2: Install Driver Run the following commands to install the driver: #> tar xzvf mlnx_en-1.0.tgz file #> cd mlnx_en-1.0 #> ./install.sh [--with-fc] The install.sh script builds the kernel modules out-of-kernel under /tmp/mlnx_en/src, places the resulting binaries under /lib/modules//updates/kernel/drivers/net/mlx4, and executes depmod. Note: Use the --with-fc option to install the FCoE modules. 5.2 Loading the Driver ---------------------- Loading the Ethernet driver: #> /etc/init.d/mlxnet start The result is a new net-device appearing in 'ifconfig -a'. See "Ethernet Driver Usage" for details on driver usage and configuration. Loading the FCoE driver: #> /etc/init.d/mlxfc start This will immediately start an FLOGI process on Ethernet ports that are up. For all responding targets, LUNs will be scanned and mapped to SCSI disk devices (/dev/sdX, /dev/sdY, ...) See "Fibre Channel over Ethernet (FCoE) Driver" for details on FCoE usage. 5.3 Unloading the Driver ------------------------ To unload the Ethernet driver run: #> /etc/init.d/mlxnet stop To unload the FCoE driver run: #> /etc/init.d/mlxfc stop Note: When both the Ethernet and Fibre Channel drivers are used, you need to unload both of them in order to close the NIC. 6. Ethernet Driver Usage and Configuration ========================================== - To assign an IP address to the interface run: #> ifconfig eth where 'x' is the OS assigned interface number. - To check driver and device information run: #> ethtool -i eth Example: #> ethtool -i eth2 driver: mlx4_en (MT_04A0140005) version: 1.0 (July 2008) firmware-version: 2.5.0 bus-info: 0000:13:00.0 - To query stateless offload status run: #> ethtool -k eth - To set stateless offload status run: #> ethtool -K eth [rx on|off] [tx on|off] [sg on|off] [tso on|off] - To query interrupt coalescing settings run: #> ethtool -c eth - By default, the driver uses adaptive interrupt moderation for the receive path, which adjusts the moderation time according to the traffic pattern. Adaptive moderation settings can be set by: #> ethtool -C eth adaptive-rx on|off - To set interrupt coalescing settings run: #> ethtool -C eth [rx-usecs N] [rx-frames N] [tx-usecs N] [tx-frames N] Note: usec settings correspond to the time to wait after the *last* packet sent/received before triggering an interrupt - To query pause frame settings run: #> ethtool -a eth - To set pause frame settings run: #> ethtool -A eth [rx on|off] [tx on|off] - To obtain additional device statistics, run: #> ethtool -S eth The driver defaults to the following parameters: - Both ports are activated (i.e., a net device is created for each port) - The number of Rx rings for each port is the number of on-line CPUs - Rx steering mode is set to RCA - Per-core NAPI is enabled - LRO is enabled with 32 concurrent sessions per Rx ring Some of these values can be changed using module parameters, which are detailed by running: #> modinfo mlx4_en To set non-default values to module parameters, the following line should be added to /etc/modprobe.conf file: "options mlx4_en = = ..." Values of all parameters can be observed in /sys/module/mlx4_en/parameters/. 7. Firmware Programming ======================= The adapter card was shipped with the most current firmware available. This section is intended for future firmware upgrades, and provides instructions for (1) installing Mellanox firmware update tools (MFT), (2) downloading FW, and (3) updating adapter card firmware. 7.1 Installing Firmware Tools ----------------------------- The driver package compiles and installs the Mellanox 'mstflint' utility under /usr/local/bin/. You may also use this tool to burn a card-specific firmware binary image. See MSTFLINT_README.txt for details. Alternatively, you can download the current Mellanox Firmware Tools package (MFT) from http://www.mellanox.com/products/management_tools.php. The tools package to download is "MFT_SW for Linux" (tarball name is mft-2.1.0.tgz). 7.2 Downloading the Firmware Image of the Adapter Card ------------------------------------------------------ To download the correct card firmware image, please visit http://www.mellanox.com/support/firmware_download.php. For help in identifying your adapter card, please visit http://www.mellanox.com/support/HCA_FW_identification.php. 7.3 Updating Adapter Card Firmware --------------------------------- Using a card specific binary firmware image file, enter the following command: #> mstflint -d mtXXXXX_pci_cr0 -i b where XXXXX stands for the PCI Device ID (decimal). For burning firmware using the MFT package, please check the MFT user's manual under http://www.mellanox.com/products/management_tools.php. 8. Known Issues =============== - For RedHat EL4, adding and removing multiple vlan interfaces over the network interface created by the mlx4_en driver may lead to printing the following: "kernel: unregister_netdevice: waiting for eth to become free. Usage count =" - iperf with multiple (> 100) streams may fail on kernel.org 2.6.25 versions earlier than 2.6.25.9. 9. Revision History =================== * Rev 1.0 (July 2008)