Mellanox Technologies ===================== =============================================================== = MTNIC Driver for XenServer 4.1 SW/FW Installation Guide = = Document No. 2959 ; Rev 0.12 ; July 4, 2007 = =============================================================== Contents: ========= 1. Overview 2. Important Notes 3. Supported Platforms and Operating Systems 4. Software Dependencies 5. Managing Firmware 5.1 Downloading Firmware Tools Package (MFT) 5.2 Updating Adapter Card Firmware 5.3 Firmware issues 6. Driver Software 6.1 Overview 6.2 Installing and Running the Driver 6.3 Removing the Driver 6.4 Performance Remarks 6.5 Limitations and Known Issues 7. Archive 8. Revision History 1. Overview =========== This document provides instructions for installing the driver and updating the firmware image for the MNEH28-X[ST]C 10Gb/s Ethernet Adapter Card in a Linux environment. The card identifies on the PCI bus as PCI DevID 25448. It is a dual-port, high- performance, stateless offload 10Gb/s Ethernet NIC, which supports HW Tx/Rx checksums, VLAN tag insertion/removal, LSO, header separation and LRO, multiple Tx queues with QoS, multiple Rx queues and RSS/VLAN-prio steering, and advanced interrupt moderation. 2. Important Notes ================== 1. The firmware images, firmware tool package, and driver software are still in the development stage and are subject to frequent changes. At their current stage, these packages should be used for basic functionality testing only and not for extensive regression testing. 2. Do not install or upgrade a single package by itself. All firmware, firmware tools and driver software should be installed / upgraded together. 3. Supported Platforms and Operating Systems ============================================ Currently, the driver software and firmware tool can only operate in a Linux environment on an x86_64 architecture. Specifically, the following operating system and kernels are currently supported: - kernel.org: 2.6.20.x, 2.6.22.x - SLES10 - RHEL5 - RHEL4 4. Software Dependencies ======================== To run the driver software and the firmware tool package, kernel sources must be installed on the machine. 5. Updating Firmware ==================== The adapter card was shipped with firmware already installed on the adapter card. As this card firmware is still in the development stage, a newer firmware version may have been produced since card shipping. This section provides instructions for (1) installing firmware update tools, and (2) updating adapter card firmware. 5.1 Downloading Firmware Tools Package --------------------------------------- Step 1: Download Mellanox Firmware Tools (MFT) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please download the current firmware tools package (MFT) and firmware image from https://docs.mellanox.com using your account login information. Both items can be found under 2 - HW Products/ 2 - Adapter Cards/ 06 - ConnectX EN PCI Express Cards/ Firmware & Software/ Current Step 2: Install and Run MFT ~~~~~~~~~~~~~~~~~~~~~~~~~~~ To install the MFT package: > tar xzf mft-2.X.X-rcX.tgz > cd mft-2.X.X-rcX/ > ./install.sh > mst start Step 3 (Optional): Check Ethernet Device Status ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To check device status run > mst status If there are no card installation problems, the status command should produce the following output: MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST Calibre (I2C) module is not loaded MST devices: ------------ /dev/mst/mt25448_pciconf0 - PCI configuration cycles access. bus:dev.fn=04:00.0 addr.reg=88 data.reg=92 Chip revision is: A0 /dev/mst/mt25448_pci_cr0 - PCI direct access. bus:dev.fn=04:00.0 bar=0xca200000 size=0x100000 Chip revision is: A0 /dev/mst/mt25448_pci_uar0 - PCI direct access. bus:dev.fn=04:00.0 bar=0xd8800000 size=0x800000 5.2 Updating Adapter Card Firmware --------------------------------- a. tar xzf fw-25448-rel-X_X_XXX.tgz b. cd fw-25448-rel-X_X_XXX c. mlxburn -d /dev/mst/mt25448_pci_cr0 -fw fw-25448-rel.mlx -c where conf_file is MNEH28-XTC_A1.ini or MNEH28-XSC_A1.ini. 5.3 Firmware issues ------------------- - API CONFIG_PORT_ADDR_STEER is not supported - API SET_RX_RING_MCAST is not supported - API ENFORCE_TX_RING_ADDR is not supported - API ARM_RX_RING_WATER_MARK is not supported - API CLOSE_NIC is not supported. you need to do a reset 6. Driver Software ================== 6.1 Overview ------------ The MTnic driver release exposes the following capabilities: - Single/Dual port - Up to 16 Tx and Rx queues per port - 2 Rx steering modes: Receive Core Affinity (RCA - default) or VLAN user-priority - 2 Tx arbitration modes: per-core (default) or VLAN user-priority - MSI-X or INTx - HW Tx/Rx checksum calculation - Large Send Offload (a.k.a TCP Segmentation Offload) - Large Receive Offload - Multi-core NAPI support - Header seperation - VLAN Tx/Rx acceleration (HW VLAN stripping/insertion) - HW VLAN filtering - HW multicast filtering - ifconfig up/down + mtu changes (up to 10K) - Ethtool support - Net device statistics - CX4 connectors (XAUI) or XFP 6.2 Installing and Running the Driver ------------------------------------- The driver is a single kernel module. The provided install script (install.sh) builds this module out-of-kernel under /tmp/mtnic/src, and places the resulting binary, mtnic.ko, under /lib/modules//kernel/drivers/net/mtnic and executes depmod. The following steps describe how to download, install, and run the driver. Step 1: Download Driver Package ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please download the current driver package from https://docs.mellanox.com using your account login information. The package can be found under 2 - HW Products/ 2 - Adapter Cards/ 06 - ConnectX EN PCI Express Cards/ Firmware & Software/ Current Step 2: Install Driver ~~~~~~~~~~~~~~~~~~~~~~ #> tar xzf mtnic-2007XXXX-XXXX.tgz #> cd mtnic-2007XXXX-XXXX #> ./install.sh Step 3: Running Driver ~~~~~~~~~~~~~~~~~~~~~~ To load the driver run: #> modprobe mtnic The result is a new netdevice appearing in 'ifconfig -a'. To assign an IP address to the interface run: #> ifconfig eth where 'x' is the OS assigned interface number. To check driver and device information run: #> ethtool -i eth Example: #> ethtool -i eth4 driver: mtnic (MT_00000MTNIC) version: 1.5.0 (Mon Apr 16 16:21:16 2007) firmware-version: 6.1.826 bus-info: 0000:04:00.0 To query stateless offload status run: #> ethtool -k eth To set stateless offload status run: #> ethtool -K eth [rx on|off] [tx on|off] [sg on|off] [tso on|off] To query interrupt coalescing settings run: #> ethtool -c eth To set interrupt coalescing settings run: #> ethtool -C eth [rx-usecs N] [rx-frames N] [tx-usecs N] [tx-frames N] Note: usec settings correspond to the time to wait after the *last* packet sent/recieved before triggering an interrupt To query pause frame settings run: #> ethtool -a eth To set pause frame settings run: #> ethtool -A eth [rx on|off] [tx on|off] To obtain additional device statistics, run: #> ethtool -S eth The driver defaults to the following parameters: - Both ports are activated (i.e., a net device is created for each port) - 1 Tx ring is used for each port - The number of Rx rings for each port is the number of online CPUs - Rx steering mode is set to RCA - Tx selection mode is set to per-core - Per-core NAPI is enabled - LRO is enabled with 32 concurrent sessions per Rx ring These values can be changed using module parameters, which are detailed by running: #> modinfo mtnic Values of all parameters can be observed in /sys/module/mtnic/parameters/. 6.3 Removing the Driver ----------------------- To remove the driver run: #> rmmod mtnic 6.4 Performance Remarks ----------------------- For best performance, use jumbo frames (an MTU of 9600 bytes) and run: #> ./scripts/mtnic_tune.sh which tunes the TCP stack defaults to match 10Gbs speeds, and spreads interrupts evenly among the machine cores. To obtain lower latency (at the expense of lower bandwidth), interrupt moderation can be disabled using ethtool as follows: #> ethtool -C eth rx-usecs 0 rx-frames 0 tx-usecs 0 tx-frames 0 Disabling LRO (lro=0 module paramter) can further improve latency. Note: in VLAN-prio Tx mode or VLAN-prio Rx steering, the driver assigns priorities evenly among available rings (up to 8). For example, priority assignment for 3 rings is: 0-2 --> ring 0 3-4 --> ring 1 6-7 --> ring 2 To assign a unique priority per ring, load the driver with 8 Tx/Rx rings: #> modprobe mtnic rx_ring_num=8 tx_ring_num=8 VLANs and socket-->VLAN priority mappings can be configured with the Linux 'vconfig' utility. 6.5 Limitations and known issues -------------------------------- - Invoking 'ethtool -S eth' does not refresh cached statistics; run 'ifconfig eth' first to do so. - Some statistic counters in 'ethtool -S eth' (e.g., #RRUNT, #RTOTFRMS) do not display correct values. 7. Archive ========== Earlier versions of the driver software, the firmware tool and/or image can be found in https://docs.mellanox.com under 2 - HW Products/ 2 - Adapter Cards/ 06 - ConnectX EN PCI Express Cards/ Firmware & Software/ Archive 8. Revision History =================== * Rev 0.12 (July 4, 2007) - New Driver mtnic-20070604-1657 - New ConnectX EN Firmware fw-25448-rel-6_0_121 - New MFT (FW tools) mft-2.0.0-rc24 * Rev 0.11 (May 3, 2007) - New Driver mtnic-20070501-1117 - New ConnectX EN Firmware fw-25448-rel-6_0_111 - New MFT (FW tools) mft-2.0.0-rc13 * Rev 0.10 (April 16, 2007): - First revision