Mellanox Technologies ===================== ======================================================================== = ConnectX EN 10GigE Adapter Card Driver Kit for Windows README = = Driver version: 1.2.0, svn revision 3971-4017 = = Document No. 3028 ; March 8, 2009 = ======================================================================== Contents: ========= 1. Overview 2. Important Notes 3. Supported Platforms and Operating Systems 4. Driver Software 4.1 Overview 4.2 Downloading the Driver Package 4.3 Installing the Driver 4.4 Performance Remarks 4.5 Limitations and Known Issues 4.6 Troubleshooting 5. Revision History 1. Overview =========== This document provides information on the ConnectX EN Windows driver and instructions for installing the driver on Mellanox adapter cards supporting 10Gb/s Ethernet. The driver is intended for ConnectX based adapter cards that identify on the PCI bus as with one of the following PCI Device IDs (decimal): 25408, 25418, 25448, 26418, 26428 and 26448. 2. Important Notes ================== 1. This driver version requires ConnectX firmware fw-25408 v2.6.000 or later. 2. LBFO is at technology preview stage. 3. For achieving best performance for 10 Gbit Ethernet, some TCP parameters should be reconfigured. This reconfiguration can be done using a special GUI. For more details on how to configure your system for best performance please refer to "Performance Tuning" section in this document. 3. Supported Platforms and Operating Systems ============================================ The following operating systems are currently supported: - Windows Server 2008 32 bits - Windows Server 2008 64 bits 4. Driver Software ================== 4.1 Overview ------------ The Windows MLNX_EN driver release introduces the following capabilities: - One or two ports - Up to 16 Rx queues per port - Rx steering mode (RSS) - Hardware Tx/Rx checksum calculation - Large Send Offload (i.e., TCP Segmentation Offload) - Hardware multicast filtering - Adaptive interrupt moderation - Polling on send completion queue to decrease the number of interrupts (default: disabled) - Polling on receive completion queue to decrease the number of interrupts (default: disabled) - MSI-X support - VLAN Tx/Rx acceleration (HW VLAN stripping/insertion) - High Availability (HA) between ports and Mellanox NICs - Load Balancing between ports and Mellanox NICs - Quality of Service (QoS) - HW VLAN filtering - Tx arbitration mode: VLAN user-priority (off by default) 4.2: Downloading the Driver Package ------------------------------------- The current driver package can be downloaded from http://docs.mellanox.com under Code Releases->MLNX EN for Windows->Windows Server 2008. NOTE: Please contact your Mellanox FAE for access. 4.3: Installing the Driver ---------------------------- Please see the MLNX_EN for Windows Installation Guide document available from http://docs.mellanox.com under Code Releases->MLNX EN for Windows->Windows Server 2008. 4.4 Performance Remarks ----------------------- 4.4.1 Performance Tuning ~~~~~~~~~~~~~~~~~~~~~~~~ To improve performance, activate the performance tuning tool as follows: - Click start-->Control Panel. - Open "Network Connections". - Right click on one of the entries "Mellanox ConnectX 10Gbit Ethernet Adapter" and select "Properties". - Select the "Performance tab". - Click "General Tuning". Clicking the "General Tuning" button will change several registry entries (described below), and will check for system services that may decrease performance. It will also generate a log including the changes that were made. Users can view this log to restore the previous values. The log path is: %HOMEDRIVE%\windows\system32\logfiles\performancetunning.log. This tuning is needed on one adapter only, and only once after the installation (as long as these entries are not changed directly in the registry, or by some other install or script). Please note that you may need to reboot for the changes to take effect. You will be asked to reboot if necessary. The registry entries that may be added/changed by this procedure are: - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters: * SackOpts, type REG_DWORD, value set to 0. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters: * FastSendDatagramThreshold, type REG_DWORD, value set to 64K. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Ndis\Parameters: * RssBaseCpu, type REG_DWORD, value set to 1. * MaxNumRssCpus, type REG_DWORD, value set to 2. Enabling Receive Side Scaling (RSS) is performed by means of the following command: "netsh int tcp set global rss = enabled" The following service is disabled: * "Base Filtering Engine (BFE)" * "Windows Firewall (MpsSvc)" On some machines the following changes may provide additional performance: 1. Change the send completion method from interrupt to polling. In order to do that: a. Open the device manager and select the Ethernet adapter that you are using (Mellanox ConnectX 10G Ethernet Adapter) and press right click properties. b. Press the advanced tab. c. Select performance options and then press properties. d. On the new dialog that will be opened, please select send completion method. e. Change the value to polling. f. Press OK twice. 2. Disable the time stamps on both sides. This is performed by means of the following command: "netsh int tcp set global timestamps=disabled" 4.4.2 Known Performance Issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - On Intel I/OAT supported systems, it is highly recommended to install and enable the latest I/OAT driver. (Download from www.intel.com.) - With I/OAT enabled, sending messages with length of 256 bytes or larger will activate I/OAT, causing significant latency increase, due to I/OAT algorithms. On the other hand, throughput will increase significantly when using I/OAT. - On some systems, reducing the receive ring size ("Receive Ring Size" value under the Advanced tab) may improve performance. - On some systems, changing the send completion method to polling ("Send Completion method" value under the Advanced tab) may improve performance. 4.5 Known Issues and Limitations -------------------------------- 1. After uninstalling the MLNX_EN for Windows package, you need to reboot the machine. 2. This release does not support installing both the MLNX_EN for Windows (Ethernet) and an InfiniBand driver (e.g., Mellanox WinOF, WinOF, Mellanox WinIB). 3. VLAN creation: a. Help windows may not update and remain in "Please wait until the VLAN installation completes..." following a new VLAN creation although operation has completed. b. Uninstalling "Mellanox Virtual Miniport Driver" from a "Network properties" page may leave system with "Mellanox Virtual Miniport Driver". c. Maximal number of VLANs is 64. d. Frequent pressing on the OK button when creating a new VLAN may cause VLAN creation to fail. e. VLANs on the same machine cannot be assigned dynamic IPs (using DHCP) of the same subnet. 4. Bundle creation (LBFO): LBFO is at technology preview stage. The following are currently know issues of LBFO: a. Disabling and re-enabling a physical adapter after defining a team may lead to a blue screen. b. Upon creating a bundle of adapters in the LBFO tab, one adapter will be automatically assigned as the primary adapter in the bundle, and all the remaining will be secondary adapters. (The user cannot decide which adapter will be assigned as the primary adapter.) c. After creating a bundle, all adapters in the bundle still appear in the Network Connections display. d. Once an adapter is part of a bundle, the following parameters cannot be changed: task offloading, RSS mode or MTU. To workaround this issue you need to (a) disassemble the bundle, (b) set the parameters for all the bundle adapters to the SAME values, and (c) reassemble the bundle. e. When creating a new bundle, it may take some time (up to one minute) until the OS presents the new bundle. f. When creating a new bundle, user may need to disable and enable the physical adapters assembling the bundle. g. LBFO does not support all load balancing options yet. h. SFT has not been fully tested. i. VLAN over LBFO may not function correctly. 5. If your machine has WDF version 1.5 installed on it, you will need to reboot the machine after the installation completes. 6. IPv6 has not been tested. 7. In "Control Panel\Programs and Features", Repair option of MLNX_EN does not repair "Mellanox Virtual Miniport Adapter" drivers. User should use Uninstall followed by install instead. 8. When using "Mellanox ConnectX 10G Ethernet Adapter" properties window, trying to change MTU, RX and TX rings size simultaneously may leave Device Manager hanging. User can change each value separately. 4.6 Troubleshooting ------------------- Issue: The installation of MLNX_EN for Windows fails with the following (or a similar) error message: "This installation package is not supported by this processor type. Contact your product vendor." Suggestion: This message is printed if you have downloaded and attempted to install an incorrect MSI -- for example, if you are trying to install a 64-bit MSI on a 32-bit machine (or vice versa). Issue: The performance is low. Suggestion: This can be due to non-optimal system configuration. See the section "Performance Tuning" to take advantage of Mellanox 10 GBit NIC performance. Issue: The driver doesn't start. Suggestion: This can happen due to an RSS configuration mismatch between the TCP stack and the Mellanox adapter. To confirm the reason for the problem, check the event log, and look for the "ConnectX" source in the system log. To resolve the problem, enable RSS by means of the following command: "netsh int tcp set global rss = enabled". Another option, which is less recommended and will cause low performance, is to disable RSS on the adapter. To do this set RSS mode to "No Dynamic Rebalancing". Issue: Packet loss Suggestion: Make sure that the port MTU has not been set to a value higher than the maximum MTU supported by the switch. Issue: No traffic on a VLAN. Suggestion: This can happen if two virtual adapters on the same port have been assigned the same VLAN ID. Issue: System reboots on an I/OAT capable system Suggestion: If you have an Intel I/OAT capable system with Direct Cache Access enabled, do not enable 9K jumbo frames on Windows Server 2008. Doing so may cause the system to reboot under normal to heavy receive traffic. Other issues: Suggestion: The MLNX_EN for Windows driver records events in the system log of the Windows event system. Using the event log, you'll be able to identify and diagnose the source of system problems, and to help predict potential system problems. To see the log of events, open the System Event Viewer as follows: Right click on My Computer, click Manage, and then click Event Viewer. OR click start-->Run and enter "evenyvwr.exe". In the event viewer, select the system log The following events are recorded: - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. - Failed to initialize Mellanox ConnectX EN 10Gbit Ethernet Adapter Port . - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. The port's network address is - The Mellanox ConnectX EN 10Gbit Ethernet NIC has been reset. - Failed to reset the Mellanox ConnectX EN 10Gbit Ethernet NIC. Try disabling then re-enabling the "Mellanox Ethernet Bus Driver" device via the Windows device manager. - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully stopped. - Mellanox ConnectX EN 10Gbit Ethernet Adapter failed to join bundle . The adapter task offloading, RSS mode or MTU configuration does not match the primary port configuration. To fix the problem, configure the task offloading, RSS mode and MTU of all the adapters in the bundle to be the same. - Mellanox ConnectX EN 10Gbit Ethernet Adapter failed to join to bundle . Number of allowed ports connected to bundle exceeded. - Failed to initialize the Mellanox ConnectX EN 10Gbit Ethernet Adapter because it uses old firmware version (). You need to burn firmware version or higher, and to restart your computer. 5. Revision History =================== * Rev 1.2.0, svn revision 3971-4017 (March 8, 2009)