Mellanox Technologies ===================== ======================================================================== = MLNX_EN for Windows Driver Kit README = = Driver version: 1.3.0 = = Document No. 3028 ; August 30, 2009 = ======================================================================== Contents: ========= 1. Overview 2. Important Notes 3. Supported Platforms and Operating Systems 4. Driver Software 4.1 Overview 4.2 Installing the Driver 4.3 Performance Remarks 4.4 Limitations and Known Issues 4.5 Troubleshooting 5. Revision History 1. Overview =========== This is the README document for MLNX_EN for Windows, the driver kit for Mellanox ConnectX(R) based adapter cards. The driver is intended for cards that identify on the PCI bus as one of the following PCI Device IDs (decimal): 25408,25418, 25448, 25458, 26418, 26428, or 26448. 2. Important Notes ================== 1. This driver version requires ConnectX firmware fw-25408 v2.6.000 or later. 2. To achieve the best performance for 10Gb/s Ethernet, some TCP parameters should be reconfigured. This reconfiguration can be done using a special GUI. For details, please see the "Performance Tuning" section below. 3. Supported Platforms and Operating Systems ============================================ The following operating systems are currently supported: - Windows Server 2003 32 bits - Windows Server 2003 64 bits - Windows Server 2008 32 bits - Windows Server 2008 64 bits - Windows Server 2008-R2 64 bits 4. Driver Software ================== 4.1 Overview ------------ The Windows MLNX_EN driver release introduces the following capabilities: - One or two ports - Up to 16 Rx queues per port - Rx steering mode (RSS) - Hardware Tx/Rx checksum calculation - Large Send Offload (i.e., TCP Segmentation Offload) - Hardware multicast filtering - Adaptive interrupt moderation - Polling on send completion queue to decrease the number of interrupts (default: disabled) - Polling on receive completion queue to decrease the number of interrupts (default: disabled) - MSI-X support (only on Windows Server 2008 and higher) - VLAN Tx/Rx acceleration (HW VLAN stripping/insertion) - High Availability (HA) between ports and Mellanox NICs - Load Balancing between ports and Mellanox NICs - Quality of Service (QoS) - HW VLAN filtering - Tx arbitration mode: VLAN user-priority (off by default) 4.2: Installing the Driver ---------------------------- 1. Download the driver package from http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=28&menu_section=35 2. Download from the same Web page the document "MLNX_EN for Windows Installation Guide" and install the driver according to the instructions in the guide. 4.3 Performance Remarks ----------------------- 4.3.1 Performance Tuning ~~~~~~~~~~~~~~~~~~~~~~~~ To improve performance, activate the performance tuning tool as follows: - Click start-->Control Panel. - Open "Network Connections". - Right click on one of the entries "Mellanox ConnectX 10Gbit Ethernet Adapter" and select "Properties". - Select the "Performance tab". - Click "General Tuning". Clicking the "General Tuning" button will change several registry entries (described below), and will check for system services that may decrease performance. It will also generate a log including the changes that were made. Users can view this log to restore the previous values. The log path is: %HOMEDRIVE%\windows\system32\logfiles\performancetunning.log. This tuning is needed on one adapter only, and only once after the installation (as long as these entries are not changed directly in the registry, or by some other install or script). Please note that you may need to reboot for the changes to take effect. You will be asked to reboot if necessary. The registry entries that may be added/changed by this procedure are: a. Windows 2003: - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters: * TcpWindowSize, type REG_DWORD, value set to 512K. * Tcp1323Opts, type REG_DWORD, value set to 1. * SackOpts, type REG_DWORD, value set to 0. * EnableRss, type REG_DWORD, value set to 1. * RssBaseCpu, type REG_DWORD, value set to 1. * MaxNumRssCpus, type REG_DWORD, value set to 2. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters: * FastSendDatagramThreshold, type REG_DWORD, value set to 64K. The following service is disabled: * "Windows Firewall/Internet Connection Sharing (ICS)" b. Windows 2008 and Windows 2008-R2: - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters: * SackOpts, type REG_DWORD, value set to 0. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters: * FastSendDatagramThreshold, type REG_DWORD, value set to 64K. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Ndis\Parameters: * RssBaseCpu, type REG_DWORD, value set to 1. * MaxNumRssCpus, type REG_DWORD, value set to 2. Enabling Receive Side Scaling (RSS) is performed by means of the following command: "netsh int tcp set global rss = enabled" Disable the time stamps on both sides. This is performed by means of the following command: "netsh int tcp set global timestamps=disabled" The following services are disabled: * "Base Filtering Engine (BFE)" * "Windows Firewall (MpsSvc)" On some machines the following change may provide additional performance: - Change the send completion method from interrupt to polling. In order to do that: a. Open the device manager and select the Ethernet adapter that you are using (Mellanox ConnectX 10G Ethernet Adapter). Right click on the adapter and select Properties. b. Press the Advanced tab. c. Select Performance Options and then click Properties. d. In the new dialog that will be opened, select Send Completion Method. e. Change the value to polling. f. Click OK twice. 4.3.2 Known Performance Issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - On Intel I/OAT supported systems, it is highly recommended to install and enable the latest I/OAT driver (download from www.intel.com). - With I/OAT enabled, sending messages with length of 256 bytes or larger will activate I/OAT, causing significant latency increase, due to I/OAT algorithms. On the other hand, throughput will increase significantly when using I/OAT. - On some systems, reducing the receive ring size ("Receive Ring Size" value under the Advanced tab) may improve performance. - On some systems, changing the send completion method to polling ("Send Completion method" value under the Advanced tab) may improve performance. 4.4 Booting Windows from an iSCSI Target -------------------------------- Booting Windows from an iSCSI Target is supported on Windows Server 2008 and 2003 with the following limitations: 2008: Installing Windows Server 2008 directly to an iSCSI Target is not supported. 2003: Windows Server 2003 must be configured using a static IP address (and not through DHCP). For more details on how to boot from a SAN using a Mellanox adapter card, please refer to http://www.etherboot.org/wiki/sanboot. Also note that Mellanox has also tested the adapter card with a Windows iSCSI Target from StarWind (build 4.1). 4.5 Known Issues and Limitations -------------------------------- 1. After uninstalling the MLNX_EN for Windows package, you need to reboot the machine. 2. This release does not support installing both the MLNX_EN for Windows (Ethernet) and an InfiniBand driver (e.g., Mellanox WinOF, WinOF, Mellanox WinIB). 3. If your machine has WDF version 1.5 installed, you will need to reboot the machine after the installation completes. 4. In "Control Panel\Add or Remove Programs", the Repair option of MLNX_EN does not repair "Mellanox Virtual Miniport Adapter" drivers. Workaround: Uninstall the MLNX_EN for Windows package and reinstall it. 5. VLAN creation: VLANs on the same machine cannot be assigned dynamic IPs (using DHCP) of the same subnet. 6. Bundle creation (LBFO): a. After creating a bundle, all adapters in the bundle still appear in the Network Connections display. b. Once an adapter is part of a bundle, the following parameters cannot be changed: task offloading, RSS mode or MTU. To workaround this issue you need to (a) disassemble the bundle, (b) set the parameters for all the bundle adapters to the SAME values, and (c) reassemble the bundle. c. When creating a new bundle, it may take some time (up to one minute) until the OS presents the new bundle. d. When VLAN is present over the network adapter, LBFO will exclude this network adapter from the adapters list in LBFO GUI. 7. Replacing an adapter card may require reconfiguring LBFO bundles and/or VLAN adapters. 8. Uninstalling "Mellanox Virtual Miniport Driver" from a "Network properties" page may not work, and VLANs and LBFO bundles may still exist. Workaround: a. Open the device manager and select the Ethernet adapter that you are using (Mellanox ConnectX 10G Ethernet Adapter). b. Double-click on the adapter and select the VLAN tab, then remove all existing VLANs. c. Select the LBFO tab, then remove all existing bundles. 4.6 Troubleshooting ------------------- Issue: The installation of MLNX_EN for Windows fails with the following (or a similar) error message: "This installation package is not supported by this processor type. Contact your product vendor." Suggestion: This message is printed if you have downloaded and attempted to install an incorrect MSI -- for example, if you are trying to install a 64-bit MSI on a 32-bit machine (or vice versa). Issue: The performance is low. Suggestion: This can be due to non-optimal system configuration. See the section "Performance Tuning" to take advantage of Mellanox 10 GBit NIC performance. Issue: The driver doesn't start. Suggestion: This can happen due to an RSS configuration mismatch between the TCP stack and the Mellanox adapter. To confirm this scenario, open the event log and look under "System" for the "mlx4eth5" or "mlx4eth6" source. If found, enable RSS as follows: a. For Windows 2003, in the TCP registry set the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRss registry value to 1. b. For windows 2008, run the following command: "netsh int tcp set global rss = enabled". Another option, which is less recommended and will cause low performance, is to disable RSS on the adapter. To do this set RSS mode to "No Dynamic Rebalancing". Issue: The Ethernet driver fails to start. In the Event log, under the mlx4_bus source, the following error message appears: RUN_FW command failed with error -22 Suggestion: The error message indicates that the wrong firmware image has been programmed on the adapter card. See http://www.mellanox.com/content/pages.php?pg=firmware_download Issue: The Ethernet driver fails to start. A yellow sign appears near the "Mellanox ConnectX 10Gb Ethernet Adapter" in the Device Manager display. Suggestion: This can happen due to a hardware error. Try to disable and re-enable "Mellanox ConnectX Adapter" from the Device Manager display. Issue: No connectivity to a Fault Tolerance bundle while using network capture tools (e.g., Wireshark). Suggestion: This can happen if the network capture tool captures the network traffic of the non-active adapter in the bundle. This is not allowed since the tool sets the packet filter to "promiscuous", thus causing traffic to be transferred on multiple interfaces. Close the network capture tool on the physical adapter card, and set it on the LBFO interface instead. Issue: No Ethernet connectivity on 1Gb/100Mb adapters after activating Performance Tuning (part of the installation). Suggestion: This can happen due to adding a TcpWindowSize registry value. To resolve this issue, remove the value key under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize or set its value to 0xFFFF. Issue: System reboots on an I/OAT capable system on Windows Server 2008. Suggestion: This may occur if you have an Intel I/OAT capable system with Direct Cache Access enabled, and 9K jumbo frames enabled. To resolve this issue, disable 9K jumbo frames. Issue: Packets are being lost. Suggestion: This may occur if the port MTU has been set to a value higher than the maximum MTU supported by the switch. Issue: Issue(s) not listed above. Suggestion: The MLNX_EN for Windows driver records events in the system log of the Windows event system. Using the event log you'll be able to identify, diagnose, and predict sources of system problems. To see the log of events, open System Event Viewer as follows: Right click on My Computer, click Manage, and then click Event Viewer. OR click start-->Run and enter "eventvwr.exe". In Event Viewer, select the system log. The following events are recorded: - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. - Failed to initialize Mellanox ConnectX EN 10Gbit Ethernet Adapter. - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. The port's network address is - The Mellanox ConnectX EN 10Gbit Ethernet was reset. - Failed to reset the Mellanox ConnectX EN 10Gbit Ethernet NIC. Try disabling then re-enabling the "Mellanox Ethernet Bus Driver" device via the Windows device manager. - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully stopped. - Failed to initialize the Mellanox ConnectX EN 10Gbit Ethernet Adapter because it uses old firmware version (). You need to burn firmware version or higher, and to restart your computer. - Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is up, and has initiated normal operation. - Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is down. This can occur if the physical link is disconnected or damaged, or if the other end-port is down. - Mismatch in the configurations between the two ports may affect the performance. When Using MSI-X, both ports should use the same RSS mode. To fix the problem, configure the RSS mode of both ports to be the same in the driver GUI. - Mellanox ConnectX EN 10Gbit Ethernet Adapter device failed to create enough MSI-X vectors. The Network interface will not use MSI-X interrupts. This may affects the performance. To fix the problem, configure the number of MSI-X vectors in the registry to be at least . 5. Revision History =================== * Aug 2009 - MLNX_EN for Windows Rev 1.3.0 * Mar 2009 - MLNX_EN for Windows Server 2008 Rev 1.2.0 * Jan 2009 - MLNX_EN for Windows Server 2003 Rev 1.1.0