Mellanox Technologies ===================== ======================================================================== = ConnectX EN 10GigE Adapter Card Driver Kit for Windows README = = Driver version: 1.1.0, svn revision 3486 = = Document No. 3008 ; November 20, 2008 = ======================================================================== Contents: ========= 1. Overview 2. Important Notes 3. Supported Platforms and Operating Systems 4. Driver Software 4.1 Overview 4.2 Downloading the Driver Package 4.3 Installing the Driver 4.4 Performance Remarks 4.5 Limitations and Known Issues 4.6 Troubleshooting 5. Revision History 1. Overview =========== This document provides information on the ConnectX EN Windows driver and instructions for installing the driver on Mellanox adapter cards supporting 10Gb/s Ethernet. The driver is intended for ConnectX based adapter cards that identify on the PCI bus as with one of the following PCI Device IDs (decimal): 25408, 25418, 25448, 26418, 26428 and 26448. 1.1 Download Path ----------------- To access the MLNX EN Windows Web page, please go to www.mellanox.com and navigate to Products --> Ethernet Software/Drivers --> MLNX_EN_Windows. 2. Important Notes ================== 1. This driver version requires ConnectX firmware fw-25408 v2.6.000 or later. 2. For achieving the best performance for 10 Gbit Ethernet, some TCP parameters should be reconfigured. This reconfiguration can be done using a special GUI. For more details on how to configure your system for best performance please refer to "Performance Tuning" section in this document. 3. Supported Platforms and Operating Systems ============================================ The following operating systems are currently supported: - Windows Server 2003 SP2 32 bits - Windows Server 2003 SP2 64 bits 4. Driver Software ================== 4.1 Overview ------------ The Windows MLNX_EN driver release introduces the following capabilities: - One or two ports - Up to 16 Rx queues per port - Rx steering mode (RSS) - Hardware Tx/Rx checksum calculation - Large Send Offload (i.e., TCP Segmentation Offload) - Hardware multicast filtering - Adaptive interrupt moderation - Polling on send completion queue to decrease the number of interrupts (default: disabled) - Polling on receive completion queue to decrease the number of interrupts (default: disabled) - VLAN Tx/Rx acceleration (HW VLAN stripping/insertion) - High Availability (HA) between ports and Mellanox NICs - Load Balancing between ports and Mellanox NICs - Quality of Service (QoS) - HW VLAN filtering - Tx arbitration mode: VLAN user-priority (off by default) 4.2: Downloading the Driver Package ------------------------------------- To download the current driver package, please go to the MLNX_EN for Windows page listed in Section 1.1 above. 4.3: Installing the Driver ---------------------------- Please see the MLNX_EN for Windows Installation Guide document available under the driver's Web page (see Section 1.1 above). NOTE: See item #2 under "Limitations and Known Issues". 4.4 Performance Remarks ----------------------- 4.4.1 Performance Tuning ~~~~~~~~~~~~~~~~~~~~~~~~ To improve performance, activate the performance tuning tool as follows: - Click start-->Control Panel. - Open "Network Connections". - Right click on one of the entries "Mellanox ConnectX 10Gbit Ethernet Adapter" and select "Properties". - Select the "Performance tab". - Click "General Tuning". Clicking the "General Tuning" button will change several registry entries (described below), and will check for system services that may decrease performance. It will also generate a log including the changes that were made. Users can view this log to restore the previous values. The log path is: %HOMEDRIVE%\windows\system32\logfiles\performancetunning.log. This tuning is needed on one adapter only, and only once after the installation (as long as these entries are not changed directly in the registry, or by some other install, or script). Please note that you may need to reboot for the changes to take effect. You will be asked to reboot if necessary. The registry entries that may be added/changed by this procedure are: - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters: * TcpWindowSize, type REG_DWORD, value set to 512K. * GlobalMaxWindowSize, type REG_DWORD, value set to 512K. * Tcp1323Opts, type REG_DWORD, value set to 1. * SackOpts, type REG_DWORD, value set to 0. * EnableRss, type REG_DWORD, value set to 1. * RssBaseCpu, type REG_DWORD, value set to 1. * MaxNumRssCpus, type REG_DWORD, value set to 2. - Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters: * FastSendDatagramThreshold, type REG_DWORD, value set to 64K. The following service is disabled: * "Windows Firewall/Internet Connection Sharing (ICS)" 4.4.2 Known Performance Issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - When using Jumbo Frames to send messages of sizes between 9561 and 16K bytes, there may be performance degradation due to NDIS Send/Receive algorithms. - On Intel I/OAT supported systems, it is highly recommended to install and enable the latest I/OAT driver. (Download from www.intel.com.) - On some systems, reducing the receive ring size ("Receive Ring Size" value under the Advanced tab) may improve performance. - If you encounter low bandwidth upon communicating with a Linux machine, please refer to the following Microsoft papers: Scalable Networking: http://technet.microsoft.com/en-us/network/bb545631.aspx Microsoft Windows Scalable Networking Pack for Windows Server 2003: http://support.microsoft.com/kb/912222 Description of Windows 2000 and Windows Server 2003 TCP Features: http://support.microsoft.com/kb/224829 Information about TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008: http://support.microsoft.com/kb/951037 Windows 2003 Scalable Networking pack and its possible effects on Exchange: http://msexchangeteam.com/archive/2007/07/18/446400.aspx Windows 2003 Scalable Networking pack and its possible effects on Exchange - Part 2: http://msexchangeteam.com/archive/2008/03/12/448421.aspx A Windows Server 2003-based computer responds slowly to RDP connections or to SMB connections that are made from a Windows Vista-based computer: http://support.microsoft.com/kb/947773 An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers: http://support.microsoft.com/?kbid=948496 4.5 Known Issues and Limitations -------------------------------- 1. After uninstalling the MLNX_EN for Windows package, you need to reboot the machine. 2. Bundle creation (LBFO): a. Upon creating a bundle of adapters in the LBFO tab, one adapter will be automatically assigned as the primary adapter in the bundle, and all the remaining will be secondary adapters. (The user cannot decide which adapter will be assigned as the primary adapter.) b. After creating a bundle, all adapters in the bundle will still appear in the Network Connections display. However, only the primary adapter will be enabled; all secondary adapters will be disabled. c. Once an adapter is part of a bundle, the following parameters cannot be changed: task offloading, RSS mode or MTU. To workaround this issue you need to (a) disassemble the bundle, (b) set the parameters for all the bundle adapters to the SAME values, and (c) reassemble the bundle. d. LBFO is of beta quality and does not support all load balancing options yet. e. In this release, it is not possible to configure a VLAN for a bundled adapter. 3. This release does not support installing both the MLNX_EN for Windows (Ethernet) and an InfiniBand driver (e.g., Mellanox WinOF, WinOF, Mellanox WinIB). 4. If your machine has WDF version 1.5 installed on it, you will need to reboot the machine after the installation completes. 5. On Windows Server 2003, the number of TCP receive rings is a parameter of the TCP stack (and NOT of NDIS). Consequently, all network adapters will have the same number of receive rings. To configure the number of receive rings, see Section 11.1 of MLNX_EN for Windows Installation Guide. 6. VLANs on the same machine cannot be assigned dynamic IPs (using DHCP) of the same subnet. 7. IPv6 was not tested 4.6 Troubleshooting ------------------- Issue: The installation of MLNX_EN for Windows fails with the following (or a similar) error message: "This installation package is not supported by this processor type. Contact your product vendor." Suggestion: This message is printed if you have downloaded and attempted to install an incorrect MSI -- for example, if you are trying to install a 64-bit MSI on a 32-bit machine (or vice versa). Issue: The performance is low. Suggestion: This can be due to non-optimal system configuration. See the section "Performance Tuning" to take advantage of Mellanox 10 GBit NIC performance. Issue: The driver doesn't start. Suggestion: This can happen due to an RSS configuration mismatch between the TCP stack and the Mellanox adapter. To confirm the reason for the problem, check the event log, and look for the "ConnectX" source in the system log. To resolve the problem, enable RSS in the TCP registry by setting the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRss registry value to 1. Another option, which is less recommended and will cause low performance, is to disable RSS on the adapter. To do this set RSS mode to "No Dynamic Rebalancing". Issue: Packet loss. Suggestion: Make sure that the port MTU has not been set to a value higher than the maximum MTU supported by the switch. Issue: No traffic on a VLAN. Suggestion: This can happen if two virtual adapters on the same port have been assigned the same VLAN ID. Other issues: Suggestion: The MLNX_EN for Windows driver records events in the system log of the Windows event system. Using the event log, you'll be able to identify and diagnose the source of system problems, and to help predict potential system problems. To see the log of events, open the System Event Viewer as follows: Right click on My Computer, click Manage, and then click Event Viewer. OR click start-->Run and enter "evenyvwr.exe". In the event viewer, select the system log The following events are recorded: - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. - Failed to initialize Mellanox ConnectX EN 10Gbit Ethernet Adapter Port . - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully initialized and enabled. The port's network address is - The Mellanox ConnectX EN 10Gbit Ethernet NIC has been reset. - Failed to reset the Mellanox ConnectX EN 10Gbit Ethernet NIC. Try disabling then re-enabling the "Mellanox Ethernet Bus Driver" device via the Windows device manager. - Mellanox ConnectX EN 10Gbit Ethernet Adapter has been successfully stopped. - Mellanox ConnectX EN 10Gbit Ethernet Adapter failed to join bundle . The adapter task offloading, RSS mode or MTU configuration does not match the primary port configuration. To fix the problem, configure the task offloading, RSS mode and MTU of all the adapters in the bundle to be the same. - Mellanox ConnectX EN 10Gbit Ethernet Adapter failed to join to bundle . Number of allowed ports connected to bundle exceeded. - Failed to initialize the Mellanox ConnectX EN 10Gbit Ethernet Adapter because it uses old firmware version (). You need to burn firmware version or higher, and to restart your computer. 5. Revision History =================== * Rev 1.1.0, svn revision 3486 (November 20, 2008)