Mellanox Technologies
                             *********************

              InfiniBand OFED Driver for VMware(R) Infrastructure 3
                                  Release Notes
                                  Version 1.2.0
                       Original release date: November 2008

                  **      Last updated: August, 2010      **


===============================================================================
Table of Contents
===============================================================================
1. Overview
   1.1 Contents
   1.2 Supported Platforms and Guest Operating Systems
   1.3 Supported HCA Cards and Firmware Versions
   1.4 Supported Switches, Gateways and Storage
2. Installation and Usage
3. Firmware Burning
4. Special Configuration
   4.1 Modules' Parameters
   4.2 Maximum Transmit Unit (MTU)
   4.3 NetQueue
   4.4 Multiple Storage and Network Adapters
   4.5 Performance
5. Fixed Bugs
6. Known Issues


===============================================================================
1. Overview
===============================================================================
These are the release notes of "InfiniBand OFED Driver for VMware(R) 
Infrastructure 3 version 3.5".
The sources of this package are based on Linux OFED 1.3.1.
The package is provided in RPM format and documentation is available from:
http://www.mellanox.com/products/ciov_ib_drivers_vi3-1.php.

1.1 Contents
------------
The package contains the following components:
o OpenFabrics based core and Upper Layer Protocols (ULPs):
  - IB HCA drivers: mthca, mlx4
  - core
  - ULPs: IPoIB (UD mode) and SRP Initiator
o ibstat utility
o Documentation

1.2 Supported Platforms and Guest Operating Systems
---------------------------------------------------
o This package version is intended for installation on VMware ESX Server 3.5
  (all released versions).

o Supported hardware compatibility list (HCL) and guest Operating Systems:
  Please refer to VMware support documentation at
  http://www.vmware.com/support/pubs and follow the "Compatibility Guides" link.

o Tested Guest Operating Systems:
  Architecture                 OS                        Kernel
  --------------------------------------------------------------------
  x86_64         SUSE Linux Enterprise Server 10         2.6.16.21-0.8
  x86_64         Red Hat Enterprise Linux Server 5.0     2.6.18-8
  x86_64         Red Hat Enterprise Linux AS 4.0-U4      2.6.9-42
  x86_64         Microsoft Windows Server 2003 R2 SP2    
  x86_64         Microsoft Windows XP Professional SP2   
  x86            SUSE Linux Enterprise Server 10         2.6.16.21-0.8
  x86            Red Hat Enterprise Linux Server 5.0     2.6.18-8
  x86            Red Hat Enterprise Linux AS 4.0-U4      2.6.9-42
  x86            Microsoft Windows Server 2003 R2 SP2    
  x86            Microsoft Windows XP Professional SP2   

1.3 Supported Host Channel Adapters (HCAs)
------------------------------------------
This release supports Mellanox Technologies InfiniBand (IB) HCAs:
o InfiniHost(TM) III Ex (MemFree: Firmware fw-25218 Rev 5.3.000 
                     with memory: Firmware fw-25208 Rev 4.8.200)
o InfiniHost(TM) III Lx (firmware fw-25204 Rev 1.2.000)
o ConnectX(TM) IB (firmware fw-25408 Rev 2.5.000 and 2.5.900)

For the latest firmware versions, visit http://www.mellanox.com and follow
the firmware download link.

1.4 Supported Switches, Gateways and Storage
--------------------------------------------
o All production InfiniBand switches and gateways are supported.
o Tested platforms:
  - This release was tested with switches provided by Mellanox, Cisco and 
    Flextronics.
  - QLogic Gateway SilverStorm 9020
  - Storage:
    o OFED 1.3.1 SRP target
    o EMC's CLARiiON CX340 Fibre Channel RAID


===============================================================================
2. Installation and Usage
===============================================================================
This section provides brief instructions for installing and using the package.
For detailed explanations, please refer to InfiniBand OFED Drvier for
VMware Virtual Infrastructure 3.5 Installation Guide.

To install the driver package on a VMware ESX Server machine, log into the 
service console as root and execute the following commands:
1. Remove any previous InfiniBand package:
   cos# rpm -e `rpm -qa | grep commsrc-infiniband-release`
2. Identify your ESX Server build number, run:
   cos# vmware -v
3. Download the suitable RPM based on the build number of your machine, RPMs 
   for all ESX 3.5 versions are available under: 
   http://www.mellanox.com/products/ciov_ib_drivers_vi3-1.php.
4. Install the RPM:
   cos# rpm -i <RPM file>
5. Reboot your machine:
   cos# reboot

The RPM following kernel modules will be installed under the directories
/usr/lib/vmware[-debug]/vmkmod :
1. ib_basic - InfiniBand basic driver
2. ib_ipoib - IP over InfiniBand ULP
3. ib_srp - SCSI RDMA Protocol ULP

To manage the InfiniBand OFED driver and ULPs on VMware ESX Server, use the
management script located at /etc/init.d/mgmt-mlx. For help, run the script 
without flags.

To disable an InfiniBand module, use esxcfg-module utility, run:
cos# esxcfg-module {ib_basic|ib_ipoib|ib_srp} -d
run this utility with --help flag for more options.

To display the InfiniBand OFED driver details, run:
cos# rpm -qi <package name>

You can also retrieve information on InfiniBand ports available on your 
machine using 'ibstat' - for usage and help, run:
cos# ibstat -h

===============================================================================
3. Firmware Burning
===============================================================================
Use MFT tools to burn a firmware image onto the Flash of an InfiniBand device.
Download the binaries and the documents for your VMware ESX Server 3.5 machine
from: http://www.mellanox.com/products/management_tools.php

===============================================================================
4. Special Configuration
===============================================================================
The InfiniBand driver and ULPs are configured upon installation to default
values. You can modify the modules configuration manually as described below.

4.1 Modules' Parameters
-----------------------
You can use the 'esxcfg-module' utility to manually configure InfiniBand 
modules. To retrieve the list and description of supported parameters, run:
cos# vmkload_mod {ib_basic|ib_ipoib|ib_srp} -s

To set the module parameters or enable/disable the module, run:
cos# esxcfg-module {ib_basic|ib_ipoib|ib_srp} <option>
run with --help flag for usage.

Note that some module parameters must be passed as a comma-separated list of 
integers, where each integer corresponds to an InfiniBand port on the machine 
according to its PCI bus order.

4.2 Maximum Transmit Unit (MTU)
-------------------------------
On VMware ESX Server, the default MTU for IPoIB is set to 1500 bytes by 
default. On other operating systems the default value is set to 2048 bytes.
It is the administrator's responsibility to make sure that all the machines 
in the network support and work with the same MTU value.
Note that IPoIB supports larger values and allows Jumbo Frames (JF) traffic:
o Up to 2044 bytes for the InfiniHost III family
o Up to 4052 bytes for ConnectX IB family

Make sure that the firmware of the HCAs and the switches supports the MTU you
wish to set. Also configure your Subnet Manager (SM) to set the MTU value in 
the configuration file. The SM configuration for MTU value is per Partition 
Key (PKey). For example, to enable 4052-byte MTUs on a default PKey using the 
OpenSM SM, log into the Linux machine (running OpenSM) and perform the 
following commands:
1. Edit the file:
   /usr/local/ofed/etc/opensm/partitions.conf
   and include the line:
   key0=0x7fff,ipoib,mtu=5 : ALL=full;
2. Restart OpenSM:
   /etc/init.d/opensmd restart

4.3 NetQueue
------------
IPoIB supports NetQueue technology that significantly improves networking 
performance in virtualized environments. This feature is disabled by default.
NetQueue support is currently limited to specific systems - see "System 
Compatibility Guide For ESX Server 3.5" for details.
Also, it can run only on ConnectX InfiniBand devices only, so make sure that 
your HCA is compatible.

To enable NetQueue, run:
cos# esxcfg-advcfg -k TRUE netNetqueueEnabled
cos# esxcfg-module ib_basic -s '--export-ksyms=linux mlx4_msi_x=1'
cos# esxcfg-module ib_ipoib -s 'NetQueue=<ib-ports-list>'
cos# reboot

After reboot, make sure that NetQueue was successfully enabled on uplink 
vmnicX, run:
cos# cat /proc/net/ipoib/vmnicX/NetQueue

4.4 Multiple Storage and Network Adapters
-----------------------------------------
Both IPoIB and SRP modules have the ability to expose multiple adapters 
(vmhbas/vmnics) over a single InfiniBand port. By default, one vmnic/vmhba is 
configured over each physical InfiniBand port, this setting is determined by
a module parameter passed to the module:
For SRP:
cos# esxcfg-module ib_srp -s 'max_vmhbas=n'
For IPoIB:
cos# esxcfg-module ib_ipoib -s 'UplinkNum=n1,n2,..'

Then reboot your machine for the changes to take effect.

4.5 Performance
---------------
For best performance, it is recommended to use ConnectX InfiniBand cards 
since they have enhanced capabilities and offloading features.
To enhance networking performance, it is recommended to enable NetQueue as 
explained in Section 4.3 and to use Jumbo Frames as explained in Section
4.2. In addition, please read and follow the instructions in the 
"Performance Tuning Best Practices for ESX Server 3" document.


===============================================================================
5. Fixed Bugs
===============================================================================
o Fixed IPoIB/SRP uplink logical names
o Fixed "Beacon Probing" failover detection method support
o Fixed Multicast and VLAN traffic interruption after uplink change
o Some IPoIB dropped packets are now handled correctly
o IPoIB memory usage was reduced
o Bogus QP problem is now fixed
o Fixed SRP loss of connection under stress

===============================================================================
6. Known Issues
===============================================================================
The following is a list of general limitations and known issues of this 
release.

IPoIB:
o IPv6 is not supported by IPoIB.
o When using VLANs, at least one packet should be transmitted for each new 
  interface created. For example:
  1. Create a new interface for a VM (e.g., <eth1> with IP address <ip1>).
  2. To guarantee that a packet is transmitted from the new interface,
     run: arping -I <eth1> <ip1> -c 1
o IPoIB module requires a lot of memory, thus some memory allocation may fail
  and you may see the following warning messages in /var/log/vmkernel:
  - "WARNING: Heap: <n>: Could not allocate <n> bytes for dynamic heap ib_ipoib"
  - "failed to create SCQ, rc -12"
  - "qpe_bulid_earp_req() failed"
  - "failed to alloc_skb"
  Workaround: 
  1. Reload IPoIB with a lower memory requirement:
     1.1. Try smaller ring sizes:
          cos# esxcfg-module ib_ipoib -s 'ipoib_sendq_size=512'
          cos# reboot
     1.2. If you are using the multiple uplinks feature per single IB port,
          try to use smaller number of uplinks:
          cos# esxcfg-module ib_ipoib -s 'ipoib_uplink_num=1'
          cos# reboot
  2. Increase the vmkernel network packet heap size:
     cos# esxcfg-advcfg -k 512 netPktHeapMaxSize
     cos# reboot
o You may see the following warning messages in /var/log/vmkernel. These 
  messages are harmless and can be ignored:
  - "arp: link address is broadcast for IP address 0.0.0.0!"
  - "post_send faild..tso hdr too long"
o A DHCP server application that is meant to manage VMs over an IPoIB fabric
  must be running from a virtual machine (and not from a native machine).
o IPoIB driver supports Unreliable Datagram (UD) mode only.

SRP:
o When shutting down ESX Server and all paths to the SRP target are dead (or 
  the SRP target is offline), the ESX Server may hang up to "dead_state_time"
  minutes. Please see the section SRP Module Parameters (above) for details.
  Workaround: Power off the ESX Server machines or wait up to "dead_state_time"
  minutes.
o After ESX Server reboot, SCSI targets connected to the SRP storage adapter
  may not be recognized.
  Workaround:
  1. Use VMware Virtual Infrastructure Client (VIC) to connect to ESX Server
  2. Go to the host "Configuration" tab
  3. Select "Storage Adapters"
  4. Press "Rescan"
o SRP storage adapters (vmhbas) appear in the VIC under "Unknown" family
  description.
o Under a heavy load of I/Os from multiple VMs running over a single ESX Server
  machine, the file system may move to a "read only" state.
  Workaround: Restart the VM.

General:
o For OFED 1.3.1 issues, check its release notes at www.openfabrics.org.
o For VI 3 issues, check its release notes at www.vmware.com.
o This (IB) package cannot be installed along with the MTNIC (EN) package.