BX Open Fabrics Enterprise Distribution (BXOFED)
                    FCoIB in BXOFED 1.5.1 Release Notes

                            August 2010


==============================================================================
Table of contents
==============================================================================

1. FCoIB Overview
2. Supported Platforms, Operating Systems and Firmware
3. Installation
4. FCoIB Basic Usage
   4.1 Starting FCoIB
   4.2 Stopping FCoIB
5. FCoIB Advanced Usage
   5.1 mlx4_fc module parameters
   5.2 mlx4_fcoib module parameters
6. Known Issues
7. Revision History

===================
1. FCoIB Overview
===================

The FCoIB feature provided by this driver allows connecting to FC targets on
FC fabric using BridgeX gateway.

Key features include:
- Complete hardware offload of SCSI operations.
- Hardware offload of FC-CRC calculations.
- Zero copy FC stack.

The mlx4_fcoib module uses FIP-alike protocol to discovers BridgeX gateways
in the Infiniband fabric.

The mlx4_fc module allows using the ConnectX/ConnectX-2 hardware offloads.
It also uses Fibre Channel library provided by open-fc project to send
and receive ELS, BLS

See http://www.open-fcoe.org/ for further information on Open-FCoE project.


=======================================================
2. Supported Platforms, Operating Systems and Firmware
=======================================================
- RedHat Enterprise Linux 5 updates 2, 3, and 4
- SLES 11


================
3. Installation
================
Step 1: Download BXOFED-1.5.1 package

Step 2: Install Driver
   Run the following commands to install the driver:
   #> tar xzvf BXOFED-1.5.1.tgz
   #> cd BXOFED-1.5.1
   #> ./install.pl

   Remember to choose mlx4_fc=y

=====================
4. FCoIB Basic Usage
=====================
After loading, log in and create a vHBA. 
The driver automatically discovers BridgX gateways.
Once a vHBA is instantiated on an Infiniband interface, it will immediately
attempt to log into the FC fabric. Provided that the FC fabric and FC targets
are well configured, LUNs will map to SCSI disk devices (/dev/sdXXX).

4.1 Starting FCoIB
--------------------
Verify the InfiniBand network is up and is configured by a
Subnet Manager (ie. opensm)
  
   #> sminfo
   #> ibstat (making sure ports state is ACTIVE)

   Then
   
   #> modprobe mlx4_fcoib
   
The vHBAs will be instantiated on active ports, and SCSI LUNs
   will be mapped.

   For Manually instantiating vHBAs, please see "vHBA Manual control".

4.2 Stopping FCoIB
-------------------------
Run:
   
   #> modprobe -r mlx4_fcoib


======================
5. FCoIB Advance Usage
======================

5.1 mlx4_fc module parameters
-----------------------------
The following mlx4_fc module parameters can change the behavior of the FC service:

+ log_exch_per_vhba: Maximum concurrent exchanges per vHBA (log).
  Meaning the maximum concurrents the SCSI IO operations can performed.
  Default is 10

+ max_vhba_per_port: Maximum number of vHBAs that can be created on each port.
  Default is (1 << (16 - log_exch_per_vhba))

+ num_reserved_xids: Maximum outstanding RFCI exchanges per vHBA
  Default is 255

+ cmd_per_lun: Maximum outstanding scsi commands can queue per lun.
  Default is 16

+ frame_size: Fibre Channel frame payload size.
  Default is 2112 (it will be enforced to be less than L2 network mtu)


5.2 mlx4_fcoib module parameters
---------------------------------
fip_debug_level: Set verbosity level of FIP-like protocol debug


================
6. Known Issues
================
- Extensive loading/unloading driver will crash the system
- Extensive bring up/down physical link will crash the system
- Fail-over, fail-back has not been extensively tested yet.
- FCoE/FCoIB host stack is in beta level. It is unstable, and might crash
  under the following conditions:
  o Extensive driver unload/load
  o Extensive GW stop/start, protocol change, port enable/disable, etc.
  o Extensive port up/down
  o Many vHBAs opened on single host
  o Many LUNs mapped to single host

- WWN allocation and persistency:
  o WWN is based on port GUID, and GW port ID.
  o Mapping function: GUID=<OUI-3B>:<Type-2B>:XX:YY:ZZ will translate 
    to WWN=2<GW port ID 12bit>:<OUI-3B>:XX:YY:ZZ
    Example: GUID 00:02:C9:03:00:11:22:33 -> WWN 20:04:00:02:c9:11:22:33
    (GW port ID = 4)
  o For Mellanox CX based devices, this mapping provides unique WWN.
  o Using <GW port ID> in WWN ensures that same host creates the same vHBA 
    WWN on the same FC switch

- Usage fcoib: modprobe mlx4_fcoib
  o New vHBAs created are visible in /sys/class/fc_host, /sys/module/mlx4_fc

- Usage fcoe: modprobe mlx4_fc && echo eth3.200,3 > /sys/class/mlx4_fc/create
  o New vHBAs created are visible in /sys/class/fc_host, /sys/module/mlx4_fc

- Using pre-T11 for CX A0 hardware:
  o When at least one host is CX A0, BX and ALL hosts need to use pre-T11 
    frame format.
  o Set pre-T11 frame format on BX system (see FabricIT-BXM User's Manual)
  o On hosts, add options mlx4_core enable_pre_t11_mode=1 in /etc/modprobe.conf
  o Restart hosts.

====================
7. Revision History
====================
* Rev 1.5.1 (Aug 2010)