BX Open Fabrics Enterprise Distribution (BXOFED) FCoE in BXOFED 1.5.1 Release Notes August 2010 ============================================================================== Table of contents ============================================================================== 1. FCoE Overview 2. Supported Platforms, Operating Systems and Firmware 3. Installation 4. FCoE Basic Usage 4.1 FCoE Configuration 4.2 Starting FCoE service 4.3 Stopping FCoE service 5. FCoE Advanced Usage 5.1 Manual vHBAs control 5.2 Creating vHBAs that use priority-flow-control 5.3 Creating vHBAs that use link pause 5.4 mlx4_fc module parameters 6. Known Issues 7. Revision History =================== 1. FCoE Overview =================== The FCoE feature provided by this driver allows connecting to FC targets on FC fabric using an FCoE-capable switch or gateway. Key features include: - Complete hardware offload of SCSI operations. - Hardware offload of FC-CRC calculations. - Zero copy FC stack. - Support VLANs and PFC (Priority-flow-control, aka. PPP) The FCoE feature is based on and interacts with the Open-FCoE project. The mlx4_fc module is designed to replace the original "fcoe" module and to allow using the ConnectX-2 hardware offloads. See http://www.open-fcoe.org/ for further information on Open-FCoE project. ======================================================= 2. Supported Platforms, Operating Systems and Firmware ======================================================= - RedHat Enterprise Linux 5 updates 2, 3, and 4 - SLES 11 ================ 3. Installation ================ Step 1: Download BXOFED-1.5.1 package Step 2: Install Driver Run the following commands to install the driver: #> tar xzvf BXOFED-1.5.1.tgz #> cd BXOFED-1.5.1 #> ./install.pl Remember to choose mlx4_fc=y ===================== 4. FCoE Basic Usage ===================== After loading driver, userspace operations should create/destroy vHBAs on required Ethernet interfaces. This can be done in one of the following ways: - manually by issuing commands to the driver using simple sysfs operations - automatically by the dcbxd daemon if the interface is connected to an FCoE switch supporting DCBX negotiation of FCoE feature like Cisco Nexus. Once a vHBA is instantiated on an Ethernet interface, it will immediately attempt to log into the FC fabric. Provided that the FC fabric and FC targets are well configured, LUNs will map to SCSI disk devices (/dev/sdXXX). vHBAs instantiated automatically by the dcbxd daemon are created on a VLAN 0 interface with VLAN priority set to the value negotiated with the switch. This will take advantage of the PFC (priority-flow-control), which allows pausing the FCoE traffic when needed without pausing the entire Ethernet link. Also, with proper configuration of the FCoE switch, the link maximum bandwidth can be divided as needed between FCoE and regular Ethernet traffic. vHBAs instantiated manually are created on VLAN interfaces with any arbitrary VLAN ID and priority as well as on the regular without VLAN Ethernet interfaces. When using the regular interface, the PFC cannot be used. In this case it is highly recommended that both the FCoE switch and the mlnx_en driver be configured to use link pause (regular flow control). Otherwise any FCoE packet drop will trigger SCSI errors and timeouts. Note: The FCoE switch might need to be configured differently to allow automatic (DCBX) or manual vHBAs creation. 4.1 FCoE Configuration ----------------------------- 1. After installation edit the file /etc/mlxfc/mlxfc.conf and set the following variables: - DCBX_IFS: Provide a space separated list of Ethernet devices to monitor using the DCBX protocol for FCoE feature availability. vHBAs are automatically created on these interfaces if FCoE switch is configured for automatic FCoE negotiation. - MTU: Enter the correct value of the MTU of the Ethernet device if the default (1500) value is changed. 2. Configure the mlx4_en Ethernet driver to support PFC. a. Add to the file /etc/modprobe.conf the line options mlx4_en pfctx=0xff pfcrx=0xff b. restart the network driver 4.2 Starting FCoE service ------------------------- Verify the network is up. Run: #> modprobe mlx4_en then #> /etc/init.d/mlxfc start The vHBAs will be instantiated on the DCBX monitored interfaces, and the SCSI LUNs will be mapped. For Manually instantiating vHBAs, please see "vHBA Manual control". 4.3 Stopping FCoE service ------------------------- Run: #> /etc/init.d/mlxfc stop Note: Only when both mlxfc service is stopped and mlx4_en module is removed can the mlx4_core module be removed as well. ====================== 5. FCoE Advance Usage ====================== Advanced usage will be probably needed when connected to FCoE switches that do not support the Cisco-like FCoE DCBX auto negotiation. 5.1 Manual vHBAs control ------------------------ Manual control allows creating and destroying vHBAs, and signaling link-up and link-down to existing vHBAs using sysfs operations. The sysfs directory is located at: /sys/class/mlx4_fc and will be referred to as $FCSYSFS To create a new vHBA on Ethernet interface (e.g. eth3). Run: #> echo eth3 > $FCSYSFS/create To destroy a vHBA previously created on interface (e.g. eth3). Run: #> echo eth3 > $FCSYSFS/destroy To signal "link-up" to existing vHBA (e.g. on eth3). Run: #> echo eth3 > $FCSYSFS/link_up To signal "link-down" to existing vHBA (e.g. on eth3). Run: #> echo eth3 > $FCSYSFS/link_down 5.2 Creating vHBAs that use priority-flow-control ------------------------------------------------- To create a vHBA that uses the PFC (priority-flow-control) feature: 1. Configure the Ethernet driver to support PFC 2. Create a VLAN ethernet interface 3. Set a priority to the VLAN ethernet interface 4. Start a vHBA on that interface To configure mlx4_en Ethernet driver to support PFC: 1. Add to the file /etc/modprobe.conf the line options mlx4_en pfctx=0xff pfcrx=0xff 2. Restart the network driver. 3. Create a VLAN with ID (e.g. 200) on interface (e.g. eth3). Run: #> vconfig add eth3 200 #> ifconfig eth3.200 up 4. Map the skb priority 0 to requested vlan priority (e.g. 3). Run: #> vconfig set_egress_map eth3.200 0 3 5. Create vHBA on the vlan device, with the required priority (e.g. 3): #> echo eth3.200,3 > $FCSYSFS/create 5.3 Creating vHBAs that use link pause -------------------------------------- The mlx4_en Ethernet driver supports link pause by default. To change this setting, you can use the following command: #> ethtool -A eth [rx on|off] [tx on|off] Then create a vHBA without priority, for instance: #> echo eth3 > $FCSYSFS/create 5.5 mlx4_fc module parameters ----------------------------- The following mlx4_fc module parameters can change the behavior of the FC service: + log_exch_per_vhba: Maximum concurrent exchanges per vHBA (log). Meaning the maximum concurrents the SCSI IO operations can performed. Default is 10 + max_vhba_per_port: Maximum number of vHBAs that can be created on each port. Default is (1 << (16 - log_exch_per_vhba)) + num_reserved_xids: Maximum outstanding RFCI exchanges per vHBA Default is 255 + cmd_per_lun: Maximum outstanding SCSI commands can queue per lun. Default is 16 + frame_size: Fibre Channel frame payload size. Default is 2112 (it will be enforced to be less than L2 network mtu) ================ 6. Known Issues ================ + To use FCoE traffic on different VLAN ID, follow the manual setup. You might also need to configure the FCoE switch to manual FCoE setup. + FCoE switch changes: Some changes in FCoE switch might require driver reload. + Extensive loading/unloading driver will crash the system. + Extensive bring up/down physical link will crash the system. + Fail-over, fail-back has not been extensively tested yet. ==================== 7. Revision History ==================== * Rev 1.5.1 (Aug 2010)