Linux Distribution Installation over IPoIB Interface
                                      README
                                     May 2011
-------------------------------------------------------------------------------

Content:
-------------------------------------------------------------------------------
1. Overview
2. Installation Instructions
   2.1 How to Install CentOS5.5 (x86_64) over an IPoIB Interface
   2.2 How to Install RHEL5.4 (x86_64) over an IPoIB Interface
   2.3 How to Install RHEL5.4 (i386) over an IPoIB Interface
   2.4 How to Install RHEL5.5 (x86_64) over an IPoIB Interface
   2.5 How to Install RHEL5.5 (i386) over an IPoIB Interface
   2.6 How to Install RHEL5.6 (x86_64) over an IPoIB Interface
   2.7 How to Install RHEL5.6 (i386) over an IPoIB Interface
   2.8 How to install SLES11/SLES11 SP1 over an IPoIB Interface
3. Example of /etc/dhcpd.conf


1. Overview
===============================================================================
This package provides support for the PXE installation and boot of diskless
clients over IPoIB interfaces.

2. Installation Instructions
===============================================================================
2.1 How to Install CentOS5.5 (x86_64) over an IPoIB Interface
-------------------------------------------------------------------------------
1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-23.el5.x86_64.rpm  
   dhcp-3.0.5-23.el5.x86_64.rpm  dhcp-devel-3.0.5-23.el5.x86_64.rpm  
   libdhcp4client-3.0.5-23.el5.x86_64.rpm  
   libdhcp4client-devel-3.0.5-23.el5.x86_64.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already):  
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart

5. Configure IPoIB interface (ib0) with the static IP address 
  (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)

6. Configure TFTP server.
   
   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT CentOS5.5x64
   prompt 1
   timeout 1400    # 140 sec
   display message.txt
  
   LABEL CentOS5.5x64
          kernel CentOS/CentOS5.5/vmlinuz.CentOS5.5x64
          append initrd=CentOS/CentOS5.5/initrd.CentOS5.5x64.img ksdevice=ib0
   ----------------------------------------------------------------------------

7. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.CentOS5.5x64.img  vmlinuz.CentOS5.5x64 /tftpboot/CentOS/CentOS5.5/

8. Start DHCP server:
   $ dhcpd -d ib0

===============================================================================
2.2 How to Install RHEL5.4 (x86_64) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-21.el5.x86_64.rpm 
   dhcp-3.0.5-21.el5.x86_64.rpm dhcp-devel-3.0.5-21.el5.x86_64.rpm 
   libdhcp4client-3.0.5-21.el5.x86_64.rpm 
   libdhcp4client-devel-3.0.5-21.el5.x86_64.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.4 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart

5. Configure IPoIB interface (ib0) with the static IP address
   (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)
6. Configure TFTP server.
  
   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.4x64
   prompt 1
   timeout 1400    # 140 sec
   display message.txt
  
   LABEL RHEL5.4x64
          kernel RH/RH5.4/vmlinuz.RHEL5.4x64
          append initrd=RH/RH5.4/initrd.RHEL5.4x64.img ksdevice=ib0
   ----------------------------------------------------------------------------
   
7. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.4x64.img  vmlinuz.RHEL5.4x64 /tftpboot/RH/RH5.4/

8. Start DHCP server:
   $ dhcpd -d ib0

9. Start PXE installation

================================================================================
2.3 How to Install RHEL5.4 (i386) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-21.el5.i386.rpm 
   dhcp-3.0.5-21.el5.i386.rpm dhcp-devel-3.0.5-21.el5.i386.rpm 
   libdhcp4client-3.0.5-21.el5.i386.rpm 
   libdhcp4client-devel-3.0.5-21.el5.i386.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.4 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart

5. Configure IPoIB interface (ib0) with the static IP address 
  (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)

6. Configure TFTP server.
   
   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.4x86
   prompt 1
   timeout 1400    # 140 sec
   display message.txt

   LABEL RHEL5.4x86
        kernel RH/RH5.4/vmlinuz.RHEL5.4x86
        append initrd=RH/RH5.4/initrd.RHEL5.4x86.img ksdevice=ib0
   ----------------------------------------------------------------------------

7. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.4x86.img  vmlinuz.RHEL5.4x86 /tftpboot/RH/RH5.4/

8. Start DHCP server:
   $ dhcpd -d ib0

9. Start PXE installation

===============================================================================
2.4 How to Install RHEL5.5 (x86_64) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-23.el5.x86_64.rpm  
   dhcp-3.0.5-23.el5.x86_64.rpm  dhcp-devel-3.0.5-23.el5.x86_64.rpm  
   libdhcp4client-3.0.5-23.el5.x86_64.rpm  
   libdhcp4client-devel-3.0.5-23.el5.x86_64.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart

5. Configure IPoIB interface (ib0) with the static IP address 
  (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)

6. Start DHCP server:
   $ dhcpd -d ib0

7. Configure TFTP server.

   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.5x64
   prompt 1
   timeout 1400    # 140 sec
   display message.txt

   LABEL RHEL5.5x64
        kernel RH/RH5.5/vmlinuz.RHEL5.5x64
        append initrd=RH/RH5.5/initrd.RHEL5.5x64.img ksdevice=ib0
   ----------------------------------------------------------------------------

8. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.5x64.img  vmlinuz.RHEL5.5x64 /tftpboot/RH/RH5.5/

9. Start PXE installation

===============================================================================
2.5 How to Install RHEL5.5 (i386) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-23.el5.i386.rpm  
   dhcp-3.0.5-23.el5.i386.rpm  dhcp-devel-3.0.5-23.el5.i386.rpm  
   libdhcp4client-3.0.5-23.el5.i386.rpm  
   libdhcp4client-devel-3.0.5-23.el5.i386.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart

5. Configure IPoIB interface (ib0) with the static IP address 
   (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)

6. Start DHCP server:
   $ dhcpd -d ib0

7. Configure TFTP server.

   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.5x86
   prompt 1
   timeout 1400    # 140 sec
   display message.txt

   LABEL RHEL5.5x86
        kernel RH/RH5.5/vmlinuz.RHEL5.5x86
        append initrd=RH/RH5.5/initrd.RHEL5.5x86 ksdevice=ib0
   ----------------------------------------------------------------------------
   
8. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.5x86  vmlinuz.RHEL5.5x86 /tftpboot/RH/RH5.5/

9. Start PXE installation

===============================================================================
2.6 How to Install RHEL5.6 (x86_64) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-23.el5_5.2.x86_64.rpm  
   dhcp-3.0.5-23.el5_5.2.x86_64.rpm  dhcp-devel-3.0.5-23.el5_5.2.x86_64.rpm  
   libdhcp4client-3.0.5-23.el5_5.2.x86_64.rpm  
   libdhcp4client-devel-3.0.5-23.el5_5.2.x86_64.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.6 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart
   
5. Configure IPoIB interface (ib0) with the static IP address
   (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)
   
6. Start DHCP server:
   $ dhcpd -d ib0

7. Configure TFTP server.
   
   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.6x64
   prompt 1
   timeout 1400    # 140 sec
   display message.txt

   LABEL RHEL5.6x64
        kernel RH/RH5.6/vmlinuz.RHEL5.6x64
        append initrd=RH/RH5.6/initrd.RHEL5.6x64.img ksdevice=ib0
   ----------------------------------------------------------------------------

8. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.6x64.img  vmlinuz.RHEL5.6x64 /tftpboot/RH/RH5.6/

9. Start PXE installation

===============================================================================
2.7 How to Install RHEL5.6 (i386) over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install (override) dhcp RPMs on the DHCP server:
   $ rpm -ihv --force dhclient-3.0.5-23.el5_5.2.i386.rpm  
   dhcp-3.0.5-23.el5_5.2.i386.rpm  dhcp-devel-3.0.5-23.el5_5.2.i386.rpm  
   libdhcp4client-3.0.5-23.el5_5.2.i386.rpm  
   libdhcp4client-devel-3.0.5-23.el5_5.2.i386.rpm

2. Install MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.6 on the DHCP/TFTP server or 
   use any other OFED or use built-in IPoIB implementation.

3. Start the driver (if not started already): 
   /etc/init.d/openibd restart

4. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart
   
5. Configure IPoIB interface (ib0) with the static IP address 
   (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)
   
6. Start DHCP server:
   $ dhcpd -d ib0

7. Configure TFTP server.

   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT RHEL5.6x86
   prompt 1
   timeout 1400    # 140 sec
   display message.txt

   LABEL RHEL5.6x86
        kernel RH/RH5.6/vmlinuz.RHEL5.6x86
        append initrd=RH/RH5.6/initrd.RHEL5.6x86.img ksdevice=ib0
   ----------------------------------------------------------------------------
   
8. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.RHEL5.6x86.img  vmlinuz.RHEL5.6x86 /tftpboot/RH/RH5.6/

9. Start PXE installation

===============================================================================
2.8 How to install SLES11/SLES11 SP1 over an IPoIB Interface
-------------------------------------------------------------------------------

1. Install MLNX_OFED_LINUX-1.5.2-2.0.0-sles<version> or 
   newer on the DHCP/TFTP server or use any other OFED or 
   use built-in IPoIB implementation.
   
2. Start the driver (if not started already): 
   /etc/init.d/openibd restart
   
3. Start Subnet Manager (OpenSM): 
   /etc/init.d/opensmd restart
   
4. Configure IPoIB interface (ib0) with the static IP address 
   (see section "Example of /etc/dhcpd.conf" which uses the IP 11.7.6.30)

5. Start DHCP server:
   $ dhcpd -d ib0
   
6. Configure TFTP server.

   Example of /tftpboot/pxelinux.cfg/0:
   ----------------------------------------------------------------------------
   DEFAULT SLES11x64
   prompt 1
   timeout 1400    # 140 sec

   LABEL SLES11x64
        kernel SLES11/x86_64/boot/x86_64/loader/linux
        append initrd=SLES11/x86_64/boot/x86_64/loader/initrd.SLES11x64 
		install=nfs://11.7.38.11/distros/SLES11/x86_64/?device=ib0 
		autoyast=nfs://11.7.38.11/distros/SLES11/x86_64/ksadd.xml
   ----------------------------------------------------------------------------

7. Place vmlinuz and initrd files under /tftpboot directory:
   $ cp initrd.SLES11x64 /tftpboot/SLES11/x86_64/boot/x86_64/loader
   /initrd.SLES11x64

8. Start PXE installation

================================================================================
3. Example of /etc/dhcpd.conf:
---------------------------------------------------------------------
   # dhcpd.conf
   # Sample configuration file for dhcpd
   ddns-update-style none ;
   ddns-updates off ;
   option space gpxe ;
   option gpxe-encap-opts code 175 = encapsulate gpxe ;
   option gpxe.bus-id code 177 = string ;
   option time-offset -8 ;
   allow booting ;
   allow bootp ;
   shared-network "IPoIB_Network" {
     subnet 11.7.0.0 netmask 255.255.0.0 {
       option dhcp-client-identifier = option dhcp-client-identifier;
       option subnet-mask 255.255.0.0 ;
       default-lease-time 86400 ;
       max-lease-time 86400 ;
       option root-path "/tftpboot" ;
       filename "pxelinux.0";
       range 11.7.6.200 11.7.6.210;
       next-server 11.7.6.30 ;
	   host sw442-ib0 {
       fixed-address 11.7.12.242 ;
       #<cid prefix>:<port gid>   
       option  dhcp-client-identifier = ff:00:00:00:00:00:02:00:00:02:c9:00:00:02:c9:03:00:0c:f2:11;
     }
   }