HPCX v2.0 README

Mellanox HPCX package contains following pre-compiled HPC packages:

* OpenMPI and OpenSHMEM v3.x from https://github.com/open-mpi/ompi/
* MXM GA
* UCX GA
* FCA v3.x (code name: "hcoll")
* libibprof (IB verbs profiler)
* MPI tests (OSU, IMB, random ring, basic examples, ...)
* MPI profiler (IPM)
* Berkeley UPC v2.18.0
* knem (http://runtime.bordeaux.inria.fr/knem/)

1. Install HPCX

   a) Extract tarball and set environment variable to HPCX location

    % tar zxvf hpcx.tar
    % cd hpcx
    % export HPCX_HOME=$PWD

   b) Load KNEM (High-Performance Intra-Node Communication Device)

   Please run following commands on all cluster nodes to enable KNEM intra-node device:

    # insmod $HPCX_HOME/knem/lib/modules/$(uname -r)/knem.ko
    # chmod 666 /dev/knem

2. Load HPCX environment from bash

    % source $HPCX_HOME/hpcx-init.sh
    % hpcx_load
    % env | grep HPCX
    % mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_usempi
    % oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem
    % hpcx_unload

3. Load HPCX environment from modules

    % module use $HPCX_HOME/modulefiles
    % module load hpcx
    % mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c
    % oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem
    % module unload hpcx

* IMPORTANT NOTE:
    
    When HPC-X is launched in an environment where there is no resource manager
    (slurm, pbs, ...) installed or if you are launching from compute node.

    In this case it will use OMPI default rsh/ssh based launcher which
    does not propagate the library path to compute nodes.

    Please pass LD_LIBRARY_PATH variable as follows:

    % mpirun -x LD_LIBRARY_PATH -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c

4. Rebuild HPCX packages from sources

The sources for SHMEM, OMPI and UCX can be found at $HPCX_HOME/sources/
Check $HPCX_HOME/sources/config.log for build details.


* The HPCX package contains OpenMPI sources which can be found at $HPCX_HOME/sources/ folder.

* The HPCX should be loaded to the user environment prior recompiling HPC-X from
  sources, see section #2 above.

    % HPCX_HOME=/path/to/extracted/hpcx
    % source $HPCX_HOME/hpcx-init.sh
    % hpcx_load
    % cd $HPCX_HOME/sources/
    % tar -zxf openmpi-gitclone.tar.gz
    % cd openmpi-gitclone
    % ./configure --prefix=${HPCX_HOME}/hpcx-ompi --with-knem=${HPCX_HOME}/knem \
                --with-mxm=${HPCX_HOME}/mxm \
                --with-hcoll=${HPCX_HOME}/hcoll \
                --with-ucx=${HPCX_HOME}/ucx \
                --with-platform=contrib/platform/mellanox/optimized \
                --with-slurm --with-pmi
    % make -j9 all && make -j9 install

  After compiling OMPI, HPC-X needs to be reloaded in order to work with the newly compiled OMPI.
  First unload the previous HPC-X.

  If loaded HPC-X from bash:

    hpcx_unload

    In the $HPCX_HOME/hpcx-init.sh file, change:
    export HPCX_MPI_DIR=$HPCX_DIR/ompi-v3.0.x
    to
    export HPCX_MPI_DIR=$HPCX_DIR/hpcx-ompi

    Then repeat #2.

  If loaded HPC-X from modules:

    module unload hpcx

    In the $HPCX_HOME/modulefiles/hpcx file, change:
    set hpcx_mpi_dir    $hpcx_dir/ompi-v3.0.x
    to
    set hpcx_mpi_dir    $hpcx_dir/hpcx-ompi

    Then repeat #3.

  Please note that the tests that are part of the HPC-X toolkit will remain in directroy of the
  previous OMPI.

* IMPORTANT NOTE FOR STATIC COMPILATION:

Static compilation with mxm/fca/hcoll/ucx products can be achieved with the following
procedure:

a. Load HPCX from package supplied module file or from shell source script
b. Run following script to fix libtool .la file with current HPCX location

    % $HPCX_DIR/utils/hpcx_fix_ladir.sh


5. Profile IB verbs API

    % export IBPROF_DUMP_FILE=ibprof_%J_%H_%T.txt
    % export LD_PRELOAD=$HPCX_IBPROF_DIR/lib/libibprof.so:$HPCX_MXM_DIR/lib/libmxm.so:$HPCX_HCOLL_DIR/lib/libhcoll.so 
    % mpirun -x LD_PRELOAD 

6. Profile MPI API

    % export IPM_KEYFILE=$HPCX_IPM_DIR/etc/ipm_key_mpi
    % export IPM_LOG=FULL
    % export LD_PRELOAD=$HPCX_IPM_DIR/lib/libipm.so 
    % mpirun -x LD_PRELOAD 
    % $HPCX_IPM_DIR/bin/ipm_parse -html outfile.xml

7. Run MPI with debug version of MXM

    % LD_PRELOAD=$HPCX_DIR/mxm/debug/lib/libmxm.so mpirun -x LD_PRELOAD 

8. Run OpenSHMEM job with debug version of MXM

    % LD_PRELOAD=$HPCX_DIR/mxm/debug/lib/libmxm.so oshrun -x LD_PRELOAD 

9. Run MPI with debug version of UCX

    % LD_PRELOAD=$HPCX_DIR/ucx/debug/lib/libucp.so mpirun -x LD_PRELOAD 

10. Run OpenSHMEM job with debug version of UCX

    % LD_PRELOAD=$HPCX_DIR/ucx/debug/lib/libucp.so oshrun -x LD_PRELOAD 


11. Generate MXM statistics for MPI/OpenSHMEM job

In order to generate statistics need to set statistics destination and trigger.

Destination is set by MXM_STATS_DEST environment variable.

It can be one of the following:

* empty string - statistics are not reported.
* stdout - Print to standard output.
* stderr - Print to standard error.
* file: - Write to a file. Following substitutions are made: %h: host, %p: pid, %c: cpu, %t: time, %e: exe.
* file::bin - Same as previous, but use a binary format when saving. The file will be smaller, but not human-readable. 
The mxm_stats_parser tool can be used to parse binary statistics files.

Examples:

    % export MXM_STATS_DEST="file:mxm_%h_%e_%p.stats"
    % export MXM_STATS_DEST="file:mxm_%h_%c.stats:bin"
    % export MXM_STATS_DEST="stdout"

Trigger is set by MXM_STATS_TRIGGER.

It can be one of the following:
* exit              - Dump statistics just before exiting the program.
* timer:  - Dump statistics periodically, interval is given in seconds.

Example:

    % export MXM_STATS_TRIGGER=exit
    % export MXM_STATS_TRIGGER=timer:3.5

12. Run MPI job with FCA v3.x (hcoll)

*** Run with defaults

%  mpirun -mca coll_hcoll_enable 1 -x HCOLL_MAIN_IB=mlx5_0:1 

13. HPCx tuning examples

# Run MPI job, map ranks to cores starting from ones which are closer to HCA,
# then use byslot policy

% mpirun --map-by dist -mca rmaps_dist_device mlx5_0:1  

# Run MPI job, map ranks to cores starting from ones which are closer to HCA,
# then use bynode policy

% mpirun --map-by dist:span -mca rmaps_dist_device mlx5_0:1  

14. Performance tuning notes

** Distribute IB IRQs across all available cores (by default it is handled by
core0). It is needed for MellanoxOFED 2.4 or earlier.

# cp $HPCX_DIR/utils/mlnx_set_irq_affinity.pl /etc/infiniband/post-start-hook.pl
# /etc/infiniband/post-start-hook.pl

** Examples of the script hooks to run from job scheduler on HPC job start/stop

Check $HPCX_DIR/utils/rc.local_mellanox script which can be used from
prolog/epilog/onboot scripts

15. HPC-X supports SHARP Accelerated Collectives. These collectives are enabled by
default if FCA 3.5 (HCOLL) and above.

# To enable SHARP acceleration:

% mpirun -x HCOLL_ENABLE_SHARP=1 

# To disable SHARP acceleration:

% mpirun -x HCOLL_ENABLE_SHARP=0 

For instructions on how to deploy SHARP in Infiniband fabric, please see the
SHARP Deployment Guide.
