Quantum Espresso 6.8 with GPU support (HPE)

Webpage

https://www.quantum-espresso.org/

Version

6.8

Build Environment

  • NVIDIA HPC SDK 22.11 (nompi version)
  • Intel MKL 2022.2.1
  • HPC-X 2.11 (Open MPI 4.1.4)
    • HPC-X 2.13.1 was used in the actual build. Runtime library was switched to HPC-X 2.11 later. In the following, it is assumed that HPC-X 2.11 was used in the build.

Files Required

  • qe-6.8-ReleasePack.tgz

Build Procedure

#!/bin/sh

QE_VERSION=6.8
BASEDIR=/home/users/${USER}/Software/QE/${QE_VERSION}
TARBALL=${BASEDIR}-hpe/qe-${QE_VERSION}-ReleasePack.tgz

INSTDIR=/apl/qe/6.8-gpu
PARALLEL=12

CUDA_HOME=/apl/nvhpc/22.11/Linux_x86_64/22.11/cuda/11.8

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/22.11-nompi
module -s load mkl/2022.2.1
module -s load openmpi/4.1.5-hpcx/nv22

export LANG=C
export LC_ALL=C
ulimit -s unlimited

cd ${INSTDIR}
if [ -d qe-${QE_VERSION} ]; then
  mv qe-${QE_VERSION} qe-erase
  rm -rf qe-erase &
fi

tar zxf ${TARBALL}
mv qe-${QE_VERSION}/* .
mv qe-${QE_VERSION}/.[a-z]* .
rmdir qe-${QE_VERSION}

sed -i -e "s/wget -O/wget –-trust-server-names -O/" \
       -e "s/curl -o/curl -L -o/" test-suite/check_pseudo.sh

export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx

FC=nvfortran F90=nvfortran F77=nvfortran CC=nvc CXX=nvc++ \
    ./configure --enable-openmp \
                --enable-openacc \
                --with-scalapack=no \
                --with-cuda=${CUDA_HOME} \
                --with-cuda-cc=80 \
                --with-cuda-runtime=11.8

# force to add curand to library list
sed -i -e "s/cusolver/cusolver,curand/" make.inc

# https://gitlab.com/QEF/q-e/-/issues/358
sed -i -e "s/libqemod.a/libqemod.a ..\/..\/upflib\/libupf.a/" \
    TDDFPT/tools/Makefile \
    GWW/simple_ip/Makefile \
    GWW/gww/Makefile \
    GWW/simple_bse/Makefile

# pwall(pw neb ph pp pwcond acfdt) cp ld1 tddfpt hp xspectra gwl
echo "==== all ===="
make -j${PARALLEL} all

cd test-suite
make pseudo
#make run-tests-serial
#make clean
#make run-tests-parallel
cd ..

Notes

  • Tests were done at GPU equipped computation nodes (GPGPU not available in login nodes).
    • (cd test-suite && make pseudo was performed at ccfep.)
    • beef-spin, beef, and vdw-d3 of pw_vdw tests failed. Changing compiler options and MPI implementation do not help.
  • Multi-node runs failed with HPC-X 2.13.1. Switching runtime library to HPC-X 2.11 fixed the problem.
View PDF