Quantum Espresso 7.2 with GPU support (HPE)

Webpage

https://www.quantum-espresso.org/

Version

7.2

Build Environment

  • NVIDIA HPC SDK 22.11 (nompi version)
  • Intel MKL 2023.1.0
  • OpenMPI 3.1.6

Files Required

  • qe-7.2-ReleasePack.tgz

Build Procedure

#!/bin/sh

QE_VERSION=7.2
BASEDIR=/home/users/${USER}/Software/QE/${QE_VERSION}
#TARBALL=${BASEDIR}/q-e-qe-${QE_VERSION}.tar.gz
TARBALL=${BASEDIR}/qe-${QE_VERSION}-ReleasePack.tar.gz

INSTDIR=/apl/qe/7.2-gpu
PARALLEL=12

CUDA_HOME=/apl/nvhpc/22.11/Linux_x86_64/22.11/cuda/11.8

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/22.11-nompi
module -s load mkl/2023.1.0
module -s load openmpi/3.1.6/nv22

export LANG=C
export LC_ALL=C
ulimit -s unlimited

cd ${INSTDIR}
if [ -d qe-${QE_VERSION} ]; then
  mv qe-${QE_VERSION} qe-erase
  rm -rf qe-erase &
fi

tar zxf ${TARBALL}
mv qe-${QE_VERSION}/* .
mv qe-${QE_VERSION}/.[a-z]* .
rmdir qe-${QE_VERSION}

sed -i -e "s/wget -O/wget –-trust-server-names -O/" \
       -e "s/curl -o/curl -L -o/" test-suite/check_pseudo.sh

export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx

rm -rf external/{devxlib,mbd}
mkdir external/devxlib
mkdir external/mbd

FC=nvfortran F90=nvfortran F77=nvfortran CC=nvc CXX=nvc++ \
    ./configure --enable-openmp \
                --enable-openacc \
                --with-scalapack=no \
                --with-cuda=${CUDA_HOME} \
                --with-cuda-cc=80 \
                --with-cuda-runtime=11.8

for i in w90; do
  echo "==== $i ===="
  make $i
done

# pwall(pw neb ph pp pwcond acfdt) cp ld1 tddfpt hp xspectra gwl
echo "==== all ===="
make -j${PARALLEL} all

cd test-suite
make pseudo
cd ..

Tests

Following job script was used to run tests, since GPUs are not available in login nodes.

#!/bin/sh
#PBS -l select=1:ncpus=4:mpiprocs=4:ompthreads=1:ngpus=4
#PBS -l walltime=48:00:00

cd $PBS_O_O_WORKDIR

umask 0022
module -s purge
module -s load nvhpc/22.11-nompi
module -s load mkl/2023.1.0
module -s load openmpi/3.1.6/nv22

export LANG=C
export LC_ALL=C
ulimit -s unlimited

cd /apl/qe/7.2-gpu
cd test-suite

make clean
make run-tests
make clean
make run-tests NPROCS=4

Test results

Failed tests:
  • pw_metaGGA/metaGGA-spin.in: libxc is necessary
  • epw related tests: epw is not used

Notes

  • HPC-X was not used, since UCX of HPC-X does not support thread parallel. Open MPI 3.1.6 is employed instead.
    • (UCX of system support thread parallel.)
    • (Rebuilding UCX of HPC-X may solve the problem.)
View PDF