Quantum Espresso 7.4 with GPU support

Webpage

https://www.quantum-espresso.org/
https://gitlab.com/QEF/q-e

Version

7.4

Build Environment

  • NVIDIA HPC SDK 24.9 (nompi version)
  • Open MPI 4.1.6 (CUDA-aware)

Files Required

  • q-e-qe-7.4.tar.gz (downloaded from gitlab)
  • (some of files are downloaded in the procedure below)

Build Procedure

#!/bin/sh

QE_VERSION=7.4
BASEDIR=/home/users/${USER}/Software/QE/${QE_VERSION}
TARBALL=${BASEDIR}/q-e-qe-${QE_VERSION}.tar.gz

GIPAW_ID="3bbf5a931fc195503c3f01565ac43cac8c05db44"

WORKDIR=/gwork/users/${USER}

INSTDIR=/apl/qe/7.4-gpu
PARALLEL=24

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/24.9-nompi
module -s load openmpi/4.1.6/nv24
## gui; not necessary while building
#module -s load itcl/3.4.4
#module -s load itk/3.4.2
#module -s load iwidgets/4.1.1

export LANG=C
export LC_ALL=C
ulimit -s unlimited

if [ ! -d ${WORKDIR} ]; then
 mkdir -p ${WORKDIR}
fi

cd ${WORKDIR}
if [ -d q-e-qe-${QE_VERSION} ]; then
 mv q-e-qe-${QE_VERSION} qe-erase
 rm -rf qe-erase &
fi
tar zxf ${TARBALL}

QE_WORKDIR=${WORKDIR}/q-e-qe-${QE_VERSION}

# QE
cd ${QE_WORKDIR}
sed -i -e "s/wget -O/wget –-trust-server-names -O/" \
      -e "s/curl -o/curl -L -o/" test-suite/check_pseudo.sh
sed -i -e "s/[^ ]* qe-gipaw/${GIPAW_ID} qe-gipaw/" \
   external/submodule_commit_hash_records

mkdir build && cd build
cmake .. \
 -DCMAKE_INSTALL_PREFIX=${INSTDIR} \
 -DCMAKE_Fortran_COMPILER=mpif90 \
 -DCMAKE_C_COMPILER=mpicc \
 -DCMAKE_CXX_COMPILER=mpicxx \
 -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
 -DESPRESSO_PSEUDO=${INSTDIR}/pseudo \
 -DQE_ENABLE_CUDA=ON \
 -DNVFORTRAN_CUDA_CC=80 \
 -DNVFORTRAN_CUDA_VERSION=12.6 \
 -DQE_ENABLE_OPENACC=ON \
 -DQE_ENABLE_OPENMP=ON \
 -DQE_ENABLE_MPI=ON \
 -DQE_ENABLE_MPI_GPU_AWARE=ON \
 -DQE_ENABLE_SCALAPACK=OFF \
 -DQE_ENABLE_ELPA=OFF \
 -DQE_ENABLE_LIBXC=OFF \
 -DQE_ENABLE_HDF5=OFF \
 -DQE_ENABLE_PLUGINS="pw2qmcpack;gipaw" \
 -DQE_ENABLE_FOX=ON \
 -DQE_WANNIER90_INTERNAL=ON \
 -DQE_MBD_INTERNAL=ON \
 -DQE_DEVICEXLIB_INTERNAL=ON \
 -DQE_ENABLE_ENVIRON=OFF \
 -DQE_ENABLE_OSCDFT=OFF

make -j${PARALLEL}
make install
ln -s ${INSTDIR}/bin ${QE_WORKDIR}/bin
cp -r ${QE_WORKDIR}/pseudo ${INSTDIR}/pseudo
mv ${QE_WORKDIR}/pseudo ${QE_WORKDIR}/pseudo.org
ln -s ${INSTDIR}/pseudo ${QE_WORKDIR}/pseudo
make test # to run "system--pw-pseudo" test for pseudo files

The last "make test" is a fake, just to download pseudo files. Actual test was performed with the script below.

Tests

(submitted as a job)

#!/bin/sh
#PBS -l select=1:ncpus=32:mpiprocs=4:ompthreads=3:ngpus=2
#PBS -l walltime=24:00:00

QE_VERSION=7.4
BUILDDIR=/gwork/users/${USER}/q-e-qe-${QE_VERSION}/build

cd $PBS_O_WORKDIR

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/24.9-nompi
module -s load openmpi/4.1.6/nv24

export LANG=C
export LC_ALL=C
ulimit -s unlimited

export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx

cd ${BUILDDIR}
make test

Test Results

The logfile was copied to /apl/qe/7.4-gpu/testlog.

  • test_qe_lax-r9-t2 => insufficient number of processes
  • test_qe_lapack_zdotc => numerical error (Expected (1.0000000308637980,6.48839729724212839E-003) got (0.000000000000000,0.000000000000000))
  • system--pw_lda+U--lda+U+V-user_ns => force killed (hang?)
  • system--pw_lda+U--lda+U+V_force_stress_ortho => (same as above)
  • system--pw_lda+U--lda+U_force_stress_ortho => (same as above)
  • system--pw_lda+U-correctness => some of tests did not work
  • system--pw_scf-correctness => numerical error
  • system--pw_twochem--nscf_twochem => not compatible with GPU
  • system--pw_twochem--relax_twochem => (same as above)
  • system--pw_twochem--scf_twochem => (same as above)
  • system--pw_twochem--vc-relax_twochem => (same as above)
  • system--pw_uspp-correctness => some of tests did not work
  • system--cp_h2o-correctness => numerical error

Notes

  • Build failed when d3q is enabled.
  • ELPA, HDF, libxc, Environ are not tested.
View PDF