Quantum ESPRESSO 7.5 with GPU support

Webpage

https://www.quantum-espresso.org/
https://gitlab.com/QEF/q-e

Version

7.5

Build Environment

  • NVIDIA HPC SDK 25.7 (nompi version)
  • Open MPI 4.1.8 (CUDA-aware)

Files Required

  • q-e-qe-7.5.tar.gz (downloaded from gitlab)
  • (Some of files will be downloaded during the installation.)

Build Procedure 

#!/bin/sh

QE_VERSION=7.5
BASEDIR=/home/users/${USER}/Software/QE/${QE_VERSION}
TARBALL=${BASEDIR}/q-e-qe-${QE_VERSION}.tar.gz

WORKDIR=/gwork/users/${USER}/qe75-gpu-wo-elpa

INSTDIR=/apl/qe/7.5-gpu
PARALLEL=24

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/25.7-nompi
module -s load openmpi/4.1.8/nv25
## gui; not necessary while building
#module -s load itcl/3.4.4
#module -s load itk/3.4.2
#module -s load iwidgets/4.1.1

export LANG=C
export LC_ALL=C
ulimit -s unlimited

if [ ! -d ${WORKDIR} ]; then
 mkdir -p ${WORKDIR}
fi

cd ${WORKDIR}
if [ -d q-e-qe-${QE_VERSION} ]; then
 mv q-e-qe-${QE_VERSION} qe-erase
 rm -rf qe-erase &
fi
tar zxf ${TARBALL}

QE_WORKDIR=${WORKDIR}/q-e-qe-${QE_VERSION}

# QE
cd ${QE_WORKDIR}
sed -i -e "s/wget -O/wget –-trust-server-names -O/" \
      -e "s/curl -o/curl -L -o/" test-suite/check_pseudo.sh

mkdir build && cd build
cmake .. \
 -DCMAKE_INSTALL_PREFIX=${INSTDIR} \
 -DCMAKE_Fortran_COMPILER=mpif90 \
 -DCMAKE_C_COMPILER=mpicc \
 -DCMAKE_CXX_COMPILER=mpicxx \
 -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
 -DESPRESSO_PSEUDO=${INSTDIR}/pseudo \
 -DQE_ENABLE_CUDA=ON \
 -DNVFORTRAN_CUDA_CC=80 \
 -DNVFORTRAN_CUDA_VERSION=12.9 \
 -DQE_ENABLE_OPENACC=ON \
 -DQE_ENABLE_OPENMP=ON \
 -DQE_ENABLE_MPI=ON \
 -DQE_ENABLE_MPI_GPU_AWARE=ON \
 -DQE_ENABLE_SCALAPACK=OFF \
 -DQE_ENABLE_ELPA=OFF \
 -DQE_ENABLE_LIBXC=OFF \
 -DQE_ENABLE_HDF5=OFF \
 -DQE_ENABLE_PLUGINS="pw2qmcpack;gipaw;legacy" \
 -DQE_ENABLE_FOX=ON \
 -DQE_WANNIER90_INTERNAL=ON \
 -DQE_MBD_INTERNAL=ON \
 -DQE_DEVICEXLIB_INTERNAL=ON \
 -DQE_ENABLE_ENVIRON=OFF \
 -DQE_ENABLE_OSCDFT=OFF

make -j${PARALLEL}
make install
ln -s ${INSTDIR}/bin ${QE_WORKDIR}/bin
cp -r ${QE_WORKDIR}/pseudo ${INSTDIR}/pseudo
mv ${QE_WORKDIR}/pseudo ${QE_WORKDIR}/pseudo.org
ln -s ${INSTDIR}/pseudo ${QE_WORKDIR}/pseudo
make test # to run "system--pw-pseudo" test for pseudo files

Test

Test was submitted as a job using the following script.

#!/bin/sh
#PBS -l select=1:ncpus=32:mpiprocs=4:ompthreads=3:ngpus=2
#PBS -l walltime=24:00:00

QE_VERSION=7.5
BUILDDIR=/gwork/users/${USER}/qe75-gpu-wo-elpa/q-e-qe-${QE_VERSION}/build

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/25.7-nompi
module -s load openmpi/4.1.8/nv25
## gui; not necessary while building
#module -s load itcl/3.4.4
#module -s load itk/3.4.2
#module -s load iwidgets/4.1.1

export LANG=C
export LC_ALL=C
ulimit -s unlimited

export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx

cd ${BUILDDIR}
make test

Test Results

Following tests have failed. Copy of log file is available in /apl/qe/7.5-gpu/testlog.

  • 54:test_qe_lax-r9-t2 => number of parallel processes is not enough to run.
  • 55:test_qe_lapack_zdotc
    • zdotc check failed. Expected (1.0000000308637980,6.48839729724212839E-003) but got (0.000000000000000,0.000000000000000)
    • (same error as qe 7.4 case)
  • 231:system--pw_plugins-correctness => Cannot find any previous test outputs.
  • 245:system--pw_scf-correctness
    • pw_scf - scf-rmm-paro-gamma.in: **FAILED**.
      ERROR: absolute error 1.10e+01 greater than 8.00e+00. (Test: 27.0. Benchmark: 16.0.)
  • 280:system--pw_twochem--nscf_twochem =>  twochem with GPU not present in this version
  • 281:system--pw_twochem--relax_twochem =>  twochem with GPU not present in this version
  • 282:system--pw_twochem--scf_twochem =>  twochem with GPU not present in this version
  • 283:system--pw_twochem--vc-relax_twochem =>  twochem with GPU not present in this version
  • 346:system--cp_al_edft--Al => Ensemble DFT for GPU not present in this version
  • 347:system--cp_al_edft--Al.uspp => USPP for GPU not present in this version
  • 349:system--cp_cluster--cluster4 => USPP for GPU not present in this version
  • 350:system--cp_cluster--cluster6 => USPP for GPU not present in this version
  • 385:system--cp_sio2--sio2-us-lda-cg => USPP for GPU not present in this version
  • 388:system--cp_sio2--sio2-us-pbe-cg => USPP for GPU not present in this version

Notes

  • While it is possible to incorporate GPU-enabled ELPA, there should not be many cases where it can actually be used.
View PDF