Quantum ESPRESSO 7.6 with GPU support

Webpage

https://www.quantum-espresso.org/
https://gitlab.com/QEF/q-e

Version

7.6

Build Environment

  • NVIDIA HPC SDK 26.5 (nompi version) (CUDA 13.2)
  • Open MPI 4.1.8 (CUDA-aware)

Files Required

  • q-e-qe-7.6.tar.gz
  • (some of files will be downloaded during installation)

Build Procedure

#!/bin/sh

QE_VERSION=7.6
BASEDIR=/home/users/${USER}/Software/QE/${QE_VERSION}
TARBALL=${BASEDIR}/q-e-qe-${QE_VERSION}.tar.gz

WORKDIR=/gwork/users/${USER}

INSTDIR=/apl/qe/7.6-gpu
PARALLEL=24

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load cmake/3.31.11
module -s load nvhpc/26.5-nompi
module -s load openmpi/4.1.8/nv26
## gui; not necessary while building
#module -s load itcl/3.4.4
#module -s load itk/3.4.2
#module -s load iwidgets/4.1.1

export LANG=C
export LC_ALL=C
ulimit -s unlimited

if [ ! -d ${WORKDIR} ]; then
 mkdir -p ${WORKDIR}
fi

cd ${WORKDIR}
if [ -d q-e-qe-${QE_VERSION} ]; then
 mv q-e-qe-${QE_VERSION} qe-erase
 rm -rf qe-erase &
fi
tar zxf ${TARBALL}

QE_WORKDIR=${WORKDIR}/q-e-qe-${QE_VERSION}

# QE
cd ${QE_WORKDIR}
sed -i -e "s/wget -O/wget –-trust-server-names -O/" \
      -e "s/curl -o/curl -L -o/" test-suite/check_pseudo.sh
#sed -i -e "s/teams distribute//" EPW/src/use_wannier.f90

mkdir build && cd build
cmake .. \
 -DCMAKE_INSTALL_PREFIX=${INSTDIR} \
 -DCMAKE_Fortran_COMPILER=mpif90 \
 -DCMAKE_C_COMPILER=mpicc \
 -DCMAKE_CXX_COMPILER=mpicxx \
 -DESPRESSO_PSEUDO=${INSTDIR}/pseudo \
 -DQE_GPU="openacc;cuda" \
 -DQE_GPU_ARCHS="sm_80" \
 -DNVFORTRAN_CUDA_VERSION=13.2 \
 -DQE_ENABLE_OPENMP=ON \
 -DQE_ENABLE_MPI=ON \
 -DQE_ENABLE_MPI_GPU_AWARE=ON \
 -DQE_ENABLE_SCALAPACK=OFF \
 -DQE_ENABLE_ELPA=OFF \
 -DQE_ENABLE_LIBXC=OFF \
 -DQE_ENABLE_HDF5=OFF \
 -DQE_ENABLE_PLUGINS="pw2qmcpack;legacy" \
 -DQE_ENABLE_FOX=OFF \
 -DQE_WANNIER90_INTERNAL=ON \
 -DQE_MBD_INTERNAL=ON \
 -DQE_DEVICEXLIB_INTERNAL=ON \
 -DQE_ENABLE_ENVIRON=OFF \
 -DQE_ENABLE_OSCDFT=OFF

make -j${PARALLEL}
make install
ln -s ${INSTDIR}/bin ${QE_WORKDIR}/bin
cp -r ${QE_WORKDIR}/pseudo ${INSTDIR}/pseudo
mv ${QE_WORKDIR}/pseudo ${QE_WORKDIR}/pseudo.org
ln -s ${INSTDIR}/pseudo ${QE_WORKDIR}/pseudo
make test

Tests

Tests were performed on ccgpu.

#!/bin/sh

QE_VERSION=7.6
BUILDDIR=/gwork/users/${USER}/q-e-qe-${QE_VERSION}/build

export OMP_NUM_THREADS=3

# --------------------------------------------------------------------
umask 0022

module -s purge
module -s load nvhpc/26.5-nompi
module -s load openmpi/4.1.8/nv26
## gui; not necessary while building
#module -s load itcl/3.4.4
#module -s load itk/3.4.2
#module -s load iwidgets/4.1.1

export LANG=C
export LC_ALL=C
ulimit -s unlimited

export MPIF90=mpif90
export MPIFC=mpif90
export MPIF77=mpif90
export MPICC=mpicc
export MPICXX=mpicxx

cd ${BUILDDIR}
make test

Test Results

Following tests have failed. Copy of test log can be found in /apl/qe/7.6-gpu/testlog.

  • 55 - test_qe_lapack_zdotc (Failed)
    • zdotc check failed. Expected (1.0000000308637980,6.48839729724212839E-003) but got (0.000000000000000,0.000000000000000)
  • 75 - system--pw_b3lyp--b3lyp-O_std (Failed)
    • bands on GPU not present in this version
  • 77 - system--pw_b3lyp--b3lyp-h2o_std (Failed)
    • bands on GPU not present in this version
  • 105 - system--pw_diag_direct--scf-direct (Failed)
    • direct diagonalization on GPU not present in this version
  • 124 - system--pw_hse--hse-si444_std (Failed)
    • bands on GPU not present in this version
  • 228 - system--pw_noncolin--noncolin-hyb_std (Failed)
    • bands on GPU not present in this version
  • 262 - system--pw_scf-correctness (Failed)
    • pw_scf - scf-rmm-paro-gamma.in: **FAILED**. ERROR: absolute error 1.20e+01 greater than 8.00e+00. (Test: 28.0.  Benchmark: 16.0.)
  • 297 - system--pw_twochem--nscf_twochem (Failed)
    • twochem with GPU not present in this version
  • 298 - system--pw_twochem--relax_twochem (Failed)
    • twochem with GPU not present in this version
  • 299 - system--pw_twochem--scf_twochem (Failed)
    • twochem with GPU not present in this version
  • 300 - system--pw_twochem--vc-relax_twochem (Failed)
    • twochem with GPU not present in this version
  • 306 - system--pw_uspp--uspp-hyb-g_std (Failed)
    • bands on GPU not present in this version
  • 308 - system--pw_uspp--uspp-hyb-k_std (Failed)
    • bands on GPU not present in this version
  • 365 - system--cp_al_edft--Al (Failed)
    • Ensemble DFT for GPU not present in this version
  • 366 - system--cp_al_edft--Al.uspp (Failed)
    • USPP for GPU not present in this version
  • 368 - system--cp_cluster--cluster4 (Failed)
    • USPP for GPU not present in this version
  • 369 - system--cp_cluster--cluster6 (Failed)
    • USPP for GPU not present in this version
  • 370 - system--cp_h2o-correctness (Failed)
    • cp_h2o - h2o-mt-blyp-6.in: **FAILED**. ERROR: absolute error 2.00e-05 greater than 1.00e-05. (Test: -17.17738.  Benchmark: -17.1774.)...
    • cp_h2o - h2o-mt-blyp-7.in: **FAILED**. ERROR: absolute error 1.42e-04 greater than 1.00e-05. (Test: -17.175103.  Benchmark: -17.174961.)...
  • 379 - system--cp_h2o_exx-correctness (Failed)
    • cp_h2o_exx - h2o-mt-b3lyp-1.in: **FAILED**. ERROR: absolute error 5.73e-03 greater than 5.00e-04. (Test: -26.26138016.  Benchmark: -26.25564791.)...
    • cp_h2o_exx - h2o-mt-exx_fraction.in: **FAILED**. ERROR: absolute error 5.47e-03 greater than 5.00e-04. (Test: -26.61866227.  Benchmark: -26.61319248.)...
  • 385 - system--cp_h2o_wf-correctness (Failed)
    • cp_h2o_wf - h2o-nspin1.in: **FAILED**. ERROR: absolute error 2.78e-01 greater than 5.00e-04. (Test: -27.83148267.  Benchmark: -27.55376092.)...
    • cp_h2o_wf - h2o-nspin2.in: **FAILED**. ERROR: absolute error 2.19e+01 greater than 5.00e-04. (Test: -48.20692815.  Benchmark: -26.32091272.)
  • 404 - system--cp_sio2--sio2-us-lda-cg (Failed)
    • USPP for GPU not present in this version
  • 407 - system--cp_sio2--sio2-us-pbe-cg (Failed)
    • USPP for GPU not present in this version

Notes

  • None