LAMMPS 29Sep21 with GPU support
Webpage
Version
29Sep21
Build Environment
- Intel oneAPI Compiler Classic 2022.2.1
- Intel MKL 2022.2.1
- HPC-X 2.11 (Open MPI 4.1.4)
- HPC-X 2.13.1 (Open MPI 4.1.5) was used on actual build. Runtime library was switched to HPC-X 2.11 to avoid the communication error.
- In the following procedure, we assume HPC-X 2.11 was used.
- CUDA 11.6
Files Required
- lammps-stable_29Sep2021.tar.gz
- (some of files will be downloaded during the installation)
Build Procedure
conda environment
Conda (miniforge) environment prepared for lammps 29Sep21 CPU version is used.
lammps
#!/bin/sh
VERSION=29Sep21
NAME=lammps-stable_29Sep2021
INSTALL_PREFIX=/apl/lammps/2021-Sep29-CUDA
BASEDIR=/home/users/${USER}/Software/LAMMPS/${VERSION}
LAMMPS_TARBALL=${BASEDIR}/${NAME}.tar.gz
WORKDIR=/gwork/users/${USER}/lammps-29Sep2021-cuda
LAMMPS_WORKDIR=${WORKDIR}/${NAME}
GPU_ARCH=sm_80
VMD_MOLFILE_INC=/home/users/${USER}/Software/VMD/1.9.4/vmd-1.9.4a57/plugins/include
PARALLEL=12
#------------------------------------------------------------------
umask 0022
export LANG=C
ulimit -s unlimited
module -s purge
# load one api compiler (oneapi 2022.3.1 compiler 2022.2.1)
. ~/intel/oneapi/compiler/2022.2.1/env/vars.sh
. /apl/lammps/2021-Sep29-CUDA/conda_init.sh
module -s load mkl/2022.2.1
module -s load cuda/11.6
module -s load openmpi/4.1.4-hpcx/intel2022.2.1
export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export MPICC=mpicc
export MPICXX=mpicxx
export MPIFC=mpif90
cd ${WORKDIR}
if [ -d ${NAME} ]; then
mv ${NAME} lammps_erase
rm -rf lammps_erase &
fi
tar zxf ${LAMMPS_TARBALL}
cd ${NAME}
sed -i -e "s/xHost/march=core-avx2/" cmake/CMakeLists.txt
mkdir build && cd build
# Disabled PKGs:
# FFMPEG, ADIOS, MDI, VTK: noavail
# MSCG: gsl too old
# MESSAGE: ZeroMQ support not enabled
# QUIP: failed to build
# ML-HDNNP: failed to build
# KIM: CDDL is imcompatible with GPL
# LATTE: technical problem of cmake? (LAPACK and BLAS)
# NETCDF: to avoid EVP_KDF_ctrl error
# MPIIO: not maintained?
cmake ../cmake \
-DLAMMPS_MACHINE=rccs-cuda \
-DENABLE_TESTING=on \
-DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DCMAKE_MPI_C_COMPILER=mpicc \
-DCMAKE_MPI_CXX_COMPILER=mpicxx \
-DCMAKE_MPI_Fortran_COMPILER=mpif90 \
-DCMAKE_CXX_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_CXX_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DCMAKE_Fortran_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_Fortran_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_Fortran_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DCMAKE_C_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_C_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DLAMMPS_EXCEPTIONS=on \
-DBUILD_SHARED_LIBS=on \
-DBUILD_TOOLS=on \
-DBUILD_MPI=on \
-DBUILD_OMP=on \
-DFFT=MKL \
-DFFT_SINGLE=on \
-DFFT_MKL_THREADS=on \
-DWITH_JPEG=yes \
-DWITH_PNG=yes \
-DWITH_GZIP=yes \
-DPKG_ASPHERE=on \
-DPKG_ATC=on \
-DPKG_AWPMD=on \
-DPKG_BOCS=on \
-DPKG_BODY=on \
-DPKG_BROWNIAN=on \
-DPKG_CG-DNA=on \
-DPKG_CG-SDK=on \
-DPKG_CLASS2=on \
-DPKG_COLLOID=on \
-DPKG_COLVARS=on \
-DPKG_COMPRESS=on \
-DPKG_CORESHELL=on \
-DPKG_DIELECTRIC=on \
-DPKG_DIFFRACTION=on \
-DPKG_DIPOLE=on \
-DPKG_DPD-BASIC=on \
-DPKG_DPD-MESO=on \
-DPKG_DPD-REACT=on \
-DPKG_DPD-SMOOTH=on \
-DPKG_DRUDE=on \
-DPKG_EFF=on \
-DPKG_EXTRA-COMPUTE=on \
-DPKG_EXTRA-DUMP=on \
-DPKG_EXTRA-FIX=on \
-DPKG_EXTRA-MOLECULE=on \
-DPKG_EXTRA-PAIR=on \
-DPKG_FEP=on \
-DPKG_GPU=on \
-DGPU_API=cuda \
-DGPU_ARCH=${GPU_ARCH} \
-DPKG_GRANULAR=on \
-DPKG_H5MD=on \
-DPKG_INTEL=on \
-DPKG_INTERLAYER=on \
-DPKG_KIM=off \
-DDOWNLOAD_KIM=no \
-DPKG_KOKKOS=off \
-DPKG_KSPACE=on \
-DPKG_LATBOLTZ=on \
-DPKG_MACHDYN=on \
-DDOWNLOAD_EIGEN3=on \
-DPKG_MANIFOLD=on \
-DPKG_MANYBODY=on \
-DPKG_MC=on \
-DPKG_MDI=off \
-DPKG_MEAM=on \
-DPKG_MESONT=on \
-DPKG_MESSAGE=on \
-DPKG_MGPT=on \
-DPKG_MISC=on \
-DPKG_ML-HDNNP=off \
-DDOWNLOAD_N2P2=no \
-DPKG_ML-IAP=on \
-DPKG_ML-PACE=on \
-DPKG_ML-QUIP=off \
-DDOWNLOAD_QUIP=no \
-DPKG_ML-RANN=on \
-DPKG_ML-SNAP=on \
-DPKG_MOFFF=on \
-DPKG_MOLECULE=on \
-DPKG_MOLFILE=on \
-DMOLFILE_INCLUDE_DIR=${VMD_MOLFILE_INC} \
-DPKG_MPIIO=off \
-DPKG_MSCG=off \
-DPKG_NETCDF=off \
-DPKG_OPENMP=on \
-DPKG_OPT=on \
-DPKG_ORIENT=on \
-DPKG_PERI=on \
-DPKG_PHONON=on \
-DPKG_PLUGIN=on \
-DPKG_PLUMED=on \
-DDOWNLOAD_PLUMED=yes \
-DPKG_POEMS=on \
-DPKG_PTM=on \
-DPKG_PYTHON=on \
-DPKG_QEQ=on \
-DPKG_QMMM=on \
-DPKG_QTB=on \
-DPKG_REACTION=on \
-DPKG_REAXFF=on \
-DPKG_REPLICA=on \
-DPKG_RIGID=on \
-DPKG_SCAFACOS=on \
-DDOWNLOAD_SCAFACOS=yes \
-DPKG_SHOCK=on \
-DPKG_SMTBQ=on \
-DPKG_SPH=on \
-DPKG_SPIN=on \
-DPKG_SRD=on \
-DPKG_TALLY=on \
-DPKG_UEF=on \
-DPKG_VORONOI=on \
-DDOWNLOAD_VORO=yes \
-DPKG_VTK=off \
-DPKG_YAFF=on \
-DBLAS_LIBRARIES="-qmkl" \
-DCMAKE_BUILD_TYPE=Release
make VERBOSE=1 -j ${PARALLEL}
export OMP_NUM_THREADS=2
#make test # will put error...
make install
cp -a ../examples ${INSTALL_PREFIX}
cd ${INSTALL_PREFIX}
for f in etc/profile.d/*; do
ln -s $f .
done
cd lib64
if [ -f liblammps_rccs-cuda.so ]; then
ln -s liblammps_rccs-cuda.so liblammps.so
fi
if [ -f liblammps_rccs-cuda.so.0 ]; then
ln -s liblammps_rccs-cuda.so.0 liblammps.so.0
fi
Enabled packages
ASPHERE ATC AWPMD BOCS BODY BROWNIAN CG-DNA CG-SDK CLASS2 COLLOID COLVARS COMPRESS CORESHELL
DIELECTRIC DIFFRACTION DIPOLE DPD-BASIC DPD-MESO DPD-REACT DPD-SMOOTH DRUDE EFF
EXTRA-COMPUTE EXTRA-DUMP EXTRA-FIX EXTRA-MOLECULE EXTRA-PAIR FEP GPU GRANULAR
H5MD INTEL INTERLAYER KSPACE LATBOLTZ MACHDYN MANIFOLD MANYBODY MC MEAM MESONT MESSAGE MGPT
MISC ML-IAP ML-PACE ML-RANN ML-SNAP MOFFF MOLECULE MOLFILE OPENMP OPT ORIENT PERI
PHONON PLUGIN PLUMED POEMS PTM PYTHON QEQ QMMM QTB REACTION REAXFF REPLICA RIGID SCAFACOS
SHOCK SMTBQ SPH SPIN SRD TALLY UEF VORONOI YAFF
Tests
("make test" was done on one of GPU computation nodes.)
Copy of test log file is available at /apl/lammps/2021-Sep29-CUDA/Testing/.
The following tests FAILED:
10 - AtomStyles (Failed)
88 - MolPairStyle:coul_diel (Failed)
94 - MolPairStyle:coul_shield (Failed)
117 - MolPairStyle:lj_charmm_coul_long_soft (Failed)
130 - MolPairStyle:lj_class2_soft (Failed)
139 - MolPairStyle:lj_cut_coul_long_soft (Failed)
145 - MolPairStyle:lj_cut_soft (Failed)
151 - MolPairStyle:lj_expand_coul_long (Failed)
164 - MolPairStyle:lj_sdk_coul_long (Failed)
165 - MolPairStyle:lj_sdk_coul_table (Failed)
169 - MolPairStyle:lj_switch3_coulgauss_long (Failed)
192 - MolPairStyle:tip4p_long_soft (Failed)
195 - MolPairStyle:wf_cut (Failed)
203 - AtomicPairStyle:buck_coul_cut_qeq_point (Failed)
204 - AtomicPairStyle:buck_coul_cut_qeq_shielded (Failed)
221 - AtomicPairStyle:edip (Failed)
224 - AtomicPairStyle:hybrid-eam (Failed)
228 - AtomicPairStyle:meam (Failed)
229 - AtomicPairStyle:meam_spline (Failed)
230 - AtomicPairStyle:meam_sw_spline (Failed)
233 - AtomicPairStyle:reaxff (Failed)
234 - AtomicPairStyle:reaxff_lgvdw (Failed)
235 - AtomicPairStyle:reaxff_noqeq (Failed)
236 - AtomicPairStyle:reaxff_tabulate (Failed)
246 - ManybodyPairStyle:bop (Failed)
247 - ManybodyPairStyle:bop_save (Failed)
248 - ManybodyPairStyle:comb (Failed)
250 - ManybodyPairStyle:drip (Failed)
251 - ManybodyPairStyle:drip_real (Failed)
256 - ManybodyPairStyle:ilp-graphene-hbn (Failed)
257 - ManybodyPairStyle:ilp-graphene-hbn_notaper (Failed)
258 - ManybodyPairStyle:kolmogorov_crespi_full (Failed)
261 - ManybodyPairStyle:lcbop (Failed)
262 - ManybodyPairStyle:lebedeva_z (Failed)
263 - ManybodyPairStyle:meam (Failed)
269 - ManybodyPairStyle:nb3b_harmonic (Failed)
270 - ManybodyPairStyle:pace_product (Failed)
271 - ManybodyPairStyle:pace_recursive (Failed)
272 - ManybodyPairStyle:polymorphic_sw (Failed)
273 - ManybodyPairStyle:polymorphic_tersoff (Failed)
281 - ManybodyPairStyle:tersoff (Failed)
286 - ManybodyPairStyle:tersoff_shift (Failed)
287 - ManybodyPairStyle:tersoff_table (Failed)
295 - BondStyle:gaussian (Failed)
312 - AngleStyle:cosine_delta (Failed)
314 - AngleStyle:cosine_shift (Failed)
336 - KSpaceStyle:ewald_tri (Failed)
338 - KSpaceStyle:pppm_ad (Failed)
339 - KSpaceStyle:pppm_cg (Failed)
341 - KSpaceStyle:pppm_cg_tiled (Failed)
347 - KSpaceStyle:pppm_disp_tip4p (Failed)
354 - KSpaceStyle:pppm_tip4p (Failed)
359 - KSpaceStyle:scafacos_direct (Failed)
360 - KSpaceStyle:scafacos_ewald (Failed)
361 - KSpaceStyle:scafacos_fmm (Failed)
362 - KSpaceStyle:scafacos_fmm_tuned (Failed)
363 - KSpaceStyle:scafacos_p2nfft (Failed)
364 - FixTimestep:adapt_coul (Failed)
367 - FixTimestep:addforce_const (Failed)
368 - FixTimestep:addforce_variable (Failed)
369 - FixTimestep:addtorque_const (Failed)
372 - FixTimestep:aveforce_variable (Failed)
374 - FixTimestep:drag (Failed)
378 - FixTimestep:heat (Failed)
381 - FixTimestep:momentum (Failed)
383 - FixTimestep:nph (Failed)
384 - FixTimestep:nph_sphere (Failed)
386 - FixTimestep:npt_iso (Failed)
387 - FixTimestep:npt_sphere_aniso (Failed)
388 - FixTimestep:npt_sphere_iso (Failed)
389 - FixTimestep:npt_sphere_tri (Failed)
397 - FixTimestep:nvt (Failed)
399 - FixTimestep:oneway (Failed)
412 - FixTimestep:rigid_npt_small (Failed)
424 - FixTimestep:shake_angle (Failed)
426 - FixTimestep:smd_couple (Failed)
429 - FixTimestep:spring_couple (Failed)
430 - FixTimestep:spring_rg (Failed)
432 - FixTimestep:spring_tether (Failed)
433 - FixTimestep:temp_berendsen (Failed)
434 - FixTimestep:temp_csld (Failed)
435 - FixTimestep:temp_csvr (Failed)
436 - FixTimestep:temp_rescale (Failed)
455 - DihedralStyle:table_cut_linear (Failed)
457 - DihedralStyle:table_linear (Failed)
458 - DihedralStyle:table_spline (Failed)
466 - ImproperStyle:harmonic (Failed)
468 - ImproperStyle:inversion_harmonic (Failed)
- Most errors are minor numerical ones or "lattice" command problem when intel compiler is employed. There may not be serious problems.
Notes
- Built on ccfep, tested on one of GPU computation nodes.
- Parallel runs failed if HPC-X 2.13.1 runtime library was used. Switching to HPC-X 2.11 solved the problem.
- If NETCDF is set to on, the build failed. We thus disabled NETCDF this time. This may be due to the problem of system library.
- The compilation errors could be avoided if python 3.6 of the system was employed... We will try it in the next time.