LAMMPS 29Sep21 with GPU support
ウェブページ
バージョン
29Sep21
ビルド環境
- Intel oneAPI Compiler Classic 2022.2.1
- Intel MKL 2022.2.1
- HPC-X 2.11 (Open MPI 4.1.4)
- 実際のビルドでは HPC-X 2.13.1 (Open MPI 4.1.5)を利用。エラーが発生したため、runtime のみ HPC-X 2.11 に切り替えている
- 以下の手順では HPC-X 2.11 を利用したものとして記述
- CUDA 11.6
ビルドに必要なファイル
- lammps-stable_29Sep2021.tar.gz
- (一部ファイルは以下スクリプト中で取得)
ビルド手順
conda 環境
lammps 29Sep21 CPU 版で作成したものを流用。lammps 本体
#!/bin/sh
VERSION=29Sep21
NAME=lammps-stable_29Sep2021
INSTALL_PREFIX=/apl/lammps/2021-Sep29-CUDA
BASEDIR=/home/users/${USER}/Software/LAMMPS/${VERSION}
LAMMPS_TARBALL=${BASEDIR}/${NAME}.tar.gz
WORKDIR=/gwork/users/${USER}/lammps-29Sep2021-cuda
LAMMPS_WORKDIR=${WORKDIR}/${NAME}
GPU_ARCH=sm_80
VMD_MOLFILE_INC=/home/users/${USER}/Software/VMD/1.9.4/vmd-1.9.4a57/plugins/include
PARALLEL=12
#------------------------------------------------------------------
umask 0022
export LANG=C
ulimit -s unlimited
module -s purge
# load one api compiler (oneapi 2022.3.1 compiler 2022.2.1)
. ~/intel/oneapi/compiler/2022.2.1/env/vars.sh
. /apl/lammps/2021-Sep29-CUDA/conda_init.sh
module -s load mkl/2022.2.1
module -s load cuda/11.6
module -s load openmpi/4.1.4-hpcx/intel2022.2.1
export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export MPICC=mpicc
export MPICXX=mpicxx
export MPIFC=mpif90
cd ${WORKDIR}
if [ -d ${NAME} ]; then
mv ${NAME} lammps_erase
rm -rf lammps_erase &
fi
tar zxf ${LAMMPS_TARBALL}
cd ${NAME}
sed -i -e "s/xHost/march=core-avx2/" cmake/CMakeLists.txt
mkdir build && cd build
# Disabled PKGs:
# FFMPEG, ADIOS, MDI, VTK: noavail
# MSCG: gsl too old
# MESSAGE: ZeroMQ support not enabled
# QUIP: failed to build
# ML-HDNNP: failed to build
# KIM: CDDL is imcompatible with GPL
# LATTE: technical problem of cmake? (LAPACK and BLAS)
# NETCDF: to avoid EVP_KDF_ctrl error
# MPIIO: not maintained?
cmake ../cmake \
-DLAMMPS_MACHINE=rccs-cuda \
-DENABLE_TESTING=on \
-DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DCMAKE_MPI_C_COMPILER=mpicc \
-DCMAKE_MPI_CXX_COMPILER=mpicxx \
-DCMAKE_MPI_Fortran_COMPILER=mpif90 \
-DCMAKE_CXX_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_CXX_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DCMAKE_Fortran_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_Fortran_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_Fortran_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DCMAKE_C_FLAGS_DEBUG="-Wall -Wextra -g" \
-DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wall -Wextra -g -O2 -DNDEBUG" \
-DCMAKE_C_FLAGS_RELEASE="-O3 -DNDEBUG" \
-DLAMMPS_EXCEPTIONS=on \
-DBUILD_SHARED_LIBS=on \
-DBUILD_TOOLS=on \
-DBUILD_MPI=on \
-DBUILD_OMP=on \
-DFFT=MKL \
-DFFT_SINGLE=on \
-DFFT_MKL_THREADS=on \
-DWITH_JPEG=yes \
-DWITH_PNG=yes \
-DWITH_GZIP=yes \
-DPKG_ASPHERE=on \
-DPKG_ATC=on \
-DPKG_AWPMD=on \
-DPKG_BOCS=on \
-DPKG_BODY=on \
-DPKG_BROWNIAN=on \
-DPKG_CG-DNA=on \
-DPKG_CG-SDK=on \
-DPKG_CLASS2=on \
-DPKG_COLLOID=on \
-DPKG_COLVARS=on \
-DPKG_COMPRESS=on \
-DPKG_CORESHELL=on \
-DPKG_DIELECTRIC=on \
-DPKG_DIFFRACTION=on \
-DPKG_DIPOLE=on \
-DPKG_DPD-BASIC=on \
-DPKG_DPD-MESO=on \
-DPKG_DPD-REACT=on \
-DPKG_DPD-SMOOTH=on \
-DPKG_DRUDE=on \
-DPKG_EFF=on \
-DPKG_EXTRA-COMPUTE=on \
-DPKG_EXTRA-DUMP=on \
-DPKG_EXTRA-FIX=on \
-DPKG_EXTRA-MOLECULE=on \
-DPKG_EXTRA-PAIR=on \
-DPKG_FEP=on \
-DPKG_GPU=on \
-DGPU_API=cuda \
-DGPU_ARCH=${GPU_ARCH} \
-DPKG_GRANULAR=on \
-DPKG_H5MD=on \
-DPKG_INTEL=on \
-DPKG_INTERLAYER=on \
-DPKG_KIM=off \
-DDOWNLOAD_KIM=no \
-DPKG_KOKKOS=off \
-DPKG_KSPACE=on \
-DPKG_LATBOLTZ=on \
-DPKG_MACHDYN=on \
-DDOWNLOAD_EIGEN3=on \
-DPKG_MANIFOLD=on \
-DPKG_MANYBODY=on \
-DPKG_MC=on \
-DPKG_MDI=off \
-DPKG_MEAM=on \
-DPKG_MESONT=on \
-DPKG_MESSAGE=on \
-DPKG_MGPT=on \
-DPKG_MISC=on \
-DPKG_ML-HDNNP=off \
-DDOWNLOAD_N2P2=no \
-DPKG_ML-IAP=on \
-DPKG_ML-PACE=on \
-DPKG_ML-QUIP=off \
-DDOWNLOAD_QUIP=no \
-DPKG_ML-RANN=on \
-DPKG_ML-SNAP=on \
-DPKG_MOFFF=on \
-DPKG_MOLECULE=on \
-DPKG_MOLFILE=on \
-DMOLFILE_INCLUDE_DIR=${VMD_MOLFILE_INC} \
-DPKG_MPIIO=off \
-DPKG_MSCG=off \
-DPKG_NETCDF=off \
-DPKG_OPENMP=on \
-DPKG_OPT=on \
-DPKG_ORIENT=on \
-DPKG_PERI=on \
-DPKG_PHONON=on \
-DPKG_PLUGIN=on \
-DPKG_PLUMED=on \
-DDOWNLOAD_PLUMED=yes \
-DPKG_POEMS=on \
-DPKG_PTM=on \
-DPKG_PYTHON=on \
-DPKG_QEQ=on \
-DPKG_QMMM=on \
-DPKG_QTB=on \
-DPKG_REACTION=on \
-DPKG_REAXFF=on \
-DPKG_REPLICA=on \
-DPKG_RIGID=on \
-DPKG_SCAFACOS=on \
-DDOWNLOAD_SCAFACOS=yes \
-DPKG_SHOCK=on \
-DPKG_SMTBQ=on \
-DPKG_SPH=on \
-DPKG_SPIN=on \
-DPKG_SRD=on \
-DPKG_TALLY=on \
-DPKG_UEF=on \
-DPKG_VORONOI=on \
-DDOWNLOAD_VORO=yes \
-DPKG_VTK=off \
-DPKG_YAFF=on \
-DBLAS_LIBRARIES="-qmkl" \
-DCMAKE_BUILD_TYPE=Release
make VERBOSE=1 -j ${PARALLEL}
export OMP_NUM_THREADS=2
#make test # will put error...
make install
cp -a ../examples ${INSTALL_PREFIX}
cd ${INSTALL_PREFIX}
for f in etc/profile.d/*; do
ln -s $f .
done
cd lib64
if [ -f liblammps_rccs-cuda.so ]; then
ln -s liblammps_rccs-cuda.so liblammps.so
fi
if [ -f liblammps_rccs-cuda.so.0 ]; then
ln -s liblammps_rccs-cuda.so.0 liblammps.so.0
fi
パッケージ
ASPHERE ATC AWPMD BOCS BODY BROWNIAN CG-DNA CG-SDK CLASS2 COLLOID COLVARS COMPRESS CORESHELL
DIELECTRIC DIFFRACTION DIPOLE DPD-BASIC DPD-MESO DPD-REACT DPD-SMOOTH DRUDE EFF
EXTRA-COMPUTE EXTRA-DUMP EXTRA-FIX EXTRA-MOLECULE EXTRA-PAIR FEP GPU GRANULAR
H5MD INTEL INTERLAYER KSPACE LATBOLTZ MACHDYN MANIFOLD MANYBODY MC MEAM MESONT MESSAGE MGPT
MISC ML-IAP ML-PACE ML-RANN ML-SNAP MOFFF MOLECULE MOLFILE OPENMP OPT ORIENT PERI
PHONON PLUGIN PLUMED POEMS PTM PYTHON QEQ QMMM QTB REACTION REAXFF REPLICA RIGID SCAFACOS
SHOCK SMTBQ SPH SPIN SRD TALLY UEF VORONOI YAFF
テスト
(make test の部分のみ GPU 演算ノードで実行)テストログのコピーを /apl/lammps/2021-Sep29-CUDA/Testing/ に置いています。
The following tests FAILED:
10 - AtomStyles (Failed)
88 - MolPairStyle:coul_diel (Failed)
94 - MolPairStyle:coul_shield (Failed)
117 - MolPairStyle:lj_charmm_coul_long_soft (Failed)
130 - MolPairStyle:lj_class2_soft (Failed)
139 - MolPairStyle:lj_cut_coul_long_soft (Failed)
145 - MolPairStyle:lj_cut_soft (Failed)
151 - MolPairStyle:lj_expand_coul_long (Failed)
164 - MolPairStyle:lj_sdk_coul_long (Failed)
165 - MolPairStyle:lj_sdk_coul_table (Failed)
169 - MolPairStyle:lj_switch3_coulgauss_long (Failed)
192 - MolPairStyle:tip4p_long_soft (Failed)
195 - MolPairStyle:wf_cut (Failed)
203 - AtomicPairStyle:buck_coul_cut_qeq_point (Failed)
204 - AtomicPairStyle:buck_coul_cut_qeq_shielded (Failed)
221 - AtomicPairStyle:edip (Failed)
224 - AtomicPairStyle:hybrid-eam (Failed)
228 - AtomicPairStyle:meam (Failed)
229 - AtomicPairStyle:meam_spline (Failed)
230 - AtomicPairStyle:meam_sw_spline (Failed)
233 - AtomicPairStyle:reaxff (Failed)
234 - AtomicPairStyle:reaxff_lgvdw (Failed)
235 - AtomicPairStyle:reaxff_noqeq (Failed)
236 - AtomicPairStyle:reaxff_tabulate (Failed)
246 - ManybodyPairStyle:bop (Failed)
247 - ManybodyPairStyle:bop_save (Failed)
248 - ManybodyPairStyle:comb (Failed)
250 - ManybodyPairStyle:drip (Failed)
251 - ManybodyPairStyle:drip_real (Failed)
256 - ManybodyPairStyle:ilp-graphene-hbn (Failed)
257 - ManybodyPairStyle:ilp-graphene-hbn_notaper (Failed)
258 - ManybodyPairStyle:kolmogorov_crespi_full (Failed)
261 - ManybodyPairStyle:lcbop (Failed)
262 - ManybodyPairStyle:lebedeva_z (Failed)
263 - ManybodyPairStyle:meam (Failed)
269 - ManybodyPairStyle:nb3b_harmonic (Failed)
270 - ManybodyPairStyle:pace_product (Failed)
271 - ManybodyPairStyle:pace_recursive (Failed)
272 - ManybodyPairStyle:polymorphic_sw (Failed)
273 - ManybodyPairStyle:polymorphic_tersoff (Failed)
281 - ManybodyPairStyle:tersoff (Failed)
286 - ManybodyPairStyle:tersoff_shift (Failed)
287 - ManybodyPairStyle:tersoff_table (Failed)
295 - BondStyle:gaussian (Failed)
312 - AngleStyle:cosine_delta (Failed)
314 - AngleStyle:cosine_shift (Failed)
336 - KSpaceStyle:ewald_tri (Failed)
338 - KSpaceStyle:pppm_ad (Failed)
339 - KSpaceStyle:pppm_cg (Failed)
341 - KSpaceStyle:pppm_cg_tiled (Failed)
347 - KSpaceStyle:pppm_disp_tip4p (Failed)
354 - KSpaceStyle:pppm_tip4p (Failed)
359 - KSpaceStyle:scafacos_direct (Failed)
360 - KSpaceStyle:scafacos_ewald (Failed)
361 - KSpaceStyle:scafacos_fmm (Failed)
362 - KSpaceStyle:scafacos_fmm_tuned (Failed)
363 - KSpaceStyle:scafacos_p2nfft (Failed)
364 - FixTimestep:adapt_coul (Failed)
367 - FixTimestep:addforce_const (Failed)
368 - FixTimestep:addforce_variable (Failed)
369 - FixTimestep:addtorque_const (Failed)
372 - FixTimestep:aveforce_variable (Failed)
374 - FixTimestep:drag (Failed)
378 - FixTimestep:heat (Failed)
381 - FixTimestep:momentum (Failed)
383 - FixTimestep:nph (Failed)
384 - FixTimestep:nph_sphere (Failed)
386 - FixTimestep:npt_iso (Failed)
387 - FixTimestep:npt_sphere_aniso (Failed)
388 - FixTimestep:npt_sphere_iso (Failed)
389 - FixTimestep:npt_sphere_tri (Failed)
397 - FixTimestep:nvt (Failed)
399 - FixTimestep:oneway (Failed)
412 - FixTimestep:rigid_npt_small (Failed)
424 - FixTimestep:shake_angle (Failed)
426 - FixTimestep:smd_couple (Failed)
429 - FixTimestep:spring_couple (Failed)
430 - FixTimestep:spring_rg (Failed)
432 - FixTimestep:spring_tether (Failed)
433 - FixTimestep:temp_berendsen (Failed)
434 - FixTimestep:temp_csld (Failed)
435 - FixTimestep:temp_csvr (Failed)
436 - FixTimestep:temp_rescale (Failed)
455 - DihedralStyle:table_cut_linear (Failed)
457 - DihedralStyle:table_linear (Failed)
458 - DihedralStyle:table_spline (Failed)
466 - ImproperStyle:harmonic (Failed)
468 - ImproperStyle:inversion_harmonic (Failed)
-
ほとんどは軽微な数値エラーと、インテルコンパイラ利用時の lattice 関連。大筋では問題無いと判断。
メモ
- ccfep でビルド。GPU 演算ノードにてテスト。
- HPC-X 2.13.1 の実行環境利用時は並列計算で通信エラー。HPC-X 2.11 の実行環境に切り替えることでエラーは解消。
- NETCDF を on にするとビルドに失敗するため、今回は回避。システム側のライブラリの問題の可能性が高い。
- システムの python 3.6 を使っていれば、conda 環境は不要であった可能性が高い。また、NETCDF も on にできたかもしれない。