[Last update] 2023-May-11

Table of Contents

• Connection
  • Login to RCCS
  • Register SSH Public Key and Password for Web
  • Login Shell
• Whole System of RCCS
• RCCS Resources
  • CPU Points and Queue Factor
  • Checking Resources
  • Individual Limitation of Resources
• Queueing System
  • Overview of Queueing System
  • Queue Classes
  • Show Job Status
  • Submit Your Jobs
  • Delete Jobs
  • Hold/Release Jobs
  • Get Information of Finished Jobs
• Build and Run
  • Command to Build
  • Running Parallel Program
  • Environment Modules
• Package Programs
• Other RCCS Special Commands
  • Queueing System Related
  • Job wait time estimator (waitest)
  • Show Resources Used
  • Utility Commands for Batch Jobs
  • Manipulation of Files on Computation Nodes
• Inquiry

Connection

Login to RCCS

• Frontend nodes (ccfep.ims.ac.jp) can be accessed using ssh with public key authentication.
• All computers will stop during 9:00-19:00 on the first Monday of each month because of maintenance. The maintenance time might be extended.
• Access to frontend nodes is allowed only from IPv4 address assigned to Japan or other registered IP addresses. See Application for SSH connection from outside Japan for details.

Register SSH Public Key and  Password for Web

Please prepare a public/private key pair of ssh.  If you do not know the procedure, please search in internet by yourself.

First registration / Missing your username or password for web

1. Open https://ccportal.ims.ac.jp/en/user/password to request mail for registration in web browser.
2. Fill your email which is written in your application, then press button "E-mail new password".
3. After you recieve an email from RCCS, open URL in the mail to login in web browser.
4. Fill your new password in "Password" and "Confirm password".
5. Paste your public key in "Public key".
6. Press "Save" button.

Using your username and password for web

1. Open https://ccportal.ims.ac.jp/en/frontpage in web browser, then fill your username and password and press "Log in" button.
2. Press "My account" which is located in top right corner.
3. Press "Edit" tab.
4. To change password, fill current password and new passwords.
5. Paste your public key.
6. Press "Save" button.

Login Shell

• /bin/bash, /bin/tcsh and /bin/zsh are available.
• You can select login shell in the sampe page as ssh public key.  It will take some time to change login shell.
• You can custumize your .login or .cshrc in your home directory, but be carefully.

Whole System of RCCS

• Whole system of RCCS is shown in the figure below.
• Interactive nodes are ccfep (4 nodes). You can build or debug applications on them.
• There are four kinds of disks, namely /home, /save, /gwork, /lwork(computation node only) and /ramd(computation node only). Access speed and data lifetime are different.
• Width of lines between disks and computers represents transfer speed. Wider is faster.
• Disk /gwork, /lwork and /ramd is temporary space for your calculation. All files will be DELETED after the completion of your job.
• Disk /lwork is NVMe SSD.  The capacity is about 1.5 TB. You can use 11.9 GB / core.
• Disk /ramd is RAM disk of each node. The size is about 240 GB or 1,008 GB, depending on node type. The sum of memory used in the job and RAM disk is controlled by the queuing system.
• There are no differences between /home and /save other than the names. (Formerly, there had been a difference regarding backuping policy.)
• Use of /tmp, /var/tmp or /dev/shm is not allowed. Jobs using those dirctories will be killed by the administrator.

RCCS Resources

CPU Points and Queue Factor

CPU points are spent when you use CPU or GPU.
Queue factors are determined as follows on each systems.

• On ccfep, CPU points will be consumed according to CPU time spent.
• On computation nodes, CPU points will be calculated from the elaps of processes.
• We don't charge money for the supercomputer usage.

If you want to know your current CPU points, run "showlim -c".

Checking Resources

• CPU points used by batch jobs and total disk usage (corresponding to "showlim -c" and "showlim -d", respectively) are updated every 10 minutes.
• CPU points used in interactive nodes are updated on 2:20.
• If CPU points run out, all running jobs of your group will be killed and further job submissions won't be allowed.
• If your disk usage exceed the limit, new job submissions will be rejected.

Individulal Limitation of Resources

Access to "Limiting Resources Page"  with your web browser.

• Only representative user can limit maximum resources of each members.
• Normal user can only view the values of maximum resources.
• Maximum available number of cpus, point and amount of disks can be limitted.

Queueing System

Overview of Queueing System

Queue Classes

Queue class for all users

• Jobs must be finished before the scheduled maintenance.
• Only around half of available nodes may accept long jobs (more than a week).
• You can omit jobtype in your jobscript except for jobtype=largemem; other types can be judged from the requested resources.
• 80 nodes of TypeC (160 vnodes) will be shared by "vnode" and "core" jobs.
• Short "vnode" jobs can be assigned to a "largemem" node.
• Short "core" jobs can be assigned to a "gpu" node.

Special queue class

The settings of queue class are following.

Show Job Status

ccfep% jobinfo [-h HOST] [-q QUEUE] [-c|-s|-m|-w] [-n] [-g GROUP|-a] [-n]

choose additional information type

One of those options can be added.

• -c  show the latest result (number of GPUs and jobtype etc. are not available)
• -s  show status summary of the queue(s)
• -m  show memory information
• -w  show working directory of jobs
• -n  show node status

jobs of other users

• -g  group member's jobs are also shown
  • If you belong to multiple groups, try -g [GROUP name] to specify a group.
• -a  show all users' jobs
  • most of information (username etc.) will be hidden.

queue specification

You don't need to specify this usually, since all the user available queues (PN and PNR[0-9]) are the default targets.

• -h HOST: specify host type (only cclx is available)
• -q QUEUE: specify queue name (such as PN or PNR1)

Example: show queue summary

In User/Group stat, number of jobs, cpu cores, and gpus can be shown by "jobinfo -s". Limitation about those resources are also shown. In Queue Status, number of waiting jobs and available nodes/cores/gpus will be shown.

ccfep% jobinfo -s

User/Group Stat:
--------------------------------------------------------------------------------
 queue: H                       | user(***)             | group(***)           
--------------------------------------------------------------------------------
   NJob (Run/Queue/Hold/RunLim) |    1/    0/    0/-    |    1/    0/    0/6400
   CPUs (Run/Queue/Hold/RunLim) |    4/    0/    0/-    |    4/    0/    0/6400
   GPUs (Run/Queue/Hold/RunLim) |    0/    0/    0/-    |    0/    0/    0/  48
   core (Run/Queue/Hold/RunLim) |    4/    0/    0/1200 |    4/    0/    0/1200
--------------------------------------------------------------------------------
note: "core" limit is for per-core assignment jobs (jobtype=core/gpu*)

Queue Status (H):
----------------------------------------------------------------------
      job         | free   |     free     | # jobs  |  requested  
      type        | nodes  | cores (gpus) | waiting | cores (gpus)
----------------------------------------------------------------------
week jobs
----------------------------------------------------------------------
1-4 vnodes        |    705 |  90240       |       0 |      0      
5+  vnodes        |    505 |  64640       |       0 |      0      
largemem          |      0 |      0       |       0 |      0      
core              |    179 |  23036       |       0 |      0      
gpu               |      0 |      0 (0)   |       0 |      0 (0)  
----------------------------------------------------------------------
long jobs
----------------------------------------------------------------------
1-4 vnodes        |    325 |  41600       |       0 |      0      
5+  vnodes        |    225 |  28800       |       0 |      0      
largemem          |      0 |      0       |       0 |      0      
core              |     50 |   6400       |       0 |      0      
gpu               |      0 |      0 (0)   |       0 |      0 (0)  
----------------------------------------------------------------------
Job Status at 2023-01-29 17:40:12

"core (Run/Queue/Hold/RunLim)" in User/Group Stat is a CPU cores limit for jobtype=core/gpu* jobs, where CPU cores used by jobtype=vnode/largemem jobs are not taken into account. For example, in this example, you can use up to 1200 cores in total.

Example: show status of jobs

You can see the latest status of jobs by specifying "-c" option. (-l option can be added but is ignored.)

ccfep% jobinfo -c

--------------------------------------------------------------------------------
Queue   Job ID Name            Status CPUs User/Grp       Elaps Node/(Reason)
--------------------------------------------------------------------------------
H       9999900 job0.csh       Run      16  zzz/---    24:06:10 ccc047
H       9999901 job1.csh       Run      16  zzz/---    24:03:50 ccc003
H       9999902 job2.sh        Run       6  zzz/---     0:00:36 ccc091
H       9999903 job3.sh        Run       6  zzz/---     0:00:36 ccc091
H       9999904 job4.sh        Run       6  zzz/---     0:00:36 ccc090
...
H       9999989 job89.sh       Run       1  zzz/---     0:00:11 ccg013
H       9999990 job90.sh       Run       1  zzz/---     0:00:12 ccg010
--------------------------------------------------------------------------------

If you don't specify "-c", you can also see some more details (jobtype and number of gpus). The information may be slightly (2-3 minutes usually) old, though.

ccfep% jobinfo
--------------------------------------------------------------------------------
Queue   Job ID Name            Status CPUs User/Grp       Elaps Node/(Reason)
--------------------------------------------------------------------------------
H(c)    9999900 job0.csh       Run      16  zzz/zz9    24:06:10 ccc047
H(c)    9999901 job1.csh       Run      16  zzz/zz9    24:03:50 ccc003
H(c)    9999902 job2.sh        Run       6  zzz/zz9     0:00:36 ccc091
H(c)    9999903 job3.sh        Run       6  zzz/zz9     0:00:36 ccc091
H(c)    9999904 job4.sh        Run       6  zzz/zz9     0:00:36 ccc090
...
H(g)    9999989 job89.sh       Run     1+1  zzz/zz9     0:00:11 ccg013
H(g)    9999990 job90.sh       Run     1+1  zzz/zz9     0:00:12 ccg010
--------------------------------------------------------------------------------

Example: show working directory

In case you forget where you ran jobs, try "-w" option. The working directories will be shown like below.

ccfep% jobinfo -w
--------------------------------------------------------------------------------
Queue   Job ID Name            Status Workdir
--------------------------------------------------------------------------------
H      9999920 H_12345.sh      Run    /home/users/zzz/gaussian/mol23
H      9999921 H_23456.sh      Run    /home/users/zzz/gaussian/mol74
...

(You can't use "-c" in this case.)

Submit Your Jobs

Description of the header part

Your have to write a script file which is written in C-shell to submit your job. An example for each system is following.

• csh, bash (/bin/sh), zsh can be used for the job submission script.
• lines started with #PBS are common, regardless of the shell type.
• Sample scripts can be found in ccfep:/local/apl/lx/(application name)/samples/.

• Nnode: # of physical node or virtual node
• Ncore: # of reserved cores per physical node or virutal node
  • largemem, vnode: 64 (for virtaul node) or 128 (for physical node)
• Nproc: # of processes per node
• Nthread: # of threads per process
• Jobtype: largemem, small, core, gpu, gpup, gpuv
  • large: 7.875GB / core
  • vnoe: 1.875GB / core
  • core: job for less than 63 cores
  • gpu: GPU jobs
• Ngpu: # of GPUs

Example of "Needed Number of CPU": Case of 192 mpi processes on 3 vnodes

Example of "Needed Number of CPU": Case of GPGPU

You can find some other examples in this page.

Job submission

After you write the script, type following command to submit.

ccfep% jsub -q (H|HR[0-9]) [-g XXX] [-W depend=(afterok|afterany):JOBID1[:JOBID2...]] script.csh

If you want to submit your jobs by 'Supercomputing Consortium for Computational Materials Science' group, use -g option.  (XXX is name of its group)
You can describe dependency of jobs using -W option.
If you want to describe dependency that a job should run after the dependent job exit successfully, use keyword "afterok".
If a job should run after the dependent job including abnormal exit, use keyword "afterany".
Sample script files exist in ccfep:/apl/*/samples/.
 

Sequential Jobs (step jobs)

You can easily submit sequential jobs by using --step or --stepany command.
 

step job 1 (afterok): run a series of jobs sequentially. If a job exited abnormally, following jobs will be deleted.

ccfep% jsub -q (H|HR[0-9]) [-g XXX] --step [-W depend=(afterok|afterany):JOBID1[:JOBID2...]] script.csh script2.csh ...

step job 2 (afterany): run a series of jobs sequentially. If a job finished, next job will run regardless of the exit status.

ccfep% jsub -q (H|HR[0-9]) [-g XXX] --stepany [-W depend=(afterok|afterany):JOBID1[:JOBID2...]] script.csh script2.csh ...

Example:

ccfep% jsub -q H --stepany job1.csh job2.csh job3.csh

Define variables which can be used in jobscript

You can define variables via -v option. The argument to the option is comma-separated list of variable definitions, (variable-name)=(value).

ccfep% jsub -v INPUT=myfile.inp,OUTPUT=myout.log script.sh

In this example, $INPUT and $OUTPUT will be set to myfile.inp and myout.log in "script.sh", respectively.

Notes:

• if multiple -v is specified, only the last one will be active.
• this can't define ncpus, jobtype etc. values in select= section of the jobscript.

Delete Jobs

First of all, you should get ID of the target jobs using "jobinfo" command. Then, run following command, where the RequestID is the job id.

ccfep% jdel RequestID

Hold/Release Jobs

You can prevent queued job to run by the following command (hold a job, in other words). (Use jobinfo to get target job id.)

ccfep% jhold RequestID

You can release the restiction using jrls command.

ccfep% jrls RequestID

Get Information of Finished Jobs

You can get information of finished jobs, such as finish time, elaps time, parallel efficiency, by joblog command.

ccfep% joblog [-d ndays] [-o item1[,item2[,...]]]

If the target period is not specified, information about jobs of current FY will be shown. Following options can be used to specify the target period.

• -d ndays: jobs finished within last "ndays"
  • -d 7 : jobs finished within last 7 days
• -y year: jobs in FY "year"
  • -y 2021 : jobs in FY 2021 (2021/4-2022/3)
• -f YYYY[MM[DD[hh[mm]]]] -t YYYY[MM[DD[hh[mm]]]]: from-to specification. (month, day, hour, etc. can be omitted)
  • -f 202107 -t 202108 : jobs finished in July or August of year 2021.

You can customize items which are displayed in -o option.  Available keywords are:

• queue: Queue name
• jobid: Job ID
• user: User name
• group: Group name
• node: Job head node name
• Node: All node names
• type: jobtype
• start: Start time（YYYY/MM/DD HH:MM）
• Start: Start time（YYYY/MM/DD HH:MM:SS）
• finish: Finish time（YYYY/MM/DD HH:MM）
• Finish: Finish time（YYYY/MM/DD HH:MM:SS）
• elaps: Elaps
• cputime: Total CPU time
• used_memory: Used memory size
• ncpu: Number of reserved cpu cores
• ngpu: Number of reserved gpus
• nproc: Number of MPI processes
• nsmp: Number of threads per process
• peff: Efficiency of job
• attention: Bad efficiency job
• command: Job name
• point: CPU points
• all: show all

e.g. 1: Show job ID, start and end datetime, CPU points of jobs finished within last 10 days.

ccfep% joblog -d 10 -o jobid,start,finish,point

e.g. 2: Show job ID, end datetime, CPU points, and workding directory of jobs in FY2020.

ccfep% joblog -y 2020 -o jobid,finish,point,Workdir

e.g. 3: Show all the parameters of jobs finished within last two days.

ccfep% joblog -d 2 -o all

Build and Run

Command to Build

• gcc, aocc, NVIDIA HPC SDK are already installed.
• For Intel oneAPI, only the libraries are installed. Compilers (icc, ifort, icpc etc.) are not installed. If you need Intel compilers, please install oneAPI Base Toolkit or/and oneAPI HPC Toolkit by yourself into your directory.
• For gcc,  system default one (8.5) and gcc-toolset ones (versions 9.2, 10.3, and 11.2) are installed. You can use gcc-toolset gccs by module command.  (e.g. module load gcc-toolset/11)

Please also check the package program list page for the available libraries and MPI environments.

Running Parallel Program

Please specify correct numbers of MPI processes and OpenMP threads in "mpiprocs" and "ompthreads" values of select line in your job script.
e.g. 12 CPUs, 4 MPI processes and 3 OpenMP threads case -> ncpus=12:mpiprocs=4:ompthreads=3

Please also check sample jobs scripts by RCCS (locating at /apl/(software name)/samples ).
 

Number of threads specification for OpenMP jobs

When submitting job using "jsub", the value specified by "ompthreads" is considered as number of OpenMP threads.
You can specify or change the value by setting OMP_NUM_THREADS envirnment variable manually.
If you run a command not via "jsub" (on frontend node for example), you should set OMP_NUM_THREADS environment variable manually.
 

Host list specification for MPI

List of hosts can be found in the file name specified in PBS_NODEFILE environment variable.
MPI environments installed by RCCS (Intel MPI and OpenMPI) automatically consider this environment variable.
Therefore, you can skip specification of machine file when invoking "mpirun" command.

e.g. 4 MPI * 3 OpenMP hybrid parallel

#!/bin/sh
#PBS -l select=1:ncpus=12:mpiprocs=4:ompthreads=3:jobtype=core
#PBS -l walltime=24:00:00
cd $PBS_O_WORKDIR
mpirun -np 4 /some/where/my/program options

• OpenMP thread number is determined from the value specified by "ompthreads" in select. (equivalent to OMP_NUM_THREADS=3)
• MPI machine file name can be omitted when invoking mpirun (if that mpirun is installed by RCCS).
  • You can specify machine file to filename specified in PBS_NODEFILE env variable. (This doesn't change anything.)

Environment Modules

Environment Modules ("module" command) is available from July 2018. See this page for detailed information.

Package Programs

• The installed pacakge programs for each systems are listed in https://ccportal.ims.ac.jp/en/installed_applications.
• Sample script files are located in ccfep:/apl/appname/samples/.
• Installed directories are located in /apl/appname/ on each systems.
• Applications compiled by center are listed in https://ccportal.ims.ac.jp/en/how_to_configure with detail description.

Request installation you want to use

Please fill the following items and send it rccs-admin[at]ims.ac.jp.

• Software name and version that you want to use
• Overview of the software and its feature
• Necessity of installation to supercomputers in RCCS
• URL of the software development

Special Commands of RCCS

Related Queueing System

You can find descriptions about "jobinfo", "jsub", "jdel", "jhold", "jrls", and "joblog" above in this page.

Submitting Gaussian Jobs

Case of Gaussian 16

ccfep% g16sub [-q "QUE_NAME"] [-j "jobtype"] [-g "XXX"] [-walltime "hh:mm:ss"] [-noedit] \
              [-rev "g16xxx"] [-np "ncpus"] [-ngpus "n"] [-mem "size"] [-save] [-mail] input_files

• Command "g09sub" are also available to use Gaussian 09.
• Default walltime is set to 72 hours.  Please set excepted time for calculation and extra to do long job or run early.
• If you want to know the meaning of options and examples, please just type "g16sub".
• %mem, %nproc, %cpu in the original input files will be overwritten by g16sub / g09sub. Please specify those values with -mem, -np etc. command line options.
  • You can prevent this automatic overwrite by -noedit option. But this is not recommended.
  • Safe maximum %mem value will be automatically assigned by g09sub/g16sub. You may not need to set that manually unless you need to reduce the amount of memory.

example case: g16c01, timelimit 24 hours, 12 cores job (Gaussian input file name is my_gaussian_input.gjf)

ccfep% g16sub -rev g16c01 -walltime 24:00:00 -np 12 my_gaussian_input.gjf

To use formchk, please check this FAQ item.

Job wait time estimator (waitest)

On ccfep, you can estimate start time of job with "waitest" command, where waitest assumes jobs will run for the time specified with "walltime" parameter". Therefore, the estimated datetime will be the worst case estimation. On the other hand, other user's jobs submitted later might run earlier than your ones. (Many parameters, such as job priority, jobtype, and remedies for (small) groups, are involved.) You shouldn't expect high accuracy for the estimation. You may also need to check queue status (jobinfo -s) additionally.

Basic Usage

Estimate start time of submitted job(s):

$ waitest [jobid1] ([jobid2] ...)

Estimate start time of not yet submitted job(s):

$ waitest -s [job script1] ([jobscript2] ...)

Example 1: estimate start time of a submitted job

[user@ccfep2]$ waitest 4923556
Current Date   : 2019-10-15 14:32:30
2019-10-15 14:32:30 ...
2019-10-15 16:40:44 ...
2019-10-15 22:26:07 ...
2019-10-16 00:43:43 ...
2019-10-16 03:03:11 ...
2019-10-16 05:58:00 ...
2019-10-16 11:34:12 ...
Job 4923556 will run at 2019-10-16 13:03:11 on ccnn500,ccnn496,ccnn497,ccnn494,ccnn495,ccnn489.
Estimation completed.

Example 2: estimate start time of not yet submitted jobs

[user@ccfep2]$ waitest -s run_small_4N.sh run_small_6N.sh run_small_8N.sh
Current Date   : 2019-10-15 14:34:39

Job Mapping "run_small_4N.sh" -> jobid=1000000000
Job Mapping "run_small_6N.sh" -> jobid=1000000001
Job Mapping "run_small_8N.sh" -> jobid=1000000002

2019-10-15 14:34:39 ...
2019-10-15 16:40:52 ...
2019-10-15 22:26:15 ...
2019-10-16 00:43:51 ...
2019-10-16 03:03:18 ...
2019-10-16 05:58:08 ...
2019-10-16 11:34:10 ...
Job 1000000001 will run at 2019-10-16 13:03:18 on ccnn500,ccnn496,ccnn497,ccnn494,ccnn495,ccnn489.
2019-10-16 13:28:41 ...
2019-10-16 16:00:36 ...
2019-10-16 20:52:10 ...
2019-10-17 01:08:27 ...
Job 1000000002 will run at 2019-10-17 03:03:18 on ccnn458,ccnn329,ccnn515,ccnn520,ccnn373,ccnn437,ccnn380,ccnn352.
2019-10-17 03:33:56 ...
2019-10-17 06:43:51 ...
2019-10-17 08:50:03 ...
2019-10-17 11:34:10 ...
2019-10-17 13:03:18 ...
2019-10-17 16:08:16 ...
2019-10-17 18:35:08 ...
2019-10-17 20:36:56 ...
2019-10-17 22:38:49 ...
Job 1000000000 will run at 2019-10-18 00:18:34 on ccnn760,ccnn789,ccnn791,ccnn787.
Estimation completed.

(In some case, larger jobs will run first due to the job priority and other parameters.)

Example 3: show periodically estimated wait time for general types of jobs

For some of basic types of jobs, wait time for those jobs are estimated periodically. The result can be accessed via the following command.

[user@ccfep2]$ waitest --showref

Showing Used Resources

ccfep% showlim (-cpu|-c|-disk|-d) [-m]

• -cpu|-c: Show used point and limited value.
• -disk|-d: Show current disk size and limited value.
• -m: Show values of each members.

example 1: show CPU points (approved and used) of YOU and WHOLE YOUR GROUP

ccfep% showlim -c

example 2: show CPU points (approved and used) of YOU, YOUR GROUP MEMBERS, and WHOLE GROUP

ccfep% showlim -c -m

example 3: show disk status (approved and used)  of YOU, YOUR GROUP MEMBERS, and WHOLE GROUP

ccfep% showlim -d -m

Utility Commands for Batch Jobs

Limit the walltime of command

• ps_walltime command which was provided from RCCS was deleted.  Please use timeout command instead.

Showing statistic of current job

• jobstatistic command which was provided from RCCS was deleted.  Please use joblog command instead.

Manipulation of Files on Computation Nodes

You can access local files on computation nodes which cannot be directly accessed from the frontend nodes (ccfep) via "remsh" command.

remsh hostname command options

• hostname: Hostname such as cccc???, cccca???, ccnn???, or ccnf???.
• command: Command to be executed on the node. Acceptable commands are ls, cat, cp, head, tail, and find.
• options: Options of command.

Example: how to access ramdisk of a computation node, ccnnXXX, by user "zzz"

remsh ccnnXXX ls /ramd/users/zzz

remsh ccnnXXX tail /ramd/users/zzz/99999/fort.10

Host names and jobids of your jobs can be found in the output of "jobinfo" command. (see above for the usage)

Inquiry

See https://ccportal.ims.ac.jp/en/contact.