Single-Node Jobs

Small scale Abaqus jobs can be run on a single node.

Job Script Template

Here is an example job submission script for a single-node Abaqus job:

 1#!/usr/bin/bash -l
 2# 
 3#SBATCH --job-name=my_job
 4#SBATCH --nodes=1 
 5#SBATCH --ntasks-per-node=1
 6#SBATCH --cpus-per-task=14
 7#SBATCH --time=0:10:00 
 8#SBATCH --mem-per-cpu=4000M
 9#SBATCH --account=aero012345
10
11# Load modules 
12module load apps/abaqus/2018
13# module load languages/intel/2020-u4              # BlueCrystal (Phase 4)
14# module load lang/intel-parallel-studio-xe/2020   # BluePebble
15
16# Unset SLURM's Global Task ID for ABAQUS's PlatformMPI to work 
17unset SLURM_GTIDS 
18
19# Launch Abaqus 
20abaqus job=<job-name> cpus=${SLURM_CPUS_PER_TASK} user=<usub-file> mp_mode=threads double=both interactive

Note

On a single node, the best choice for best performance between mp_mode=threads and mp_mode=mpi will depend on your specific problem. threads is recommended and known to work well; mpi may be more efficient for larger problems.

How to use

  1. Create a new job file (e.g. job.sh) with the template above in a folder in your scratch space

    • Copy your Abaqus job file (.inp) and Fortran user subroutine (if using) into the same folder

  2. Update the number of processors to use (line 6)

    • On BluePebble the maximum number of processors per standard node is 24

    • On BlueCrystal the maximum number of processors per node is 28

  3. Update the time limit (line 7)

    • You should update this to however long you require for your Abaqus job.

    • The format for the time limit is days-hours:minutes:seconds

    • See here for more help on sbatch --time

  4. Update the memory per cpu (line 8)

    • Increase this value if you encounter an oom-kill or out-of-memory error message in your output file

    • Requesting too little memory (<500M) may cause the compilation or linking of large user subroutines to fail

  5. Update the project account (line 9) to match your HPC project code

    • If you don’t know your HPC project code, you can run the command:

    sacctmgr show user withassoc format=account where user=$USER

  6. If you are using custom user subroutines:

    • Uncomment either line 13 or line 14 to add the Intel Fortran compiler depending on which system you are running on (BlueCrystal or BluePebble)

    • Replace <usub-file> on the last line with the name of your Fortran user subroutine source file (e.g. usub_czm.f)

  7. Replace <job-name> on the last line with the name of your Abaqus job input file (.inp)

  8. Submit the job to slurm:

    • Change directory (cd) into the folder containing your job files and job script

    • Run $ sbatch job.sh where job.sh is the name of your job script

Important

Make sure that you run your jobs using your scratch space in /user/work/<username>/ and not from your $HOME directory in /user/home/<username>. Your $HOME storage only has 20 GB (BlueCrystal) or 25 GB (BluePebble) of space, whereas your scratch space has 1 TB.