Batch High Performance Computing

The large high performance computing resources at MSI are often in high demand, with over 900 research groups and four to five thousand unique researcher logins competing for time. There are not enough resources to run every job immediately when it gets submitted, so a job scheduling program is required to keep things efficient and also allocate time fairly.

If you have not already connected your personal computer to a shell on the MSI cluster, review the guide on Connecting to HPC Resources first, and then come back here to learn how to run your programs on these systems.

SLURM is the name of the job scheduler that MSI uses to control the flow of research jobs flowing through the compute resources. To place jobs in the queue, a user must submit their jobs to the scheduler using a “batch script” that describes the resources necessary to complete the job.

This guide on Job Submission and Scheduling provides instructions on how to write the small “batch scripts” required to describe your job. Another guide that will also be useful for submitting batch jobs is this guide that helps you choose which SLURM partition to insert into your batch script. Once a script has been submitted, the scheduler will either begin running the job or place it in the queue to begin once the necessary compute resources become available.

Note that MSI also supports ways to run applications on the HPC resources in “Interactive” mode. See the guide for Connecting to Interactive HPC Resources to learn how to submit jobs via an interactive interface. Jobs submitted via the interactive methods are all scheduled behind the scenes by the same SLURM job scheduler that controls the batch jobs.