What is High Performance Computing (HPC)?
MSI's High Performance Computing (HPC) systems are designed with high speed networking, high performance storage, a range of GPUs, and large amounts of memory in order to support some of the most hardware intensive programs developed today.
MSI currently has one main HPC system, called Agate. Agate was put into service in Spring 2022 and was expanded with hardware from decommissioned Mangi nodes in Summer 2024. Agate nodes are built with some of the newest AMD processors and several nodes contain the latest NVIDIA A100 and A40 GPUs.
What can I do with HPC?
MSI’s HPC system combines incredible computing power with direct access to large high performance storage volumes. The system also gives researchers access to a wide variety of software packages, and includes popular programming languages such as Python, R, Matlab, and C compilers. This integration creates a computational environment that is flexible and powerful enough to accommodate most research needs. Researchers from departments across the University use MSI’s HPC resources every day to run simulations and computational jobs via either a Batch system or through a browser-based interactive interface.
Utilizing HPC
HPC Scheduling and Fairshare
For all MSI users to be able to access HPC resources efficiently and fairly, MSI employs a “fair share” job scheduler to help ensure an equitable mix of jobs.
The Slurm scheduler tracks system usage behind the scenes and uses this information to modify the fairshare score of the project that submitted the job. The goal of fairshare is to encourage researchers to utilize HPC resources efficiently by increasing the priority when scheduling jobs belong to projects who right-size their jobs, and decrease the priority of jobs belonging to projects who request HPC resources that go unused. In general, this means that when a group has recently used a large amount of resources, the priorities of their waiting jobs will be negatively affected until their usage decreases to their fairshare target once again.
The scheduler considers many factors when scheduling jobs and calculating priorities of waiting jobs, and the submitting project's fairshare score is only one component. MSI also uses queue time - the time that a job has been waiting to run - to affect the priority of any given job. The longer a job waits, the more the queue time will add to the job's overall priority. Other factors include the type of resources requested (such as GPUs), and the amount of resources requested, and the length of time the resources are requested for (referred to as the job's walltime).
Fairshare Priority Breakdown
To help groups determine what is impacting their job priority, the command sprio -u $USER gives a breakdown of the three main factors that determine priority. Using the sprio -u $USER command will display the AGE, JOBSIZE, and FAIRSHARE components that are used to calculate the job priority. The description of each of these three components can be broken down as follows:
- AGE component: This component of priority will increase as a job waits in the queue. Each group has a limited number of jobs that will gain AGE priority at a given time. The longer a job waits in the queue, the higher priority it will be given due to this AGE component.
- JOBSIZE component: This component of priority is based on the number of resources requested. A larger job will receive a higher priority.
- FAIRSHARE component: The FAIRSHARE component is adjusted based on the recent usage of their group members. Fairshare usage is proportional to the resources used by the group. CPU, Memory, and GPU resources are tracked separately and have different weights for Fairshare. For example; a single hour on a V100 GPU counts the same as 88 CPU·hours or 602 GB·hours of memory. The resources for each job are summed and multiplied by the duration of the job. The impact on priority decays over time, so a recently completed job has a greater impact on priority than a job that completed last week.
Additionally, the Slurm scheduler is configured to first try to schedule jobs requesting a large amount of resources, and then schedule smaller jobs around the larger jobs. Jobs requesting a large amount of resources need to reserve those resources in order to run, and they cannot run until there are sufficient free resources to fit such jobs. It is undesirable to have unused resources, so the scheduler uses smaller jobs to fill in the gaps created by the reservations of the large jobs. This scheduling behavior is called "backfill." It is overall more efficient to backfill smaller jobs around larger jobs. Accurate estimates of wall clock time on your jobs, especially small jobs, will help the scheduler schedule your jobs promptly.
MSI understands that no one wants to wait. It is also true that no scheduling policy can guarantee that no one will wait - only impossibly large machines can guarantee that - so the scheduler does its best to ensure a mix of jobs from all users can utilize the resources efficiently and fairly. We monitor queues and often adjust parameters to get better turnaround times on jobs. Your comments are always welcome.