Skip to main content

CAE CPU Pinning Recommendations

This document records recommendations about CPU pinning when using CAE software. It applies to MPI applications.

Introduction​

CAE software usually uses many processes for parallel computation and uses MPI for inter-process communication. When building the MPI runtime environment, there are two common ways under the SLURM scheduler:

  • Direct: start MPI processes directly with the srun command.
  • Indirect: start MPI processes with mpirun or mpiexec.

Under these two approaches, the pinning method and results differ.

Direct​

  • Resources are allocated by the scheduler.
  • The scheduler performs pinning automatically based on resource allocation.

Indirect​

  • Resources are allocated by the scheduler.
  • The MPI runtime performs pinning automatically based on resource allocation.
caution

Note: In this mode, OpenMPI gets pinning information through the cgroup cpuset.

CPU Pinning Recommendations​

Because OPENMPI obtains pinning information through the cgroup cpuset, the cluster must enable cgroup and use the following configuration:

TaskPlugin = task/affinity,task/cgroup
TaskPluginParam = autobind=threads

CGroup_ConstrainCores=Yes
CGroup_TaskAffinity=no # Not required for fsched 10.67+

Where:

  • task/affinity: enables pinning.
  • task/cgroup: enables cgroup.
  • autobind=threads: automatically pins to threads.