Change Job Resource Requests
fsched (fsched-10.61+) supports changing resource requests for running jobs. Currently supported parameters include:
MinMemoryNode- Change the minimum memory required per node for a job (in MB).
MinCPUsNode- Change the minimum CPU count required per node for a job.
Purpose
By changing resource requests for running jobs, you can reduce over-allocated resources during execution so that other jobs pending due to insufficient resources can run.
warning
- Reducing the requested resources while a job is running does not reduce the job's actual resource usage, so other jobs may cause node resource load to become too high.
tip
- The root user can modify resource requests for any user's jobs.
- Regular users can modify resource requests only for their own jobs.
- Currently
MinCPUsNodeonly supports decreasing the value; increasing it will produce an error (the dev version will add support for increasing it).
Cluster Configuration
Modify the following configuration to use the plugin select/cons_tres_ex, which supports changing resource requests for running jobs.
SelectType=select/cons_tres_ex
Example 1: How to Modify
-
Submit a job, wait for it to run, and modify
MinMemoryNode.[root@head-1 ~]# srun -w compute-1 --mem 8000 sleep 120&
[1] 17033
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
289 partition sleep root R 0:02 1 compute-1[root@head-1 ~]# scontrol update job 289 MinMemoryNode=3000 -
View the modified
MinMemoryNode.[root@head-1 ~]# scontrol show job 289
JobId=289 JobName=sleep
UserId=root(0) GroupId=root(0) MCS_label=N/A
Priority=4294901751 Nice=0 Account=root QOS=normal WCKey=*
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
RunTime=00:00:24 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2024-12-05T14:01:30 EligibleTime=2024-12-05T14:01:30
AccrueTime=Unknown
StartTime=2024-12-05T14:01:30 EndTime=Unknown Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-12-05T14:01:30
Partition=partition-9C3RA AllocNode:Sid=head-1:14475
ReqNodeList=compute-1 ExcNodeList=(null)
NodeList=compute-1
BatchHost=compute-1
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=1,mem=3000M,node=1,billing=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=3000M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=sleep 120
WorkDir=/root
Power= -
Submit a job, wait for it to run, and modify
MinCPUsNode.[root@head-1 ~]# srun -w compute-1 --mincpus=3 sleep 120&
[1] 18209
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
290 partition sleep root R 0:02 1 compute-1[root@head-1 ~]# scontrol update job 290 MinCPUsNode=2 -
View the modified
MinCPUsNode.[root@head-1 ~]# scontrol show job 290
JobId=290 JobName=sleep
UserId=root(0) GroupId=root(0) MCS_label=N/A
Priority=4294901750 Nice=0 Account=root QOS=normal WCKey=*
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
RunTime=00:00:28 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2024-12-05T14:09:32 EligibleTime=2024-12-05T14:09:32
AccrueTime=Unknown
StartTime=2024-12-05T14:09:32 EndTime=Unknown Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-12-05T14:09:32
Partition=partition-9C3RA AllocNode:Sid=head-1:14475
ReqNodeList=compute-1 ExcNodeList=(null)
NodeList=compute-1
BatchHost=compute-1
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=2,mem=1M,node=1,billing=2
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=2 MinMemoryNode=1M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=sleep 120
WorkDir=/root
Power=
Example 2: Allow Other Jobs to Run After Modification
-
Modify
MinMemoryNode.[root@head-1 ~]# sinfo -Nel
Thu Dec 5 14:35:37 2024
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
compute-1 1 partition-9C3RA* allocated 4 4:1:1 13926 0 1 (null) none
compute-2 1 partition-9C3RA* idle 4 4:1:1 13926 0 1 (null) none[root@head-1 ~]# srun -w compute-1 --mem 8000 sleep 120&
[3] 22415
[2] Done srun -w compute-1 --mincpus=3 sleep 120
[root@head-1 ~]# srun -w compute-1 --mem 8000 sleep 120&
[4] 22428
[root@head-1 ~]# srun: job 300 queued and waiting for resources
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
300 partition sleep root PD 0:00 1 (Resources)
299 partition sleep root R 0:04 1 compute-1[root@head-1 ~]# scontrol update job 299 MinMemoryNode=2000
[root@head-1 ~]# srun: job 300 has been allocated resources
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
299 partition sleep root R 0:31 1 compute-1
300 partition sleep root R 0:03 1 compute-1 -
Modify
MinCPUsNode.[root@head-1 ~]# sinfo -Nel
Thu Dec 5 14:35:37 2024
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
compute-1 1 partition-9C3RA* allocated 4 4:1:1 13926 0 1 (null) none
compute-2 1 partition-9C3RA* idle 4 4:1:1 13926 0 1 (null) none[root@head-1 ~]# srun -w compute-1 --mincpus=3 sleep 120&
[1] 21997
[root@head-1 ~]# srun -w compute-1 --mincpus=3 sleep 120&
[2] 22014
[root@head-1 ~]# srun: job 298 queued and waiting for resources
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
298 partition sleep root PD 0:00 1 (Resources)
297 partition sleep root R 0:03 1 compute-1[root@head-1 ~]# scontrol update job 297 MinCPUsNode=1
[root@head-1 ~]# srun: job 298 has been allocated resources
[root@head-1 ~]# squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
297 partition sleep root R 0:28 1 compute-1
298 partition sleep root R 0:04 1 compute-1