bsub

Overview

bsub is used to submit jobs to the FSCHED cluster and is compatible with the LSF command-line interface. It supports both batch and interactive job submission and provides rich resource management and scheduling options.

---

Options

I/O and Working Directory

Option	Requires Value	Format	Description
-cwd	Yes	Path string	Specify the job working directory. Supports relative and absolute paths.
-i	Yes	File path	Read job standard input from the specified file.
-o	Yes	File path (supports %J, %I placeholders)	Append standard output to the specified file.
-oo	Yes	File path (supports %J, %I placeholders)	Overwrite standard output to the specified file.
-e	Yes	File path (supports %J, %I placeholders)	Append standard error to the specified file.
-eo	Yes	File path (supports %J, %I placeholders)	Overwrite standard error to the specified file.

Placeholder Notes

• %J - Job ID (automatically converted to %A in FSCHED)
• %I - Array index (automatically converted to %a in FSCHED)

Resource Allocation

Option	Requires Value	Format	Description
-n	Yes	min_tasks[,max_tasks]	Specify the number of tasks for the job. You can provide a minimum or a min/max range.
-m	Yes	Hostname list (space-separated)	Specify the host nodes on which the job runs.
-R	Yes	Resource requirement expression	Specify resource requirements; supports span, rusage[mem=...], rusage[tmp=...].
-x	No	None	Exclusive mode; the job exclusively uses the allocated nodes.
-gpu	Yes	number[/mode][/j_exclusive][/mps]	Specify GPU resource requirements.

Resource Expression Notes

The -R option supports the following formats:

• span[hosts=N] - Allocate across N hosts
• span[ptile=N] - Allocate N tasks per host
• rusage[mem=SIZE] - Memory requirement (for example rusage[mem=4G])
• rusage[tmp=SIZE] - Temporary space requirement (for example rusage[tmp=10G])

Process Resource Limits (ulimit)

Option	Requires Value	Format	Description
-M	Yes	Memory size (MB)	Set per-process memory limit (ulimit -m).
-C	Yes	Size or unlimited	Set core file size limit (ulimit -c).
-c	Yes	Time (milliseconds)	Set CPU time limit (ulimit -t).
-D	Yes	Size or unlimited	Set data segment size limit (ulimit -d).
-F	Yes	Size or unlimited	Set file size limit (ulimit -f).
-S	Yes	Size or unlimited	Set stack size limit (ulimit -s).
-v	Yes	Size or unlimited	Set virtual memory limit (ulimit -v).
-p	Yes	Count or unlimited	Set process count limit (ulimit -u).
-T	Yes	Count	Set thread count limit.
-ul	No	None	Inherit resource limits from the current shell.

Time and Scheduling

Option	Requires Value	Format	Description
-W	Yes	[hour:]minute	Set job run time limit. For example 2:30 means 2 hours 30 minutes.
-b	Yes	Date-time string	Set the earliest start time for the job.
-t	Yes	Date-time string	Set the termination deadline for the job.
-w	Yes	Dependency expression	Set job dependencies. Supports done(), ended(), exit(), started().
-H	No	None	Put the job into suspended state after submission.
-ti	No	None	Enable automatic orphan job termination.

Dependency Expression Examples

• done(12345) - Wait for job 12345 to finish successfully
• ended(12345) - Wait for job 12345 to finish (success or failure)
• exit(12345,0) - Wait for job 12345 to exit with code 0
• started(12345) - Wait for job 12345 to start running

Job Attributes

Option	Requires Value	Format	Description
-J	Yes	jobname[index_list]	Specify the job name. Supports arrays like myjob[1-10].
-Jd	Yes	Description text	Specify job description.
-q	Yes	Queue name	Specify the submission queue.
-P	Yes	Project name	Specify the project the job belongs to.
-G	Yes	User group name	Specify the fair-share group.
-sp	Yes	Priority (0-100)	Set the user-specified job priority; higher value means higher priority.
-r	No	None	Enable automatic retry on job failure.
-rn	No	None	Disable automatic retry on job failure.

Interactive Jobs

Option	Requires Value	Format	Description
-I	No	None	Submit an interactive job with direct input/output forwarding.
-Is	No	None	Submit an interactive job with a pseudo-terminal and shell.
-Ip	No	None	Submit an interactive job with a pseudo-terminal, supports terminal control sequences.
-IX	No	None	Submit an interactive job with X11 GUI.
-L	Yes	Shell path (absolute path)	Specify the login shell.
-tty	No	None	Display job output and error messages on the screen.

Mutually Exclusive Options

-I, -Is, -Ip, and -IX are mutually exclusive; only one can be used.

Execution Commands

Option	Requires Value	Format	Description
-E	Yes	Command string	Pre-execution command run before the job starts.
-Ep	Yes	Command string	Post-execution command run after the job ends.

Environment Variable Control

Option	Requires Value	Format	Description
-env	Yes	Environment variable spec	Control environment variable propagation. Supports all, none, selective propagation, etc.

Environment Variable Spec Format

• all - Propagate all environment variables (default)
• none - Do not propagate any environment variables
• VAR1,VAR2 - Propagate only specified variables
• ~VAR1,~VAR2 - Exclude specified variables
• VAR=value - Set variable values
• Combinations are allowed, for example all,~TMPDIR,MYVAR=newvalue

Notifications

Option	Requires Value	Format	Description
-B	No	None	Send an email notification when the job starts.
-N	No	None	Send an email notification when the job ends.
-Ne	No	None	Send an email notification when the job exits.
-u	Yes	Email address	Specify the email recipient address (not fully implemented yet).

Other

Option	Requires Value	Format	Description
-K	No	None	Wait for the job to complete and return the exit code. Batch jobs only.
-V	No	None	Print version information and exit.

Option Limitations

The -K option applies only to batch jobs and is mutually exclusive with interactive options (-I, -Is, -Ip, -IX).

---

Examples

Basic Submission

# Submit a simple batch job
bsub ./myjob.sh

# Submit a job with a name
bsub -J myjob ./run.sh

# Submit an array job
bsub -J "arrayjob[1-10]" ./process.sh

Resource Allocation

# Specify task count and run time
bsub -n 8 -W 2:30 ./parallel_job.sh

# Specify memory and temporary space requirements
bsub -R "rusage[mem=4G,tmp=10G]" ./data_process.sh

# Allocate across multiple nodes
bsub -n 16 -R "span[hosts=4]" ./mpi_job.sh

# Specify GPU resources
bsub -gpu 2 ./gpu_task.sh

Process Limits

# Set memory and CPU time limits
bsub -M 2048 -c 3600000 ./limited_job.sh

# Set multiple resource limits
bsub -M 4096 -S 102400 -F 1048576 ./constrained_job.sh

# Inherit current shell resource limits
bsub -ul ./inherit_limits.sh

I/O Redirection

# Redirect output to files
bsub -o output.%J.log -e error.%J.log ./task.sh

# Overwrite output files
bsub -oo output.log -eo error.log ./task.sh

# Use array task placeholders
bsub -J "job[1-5]" -o "out.%J.%I.log" ./array_task.sh

Scheduling Control

# Delay job start
bsub -b "2025-01-01:09:00" ./morning_job.sh

# Set dependencies
bsub -w "done(12345)" ./dependent_job.sh

# Suspend job after submission
bsub -H ./paused_job.sh

# Set priority
bsub -sp 80 ./high_priority.sh

Interactive Jobs

# Basic interactive job
bsub -I bash

# Interactive job with pseudo-terminal
bsub -Ip -n 4 python

# X11 GUI job
bsub -IX matlab

# Use a login shell
bsub -Is -L /bin/bash

Environment Variable Control

# Do not propagate any environment variables
bsub -env "none" ./clean_env.sh

# Propagate only specified variables
bsub -env "PATH,HOME,USER" ./selective.sh

# Exclude specific variables
bsub -env "all,~TMPDIR,~DISPLAY" ./filtered.sh

# Set variable values
bsub -env "DEBUG=1,WORKERS=8" ./configured.sh

Notifications and Waiting

# Send emails on start and end
bsub -B -N ./notify_job.sh

# Wait for job completion
bsub -K ./wait_job.sh
echo "Job completed, exit code: $?"

Combined Usage

# Full job configuration example
bsub \
  -J "production[1-100]" \
  -q normal \
  -n 8 \
  -W 4:00 \
  -R "span[hosts=1] rusage[mem=8G]" \
  -M 8192 \
  -o "logs/output.%J.%I.log" \
  -e "logs/error.%J.%I.log" \
  -P myproject \
  -sp 50 \
  -B -N \
  ./production_task.sh

---

Notes

Limitations

1. Resource expression support

   • The -R option supports only span[hosts=N], span[ptile=N], rusage[mem=SIZE], and rusage[tmp=SIZE].
   • Advanced LSF resource syntax such as select, cu, and affinity is not supported.

2. Email notifications

   • The -u option is not fully implemented yet; email notifications may be limited by system configuration.

3. GPU resources

   • The -gpu option supports parameter parsing; full functionality depends on FSCHED GPU scheduling configuration.

4. Priority

   • The -sp option accepts priorities 0-100; actual scheduling behavior depends on system configuration.

Placeholder Conversion

• LSF %J (job ID) is automatically converted to FSCHED %A.
• LSF %I (array index) is automatically converted to FSCHED %a.
• Placeholders are valid only for -o, -oo, -e, and -eo.

Option Conflicts

• Interactive options (-I, -Is, -Ip, -IX) are mutually exclusive.
• -K cannot be used together with interactive options.
• -o conflicts with -oo, and -e conflicts with -eo (the latter overwrites the former).

Default Behavior

• If -o or -e is not specified, standard output and error are discarded by default (sent to /dev/null).
• If -cwd is not specified, the current working directory at submission time is used.
• Automatic job retry is enabled by default (-r), and can be disabled with -rn.

Unsupported Options

The following LSF options are not supported in the current version. Using them will show a warning but will not affect job submission:

• CSM-related options (for example -alloc_flags, -cn_cu)
• Advanced LSF features (for example -app, -sla, -clusters)
• File staging (for example -f, -stage)
• Checkpointing and migration (for example -k, -mig)
• Alternative submission formats (for example -json, -yaml)

For the complete list of unsupported options, see the FSCHED technical documentation.