Slurm Workload Manager

.SH "DESCRIPTION"
\fBscontrol\fR is used to view or modify Slurm configuration including: job,
job step, node, partition, reservation, and overall system configuration. Most
of the commands can only be executed by user root or an Administrator. If an
attempt to view or modify configuration information is made by an unauthorized
user, an error message will be printed and the requested action will not occur.
If no command is entered on the execute line, \fBscontrol\fR will operate in an
interactive mode and prompt for input. It will continue prompting for input and
executing commands until explicitly terminated. If a command is entered on the
execute line, \fBscontrol\fR will execute that command and terminate. All
commands and options are case\-insensitive, although node names, partition
names, and reservation names are case\-sensitive (node names "LX" and "lx" are
distinct). All commands and options can be abbreviated to the extent that the
specification is unique. A modified Slurm configuration can be written to
a file using the \fIscontrol write config\fR command. The resulting file
will be named using the convention "slurm.conf.<datetime>" and located in the
same directory as the original "slurm.conf" file. The directory containing
the original slurm.conf must be writable for this to occur.

.SH "OPTIONS"
.TP
\fB\-a\fR, \fB\-\-all\fR
When the \fIshow\fR command is used, then display all partitions, their jobs
and jobs steps. This causes information to be displayed about partitions
that are configured as hidden and partitions that are unavailable to user's
group.
.IP

.TP
\fB\-M\fR, \fB\-\-clusters\fR=<\fIstring\fR>
The cluster to issue commands to. Only one cluster name may be specified.
Note that the \fBslurmdbd\fR must be up for this option to work properly, unless
running in a federation with either \fBFederationParameters=fed_display\fR
configured or the \fB\-\-federation\fR option set.
This option implicitly sets the \fB\-\-local\fR option.
.IP

.TP
\fB\-d\fR, \fB\-\-details\fR
Causes the \fIshow\fR command to provide additional details where available.
.IP

.TP
\fB\-\-federation\fR
Report jobs from federation if a member of one.
.IP

.TP
\fB\-F\fR, \fB\-\-future\fR
Report nodes in FUTURE state.
.IP
\f3\-\-json\fP, \f3\-\-json\fP=\fIlist\fR, \f3\-\-json\fP=<\fIdata_parser\fR>
Dump information as JSON using the default data_parser plugin or explicit
data_parser with parameters. All information is dumped, even if it would
normally not be. Sorting and formatting arguments passed to other options are
ignored; however, most filtering arguments are still used. This option is not
available for every command.
This option implicitly sets the \fB\-\-details\fR option.
.IP

.TP
\fB\-\-local\fR
Show only information local to this cluster. Ignore other clusters in the
federated if a member of one. Overrides \-\-federation.
.IP

.TP
\fB\-o\fR, \fB\-\-oneliner\fR
Print information one line per record.
.IP

.TP
\fB\-Q\fR, \fB\-\-quiet\fR
Print no warning or informational messages, only fatal error messages.
.IP

.TP
\fB\-\-sibling\fR
Show all sibling jobs on a federated cluster. Implies \-\-federation.
.IP

.TP
\fB\-u\fR, \fB\-\-uid\fR=<\fIuid\fR>
Attempt to update a job as user <uid> instead of the invoking user id.
.IP

.TP
\fB\-v\fR, \fB\-\-verbose\fR
Print detailed event logging. Multiple '\fB\-v\fR's will further increase
the verbosity of logging. By default only errors will be displayed.
.IP

.TP
\fB\-V\fR , \fB\-\-version\fR
Print version information and exit.
.IP

.TP
\f3\-\-yaml\fP, \f3\-\-yaml\fP=\fIlist\fR, \f3\-\-yaml\fP=<\fIdata_parser\fR>
Dump information as YAML using the default data_parser plugin or explicit
data_parser with parameters. All information is dumped, even if it would
normally not be. Sorting and formatting arguments passed to other options are
ignored; however, most filtering arguments are still used. This option is not
\fBcreate\fR <\fISPECIFICATION\fR>
Create a new node, partition, or reservation. See the full list of parameters
below.
.IP

.TP
\fBcompleting\fR
Display all jobs in a COMPLETING state along with associated nodes in either a
COMPLETING or DOWN state.
.IP

.TP
\fBdelete\fR <\fISPECIFICATION\fR>
Delete the entry with the specified \fISPECIFICATION\fR. The following
\fISPECIFICATION\fR choices are available:
.IP

.RS
.TP
\fBNodeName\fR=<\fInodelist\fR>
Node name or list of node names to delete. Only dynamic nodes that have no
running jobs and that are not part of a reservation can be deleted.

Multiple node names may be specified using simple node range expressions
(e.g., "lx[10\-20]"). Nodesets can also be specified by themselves or mixed with
node range expressions, using a comma as a list separator. If the keyword "ALL"
is specified alone, then the delete will be attempted against all the nodes in
the local cluster (and will fail on nodes ineligible for deletion).
.IP
.TP
\fBPartitionName\fR=<\fIname\fR>
Name of partition to delete. The partition must have no associated jobs running
or pending, otherwise the request is denied. It may be helpful to modify pending
jobs or to change the partition state ahead of time.
Note that if this partition is present in the \fBslurm.conf\fR file, it will be
recreated the next time the controller is restarted or reconfigured.
.IP
.TP
\fBReservationName\fR=<\fIname\fR>
Name of reservation to delete. The reservation must have no associated jobs,
otherwise the request is denied. It may be helpful to modify pending jobs to not
use the reservation. You can also plan the reservation's deletion ahead of time
by setting an \fBEndTime\fR or \fBDuration\fR.
.IP
.RE
.IP

.TP
\fBerrnumstr\fR <\fIERRNO\fP>
Given a Slurm error number, return a descriptive string.
.IP


.TP
\fBhold\fR <\fIjob_list\fR>
Prevent a pending job from being started (sets its priority to 0).
Use the \fIrelease\fP command to permit the job to be scheduled.
The job_list argument is a comma separated list of job IDs OR
"jobname=" with the job's name, which will attempt to hold all jobs having
that name.
Note that when a job is held by a system administrator using the \fBhold\fP
command, only a system administrator may release the job for execution (also
see the \fBuhold\fP command). When the job is held by its owner, it may also
be released by the job's owner.
Additionally, attempting to hold a running job will have not suspend or cancel
it. But, it will set the job priority to 0 and update the job reason field,
which would hold the job if it was requeued at a later time.
.IP

.TP
\fBlistjobs\fR [<\fINodeName\fR>]
Print jobs running on the host that runs this. This contacts any slurmstepd's
running locally, and does not contact slurmctld

Use <\fINodeName\fR> if using --enable-multiple-slurmd.
.IP

.TP
\fBlistpids\fR [<\fIjob_id\fR>[.<\fIstep_id\fR>]] [<\fINodeName\fR>]
Print a listing of the process IDs in a job step (if JOBID.STEPID is provided),
or all of the job steps in a job (if \fIjob_id\fP is provided), or all of the job
steps in all of the jobs on the local node (if \fIjob_id\fP is not provided
or \fIjob_id\fP is "*"). This will work only with processes on the node on
which scontrol is run, and only for those processes spawned by Slurm and
their descendants. Note that some Slurm configurations
(\fIProctrackType\fP value of \fIpgid\fP)
are unable to identify all processes associated with a job or job step.

Note that the NodeName option is only really useful when you have multiple
slurmd daemons running on the same host machine. Multiple slurmd daemons on
one host are, in general, only used by Slurm developers.
.IP

.TP
\fBliststeps\fR [<\fINodeName\fR>]
Print steps running on the host that runs this. This contacts any slurmstepd's
running locally, and does not contact slurmctld

Use <\fINodeName\fR> if using --enable-multiple-slurmd.
.IP

.TP
\fBnotify\fR <\fIjob_id\fR> <\fImessage\fR>
Send a message to standard error of the salloc or srun command or batch job
they are responding. If \fB\-\-clusters\fR is specified it will only ping the
primary slurmctld daemon of the specified cluster.
.IP

.TP
\fBpower\fR  {up|down} [asap|force] {ALL|<\fINodeList\fR>|<\fINodeSet\fR>} [Reason=<\fIreason\fR>]
Control power state of the provided node list/set. For 'power down', the
optional ASAP/FORCE flag will be added to the power down request, but will
otherwise be rejected for power up requests. All arguments will be processed
insensitive of case except for the node list/set. Optional \fIreason\fR may be
be specified only when powering down. This subcommand obsoletes the
prior usage of scontrol's update command:

scontrol update NodeName=<\fInodes\fR>
                State={POWER_UP|POWER_DOWN|POWER_DOWN_ASAP|POWER_DOWN_FORCE}

.RS
Commands:
.TP
\fBdown\fR
Will use the configured \fISuspendProgram\fR program to explicitly place
node(s) into power saving mode. If a node is already in the process of being
powered down, the command will only change the state of the node but won't have
any effect until the configured \fISuspendTimeout\fR is reached. Use of this
command can be useful in situations where a \fIResumeProgram\fR, like
\fIcapmc\fR in Cray machines, is stalled and one wants to restore the node to
"IDLE" manually. In this case rebooting the node and setting the state to
"power down" will cancel the previous "power up" state and the node will become
"IDLE".
.IP

.TP
\fBdown asap\fR
Will drain the node(s) and mark them for power down. Currently running jobs
will complete first and no additional jobs will be allocated to the node(s).
.IP

.TP
\fBdown force\fR
Will cancel all jobs on the node(s), power them down, and reset their state to
"IDLE".
.IP

.TP
\fBup\fR
Will use the configured \fIResumeProgram\fR program to explicitly move node(s)
out of power saving mode. If a node is already in the process of being powered
up, the command will only change the state of the node but won't have any
effect until the configured \fIResumeTimeout\fR is reached.
.IP
.RE

"ASAP" will also set the node reason to "Reboot ASAP" if the "reason" option
isn't specified and will set nextstate=UNDRAIN if nextstate isn't specified.
If the "nextstate" option is specified
as "DOWN", then the node will remain in a down state after rebooting. If
"nextstate" is specified as "RESUME", then the nodes will resume as normal
and the node's reason and "DRAIN" state will be cleared.
Resuming nodes will be considered as available in
backfill future scheduling and won't be replaced by idle nodes in a reservation.
The "reason" option sets each node's reason to a user\-defined message.
A default reason of "reboot requested" is set if no other reason is set on the
node.
The reason will be appended with: "reboot issued" when the reboot is issued;
"reboot complete" when the node registers and has a "nextstate" of "DOWN"; or
"reboot timed out" when the node fails to register within \fBResumeTimeout\fR.
You must specify either a list of nodes or that ALL nodes are to be rebooted.

\fBNOTE\fR: The reboot request will be ignored for hosts in the following
states: FUTURE, POWER_DOWN, POWERED_DOWN, POWERING_DOWN, REBOOT_ISSUED,
REBOOT_REQUESTED

\fBNOTE\fR: By default, this command does not prevent additional jobs from being
scheduled on any nodes before reboot.
To do this, you can either use the "ASAP" option or explicitly drain the nodes
beforehand.
You can alternately create an advanced reservation to
prevent additional jobs from being initiated on nodes to be rebooted.
Pending reboots can be cancelled by using "scontrol cancel_reboot <node>" or
setting the node state to "CANCEL_REBOOT".
A node will be marked "DOWN" if it doesn't reboot within \fBResumeTimeout\fR.
.IP

.TP
\fBreconfigure\fR
Instruct all slurmctld and slurmd daemons to re\-read the configuration file.
This mechanism can be used to modify configuration parameters set in
\fBslurm.conf\fR(5) without interrupting running jobs.
Starting in 23.11, this command operates by creating new processes for the
daemons, then passing control to the new processes when or if they
start up successfully. This allows it to gracefully catch configuration
problems and keep running with the previous configuration if there is a problem.
This will \fBnot\fR be able to change the daemons' listening TCP port settings
or authentication mechanism.
.IP

.TP
\fBrelease\fR <\fIjob_list\fR>
Release a previously held job to begin execution.
The job_list argument is a comma separated list of job IDs OR
"jobname=" with the job's name, which will attempt to release all jobs having
that name.
Also see \fBhold\fR.
.IP
.IP

.TP
\fBrequeuehold\fR [<\fIoption\fR>] <\fIjob_list\fR>
Requeue a running, suspended or finished Slurm batch job into pending state,
moreover the job is put in held state (priority zero).
The job_list argument is a comma separated list of job IDs.
A held job can be released using scontrol to reset its priority (e.g.
"scontrol release <job_id>"). The command accepts the following options:
.IP
.RS
.TP
\fBIncomplete\fR
Operate only on jobs (or tasks of a job array) which have not completed.
Specifically only jobs in the following states will be requeued:
CONFIGURING, RUNNING, STOPPED or SUSPENDED.
.IP

.TP
\fBState\fR=SpecialExit
The "SpecialExit" keyword specifies that the job
has to be put in a special state \fBJOB_SPECIAL_EXIT\fP.
The "scontrol show job" command will display the JobState as
\fBSPECIAL_EXIT\fP, while the "squeue" command as \fBSE\fP.
.RE
.IP

.TP
\fBresume\fR <\fIjob_list\fR>
Resume a previously suspended job.
The job_list argument is a comma separated list of job IDs.
Also see \fBsuspend\fR.

\fBNOTE\fR: A suspended job releases its CPUs for allocation to other jobs.
Resuming a previously suspended job may result in multiple jobs being
allocated the same CPUs, which could trigger gang scheduling with some
configurations or severe degradation in performance with other configurations.
Use of the scancel command to send SIGSTOP and SIGCONT signals would stop a
job without releasing its CPUs for allocation to other jobs and would be a
preferable mechanism in many cases.
If performing system maintenance you may want to use suspend/resume in the
following way. Before suspending set all nodes to draining or set all
partitions to down so that no new jobs can be scheduled. Then suspend
jobs. Once maintenance is done resume jobs then resume nodes and/or set all
partitions back to up.
Use with caution.
Only an account coordinator, operator, administrator, SlurmUser, or root can
resume jobs.
.IP

.TP
\fBschedloglevel\fR <\fILEVEL\fR>
same values as \fISlurmctldDebug\fP in the \fIslurm.conf\fP file) or
the name of the most detailed message type to be printed:
"quiet", "fatal", "error", "info", "verbose", "debug", "debug2", "debug3",
"debug4", or "debug5".
This value is temporary, and will be overwritten whenever the daemon reads the
slurm.conf configuration file. (Such as when the daemon is restarted or
\fBscontrol reconfigure\fR is executed).
.IP
.RS
.TP
\fBnodes\fR=<\fINodeName\fR>
If set, the request to change debug level is sent to the slurmd processes on
the nodes instead of to the slurmctld. A node range expression may be used for
NodeName.
.RE
.IP

.TP
\fBsetdebugflags\fR {+|\-}<\fIFLAG\fR> [{+|\-}<\fIFLAG\fR>] [nodes=<\fINODES\fR>]
Add or remove DebugFlags of the slurmctld daemon.
See "man slurm.conf" for a list of supported DebugFlags.

\fBNOTE\fR: Changing the value of some DebugFlags will have no effect without
restarting the slurmctld daemon, which would set DebugFlags based upon the
contents of the slurm.conf configuration file or the \fBSLURM_DEBUG_FLAGS\fR
environment variable. The environment variable takes precedence over the
setting in the slurm.conf.
.IP
.RS
.TP
\fBnodes\fR=<\fINodeName\fR>
The argument is optional and if used the request to change debug level
is sent to slurmd instead of slurmctld. A node range expression may be
used for NodeName.
.RE
.IP

.TP
\fBshow\fR <\fIENTITY\fR>[=<\fIID\fR>] or <\fIENTITY\fR> [<\fIID\fR>]
Display the state of the specified entity with the specified identification.
.IP
.RS
.TP
\fBaliases\fR
Returns all \fINodeName\fP values associated with a given \fINodeHostname\fP
(useful to get the list of virtual nodes associated with a
real node in a configuration where multiple slurmd daemons execute on a single
compute node).
.IP

.TP
\fBassoc_mgr\fR
.IP

.TP
\fBqos\fR=<\fIqos1\fR>[...,<\fIqosN\fR>]
Limit the QOS Records displayed to those with the specified QOS name(s).
.IP

.TP
\fBflags\fR={users|assoc|qos}
Specify the desired record type to be displayed. If no flags are specified,
all record types are displayed.
.RE
.IP

.TP
\fBbbstat\fR
Displays output from the current burst buffer plugin's status tool
(slurm_bb_get_status for lua or dwstat for datawarp). Options following
\fBbbstat\fR are passed directly to the status tool by the slurmctld daemon
and the response returned to the user. Equivalent to \fBdwstat\fR.
.IP

.TP
\fBburstbuffer\fR
Displays the current status of the BurstBuffer plugin.
.IP

.TP
\fBconfig\fR
Displays parameter names from the configuration files in mixed
case (e.g. SlurmdPort=7003) while derived parameters names are in upper case
only (e.g. SLURM_VERSION).
.IP

.TP
\fBdaemons\fR
Reports which daemons should be running on this node.
.IP

.TP
\fBdwstat\fR
Displays output from the current burst buffer plugin's status tool
(slurm_bb_get_status for lua or dwstat for datawarp). Options following
\fBdwstat\fR are passed directly to the status tool by the slurmctld daemon
and the response returned to the user. Equivalent to \fBbbstat\fR.
.IP

.TP
\fBfederation\fR
The federation name that the controller is part of and the
sibling clusters part of the federation will be listed.
.IP
\fBhostlistsorted\fR
Takes a list of host names and prints a sorted (but not unique) hostlist
expression for them. See \fBhostlist\fR.
.IP

.TP
\fBhostnames\fR
Takes an optional hostlist expression as input and
writes a list of individual host names to standard output (one per
line). If no hostlist expression is supplied, the contents of the
SLURM_JOB_NODELIST environment variable is used. For example "tux[1\-3]"
is mapped to "tux1","tux2" and "tux3" (one hostname per line).
.IP

.TP
\fBjob\fR
Displays statistics about all jobs by default. If an optional jobid is
specified, details for just that job will be displayed.
If the job does not specify socket\-per\-node, cores\-per\-socket or
threads\-per\-core then it will display '*' in the ReqS:C:T=*:*:* field.
.IP

.TP
\fBlicenses\fR
Displays statistics about all configured licenses (local and remote) by
default. If an optional license name is specified, details for just that
license will be displayed.

.TP
\fBnode\fR
Displays statistics about all nodes by default. If an optional nodename is
specified, details for just that node will be displayed.
.IP

.TP
\fBpartition\fR
Displays statistics about all partitions by default. If an optional partition
name is specified, details for just that partition will be displayed.
.IP

.TP
\fBreservation\fR
Displays statistics about all reservations by default. If an optional
reservation name is specified, details for just that reservation will be
displayed.
.IP

.TP
\fBslurmd\fR
Displays statistics for the slurmd running on the current node.
.IP


.TP
\fBtopology\fR [topology_name] [unit=NAME] [node=NAME]
Displays information about the selected topology layout. If no arguments are
provided, the default topology will be returned.

If a unit is specified, information about that unit will be shown. The
\fBunit\fR keyword can be replaced with \fBswitch\fR or \fBblock\fR (e.g.
block=b2).

If one node name is specified, all units connected to that node (and
their parent switches) will be shown.
If more than one node name is specified, only units that connect to all
named nodes will be shown.
.RE
.IP

.TP
\fBshutdown\fR <\fIOPTION\fR>
Instruct Slurm daemons to save current state and terminate.
By default, the Slurm controller (slurmctld) forwards the request all
other daemons (slurmd daemon on each compute node).
An \fIOPTION\fP of \fIslurmctld\fP or \fIcontroller\fP results in
only the slurmctld daemon being shutdown and the slurmd daemons
remaining active.
.IP

.TP
\fBsuspend\fR <\fIjob_list\fR>
Suspend a running job.
The job_list argument is a comma separated list of job IDs.
Use the \fIresume\fP command to resume its execution.
User processes must stop on receipt of SIGSTOP signal and resume
upon receipt of SIGCONT for this operation to be effective.
Not all architectures and configurations support job suspension.
If a suspended job is requeued, it will be placed in a held state.
The time a job is suspended will not count against a job's time limit.
Only an account coordinator, operator, administrator, SlurmUser, or root can
suspend jobs.
.IP

.TP
\fBtakeover\fR [<\fIINDEX\fR>]
Instruct one of Slurm's backup controllers (slurmctld) to take over system
control. By default the first backup controller (INDEX=1) requests control
from the primary and waits for its termination. After that, it switches from
backup mode to controller mode. If primary controller can not be contacted, it
directly switches to controller mode. This can be used to speed up the Slurm
controller fail\-over mechanism when the primary node is down.
This can be used to minimize disruption if the computer executing the
primary Slurm controller is scheduled down.

SlurmUser, or root) users. This operation may be enabled for non\-privileged
users by the system administrator by including the option "enable_user_top" in
the SchedulerParameters configuration parameter.
.IP

.TP
\fBtoken\fR [lifespan=<\fIlifespan\fR>] [username=<\fIusername\fR>]
Return an auth token which can be used to support JWT authentication if
\fIAuthAltTypes=auth/jwt\fP has been enabled on the system.
Supports two optional arguments. \fBlifespan=\fP may be used to specify the
token's lifespan in seconds. \fBusername\fP (only available to SlurmUser/root)
may be used to request a token for a different username. Lifespan can be set to
"infinite" to have a token that will not expire. Sites are suggested to use the
smallest lifespan required and generate tokens more often instead of using
longer lived tokens.
.IP

.TP
\fBuhold\fR <\fIjob_list\fR>
Prevent a pending job from being started (sets its priority to 0).
The job_list argument is a space separated list of job IDs or job names.
Use the \fIrelease\fP command to permit the job to be scheduled.
This command is designed for a system administrator to hold a job so that
the job owner may release it rather than requiring the intervention of a
system administrator (also see the \fBhold\fP command).
.IP

.TP
\fBupdate\fR <\fISPECIFICATION\fR>
Update job, step, node, partition, or reservation configuration per
the supplied specification. \fISPECIFICATION\fP is in the same format as the
Slurm configuration file and the output of the \fIshow\fP command described
above. It may be desirable to execute the \fIshow\fP command (described above)
on the specific entity you want to update, then use cut\-and\-paste tools to
enter updated configuration values to the \fIupdate\fP. Note that while most
configuration values can be changed using this command, not all can be changed
using this mechanism. In particular, the hardware configuration of a node or
the physical addition or removal of nodes from the cluster may only be
accomplished through editing the Slurm configuration file and executing
the \fIreconfigure\fP command (described above).
.IP

.TP
\fBupdate\fR <SuspendExc*>[=|+=|\-=]<\fILIST\fR>
Update \fISuspendExcNodes\fP, \fISuspendExcParts\fP, or \fISuspendExcStates\fP.
<\fILIST\fR> is either a NodeList, list of partitions, or list of node states
respectively. Use +=/\-= to add/remove nodes, partitions, or states to/from the
currently configured list. Use = to replace the current list.
SuspendExcNodes does not support "+="/"\-=" when the ":" option is used,
however, direct assignment "=" is always supported.
Consider using "scontrol show config | grep SuspendExc" to see current state of
these settings.
\fBNOTE\fP: Don't use scontrol wait_job in PrologSlurmctld or Prolog with
PrologFlags=Alloc as this will result in a deadlock.
.IP

\fBNOTE\fR: When using \fBwait_job\fR for an array job, use the
\fBSLURM_JOB_ID\fR environment variable to reference the job rather than the
\fBSLURM_ARRAY_JOB_ID\fR variable.

.TP
\fBwrite batch_script\fR <\fIjob_id\fR> [<\fIoptional_filename\fR>]
Write the batch script for a given job_id to a file or to stdout. The file will
default to \fIslurm\-\<job_id\>.sh\fP if the optional filename argument is not
given. The script will be written to stdout if \- is given instead of a filename.
The batch script can only be retrieved by an admin or operator, or by the
owner of the job.
.IP

.TP
\fBwrite config\fR <\fIoptional_filename\fR>
Write the current configuration to a file with the naming convention of
"slurm.conf.<datetime>" in the same directory as the original slurm.conf
file. If a filename is given that file location with a .<datetime> suffix is
created.
.IP

.SH "INTERACTIVE COMMANDS"
\fBNOTE\fR:
All commands listed below can be used in the interactive mode, but \fINOT\fP
on the initial command line.

.TP
\fBall\fR
Show all partitions, their jobs and jobs steps. This causes information to be
displayed about partitions that are configured as hidden and partitions that
are unavailable to user's group.
.IP

.TP
\fBcluster\fR <\fICLUSTER_NAME\fR>
The cluster to issue commands to. Only one cluster name may be specified.
.IP

.TP
\fBdetails\fR
Causes the \fIshow\fP command to provide additional details where available.
Job information will include CPUs and NUMA memory allocated on each node.
Note that on computers with hyperthreading enabled and Slurm configured to
allocate cores, each listed CPU represents one physical core.
Each hyperthread on that core can be allocated a separate task, so a job's
CPU count and task count may differ.
See the \fB\-\-cpu\-bind\fR and \fB\-\-mem\-bind\fR option descriptions in
srun man pages for more information.
.IP

.TP
\fBoneliner\fR
Print information one line per record.
.IP

.TP
\fBquiet\fR
Print no warning or informational messages, only fatal error messages.
.IP

.TP
\fBquit\fR
Terminate the execution of scontrol.
.IP

.TP
\fBverbose\fR
Print detailed event logging.
This includes time\-stamps on data structures, record counts, etc.
.IP

.TP
\fB!!\fR
Repeat the last command executed.
.IP

.SH "JOBS \- SPECIFICATIONS FOR UPDATE COMMAND"
Note that update requests done by either root, SlurmUser or Administrators are
not subject to certain restrictions. For instance, if an Administrator changes
the QOS on a pending job, certain limits such as the TimeLimit will not be
changed automatically as changes made by the Administrators are allowed to
violate these restrictions.

.TP
\fBJobName\fR=<\fIname\fR>
Identify the name of jobs to be modified or set the job's name to the
specified value.
When used to identify jobs to be modified, all jobs belonging to all users
are modified unless the \fIUserID\fP option is used to identify a specific user.
Either \fIJobId\fP or \fIJobName\fP is required.
.IP

.TP
\fBAccount\fR=<\fIaccount\fR>
Account name to be changed for this job's resource use.
Value may be cleared with blank data value, "Account=".
.IP

.TP
\fBAdminComment\fR=<\fIspec\fR>
.TP
\fBBurstBuffer\fR=<\fIspec\fR>
Burst buffer specification to be changed for this job's resource use.
Value may be cleared with blank data value, "BurstBuffer=".
Format is burst buffer plugin specific.
.IP

.TP
\fBClusters\fR=<\fIspec\fR>
Specifies the clusters that the federated job can run on.
.IP

.TP
\fBClusterFeatures\fR=<\fIspec\fR>
Specifies features that a federated cluster must have to have a sibling job
submitted to it. Slurm will attempt to submit a sibling job to a cluster if it
has at least one of the specified features.
.IP

.TP
\fBComment\fR=<\fIspec\fR>
Arbitrary descriptive string.
.IP

.TP
\fBContiguous\fR={yes|no}
Set the job's requirement for contiguous (consecutive) nodes to be allocated.
Possible values are "YES" and "NO".
Only a Slurm administrator or root can change this parameter.
.IP

.TP
\fBCoreSpec\fR=<\fIcount\fR>
Number of cores to reserve per node for system use.
The job will be charged for these cores, but be unable to use them.
Will be reported as "*" if not constrained.
.IP

.TP
\fBCPUsPerTask\fR=<\fIcount\fR>
Change the CPUsPerTask job's value.
.IP

.TP
\fBDeadline\fR=<\fItime_spec\fR>
Remove the job if no ending is possible before this deadline
(start > (deadline \- time[\-min])). Note that if neither \fBDefaultTime\fR nor
\fBMaxTime\fR are configured on the partition the job is in, the job will
need to specify some form of time limit (TimeMin/TimeLimit) if a deadline
is to be used. The specified deadline must be later than
the current time. Only pending jobs can have the deadline updated. Only a
Slurm administrator or root can change this parameter.

One or more time strings may be specified (e.g., 'tomorrow18:00'). If there is
a conflict between them, the last one will silently take precedence.
.IP

.TP
\fBDelayBoot\fR=<\fItime_spec\fR>
Change the time to decide whether to reboot nodes in order to satisfy job's
feature specification if the job has been eligible to run for less than this
time period. See salloc/sbatch man pages option \-\-delay\-boot.
.IP

.TP
\fBDependency\fR=<\fIdependency_list\fR>
Defer job's initiation until specified job dependency specification
is satisfied. Once a dependency is satisfied, it is removed from the job.
Cancel dependency with an empty dependency_list (e.g. "Dependency=").
<\fIdependency_list\fR> is of the form
<\fItype:job_id[:job_id][,type:job_id[:job_id]]\fR>.
Many jobs can share the same dependency and these jobs may even belong to
different  users.
.IP
.PD
.RS
.TP
\fBafter:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have begun
execution or been canceled.
.IP

.TP
\fBafterany:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have terminated.
.IP

.TP
\fBafternotok:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have terminated
in some failed state (non\-zero exit code, node failure, timed out, etc).
This dependency must be added while the specified job is still active or
within \fBMinJobAge\fR seconds after the specified job has ended.
.IP

.TP
\fBafterok:job_id[:jobid...]\fR
This job can begin execution after the specified jobs have successfully
executed (ran to completion with an exit code of zero).
This dependency must be added while the specified job is still active or
within \fBMinJobAge\fR seconds after the specified job has ended.
.IP

.TP
\fBEndTime\fR
The time the job is expected to terminate based on the job's time
limit. When the job ends sooner, this field will be updated with the
actual end time.
.IP

.TP
\fBExcNodeList\fR=<\fInodes\fR>
Set the job's list of excluded node. Multiple node names may be
specified using simple node range expressions (e.g. "lx[10\-20]").
Value may be cleared with blank data value, "ExcNodeList=".
.IP

.TP
\fBExtra\fR=<\fIspec\fR>
An arbitrary string enclosed in single or double quotes if using spaces or some
special characters. See <https://slurm.schedmd.com/extra_constraints.html> for
more details.
.IP

.TP
\fBFeatures\fR=<\fIfeatures\fR>
Set the job's required node features.
The list of features may include multiple feature names separated
by ampersand (AND) and/or vertical bar (OR) operators.
For example: \fBFeatures="opteron&video"\fR or \fBFeatures="fast|faster"\fR.
In the first example, only nodes having both the feature "opteron" AND
the feature "video" will be used.
There is no mechanism to specify that you want one node with feature
"opteron" and another node with feature "video" in case no
node has both features.
If only one of a set of possible options should be used for all allocated
nodes, then use the OR operator and enclose the options within square brackets.
For example: "\fBFeatures=[rack1|rack2|rack3|rack4]"\fR might
be used to specify that all nodes must be allocated on a single rack of
the cluster, but any of those four racks can be used.
A request can also specify the number of nodes needed with some feature
by appending an asterisk and count after the feature name.
For example "\fBFeatures=graphics*4"\fR
indicates that at least four allocated nodes must have the feature "graphics."
Parenthesis are also supported for features to be ANDed together.
For example "\fBFeatures=[(knl&a2a&flat)*4&haswell*2]"\fR indicates the
resource allocation should include 4 nodes with ALL of the features "knl",
"a2a", and "flat" plus 2 nodes with the feature "haswell".
Constraints with node counts may only be combined with AND operators.
Value may be cleared with blank data value, for example "Features=".
.IP

.TP
\fBGres\fR=<\fIlist\fR>
Specifies a comma\-delimited list of generic consumable resources requested per
node.
\fBJobId\fR=<\fIjob_list\fR>
Identify the job(s) to be updated.
The job_list may be a comma separated list of job IDs.
Either \fIJobId\fP or \fIJobName\fP is required.
If the \fIJobId\fP is equal to the \fIArrayJobID\fP then the update will affect
all of the individual jobs of the array. In that case, to update the specific
individual job, the form <\fIArrayJobID\fP>_<\fIArrayTaskId\fP> must be used.
.IP

.TP
\fBLicenses\fR=<\fIname\fR>
Specification of licenses (or other resources available on all nodes
of the cluster) as described in salloc/sbatch/srun man pages.
.IP

.TP
\fBMailType\fR=<\fItypes\fR>
Set the mail event types. Valid type values are NONE, BEGIN, END, FAIL,
REQUEUE, ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT),
STAGE_OUT (burst buffer stage out and teardown completed), TIME_LIMIT,
TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80
percent of time limit), TIME_LIMIT_50 (reached 50 percent of time limit) and
ARRAY_TASKS (send emails for each array task). Multiple type values may be
specified in a comma separated list. Unless the ARRAY_TASKS option is
specified, mail notifications on job BEGIN, END and FAIL apply to a job array
as a whole rather than generating individual email messages for each task in
the job array.
.IP

.TP
\fBMailUser\fR=<\fIname\fR>
Set the user to receive email notification of state changes. A blank string
will set the mail user to the default which is the submitting user.
.IP

.TP
\fBMCSLabel\fR=<\fIname\fR>
Set the MCS label for the job, which will take effect only if a MCS plugin
is loaded in slurmctld.
.IP

.TP
\fBMinCPUsNode\fR=<\fIcount\fR>
Set the job's minimum number of CPUs per node to the specified value.
.IP

.TP
\fBMinMemoryCPU\fR=<\fImegabytes\fR>
Set the job's minimum real memory required per allocated CPU to the specified
value. Either \fIMinMemoryCPU\fP or \fIMinMemoryNode\fP may be set, but not both.
.IP

See JobName.
.IP

.TP
\fBNice\fR[=<\fIadjustment\fR>]
Update the job with an adjusted scheduling priority within Slurm. With no
adjustment value the scheduling priority is decreased by 100. A negative nice
value increases the priority, otherwise decreases it. The adjustment range is
+/\- 2147483645. Only privileged users can specify a negative adjustment.
.IP

.TP
\fBNodeList\fR=<\fInodes\fR>
Change the nodes allocated to a running job to shrink its size.
The specified list of nodes must be a subset of the nodes currently
allocated to the job. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10\-20]"). After a job's allocation
is reduced, subsequent \fBsrun\fR commands must explicitly specify node and
task counts which are valid for the new allocation.

\fBNOTE\fR: The allocated nodes of jobs with arbitrary distribution can not be
updated.
.IP

.TP
\fBNumCPUs\fR=<\fImin_count\fR>[\-<\fImax_count\fR>]
Set the job's minimum and optionally maximum count of CPUs to be allocated.
.IP

.TP
\fBNumNodes\fR=<\fImin_count\fR>[\-<\fImax_count\fR>]
Set the job's minimum and optionally maximum count of nodes to be allocated.
If the job is already running, use this to specify a node count less than
currently allocated and resources previously allocated to the job will be
relinquished. After a job's allocation is reduced, subsequent \fBsrun\fR
commands must explicitly specify node and task counts which are valid for the
new allocation. Also see the \fINodeList\fP parameter above. This is the same
as ReqNodes.

\fBNOTE\fR: The node count of jobs with arbitrary distribution can not be
updated.
.IP

.TP
\fBNumTasks\fR=<\fIcount\fR>
Set the job's count of requested tasks to the specified value.
The number of tasks started in a specific step inside the allocation may differ
from this value, for instance when a different number of tasks is requested on
step creation. This is the same as ReqProcs.
.IP

.TP
like \fIFeatures\fP is. This list will override what is requested in Features.
See \fIFeatures\fP option above.
.IP

.TP
\fBPriority\fR=<\fInumber\fR>
Set the job's priority to the specified value.
Note that a job priority of zero prevents the job from ever being scheduled.
By setting a job's priority to zero it is held.
Set the priority to a non\-zero value to permit it to run.
Explicitly setting a job's priority clears any previously set nice value and
removes the priority/multifactor plugin's ability to manage a job's priority.
In order to restore the priority/multifactor plugin's ability to manage a
job's priority, hold and then release the job.
Only a Slurm administrator or root can increase job's priority.
.IP

.TP
\fBQOS\fR=<\fIname\fR>
Set the job's QOS (Quality Of Service) to the specified value,
or comma separated list of QOS.
If requesting a list it will be ordered based on the priority of the QOS given
with the first being the highest priority.
Value may be cleared with blank data value, "QOS=".
.IP

.TP
\fBReboot\fR={yes|no}
Set the job's flag that specifies whether to force the allocated nodes to reboot
before starting the job. This is only supported with some system configurations
and therefore it could be silently ignored.
.IP

.TP
\fBReqCores\fR=<\fIcount\fR>
Change the job's requested Cores count.
.IP

.TP
\fBReqNodeList\fR=<\fInodes\fR>
Set the job's list of required node. Multiple node names may be specified using
simple node range expressions (e.g. "lx[10\-20]").
Value may be cleared with blank data value, "ReqNodeList=".
.IP

.TP
\fBReqNodes\fR=<\fImin_count\fR>[\-<\fImax_count\fR>]
See NumNodes.
.IP

.TP
\fBReqProcs\fR=<\fIcount\fR>
\fBRequeue\fR={0|1}
Stipulates whether a job should be requeued after a node failure: 0
for no, 1 for yes.
.IP

.TP
\fBReservationName\fR=<\fIname\fR>
Set the job's reservation to the specified value.
Value may be cleared with blank data value, "ReservationName=".
.IP

.TP
\fBResetAccrueTime\fR
Set the job's accrue time value to 'now' meaning it will lose any time
previously accrued for priority. Helpful if you have a large queue of jobs
already in the queue and want to start limiting how many jobs can accrue time
without waiting for the queue to flush out.
.IP

.TP
\fBSiteFactor\fR=<\fIaccount\fR>
Specify the job's admin priority factor in the range of +/\-2147483645.
Only privileged users can modify the value.
.IP

.TP
\fBStdErr\fR=<\fIfilepath\fR>
Set the batch job's stderr file path.
Value may be reset to job default with blank data value, "StdErr=".

\fBNOTE\fR: By default, StdErr will be merged into StdOut.
.IP

.TP
\fBStdIn\fR=<\fIfilepath\fR>
Set the batch job's stdin file path.
Value may be reset to job default with blank data value, "StdIn=".

\fBNOTE\fR: By default, StdIn will be '/dev/null'.
.IP

.TP
\fBStdOut\fR=<\fIfilepath\fR>
Set the batch job's stdout file path.
Value may be reset to job default with blank data value, "StdOut=".

\fBNOTE\fR: By default, StdOut will be based on the \fIJobId\fP.
.IP

.TP
\fBShared\fR={yes|no}
See \fIOverSubscribe\fP option above.
give times like \fInow + count time\-units\fR, where the time\-units
can be \fIseconds\fR (default), \fIminutes\fR, \fIhours\fR, \fIdays\fR,
or \fIweeks\fR and you can tell Slurm to run the job today with the keyword
\fItoday\fR and to run the job tomorrow with the keyword
\fItomorrow\fR.
.IP
.RS
.PP
Notes on date/time specifications:
 \- although the 'seconds' field of the HH:MM:SS time specification is
allowed by the code, note that the poll time of the Slurm scheduler
is not precise enough to guarantee dispatch of the job on the exact
second. The job will be eligible to start on the next poll
following the specified time. The exact poll interval depends on the
Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
 \- if no time (HH:MM:SS) is specified, the default is (00:00:00).
 \- if a date is specified without a year (e.g., MM/DD) then the current
year is assumed, unless the combination of MM/DD and HH:MM:SS has
already passed for that year, in which case the next year is used.
.RE
.IP

.TP
\fBSwitches\fR=<\fIcount\fR>[@<\fImax\-time\-to\-wait\fR>]
When a tree topology is used, this defines the maximum count of switches
desired for the job allocation. If Slurm finds an allocation containing more
switches than the count specified, the job remain pending until it either finds
an allocation with desired switch count or the time limit expires. By default
there is no switch count limit and no time limit delay. Set the count
to zero in order to clean any previously set count (disabling the limit).
The job's maximum time delay may be limited by the system administrator using
the \fBSchedulerParameters\fR configuration parameter with the
\fBmax_switch_wait\fR parameter option.
Also see \fIwait\-for\-switch\fP.
.IP

.TP
\fBTasksPerNode\fR=<\fIcount\fR>
Change the job's requested TasksPerNode.
.IP

.TP
\fBThreadSpec\fR=<\fIcount\fR>
Number of threads to reserve per node for system use.
The job will be charged for these threads, but be unable to use them.
Will be reported as "*" if not constrained.
.IP

.TP
\fBTimeLimit\fR=<\fItime\fR>
The job's time limit.
Output format is [days\-]hours:minutes:seconds or "UNLIMITED".

.TP
\fBTimeMin\fR=<\fItimespec\fR>
Change TimeMin value which specifies the minimum time limit minutes of the job.
.IP

.TP
\fBUserID\fR=<\fIUID\fR or \fIname\fR>
Used with the \fIJobName\fP option to identify jobs to be modified.
Either a user name or numeric ID (UID), may be specified.
.IP

.TP
\fBwait\-for\-switch\fR=<\fIseconds\fR>
Change max time to wait for a switch <seconds> secs.
.IP

.TP
\fBWCKey\fR=<\fIkey\fR>
Set the job's workload characterization key to the specified value.
.IP

.TP
\fBWorkDir\fR=<\fIdirectory_name\fR>
Set the job's working directory to the specified value. Note that this may
only be set for jobs in the PENDING state, and that jobs may fail to launch
if they rely on relative paths to the originally submitted WorkDir.
.IP

.SH "JOBS \- SPECIFICATIONS FOR SHOW COMMAND"
The "show" command, when used with the "job" or "job <jobid>"
entity displays detailed information about a job or jobs. Much of
this information may be modified using the "update job" command as
described above. However, the following fields displayed by the show
job command are read\-only and cannot be modified:

.TP
\fBAllocNode:Sid\fR
Local node and system id making the resource allocation.
.IP

.TP
\fBBatchFlag\fR
Jobs submitted using the sbatch command have BatchFlag set to 1. The BatchFlag
will be incremented past 1 if the job is requeued due to a failure. Jobs
submitted using other commands have BatchFlag set to 0 and will not be
incremented.
.IP

.TP
\fBExitCode\fR=<\fIexit\fR>:<\fIsig\fR>
Exit status reported for the job by the wait() function.

.TP
\fBNodeListIndices\fR
The NodeIndices expose the internal indices into the node table
associated with the node(s) allocated to the job.
.IP

.TP
\fBNtasksPerN:B:S:C\fR=<\fItasks_per_node\fR>:<\fItasks_per_baseboard\fR>:<\fItasks_per_socket\fR>:<\fItasks_per_core\fR>
Specifies the number of tasks to be started per hardware component (node,
baseboard, socket and core).
Unconstrained values may be shown as "0" or "*".
.IP

.TP
\fBPreemptEligibleTime\fR
Time the job becomes eligible for preemption. Modified by PreemptExemptTime,
either from the global option in slurm.conf or the job QOS. This is hidden if
the job has not started or if PreemptMode=OFF.
.IP

.TP
\fBPreemptTime\fR
Time at which job was signaled that it was selected for preemption.
This value is only meaningful for \fIPreemptMode=CANCEL\fR and
\fIPreemptMode=REQUEUE\fR and for jobs in a partition or QOS that has a
GraceTime value designated. This is hidden if the job has not started or
if \fIPreemptMode=OFF\fR.
.IP

.TP
\fBPreSusTime\fR
Time the job ran prior to last suspend.
.IP

.TP
\fBReason\fR
The reason a job has not been started by the scheduler: e.g., waiting for
"Resources". Details of job reason codes are found on this page: