Important Commands
==================

This is is a list of the core PBS related commands.

Submitting
----------

Submit a job description stored in a file called ``jobfile``. The return value
of qsub an error or the job ID in the form of ``ID@tardis.mpib-berlin.mpg.de``.
You can use that id to query information about that specific job either with
the command line tools or on the `website`_.

.. code-block:: bash

   qsub job.pbs

Submit a job from a jobfile specifying resource parameters (use ``man qsub``) for
an extensive list of parameters:

.. code-block:: bash

   qsub job.pbs -N jobname -l walltime=10:0:0,mem=32gb

Directly submit a job through a pipe containing only a single command without
the need for a jobfile:

.. code-block:: bash

   echo "sleep 1m" | qsub

Get a quick shell on the testing queue, which should often be empty, because
only short jobs are allowed.

.. code-block:: bash

   qsub -I -q testing


Examples:

.. code-block:: bash

   krause@master:~/sleep> $ qsub job.pbs -l mem=32gb
   4294349.master.tardis.mpib-berlin.mpg.de
   krause@master:~/sleep> $
   krause@master:~> $ qsub  -I -l nodes=1:ppn=16
   qsub: waiting for job 4294350.master.tardis.mpib-berlin.mpg.de to start
   qsub: job 4294350.master.tardis.mpib-berlin.mpg.de ready
   krause@ood-32:~> $


Querying
--------

To query the state of a single job, run:

.. code-block:: bash

   qstat <job-id>

or in an alternate output format:

.. code-block:: bash

    qstat -a <job-id>

List all running jobs:

.. code-block:: bash

   qstat -r

List an overview of all queues (note that the Lm column truncates after 2 characters):

.. code-block:: bash

   qstat -q


Examples:

.. code-block:: bash

    krause@master:~> $ echo "sleep 5m" | qsub
    4294351.master.tardis.mpib-berlin.mpg.de
    krause@master:~> $ qstat -a 4294351

    master.tardis.mpib-berlin.mpg.de:
                                                                             Req'd  Req'd   Elap
    Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time
    -------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
    4294351.master.t     krause   default  STDIN              5712   --   -- 4000mb 24:00 R 00:00


    krause@master:~> $ qstat -q

    server: master

    Queue            Memory CPU Time Walltime Node  Run Que Lm  State
    ---------------- ------ -------- -------- ----  --- --- --  -----
    gpu                --      --       --      --    0   0 --   E R
    route              --      --       --      --    0   0 --   E R
    testing            --      --    01:00:00   --    0   0  2   E R
    default            --      --    36:00:00   --    1   0 --   E R
    longwall           --      --       --      --    1   0 51   E R
                                                   ----- -----
                                                       2     0

Selecting
---------

The reverse command to ``qsub <jobid>`` is ``qselect``. This is useful to
generate a number of active job IDs matching some properties. This is extremely
useful in scripts as the output is consistently formatted. For example the
option ``-s`` lets you query job with a specific state and the option ``-N``
for specific job name. For an extensive list consult the man page (``man
qselect``).

Examples:

.. code-block:: bash

    krause@master:~> $ qselect -s R
    4294352.master.tardis.mpib-berlin.mpg.de
    4294353.master.tardis.mpib-berlin.mpg.de
    4294354.master.tardis.mpib-berlin.mpg.de
    krause@master:~> $ qselect -N STDIN
    4294352.master.tardis.mpib-berlin.mpg.de
    krause@master:~> $ qselect -N STDIN | cut -d"." -f1
    4294352


Deleting
--------

Sometimes it is necessary to delete jobs from the queue with ``qdel [job id]``.
Either because you realized the code is not doing what it's supposed to be
doing or you want to start over for some other reason. The command will not
return any text by default, but throw an error when it couldn't find the job(s)
specified. The option ``-W <seconds>`` will send a signal to the job and wait
an amount of time to let it stop itself before being killed by the operating
system.

Examples:

.. code-block:: bash

    krause@master:~> $ qdel 4294357
    krause@master:~> $ qdel all
    krause@master:~> $ qdel all
    qdel: cannot find any jobs to delete

``qdel`` cannot handle wildcards to remove jobs matching the name 'project-*',
but you can use grep and xargs:

.. code-block:: bash

   qstat -a | grep 'project-' | cut -d"." -f1 | xargs -n1 qdel



Altering
--------

It may be useful to change job parameters while they are waiting in the queue.
This is more efficient than deleting and re-submitting the jobs because their
waiting time accounts to a priority value for the next scheduling decision. The
most useful feature here is to increase or decrease some resource requests with
the parameter ``-l``. As always, consult the man page ``man qalter`` or ``man
pbs_resources`` for details.
This will only work while the jobs is still queued. Afterwards only the system
administrator is able to change those values (send us an e-mail in that case).

Examples:

.. code-block:: bash



    krause@master:~> $ qstat -a 4294516

    master.tardis.mpib-berlin.mpg.de: 
                                                                             Req'd  Req'd   Elap
    Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time
    -------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
    4294516.master.t     krause   default  foo                 --      1   0 4000mb 24:00 Q   -- 
    krause@master:~> $ qalter -l walltime=10:0:0 4294516
    krause@master:~> $ qalter -l mem=32gb 4294516
    krause@master:~> $ qstat -a 4294516

    master.tardis.mpib-berlin.mpg.de: 
                                                                             Req'd  Req'd   Elap
    Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time
    -------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
    4294516.master.t     krause   default  foo                 --      1   0   32gb 10:00 Q   -- 




.. _website: https://tardis.mpib-berlin.mpg.de
