Commit 78eb9bc6 authored by Michael Krause's avatar Michael Krause 🎉
Browse files

software: change examples from qsub to sbatch

parent 3c2db3f6
...@@ -124,12 +124,14 @@ Usually users want to collect a number of jobs into batches and submit them with ...@@ -124,12 +124,14 @@ Usually users want to collect a number of jobs into batches and submit them with
This can be condensed down into a single line with the ``--wrap`` option to This can be condensed down into a single line with the ``--wrap`` option to
sbatch: sbatch. Here SLURM will create a job file on the fly, add the #!-line and
append the wrapped string to that file. This is syntax is being used a lot in
the examples in this document.
.. code-block:: bash .. code-block:: bash
for sub in $(seq -w 1 15) ; do\ for sub in $(seq -w 1 15) ; do
sbatch -c 2 -J main_$sub --wrap "python3 main.py $sub" ;\ sbatch -c 2 -J main_$sub --wrap "python3 main.py $sub"
done done
.. _slurm documentation: https://slurm.schedmd.com/quickstart.html .. _slurm documentation: https://slurm.schedmd.com/quickstart.html
...@@ -39,7 +39,7 @@ All other programs need to be wrapped into a job file to be run on the cluster n ...@@ -39,7 +39,7 @@ All other programs need to be wrapped into a job file to be run on the cluster n
.. code-block:: bash .. code-block:: bash
qsub -N ants -l walltime=100:0:0,mem=16gb complex-ants-pipeline.sh sbatch -J ants --time 100:0:0 --mem 16gb complex-ants-pipeline.sh
The file ``complex-ants-pipeline.sh`` contains your ANTS commands *preceeded* The file ``complex-ants-pipeline.sh`` contains your ANTS commands *preceeded*
......
...@@ -92,8 +92,8 @@ Old approach ...@@ -92,8 +92,8 @@ Old approach
.. code-block:: bash .. code-block:: bash
[krause@master ~] module load ashs module load ashs
[krause@master ~] $ASHS_ROOT/bin/ashs_main.sh \ $ASHS_ROOT/bin/ashs_main.sh \
-a /opt/software/ashs/data/1.0/ashs_atlas_mpib_20180208\ -a /opt/software/ashs/data/1.0/ashs_atlas_mpib_20180208\
-g T1.nii -f T2.nii -w wdir -T -Q -g T1.nii -f T2.nii -w wdir -T -Q
...@@ -136,13 +136,14 @@ Since ASHS version 1.0 this is the preferred way to submit segmentations for a n ...@@ -136,13 +136,14 @@ Since ASHS version 1.0 this is the preferred way to submit segmentations for a n
.. code-block:: bash .. code-block:: bash
[krause@master ~] qsub ashs_job.pbs -l nodes=1:ppn=8,mem=10gb sbatch ashs_job.jobs -c 8 --mem 10gb
with :file:`ashs_job.pbs` containing: with :file:`ashs_job.sh` containing:
.. code-block:: bash .. code-block:: bash
#!/bin/bash
module load ashs module load ashs
export SUBJECT_ID=23 export SUBJECT_ID=23
$ASHS_ROOT/bin/ashs_main.sh \ $ASHS_ROOT/bin/ashs_main.sh \
...@@ -155,14 +156,10 @@ Looping over a number of subject IDs would look something like this. ...@@ -155,14 +156,10 @@ Looping over a number of subject IDs would look something like this.
.. code-block:: bash .. code-block:: bash
for id in 01 03 10 32 ; do for id in 01 03 10 32 ; do
export SUBJECT_ID=$id export SUBJECT_ID=$id # all exports are passed to SLURM
qsub ashs_job.pbs -l nodes=1:ppn=8,mem=10gb -v SUBJECT_ID sbatch ashs_job.sh -c 8 --mem 10gb
done done
In this case the option `-v` to qsub instructs qsub to inherit a named,
formerly exported, variable. This way the variable `SUBJECT_ID` becomes
accessible in the job context.
Known Issues Known Issues
------------ ------------
......
...@@ -18,16 +18,14 @@ The simple subject loop shown in section :ref:`job_wrappers` and can be adapted ...@@ -18,16 +18,14 @@ The simple subject loop shown in section :ref:`job_wrappers` and can be adapted
SUBJECTS_DIR="./subjects" # The freesurfer import location SUBJECTS_DIR="./subjects" # The freesurfer import location
for subject in $SUBJECTS_IDS ; do for subject in $SUBJECTS_IDS ; do
echo "#PBS -m n" > tmp.pbs echo '#!/bin/bash' > tmp.sh
echo "#PBS -o $HOME/logs/" >> tmp.pbs echo "#SBATCH --output $HOME/logs/slurm-%j.out" >> tmp.sh
echo "#PBS -j oe" >> tmp.pbs echo "export FREESURFER_HOME=/opt/freesurfer/5.3.0" >> tmp.sh
echo "#PBS -d ." >> tmp.pbs echo "source \$FREESURFER_HOME/SetUpFreeSurfer.sh" >> tmp.sh
echo "export FREESURFER_HOME=/opt/freesurfer/5.3.0" >> tmp.pbs echo "export SUBJECTS_DIR=$SUBJECTS_DIR" >> tmp.sh
echo "source \$FREESURFER_HOME/SetUpFreeSurfer.sh" >> tmp.pbs echo "recon-all -all -i ${subject}.nii -subjid ${subject}" >> tmp.sh
echo "export SUBJECTS_DIR=$SUBJECTS_DIR" >> tmp.pbs
echo "recon-all -all -i ${subject}.nii -subjid ${subject}" >> tmp.pbs sbatch tmp.sh
qsub tmp.pbs
done done
......
...@@ -47,7 +47,7 @@ for all combinations of subjects and runs in parallel all in one go: ...@@ -47,7 +47,7 @@ for all combinations of subjects and runs in parallel all in one go:
sed -i "s|%RUN%|$run|g" $featfile sed -i "s|%RUN%|$run|g" $featfile
# submit the feat command with the current design file # submit the feat command with the current design file
echo "module load fsl ; feat $featfile" | qsub -d. sbatch --wrap "module load fsl ; feat $featfile"
done done
done done
...@@ -62,11 +62,11 @@ parallelize over a number of subjects. Example: ...@@ -62,11 +62,11 @@ parallelize over a number of subjects. Example:
.. code-block:: bash .. code-block:: bash
for icadir in *ica ; do for icadir in *ica ; do
echo "module load fsl_fix" >> tmpjob echo '#!/bin/bash" > tmp.sh
echo "source /etc/fsl/5.0/fsl.sh" >> tmpjob echo "module load fsl_fix" >> tmp.sh
echo "fix ${icadir} /opt/software/fix/1.06/training_files/Standard.RData 20" >> tmpjob echo "source /etc/fsl/5.0/fsl.sh" >> tmp.sh
qsub -N fix_foobar -j oe -d. tmpjob echo "fix ${icadir} /opt/software/fix/1.06/training_files/Standard.RData 20" >> tmp.sh
rm -f tmjob sbatch -J fix_foobar tmp.sh
done done
The :file:`fsl_fix` module will also load a special R environment module which will The :file:`fsl_fix` module will also load a special R environment module which will
......
...@@ -5,20 +5,27 @@ Matlab is a bit of a problem child on the Tardis. While the `MATLAB Distributed ...@@ -5,20 +5,27 @@ Matlab is a bit of a problem child on the Tardis. While the `MATLAB Distributed
Computing Server`_ product aims to implement a compatibility layer to a number Computing Server`_ product aims to implement a compatibility layer to a number
of PBS based clusters it just doesn't work reliably for a number of reasons. of PBS based clusters it just doesn't work reliably for a number of reasons.
Because there is only a limited number of shared licenses available it's also Because there are only a limited number of shared licenses available, it's
not feasible to run an arbitrary number of Matlab sessions in the form of jobs. generally not feasible to run an arbitrary number of Matlab sessions in the
A workaround is to "compile" a script and create a standalone redistribution form of jobs. A workaround is to "compile" a script and create a standalone
environment, which does not require a license to run. redistribution environment, which does not require a license to run.
.. important::
You can check https://lipsupport.mpib-berlin.mpg.de/licstat for the current
license usage at the institute. Keep in mind that *one* license is necessary
for each user on each unique host. So 3 users, each submitting jobs to
any 4 hosts will result in 12 licenses.
Different Matlab versions are available via environment modules. You can list Different Matlab versions are available via environment modules. You can list
them with :program:`module avail matlab` and activate a specific version with :program:`module them with :program:`module avail matlab` and activate a specific version with
load matlab/<version>`. :program:`module load matlab/<version>`.
Regular sessions Regular sessions
---------------- ----------------
**If** there are free licenses available and you just need a quick way to spawn a **If** there are free licenses available and you just need a quick way to spawn
single Matlab session, there is nothing wrong with just running Matlab as is. a single Matlab session, there is nothing wrong with just running Matlab as is.
This might especially be useful if you simply need a node with lot's of memory This might especially be useful if you simply need a node with lot's of memory
or if you want to test your code. In an interactive job you could simply enter or if you want to test your code. In an interactive job you could simply enter
"matlab" and it will warn you about there not being a display available and "matlab" and it will warn you about there not being a display available and
...@@ -27,12 +34,10 @@ start in command line mode. ...@@ -27,12 +34,10 @@ start in command line mode.
.. code-block:: bash .. code-block:: bash
[krause@master ~] qsub -i -l mem=100gb -q testing srun --mem 100gb -p test --pty /bin/bash # run an interactive job
qsub: waiting for job 5395814.master.tardis.mpib-berlin.mpg.de to start
qsub: job 5395814.master.tardis.mpib-berlin.mpg.de ready
[krause@ood-9.tardis.mpib-berlin.mpg.de ~] module load matlab/R2012b module load matlab/R2012b # in that job, run matlab
[krause@ood-9.tardis.mpib-berlin.mpg.de ~] matlab matlab
< M A T L A B (R) > < M A T L A B (R) >
...@@ -52,11 +57,8 @@ In a job context you would just run :program:`matlab -r main` with main.m contai ...@@ -52,11 +57,8 @@ In a job context you would just run :program:`matlab -r main` with main.m contai
.. code-block:: bash .. code-block:: bash
[krause@master ~] echo "module load matlab/R2014b; matlab -r main" | qsub -d. sbatch --wrap "module load matlab/R2014b; matlab -r main"
.. important:
Always check `http://lipsupport.mpib-berlin.mpgde/licstat`_ to see if there is an available license.
Compiling Compiling
...@@ -96,8 +98,8 @@ Example: ...@@ -96,8 +98,8 @@ Example:
.. code-block:: matlab .. code-block:: matlab
[krause@master ~] mcc -m project.m mcc -m project.m
[krause@master ~] ./run_project.sh /opt/matlab/interactive 42 5 ./run_project.sh /opt/matlab/interactive 42 5
------------------------------------------ ------------------------------------------
Setting up environment variables Setting up environment variables
--- ---
...@@ -107,8 +109,6 @@ Example: ...@@ -107,8 +109,6 @@ Example:
'42' '42' '42' '42' '42' '42' '42' '42' '42' '42'
[krause@master ~]
To include toolboxes in your script you have to add them during the compile To include toolboxes in your script you have to add them during the compile
step so they get included in your package. Matlab built-in toolboxes such as step so they get included in your package. Matlab built-in toolboxes such as
...@@ -124,7 +124,7 @@ package: ...@@ -124,7 +124,7 @@ package:
.. code-block:: bash .. code-block:: bash
[krause@master ~] cat matlab/tools/project/myrnd.m cat matlab/tools/project/myrnd.m
function X = myrnd(arg) function X = myrnd(arg)
X = normrnd(0, 1, arg, arg); X = normrnd(0, 1, arg, arg);
...@@ -141,13 +141,13 @@ version as MCR upon script invocation with :program:`run_project.sh`. ...@@ -141,13 +141,13 @@ version as MCR upon script invocation with :program:`run_project.sh`.
.. code-block:: matlab .. code-block:: matlab
[krause@master ~] module load matlab/R2014b module load matlab/R2014b
[krause@master ~] cat project.m cat project.m
function project(arg1) function project(arg1)
myrnd(str2num(arg1)) myrnd(str2num(arg1))
[krause@master ~] mcc -m project.m -a matlab/tools/project mcc -m project.m -a matlab/tools/project
[...] [...]
[krause@master ~] ./run_project.sh /opt/matlab/R2014b 3 ./run_project.sh /opt/matlab/R2014b 3
------------------------------------------ ------------------------------------------
Setting up environment variables Setting up environment variables
--- ---
...@@ -159,15 +159,13 @@ version as MCR upon script invocation with :program:`run_project.sh`. ...@@ -159,15 +159,13 @@ version as MCR upon script invocation with :program:`run_project.sh`.
1.8339 0.3188 0.3426 1.8339 0.3188 0.3426
-2.2588 -1.3077 3.5784 -2.2588 -1.3077 3.5784
[krause@master ~]
.. note:: .. note::
You only have to compile your project once and can then use it any number You only have to compile your project once and can then use it any number
of times. Matlab extracts your package to a shared hidden folder called of times. Matlab extracts your package to a shared hidden folder called
`.mcrCache<Version-Number>`. Those folders sometimes get corrupted by `.mcrCache<Version-Number>`. Those folders sometimes get corrupted by
Matlab, especially when multiple jobs start at exactly the same time. The Matlab, especially when multiple jobs start at exactly the same time. The
only workaround so far is to add a sleep 1s between qsub calls and hope only workaround so far is to add a sleep 1s between qsub/sbatch calls and hope
there is no collision. Also, it makes sense to regularly remove those there is no collision. Also, it makes sense to regularly remove those
directories. But make sure all your jobs have finished before removing directories. But make sure all your jobs have finished before removing
them with :file:`rm -rf .mcrCache*`. them with :file:`rm -rf .mcrCache*`.
...@@ -176,10 +174,10 @@ SPM ...@@ -176,10 +174,10 @@ SPM
--- ---
SPM already comes as a pre-compiled version and can, identical to the examples SPM already comes as a pre-compiled version and can, identical to the examples
above, be started with :program:`run_spm8.sh` or :program:`run_spm12.sh`. Usually users are above, be started with :program:`run_spm12.sh`.
exporting a number of batch files with the spm gui on their local machine, Usually users are exporting a number of batch files with the spm gui on their
change the paths to reflect the names on the tardis and then call local machine, change the paths to reflect the names on the tardis and then
:program:`run_spm12.sh` with the **run** parameter for each batch file. call :program:`run_spm12.sh` with the **run** parameter for each batch file.
Example: segmentation for a number of nifti images. The file batch.template contains the string :`%%IMAGE%%` as a placeholder so we can easily replace it with the current image path and create a number of new batches from a single template: Example: segmentation for a number of nifti images. The file batch.template contains the string :`%%IMAGE%%` as a placeholder so we can easily replace it with the current image path and create a number of new batches from a single template:
...@@ -192,7 +190,7 @@ Example: segmentation for a number of nifti images. The file batch.template cont ...@@ -192,7 +190,7 @@ Example: segmentation for a number of nifti images. The file batch.template cont
for image in tp2/Old/*.nii ; do for image in tp2/Old/*.nii ; do
fullpath=$PWD/$image fullpath=$PWD/$image
sed "s#%%IMAGE%%#$fullpath#" batch.template > batch_${i}.m sed "s#%%IMAGE%%#$fullpath#" batch.template > batch_${i}.m
echo "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m" | qsub -d. sbatch --wrap "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m"
i=$((i+1)) i=$((i+1))
done done
...@@ -203,7 +201,7 @@ toolboxes to SPM (e.g. cat12). ...@@ -203,7 +201,7 @@ toolboxes to SPM (e.g. cat12).
.. code-block:: matlab .. code-block:: matlab
[krause@master ~] matlab matlab
Warning: No display specified. You will not be able to display graphics on the screen. Warning: No display specified. You will not be able to display graphics on the screen.
< M A T L A B (R) > < M A T L A B (R) >
......
...@@ -20,14 +20,14 @@ Example: ...@@ -20,14 +20,14 @@ Example:
.. code-block:: bash .. code-block:: bash
# system libraries # system libraries
[krause@master ~] python --version python --version
Python 2.7.13 Python 2.7.13
[krause@master ~] python3 --version python3 --version
Python 3.5.3 Python 3.5.3
# load new version for python 3 # load new version for python 3
[krause@master ~] module load python/3.6 module load python/3.6
[krause@master ~] python3 --version python3 --version
Python 3.6.3 Python 3.6.3
...@@ -40,7 +40,7 @@ To see if they are available, simply try to import them: ...@@ -40,7 +40,7 @@ To see if they are available, simply try to import them:
.. code-block:: python .. code-block:: python
[krause@master ~] python3 python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54) Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux [GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information. Type "help", "copyright", "credits" or "license" for more information.
...@@ -55,9 +55,9 @@ Of course it comes separately for Python 2 and 3: ...@@ -55,9 +55,9 @@ Of course it comes separately for Python 2 and 3:
.. code-block:: bash .. code-block:: bash
[krause@master ~] pip --version pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7) pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
[krause@master ~] pip3 --version pip3 --version
pip 9.0.1 from /opt/software/python/3.6.3/lib/python3.6/site-packages (python 3.6) pip 9.0.1 from /opt/software/python/3.6.3/lib/python3.6/site-packages (python 3.6)
...@@ -68,13 +68,13 @@ Example (install numpy): ...@@ -68,13 +68,13 @@ Example (install numpy):
.. code-block:: bash .. code-block:: bash
[krause@master ~] pip3 install --user numpy pip3 install --user numpy
Collecting numpy Collecting numpy
Downloading numpy-1.11.2-cp35-cp35m-manylinux1_x86_64.whl (15.6MB) Downloading numpy-1.11.2-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
100% |████████████████████████████████| 15.6MB 58kB/s 100% |████████████████████████████████| 15.6MB 58kB/s
Installing collected packages: numpy Installing collected packages: numpy
Successfully installed numpy-1.11.2 Successfully installed numpy-1.11.2
[krause@master ~] python3 python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54) Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux [GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information. Type "help", "copyright", "credits" or "license" for more information.
...@@ -83,7 +83,7 @@ Example (install numpy): ...@@ -83,7 +83,7 @@ Example (install numpy):
('1.11.2', ['/home/mpib/krause/.local/lib/python3.5/site-packages/numpy']) ('1.11.2', ['/home/mpib/krause/.local/lib/python3.5/site-packages/numpy'])
Remember to always install on the tardis master node in case a package needs Remember to always install on the tardis login node in case a package needs
special development files which aren't installed on the computation hosts. special development files which aren't installed on the computation hosts.
.. note:: .. note::
...@@ -108,8 +108,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_, ...@@ -108,8 +108,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
.. code-block:: bash .. code-block:: bash
[krause@master ~] mkvirtualenv --python=/usr/bin/python3 project $ mkvirtualenv --python=/usr/bin/python3 project
(project) [krause@master ~] (project) $
**Activate and use the virtual environment** **Activate and use the virtual environment**
...@@ -118,16 +118,16 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_, ...@@ -118,16 +118,16 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
# without virtual environment # without virtual environment
[krause@master ~] python --version python --version
Python 2.7.13 Python 2.7.13
# with virtual environment # with virtual environment
[krause@master ~] workon project workon project
(project) [krause@master ~] python --version python --version
Python 3.5.3 Python 3.5.3
(project) [krause@master ~] which python3 which python3
/home/mpib/krause/.virtualenvs/project/bin/python /home/mpib/krause/.virtualenvs/project/bin/python
[krause@master ~] pip --version pip --version
pip 9.0.1 from /home/mpib/krause/.virtualenvs/project/lib/python3.5/site-packages (python 3.5) pip 9.0.1 from /home/mpib/krause/.virtualenvs/project/lib/python3.5/site-packages (python 3.5)
...@@ -135,8 +135,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_, ...@@ -135,8 +135,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
.. code-block:: bash .. code-block:: bash
(project) [krause@master ~] deactivate (project) $ deactivate
[krause@master ~] $
Note how :file:`virtualenv` is also managing your shell prompt, so you always know which Python environment you are currently running. Note how :file:`virtualenv` is also managing your shell prompt, so you always know which Python environment you are currently running.
......
...@@ -10,7 +10,7 @@ Packages ...@@ -10,7 +10,7 @@ Packages
-------- --------
Some packages are already installed globally, for every other dependency you Some packages are already installed globally, for every other dependency you
can just go ahead and install them on your own. All the nodes and the master can just go ahead and install them on your own. All the nodes and the login
node share your home directory so you only need to install the packages once node share your home directory so you only need to install the packages once
and they'll be available with your jobs. and they'll be available with your jobs.
...@@ -66,9 +66,9 @@ The external loop can then be constructed with a job array (see :ref:`job_array` ...@@ -66,9 +66,9 @@ The external loop can then be constructed with a job array (see :ref:`job_array`
.. code-block:: bash .. code-block:: bash
echo 'Rscript simulation_loop_body.R $PBS_O_ARRAYID' | qsub -N simulation -d. -t 1-1000 sbatch --array 1-1000 --wrap 'Rscript simulation_loop_body.R $SLURM_ARRAY_TASK_ID'
This will create 1000 jobs with ``$PBS_O_ARRAYID`` holding the current index This will create 1000 jobs with ``$SLURM_ARRAY_TASK_ID`` holding the current index
that will be passed to the R script shown above. that will be passed to the R script shown above.
Parallel Vector Operations Parallel Vector Operations
...@@ -78,17 +78,17 @@ You can also use the multi core nodes of the tardis to run a single multi core ...@@ -78,17 +78,17 @@ You can also use the multi core nodes of the tardis to run a single multi core
aware job with the :file:`parallel` package. You *have to make sure* your aware job with the :file:`parallel` package. You *have to make sure* your
requested number of cores matches the number you register with the script. requested number of cores matches the number you register with the script.
Assume you submit a job with :samp:`-l nodes=1:ppn=16` you can safely request 16 Assume you submit a job with :samp:`-c 16` you can safely request 16 cores in
cores in the script then. the script then.
Example :samp:`qsub -I -l nodes=1:ppn=16` Example :samp:`srun -c 16 --pty /bin/bash`
.. code-block:: r .. code-block:: r
library(parallel) library(parallel)
library(gmp) library(gmp)
cores <- Sys.getenv("PBS_NUM_PPN") cores <- Sys.getenv("SLURM_ARRAY_TASK_ID")
cores <- if (cores == '') 1 else as.numeric(cores) cores <- if (cores == '') 1 else as.numeric(cores)
data <- matrix(as.bigz(2**521)-1, 16) data <- matrix(as.bigz(2**521)-1, 16)
...@@ -112,7 +112,7 @@ Alternatively you can use `%dopar%` and `foreach`: ...@@ -112,7 +112,7 @@ Alternatively you can use `%dopar%` and `foreach`:
library(doMC) library(doMC)
library(gmp) library(gmp)
cores <- Sys.getenv("PBS_NUM_PPN") cores <- Sys.getenv("SLURM_ARRAY_TASK_ID")
cores <- if (cores == '') 1 else as.numeric(cores) cores <- if (cores == '') 1 else as.numeric(cores)
registerDoMC(cores) registerDoMC(cores)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment