Skip to content

GitLab

Commit 78eb9bc6 authored by Michael Krause's avatar Michael Krause 🎉
Browse files

software: change examples from qsub to sbatch

parent 3c2db3f6
Show whitespace changes
Inline Side-by-side
rm/slurm/jobs.rst
View file @ 78eb9bc6
......@@ -124,12 +124,14 @@ Usually users want to collect a number of jobs into batches and submit them with
This can be condensed down into a single line with the ``--wrap`` option to
sbatch:
sbatch. Here SLURM will create a job file on the fly, add the #!-line and
append the wrapped string to that file. This is syntax is being used a lot in
the examples in this document.
.. code-block:: bash
for sub in $(seq -w 1 15) ; do\
sbatch -c 2 -J main_$sub --wrap "python3 main.py $sub" ;\
for sub in $(seq -w 1 15) ; do
sbatch -c 2 -J main_$sub --wrap "python3 main.py $sub"
done
.. _slurm documentation: https://slurm.schedmd.com/quickstart.html
software/ants.rst
View file @ 78eb9bc6
......@@ -39,7 +39,7 @@ All other programs need to be wrapped into a job file to be run on the cluster n
.. code-block:: bash
qsub -N ants -l walltime=100:0:0,mem=16gb complex-ants-pipeline.sh
sbatch -J ants --time 100:0:0 --mem 16gb complex-ants-pipeline.sh
The file ``complex-ants-pipeline.sh`` contains your ANTS commands *preceeded*
......
software/ashs.rst
View file @ 78eb9bc6
......@@ -92,8 +92,8 @@ Old approach
.. code-block:: bash
[krause@master ~] module load ashs
[krause@master ~] $ASHS_ROOT/bin/ashs_main.sh \
module load ashs
$ASHS_ROOT/bin/ashs_main.sh \
-a /opt/software/ashs/data/1.0/ashs_atlas_mpib_20180208\
-g T1.nii -f T2.nii -w wdir -T -Q
......@@ -136,13 +136,14 @@ Since ASHS version 1.0 this is the preferred way to submit segmentations for a n
.. code-block:: bash
[krause@master ~] qsub ashs_job.pbs -l nodes=1:ppn=8,mem=10gb
sbatch ashs_job.jobs -c 8 --mem 10gb
with :file:`ashs_job.pbs` containing:
with :file:`ashs_job.sh` containing:
.. code-block:: bash
#!/bin/bash
module load ashs
export SUBJECT_ID=23
$ASHS_ROOT/bin/ashs_main.sh \
......@@ -155,14 +156,10 @@ Looping over a number of subject IDs would look something like this.
.. code-block:: bash
for id in 01 03 10 32 ; do
export SUBJECT_ID=$id
qsub ashs_job.pbs -l nodes=1:ppn=8,mem=10gb -v SUBJECT_ID
export SUBJECT_ID=$id # all exports are passed to SLURM
sbatch ashs_job.sh -c 8 --mem 10gb
done
In this case the option `-v` to qsub instructs qsub to inherit a named,
formerly exported, variable. This way the variable `SUBJECT_ID` becomes
accessible in the job context.
Known Issues
------------
......
software/freesurfer.rst
View file @ 78eb9bc6
......@@ -18,16 +18,14 @@ The simple subject loop shown in section :ref:`job_wrappers` and can be adapted
SUBJECTS_DIR="./subjects" # The freesurfer import location
for subject in $SUBJECTS_IDS ; do
echo "#PBS -m n" > tmp.pbs
echo "#PBS -o $HOME/logs/" >> tmp.pbs
echo "#PBS -j oe" >> tmp.pbs
echo "#PBS -d ." >> tmp.pbs
echo "export FREESURFER_HOME=/opt/freesurfer/5.3.0" >> tmp.pbs
echo "source \$FREESURFER_HOME/SetUpFreeSurfer.sh" >> tmp.pbs
echo "export SUBJECTS_DIR=$SUBJECTS_DIR" >> tmp.pbs
echo "recon-all -all -i ${subject}.nii -subjid ${subject}" >> tmp.pbs
qsub tmp.pbs
echo '#!/bin/bash' > tmp.sh
echo "#SBATCH --output $HOME/logs/slurm-%j.out" >> tmp.sh
echo "export FREESURFER_HOME=/opt/freesurfer/5.3.0" >> tmp.sh
echo "source \$FREESURFER_HOME/SetUpFreeSurfer.sh" >> tmp.sh
echo "export SUBJECTS_DIR=$SUBJECTS_DIR" >> tmp.sh
echo "recon-all -all -i ${subject}.nii -subjid ${subject}" >> tmp.sh
sbatch tmp.sh
done
......
software/fsl.rst
View file @ 78eb9bc6
......@@ -47,7 +47,7 @@ for all combinations of subjects and runs in parallel all in one go:
sed -i "s|%RUN%|$run|g" $featfile
# submit the feat command with the current design file
echo "module load fsl ; feat $featfile" | qsub -d.
sbatch --wrap "module load fsl ; feat $featfile"
done
done
......@@ -62,11 +62,11 @@ parallelize over a number of subjects. Example:
.. code-block:: bash
for icadir in *ica ; do
echo "module load fsl_fix" >> tmpjob
echo "source /etc/fsl/5.0/fsl.sh" >> tmpjob
echo "fix ${icadir} /opt/software/fix/1.06/training_files/Standard.RData 20" >> tmpjob
qsub -N fix_foobar -j oe -d. tmpjob
rm -f tmjob
echo '#!/bin/bash" > tmp.sh
echo "module load fsl_fix" >> tmp.sh
echo "source /etc/fsl/5.0/fsl.sh" >> tmp.sh
echo "fix ${icadir} /opt/software/fix/1.06/training_files/Standard.RData 20" >> tmp.sh
sbatch -J fix_foobar tmp.sh
done
The :file:`fsl_fix` module will also load a special R environment module which will
......
software/matlab.rst
View file @ 78eb9bc6
......@@ -5,20 +5,27 @@ Matlab is a bit of a problem child on the Tardis. While the `MATLAB Distributed
Computing Server`_ product aims to implement a compatibility layer to a number
of PBS based clusters it just doesn't work reliably for a number of reasons.
Because there is only a limited number of shared licenses available it's also
not feasible to run an arbitrary number of Matlab sessions in the form of jobs.
A workaround is to "compile" a script and create a standalone redistribution
environment, which does not require a license to run.
Because there are only a limited number of shared licenses available, it's
generally not feasible to run an arbitrary number of Matlab sessions in the
form of jobs. A workaround is to "compile" a script and create a standalone
redistribution environment, which does not require a license to run.
.. important::
You can check https://lipsupport.mpib-berlin.mpg.de/licstat for the current
license usage at the institute. Keep in mind that *one* license is necessary
for each user on each unique host. So 3 users, each submitting jobs to
any 4 hosts will result in 12 licenses.
Different Matlab versions are available via environment modules. You can list
them with :program:`module avail matlab` and activate a specific version with :program:`module
load matlab/<version>`.
them with :program:`module avail matlab` and activate a specific version with
:program:`module load matlab/<version>`.
Regular sessions
----------------
**If** there are free licenses available and you just need a quick way to spawn a
single Matlab session, there is nothing wrong with just running Matlab as is.
**If** there are free licenses available and you just need a quick way to spawn
a single Matlab session, there is nothing wrong with just running Matlab as is.
This might especially be useful if you simply need a node with lot's of memory
or if you want to test your code. In an interactive job you could simply enter
"matlab" and it will warn you about there not being a display available and
......@@ -27,12 +34,10 @@ start in command line mode.
.. code-block:: bash
[krause@master ~] qsub -i -l mem=100gb -q testing
qsub: waiting for job 5395814.master.tardis.mpib-berlin.mpg.de to start
qsub: job 5395814.master.tardis.mpib-berlin.mpg.de ready
srun --mem 100gb -p test --pty /bin/bash # run an interactive job
[krause@ood-9.tardis.mpib-berlin.mpg.de ~] module load matlab/R2012b
[krause@ood-9.tardis.mpib-berlin.mpg.de ~] matlab
module load matlab/R2012b # in that job, run matlab
matlab
< M A T L A B (R) >
......@@ -52,11 +57,8 @@ In a job context you would just run :program:`matlab -r main` with main.m contai
.. code-block:: bash
[krause@master ~] echo "module load matlab/R2014b; matlab -r main" | qsub -d.
.. important:
sbatch --wrap "module load matlab/R2014b; matlab -r main"
Always check `http://lipsupport.mpib-berlin.mpgde/licstat`_ to see if there is an available license.
Compiling
......@@ -96,8 +98,8 @@ Example:
.. code-block:: matlab
[krause@master ~] mcc -m project.m
[krause@master ~] ./run_project.sh /opt/matlab/interactive 42 5
mcc -m project.m
./run_project.sh /opt/matlab/interactive 42 5
------------------------------------------
Setting up environment variables
---
......@@ -107,8 +109,6 @@ Example:
'42' '42' '42' '42' '42'
[krause@master ~]
To include toolboxes in your script you have to add them during the compile
step so they get included in your package. Matlab built-in toolboxes such as
......@@ -124,7 +124,7 @@ package:
.. code-block:: bash
[krause@master ~] cat matlab/tools/project/myrnd.m
cat matlab/tools/project/myrnd.m
function X = myrnd(arg)
X = normrnd(0, 1, arg, arg);
......@@ -141,13 +141,13 @@ version as MCR upon script invocation with :program:`run_project.sh`.
.. code-block:: matlab
[krause@master ~] module load matlab/R2014b
[krause@master ~] cat project.m
module load matlab/R2014b
cat project.m
function project(arg1)
myrnd(str2num(arg1))
[krause@master ~] mcc -m project.m -a matlab/tools/project
mcc -m project.m -a matlab/tools/project
[...]
[krause@master ~] ./run_project.sh /opt/matlab/R2014b 3
./run_project.sh /opt/matlab/R2014b 3
------------------------------------------
Setting up environment variables
---
......@@ -159,15 +159,13 @@ version as MCR upon script invocation with :program:`run_project.sh`.
1.8339 0.3188 0.3426
-2.2588 -1.3077 3.5784
[krause@master ~]
.. note::
You only have to compile your project once and can then use it any number
of times. Matlab extracts your package to a shared hidden folder called
`.mcrCache<Version-Number>`. Those folders sometimes get corrupted by
Matlab, especially when multiple jobs start at exactly the same time. The
only workaround so far is to add a sleep 1s between qsub calls and hope
only workaround so far is to add a sleep 1s between qsub/sbatch calls and hope
there is no collision. Also, it makes sense to regularly remove those
directories. But make sure all your jobs have finished before removing
them with :file:`rm -rf .mcrCache*`.
......@@ -176,10 +174,10 @@ SPM
---
SPM already comes as a pre-compiled version and can, identical to the examples
above, be started with :program:`run_spm8.sh` or :program:`run_spm12.sh`. Usually users are
exporting a number of batch files with the spm gui on their local machine,
change the paths to reflect the names on the tardis and then call
:program:`run_spm12.sh` with the **run** parameter for each batch file.
above, be started with :program:`run_spm12.sh`.
Usually users are exporting a number of batch files with the spm gui on their
local machine, change the paths to reflect the names on the tardis and then
call :program:`run_spm12.sh` with the **run** parameter for each batch file.
Example: segmentation for a number of nifti images. The file batch.template contains the string :`%%IMAGE%%` as a placeholder so we can easily replace it with the current image path and create a number of new batches from a single template:
......@@ -192,7 +190,7 @@ Example: segmentation for a number of nifti images. The file batch.template cont
for image in tp2/Old/*.nii ; do
fullpath=$PWD/$image
sed "s#%%IMAGE%%#$fullpath#" batch.template > batch_${i}.m
echo "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m" | qsub -d.
sbatch --wrap "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m"
i=$((i+1))
done
......@@ -203,7 +201,7 @@ toolboxes to SPM (e.g. cat12).
.. code-block:: matlab
[krause@master ~] matlab
matlab
Warning: No display specified. You will not be able to display graphics on the screen.
< M A T L A B (R) >
......
software/python.rst
View file @ 78eb9bc6
......@@ -20,14 +20,14 @@ Example:
.. code-block:: bash
# system libraries
[krause@master ~] python --version
python --version
Python 2.7.13
[krause@master ~] python3 --version
python3 --version
Python 3.5.3
# load new version for python 3
[krause@master ~] module load python/3.6
[krause@master ~] python3 --version
module load python/3.6
python3 --version
Python 3.6.3
......@@ -40,7 +40,7 @@ To see if they are available, simply try to import them:
.. code-block:: python
[krause@master ~] python3
python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
......@@ -55,9 +55,9 @@ Of course it comes separately for Python 2 and 3:
.. code-block:: bash
[krause@master ~] pip --version
pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
[krause@master ~] pip3 --version
pip3 --version
pip 9.0.1 from /opt/software/python/3.6.3/lib/python3.6/site-packages (python 3.6)
......@@ -68,13 +68,13 @@ Example (install numpy):
.. code-block:: bash
[krause@master ~] pip3 install --user numpy
pip3 install --user numpy
Collecting numpy
Downloading numpy-1.11.2-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
100% |████████████████████████████████| 15.6MB 58kB/s
Installing collected packages: numpy
Successfully installed numpy-1.11.2
[krause@master ~] python3
python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
......@@ -83,7 +83,7 @@ Example (install numpy):
('1.11.2', ['/home/mpib/krause/.local/lib/python3.5/site-packages/numpy'])
Remember to always install on the tardis master node in case a package needs
Remember to always install on the tardis login node in case a package needs
special development files which aren't installed on the computation hosts.
.. note::
......@@ -108,8 +108,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
.. code-block:: bash
[krause@master ~] mkvirtualenv --python=/usr/bin/python3 project
(project) [krause@master ~]
$ mkvirtualenv --python=/usr/bin/python3 project
(project) $
**Activate and use the virtual environment**
......@@ -118,16 +118,16 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
# without virtual environment
[krause@master ~] python --version
python --version
Python 2.7.13
# with virtual environment
[krause@master ~] workon project
(project) [krause@master ~] python --version
workon project
python --version
Python 3.5.3
(project) [krause@master ~] which python3
which python3
/home/mpib/krause/.virtualenvs/project/bin/python
[krause@master ~] pip --version
pip --version
pip 9.0.1 from /home/mpib/krause/.virtualenvs/project/lib/python3.5/site-packages (python 3.5)
......@@ -135,8 +135,8 @@ For convenience reasons we also installed a module called `virtualenvwrapper`_,
.. code-block:: bash
(project) [krause@master ~] deactivate
[krause@master ~]
(project) $ deactivate
$
Note how :file:`virtualenv` is also managing your shell prompt, so you always know which Python environment you are currently running.
......
software/r.rst
View file @ 78eb9bc6
......@@ -10,7 +10,7 @@ Packages
--------
Some packages are already installed globally, for every other dependency you
can just go ahead and install them on your own. All the nodes and the master
can just go ahead and install them on your own. All the nodes and the login
node share your home directory so you only need to install the packages once
and they'll be available with your jobs.
......@@ -66,9 +66,9 @@ The external loop can then be constructed with a job array (see :ref:`job_array`
.. code-block:: bash
echo 'Rscript simulation_loop_body.R $PBS_O_ARRAYID' | qsub -N simulation -d. -t 1-1000
sbatch --array 1-1000 --wrap 'Rscript simulation_loop_body.R $SLURM_ARRAY_TASK_ID'
This will create 1000 jobs with ``$PBS_O_ARRAYID`` holding the current index
This will create 1000 jobs with ``$SLURM_ARRAY_TASK_ID`` holding the current index
that will be passed to the R script shown above.
Parallel Vector Operations
......@@ -78,17 +78,17 @@ You can also use the multi core nodes of the tardis to run a single multi core
aware job with the :file:`parallel` package. You *have to make sure* your
requested number of cores matches the number you register with the script.
Assume you submit a job with :samp:`-l nodes=1:ppn=16` you can safely request 16
cores in the script then.
Assume you submit a job with :samp:`-c 16` you can safely request 16 cores in
the script then.
Example :samp:`qsub -I -l nodes=1:ppn=16`
Example :samp:`srun -c 16 --pty /bin/bash`
.. code-block:: r
library(parallel)
library(gmp)
cores <- Sys.getenv("PBS_NUM_PPN")
cores <- Sys.getenv("SLURM_ARRAY_TASK_ID")
cores <- if (cores == '') 1 else as.numeric(cores)
data <- matrix(as.bigz(2**521)-1, 16)
......@@ -112,7 +112,7 @@ Alternatively you can use `%dopar%` and `foreach`:
library(doMC)
library(gmp)
cores <- Sys.getenv("PBS_NUM_PPN")
cores <- Sys.getenv("SLURM_ARRAY_TASK_ID")
cores <- if (cores == '') 1 else as.numeric(cores)
registerDoMC(cores)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment