Commit 489ea036 authored by Michael Krause's avatar Michael Krause 🎉
Browse files

Merge branch 'master' of gitlab.mpib-berlin.mpg.de:lip-support/tardis-doc

parents e749f014 5f5e24df
......@@ -56,9 +56,9 @@ author = 'Michael'
# built documents.
#
# The short X.Y version.
version = '3.2'
version = '3.4.2'
# The full version, including alpha/beta/rc tags.
release = '3.2'
release = '3.4.2'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......@@ -137,7 +137,7 @@ html_logo = 'img/tardis_small.png'
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
html_favicon = 'img/tardis_small.png'
html_favicon = 'img/tardis_small.ico'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
......
......@@ -4,6 +4,15 @@ Welcome to Tardis's documentation!
Changelog
=========
3.4.2 (11.11. 2016)
+ added python section
3.4.1 (18.10. 2016)
+ added fieldtrip compilation notes
3.4 (19.09. 2016)
+ added matlab section
3.3 (15.08. 2016)
+ added minimal example for fsl fix
+ fixed a typo with the `qdel` example
......@@ -42,12 +51,13 @@ List of Contents
:maxdepth: 1
:caption: Software
software/freesurfer
software/fsl
software/ashs
software/ants
software/r
software/freesurfer
software/fsl
software/matlab
software/python
software/r
.. toctree::
:maxdepth: 1
......
......@@ -29,6 +29,14 @@ the need for a jobfile:
echo "sleep 1m" | qsub
Get a quick shell on the testing queue, which should often be empty, because
only short jobs are allowed.
.. code-block:: bash
qsub -I -q testing
Examples:
.. code-block:: bash
......
......@@ -38,6 +38,7 @@ Each queue has different parameter sets and resource targets. On the tardis ther
+ ``default``
+ ``longwall`` for jobs that need more than 36 hours
+ ``testing`` for very short jobs only
+ ``gpu`` for jobs that need a GPU
Resources
......
......@@ -11,7 +11,7 @@ A workaround is to "compile" a script and create a standalone redistribution
environment, which does not require a license to run.
Different Matlab versions are available via environment modules. You can list
them with `module avail matlab` and activate a specific version with `module
them with :program:`module avail matlab` and activate a specific version with :program:`module
load matlab/<version>`.
Regular sessions
......@@ -44,15 +44,15 @@ start in command line mode.
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> _
>> _
In a job context you would just run `matlab -r main` with main.m containing your script:
In a job context you would just run :program:`matlab -r main` with main.m containing your script:
.. code-block:: bash
[krause@master ~] echo "module load matlab/R2014b; matlab -m main" | qsub -d.
[krause@master ~] echo "module load matlab/R2014b; matlab -r main" | qsub -d.
.. important:
......@@ -79,7 +79,7 @@ over. A little like this maybe:
%
% subject_id: a string encoding the subject id
% sigma: a string encoding values for sigma
sigma = str2num(sigma);
repmat(cellstr(subject_id), 1, sigma)
......@@ -87,10 +87,10 @@ over. A little like this maybe:
Running :program:`mcc -m project.m` would then "compile" (or rather encrypt and
package) your function and output a system dependent binary named
:file:`project` a wrapper script file:`run_project.sh`. To run your script you
now have to combine the shell wrapper script, the location of a Matlab Compile
Runtime or the local installation path of the matlab instance that was used by
mcc and a sufficient number of arguments for the function project().
:file:`project` and a wrapper script :file:`run_project.sh`. To run it you
now have to combine the wrapper script, the location of a Matlab Compile
Runtime or the local installation path of the matlab instance, that was used by
mcc, and a sufficient number of arguments for the function project().
Example:
......@@ -104,13 +104,13 @@ Example:
LD_LIBRARY_PATH is .:/opt/matlab/interactive/runtime/glnxa64:/opt/matlab/interactive/bin/glnxa64:/opt/matlab/interactive/sys/os/glnxa64:/opt/matlab/interactive/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/opt/matlab/interactive/sys/java/jre/glnxa64/jre/lib/amd64/server:/opt/matlab/interactive/sys/java/jre/glnxa64/jre/lib/amd64/client:/opt/matlab/interactive/sys/java/jre/glnxa64/jre/lib/amd64
Warning: No display specified. You will not be able to display graphics on the screen.
ans =
'42' '42' '42' '42' '42'
[krause@master ~]
To include toolboxes in you script you have to add them during the compile step
To include toolboxes in your script you have to add them during the compile step
so they get included in your package. Matlab built-in toolboxes like signal
processing or statistics are detected automatically by scanning the functions
used in your script and don't need to be added explicitly.
......@@ -119,18 +119,21 @@ Example: Suppose you collect your project library in a toolbox called project,
which in turn uses the function `normrnd()` from the statistics package:
.. code-block:: bash
[krause@master ~] cat matlab/tools/project/myrnd.m
function X = myrnd(arg)
X = normrnd(0, 1, arg, arg);
You can then either use the "-a" or the "-I" switch of mcc to add your own toolbox.
+ `-a`: will add the functions or directories listed directly to the compiled package/archive
+ `-I`: will add the location to the mcc search path so it get's included implicitly
+ **-a** will add the functions or directories listed directly to the compiled package/archive
+ **-I** (uppercase i) will add the location to the mcc search path so it get's included implicitly
Both options should work fine.
Both options should work fine. The example below uses mcc from matlab R2014b,
but you can use any version. The important part is to use the same Matlab
version as MCR upon script invocation with :program:`run_project.sh`.
.. code-block:: matlab
......@@ -145,48 +148,48 @@ Both options should work fine.
Setting up environment variables
---
LD_LIBRARY_PATH is .:/opt/matlab/R2014b/runtime/glnxa64:/opt/matlab/R2014b/bin/glnxa64:/opt/matlab/R2014b/sys/os/glnxa64:/opt/matlab/R2014b/sys/opengl/lib/glnxa64
ans =
0.5377 0.8622 -0.4336
1.8339 0.3188 0.3426
-2.2588 -1.3077 3.5784
[krause@master ~]
[krause@master ~]
.. note::
You only have to compile your project once and can then use it any number
of times. Matlab extracts your package a shared hidden folder called
of times. Matlab extracts your package to a shared hidden folder called
`.mcrCache<Version-Number>`. Those folders sometimes get corrupted by
Matlab, especially when multiple jobs start at exactly the same time. The
only workaround so far is to add a sleep 1s between qsub calls and hope
there is no collision. Also, it makes sense to regularly remove those
directories. But make sure all your jobs have finished before removing
them with `rm -rf .mcrCache*`.
them with :file:`rm -rf .mcrCache*`.
SPM
---
SPM already comes as a pre-compiled version and can, identical to the examples
above, be started with `run_spm8.sh` or `run_spm12.sh`. Usually users are
above, be started with :program:`run_spm8.sh` or :program:`run_spm12.sh`. Usually users are
exporting a number of batch files with the spm gui on their local machine,
change the paths to reflect the names on the tardis and then call
:program:`run_spm12.sh` with the **run** parameter for each batch file.
Example: segmentation for a number of nifti images. The file batch.template contains the string `%%IMAGE%%` as a placeholder so we can easily replace it with the current image path and create a number of new batches from a single template:
Example: segmentation for a number of nifti images. The file batch.template contains the string :`%%IMAGE%%` as a placeholder so we can easily replace it with the current image path and create a number of new batches from a single template:
.. code-block:: bash
#!/bin/bash
i=0
for image in tp2/Old/*.nii ; do
fullpath=$PWD/$image
sed "s#%%IMAGE%%#$fullpath#" batch.template > batch_${i}.m
echo "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m" | qsub -d.
i=$((i+1))
sed "s#%%IMAGE%%#$fullpath#" batch.template > batch_${i}.m
echo "run_spm12.sh /opt/matlab/interactive run $PWD/batch_${i}.m" | qsub -d.
i=$((i+1))
done
......@@ -195,30 +198,63 @@ for instance if you need a specific version or if you want to add external
toolboxes to SPM (e.g. cat12).
.. code-block:: matlab
[krause@master ~] matlab
Warning: No display specified. You will not be able to display graphics on the screen.
< M A T L A B (R) >
Copyright 1984-2012 The MathWorks, Inc.
R2012a (7.14.0.739) 64-bit (glnxa64)
February 9, 2012
< M A T L A B (R) >
Copyright 1984-2012 The MathWorks, Inc.
R2012a (7.14.0.739) 64-bit (glnxa64)
February 9, 2012
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> addpath(genpath('/home/mpib/krause/matlab/tools/spm12'))
>> spm_make_standalone()
[... lot's of output and warnings ...]
Processing /opt/matlab/R2012a/toolbox/matlab/mcc.enc
[... lot's of output and warnings ...]
Generating file "/home/mpib/krause/matlab/tools/spm_exec/readme.txt".Gen
Generating file "/home/mpib/krause/matlab/tools/spm12/../spm_exec/run_spm12.sh".
>>
This should create a folder :file:`spm_exec` below the spm toolbox location
containing the fresh :program:`spm12` and :program:`run_spm12.sh` which you can
then use in your jobs just like above.
Fieldtrip
---------
To properly add the fieldtrip toolbox we have to jump through some more hoops.
For now the only reliable and flexible way is to run :program:`mcc()` from
within a Matlab session and make sure to run :program:`ft_defaults()` first.
Also, some of the provided mex files won't work out of the box, so we have to
recompile them using :program:`ft_compile_mex()`. This however stumbles over
some external C file called :file:`CalcMD5.c` using non-standard comments.
The following Matlab Script has been successfully used to create a compiled
script from a :file:`main.m` file, which relies on internal fieldtrip
functions.
.. code-block:: matlab
% setup path
basepath='/home/mpib/krause/matlab/tools/ConMemEEGTools/'
addpath([basepath, '/fieldtrip-20150930'])
ft_defaults()
% re-compile mex functions (this has to be done only once per fieldtrip version)
% "fix" the CalcMD5.c file
system(['sed -i ''s#//.*##g'' ', basepath, '/fieldtrip-20150930/external/fileexchange/CalcMD5.c'])
% and compile
ft_compile_mex(true)
% build the runtime environment
mcc('-m', 'main.m')
.. _`MATLAB Distributed Computing Server`: http://de.mathworks.com/help/mdce/index.html
Python
======
Python is probably the most versatile programming ecosystem available on the Tardis.
It's an interpreted language that is easy to read, quick to prototype in and comes with a huge number of useful libraries.
Versions
--------
Users need to be aware, that there are still two major versions around that are not 100% compatible to each other.
Although Python 3.0 was released back in 2008, its predecessor Python 2.7 is still around and not all libraries and packages have migrated.
Since the system version on Debian/stable is still 2.7, the :file:`python` program always points to Python 2 (2.7.3).
The :file:`python3` program designates the counterpart (3.2).
The system and core python libraries are, though stable, still quite old (even in the Python 3 branch).
You can use environment modules to switch to a newer version.
Example:
.. code-block:: bash
# system libraries
[krause@master ~] python --version
Python 2.7.3
[krause@master ~] python3 --version
Python 3.2.3
# load new version for python 3
[krause@master ~] module load python/3.5
[krause@master ~] python3 --version
Python 3.5.1
# load new version for python 2
[krause@master ~] module unload python
[krause@master ~] module load python/2.7
[krause@master ~] python --version
Python 2.7.12
Packages
--------
Some important packages are already installed system-wide.
To see if they are available, simply try to import them:
.. code-block:: python
[krause@master ~] python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
>>> lxml.__path__
['/opt/software/python/3.5/lib/python3.5/site-packages/lxml']
If you happen to need a more recent one or something that hasn't been installed already, the easiest way is to use the default Python package manager :file:`pip`.
Of course it comes separately for Python 2 and 3:
.. code-block:: bash
[krause@master ~] pip --version
pip 1.1 from /usr/lib/python2.7/dist-packages (python 2.7)
[krause@master ~] pip3 --version
pip 9.0.1 from /home/mpib/krause/.local/lib/python3.5/site-packages (python 3.5)
To install a package in your home directory, you can simply run :program:`pip3 install --user <packagename>`.
Example (install numpy):
.. code-block:: bash
[krause@master ~] pip3 install --user numpy
Collecting numpy
Downloading numpy-1.11.2-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
100% |████████████████████████████████| 15.6MB 58kB/s
Installing collected packages: numpy
Successfully installed numpy-1.11.2
[krause@master ~] python3
Python 3.5.1 (default, Mar 14 2016, 16:32:54)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__, np.__path__
('1.11.2', ['/home/mpib/krause/.local/lib/python3.5/site-packages/numpy'])
Remember to always install on the tardis master node in case a package needs
special development files which aren't installed on the computation hosts.
.. note::
The Python Package Index (`PyPI`_), pip's source, contains a lot of user provided, custom and sometimes old and unstable software. Make sure that what you're installing is actually the package that you want. Usually the project's installations notes tell you what the package is called in pypi.
Virtual Environments
--------------------
Sometimes, especially for reproducibility reasons, it may be useful to freeze
the python package versions to a specific release. You could create a
:file:`requirements.txt` with version numbers and then always run `pip3 install
--user -r requirements.txt`, when you switch a project, but it's way more
convenient to use different python environments for each project.
Starting with Python 3.4 the program `pyvenv`_ will help you manage different environments.
Once created it will copy the current system version's python and pip to a new directory with the environment's name.
Every package installed or upgraded will be contained to that specific directory and you can switch between them very easily.
**Create a new virtual environment**
.. code-block:: bash
[krause@master ~] module load python/3.5 # global tardis environment
[krause@master ~] pyvenv .venvs/project # python environment
**Activate and use the virtual environment**
.. code-block:: bash
[krause@master ~] source .venvs/project/bin/activate
(project) [krause@master ~] which python3
/home/mpib/krause/.venvs/project/bin/python3
(project) [krause@master ~] which pip3
/home/mpib/krause/.venvs/project/bin/pip3
(project) [krause@master ~] pip3 --version
ppip 7.1.2 from /home/mpib/krause/.venvs/project/lib/python3.5/site-packages (python 3.5)
(project) [krause@master ~] python3 --version
Python 3.5.1
**Deactivate the virtual environment**
.. code-block:: bash
(project) [krause@master ~] deactivate
[krause@master ~]
Note how :file:`pyvenv` is also managing your shell prompt, so you always know which Python environment you are currently running.
.. _PyPI: https://pypi.python.org/pypi
.. _pyvenv: https://packaging.python.org/installing/#creating-virtual-environments
SPM
===
1. Matlab Installationen
2. SPM8 bauen
3. SPM12b bauen
### 1. Matlab Installationen
### 2. SPM8 bauen
### 3. SPM12b bauen
+ spm package besorgen und extrahieren
+ mex dateien löschen und danach neu bauen:
rm -rf *.mex*
cd src
make && make install
+ Matlab fuers compilen starten, ggf mit anderer Lizenz:
MLM_LICENSE_FILE="/opt/matlab/compiler.lic" matlab
cd ../tools/spm12b/
addpath(genpath('.'))
spm_make_standalone
cd ..
spm_make_standalone % [sic]
% warten..
+ in der Shell dann das .ctf Archiv (eigentlich nur ein zip) einmalig entpacken:
./run_spm12b.sh /opt/matlab/interactive dir
+ als root dann den erzeugten Ordner spm_exec deployen:
rm -rf /import/client/matlab/interactive/toolbox/spm12b # altes loeschen
cp -r spm_exec /import/client/matlab/interactive/toolbox/spm12b
chown -R root:root /import/client/matlab/interactive/toolbox/spm12b
chmod -R og+rX /import/client/matlab/interactive/toolbox/spm12b
# kaputte Pfade fixen
cd /import/client/matlab/interactive/toolbox/spm12b/spm12b_mcr/spm12b/
ln -s ../EEGtemplates EEGtemplates
ln -s ../toolbox toolbox
+ warten bis es deployed wird oder selber nach /opt syncen
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment