BIDSApps
========

With the rise of the `BIDS`_ data layout standard software engineers in the
neuroimaging community started to build a commonly used interface for
applications called `BIDSApps`_. This approach has the potential to greatly
simplify and automate common analysis patterns. In general, a BIDS app can be
used either on a group or on a subject level. Since the data is already in
a well defined format, all you have to pass is the path to a BIDS folder, some
output location and an optional filter (for a specific subject for example):

.. code-block:: bash

   # on a participant level
   some_bids_app /input /output participant --participant-label 01

   # summary analysis on the whole group in the bids dataset
   some_bids_app /input /output group


This general pattern (or interface) allows for a consistent analysis
development and, in the case of participant level scripts, a very simple way of
looping over all participants.
Notable examples that use this pattern are `fMRIPrep`_, `MRIQC`_, and `QSIprep`_.

Because the interface is so well defined and it allows for automatic or at least
semi-automatic runs, BIDSApps are usually distributed in the form of
containers to further improve reproducibility and robustness.

So in the simplest case the pattern would extend to:

.. code-block:: bash

   singularity run bids/someapp:version /input /output participant --participant-label 01


To make this even more robust a number of additional parameters to singularity
are recommended. The next sections list the reference singularity calls for
often used BIDSApps on Tardis.


MRIQC
-----

To run a reproducible, single participant level instance of MRIQC, all you need is a bids valid
directory and the following block:

.. code-block:: bash

   export SINGULARITYENV_TEMPLATEFLOW_HOME=/cache
   singularity run --contain --cleanenv \
      -B $BIDSDIR:/input:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $CACHEDIR:/cache \
      /data/container/mriqc/mriqc-0.16.1.sif /input /output participant \
        --participant-label 01 -w /work [other mriqc specific options]


TemplateFlow
~~~~~~~~~~~~

Note that the above example needs to override the default Templateflow cache
directory. This is a common side effect of locking up the container environment
to only explicitly added directories using ``-B / --bind``.  Some neuroimaging
projects are using the `Templateflow`_ library, which downloads, and caches
some commonly used template images. Those files usually reside in your user
home in a hidden directory ``.cache``. You can either add this cache using ``-B
$HOME/.cache:$HOME/.cache`` or use a seperate folder for the cache as configured
above. When you don't want to use a shared cache directory (arguably a bit
paranoid), the library expects an evironment variable called
``TEMPLATEFLOW_HOME`` to be set and we pass that into the container by
prefixing it with ``SINGULARITYENV_`` and exporting it.

It may be a good idea, to pre-populate a new cache directory, by running
a single participant, before starting multiple participants in parallel to
avoid conflicts from concurrent downloads to that folder.


Large Datasets
~~~~~~~~~~~~~~

Sometimes, with *very* large datasets (1000+ sessions) running MRIQC can be
extremely slow, because each instance, upon start, is indexing the whole
dataset before it can access files in it. The following pattern exploits the
convenient bind-mount option of singularity to create ad-hoc, minimal
sub-datasets that mriqc can run on. It has been succesfully used on the
UKBioBank datset on Tardis with a drastic decrease of startup time.

.. code-block:: bash

   # create smaller, fake datasets using individual subjects only
   CHUNK="sub-990 sub-991 sub-992"
   BIND_LIST=""
   for subject in $CHUNK ; do
       BIND_LIST="${BIND_LIST} -B $BIDSDIR/$subject:/input/$subject:ro"
   done

   # only pass each sub-X directory and dataset_description.json into the container
   singularity run --contain --cleanenv \
      $BIND_LIST \
      -B $BIDSDIR/dataset_description.json:/input/dataset_description.json:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $HOME/.cache:$HOME/.cache \
      /data/container/mriqc/mriqc-0.16.1.sif /input /output participant \
        --participant-label 01 -w /work [other mriqc specific options]



fMRIPrep
--------

The usage of fmriprep containers is very similar to that of MRIQC as discussed
above. Additionally to input, output, and cache folders you also need to pass
a freesurfer "license" file (you can map the Tardis files) and a recommended
option to disable software tracking:


.. code-block:: bash

   singularity run --contain --cleanenv \
     -B /opt/freesurfer/7.2.0/.license:/.license:ro \
     -B $PWD/dsone:/input \
     -B $PWD/outtest:/output \
     -B $HOME/.cache:$HOME/.cache \
     -B $PWD/work:/work \
     /data/container/fmriprep/fmriprep-20.2.6.sif \
       /input /output participant  --participant-label 01 \
       -w /work --fs-license-file /.license --notrack \
       --omp-nthreads X --nthreads X

Here, `X` denotes the number of CPU cores you would like to utilize. This
usually corresponds to the amount of cores you requested for a Tardis job. The
default here is all (seemingly) available cores, which in the default job case
will result in up to 40 threads being run on a single core. This is quite
inefficient, and it's therefore a good idea to always explicitly set the amount
of cores available to evenly distribute tasks across real cores.

Based on some simple benchmarks below (used on `DS000001`_ and ``sub-01``) we
recommend to start with 6 cores and 5GB of memory. Results may vary depending
on image size however.

.. list-table:: fMRIPrep benchmarks
   :header-rows: 1

   * - Number of cores (HT)
     - Memory Usage (maxRSS)
     - Elapsed Time
   * - 2
     - 3143 MB
     - 09:52:22
   * - 4
     - 3565 MB
     - 08:14:03
   * - 6
     - 4087 MB
     - 07:54:47
   * - 8
     -/ 4800 MB
     - 07:59:50
   * - 10
     - 4947 MB
     - 07:22:35
   * - 12
     - 5519 MB
     - 06:29:21

QSIprep
-------

`QSIprep`_ is a comprehensive preprocessing pipeline for all kinds of diffusion
weighted MRI data. There is a large number of tunables and options to qsiprep.
The basic command structure follows the BIDSApps scheme and can be extended
from this recommended base:

.. code-block:: bash

   singularity run --contain --cleanenv --nv \
      -B $BIDSDIR:/input:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $EDDY_PARAMS:/eddy_params.json \
      -B /data/software/archive/freesurfer/7.1.1/.license:/fs-license.txt \
      /data/container/qsiprep/qsiprep-0.14.3.sif /input /output participant \
         --eddy-config /eddy_params.json --fs-license-file /fs-license.txt \
         --skip_bids_validation --notrack -w /work \
         --output-resolution 1.5 --participant-label ...

Eddy
~~~~

One part of the QSIprep pipeline is FSL's eddy correction tool. To use this
tool most efficienctly it's highly recommended to run it on a gpu node and
enable the cuda version of eddy. The default configuration parameters are
listed on `Github
<https://github.com/PennLINC/qsiprep/blob/master/qsiprep/data/eddy_params.json>`_.
To enable `eddy_cuda`, you need to

#. Create a copy of ``eddy_params.json`` and change the value of ``use_cuda`` to ``true``
#. Request a gpu job by adding ``--gres gpu:pascal:1 -p gpu`` to your batch command
#. Run qsiprep with ``--eddy-config path/to/eddy_params.json``
#. Add the nvidia flag ``--nv`` to your singularity call


.. note::

    With QSIprep up to at least version **0.14.3** the bundled CUDA runtime
    libraries 9.1 are not entirely functional with all GPUs, so you need to
    explicitly request a pascal architecture card as shown in the list.



.. _`BIDS`: https://bids.neuroimaging.io
.. _`BIDSApps`: https://bids-apps.neuroimaging.io/apps
.. _`fMRIPrep`: https://fmriprep.org/en/stable/usage.html#command-line-arguments
.. _`MRIQC`: https://mriqc.readthedocs.io/en/stable/running.html#command-line-interface
.. _`QSIprep`: https://qsiprep.readthedocs.io/en/latest
.. _`Templateflow`: https://www.templateflow.org
.. _`DS000001`: https://openneuro.org/datasets/ds000001
