BIDSApps
========

With the rise of the `BIDS`_ data layout standard software engineers in the
neuroimaging community started to build a commonly used interface for
applications called `BIDSApps`_. This approach has the potential to greatly
simplify and automate common analysis patterns. In general, a BIDS app can be
used either on a group or on a subject level. Since the data is already in
a well defined format, all you have to pass is the path to a BIDS folder, some
output location and an optional filter (for a specific subject for example):

.. code-block:: bash

   # on a participant level
   some_bids_app /input /output participant --participant-label 01

   # summary analysis on the whole group in the bids dataset
   some_bids_app /input /output group


This general pattern (or interface) allows for a consistent analysis
development and, in the case of participant level scripts, a very simple way of
looping over all participants.
Notable examples that use this pattern are `fMRIPrep`_, `MRIQC`_, and `QSIprep`_.

Because the interface is so well defined and it allows for automatic or at least
semi-automatic runs, BIDSApps are usually distributed in the form of
containers to further improve reproducibility and robustness.

So in the simplest case the pattern would extend to:

.. code-block:: bash

   singularity run bids/someapp:version /input /output participant --participant-label 01


To make this even more robust a number of additional parameters to singularity
are recommended. The next sections list the reference singularity calls for
often used BIDSApps on Tardis.


MRIQC
-----

To run a reproducible, single participant level instance of MRIQC, all you need is a bids valid
directory and the following block:

.. code-block:: bash

   # set paths and files
   export BIDSDIR=path/to/dataset_dir
   export OUTPUTDIR=path/to/output_dir
   export WORKDIR=path/to/work_dir
   export CACHEDIR=path/to/cache_dir   # could be $HOME/.cache

   export SINGULARITYENV_TEMPLATEFLOW_HOME=/cache
   singularity run --contain --cleanenv \
      -B $BIDSDIR:/input:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $CACHEDIR:/cache \
      /data/container/mriqc/mriqc-0.16.1.sif /input /output participant \
        --participant-label 01 -w /work [other mriqc specific options]


TemplateFlow
~~~~~~~~~~~~

Note that the above example needs to override the default Templateflow cache
directory. This is a common side effect of locking up the container environment
to only explicitly added directories using ``-B / --bind``.  Some neuroimaging
projects are using the `Templateflow`_ library, which downloads, and caches
some commonly used template images. Those files usually reside in your user
home in a hidden directory ``.cache``. You can either add this cache using ``-B
$HOME/.cache:$HOME/.cache`` or use a seperate folder for the cache as configured
above. When you don't want to use a shared cache directory (arguably a bit
paranoid), the library expects an evironment variable called
``TEMPLATEFLOW_HOME`` to be set and we pass that into the container by
prefixing it with ``SINGULARITYENV_`` and exporting it.

It may be a good idea, to pre-populate a new cache directory, by running
a single participant, before starting multiple participants in parallel to
avoid conflicts from concurrent downloads to that folder.


Large Datasets
~~~~~~~~~~~~~~

Sometimes, with *very* large datasets (1000+ sessions) running MRIQC can be
extremely slow, because each instance, upon start, is indexing the whole
dataset before it can access files in it. The following pattern exploits the
convenient bind-mount option of singularity to create ad-hoc, minimal
sub-datasets that mriqc can run on. It has been succesfully used on the
UKBioBank datset on Tardis with a drastic decrease of startup time.

.. code-block:: bash

   # create smaller, fake datasets using individual subjects only
   CHUNK="sub-990 sub-991 sub-992"
   BIND_LIST=""
   for subject in $CHUNK ; do
       BIND_LIST="${BIND_LIST} -B $BIDSDIR/$subject:/input/$subject:ro"
   done

   # only pass each sub-X directory and dataset_description.json into the container
   singularity run --contain --cleanenv \
      $BIND_LIST \
      -B $BIDSDIR/dataset_description.json:/input/dataset_description.json:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $HOME/.cache:$HOME/.cache \
      /data/container/mriqc/mriqc-0.16.1.sif /input /output participant \
        --participant-label 01 -w /work [other mriqc specific options]



fMRIPrep
--------

The usage of fmriprep containers is very similar to that of MRIQC as discussed
above. Additionally to input, output, and cache folders you also need to pass
a freesurfer "license" file (you can map the Tardis files) and a recommended
option to disable software tracking:


.. code-block:: bash

   # set paths and files
   export BIDSDIR=path/to/dataset_dir
   export OUTPUTDIR=path/to/output_dir
   export WORKDIR=path/to/work_dir
   export CACHEDIR=path/to/cache_dir   # could be $HOME/.cache

   export SINGULARITYENV_TEMPLATEFLOW_HOME=/cache
   singularity run --contain --cleanenv \
     -B /opt/freesurfer/7.2.0/.license:/.license:ro \
     -B $BIDSDIR:/input \
     -B $OUTPUTDIR:/output \
     -B $WORKDIR:/work \
     -B $CACHEDIR:/cache \
     /data/container/fmriprep/fmriprep-20.2.6.sif \
       /input /output participant  --participant-label 01 \
       -w /work --fs-license-file /.license --notrack \
       --omp-nthreads X --nthreads X

Here, `X` denotes the number of CPU cores you would like to utilize. This
usually corresponds to the amount of cores you requested for a Tardis job. The
default here is all (seemingly) available cores, which in the default job case
will result in up to 40 threads being run on a single core. This is quite
inefficient, and it's therefore a good idea to always explicitly set the amount
of cores available to evenly distribute tasks across real cores.

Based on some simple benchmarks below (used on `DS000001`_ and ``sub-01``) we
recommend to start with 6 cores and 5GB of memory (lower to 4 cores if you have
a more subjects than free slots on Tardis). Results may vary depending on image
size and other parameters of course.

.. list-table:: fMRIPrep benchmarks
   :header-rows: 1

   * - Number of cores (HT)
     - Memory Usage (maxRSS)
     - Elapsed Time (H:M:S)
   * - 2
     - 3143 MB
     - 09:52:22
   * - 4
     - 3565 MB
     - 08:14:03
   * - 6
     - 4087 MB
     - 07:54:47
   * - 8
     - 4800 MB
     - 07:59:50
   * - 10
     - 4947 MB
     - 07:22:35
   * - 12
     - 5519 MB
     - 06:29:21

QSIprep
-------

`QSIprep`_ is a comprehensive preprocessing pipeline for all kinds of diffusion
weighted MRI data. There is a large number of tunables and options to qsiprep.
The basic command structure follows the BIDSApps scheme and can be extended
from this recommended base:

.. code-block:: bash

   # set paths and files
   export BIDSDIR=path/to/dataset_dir
   export OUTPUTDIR=path/to/output_dir
   export WORKDIR=path/to/work_dir
   export EDDY_PARAMS=path/to/eddy_params.json

   singularity run --contain --cleanenv --nv \
      -B $BIDSDIR:/input:ro \
      -B $OUTPUTDIR:/output \
      -B $WORKDIR:/work \
      -B $EDDY_PARAMS:/eddy_params.json \
      -B /opt/freesurfer/7.2.0/.license:/.license:ro \
      /data/container/qsiprep/qsiprep-0.14.3.sif /input /output participant \
         --eddy-config /eddy_params.json --fs-license-file /.license \
         --skip_bids_validation --notrack -w /work \
         --output-resolution 1.5 --participant-label ...

Eddy
~~~~

One part of the QSIprep pipeline is FSL's eddy correction tool. To use this
tool most efficienctly it's recommended to run it on a gpu node and enable the
cuda version of eddy, especially when the slice-to-volume correction method is
being used.



To enable ``eddy_cuda``, you need to

#. Create a copy of ``eddy_params.json`` and change the value of ``use_cuda`` to ``true``
#. Request a gpu job by adding ``--gres gpu:pascal:1 -p gpu`` to your batch command
#. Run qsiprep with ``--eddy-config path/to/eddy_params.json``
#. Add the nvidia flag ``--nv`` to your singularity call

.. note::

    With QSIprep up to at least version **0.14.3** the bundled CUDA runtime
    libraries 9.1 are not entirely functional with all GPUs, so you need to
    explicitly request a pascal architecture card as shown in the list, use the
    unofficial CUDA10 containers or wait for upstream to update their
    containers.

In general, to configure eddy, you need to pass custom ``eddy_params.json`` to
qsiprep. The default configuration parameters are listed on `Github
<https://github.com/PennLINC/qsiprep/blob/master/qsiprep/data/eddy_params.json>`_.
One example that has been used for **high movement** data **from children** is
the following. Please consult the `eddy user-guide`_ for an explanation of all
options and **always tune those options to your current study**.


.. code-block:: json
   :caption: example ``eddy_params.json`` for high movement, **not for general use**

   {
      "flm": "quadratic",
      "slm": "linear",
      "fep": false,
      "interp": "spline",
      "nvoxhp": 1000,
      "fudge_factor": 10,
      "dont_sep_offs_move": false,
      "dont_peas": false,
      "niter": 8,
      "method": "jac",
      "repol": true,
      "is_shelled": false,
      "use_cuda": true,
      "cnr_maps": true,
      "residuals": false,
      "output_type": "NIFTI_GZ",
      "mporder": 8,
      "slice2vol_niter": 8,
      "estimate_move_by_susceptibility": true,
      "args": "--fwhm=10,6,4,2,0,0,0,0"
   }


Using the parameters above we ran some benchmarks on ``sub-10159`` from
``ds000030``.  The results in the following table suggest a recommended
resource list of **8 CPUs** and **8 GB** of memory (to be safe). Beyond 8 cores
there is no additional significant speedup expected. The last line is
a comparison run without a GPU/CUDA.

.. list-table:: QSIpreps benchmarks
   :header-rows: 1

   * - Number of cores (HT)
     - Memory Usage (maxRSS)
     - Elapsed Time (H:M:S)
   * - 2
     - 3454 MB
     - 05:09:29
   * - 4
     - 3639 MB
     - 02:36:20
   * - 8
     - 4456 MB
     - 01:35:53
   * - 12
     - 4752 MB
     - 01:24:22
   * - 16
     - 6777 MB
     - 01:10:05
   * - 8 (no CUDA)
     - 4540 MB
     - 03:42:18


.. _`BIDS`: https://bids.neuroimaging.io
.. _`BIDSApps`: https://bids-apps.neuroimaging.io/apps
.. _`fMRIPrep`: https://fmriprep.org/en/stable/usage.html#command-line-arguments
.. _`MRIQC`: https://mriqc.readthedocs.io/en/stable/running.html#command-line-interface
.. _`QSIprep`: https://qsiprep.readthedocs.io/en/latest
.. _`Templateflow`: https://www.templateflow.org
.. _`DS000001`: https://openneuro.org/datasets/ds000001
.. _`eddy user-guide`: https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/eddy/UsersGuide
