Commit b8718567 authored by Hansen's avatar Hansen
Browse files

software: singularity added pro contra of container building methods

parent 00792993
......@@ -4,21 +4,22 @@ Singularity
Singularity enables users to package their computational research environments,
making them more mobile and improving repeatability
and reproducibility of results.
It can build and run so called container,
It can build and run so called containers,
whose content can be predefined in special text files called recipes.
A container, once build,
can be run on any system that has singularity itself installed,
can be run on any system, that has singularity itself installed,
even if it contains software not normally supported for it.
Since a container is (outwardly) only comprised of one file,
it is easy to distribute it to other pcs,
improving mobility and guarantying a consistent work environment.
This also enables users to execute their software
of choice in environments like HPC-Clusters,
it is easy to distribute it to other computers.
This improves mobility and guaranties a consistent work environment
even if workflows are performed over heterogenous set of computers.
It also enables users to choose their software
to use in environments like HPC-Clusters
without root privileges
and without being dependent on system administrators.
All that is needed, is a singularity installation on the system.
A container can also be published with your research data and results,
helping to keep good scientific practice
A container can also be published with users research data and results,
helping to ensure good scientific practice
by making results more easily testable.
Singularity is however limited to software
executable in linux based operating systems.
......@@ -26,7 +27,7 @@ executable in linux based operating systems.
**Singularity in short:**
* Helps users to build their software environment
how they want it and use it where they want it. (with limits)
* Helps the research community
* Helps the research community,
by supporting the improvement of research softwares reproducibility.
* Needs root privileges to build containers but not to use them.
* Can "only" run software compatible with linux distributions.
......@@ -45,13 +46,18 @@ Getting Containers
------------------
There exists an official repository for singularity containers.
It is called `Singularity Hub <https://www.singularity-hub.org/>`_ and hosts containers created by the user community.
It is called `Singularity Hub <https://www.singularity-hub.org/>`_
and hosts containers created by the user community.
Singularity Hub allows users
to link their GitHub account and projects with it and builds containers
from recipe_ files stored in the chosen GitHub projects.
The repository can be browsed and searched through, using the webpage.
to link their GitHub account and projects with its repository.
`Recipe <recipe_>`_ files, stored in chosen branches and
`conforming to specific naming conventions
<https://github.com/singularityhub/singularityhub.github.io/wiki/Automated-Build#finding-recipes>`_,
are then build and published through on the repository.
The repositories content can be browsed
and searched through, using the webpage.
Containers are grouped in collections, defined by the username of the owner.
Ever container in a collection has a name and a version_.
Every container in a collection has a name and a version_.
**Every container visible on singularity-hub can be downloaded,
using singularity with the pull command:**
......@@ -59,7 +65,7 @@ using singularity with the pull command:**
singularity pull shub://collection_name/container_name:version
They can also be directly `executed <run>`_ using:
They can also be directly `executed <run_>`_ using:
.. code-block:: shell
......@@ -73,6 +79,9 @@ or entered through a shell_ using:
.. _mpibhub_reg:
The :code:`shub://` prefix tells singularity, that the source
to download from is a singularity registry (sregistry).
Private Registry (mpibhub)
~~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -103,7 +112,7 @@ There is however a lot of software already containerized
and available at their container repository
`Docker Hub <https://hub.docker.com/>`_.
Singularity can load docker-images and convert them
to singularity container, which can shrink the
into the singularity format, which can shrink the
workload of building containers.
The command to pull a docker-container would look somewhat
like this:
......@@ -115,8 +124,8 @@ like this:
singularity pull docker://python:3.7.0b3-alpine
Docker Hub stores some images, that are from official partners,
like the python example.
Those do not have a name in the form of :code:`username/container`.
like the python example.
Those do not have names in the form of :code:`username/container`.
Images, that are created my normal users do however have
this naming schema.
......@@ -145,6 +154,7 @@ The following statements should therefore be equivalent:
singularity pull shub://collection/container
singularity pull shub://collection/container:latest
To improve reproducibility, a version tag should always be specified.
.. _build:
......@@ -155,6 +165,7 @@ Building a Container
Building a container requires root privileges.
Users, that want to build singularity containers themselves
need access to a computer where it is possible to attain those root privileges.
This will most likely be their privately owned or work supplied computer.
The most common way to build a container is by using a
recipe_ file and the singularity build command:
......@@ -165,9 +176,9 @@ recipe_ file and the singularity build command:
.. _recipe:
Recipe Files
~~~~~~~~~~~~
~~~~~~~~~~~~~
A Recipe is a plain text file,
A recipe_ is a plain text file,
that is a blueprint for the building of a container.
It should begin with two lines looking somewhat like this:
......@@ -177,7 +188,7 @@ It should begin with two lines looking somewhat like this:
From: collection/container:version
This defines the basis of the image.
:code:`From:` define
:code:`From:` defines
the base-image of the container and :code:`Bootstrap:` the type
of image. The target of :code:`From:` can also be a container at
`the private sregistry <mpibhub_reg_>`_
......@@ -189,9 +200,16 @@ of image. The target of :code:`From:` can also be a container at
The :code:`%post` section can be used
to install software and change the container in general.
It is mostly made up of commands, that would just be typed into the
users terminal during manual installation.
To install the the editor vim, one would for example
add the following to the recipe:
.. code-block:: shell
%post
apt-get update
apt-get -y install vim
......@@ -200,25 +218,27 @@ the building of the image is not interactive.
The command :code:`apt-get install` would prompt for user input
and cause the build to fail.
It should be kept in mind,
that setting such flags for commands, that would otherwise
prompt for user input, must always be set.
that such commands must **always** be told to run noninteractively,
to not cause errors
and stop a potentially very long building process prematurely.
It is advised to delete no longer needed packages and files
at the end of the post section:
.. code-block:: shell
# Remove software no longer needed:
apt-get purge software_i_used_only_to_do_installation
# This removes the package list
# Remove the package list
# created by "apt-get update",
# downloaded package archives and
# packages not used by anything.
# packages not used by anything:
apt-get autoclean -y
apt-get autoremove -y
rm -rf /var/lib/apt/lists/*
A detailed documentation of singularity recipes is
given at the `official recipe documentation <http://singularity.lbl.gov/docs-recipes>`
given at the `official recipe documentation <http://singularity.lbl.gov/docs-recipes>`_
Problems with Consistency
~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -232,6 +252,9 @@ installed at build time.
Installing from software repositories using :code:`apt-get install`
for example can lead to the installation
of different versions at different times.
Base images used to build upon can also be subject to change.
Either because they also rely on software repositories for building
or because they where deliberately modified.
This means that adding to the recipe_
file and building a container with additional functionality
could compromise its consistent functionality as a whole.
......@@ -258,7 +281,7 @@ already installed software.
Bootstrap: localimage
From: oldimage.simg
2. Creating a writable container or sandbox and manually installing software through its shell.
2. Creating a sandbox and manually installing software through its shell.
.. code-block:: shell
......@@ -276,29 +299,38 @@ already installed software.
sudo singularity build newimage.simg sandbox_folder
sudo rm -rf sandbox_folder
3. Creating a writable overlay.
Just for testing new software, a writable overlay can be used.
.. code-block:: shell
# Create overlay
singularity image.create overlay.img
# Run overlay
sudo singularity shell --overlay overlay.img oldimage.simg
# Install software
Method 2 and 3 will break the
To use the changes made in the overlay, its use has to always be specified
with the :code:`--overlay` flag.
Using a sandbox will break the
reproducibility of the used container, since it is no
longer possible to follow the installation process inside a recipe_ file.
The building with the :code:`--writable` flag could furthermore lead to
corruption of the created image. This is however only an issue if the
An overlay on the other hand will
break the reproducibility of produced results,
since it changed the behavior of the used container base container.
This is however only an issue if the
container is to be published or shared with others.
Using method 1 reproducibility is kept,
if the base image is publicly available or made available.
This can however lead to a, hard to follow, chain of recipes and images.
It is also possible to try to obtain third party software only from sources
that are sure to provide always the
that are sure to always provide the
exact same version and that also enable the
user to choose specific versions.
A best practice for developing containers
is described in `Keeping Containers Reproducible`_.
Keeping Containers Reproducible
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -315,6 +347,60 @@ container can be build. Container and recipe_ can then be distributed.
It is however possible, that the final container is not a perfect replication
of the sandbox (see `Problems with Consistency <consistent_>`_)
Pros and Cons of Container Creation Methods
+++++++++++++++++++++++++++++++++++++++++++
Installing into Sandbox and creating a container from it:
* Pro:
* Interactive, errors can be corrected immediately.
* No redundant installation processes: Shorter installation time.
* No need to keep recipe file up to date.
* Contra:
* Changes, breaking the functionality, cannot be easily reversed.
* Hampers reproducibility.
Iteratively adding to the recipe file and building from it:
* Pro:
* Makes used software transparent: Helps reproducibility.
* Functionality breaking changes can be reversed.
* Can be published on Singularity Hub.
* Contra:
* May lead to containers,
that are different in more than the additional software.
* Recipe file has to be maintained during development.
Iteratively using old containers as base
(This would have to incorporate
always letting Singularity Hub build the container)
* Pro:
* The already functioning part of the environment is not build again.
* Comprehension of the building process is less easy but still possible.
* Functionality breaking changes can be reversed.
* Can be published on Singularity Hub.
* Contra:
* A more or less complicated chain of containers has to be inspected to
comprehend the resulting containers function.
* Several recipe files have to be maintained.
* Software only needed for building, like make or git, have to be installed
in every iteration, if the size size of the containers is not supposed be
unnecessarily large.
* Needs Singularity Hub to be reproducible.
None of these methods will protect against changes caused by the installation
of the new software.
.. _arch:
Container Architecture
......@@ -358,7 +444,7 @@ on shared systems.)
Run Container
-------------
During the `creation <build>`_ of a container,
During the `creation <build_>`_ of a container,
a run script can be specified in the :code:`%runscript`
section of its recipe_.
This script will be executed upon calling
......@@ -377,7 +463,8 @@ functions interactively.
Keep in mind that all changes made to the container
itself will be lost upon exiting the shell.
Only changes made in mounted files and folders
will persist. (see `Container Architecture <arch>`_)
will persist. (see `Container Architecture <arch_>`_)
Singularity Documentation
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment