Use Singularity on CentOS 7

There are several important caveats to note here: Singularity is still being evaluated as a permanent offering here. This test run may expand into an MSI Beta program at some point this summer, but for now please keep in mind that this tool may go away. Along the same lines, this documentation is a work in progress; it will expand as Singularity's footprint at MSI does. If you have questions that are not addressed below or in Singularity's user documentation, please submit a ticket to help@msi.umn.edu for clarification.

Which brings us to the most important caveat: MSI can provide general guidance and best practices with Singularity. However, MSI cannot provide in-depth troubleshooting on third-party containers running in any number of unsupported applications, scripts, and OS flavors.  

(If you're just looking for a cheat sheet of Singularity commands, they're at the bottom.)

What is Singularity? How is it different from Docker?

Singularity is a containerization platform developed at the Lawrence Berkeley National Laboratory. It addresses many of the tasks otherwise completed by Docker and its cousins, but was built from the ground up with a different security paradigm: While Docker assumes (and essentially requires) its users have elevated privileges on its host systems, Singularity was designed for use in high-performance computing environments in which the protection of shared resources is paramount.

So, while any Docker user would recognize the commands documented below (singularity run my_container does exactly what you think it does), there are a couple major differences with how containers run in Singularity at MSI:

• Your container runs as you. This has several implications (no need for sudo docker run xyz!), but the main difference for users is that the user executing commands inside your container is you as well. If user fredx starts a container, and that container begins executing scripts "inside" of it, there is a fredx user in the container that's doing it. For a container to execute a command as root, it needs to be started by the root user on the host. This is likely the biggest difference people will run into if they try to run Docker containers with Singularity.
• No user context switching inside containers. This is related to the previous point, and is also a major difference from Docker: "sudo," by design, doesn't work in Singularity.

Launching container images on Mesabi

Fundamentally, singularity works exactly like any other command you would run as part of a job: You can run a container as part of a batch job, for example, and parse its output just as you would a command like cat or ls. For this example, we'll use an interactive job. To request one:

qsub -I -l walltime=30:00,nodes=1:ppn=1,mem=4gb

(Requesting an interactive session with Singularity looks the same as any other context, except you must specify one of the queues that is associated with the nodes that have Singularity available: centos7 and centos7gpu.)

Now, there's only one more step to getting a simple container running. Here's an example, if you don't have a particular container in mind:

singularity run shub://GodloveD/lolcow

At this point, Singularity will call out to Singularity Hub (that's the "shub://") and download the lolcow image from user GodloveD, and run it. If your terminal now features a cow saying something obnoxious, you successfully loaded a container and ran it on Mesabi!

What container images can I run?

Singularity technically supports six container "formats." Effectively, this means you can use Singularity to run Docker and Singularity images, plus, in theory, any alternative container that uses the ext3 format.

How do I run an image from the internet?

To download and run a container from Docker Hub, preface the user name and container name with "docker://" in the run command. For example, to reference the container available at https://hub.docker.com/_/python/ you would run:

singularity run docker://library/python

(In this case, the URL looks a little weird because the "official" Docker user that publishes the Python images is named "_" in URLs but "library" in repository names.)

The Singularity equivalent of Docker Hub is called, as you'd have guessed, Singularity Hub. To reference the container called "quantum_state_diffusion" published by user "researchapps," you would run:

singularity run shub://researchapps/quantum_state_diffusion

If you have downloaded a container image locally (see below for how to build or convert your own), you can replace the URL in the above examples with the relative path to the container image.

singularity run my_images/image1.simg

How do I execute commands inside of a container?

Passing commands into a container with Singularity works the same way it does with Docker: Rather than running singularity run containerName, you would instead use the singulary exec command. For example, to execute a command within the Python container referenced above:

singularity exec docker://library/python python cool_script1.py --cooloption=15

This would download and start the "python" image from Docker Hub, then run the command python cool_script1.py --cooloption=15 once it starts.

Accessing files from inside a container

Several directories from the host machine are automatically bind-mounted to every container:

• /tmp
• /proc
• /sys
• /dev
• plus whatever directory is specified by the $HOME environment variable. Probably your actual home directory.

Because the active user's identity is the same on the host and within the container, file permissions all apply as they normally would. Your home directory will be mounted at the same location as its location on the host machine—if running cd on the host machine takes you to /home/msistaff/rabdill, then running cd; pwd within the container will output /home/msistaff/rabdill too.

To mount other directories to specific locations within your container, you can use the "bind" option: a -B flag, followed by the host location and intended location in the container, will create a mount point there. For example:

singularity run -B /data/richard:/good_stuff my_images/coolimage1.simg

This command will start up the container defined in the file at my_images/coolimage1.simg, but it will also bind the directory on the host machine at /data/richard to a directory in the container called /good_stuff. That is to say, if you ran ls /good_stuff in the container, you would see the same results as if you ran ls /data/richard on the host machine.

Building custom images

This section is under construction, but MSI resources can NOT be used to build container images: Root access is required for this, so a virtual machine or workstation will be necessary for this step. The Singularity documentation for "Container Recipes" has what you need. Once you create an image (probably a squashfs file with a ".sif" extension", you can import it to your home directory on Panassas and use it from there without issue. Another option is to use the Singularity remote builder documented here.

Troubleshooting and feedback

If you are reading this, you are likely among the first people to run a Linux container on a supercomputer at the University of Minnesota. We would love to hear about your experience! Trouble tickets should still be routed to help@msi.umn.edu as usual, but please feel free to send feedback there as well. What worked well for you? What didn't? We're interested in the experiences users are having with containers, so don't be shy letting us know.

Quick reference

• Run container from Docker Hub:
  singularity run docker://publishing_user/container_name
• Running container from Singularity Hub:
  singularity run shub://publishing_user/container_name
• Running a local container image:
  singularity run path_to_container
• Passing a command to a container:
  singularity exec path_to_container command_goes_here so_do_parameters
• Mounting a directory in a container:
  singularity run -B path_on_host:path_in_container shub://publishing_user/container_name