Overview
In a Jupyter Notebook, there are some cases where you may need a specialized software stack that does not work with the default Python environment. The vast majority of cases only require that you set up a custom environment for your set of modules. An example of that can be found here.
The example below demonstrates how to launch a python environment encapsulated within an Apptainer (also known as Singularity) container. In this example, we launch python from within an Apptainer container on an Agate node in the A40 partition. The container is based off of Ubuntu 20.04, and includes a stack of software for performing GPU-accelerated simulations of Quantum Circuits using IBM Qiskit, Google Cirq, ProjectQ, and QuTiP.
MSI Features Used in This Example
- Agate cluster
- A40 GPU
- Interactive GPU partition
- JupyterLab Interface
- Open OnDemand
- Apptainer
Find the procedure in the tabs below
- 1. Build or Locate a Apptainer Image
- 2. Make the Custom Kernel Selectable by Assigning a Display Name
- 3. Edit the Kernel Description File
- 4. Launch a Notebook on an A40 Node
- 5. Connect to Jupyter Notebook
- 6. Select kernel
For the example below, you can use a public Apptainer image provided. The definition file for this example image is also provided, so you can build the image yourself. If you have never used Apptainer before, please start here.
To build an image yourself, you can start with an Apptainer Definition File. The Definition File contains all the software packages (including JupyterLab) that are needed. If you are running Apptainer on your laptop, you can build a new image with the command:
apptainer build my-new-image.sif my-definition-file.def
If you setup an account with a remote build tool, you can execute a command like this from MSI machines, or your laptop:
apptainer build --remote my-new-image.sif my-definition-file.def
Create a skeleton .json file for the python kernel that you want to run from an Apptainer container. Within the container, run
python3 -m ipykernel install --user --name singularity-qiskit --display-name "Python 3.8 Singularity QISKIT"
The Jupyter kernel will be populated with paths that are local to the container; these will need to be changed to external paths. In a shell (through OnDemand or ssh) open the file
~/.local/share/jupyter/kernels/singularity-qiskit/kernel.json
The contents should be changed to something like this:
{
"argv": [
"/usr/bin/singularity",
"exec",
"--nv",
"-B",
"/users/number/myuname:/users/number/myuname",
"-B",
"/projects:/projects",
"-B",
"/scratch.global:/scratch.global",
"https://public.s3.msi.umn.edu/singularity-images/quantum-oct-2022.sif",
"/usr/bin/python3",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"interrupt_mode": "message",
"display_name": "Python 3.8 Singularity QISKIT",
"language": "python"
}
In the example above, replace /users/number/myuname with your home directory. You can type
echo $HOME
in a shell to get the full path.
After you have an Apptainer image, note the location. It can be placed somewhere on Primary storage (home directory or group project space) or it can be placed as a publicly accessible object on an S3 bucket. For example, https://public.s3.msi.umn.edu/singularity-images/quantum-oct-2022.sif
From OnDemand,
Connect to the Notebook
After you connect to the notebook, you should be able to see the Display Name you set for your kernel in step 2. Select that kernel when you start a new notebook.