xargs --null --arg-file=<(find "$OLDHOME" -maxdepth 1 -mindepth 1 \( -type f -o -type l -o -type d \) -name '.*' -not -name .trash -not -name .Trash -not -name .conda -not -name .cache -not -name .apptainer -not -name .singularity -not -name .bashrc -not -name .bash_profile -not -name .profile -print0) -n 1 -P 4 cp --preserve=mode,timestamps --target-dir="$HOME" -Rv

Singularity and Apptainer config folders

The Singularity and Apptainer configuration folders largely only contain cache files.  There are key files however that you may wish to preserve.  

If you use either of these applications, copy and paste the code block below to copy the application keys over to your new home directory (without copying over all the cache data)

if [[ -d "$OLDHOME/.apptainer/keys" ]]; then 
    mkdir "$HOME/.apptainer"
    cp -r --preserve=mode,timestamps "$OLDHOME/.apptainer/keys" "$HOME/.apptainer"
fi

if [[ -d "$OLDHOME/.singularity/sypgp" ]]; then
    mkdir "$HOME/.singularity"
    cp -r --preserve=mode,timestamps "$OLDHOME/.singularity/sygpg" "$HOME/.singularity"
fi

Conda

Due to how conda creates its environments and to recent changes to the Anaconda terms of service, it is not possible for MSI to simply copy your conda environments over for you. You will need to reinstall your environments in your new home directory, using a module that is compliant with Anaconda's terms of service. Currently we are providing the `miniforge` module for this purpose.

Your existing conda environments are stored by default at the following path:

$OLDHOME/.conda/envs

If you used miniconda, micromamba, or other convenience tools to install your environments, then the location of your default install directory may be different than this. You should consult the documentation for the particular tool you are using to see where it installs environments by default. It is also possible that you installed your conda environments to a different location if you wanted to share them with group members; the same instructions will apply for these cases but with a different origin directory.

To avoid package channel conflicts with our install method, let's first make sure that we don't have any specific channels selected in our ~/.condarc file. For instance, my ~/.condarc file might look like the following:

channels:
- bioconda
- conda-forge
- defaults
default_threads: 1

Yours might look different if you have other settings configured for your conda usage. For my file it might look different if you have other settings configured for your conda usage. If you do have a ~/.condarc file that includes a 'channels' section, it should be removed before we proceed. For my file, after removing the channels section it would contain only:

default_threads: 1

After the new environment is installed you can consider re-adding the channels to your ~/.condarc file. If you do so please omit the 'default' and 'anaconda' channels, as using these at MSI would currently violate the Anaconda terms of service.

Once you've removed the 'channels' section, or if you don't have a 'channels' section in yours at all, then you can continue to the next step.

In order to reinstall your environment in the new location, we will first capture the state of the environment in $OLDHOME, then attempt to reinstall it to the new home directory.

First, let's create a yaml file with the versions of packages installed to this environment. We'll use this yaml file to attempt a reinstallation later.

module load miniforge
conda env export --no-builds -p $OLDHOME/.conda/envs/my_tensorflow > my_tensorflow.yaml

It is important that we use the miniforge module here, rather than one of our other modules that provides conda. Using any other module as the source of conda to reinstall these environments is likely to install the new environment in a way that will break in the near future.

This creates a file `my_tensorflow.yaml` that contains the version specification for each of the packages that conda installed in your environment. It also contains the `name` and `prefix` headings, which you will want to update or remove.

If you intend to install the environment to the default location, make sure that the `name` field is correct and remove the line containing `prefix` and its value. If you want to install to a non-default location, update the value of the `prefix` line accordingly and replace the value of `name` with `null` if it is not already set.

We also need to edit the contents of the 'channels' heading in our yaml file. If either 'defaults' or 'anaconda' appears in this section, they should be removed and replaced by 'conda-forge'. For instance, I might have a yaml file that contains the following 'channels' section:

channels:
- bioconda
- defaults

I would need to edit this section of the yaml file to read:

channels:
- bioconda
- conda-forge

Once you've made the necessary edits to your yaml file, you can attempt to reinstall it to the new location via:

conda env create -f my_tensorflow.yaml

This may look a little different than a typical conda installation as it handles pip-installed dependencies a bit differently, but if it completes without error then you should be able to activate and use your conda environment from the new location you've selected.

In certain cases, the environment may fail to reinstall because the specific versions of the packages you are requesting are not available anymore, or the developers have updated the dependencies to require or forbid specific versions of other packages. In these cases, conda will print out a message that looks like the following.

PackagesNotFoundError: The following packages are not available from current channels:
  - numpy=3.1*

For a version conflict:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package numpy-1.18.1-py36h95a1406_0 requires python >=3.6,<3.7.0a0, but none of the providers can be installed

You can work around this by relaxing the version specificity of your environment install file. To do so, find the conflicting package(s) in your yaml file, and remove the version from the line specifying that package. For instance, if I have a conflict with the packages python and numpy, I would find the lines in my yaml file in the 'dependencies' section containing:

...
- ncurses=6.5
- numpy=1.18
- openssl=3.3.1
- pip=24.2
- python=3.12
- python_abi=3.12
...

I would then remove the version specifiers from these lines, so they would read:

...
- ncurses=6.5
- numpy
- openssl=3.3.1
- pip=24.2
- python
- python_abi=3.12
...

And attempt to install the environment again from this updated file:

conda env create -f my_tensorflow.yaml

Depending on your specific situation you may need to remove additional version specifiers from packages if another conflict is reported, but eventually a working environment should be installed from your yaml file.

The resulting environment will not be identical to the original, but it will be as close as we can get via a fresh installation at this time.

We are still working on the documentation here, so if you run into issues we haven't covered yet please contact [email protected] for direct assistance with migrating your old conda environments.  We will update this section with additional guidance as it becomes available. 

R and RStudio Profile, History, and Libraries

You can either copy over your old libraries/packages, history, and profile, or start over from scratch in your new directory (highly recommended).

If you want to copy over your history and profile, you can use:

cp -ra "$OLDHOME/.Rprofile" "$HOME"
cp -ra "$OLDHOME/.Rhistory" "$HOME"

As with all data in your old home directory, we encourage you to consider if the R libraries you have installed there are best located in your (new) home directory, or if they would be better located in your project's "shared" folder.  If you determine your home directory is the correct location for your R libraries, copying them over should be done selectively.

If you used the default location for R user libraries, you can view your old R libraries with

find "$OLDHOME/R" -maxdepth 2 -type d

You can then selectively copy over each library you want to maintain. 

For example if you want to copy over

$OLDHOME/R/x86_64-pc-linux-gnu-library/4.4

You can use this series of commands to copy it over

mkdir -p "$HOME/R/x86_64-pc-linux-gnu-library/4.4"
cp -ra "$OLDHOME/R/x86_64-pc-linux-gnu-library/4.4" "$HOME/R/x86_64-pc-linux-gnu-library"

If your R libraries are in, or end up at, a non-standard location (e.g. your project SHARED folder), then you can tell R the location of the R libraries you want to use either

• Within R itself via .libPaths()
• In the shell via the R_LIBS_USER environment variable 

Please keep in mind that R libraries always need to match the version of R you're using.

How to set .libPaths within R

Replace "/path/to/your/R/libraries" with the actual path to where the R libraries you want to use are stored.

NOTE: While you can point at your OLDHOME (variable) for this libPaths setting, it's strongly encouraged to copy/move these libraries to either your new HOME directory (variable) or your Project's SHARED folder (variable).

# Example
.libPaths(paste0(Sys.getenv("SHARED"),"/R/x86_64-pc-linux-gnu-library/4.4"))
# or if you have your libraries installed elsewhere use that path
.libPaths("/path/to/your/R/libraries")

How to set R_LIBS_USER in the shell or bash script

Replace "/path/to/your/R/libraries" with the actual path to where the R libraries you want to use are stored

NOTE: You can use the new Variables like $PROJECT and $SHARED in this type of declaration

export R_LIBS_USER="/path/to/your/R/libraries"
# Example
export R_LIBS_USER="$SHARED/R/x86_64-pc-linux-gnu-library/4.4"

Could not chdir to home directory /users/#/name: Input/output error
/usr/bin/xauth:  error in locking authority file /users/#/name/.Xauthority