MSI Users Bulletin - December 2022
The Users Bulletin provides a summary of new policies, procedures, and events of interest to MSI users. It is published quarterly.
1. MSI Leadership Changes
We are very happy to announce that Dr. Ben Lynch has accepted a new position as Director of MSI. Ben will replace Dr. Josh Baller, who is moving to the University of Virginia where he will serve as its inaugural Associate VP for Research Computing. While we are sad to see Josh go, we also recognize that this is an exciting opportunity for him and we expect that there will be opportunities to collaborate with Josh in his new role. Along with Ben’s promotion, we are making some other organizational changes within Research Computing that will help us to operate more effectively and provide a more coherent view of our services. Effective immediately, all of UMII staff will report under a larger MSI informatics group led by Drs. Thomas Pengo and Christy Henzler. We’re also introducing the idea of “Homes'' and “Teams,” in part to make explicit what we already do implicitly. Our goal with this change is to better align similarly skilled staff under homes and provide more explicit opportunities for supervision and evaluation within the various teams under which many of our staff now participate. The type of research and the number of staff that Research Computing supports continues to grow; therefore, it is critical that we regularly evaluate and update our internal structures and processes. We're confident that these changes will help us scale up to address new partnership opportunities while continuing to provide high quality core services. Please feel free to reach out to Jim Wilgenbusch (jwilgenb@umn.edu) if you have any questions or comments.
2. Open OnDemand
Open OnDemand, a portal for interactive access to MSI’s compute clusters, is now available following two months of MSI staff testing and a period of ~3 months when all MSI users have had access to the service to try it out. Dozens of researchers, including tutorial students and those who heavily use interactive workflows, used Open OnDemand and provided feedback. This service, approved by the MSI Advisory Group and installed at over 250 academic and commercial sites around the world, is proven and under active support from its creators. An email explaining this service was sent to all MSI users on December 8, 2022.
Through the Open OnDemand interface, which is accessed through your web browser, you will be able to access your MSI files, see a display of your current jobs on the MSI clusters, use a command line on either of our compute clusters, or generate an interactive desktop on a compute node of either cluster with configurable memory and compute resources. In addition, you will have access to familiar servers, interactive development environments, and graphical user interfaces that have been adapted to use the Open OnDemand platform. Instructions for use can be found on the MSI website.
NICE and NX No Machine, existing tools that provide some similar interactive functionality to Open OnDemand, will be retired on March 1, 2023. Current users of NICE and NX should either move their interactive workflows to Open OnDemand or move them to the SLURM scheduler, using the “interactive” partition for workflows. The sooner this is done, the more time there will be to work out any issues in OnDemand or SLURM. As always, please contact our Helpdesk with any issues.
3. Disaster Recovery Directories
This is a reminder that the only data that is backed up in your Tier 1 storage at MSI is the data located in your "disaster_recovery" folders found in your "shared" and "public" folders in your group's account. No other data is backed up in a location away from the main MSI datacenter, and thus it is vulnerable to loss if a disaster-scale event occurs there.
Please move any key datasets that cannot be replaced in the event of a datacenter disaster into the "disaster_recovery" folders. No other files in primary storage are backed up long term.
“Snapshots” – which are accessible by users, and provide a way to restore files to a very recent point in time – are a completely separate system and have not changed. Nothing is saved in snapshots for more than a month.
Disaster recovery backups are in place to restore key datasets in case of a significant incident such as a datacenter fire, flood, etc. The backups are implemented by writing user data to a tape storage system located in a separate building than our primary datacenter. These backups are not directly accessible by users, and can only be restored to the system with staff intervention.
As of late June 2022, only the “disaster_recovery” folders have been written to the disaster recovery backup. (If you do not find these “disaster_recovery” folders in your shared and public group folders, contact help@msi.umn.edu and we will add them.) These changes, approved by the MSI Advisory Group, allow MSI to increase the frequency with which files are written to the backups, and control the growth of our disaster recovery system, which has improved our ability to provide affordable Tier 1 storage solutions going forward.
4. Global Scratch Storage
A new high-performance scratch storage system from VAST Data is now available to all users for early release on the HPC clusters on /scratch.test. We encourage you to test your workloads using the new storage; please report any issues to help@msi.umn.edu. The existing global scratch filesystem (scratch.global) is unchanged, and the overall capacity and group quotas on the old and new volumes are the same.
This configuration will remain in place until the next maintenance day, January 4, 2023. If at that time there are no serious issues, the new scratch will be remounted at the standard location (/scratch.global) with no change to its contents. The old scratch volume will be remounted under /scratch.old in a read-only state.
We anticipate that the old scratch volume will be decommissioned on February 1. MSI will not migrate data between the old and new scratch volumes. If you need to retain any data from the old volume, please move it to the new volume by February 1.
5. Data Release/Data Delivery Expirations
Quarterly data deletions of UMGC data stored at MSI will occur on January 1, 2023. Affected group managers and PIs received detailed notes regarding these removals in early October (for the data_release files), and early November (for the data_delivery files). There are two locations where you may have old sequencing data removed:
Sequencing data that will be removed from your group's data_delivery/umgc directory will be in a folder labeled 4q-2021. This directory and the files within it have NOT been backed up to your Google Drive, or anywhere else on MSI storage.
Sequencing data that will be removed from your group's data_release directory are all dated between Oct 1 and Dec 31, 2017. These files were previously backed up to Google Share.
Please see the UMGC Sequencing Data Storage at MSI page for more information about storage of UMGC sequencing data at MSI. For more information about MSI's data delivery service (called "SURFS"), please see the Shared User Research Facilities Storage page.
6. Research Computing Exhibition
Save the Date! The 2023 Research Computing Exhibition (formerly the MSI Research Exhibition) will be held on Tuesday, April 25, 2023.
The Research Computing Exhibition will include a judged poster competition with prizes (travel awards), a panel discussion with industry representatives about careers in HPC, presentations about projects supported by Research Computing, and tours.
The Call for Posters will be sent out in early February. The Research Computing Exhibition webpage will be updated as information becomes available.
7. Spring Tutorials and Workshop
a. MSI Tutorial Series: MSI’s tutorials are intended to help users achieve the best results using MSI systems. Spring 2023 tutorials will be hybrid, with both in-person and online (Zoom) participation. All tutorials are held in room 575 Walter Library. Descriptions and registration links will be posted on the MSI Events page and the MSI Tutorial Calendar.
Past tutorials can be viewed on the MSI YouTube channel.
b. Software Carpentry: Software Carpentry workshops are hands-on events that cover core computational research skills to be productive as an individual scientist or in small research teams. These workshops offer graduate students, staff, and faculty practical exercises on basic computational research skills. No previous experience is needed and each workshop is taught by experienced researchers who use these tools in their own work. This is an excellent opportunity to learn about tools that will help your research process become more efficient, rigorous, and reproducible. Descriptions of Software Carpentry classes and links for registration for the Spring 2023 semester will be posted on the DASH website.
c. GEMS Learning: GEMS Learning provides modular non-credit digital and data science education for working professionals and students in food, agriculture, and natural resource application areas. Across the curriculum, instructors have built their course content from their own work executing large-scale data science projects to solve agricultural problems. Registration for GEMS Learning Spring 2023 courses is now open.
d. U-Spatial Workshops: U-Spatial offers a variety of instructor-led workshops focusing on different geospatial tools and technologies. These workshops are offered at a minimal cost to U of M students, faculty, and staff. The schedule for Spring 2023 can be found on the U-Spatial website.
8. U-Spatial Mapping Prize
The Mapping Prize competition, which is open to students at the University of Minnesota, honors provocative and innovative map-making. Information about the 2023 competition can be found at the Spatial University website.
The submission deadline for the 2023 competition is May 12, 2023.
9. 2023 DMD Conference
The annual Design of Medical Devices Conference will be held April 17-19, 2023. Registration will open on January 1, 2023. Complete information can be found on the conference website.
10. MSI Operations
MSI staff are now working under a hybrid model; not all staff are working on-site full time.
All MSI systems are operational and can be accessed remotely.
To request technical assistance with your MSI account, contact the Helpdesk:
In-person walk-ins: Monday, Wednesday and Friday 09h00 - 16h00; 587 Walter Library
Zoom walk in's Tuesdays and Thursdays, 9 - 11 am and 1 - 3 pm; go to the zoom help room
Email: help@msi.umn.edu
Phone: 612-626-0802
Online Support Request Form
Video consultation
11. Useful Webpages
Looking for help with using MSI? One of these pages may have the information you