Storage Modernization Initiative: Frequently Asked Questions (FAQs)

What is the difference between the new Compute Storage and Lab Storage?

Compute Storage is designed as an active storage location for data analysis, with data being readily accessed and utilized. It is a highly performant, cluster adjacent storage, optimized for high throughput data access, including AI and ML. Lab Storage is a general-purpose storage for hosting raw and project data. It is not intended for heavy computational workflows but can be used as a buffer storage for lab instruments. Some lab groups may decide to utilize multiple storage offerings, including both Compute Storage and Lab Storage. A full listing of the features on Compute Storage and Lab Storage including associated costs, backups, and security levels, is available on our Data Workflows webpage.

My data currently resides on FASSE. What is happening to it?

All data currently residing in a FASSE storage folder will be moving to our new FASSE storage. Information about the new features, costs, and security level is provided on our Data Workflows webpage:

https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/

Where will my data be moving? When will it be moved?

The full Storage Transition schedule is available on our Storage Modernization Initiative webpage:

https://www.rc.fas.harvard.edu/services/data-storage

Specific timelines and dates will be communicated directly to affected labs prior to the migration.

Netscratch, holylabs, and home folders will not be affected by this migration.

Who in my lab will receive the emails about upcoming storage migration efforts?

All active group members will receive email communications about upcoming storage migrations. PIs can review the list of group members in the Coldfront Allocation dashboard.

To log in to Coldfront, please use your FAS RC username and password:

https://docs.rc.fas.harvard.edu/kb/coldfront-allocation-management/

If you have difficulties with your password, you can reset it:

https://docs.rc.fas.harvard.edu/kb/reset-password/

We recommend accessing the Coldfront dashboard while connected to the FAS RC VPN:

https://docs.rc.fas.harvard.edu/kb/vpn-setup/

What are the new features of Compute Storage? Does it have backups?

The full list of features for Compute Storage is available on the Data Workflows webpage:

https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/

Compute Storage comes with weekly snapshots every two weeks. No disaster recovery option is provided.

It is important to have a post-processing plan for Compute Storage, including the option to move post-processed data to a location with disaster recovery, if needed:

https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/

What are the new features of Lab Storage? Does it have backups?

The full list of features for Lab Storage is available on the Data Workflows webpage:

https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/

Lab Storage will include daily backups for seven days and weekly backups for four weeks.

How do I access my new storage folder on Lab Storage?

On the cluster via terminal or OpenOnDemand the path is
/n/lab_storage/pi_lab

To access your data via mapped network drive, the path is
\\lab-storage-smb.rc.fas.harvard.edu\pi_lab (windows)
smb://lab-storage-smb.rc.fas.harvard.edu/pi_lab (mac)

On Globus in the “Harvard FAS RC Boston” collection, the path is
/n/lab_storage/pi_lab

All paths not containing ‘lab_storage’ or ‘lab-storage-smb’ are to be distrusted. If, somehow, paths with ‘net’, ‘fs’, ‘isilon’, ‘/n/pi_lab’ or anything else do work, you will find that they are read-only. You will be seeing old data from before the move and will not be able to write new data. Therefore, you should switch scripts and habits to use lab_storage immediately because other paths will eventually go away (maybe soon).

You can follow the steps in the Getting Started with FAS RC Storage documentation to locate your lab folder:

https://docs.rc.fas.harvard.edu/kb/getting-started-with-fasrc-storage/

I received an email about the upcoming storage migration, but I’m no longer associated with that lab. Can I be removed?

Please email rchelp@rc.fas.harvard.edu to request removal from a lab group.

We want our data to be moved somewhere else, rather than to Lab Storage.
How do we do that?

Please email rdm@rc.fas.harvard.edu to discuss migrating the data to another storage offering.

We no longer use this storage folder; can we delete the data or move it to Tape?

1. PIs should review their data storage using the Coldfront Allocation Dashboard and the Starfish Data Visualization tool to confirm that the data can be removed.

Coldfront Allocation Dashboard:
https://docs.rc.fas.harvard.edu/kb/coldfront-allocation-management/

Starfish Data Visualization tool:
https://docs.rc.fas.harvard.edu/kb/starfish-data-management/

Please review Harvard’s Data Retention FAQs to ensure compliance with the university’s data-retention policy. Please also consider any applicable IRB requirements.

Harvard’s Data Retention FAQs:
https://research.harvard.edu/files/2022/10/research_records_and_data_retention_and_maintenance_guidance_rev_2017.pdf

2. If the data can be deleted, the lab can delete the data and then request that the storage allocation be decommissioned through the Coldfront Allocation Dashboard:

https://docs.rc.fas.harvard.edu/kb/coldfront-allocation-management/

If you have difficulty deleting the data, please email rdm@rc.fas.harvard.edu. They will work with you to delete the data. Please provide the exact directory path to ensure that the correct data is deleted.

3. If you would like to migrate the data to Tape, request a new Tape storage allocation using the Coldfront Allocation Dashboard:

https://docs.rc.fas.harvard.edu/kb/coldfront-allocation-management/

A member of FAS RC will contact you once the Tape allocation is ready to use.

4. You can migrate data to Tape using Globus:

https://docs.rc.fas.harvard.edu/kb/tape-globus-access/

You may need to tar the dataset before moving it to Tape.

Tape has restrictions on how many files can be moved per directory. We have recommendations for tarring data in preparation for Tape:

https://docs.rc.fas.harvard.edu/kb/tips-for-tar-archiving/

Do I need to do anything to migrate the data? How will the migration process work?

FAS RC will migrate existing folders on each designated storage system to the new Lab Storage.

Before the cutover date, the data in your lab folder will be marked read-only. Once FAS RC has completed the migration, you will receive an email with the new folder path.

The older version on the previous storage system will then be decommissioned.

What will change with my workflow with this migration? Will I have a different folder path? Will the permissions be the same?

Folder paths for all groups will change. You may need to update your existing scripts or documentation to reflect this change.

Two subdirectories will be present by default within the folder:

Lab:
This directory is intended as the primary working directory. It is also the directory shared through Globus:

https://docs.rc.fas.harvard.edu/kb/globus-file-transfer/

By default, folders in this subdirectory are visible to the entire lab. Individual users may update their permissions to adjust access as needed. However, we highly recommend keeping access open to all lab members to allow for easier collaboration and data cleanup after you leave the university.

Everyone:
This directory is visible to anyone on the HPC cluster and is intended for collaboration with other labs on the cluster.

Data in this directory is owned by the lab that hosts it by default. This directory is not available through Globus and is intended only for internal sharing.

Directory structures may differ depending on when they were created. Some older storage folders may have a third subdirectory called Users.

Use of this folder has been deprecated due to issues related to data access:

https://docs.rc.fas.harvard.edu/kb/data-ownership-and-access-policy/

Will this impact my other storage on /n/holylabs or /n/netscratch?

Netscratch, holylabs, and home folders will not be affected by the data migrations. They will remain in their current locations.

No data from these file systems will be migrated.

I need to request more storage on Lab Storage following the migration. How do I do this?

1. To request a new storage allocation or modify an existing storage allocation, log in to the Coldfront Allocation Dashboard:

https://docs.rc.fas.harvard.edu/kb/coldfront-allocation-management/

Use your FAS RC username and password. If you have difficulties with your password, you can reset it:

https://docs.rc.fas.harvard.edu/kb/reset-password/

You may also need to clear your web browser’s cache.

If requesting a new storage allocation, indicate which storage offering you would like to acquire and provide the associated 33-digit billing code.

If you do not have a FAS RC account, request one before logging in to Coldfront:

https://docs.rc.fas.harvard.edu/kb/how-do-i-get-a-research-computing-account/

2. PIs, General Managers, and Storage Managers can request new allocations or make changes to existing allocations:

https://docs.rc.fas.harvard.edu/kb/roles-responsibilities/

PIs can assign a General Manager or Storage Manager role to their lab through Coldfront. This allows the lab member to add or modify storage allocations.

How much does Lab Storage cost? How does it compare to my previous storage?

– Lab Storage costs $125 per TiB per year, billed monthly.
– For example, if you have a 20 TiB storage allocation, the cost will be:

($125 x 20 TiB) / 12 = approximately $208 per month

– Billing is based on the total allocation size, not on usage.

How can I update my current scripts so they point to Lab Storage?

You can use a variable for pathnames in your scripts.

For example, set lab_storage to the appropriate path, then change hard-coded paths to:

$lab_storage/Lab/data

To find which scripts contain hard-coded paths, use the grep command recursively in a directory.

For example, in your home directory, you might use:

grep -R “isilon” /n/home00/jharvard

Or:

grep -R “isilon” [your new lab path]/Lab/jharvard

Can I mount Lab Storage and Compute Storage via Samba?

Lab Storage will be available through SMB.

Compute Storage will not be available through SMB.

For instructions on mounting Lab Storage, see the Mounting Storage documentation:

https://docs.rc.fas.harvard.edu/kb/mounting-storage/

Will ACLs or fACLs carry over to the new storage?

No. Unix-style fACLs cannot be moved to the new storage.

If you require specific control over subdirectories, please contact FAS RC for help creating a group to manage access.

Very granular permissions—such as including one person while excluding another—will not be possible.

I can’t access a file that existed prior to the migration

It’s possible that the file was soft-linked, or symlinked, to another location. Open the folder in a terminal and view a long listing using:

ls -l

You’ll see something that looks like:

lrwxrwx—. 1 jharvard mcfee_lab 36 Aug 22 2024 myfile -> /n/mcfee_lab/Lab/scripts/myfile

This confirms that the file was soft-linked to another location. You’ll need to update the link to point to /n/lab_storage/… or use a relative path instead.

Is Lab Storage available on Globus for file transfers?

Lab Storage is available through the Harvard FAS RC Boston collection in Globus.

This may be different if you previously used Harvard FAS RC Holyoke.

Be sure to use the /n/lab_storage/pi_lab path. The direct links, like /n/pi_lab, will be removed soon.

For more information about file transfers using Globus, see:

https://docs.rc.fas.harvard.edu/kb/globus-file-transfer/

I have additional questions about my storage. Who should I contact and how do I get in touch with them?

Please email rdm@rc.fas.harvard.edu if you have additional questions about the Storage Modernization Initiative or future storage migrations.