FASRC DOCS

Skip to main content

Search Docs by Keyword

Table of Contents

Globus: Transfer Data to Tape

Globus is one of the access mechanisms to transfer data to and from the NESE Tape System.  Please review Storage Service Center and Cold Storage (tape) docs as this has references for tools like Coldfront and allocations.

See the Globus Overview documentation to learn FASRC-specific details that are needed to use Globus.

Globus login

See a video and step-by-step instructions in the Globus login documentation.

Tape Storage via Globus

Each NESE Tape allocation comes with a disk-based staging/buffer area that is available via Globus. Users write to the staging area and then the data is migrated to tape. Please see NESE Documentation.

Permission to access your lab’s tape collection

Access to Tape via Globus, by default, is only given to the PI and General Manager (Project Membership and Roles). The PI and General Manager can also add other lab members to access tape, but the entire group does NOT have access by default to tape storage.

If you follow the instructions below and do not see the tape Collection, that means that the PI or General Manager has not shared the Collection with you. Ask them to add you so you can access it.

If you are the PI and don’t see a tape collection, go to the Storage Service Center and request allocation using Coldfront. To update the existing allocation, go to Coldfront and update the allocation.

There is no way to restrict permissions on individual folders within your lab tape collection. Read/write access applies to the whole lab collection on Globus. The underlying filesystem is not owned by FASRC so we cannot set permissions. Everyone with access is essentially a guest on the NESE tape system.

Access your lab’s tape collection

After you’ve logged in to Globus

  1. On the left-hand menu, go to the “File Manager”
  2. Click on the “Collections” bar and type your “Name Research Lab Tape HU FASRC”, where Name is your lab’s name (e.g., jharvard_lab will be Jharvard Research Lab Tape HU FASRC).
  3. Check the box “Search All Collections” below the “Collections” bar
    In the screenshot below is an example when searching “Research Lab Tape HU FASRC” (without Name).
  4. Click on your Research Lab collection

Share Lab Tape Collection

PI’s and General Managers can share the collection with Lab members.

  1. Go to the Collection > Shared with you
  2. Click on the lab tape collection.
    You can also search for your lab collection in the search area on the top menu.
  3. Click on the Permissions tab.
  4. On the top right corner, there is an “Add Permissions” button to add users. Search for a user there.
    1. To add a user, search for their name in the Username or Email field. If you are not able to search for that user, there are a few possibilities:
      1. You have the wrong Globus identity. Ask them what their Globus identity is.
      2. They don’t have a Globus account yet. Please send them the Globus File Transfer page and ask them to log in to Globus.
      3. If you need further assistance, feel free to write to rchelp@rc.fas.harvard.edu or join our office hours.
    2. Give read and write access. If it’s for a collaborator and you want to share only part of the data, please follow How To Share Data Using Globus

Transferring Data from FASRC storage to Tape

To move data to tape, you should make sure the data is tarred and compressed. File sizes should be between 100MB and 1TB before transferring any data. We recommend maximum file size of 250GB in order to make transfers and retrievals reasonable and to protect your data against bit rot. You can review the files after login to the cluster and creating compressed files for folders with small files. You can use zip, tar, gzip, or your favorite tools for compression or review this. Also, make sure you name the files appropriately so you can find them from the Globus interface. In the example below, we have created a folder with the month and year for these files. You can use the project name, user, etc to make the discovery easier.  Also see Tips for using tar to archive data.

Carefully read the Tips for using tar to archive data documentation

File Manager in Globus is to transfer data to and from Tape and other FASRC storage. We have Globus File Transfer doc to provide you with more details about Globus and its features. Here, we have a simple example to transfer data from Tier 0 storage in Holyoke to Tape.

In the Globus File Manager, there is a Panels button on the top right, and I have selected the split panel as below. We have Harvard FAS RC Holyoke collection on the left collection, where we have most of the Holyoke storage mounted (like holylfsxx, holystore01, etc,) and on right, I have the Tape storage we have for testing.

You will need to review the Globus menu as shown below to see what all you can do with the share you have.
After you select the data you want to transfer from storage to tape, the start button on the top will start the transfer. To transfer the data from the tape, the start button on the right menu can be used. Data transfer from tape can take time as we only have data for the last few hours on buffer storage and the remaining data is stored on tape as it may take some time to move the data back to buffer storage so it can be transferred to the requested storage/collection.

Important notes

Note 1: Please see NESE documentation. It contains vital information to understand the buffer and tape system in order to properly plan and execute the safe-keeping of your data!

Note 2: Do not delete, move, or rename data after it is transferred to the tape buffer

The Globus buffer is designed so that you have write access to upload your data that needs to go to tape storage. Then, users should not delete or move any data once it is uploaded. All changes within your lab’s Globus collection are automatically picked by the tape robot. Deletions on the Globus buffer may result in permanent deletion on tape! Mass deletions and moves or file renaming require cumbersome manual intervention. In addition, deleting files that have already been placed on tape removes the “link” and makes retrieval impossible. Please work with your PI or lab data manager for tape storage protocols in your lab and let us know if you have any other questions or concerns.

Note 3: The tape system is designed to place large amounts of data onto tape cartridges for cold storage documentation

  1. This is designed to be a slow I/O system, data is placed on tapes for long-term storage. Retrieving data from tape by the user should only be at low volume scales.
  2. Bulk retrieval of data from tape needs to be handled by the FASRC/NESE team. A retrieval request must be made well in advance of data needs.

Please follow NESE documentation and Globus documentation, or write to rchelp@rc.fas.harvard.edu or join our office hours for further assistance.

Last Updated
  • © The President and Fellows of Harvard College.
    Except where otherwise noted, this content is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.