Search Docs by Keyword

Table of Contents

Globus: Transfer Data to Tape

Globus is one of the access mechanisms to transfer data to and from the NESE Tape System.  Please review Storage Service Center and Cold Storage (tape) docs as this has references for tools like Coldfront and allocations.

See the Globus Overview documentation to learn FASRC-specific details that are needed to use Globus.

Globus login

See a video and step-by-step instructions in the Globus login documentation.

Tape Storage via Globus

Permission to access your lab’s tape collection

Access to Tape via Globus, by default, is only given to the PI and General Manager (Project Membership and Roles). The PI and General Manager can also add other lab members to access tape, but the entire group does NOT have access by default to tape storage.

If you follow the instructions below and do not see the tape Collection, that means that the PI or General Manager has not shared the Collection with you. Ask them to add you so you can access it.

If you are the PI and don’t see a tape collection, go to the Storage Service Center and request allocation using Coldfront. To update the existing allocation, go to Coldfront and update the allocation.

Access your lab’s tape collection

After you’ve logged in to Globus

  1. On the left-hand menu, go to the “File Manager”
  2. Click on the “Collections” bar and type your “Name Research Lab Tape HU FASRC”, where Name is your lab’s name (e.g., jharvard_lab will be Jharvard Research Lab Tape HU FASRC).
  3. Check the box “Search All Collections” below the “Collections” bar
    In the screenshot below is an example when searching “Research Lab Tape HU FASRC” (without Name).
  4. Click on your Research Lab collection

Share Lab Tape Collection

PI’s and General Managers can share the collection with Lab members.

  1. Go to the Collection > Shared with you
  2. Click on the lab tape collection.
    You can also search for your lab collection in the search area on the top menu.
  3. Click on the Permissions tab.
  4. On the top right corner, there is an “Add Permissions” button to add users. Search for a user there.
    1. To add a user, search for their name in the Username or Email field. If you are not able to search for that user, there are a few possibilities:
      1. You have the wrong Globus identity. Ask them what their Globus identity is.
      2. They don’t have a Globus account yet. Please send them the Globus File Transfer page and ask them to log in to Globus.
      3. If you need further assistance, feel free to write to rchelp@rc.fas.harvard.edu or join our office hours.
    2. Give read and write access. If it’s for a collaborator and you want to share only part of the data, please follow How To Share Data Using Globus

Transferring Data from FASRC storage to Tape

Note 1: The tape system is designed to place large amounts of data onto tape cartridges for cold storage documentation

  1. This is designed to be a slow I/O system, data is placed on tapes for long-term storage. Retrieving data from tape by the user should only be at low volume scales.
  2. Bulk retrieval of data from tape needs to be handled by the FASRC/NESE team. A retrieval request must be made well in advance of data needs.

Note 2: Carefully read the Tips for using tar to archive data documentation

File Manager in Globus is to transfer data to and from Tape and other FASRC storage. We have Globus File Transfer doc to provide you with more details about Globus and its features. Here, we have a simple example to transfer data from Tier 0 storage in Holyoke to Tape.

In the Globus File Manager, there is a Panels button on the top right, and I have selected the split panel as below. We have Harvard FAS RC Holyoke collection on the left collection, where we have most of the Holyoke storage mounted (like holylfsxx, holystore01, etc,) and on right, I have the Tape storage we have for testing.

To move data to tape, you should make sure the data is either compressed or files are greater than 100MB before transferring any data. You can review the files after login to the cluster and creating compressed files for folders with small files. You can use zip, tar, gzip, or your favorite tools for compression or review this. Also, make sure you name the files appropriately so you can find them from the Globus interface. In the example below, we have created a folder with the month and year for these files. You can use the project name, user, etc to make the discovery easier.  Also see Tips for using tar to archive data.
You will need to review the Globus menu as shown below to see what all you can do with the share you have.
After you select the data you want to transfer from storage to tape, the start button on the top will start the transfer. To transfer the data from the tape, the start button on the right menu can be used. Data transfer from tape can take time as we only have data for the last few hours on buffer storage and the remaining data is stored on tape as it may take some time to move the data back to buffer storage so it can be transferred to the requested storage/collection.

Please follow NESE documentation and Globus documentation, or write to rchelp@rc.fas.harvard.edu or join our office hours for further assistance.

© The President and Fellows of Harvard College
Except where otherwise noted, this content is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.