Search Docs by Keyword

Table of Contents

Tape Globus Access

This document is derived from Log In and Transfer Files with Globus to help FASRC users to use Globus to transfer data from different storage tiers. Globus is one of the access mechanisms to transfer data to and from the NESE Tape System so it’s important to set up a Globus account so FASRC can provide you access to the Tape endpoint. Please review Storage Service Center and Tier 3 docs as this has references for tools like Coldfront and allocations.

 

Log in with an existing identity

Visit www.globus.org and click “Login” at the top of the page. On the Globus login page, choose an organization you’re already registered with, such as Harvard University (Try typing a few letters of Harvard to narrow the list.) When you find it, click “Continue.

Globus Harvard University

You’ll be redirected to the Harvard Key Login page and use your university credentials to log in.
Harvard key

Harvard Key will ask for your permission to release your account information to Globus.

Once you’ve logged in with your organization, Globus will ask if you’d like to link to an existing account. If this is your first time logging in to Globus, click “Continue.” If you’ve already used another account with Globus, you can choose “Link to an existing account.”

 

Access Tape Storage

After you’ve signed up and logged in to Globus, you will go to the File Manager page with a left-hand menu with different icons, you’ll need to click ENDPOINTS and then click the Shared With You tab on the top menu. In the list below, you will see the Tape storage if its already set for your lab. If you don’t see the share for your lab, please check with your PI. If you are the PI, please go to the Storage Service Center and request allocation using Coldfront. To update existing allocation, also go to Coldfront and request/update allocation.

 

 

 

Share Lab Tape Endpoint

PI’s and Lab admin can share the endpoint with Lab members.  Go to the Endpoint > Shared with you and click on the lab tape endpoint. You can also search for your lab endpoints on the search area on the top menu.  After you have the share then click on the Permissions tab. On the right top corner, there is Add permissions button to add users. Search for a user these. If you can’t find your lab tape endpoint, it may not be yet set up and you may still need to request an allocation going to Coldfront.

To add, the user needs to have an existing Globus account and if you are not able to search for that user, please share this page with them to set up access using the existing identity. After you are able to find a user, give read and write access. If it’s for a collaborator and you want to share part of the data, please follow Globus documentation. Feel free to write to rchelp@rc.fas.harvard.edu or join our office hours for further assistance.

Transferring Data from FASRC storage to Tape

File Manager in Globus is to transfer data to and from Tape and other FASRC storage. We have Globus File Transfer doc to provide you with more details about Globus and its features. Here, we have a simple example to transfer data from Tier 0 storage in Holyoke to Tape.

In the Globus File Manager, there is a Panels button on the top right and I have selected the split panel as below. We have Harvard FAS RC Holyoke endpoint on the left collection, where we have most of the Holyoke storage mounted (like holylfsxx, holystore01, etc,) and on right, I have the Tape storage we have for testing.

You will need to review the Globus menu as shown below to see what all you can do with the share you have. To move data to tape, you should make sure the data is either compressed or files are greater than 100MB before transferring any data. You can review the files after login to the cluster and creating compressed files for folders with small files. You can use zip, tar, gzip, or your favorite tools for compression or review this. Also, make sure you name the files appropriately so you can find them from the Globus interface. In the example below, we have created a folder with the month and year for these files. You can use the project name, user, etc to make the discovery easier.
After you select the data you want to transfer from storage to tape, the start button on the top will start the transfer. To transfer the data from the tape, the start button on the right menu can be used. Data transfer from tape can take time as we only have data for the last few hours on buffer storage and the remaining data is stored on tape as it may take some time to move the data back to buffer storage so it can be transferred to the requested storage/endpoint.

Note: The tape system is designed to place large amounts of data onto tape cartridges for cold storage.

  1. This is designed to be a slow I/O system, data is placed on tapes for long-term storage. Retrieving data from tape by the user should only be at low volume scales.
  2. Bulk retrieval of data from tape needs to be handled by the FASRC/NESE team. A retrieval request must be made well in advance of data needs.

Please follow Globus documentation, or write to rchelp@rc.fas.harvard.edu or join our office hours for further assistance.

© The President and Fellows of Harvard College
Except where otherwise noted, this content is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.