Tag: Storage service center

Starfish  – Data Management

Starfish – Data Management

To scan and manage data on different filesystems, FASRC is using Starfish. We have 2 different Starfish servers to scan our storage servers and the information is below:

Starfish Service URL Storage Servers
https://starfish01.rc.fas.harvard.edu/  holylfs04 (Tier0), holylfs05 (Tier0), holystore01
https://starfish02.rc.fas.harvard.edu/  holylfs02, holy-isilon(Tier1), bos-isilon(Tier1), boslfs02 (Tier 0)

Note: If your lab has not requested access to starfish already, please reach out to us: rchelp@rc.fas.harvard.edu

 

Starfish Features:

  • View of all your lab data folders with basic statistics per folder (usage, owner, creation, last modified, last access data)
  • View access only to your lab’s data, using a zone.
  • Export basic statistics to a CSV file to share and offline review
  • Tagging of data based on project/owners
  • Summary of usage based on file extensions, users, and tags.

 

Coldfront – Allocation management

Coldfront – Allocation management

Please check the Storage Service Center page for the dataflow to ColdFront

ColdFront is an open-source resource allocation management system designed to provide a central portal for administration, usage reporting, and allocation management of HPC resources.  FASRC adapted the software to manage allocations on the FASRC cluster.  Right now, the service has 2 major components:

Project: Lab groups and projects, if your lab has more than one group or you are part of multiple groups you will see more than one project.

Allocation: Storage Allocation on one or more storage devices.

Users need to connect to @fasrc VPN and can log in to Coldfront using FASRC credentials:

https://coldfront.rc.fas.harvard.edu/

After you log in to coldfront, you will see one or more projects on the left and one or more allocations on the right. Click, project link or Allocation active button to view details about your projects/allocation.

Project/s:

Project list all the users in your lab group. These are 2 main roles, Manager and User. Only the users with the manager role are allowed to request new allocations or request changes to the current allocation.  A user role is to view their current projects and allocations.  The allocation tab shows your current allocations and usage.

Note: If you should be authorized to request allocations, please ask the PI to log in to coldfront and promote your user as manager.

Allocations:

The page shows the total allocation size, total usage, and estimated cost per month. It also has a table to show usage per user and the information is updated once a week from the data management system (Starfish). It shows the estimated cost per month after aggregating their usage across different folders. Starfish is used to get this data and Labs can request access to starfish or its data to better understand the usage and remove any old project or user data.

Currently, allocation requests are for different storage Tiers (Tier0, Tier1, Tier3). We are still working on Tier2 to offer it as a service.

Allocation Requests:

PI or Managers of a lab project should be able to make a new allocation request or request changes to the current allocation. To update the current allocation, please add the new total size and add a comment in the justification. Please review https://www.rc.fas.harvard.edu/services/data-storage/#Offerings_Tiers_of_Service for the storage features and updates

If you have any questions. please feel free to reach us here: rchelp@rc.fas.harvard.edu

Storage Service Center

Storage Service Center

This page is provides the necessary information for requesting, managing data storage allocations and billing.  It is important that you review the Storage Billing FAQ and Data Storage Service pages.  Below we provide information on the three different software applications we use to help PIs (Coldfront), Finance managers (FIINE), and Lab/Data managers (Starfish) perform their roles.

To get more information about the storage tiers and their features please visit storage tiers on our data-storage services page. Please feel free to reach out to us at  rchelp@rc.fas.harvard.edu with any questions.

 

REQUEST OR MANAGE AN ALLOCATION

  • To request a new allocation or manage and existing one, you will use Coldfront
  • If you have an existing allocation and cannot access Coldfront, please contacts FASRC

 

MANAGE BILLING FOR ALLOCATIONS

Charges for storage allocations are billed monthly. Expense code(s) can be applied to each allocation and can be sub-divided amongst multiple codes.
See our Service Center FAQ for answers to common questions.

To manage billing for an existing allocation, you will need:

Instructions for expense code management and billing record review in FIINE are available at:
https://ifx.rc.fas.harvard.edu/docs/user/fiine.html

 

Starfish – Data Management

Starfish – Scans the different storage servers to provide a view, usage details, metadata, and tagging based on the projects. Check here for more details about starfish and examples to query the data.

Coldfront – Lab and Allocation management:

Coldfront – Provides a view on PI projects and allocations. New allocation and updates to existing allocation can be requested using Coldfront. Check here for more details about Coldfront and its use.

Fiine -FAS Instrument Invoicing Environment

Fiine – For lab/finance administrator to manage the expense codes per project/user and view invoices. Check here for more information about using the Fiine system.

 

© The President and Fellows of Harvard College
Except where otherwise noted, this content is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.