Home directory full

If you receive an error that your home directory is full (“no space left on device [your home directory path]”)  or an email saying you are over your 100GB home directory quota (and a 95G soft quota that triggers notifications), you will need to remove files to get back under quota. Ordinarily you would just use rm to remove some files and reduce your usage.

However, a situation may arise where you are not at quota but over quota and when trying to remove files using rm, you receive the error

rm: cannot remove ‘{somefilename}’: No space left on device

NOTE: Any time you are deleting files, it is important that you check to ensure you enter the correct filename. A good rule of thumb is to use the full path to a file (instead of relative path) or cd to the directory containing the file first. Also, be extra cautious when using wildcards like * .

 

WORKAROUND

A workaround is to identify some larger file(s) to remove and to reduce that file(s) size to zero bytes. Once enough space is recovered to get you under quota, you should be able to use rm again. To do this on files you’ve identified for removal, use the truncate command:

truncate -s 0 FILENAME
To truncate a single file down to zero bytes.

or

truncate -s 0 FILE1 FILE2 FILE3
To truncate multiple files down to zero bytes

Example:

truncate -s 0 ~/Jobfiles/August/job12653287.out
It’s always safer to use the full path to a file.
~ as used here is a Unix shortcut for the path to your home directory.

Alternately, you could also use the cat command to insert zero bytes of data into a large file:

cat /dev/null > ~/mybig.file

/dev/null is a special Unix device that is always zero bytes in size

CHECKING USED SPACE

You can check to see your current total home directory (shortcut “~”) usage using the du command (plus the summary, total, and human-readable options) like so:

[jharvard@holylogin01 ~]$ du -sch ~
80G .
80G total

If you are on a login node, you can also view your computed quota directly like so:

[jharvard@holylogin01 ~]$ df -h .
Filesystem                      Size Used Avail Use% Mounted on
rcstorenfs:/ifs/rc_homes/home13 95G  80G   15G  9%   /n/home13

This shows that the user jharvard has used 80GB out of 100GB. The 95GB shown as the Size is called a soft quota. That is the threshold at which the system will notify you that you are going over.

Please bear in mind that due to the size of our home directories neither is instantaneous. The notification and re-calculation of quotas happens some time during a 24 hour period. So if you manage to go over quota before the next calculation is done, you won’t receive a soft notice. Similarly, it may take some time for your actual usage and computed quota to match again.

To find which files or directories are using the most space:

[jharvard@holylogin01 ~]$ cd ~
[jharvard@holylogin01 ~]$ du -h --max-depth=1 .
384K ./.config
232K ./Test
2.0G ./spack
...

This will show a listing of files and sizes and you can repeat the command down the directory tree to find files to delete.

What To Do If du and df Are Different

If you find that df says you are at quota while du show a lower number, you may have sparse files which are not being accounted for properly in your du(but are accounted for by the filesystem’s quota check).

To check for this, you will need to use the --apparent-size flag in du to show the logical size and find the culprit(s)

cd ~

du -ch --apparent-size --max-depth=1 .

This will show the logical size of directories and should point you to the cause.

Clearing Disk Space

.local

This hidden folder is located in your $HOME . It typically grows in size when pip install is executed outside of a conda/mamba or python virtual environments to install packages, for example while in a Jupyter or interactive session. See the warning on pip installs. This is because such installations get placed in your ~/.local , resulting in $HOME getting full.

In order to manage ~/.local, do the following:

1. Make sure that there are no jobs currently running under your profile by executing: squeue -u <username
2. Rename/turn-off .local folder: mv ~/.local ~/.local.off

.conda

Conda/Mamba environments can be quite bulky based on the number and types of packages installed inside them and should be stored in your PI’s $LAB directory. See Mamba environments in a desired location. However, if such environments are created using the default location, $HOME/.conda, then the storage size of ~/.conda can be managed as follows:

1. Remove unused packages and clear caches of Conda/Mamba:
   module load python
source activate <your-environment>
conda clean --all 
   This deletes only the unused packages in your ~/.conda/pkgs directory.
2. Remove unused conda/mamba environments:
   module load python
conda info -e
conda env remove --name <your-environment>

.cache

This directory, located in ~/.cache, can grow in size with the general use of the cluster, Open OnDemand, or VSCode. In order to manage this space, do the following:

1. Make sure that there are no jobs currently running under your profile by executing: squeue -u <username
2. Remove the folder: rm -r ~/.cache

.singularity

This folder, located in ~/.singularity, typically grows in size when a container is pulled to the cluster using Singularity. You can manage the size of this folder by cleaning its corresponding cache: singularity cache clean all

In order to avoid ~/.singularity folder from filling up, you can set a temporary directory while pulling a container and redirect the location for storing its cache. For example: export SINGULARITY_TMPDIR=/tmp/

Then, pull the container using Singularity as usual.

Note: It is best to run the above commands from a compute node, in an interactive session, as login nodes are not performant and are meant for lightweight activities only. One can ignore the interactive session being used for managing $HOME size and move forward with turning off ~/.local or deleting ~/.cache.