Installing Software Yourself

NOTE: This documentation assumes you are familiar with the module system. See this page for details.

Numerous software modules are available for your use on the cluster. Please search the installed modules if you’re unsure.  However, if you need to install a tool for your own use, this document outlines how you might do so.

Compiling and installing software

We have hundreds of high performance, scientific computing applications installed and available through our modules system. If there’s a software app that you’d like to use, is commonly used by a group of labs or your scientific domain, and is not available on the cluster, we encourage you to request that we install it as a module by submitting a help request on the RC Portal. We want the cluster to be as immediately usable as possible for all users, and having all the common apps available is part of that.
If you are looking to use a ‘bleeding edge’ version of software or something specific only to your lab, you can install apps yourself in your home directory, any other space to which you can write, or a shared lab directory. Writing and installing your own software may be part of your research, too. Below you will find several approaches to compiling and installing personal software on the cluster. If you are having problems during your installation, submit a help request on the RC Portal or stop by Office Hours.

Compiler choice

As explained in our documentation on modules, the cluster environment is very basic by default. You may wish to load newer or alternative versions of compilers such as gcc and icc. See this page for more details about compiler choices.

Installing GNU-toolchain-style apps

Many apps are provided in source code form and require conversion to binary form — a process called compilation or building. Such apps are often packaged to work with the GNU toolchain — you use the commands ./configure, make, and make install to build and install them. The important thing is that you specify a --prefix in order to write the software to a non-default location, i.e. a location writable by you. The examples below install personal software to a directory ~/sw/.
Software source is often distributed as a zipped tarball, such as APP-X.Y.Z.tar.gz. The first step is to unpack it. This can be done in whatever directory you like, even a temporary space that you later delete, since the installation step below will copy stuff out of this particular directory and into the prefix you specify:
tar xvf APP-X.Y.Z.tar.gz

Next, cd into the unpacked directory:
cd APP-X.Y.Z

(Often the newly created directory won’t exactly match the tarball, or the files will have been dumped to the current working directory; adjust accordingly.)
Next, configure the software, including telling it to install to somewhere in your home directory:
./configure --prefix=~/sw/

If you want to install every app in its own directory, instead of having them all share one (e.g. if you want to have different versions of the same software available), use a more specific prefix, like --prefix=~/sw/APP-X.Y.Z.
Some apps don’t actually use the full GNU toolchain, and don’t have a configure script; they have just a Makefile. In that case, you’ll have to look at the Makefile to see if it has an adjustable installation location. If not, you’ll have to manually install files.
Next, build the software:
make

This is the step that actually compiles the software. Once you work through any issues that may come up and the software compiles successfully, install it:
make install

If you get any Permission denied issues at this point, it’s possible the software did not properly use the --prefix you have it. Likewise, you may get the error No rule to make target `install'. In fact, many packages only mimic the GNU-toolchain style but actually work slightly differently. In such cases, you’ll have to manually copy the build outputs to their destination.
As a final step, you’ll want to modify your environment to make the software available for use. See this section below for details.

Adding packages to Python, Ruby, Perl, R, etc.

Please see our dedicated pages on these languages and environments:

• Python
• Ruby
• Perl
• R

Developing your own software

See this doc for information about software development on the cluster.

Making software available

The shell environment relies on environment variables that name where apps and libraries are located. When you install software to non-default locations such as your home directory, you’ll need to update these variables. Without doing so, you’ll get errors like command not found, or error while loading shared libraries.
For example, the environment variable PATH names directories in which the apps you invoke on the command line reside. Conventionally, when you install software under a --prefix, the apps are in a sub-directory of that prefix named bin. Similarly, the variable LD_LIBRARY_PATH controls what libraries are available for dynamic dependency loading, and those libraries are often put in a directory named lib.
We provide a convenience script to search a directory and print these common environment modifications. For example, if you’ve installed software using --prefix=~/sw/, you may find:
$ module load fasrc
$ generate_setup.sh --action echo ~/sw/
export PATH="/home/username/sw/bin:$PATH"
export LD_LIBRARY_PATH="/home/username/sw/lib:$LD_LIBRARY_PATH"
export LIBRARY_PATH="/home/username/sw/lib:$LIBRARY_PATH"
export PKG_CONFIG_PATH="/home/username/sw/lib/pkgconfig:$PKG_CONFIG_PATH"
export CPATH="/home/username/sw/include:$CPATH"
export FPATH="/home/username/sw/include:$FPATH"
export PYTHONPATH="/home/username/sw/lib/python2.7/site-packages:$PYTHONPATH"
export MANPATH="/home/username/sw/share/man:$MANPATH"

You can copy these variable definitions to the end of your ~/.bashrc file, and then the software will be available by default. The script is very aggressive, so you may wish to prune some if its output (e.g. FPATH is only relevant if your app provides fortran source code). Also, the output is only applicable to sh-family shells (e.g. bash), you’ll have to translate for csh-family shells (e.g. tcsh).
Alternatively, you can save this output to a file such as ~/sw/setup.sh, and then every time you want to use this software, you run:
source ~/sw/setup.sh

This is necessary if you want to install multiple incompatible apps, or different versions of the same app, and pick and choose between them at will (you’ll have to use a different --prefix for each).
You may also consider writing your own lmod modulefiles so that you can integrate your own software with our modules system.