Jupyter notebook server on the cluster without VDI

NOTE: The below method is not generally recommended. You can easily schedule Jupyter notebook or Jupyter Lab sessions on compute nodes using our interactive computing portal vdi.rc.fas.harvard.edu .  See more information at this page

NOTE: If you are a FAS user and looking for Jupyter Hub, you can request from FAS here
This document is to help you start a Jupyter notebook server on a compute node from the command line and connect to your computational notebook using a local browser of your workstation/laptop.
While using the App on our interactive computing portal vdi.rc.fas.harvard.edu is the recommended solution for most use cases, scheduling the job from the command line might give you more flexibility and control on advanced Slurm submission options and could be useful in some specific situations.

Start a job

After login to the login node on the cluster, you will need to start a job on the cluster. While you are setting up an environment, we recommend using an interactive node to configure your conda environment.

$ ssh user@login.rc.fas.harvard.edu
[jharvard@holylogin01 ~]$ salloc --partition test --time 60 --mem 4000

Mamba/conda environments

To set up the necessary python packages.

[jharvard@holy7c24102 ~]$ module load Mambaforge/22.11.1-fasrc01

If you already have your own Jupyter environment, activate your environment source activate environment_name and skip to the next step

[jharvard@holy7c24102 ~]$ mamba create -n jupyter_py3.11 python=3.11 jupyter    ## you can add any other package to install you might need.
[jharvard@holy7c24102 ~]$ source activate jupyter_py3.11

If you want python2, you can create a conda environment with version 2 like conda create -n jupyter_2_7 python=2.7 jupyter. You can create as many conda environments needed and load the one you want to work within your notebook server.

If you forget to activate the environment Jupyter will not see packages you installed. You have less control over the versions of packages which are part of Anaconda so we recommend you to have a conda environment for you or your lab to share.

Returning Users

You can request an interactive node based on the walltime and memory of the job or create your sbatch script to start your Jupyter server.
Whichever way you are working with SLURM to get access to a compute node, you need to load the modules and source the conda environment created during installation. In case of an interactive node,

[jharvard@holy7c24102 ~]$ module load Mambaforge/22.11.1-fasrc01
[jharvard@holy7c24102 ~]$ source activate jupyter_py3.11

Use conda env list to list available environments.

To help you connect to the Jupyter server, run the following command to get the hostname.
Select an available port between 6818 and 11845 (in this example, 6820 is the first such available port):

[jharvard@holy7c24102 ~]$ for myport in {6818..11845}; do ! nc -z localhost ${myport} && break; done
[jharvard@holy7c24102 ~]$ echo "ssh -NL $myport:$(hostname):$myport $USER@login.rc.fas.harvard.edu"
ssh -NL 6820:holy7c05314.rc.fas.harvard.edu:6820 jharvard@login.rc.fas.harvard.edu

Copy the output of the command for your user and hostname.

Starting the notebook server

Finally, you have all the components and you can start Jupyter and connect to it. On the cluster where you did source active, run the command:

(jupyter_py3.11) [jharvard@holy7c24102 ~]$jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[W 15:39:35.341 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[C 15:39:35.342 NotebookApp] You must use Jupyter Server v1 to load JupyterLab as notebook extension. You have v2.5.0 installed.
You can fix this by executing:
pip install -U "jupyter-server<2.0.0"
[I 15:39:35.391 NotebookApp] Serving notebooks from local directory: /n/home_rc/paulasan
[I 15:39:35.391 NotebookApp] Jupyter Notebook 6.5.4 is running at:
[I 15:39:35.391 NotebookApp] http://holy7c24102.rc.fas.harvard.edu:6820/?token=4abccf045e36436f72d42f292a25bf9ad84ac403793abdc1
[I 15:39:35.391 NotebookApp] or http://127.0.0.1:6820/?token=4abccf045e36436f72d42f292a25bf9ad84ac403793abdc1
[I 15:39:35.391 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

New Terminal

In a new terminal on your machine, connect to the cluster using an output of your echo command.

$ ssh -NL 6820:holy7c05314.rc.fas.harvard.edu:6820 jharvard@login.rc.fas.harvard.edu
Password:
Verification code:

Note: If this works you will not see anything further output. This means you are able to connect to port 6820 on the interactive node.

Access Notebook

Then in your workstation/laptop browser. Make sure to copy the token from the Jupyter notebook server and update the token below.
http://localhost:6820/?token=<TokenFromYourServer>

Adding an R Kernel to Jupyter

After you activate the jupyter conda environment. Please follow the steps below to setup R Kernel

(jupyter) [jharvard@holy7c19315 ~]$ module load R
(jupyter) [jharvard@holy7c19315 ~]$ export R_LIBS_USER=/your/r/package/folder:$R_LIBS_USER
(jupyter) [jharvard@holy7c19315 ~]$ R
>install.packages("devtools")
>devtools::install_github('IRkernel/IRkernel') >IRkernel::installspec()

This will take few mins to install required libraries for R and add the R kernel to Jupyter. For more details please check here. Got to the Returning User section to start Jupyter.