Jupyter notebook server on the cluster without VDI

NOTE: The below method is not generally recommended. You can easily schedule Jupyter notebook or Jupyter Lab sessions on compute nodes using our interactive computing portal vdi.rc.fas.harvard.edu .  See more information at this page

NOTE: If you are a FAS user and looking for Jupyter Hub, you can request from FAS here
This document is to help you start a Jupyter notebook server on a compute node from the command line and connect to your computational notebook using a local browser of your workstation/laptop.
While using the App on our interactive computing portal vdi.rc.fas.harvard.edu is the recommended solution for most use cases, scheduling the job from the command line might give you more flexibility and control on advanced Slurm submission options and could be useful in some specific situations.

Start a job:

After login to the login node on the cluster, you will need to start a job on the cluster. While you are setting up an environment, we recommend using an interactive node to configure your conda environment.
$ ssh user@login.rc.fas.harvard.edu
$ salloc -p test -t 60 --mem 4000


Conda environments:

To set up the necessary python packages.
holy7c05314>$ module load Anaconda3/5.0.1-fasrc01

If you already have your own Jupyter environment, activate your environment source activate environment_name and skip to the next step
holy7c05314>$ conda create -n jupyter_3.6 python=3.6 jupyter  ## you can add any other package to install you might need.
holy7c05314>$ source activate jupyter_3.6
If you want python2, you can create a conda environment with version 2 like conda create -n jupyter_2_7 python=2.7 jupyter. You can create as many conda environments needed and load the one you want to work within your notebook server.
If you forget to activate the environment Jupyter will not see packages you installed. You have less control over the versions of packages which are part of Anaconda so we recommend you to have a conda environment for you or your lab to share.

Returning Users:

You can request an interactive node based on the walltime and memory of the job or create your sbatch script to start your Jupyter server.
Whichever way you are working with SLURM to get access to a compute node, you need to load the modules and source the conda environment created during installation. In case of an interactive node,
holy7c05314>$ module load Anaconda3/5.0.1-fasrc01

 holy7c05314>$ source activate jupyter_3.6

Use conda env list to list available environments.
To help you connect to the Jupyter server, run the following command to get the hostname.
Select an available port between 6818 and 11845 (in this example, 6820 is the first such available port):
$ for myport in {6818..11845}; do ! nc -z localhost ${myport} && break; done
$ echo "ssh -NL $myport:$(hostname):$myport $USER@login.rc.fas.harvard.edu"
ssh -NL 6820:holy7c05314.rc.fas.harvard.edu:6820 user@login.rc.fas.harvard.edu
Copy the output of the command for your user and hostname.

Starting the notebook server:

Finally, you have all the components and you can start Jupyter and connect to it. if the port is
(jupyter_3.6) holy7c05314>$jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'
[I 16:04:57.029 NotebookApp] Serving notebooks from local directory: /n/home_rc/user
[I 16:04:57.029 NotebookApp] 0 active kernels
[I 16:04:57.030 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:6820/?token=f5745c5674fec1dad756a1cf339f3c1e12dc2ee5c175a770

New Terminal: 

In a new terminal on your machine, connect to the cluster using an output of your echo command.
$ ssh -NL 6820:holy7c05314.rc.fas.harvard.edu:6820 user@login.rc.fas.harvard.edu
Password:
Verification code:
Note: If this works you will not see anything further output. This means you are able to connect to port 6820 on the interactive node.

Access Notebook:

Then in your workstation/laptop browser. Make sure to copy the token from the Jupyter notebook server and update the token below.
http://localhost:6820/?token=<TokenFromYourServer>

Adding an R Kernel to Jupyter:

After you activate the jupyter conda environment. Please follow the steps below to setup R Kernel
(jupyter) [rjsingh@holy7c19315 ~]$ module load R
(jupyter) [rjsingh@holy7c19315 ~]$ export R_LIBS_USER=/your/r/package/folder:$R_LIBS_USER
(jupyter) [rjsingh@holy7c19315 ~]$ R
>install.packages('devtools')

>devtools::install_github('IRkernel/IRkernel') >IRkernel::installspec()

This will take few mins to install required libraries for R and add the R kernel to Jupyter. For more details please check here. Got to the Returning User section to start Jupyter.