Notes on running CMS ML Studies

The following page contains some notes/hot tips on doing CMS ML studies.

Logging onto the auton cluster

You first have to ssh through a gateway, either bash.autonlab.org (or lop1.autonlab.org, depending how Predrag set you up)

e.g: ssh USERNAME@bash.autonlab.org

(You can add your public key to allow passwordless access)

From the gateway you can then access the various gpu servers through ssh.

e.g: ssh gpu10

(This can also be setup to allow passwordless access)

Configuring local environment

The instructions here explain how the pytorch environment is configured for the first install.

In subsequent sessions you can activate the environment with:

conda activate pytorch

Git on the auton cluster

Note that git only seems to work on the gateway on the worker servers (e.g: gpu10). However, emacs is not installed on the

Local data exploration

Setup a python virtual environment with python3 and matplotlib, numpy, and h5py.

Activate via something like:

source python-virtual-environments/env/bin/activate

You can then use some of the (very much) work in progress scripts (barely) documented here to make some basic plots.

Running a remote jupyter session

To launch a remote jupyter session in the auton cluster from your local machine, you need to chain 2 tunnels: first into bash.autonlab.org (or lop1.autonlab.org, depending how Predrag set you up) then again into your target machine, e.g. gpu10:

local$ ssh -i ~/.ssh/auton -L 8010:localhost:8010 mbandrews@bash.autonlab.org -t ssh -i /zfsauton/home/mbandrews/.ssh/auton -L 8010:localhost:8010 gpu10

where the first ~/.ssh/auton is the private key in my local machine for accessing bash.autonlab.org, and the second /zfsauton/home/mbandrews/.ssh/auton is the private key in my auton home folder for accessing the actual machines from the bash.autonlab.org gateway. The port number is arbitrary but should not be in use (to check: ps aux | grep ssh). Then activate whatever environment you'll need to do your jupyter stuff:

gpu10$ conda activate my_env

Launch jupyter through whatever port you specified in your tunnel:

(my_env) gpu10$ jupyter notebook --no-browser --port=8010

In addition to notebooks, the jupyter browser also doubles as a neat file/image browser allowing you to remotely look at whatever validation plots your training scripts might be producing in the background. They don't necessarily need to be produced from your notebook — the browser gives you access to the complete local file system starting from the directory you launched jupyter from.

Also, if you want to install ROOT in your conda environment:

conda install -c conda-forge root