Slurm

class runner.runners.slurm.SlurmRunner(name, database='database.db', interpreter='#!/bin/bash', pre_runner_data=None, max_jobs=50, cycle_time=30, keep_run=False, run_folder='./', multi_fail=0, logfile=None)[source]

Bases: BaseRunner

Slurm runner

Parameters:

  • name (str) – The name of the runner. It is saved as slurm:<given_name>. Used to identify from other runners attached to the database.

  • database (str) – ASE database to connect

  • interpreter (str) – the interpreter for the shell

  • pre_runner_data (RunnerData) – pre-run runnerdata Files, tasks, and scheduler_options can be added to all the runs handled by this runner.

  • max_jobs (int) – maximum number of jobs running at an instance

  • cycle_time (int) – time in seconds

  • keep_run (bool) – keep the folder in which the run was performed

  • run_folder (str) – the folder that needs to be populated

  • multi_fail (int) – The number of re-runs on failure

  • logfile (str) – the log filename for logging

Setting up a slurm runner:

>>> from runner import SlurmRunner
>>> runner_ = SlurmRunner('myRunner',
...                       'myDatabase.db',
...                       scheduler_options={'--account': 'myAccount'},
...                       tasks=[['shell', 'module load anaconda3']],
...                       max_jobs=5,
...                       cycle_time=30,
...                       keep_run=False,
...                       run_folder='./')

This runner can start spooling via:

>>> runner_.spool()

or the runner can be attached to the database:

>>> runner_.to_database()

to be run from the respective machine:

>>> runner_ = SlurmRunner.from_database('slurm:myRunner', 'myDatabase.db')
>>> runner_.spool()

or can be run via Command Line Interface (CLI) tools:

$ runner start slurm:myRunner -db database.db
...

Slurm scheduler options in RunnerData

Schduler options for slurm are placed in the slurm file as #SBATCH options:

>>> scheduler_data = {'-N': 1,
...                   '-n': 16,
...                   '-t': '0:5:0:0',
...                   '--mem-per-cpu': 2000}

This will result in a slurm file as:

#SBATCH -N 1
#SBATCH -n 16
#SBATCH -t 0:5:0:0
#SBATCH --mem-per-cpu=2000

See https://slurm.schedmd.com/pdfs/summary.pdf for further options.