Runners

Runners define the execution environment in which the tasks are execute. They can be also used globally across multiple tasklists. This reduces the amount of clutter in each task definition and makes tasklists portable across multiple environments.

The desired runner is selected using the --runner option to the PyTaskFarmer program.

The BasicRunner is always available under the name default. See Provided Runners for the list of runners shipped with PyTaskFarmer.

Defining Runners

Custom runners can be defined inside the ~/.pytaskfarmer/runners.d directory or the current working directory as INI files. All files ending in .ini are loaded. There can be multiple runners defined in a single file.

The format of a single runner definition is

[runnername]
Runner = runner.python.class
Arg0 = value0
Arg1 = value1

where runnername is the name of the runner and Runner is the Python class (along with package and module) of the implementation. The remaining key:value pairs are passed to the runner.python.class constructor as keyword arguments.

Provided Runners

class taskfarmer.runners.BasicRunner

Simple runner that runs a command.

Executes the command as it is given to it. It uses subprocess.Popen to execute the task in the same environment as the worker.

__init__()

class taskfarmer.runners.ShifterRunner(image, setup='', volumes='', modules='', tempdir=False)

Executes each task inside a Shifter container. This can be preferable over starting PyTaskFarmer inside Shifter as it does not require a recent version of Python in the image. Shifter itself is started using subprocess module with the following command.

shifter --image image -- /bin/bash -c "setup && task"

The setup is user-configurable set of commands to setup the environment (ie: source ALRB) in Shifter.

See the constructor for the list of available options.

Example (ATLAS Athena Release 22):

[reco22]
Runner = taskfarmer.runners.ShifterRunner
image = zlmarshall/atlas-grid-centos7:20191110
setup = source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh && source ${AtlasSetup}/scripts/asetup.sh Athena,22.2,latest
modules = cvmfs
tempdir = True

__init__(image, setup='', volumes='', modules='', tempdir=False)

Parameters

imagestr: Name of Shfter image.
setupstr, optional: Setup command to run before executing task.
volumesstr, optional: List of volume bindings as a space separated string.
modulestr, optional: List of modules as a space separated string.
tempdirbool, optional: Each task should be run in own temporary directory.