runners

Submodules

runners.config

runners.config.artifacts_directory = None

The experiment scripts all output their results in custom named folders whose names are randomly generated (by comet.ml). These folders get saved to mapped to the same location inside the docker container. Good to know where these are so that you know where your results are.

runners.config.cache_directory = None

The training scripts generate a cache of input/output pairs for the network. These caches are zarr files that contain all the input/output for the network and can be substantial in size. It’s good to know where they are so you can free up hard drive space from time to time as needed.

runners.config.code_directory = None

This tells the Docker container where the code containing all of your scripts are. When the container starts, this is the folder you will be in. You can assume relative paths from the root of this code directory in your script. It now just uses the current working directory.

runners.config.dataset_directory = None

This folder is where all of your data lives for training and evaluating. This folder will be mapped to /storage/data/ in your docker container, allowing you to write the scripts in reference to those locations. Make sure you have read/write permissions for the folder you are pointing to.

runners.config.default_script_args = {'blocking': False, 'num_gpus': 0, 'num_workers': 1, 'run_in': 'host'}

Arguments given by default when running a script.

runners.config.nussl_directory = None

This tells the Docker container where nussl is, so the scripts can import your version of nussl. This is useful if are editing nussl continuously and testing it. This is optional as you could just use the version of nussl on Github. But if you’re editing core nussl features, this is useful.

runners.docker_runner

class runners.docker_runner.DockerRunner(image=None)

Bases: object

This class just creates a Docker container from the given image and runs it. If the image is not specified, it looks in the environment variable called ‘DOCKER_IMAGE_NAME’. If that doesn’t exist, it throws a warning.

If there are gpus available, it uses the NVIDIA docker runtime to access them through GPU passthrough. If no GPUs are available, it uses the standard runtime.

The class mounts volumes in the host into the container at specified paths. These paths are specified in config.py and are accessible via environment variables.

Parameters

image (str) – Image to use when running inside Docker. If not specified, it uses whatever is in the environment variable DOCKER_IMAGE_NAME. If that isn’t specified, it throws a warning.

run(command, gpus, name=None, detach=True)

Run a given command in a Docker container. This detaches from the container completely by default.

Parameters

• command (str) – Command to run inside the container.

• gpus (str) – What GPUs to use. For example gpus=’0,1,2’ will result in the container having access to the first 3 GPUs on the system.

• name (str) – The container should be named something. If name is not provided (default) then a name is chosen by parsing the command. Docker container names must be unique.

• detach (bool) – Whether to detach from the container after launching (running it in the background) or to stay attached to the container.

runners.experiment_utils

runners.experiment_utils.load_experiment(path_to_yml_file)

Loads an experiment given the path to the configuration file. If there is no experiment key in the info dictionary in config, then this is an experiment that should be instantiated. If comet is available, then the experiment is instantiated by comet. If it is not available, then it is instantiated by producing a directory with a random name. The name of this directory is the experiment key used to keep track of artifacts, results, and so on.

Parameters

path_to_yml_file (str) – Path to the yml file containing the experiment configuration.

Returns

3-element tuple containing

• config (dict): A dictionary containing the configuration of the experiment.

• exp (comet_ml.Experiment): An instantiated experiment if comet.ml is needed, otherwise it is None.

• path_to_yml_file (str): The path to the saved yml file (will be different from the input path if the experiment in the input yml was not yet instantiated).

Return type

tuple

runners.experiment_utils.save_experiment(config, exp=None)

Saves a configuration of an experiment to an experiment directory. If exp is defined, it is an Experiment object from comet.ml whose parameters will be updated via config.

Parameters

• config (dict) – A dictionary containing the experiment configuration. This should have all necessary parameters needed to recreate the experiment given the current codebase.

• exp (Experiment) – An Experiment object that is used by comet.ml. The settings in the configuration dictionary are logged to the Experiment object.

Returns

Path to the yml file that the config was saved to.

Return type

str

runners.run_in_docker

runners.run_in_docker.run_in_docker()

Runs a command in a Docker container. Can be configured to name the container and give it GPUs. The configuration of the docker is controlled in runners.config.

For example:

# Runs a Docker container with two GPUs.
python -m runners.run_in_docker nvidia-smi --name testing --gpus 0,1
# Simple command to ls the workspace in the Docker.
python -m runners.run_in_docker ls

or use the Makefile:

# Runs a Docker container with two GPUs.
make run_in_container command="nvidia-smi" name=testing gpus=0,1
# Simple command to ls the workspace in the Docker.
make run_in_container command="ls"

runners.script_runner_pool

class runners.script_runner_pool.ScriptRunnerPool(max_workers=10)

Bases: object

Class for launching scripts in sequence or parallel with corresponding arguments. Keeps track of GPU resources for allocating jobs, takes care of blocking on certain jobs (e.g. dataset generation), and so on. Called by scripts/pipeline.yml or by ‘make pipeline yml=path/to/.yml’.

Parameters

max_workers (int, optional) – Number of workers to use in the pool. Controls how many jobs run at the same time. Defaults to 10.

submit(scripts)

Takes in a list of scripts and allocates each of them to a ThreadPoolExecutor. Each script is run via the Makefile, so each thread spawns a process by using subprocess.run (see runners.script_runner_pool.job_runner())

The logic of this script is to go through every script in order. If the script is blocking, then it will wait until the script finishes executing before moving on to the next script. If the script requires GPU (num_gpus > 0), it will check the availability of GPUs using GPUtil and only schedule the job to be run if there is an available GPU for it. If not, it will try again every 5 seconds until all jobs have been scheduled.

Parameters

scripts (list) – A list of configurations for scripts that get run.

runners.script_runner_pool.job_runner(script)

Runs a single script in a subprocess. Uses the Makefile to run the script (which is always a Python script). The Makefile can run things either in the container or on the host.

Parameters

script (dict) – Configuration for the script.

runners.utils

runners.utils.deep_update(source, overrides)

Update a nested dictionary or similar mapping. Modify source in place.

Parameters

• source (dict) – Source dictionary that gets updated

• overrides (dict) – Dictionary with items to update in source dict.

Returns

Updated source dictionary.

Return type

(dict)

runners.utils.disp_script(spec)

Displays the arguments for a script in a readable fashion in logging.

Parameters

spec (dict) – Dictionary containing script parameters.

runners.utils.dump_yaml(data, path_to_yml)

Dump data to a yml file at a specified location.

Parameters

• data (obj) – Whatever data to dump to the yml file, as long as it can be serialized to YAML. Typically a dictionary.

• path_to_yml (str) – Where to save the data.

runners.utils.env_variables = ['DATA_DIRECTORY', 'CACHE_DIRECTORY', 'ARTIFACTS_DIRECTORY', 'NUSSL_DIRECTORY']

List of environment variables to look for and replace.

runners.utils.flatten(d, parent_key='', sep='_')

Flattens a dictionary so that it only has one level. A sequence of keys will result in a key that is like:

{ key1_key2_key3: value }

from:

{ key1: {key2: {key3: val} } }

This is done recursively.

Parameters

• d ([type]) – Dictionary that is being flattened.

• parent_key (str, optional) – The key above this one (used in recursion). Defaults to ‘’.

• sep (str, optional) – Delimiter between keys. Defaults to ‘_’.

Returns

flattened dictionary

Return type

dict

runners.utils.load_yaml(path_to_yml, env_variables=['DATA_DIRECTORY', 'CACHE_DIRECTORY', 'ARTIFACTS_DIRECTORY', 'NUSSL_DIRECTORY'])

Loads a YAML file and modifies it according to the environment variables using runners.utils.modify_yml_with_env().

Parameters

• path_to_yml (str) – Path to the YML file.

• env_variables (list) – A list of strings containing what environment variables to replace.

Returns

Parsed and loaded YAML into a dictionary.

Return type

dict

runners.utils.make_random_string(length=10)

Makes a random alphanumeric string of some length

Parameters

length (int, optional) – Length of the random string to return. Defaults to 10.

Returns

Random alphanumeric string of the specified length.

Return type

str

runners.utils.modify_yml_with_env(yml, env_variables)

Replaces specific substrings in a string or elsewhere with their corresponding environment variables. The environment variables it currently replaces are: DATA_DIRECTORY, CACHE_DIRECTORY, ARTIFACTS_DIRECTORY, and NUSSL_DIRECTORY. Descriptions of these are in setup/environment/default.sh.

Parameters

• yml (str) – A string containing the YML code (unparsed). Things in curly braces in the string are modified by passing in the data in runners.utils.env_variables.

• env_variables (list) – A list of strings containing what environment variables to replace.

Returns

YML string with the environment variables replaced.

Return type

str

runners.utils.parse_yaml(path_to_yml, jobs=True)

Parses a YAML file, replacing necessary environment variables and putting it in an expected form for the scripts.

Parameters

• path_to_yml (str) – Path to yml file to be parsed.

• jobs (bool, optional) – Whether to convert it so that it’s a sequence of jobs if jobs is not defined in spec. Defaults to True.

Returns

Loaded dictionay, modified by environment variables and depending on

jobs.

Return type

dict

runners.utils.prepare_script_args(spec)

Uses the default script args if those items are not specified for the script.

Parameters

spec (dict) – Script args to modify with defaults as needed.

Returns

Modified dictionary with values as needed.

Return type

dict

Module contents