experiment_runner.config

Config loading and validation for the experiment runner.

Resolves user YAML against master.yaml, applies per-method defaults, and validates model and explainer names against the live runtime registries.

Attributes

logger

Classes

ConfigIssues

Collector for config problems, so one run reports all of them in one place.

Functions

result_identities(→ tuple[list[str], list[int]])

Derive the display identity and the occurrence number of every entry.

register_external_components(→ None)

Load and register external model/explainer/metric classes from the config.

validate_user_config(→ None)

Check that a user config supplies everything that cannot be defaulted.

config_resolve(→ dict)

Resolve a user config against the master config.

config_sanity_check(→ bool)

Check whether the supplied config contains errors.

Module Contents

experiment_runner.config.logger
experiment_runner.config.result_identities(names: list[str]) → tuple[list[str], list[int]]

Derive the display identity and the occurrence number of every entry.

One pass over the names serves both, because both answer the same question - which config entry is this - for two consumers that need different trade-offs:

  • the display identity names output paths and metrics.csv rows, so two entries sharing a name would otherwise overwrite each other’s artifacts and produce indistinguishable rows. Only the repeated names are suffixed _1.._N, which keeps every unique name - the overwhelmingly common case - byte-identical to what earlier versions wrote.

  • the occurrence number keys a result’s derived seed (boundary_seed()), and is always present for exactly that reason. The identity is not usable there: it suffixes only duplicates, so ["A"] yields ["A"] while ["A", "A"] yields ["A_1", "A_2"], and adding a second copy of an entry would silently change the first copy’s seed. A running count leaves the first entry’s number alone while still separating the copies, so duplicating an entry to observe variance still shows variance.

A name’s total count is its highest occurrence number, which is what decides whether it needs a suffix - so the identity is derived from the occurrences rather than counted separately.

Parameters:

names (list of str) – Display names in config order, one per entry at a single level (datasets, models, explainers or metrics).

Returns:

(identities, occurrences), both positionally aligned with names.

Return type:

tuple[list[str], list[int]]

Examples

>>> result_identities(["TSHAP", "TSHAP", "SIGN"])
(['TSHAP_1', 'TSHAP_2', 'SIGN'], [1, 2, 1])
>>> result_identities(["TSHAP"])[1] == result_identities(["TSHAP", "TSHAP"])[1][:1]
True
experiment_runner.config.register_external_components(config: dict) → None

Load and register external model/explainer/metric classes from the config.

Silent extension fallback used by config_sanity_check(): when a configured component name is absent from its runtime registry but the item supplies class_name and class_path, the class is imported from that source file and registered, so the rest of the pipeline treats it like any built-in. This lets users add components without editing package source.

Gated by general.allow_external_code (default False). If a path is supplied while the flag is off, a warning is logged and the component is left unregistered, so the subsequent validation in config_sanity_check() raises a clear error.

Parameters:

config (dict) – Resolved configuration dict.

experiment_runner.config.validate_user_config(user_config: dict, issues: ConfigIssues | None = None) → None

Check that a user config supplies everything that cannot be defaulted.

Operates on the raw config, before config_resolve() merges master in. That ordering is essential: master declares the four component lists as a catalogue, so after resolution an omitted list is indistinguishable from an inherited one. Settings describe how an experiment runs and are defaulted from master; the four lists describe what it runs and must be stated.

Parameters:

  • user_config (dict) – Raw config dict as loaded from the user YAML.

  • issues (ConfigIssues, optional) – Collector for the empty-list warnings. This check must run before the results directory exists, hence before file logging is attached, so deferring its warnings is what gets them into experiment.log.

Raises:

ValueError – If any required setting or component list is missing, naming all of them.

experiment_runner.config.config_resolve(user_config: dict, master_path: pathlib.Path, issues: ConfigIssues | None = None) → dict

Resolve a user config against the master config.

Steps performed:

  1. Load master.yaml (annotated reference with all known keys and the default values of the general and default_settings blocks).

  2. Warn + drop any unknown top-level or nested keys found in user_config.

Master’s four component lists (datasets, models, explainers, metrics) are a catalogue, not a source of values: parameter defaults belong to the classes that consume them, so an omitted parameter falls through to its constructor default via dict_to_args() rather than being filled in here. That keeps the package and the runner in agreement about what a given method is. Because validate_user_config() requires all four lists, master’s own entries are always overlaid and never leak into a run.

Parameters:

  • user_config (dict) – Raw config dict as loaded from the user YAML.

  • master_path (Path) – Path to master.yaml.

  • issues (ConfigIssues, optional) – Collector for the unknown keys dropped here, so they are reported with every other config problem instead of on their own.

Returns:

Resolved config with defaults applied and unknown keys removed.

Return type:

dict

class experiment_runner.config.ConfigIssues

Collector for config problems, so one run reports all of them in one place.

Split by consequence: an error means the experiment cannot run as configured, a warning means the run proceeds but not as the user wrote it. Every config check appends here rather than logging as it goes - unknown keys in the settings blocks, deprecated values, unknown entry keys, unregistered component names - so the report is a single block with an honest count, rather than warnings scattered through the setup output.

Warnings are logged even when errors follow, so a config with several mistakes takes one run to fix rather than one run per mistake.

errors: list[str] = []
warnings: list[str] = []
error(where: str, message: str) → None

Record a problem that prevents the run.

warn(where: str, message: str) → None

Record something the run will proceed without honouring.

report(repeat: bool = False) → None

Log everything collected as one block, then raise if anything was fatal.

Parameters:

repeat (bool) – True when re-printing at the end of a finished run, which only re-states the warnings and never raises. A long run scrolls its configuration report out of view, so the warnings are shown again where the user will actually be looking.

Raises:

ValueError – If any error was recorded and repeat is false; the message carries every one of them.

experiment_runner.config.config_sanity_check(config: dict, issues: ConfigIssues | None = None) → bool

Check whether the supplied config contains errors.

First registers any external components declared via class_name / class_path (see register_external_components()), then validates every component list entry against the live runtime registries, so that both custom entries added via register_model() / register_explainer() / register_metric() and external classes loaded from source are accepted.

Problems are split by consequence and collected into one report:

  • an unregistered component name is an error - the entry cannot run;

  • a key the runner will never read is a warning - the entry runs, but not as the user wrote it. This is where most real typos land, because a parameter name that does not match is dropped by dict_to_args() and the component silently uses its own default.

Keys in the mapping sections are handled separately, by _merge_warn_drop() during resolution; this function owns the list entries, which that merge cannot see into.

Parameters:

  • config (dict) – Configuration to check.

  • issues (ConfigIssues, optional) – Collector to append to, so these problems are reported alongside those found during resolution. When omitted, one is created and reported here.

Returns:

True if the configuration contains no errors.

Return type:

bool

Raises:

ValueError – If any entry names an unregistered component, or a domain-restricted metric has no explainer that can satisfy it.