xai4tsc.evaluation.evaluate

evaluate() entry point and the METRICS / QUANTUS_METRICS registries.

Attributes

logger

QUANTUS_METRICS

METRICS

Functions

evaluate(→ float | numpy.ndarray | None)

Evaluate an explanation using the named metric from the METRICS registry.

ground_truth_missing(→ str | None)

Return why metric cannot score explanation, or None if it can.

required_domains_for(→ set[xai4tsc.xai._types.Domain])

Return the explanation domains a registered metric can score.

register_metric(→ None)

Register a custom metric class.

Module Contents

xai4tsc.evaluation.evaluate.logger
xai4tsc.evaluation.evaluate.evaluate(model: torch.nn.Module, metric: str, explanation: xai4tsc.xai.explain.Explanation, data: numpy.ndarray | None = None, labels: numpy.ndarray | None = None, metric_class_params: dict | None = None, metric_call_params: dict | None = None, device: str = 'cpu', seed: int | None = None) → float | numpy.ndarray | None

Evaluate an explanation using the named metric from the METRICS registry.

Parameters:

  • model – A Module (typically a ModelBase instance).

  • metric (str) – Registry key for the metric to run.

  • explanation (Explanation) – Explanation dataclass containing exp_values.

  • data (np.ndarray, optional) – Input samples.

  • labels (np.ndarray, optional) – Ground-truth labels.

  • metric_class_params (dict, optional) – Keyword arguments for the metric constructor.

  • metric_call_params (dict, optional) – Keyword arguments for the metric __call__.

  • device (str) – Compute device.

  • seed (int or None) – Seed for a metric that randomises something of its own, such as the MPRT family’s weight randomisation. Forwarded to the evaluator, which offers it only to a backend that declares a seed; the rest ignore it. A seed in metric_class_params takes precedence.

Returns:

Metric score, or None if the metric is not registered.

Return type:

float or np.ndarray or None

xai4tsc.evaluation.evaluate.ground_truth_missing(metric: str, explanation: xai4tsc.xai.explain.Explanation) → str | None

Return why metric cannot score explanation, or None if it can.

The localisation metrics compare an attribution against the regions a dataset declares as discriminative. Most datasets declare none — the discriminative region of a real recording is not known — so this is an ordinary unmet requirement rather than a failure, and the caller is expected to skip and say so.

Parameters:

  • metric (str) – A key of the METRICS registry.

  • explanation (Explanation) – The explanation to score; its metadata carries the regions.

Returns:

A reason the metric cannot be scored, or None when it can — including for every metric that does not need ground truth at all.

Return type:

str or None

xai4tsc.evaluation.evaluate.required_domains_for(metric: str, data_domain: xai4tsc.xai._types.Domain = Domain.TIME) → set[xai4tsc.xai._types.Domain]

Return the explanation domains a registered metric can score.

Reads the declaration off the registry value, so it covers native evaluator classes, user-registered subclasses, and the Quantus entries alike (those carry their per-name declaration on the factory; see _quantus_entry()). A metric that requires the model-native domain resolves to {data_domain}.

Parameters:

  • metric (str) – Registry key for the metric.

  • data_domain (Domain) – The domain the model’s inputs live in, i.e. xai4tsc.Explanation.data_domain. The default reflects that every shipped loader produces time-domain data; pass the explanation’s own value where it is known.

Returns:

The domains the metric applies to; an empty set means any domain, which is also what an unregistered name returns.

Return type:

set of Domain

xai4tsc.evaluation.evaluate.register_metric(name: str, metric_class: type) → None

Register a custom metric class.

Mirrors register_model() / register_explainer(): metric_class must be an EvaluatorBase subclass (instantiated with its constructor params at evaluation time). The fixed set of Quantus-library metrics is wired in package-side (via QUANTUS_METRICS and the single QuantusEvaluator), so users only ever register their own EvaluatorBase subclasses.

Parameters:

  • name – Key used to reference the metric in experiment configs.

  • metric_class – The metric class to register.

Raises:

TypeError – If metric_class is not an EvaluatorBase subclass.

xai4tsc.evaluation.evaluate.QUANTUS_METRICS
xai4tsc.evaluation.evaluate.METRICS: dict[str, type | collections.abc.Callable]