xai4tsc.evaluation.base

Evaluator base classes: EvaluatorBase ABC and QuantusEvaluator.

Attributes

logger

eval_metric_defaults

Classes

EvaluatorBase

Base class for all XAI evaluation metrics.

QuantusEvaluator

The single adapter for every Quantus metric.

Module Contents

xai4tsc.evaluation.base.logger
xai4tsc.evaluation.base.eval_metric_defaults
class xai4tsc.evaluation.base.EvaluatorBase

Bases: abc.ABC

Base class for all XAI evaluation metrics.

Subclass this and implement evaluate(), then register the subclass with xai4tsc.register_metric() to make it available by name. The registry value is a callable producing an evaluator, so a subclass (instantiated with its constructor params) satisfies the contract directly.

Example:

class MyEvaluator(EvaluatorBase):
    metric_name = "my_metric"

    def evaluate(self, model, explanation, data, labels, device="cpu", **kw):
        return float(np.mean(np.abs(explanation.exp_values)))

xai4tsc.register_metric("MyMetric", MyEvaluator)

Domain applicability

Like explainers, evaluators declare where they apply via Quantus-style class attributes: data_applicability (a set of DataType members) and, on two axes, which explanation domains they can score.

  • required_domains names the domains outright (an empty set means any), which is what a metric defined for a domain uses, e.g. a frequency perturbation curve.

  • requires_native_domain instead ties the metric to the model-native domain: the domain the model’s inputs live in (xai4tsc.Explanation.data_domain). A metric that perturbs the input and re-queries the model needs this, because that is the space it can predict in - the time domain for a model trained on raw series, the frequency domain for one trained on spectra.

Read the two together via required_domains_for(), which resolves the native rule against a given data domain; the runner skips a metric whose result does not contain the explanation’s domain.

Contract — attributions are real-valued

Metric backends operate on real attributions: Quantus preprocessing and the perturbation metrics’ np.argsort are undefined on complex arrays. Frequency / time-frequency explanations may be complex (the explanation-space wrappers transform the relevance itself), while others in the same domain are real (FreqRISE, random baselines). Backend evaluators must therefore pass attributions through real_attribution() before handing them to the underlying metric.

metric_name: str = ''

Human-readable name of the metric.

data_applicability: ClassVar[set[xai4tsc.xai._types.DataType]]

Data domains this evaluator applies to (Quantus-style; defaults to agnostic).

required_domains: ClassVar[set[xai4tsc.xai._types.Domain]]

Explanation domains this evaluator requires; empty means any domain.

requires_native_domain: ClassVar[bool] = False

Whether the explanation must be in the model-native domain (see above).

static real_attribution(a_batch: numpy.ndarray | None) → numpy.ndarray | None

Reduce a complex attribution to its magnitude; pass real arrays through.

Frequency / time-frequency explanations may be complex; metric backends operate on real values, so complex attributions are reduced to magnitude (np.abs). Real attributions are returned unchanged so the metric’s own abs handling still governs them.

Parameters:

a_batch (np.ndarray or None) – The attribution array (possibly complex), or None.

Returns:

A real-valued attribution, or None if a_batch was None.

Return type:

np.ndarray or None

abstractmethod evaluate(model: torch.nn.Module, explanation: xai4tsc.xai._types.Explanation, data: numpy.ndarray, labels: numpy.ndarray, device: str = 'cpu', **kwargs: object) → float | numpy.ndarray

Run the evaluation metric.

Parameters:

  • model – The PyTorch model being explained (an Module).

  • explanation – Explanation dataclass containing exp_values, data, labels, and encoder.

  • data (np.ndarray) – Input samples to evaluate on.

  • labels (np.ndarray) – Ground-truth labels.

  • device (str) – Compute device.

  • **kwargs – Additional metric-specific options.

Returns:

Metric score(s).

Return type:

float or np.ndarray

class xai4tsc.evaluation.base.QuantusEvaluator(name: str, **metric_class_params: object)

Bases: EvaluatorBase

The single adapter for every Quantus metric.

Initialized with a metric name (a key of the QUANTUS_METRICS lookup table), not a class — so the registry can map every Quantus metric in via one class, e.g. partial(QuantusEvaluator, name).

Example:

evaluator = QuantusEvaluator(
    "Faithfulness Correlation",
    normalise=True,
    abs=True,
    disable_warnings=True,
)
score = evaluator.evaluate(model, explanation, data, labels)

Domain applicability

Which explanation domains a Quantus metric can score depends on what the metric reads: one that reads only the attribution applies in any domain, while one that perturbs the input and re-queries the model needs the model-native domain. requires_native_domain is therefore set per metric name from QUANTUS_GROUPS; the class attribute is the conservative True default.

requires_native_domain: ClassVar[bool] = True

Conservative default; the instance relaxes it per metric name.

metric_name

Human-readable name of the metric.

evaluate(model: torch.nn.Module, explanation: xai4tsc.xai._types.Explanation, data: numpy.ndarray, labels: numpy.ndarray, device: str = 'cpu', channel_first: bool = True, seed: int | None = None, **call_params: object) → float | numpy.ndarray

Instantiate the Quantus metric and run it.

The metric is reinstantiated on each call to avoid state leakage between evaluations.

Parameters:

  • model – The model being explained.

  • explanation (Explanation) – The explanation to score; supplies the attribution and its domain.

  • data (np.ndarray) – Input samples in the framework’s canonical (B, C, T) order.

  • labels (np.ndarray) – Target labels.

  • device (str) – Compute device.

  • channel_first (bool) – Axis order of data, passed straight to the Quantus metric. The default matches the framework’s canonical (B, C, T); set it to False for channel-last inputs. Left to Quantus it is guessed from the shape, and the guess is wrong for an input with more channels than timesteps.

  • seed (int or None) – Seed for the metrics that randomise something of their own (the MPRT family’s weight randomisation, Random Logit’s class draw). Offered only to the metrics that declare a seed, since the others raise on an argument they do not name. A seed in metric_class_params wins. Named explicitly rather than left to **call_params so it reaches the metric’s constructor instead of its __call__.

  • **call_params – Further keyword arguments for the Quantus metric’s __call__, filtered to its signature.

Returns:

The metric’s score(s).

Return type:

float or np.ndarray