Setting Release Thresholds for Spatial Agents

A release gate needs a number, and the number is usually chosen by someone saying “let’s require ninety per cent”. That figure has no relationship to the system’s actual behaviour, so it either blocks every release or none, and within two months it has been removed. This guide derives thresholds from measured history instead, as the decision layer over evaluation and benchmarking for spatial LLMs.

When to Use This Approach

Set thresholds once you have enough history to see the noise floor — five or more sweeps of a stable system. Before that, publish the numbers without gating on them; a gate calibrated against two data points measures the two data points.

The fourth row is the exception to the whole approach. Most thresholds should be relative to observed behaviour, and a small number — an unfounded-answer rate, a refusal-path failure — deserve an absolute floor that does not move when the baseline drifts.

Implementation

The proposal function reads history, checks that the history is stable enough to derive from, and returns a threshold with the evidence behind it.

import logging
import statistics
from dataclasses import dataclass
from typing import Sequence

log = logging.getLogger("release_thresholds")

MIN_HISTORY = 5


@dataclass(frozen=True)
class Threshold:
    metric: str
    value: float
    baseline: float
    spread: float
    basis: str


def propose(metric: str, history: Sequence[float],
            allowed_regression: float = 0.02,
            absolute_floor: float | None = None) -> Threshold | None:
    """Derive a gate from observed behaviour. Returns None when history is too thin."""
    values = [v for v in history if v is not None]
    if len(values) < MIN_HISTORY:
        log.info("%s: only %d sweep(s) of history — publishing without a gate",
                 metric, len(values))
        return None

    baseline = statistics.median(values)
    spread = statistics.pstdev(values) if len(values) > 1 else 0.0
    # Sit below both the baseline-minus-allowance and the observed noise band.
    candidate = min(baseline - allowed_regression, baseline - 2 * spread)
    if absolute_floor is not None:
        candidate = max(candidate, absolute_floor)
    value = round(max(0.0, candidate), 4)

    basis = (f"median {baseline:.4f} over {len(values)} sweep(s), "
             f"spread {spread:.4f}, allowance {allowed_regression}")
    if absolute_floor is not None and value == absolute_floor:
        basis += f"; raised to the absolute floor {absolute_floor}"
    return Threshold(metric, value, baseline, spread, basis)

Taking the minimum of two candidates is what makes the gate robust. An allowance alone ignores how noisy the metric is, so a metric that varies by five points between identical sweeps will trip a two-point allowance constantly; two standard deviations alone can produce an absurdly low gate for a very stable metric. The minimum respects both.

The gate itself needs an override path, and the override needs to be recorded rather than merely possible.

@dataclass(frozen=True)
class GateResult:
    passed: bool
    failures: tuple[str, ...]
    overridden: bool
    reason: str


def evaluate(report: dict, thresholds: dict[str, Threshold],
             override_reason: str | None = None) -> GateResult:
    """Apply thresholds to a report. An override is allowed and always recorded."""
    failures = tuple(
        f"{name}: {report[name]['p10']:.4f} below {t.value:.4f}"
        for name, t in thresholds.items()
        if name in report and report[name]["p10"] < t.value
    )
    if not failures:
        return GateResult(True, (), False, "")
    if override_reason:
        log.warning("release gate overridden (%s): %s", override_reason, "; ".join(failures))
        return GateResult(True, failures, True, override_reason)
    log.error("release blocked: %s", "; ".join(failures))
    return GateResult(False, failures, False, "")

Allowing the override is not a weakness. A gate with no override will be deleted the first time it blocks an urgent security fix, and its absence afterwards is permanent; an override that requires a reason and appears in the release record keeps the gate alive and turns each bypass into a decision somebody signed.

Validation & Testing

def test_thin_history_yields_no_gate():
    assert propose("placement", [0.9, 0.91, 0.89]) is None


def test_noisy_metric_gets_a_lower_gate_than_a_stable_one():
    stable = propose("a", [0.90, 0.90, 0.91, 0.90, 0.90])
    noisy = propose("b", [0.90, 0.82, 0.95, 0.86, 0.93])
    assert noisy.value < stable.value


def test_absolute_floor_wins_when_it_is_higher():
    t = propose("unfounded_answers", [0.2] * 6, absolute_floor=0.5)
    assert t.value == 0.5 and "absolute floor" in t.basis


def test_override_is_recorded_not_silent(caplog):
    result = evaluate(FAILING_REPORT, THRESHOLDS, override_reason="hotfix for outage")
    assert result.passed and result.overridden and result.failures
    assert any("overridden" in r.message for r in caplog.records)

The fourth test is the one that keeps the override honest. An override implementation that returns passed=True and discards the failures is indistinguishable, from the caller’s perspective, from a genuine pass — and a release record that cannot show which releases bypassed the gate cannot answer the only question anyone asks about gates afterwards.

Store the derived thresholds as data rather than recomputing them on every run. A gate whose value is recalculated from recent history at evaluation time will drift downward automatically as behaviour degrades, which is the opposite of what a gate is for.

Gotchas & Edge Cases

A threshold derived from a period that included a regression. The baseline absorbs the bad sweeps and the gate is set below the degraded behaviour, permanently. Exclude known-bad periods explicitly, and record which sweeps contributed.

Gating on a metric with no history because it is new. New metrics should publish for several sweeps before gating; a gate on a metric nobody has watched will fire on its own noise and be blamed on the release.

One allowance applied to every metric. Two points of regression is generous for a stable metric and meaningless for a noisy one. Derive per metric, from that metric’s own spread.

Thresholds that only ever go down. Each override or recalibration lowers the bar slightly and nothing raises it, so after a year the gate is below anything that could fail. Recalibrate upward too when the baseline improves, and record when you last did.

A gate on an aggregate rather than a family. A blended score can pass while one family regresses badly. Gate per family and per region, which is also what makes a failure actionable.

Overrides that become routine. A gate overridden every release is not protecting anything; it is generating a ritual. Review the override log periodically and either fix the underlying regression or retire the gate deliberately.

Frequently Asked Questions

Related

• Up to the parent topic: Evaluation and Benchmarking for Spatial LLMs
• Building Regression Test Harnesses for Spatial Agents
• Measuring Spatial IoU for LLM-Generated Geometries
• Related technique: Tuning Fusion Weights for Toponym-Heavy Queries