Evaluation and Benchmarking for Spatial LLMs

General language benchmarks say nothing useful about whether a system puts things in the right place. A spatial agent can score well on every text metric available and still put a building in the wrong county, because the failure is geometric and the metric was lexical. This topic builds the evaluation that measures what actually breaks: where the geometry landed, whether the coordinates were real, whether the relations hold, and whether the system refused when it should have.

It belongs to spatial LLM architecture and core concepts and is the gate on everything else here — a normalization rule, a tokenization policy or a routing change is only an improvement if something measures it. Its measurements depend on the verification machinery in spatial reasoning and relation inference, which supplies the ground truth for relation scoring.

Foundational Principles

Every score needs a defined degenerate case. Geometry evaluation is full of inputs that have no natural score: disjoint shapes, empty unions, unparseable output. Decide what each one scores before running anything, and make the decision visible in the code rather than emerging from an exception handler.

Parse failures are results, not exclusions. Dropping output the harness could not read flatters the model exactly where it is weakest. Score an unparseable answer as a failure and report the parse rate separately, so two systems with identical scores over their parseable outputs are still distinguishable.

A refusal can be correct. An agent that declines a question it lacks the data to answer has behaved well, and an evaluation that scores every non-answer as a miss will reward a system that guesses. Build unanswerable cases into the set deliberately and score refusal on them as a success.

Step-by-Step Implementation Pipeline

1. Build a case set that spans the failure families

The set is the asset; the metrics are derived from it. Aim for coverage of each family rather than volume, and label each case with the family it exercises so the report can be broken down.

import logging
from dataclasses import dataclass
from typing import Optional

log = logging.getLogger("spatial_eval")


@dataclass(frozen=True)
class Case:
    case_id: str
    family: str                # placement | plausibility | relations | refusal
    question: str
    truth_geometry: Optional[object] = None
    truth_relations: tuple[tuple[str, str, str], ...] = ()
    answerable: bool = True
    region: Optional[str] = None

Include the region on every case. Spatial systems fail unevenly across geography — a model that handles one country well may be hopeless in another — and an aggregate score that mixes regions will report a competent system as mediocre and a regionally broken one as fine.

2. Score placement with a robust overlap ratio

Placement is measured by the overlap between predicted and true footprints, computed in an equal-area projection and defined on every degenerate input. The implementation, including repair and clamping, is in measuring spatial overlap for model-generated geometries.

def score_placement(predicted, truth, spatial_iou) -> tuple[float, str]:
    """Overlap ratio with an explicit note for every degenerate case."""
    if predicted is None:
        return 0.0, "no geometry produced"
    try:
        value = spatial_iou(predicted, truth)
    except Exception as exc:                       # the scorer itself must not abort a sweep
        log.warning("scorer failed: %s", exc)
        return 0.0, f"scorer error: {exc}"
    if value == 0.0:
        return 0.0, "disjoint or unusable"
    return value, ""

3. Screen for implausible coordinates before scoring anything

A coordinate in the wrong hemisphere scores zero on overlap and tells you nothing about how wrong it was. Screening separates “produced a bad shape” from “produced a fictional place”, which are different problems with different fixes. The screening rules are in detecting hallucinated coordinates in model output.

def screen_plausibility(geom, expected_region_bbox) -> tuple[bool, str]:
    """Cheap checks that catch fiction before expensive geometry runs."""
    if geom is None or geom.is_empty:
        return False, "no geometry"
    x, y = geom.representative_point().x, geom.representative_point().y
    if not (-180.0 <= x <= 180.0 and -90.0 <= y <= 90.0):
        return False, "outside the coordinate domain"
    if abs(x) < 1e-9 and abs(y) < 1e-9:
        return False, "null island"
    w, s, e, n = expected_region_bbox
    if not (w - 1 <= x <= e + 1 and s - 1 <= y <= n + 1):
        return False, "outside the expected region"
    return True, ""

The null-island check earns its place. A coordinate pair of exactly zero is almost always a parsing failure or a missing value rendered as a number, and it is common enough to be worth naming rather than letting it score as an ordinary miss thousands of kilometres away.

4. Score relations against real predicates

Relation scoring compares the relations an answer asserted against the relations that actually hold, computed over the truth geometry. Precision and recall are both meaningful here and measure different failures: precision falls when the system asserts relations that do not hold, recall falls when it misses ones that do.

def score_relations(asserted: set, truth: set) -> dict:
    """Precision and recall over relation triples, with the disagreements listed."""
    if not truth and not asserted:
        return {"precision": 1.0, "recall": 1.0, "false": [], "missed": []}
    false_positives = sorted(asserted - truth)
    missed = sorted(truth - asserted)
    correct = len(asserted & truth)
    precision = correct / len(asserted) if asserted else 1.0
    recall = correct / len(truth) if truth else 1.0
    return {"precision": round(precision, 4), "recall": round(recall, 4),
            "false": false_positives, "missed": missed}

Returning the actual disagreements rather than only the numbers is what makes this metric usable. A recall of 0.6 is a number; a list of the four relations that were missed is a bug report.

5. Score refusal on the cases that cannot be answered

Refusal is scored on a separate axis with its own truth: a case marked unanswerable expects a decline, and a case marked answerable expects an attempt. Scoring them together with placement produces a metric that improves when the system becomes more reckless.

def score_refusal(answered: bool, answerable: bool) -> tuple[float, str]:
    """Refusing an unanswerable case is a success, not a miss."""
    if answerable and answered:
        return 1.0, ""
    if answerable and not answered:
        return 0.0, "declined a question it had the data for"
    if not answerable and not answered:
        return 1.0, "correctly declined"
    return 0.0, "answered a question it had no basis for"

The last branch deserves the heaviest weight in any report a reader will act on. An unnecessary refusal costs usefulness and is visible to the user, who can rephrase; an unfounded answer costs trust and is invisible, because it looks exactly like a good one.

6. Report per family and per region, never as one number

The report is a table, not a scalar. A single blended score can improve while two of its components regress, and the blend weights encode a judgement nobody wrote down.

from collections import defaultdict


def summarise(results: list[dict]) -> dict:
    """Break results down by family and region; no aggregate is produced."""
    buckets = defaultdict(list)
    for r in results:
        buckets[(r["family"], r.get("region", "all"))].append(r)
    report = {}
    for (family, region), rows in sorted(buckets.items()):
        scores = [r["score"] for r in rows]
        scores.sort()
        report[f"{family}/{region}"] = {
            "n": len(rows),
            "median": scores[len(scores) // 2],
            "p10": scores[max(0, int(0.1 * len(scores)) - 1)],
            "parse_rate": round(sum(r["parsed"] for r in rows) / len(rows), 4),
        }
    return report

Reporting the tenth percentile alongside the median is deliberate. Spatial quality is bimodal — most answers are close and a few are in another country — and a mean sits between two peaks describing neither. The percentile is what a release gate should read.

7. Fix the harness before fixing the model

An evaluation harness has bugs, and its bugs look exactly like model regressions. Before acting on a score change, verify that the truth data has not moved, the library versions have not changed, and the scoring code produces identical results on a frozen fixture.

FROZEN_EXPECTED = {"case-001": 0.8734, "case-014": 0.0, "case-102": 1.0}


def harness_self_check(score_case) -> None:
    """Fail loudly if the harness itself has changed behaviour."""
    for case_id, expected in FROZEN_EXPECTED.items():
        got = round(score_case(case_id), 4)
        if abs(got - expected) > 1e-4:
            raise AssertionError(
                f"harness drift on {case_id}: expected {expected}, got {got}; "
                "check library versions and truth data before blaming the model")

8. Set thresholds from the distribution, not from a round number

A release gate needs a number, and the number should come from observed behaviour rather than from taste. The procedure and its pitfalls are in setting release thresholds for spatial agents.

def propose_threshold(historical: list[float], allowed_regression: float = 0.02) -> float:
    """A gate just below current behaviour, so noise does not block a release."""
    if len(historical) < 5:
        raise ValueError("not enough history to propose a threshold")
    ordered = sorted(historical)
    baseline = ordered[len(ordered) // 2]
    return round(max(0.0, baseline - allowed_regression), 4)

9. Wire the gate into the release, with an override that is recorded

A gate nobody can override will be removed the first time it blocks an urgent fix. A gate whose override is silent is not a gate. The workable arrangement is an override that requires a reason and appears in the release record.

def release_gate(report: dict, thresholds: dict, override_reason: str | None) -> bool:
    failures = [k for k, v in report.items()
                if k in thresholds and v["p10"] < thresholds[k]]
    if not failures:
        return True
    if override_reason:
        log.warning("release gate overridden for %s: %s", ", ".join(failures), override_reason)
        return True
    log.error("release blocked by %s", ", ".join(failures))
    return False

Operating This Stage Over Time

Evaluation sets rot in a specific way: they stop being hard. Cases that were failing get fixed, the set fills with cases the system passes, and the score rises toward one while real-world quality plateaus. The remedy is to add cases from production failures continuously and to retire nothing — a case that has passed for a year costs a second to run and is the thing that will catch a regression when someone rewrites the tokenizer.

Truth data is the other durable problem. Geometry truth ages: parcels are resurveyed, boundaries are corrected, and a case labelled three years ago may now disagree with reality rather than with the model. Version the truth alongside the cases, record which version a score was measured against, and treat a truth correction as an event that invalidates historical comparisons rather than one that silently changes them.

Library versions belong in the same category. A geometry engine upgrade can move overlap scores by a percent or two through changes in repair behaviour, which is indistinguishable from a model regression unless the versions are pinned and recorded. The self-check in step 6 exists precisely to make that distinguishable, and it should run before every sweep rather than occasionally.

Finally, resist the pull toward a single headline number. It will be asked for, repeatedly, and the honest response is a small table — four families, a median and a tenth percentile each. The reason is not pedantry: the families fail independently and are fixed by different work, and a blended score is a decision about their relative importance that is better made explicitly, in a release conversation, than implicitly, in a weighting constant.

A last note on culture rather than code. Evaluation only changes behaviour when its output is read by the people making decisions, which means the report has to be short enough to read and stable enough to compare. Four families, two statistics each, a parse rate and a region breakdown fits on a screen; a hundred-metric dashboard does not get read and therefore does not gate anything, however carefully it was built.

Failure Modes & Root Causes

The flattering filter. Unparseable outputs are excluded and the score rises. Root cause: treating a parse failure as a harness problem rather than a result. Mitigation: score it zero, report the parse rate separately.

The guessing reward. Every refusal counts as a miss, so the highest-scoring system is the one that never declines. Root cause: no unanswerable cases in the set. Mitigation: build them in and score refusal as success on them.

The regional average. A system that works in one region and fails in another reports as uniformly mediocre. Root cause: aggregation across regions. Mitigation: region on every case, breakdown in every report.

The harness regression. A library upgrade moves scores and the team spends a week investigating the model. Root cause: unpinned dependencies and no self-check. Mitigation: frozen fixtures with expected values, checked before every sweep.

Production Validation Protocols

1. Harness self-check first. Run the frozen fixture check before any sweep; a drift there invalidates everything downstream.
2. Parse-rate reporting. Publish the share of outputs that parsed, per family; a score without it is not comparable across runs.
3. Refusal scoring. Assert the set contains unanswerable cases and that refusal on them scores as success.
4. Region breakdown. Assert every case carries a region and every report is broken down by it.
5. Percentile gate. Gate releases on the tenth percentile rather than the mean, and record the threshold with the release.
6. Override audit. Assert that any gate override carries a recorded reason, and review overrides periodically — a gate overridden every release is a gate that needs rethinking, not repeating.

Frequently Asked Questions

Related

• Up to the section overview: Spatial LLM Architecture and Core Concepts
• Technique: Measuring Spatial Overlap for Model-Generated Geometries
• Technique: Detecting Hallucinated Coordinates in LLM Output
• Technique: Building Regression Test Harnesses for Spatial Agents
• Technique: Setting Release Thresholds for Spatial Agents
• Peer topic: Spatial Reasoning and Relation Inference