Evaluation and Benchmarking for Spatial LLMs

A spatial model that produces fluent, well-formed geometry can still be wrong in ways no language metric detects: a polygon in the right shape but the wrong place, a coordinate invented from nothing, a boundary that crosses itself. This guide, part of the Spatial LLM Architecture & Core Concepts area, defines the correctness signals that actually matter for spatial agents and shows how to wire them into a repeatable harness that gates deployments.

Foundational Principles

Fluency is not correctness. Perplexity, BLEU, and answer-formatting checks say nothing about whether a geometry lands on the real feature. Spatial evaluation must compare against ground-truth geometry with geometric operators, not string overlap. A model that routes uncertain cases through fallback routing for geospatial queries should be scored on what it returns, including its abstentions.
Every metric names the failure it catches. Placement error, coordinate invention, and topological illegality are distinct pathologies with distinct fixes. Reporting one aggregate score hides which one is regressing. Track spatial IoU for placement, a hallucination rate for invention, and a topology-violation rate for legality as separate, independently gated numbers.
The harness is deterministic and pinned. The same predictions against the same truth set must yield identical scores on every run and every machine. That means pinned GEOS and PROJ versions, a fixed CRS for area computation, and repair steps that are themselves deterministic. Non-reproducible scores make regressions unfalsifiable, which is worse than having no harness.

Step-by-Step Implementation Pipeline

1. Pair predictions to ground truth and repair validity

Before any metric, align each prediction to its truth geometry and make both valid. An LLM-emitted polygon frequently self-intersects; scoring it raw either crashes the operator or silently returns a misleading area. Repair once, deterministically, and record that a repair happened so it can be audited. Precise IoU computation is covered in measuring spatial IoU for LLM-generated geometries.

import logging
from shapely.geometry.base import BaseGeometry
from shapely.validation import make_valid
from shapely.errors import GEOSException

logger = logging.getLogger("spatial_eval")


def prepare(geom: BaseGeometry) -> BaseGeometry | None:
    """Return a valid geometry, or None if it cannot be repaired."""
    if geom is None or geom.is_empty:
        return None
    if geom.is_valid:
        return geom
    try:
        repaired = make_valid(geom)
        return repaired if not repaired.is_empty else None
    except GEOSException as exc:
        logger.warning("make_valid failed: %s", exc)
        return None

2. Score placement with spatial IoU

Intersection-over-union quantifies how much of the predicted footprint coincides with the truth footprint. It is defined as

\mathrm{IoU} = \frac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}

for predicted area $A$ and truth area $B$ . Compute it in an equal-area CRS so the units are honest, and return 0.0 on an empty union rather than dividing by zero.

def spatial_iou(pred: BaseGeometry, truth: BaseGeometry) -> float:
    """IoU in the caller's equal-area CRS; 0.0 on any degenerate case."""
    p, t = prepare(pred), prepare(truth)
    if p is None or t is None:
        return 0.0
    try:
        inter = p.intersection(t).area
        union = p.union(t).area
    except GEOSException as exc:
        logger.warning("IoU overlay failed: %s", exc)
        return 0.0
    return inter / union if union > 0.0 else 0.0

3. Count topology violations across the set

Legality is a separate axis from placement. Count self-intersections, unclosed rings, and other invalidities across the prediction set so a model that scores well on IoU but emits illegal geometry is still flagged. Feed embedding-level regressions back through spatial embedding models when the violation rate correlates with a specific geometry family.

def evaluate_batch(pairs: list[tuple[BaseGeometry, BaseGeometry]],
                   iou_floor: float = 0.5) -> dict:
    """Aggregate IoU and topology-violation rate over prediction/truth pairs."""
    if not pairs:
        return {"n": 0, "mean_iou": 0.0, "topology_violation_rate": 0.0, "pass": False}

    ious, violations = [], 0
    for pred, truth in pairs:
        try:
            if pred is None or not pred.is_valid:
                violations += 1
            ious.append(spatial_iou(pred, truth))
        except GEOSException as exc:
            logger.error("pair evaluation failed, scoring as miss: %s", exc)
            violations += 1
            ious.append(0.0)                       # deterministic: a failure is a zero

    n = len(pairs)
    mean_iou = sum(ious) / n
    viol_rate = violations / n
    return {
        "n": n,
        "mean_iou": mean_iou,
        "topology_violation_rate": viol_rate,
        "pass": mean_iou >= iou_floor and viol_rate == 0.0,
    }

Failure Modes & Root Causes

Area measured in degrees. Computing IoU on geometry stored in EPSG:4326 makes area a meaningless product of degrees, and the score varies with latitude. Root cause: no reprojection to an equal-area CRS before the overlay.

Invalid geometry scored as a crash or a zero-by-accident. An unrepaired self-intersecting polygon can throw inside intersection, aborting the whole batch, or return an empty result that reads as a legitimate miss. Root cause: skipping the make_valid gate and conflating a repair failure with a genuine low score.

Hallucinations invisible to IoU. A confidently invented geometry over open ocean may still overlap a truth polygon partially and post a non-zero IoU. Root cause: relying on placement overlap alone without a dedicated invention check, covered under detecting hallucinated coordinates below.

Regression masked by aggregation. A rising topology-violation rate hides behind a stable mean IoU when reported as one blended number. Root cause: collapsing independent failure axes into a single score.

Production Validation Protocols

Reproject predictions and truth to a fixed equal-area CRS before any area computation, and assert the CRS in the harness.
Gate three numbers independently: mean IoU above a floor, topology-violation rate at zero, and coordinate-hallucination rate below a ceiling.
Pin GEOS and PROJ versions in the evaluation container so scores are identical across machines.
Store per-pair results, not just aggregates, so a regression can be traced to the offending predictions.
Run the harness on every model or prompt change in CI and block the deployment on any gate failure, using a stable, versioned truth corpus.