Measuring Spatial IoU for LLM-Generated Geometries

Intersection-over-union is the workhorse metric for asking whether a model put a shape in the right place, but model-generated geometry breaks the naive implementation: it self-intersects, arrives in the wrong projection, and occasionally has no overlap at all. This guide builds a robust IoU function that repairs invalid input, measures area in honest units, and returns a defined value on every degenerate case — the scoring primitive the rest of evaluation and benchmarking for spatial LLMs is built on. It belongs at the scoring stage of the evaluation harness, after generation and before any threshold is applied.

When to Use This Approach

Use spatial IoU whenever the ground truth is a footprint and you care both how well the prediction covers it and how much it over-claims beyond it: extracted parcels, delineated hazard zones, service areas, building outlines. The metric is

$$\mathrm{IoU} = \frac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}$$

for predicted footprint $A$ and truth footprint $B$, ranging from 0.0 (disjoint) to 1.0 (identical). Its virtue is that the denominator grows when the model draws too much, so a prediction that swallows the whole city cannot score well by containing the right building. Reach for a different metric when the truth is a point or a line, where both areas collapse to zero and a distance measure carries the information instead.

IoU is also the wrong tool when the task is classification dressed up as geometry. If the model is choosing among a fixed set of pre-existing polygons — census tracts, administrative units, delivery zones — then the answer is a label, and a confusion matrix over those labels tells you more than an area ratio ever will. Reserve IoU for the case where the model actually drew the boundary.

Implementation

The function projects both geometries to a caller-supplied equal-area projection, repairs invalidity with make_valid, and treats an empty union as a clean 0.0 rather than a division error. It never raises on data quality; a geometry it cannot use scores zero and is logged. That distinction — programmer error raises, data error scores — is what lets the harness run a thousand cases unattended without a single bad prediction aborting the sweep.

import logging
from typing import Optional

from shapely.geometry.base import BaseGeometry
from shapely.validation import make_valid
from shapely.ops import transform
from shapely.errors import GEOSException
from pyproj import Transformer, CRS

logger = logging.getLogger("spatial_iou")


class IoUInputError(ValueError):
    """Raised only for programmer error, never for bad geometry data."""


def _to_equal_area(geom: BaseGeometry, src_epsg: int, area_epsg: int) -> Optional[BaseGeometry]:
    """Reproject into an equal-area projection so that .area is in real square metres."""
    try:
        if src_epsg == area_epsg:
            return geom
        transformer = Transformer.from_crs(
            CRS.from_epsg(src_epsg), CRS.from_epsg(area_epsg), always_xy=True
        )
        return transform(transformer.transform, geom)
    except Exception as exc:                        # malformed CRS, pyproj failure
        logger.warning("reprojection %s->%s failed: %s", src_epsg, area_epsg, exc)
        return None


def _validate(geom: BaseGeometry) -> Optional[BaseGeometry]:
    """Repair an invalid geometry; return None if it is unusable."""
    if geom is None or geom.is_empty:
        return None
    if geom.is_valid:
        return geom
    try:
        fixed = make_valid(geom)              # deterministic repair
        return fixed if not fixed.is_empty else None
    except GEOSException as exc:
        logger.warning("make_valid guard failed: %s", exc)
        return None


def spatial_iou(
    predicted: BaseGeometry,
    truth: BaseGeometry,
    src_epsg: int = 4326,
    area_epsg: int = 6933,          # World Equal-Area Cylindrical (metres)
) -> float:
    """Return IoU in [0, 1]. Never crashes; returns 0.0 on any degenerate case."""
    if not isinstance(src_epsg, int) or not isinstance(area_epsg, int):
        raise IoUInputError("EPSG codes must be integers")

    pred = _validate(predicted)
    tru = _validate(truth)
    if pred is None or tru is None:
        return 0.0

    pred = _to_equal_area(pred, src_epsg, area_epsg)
    tru = _to_equal_area(tru, src_epsg, area_epsg)
    if pred is None or tru is None:
        return 0.0

    try:
        intersection_area = pred.intersection(tru).area
        union_area = pred.union(tru).area
    except GEOSException as exc:
        logger.warning("overlay failed, scoring as no-overlap: %s", exc)
        return 0.0

    # Fallback: an empty union means both footprints vanished under repair;
    # define IoU as 0.0 rather than dividing by zero.
    if union_area <= 0.0:
        return 0.0

    iou = intersection_area / union_area
    return max(0.0, min(1.0, iou))     # clamp against floating-point overshoot

Four things are load-bearing in that order of operations. Repair runs first, because an invalid ring makes every subsequent area meaningless rather than merely wrong. Reprojection runs second, because repairing after a projection change can move vertices that the repair had just reconciled. The overlay is wrapped, because GEOS raises on inputs that survived make_valid but still confound the noding step. And the clamp runs last, because the ratio of two floating-point areas is not guaranteed to land inside the closed unit interval even when the geometry is perfect.

Validation & Testing

from shapely.geometry import box, Polygon


def test_identical_footprints_score_one():
    a = box(0, 0, 10, 10)
    assert abs(spatial_iou(a, a, src_epsg=6933, area_epsg=6933) - 1.0) < 1e-9


def test_disjoint_footprints_score_zero():
    a, b = box(0, 0, 1, 1), box(50, 50, 51, 51)
    assert spatial_iou(a, b, src_epsg=6933, area_epsg=6933) == 0.0


def test_self_intersecting_prediction_is_repaired_not_crashed():
    bowtie = Polygon([(0, 0), (2, 2), (2, 0), (0, 2), (0, 0)])  # invalid
    truth = box(0, 0, 2, 2)
    iou = spatial_iou(bowtie, truth, src_epsg=6933, area_epsg=6933)
    assert 0.0 <= iou <= 1.0            # defined, in range, no exception


def test_over_claim_is_penalised_more_than_partial_cover():
    truth = box(0, 0, 10, 10)
    spilled = box(0, 0, 20, 20)         # covers everything, claims four times the area
    clipped = box(0, 0, 10, 5)          # covers half, claims nothing extra
    assert spatial_iou(spilled, truth, 6933, 6933) < spatial_iou(clipped, truth, 6933, 6933)

Run these in CI with pinned GEOS and PROJ versions so the equal-area reprojection and repair produce identical scores across environments. A version bump that changes make_valid’s output on a bow-tie will move scores by a few percent, and without pinning that drift is indistinguishable from a regression in the model itself. Record both library versions alongside every score you publish, the same way you would record the model version.

The fourth test is the one worth defending in review. It encodes the property the metric exists for, not an arithmetic identity, and it fails loudly if somebody “optimises” the implementation into a recall calculation.

Gotchas & Edge Cases

Area measured in degrees. Calling .area on EPSG:4326 geometry returns square degrees, which vary with latitude and make IoU incomparable across regions. The _to_equal_area step exists precisely to avoid this; never skip it because the inputs “look small”. A test suite drawn entirely from one city will not catch the omission, because within one city the distortion is a constant factor that cancels in the ratio.

Empty union from over-aggressive repair. A degenerate zero-width sliver can survive parsing but collapse to empty under make_valid, producing a zero union. The union_area <= 0.0 guard returns 0.0 instead of raising ZeroDivisionError.

Floating-point overshoot above 1.0. Overlay arithmetic can yield an intersection marginally larger than the union at the last decimal. The final min(1.0, ...) clamp keeps the score in range so downstream thresholds behave.

MultiPolygon truth with holes. IoU handles multipart and holed geometry correctly only if both sides are valid; an unrepaired hole becomes a phantom overlap. The _validate gate must run before any area is taken.

Averaging scores across wildly different footprint sizes. A mean IoU over a set containing both a bus shelter and a national park is dominated by whichever cases happen to be easy at that scale. Report the distribution — median plus the tenth percentile — or weight by truth area deliberately, and say which you did.

Frequently Asked Questions

Related

• Up to the parent topic: Evaluation and Benchmarking for Spatial LLMs
• Detecting Hallucinated Coordinates in LLM Output
• Building Regression Test Harnesses for Spatial Agents
• Setting Release Thresholds for Spatial Agents
• Coordinate Reference System Normalization