Carrying Frame and Extent Metadata Into Every Chunk

A chunk is retrieved alone. Whatever context sat around it in the source document — the header declaring the projection, the caption naming the survey, the sentence explaining the units — is gone by the time a model reads it. Metadata is how that context survives the trip. This guide covers computing and attaching the four fields that make a spatial chunk self-describing, as the final stage of chunk-boundary strategies for spatial corpora.

When to Use This Approach

Always, for any corpus containing coordinates. The only real decision is how much to compute rather than copy, and that decision turns on whether the source can be trusted.

The last row is the one people skip. A document’s declared extent describes what the author thought the file contained, often before an edit removed half of it, and a chunk’s extent must describe what the chunk actually holds — otherwise the spatial pre-filter in spatial context retrieval and reranking is filtering on fiction.

Implementation

The function parses each chunk’s geometries, computes the true extent from them, verifies the declared frame against that extent, and returns a record that carries everything a downstream filter or citation layer needs.

import logging
from dataclasses import dataclass, asdict
from typing import Iterable, Optional

from shapely import wkt
from shapely.errors import GEOSException
from shapely.geometry.base import BaseGeometry
from shapely.ops import unary_union
from pyproj import CRS
from pyproj.exceptions import CRSError

log = logging.getLogger("chunk_metadata")


@dataclass(frozen=True)
class ChunkMeta:
    epsg: int
    bbox: tuple[float, float, float, float]
    feature_ids: tuple[str, ...]
    geometry_count: int
    frame_verified: bool          # False means the frame is copied, not confirmed
    notes: tuple[str, ...]


def _parse_all(literals: Iterable[str]) -> list[BaseGeometry]:
    """Parse what parses; log and skip what does not, never abort the chunk."""
    out = []
    for lit in literals:
        try:
            geom = wkt.loads(lit)
        except (GEOSException, ValueError) as exc:
            log.warning("unparseable geometry skipped during metadata build: %s", exc)
            continue
        if not geom.is_empty:
            out.append(geom)
    return out


def _frame_plausible(epsg: int, bbox) -> tuple[bool, Optional[str]]:
    """Check the extent sits inside the frame's declared area of use."""
    try:
        crs = CRS.from_epsg(epsg)
    except CRSError as exc:
        return False, f"frame will not construct: {exc}"
    if crs.area_of_use is None:
        return False, "frame declares no area of use"
    if not crs.is_geographic:
        return False, "projected frame — extent check needs a transform first"
    w, s, e, n = crs.area_of_use.bounds
    inside = (w - 1 <= bbox[0] and bbox[2] <= e + 1
              and s - 1 <= bbox[1] and bbox[3] <= n + 1)
    return inside, None if inside else "extent falls outside the frame's area of use"


def build_chunk_meta(
    geometry_literals: Iterable[str],
    feature_ids: Iterable[str],
    declared_epsg: int,
) -> ChunkMeta:
    """Compute self-describing metadata for one chunk. Never raises on data quality."""
    geoms = _parse_all(geometry_literals)
    ids = tuple(str(i) for i in feature_ids)
    notes: list[str] = []

    if not geoms:
        # Deterministic fallback: a chunk with no usable geometry still gets a record,
        # with a degenerate extent that no bounding-box filter will match by accident.
        notes.append("no parseable geometry in chunk")
        return ChunkMeta(declared_epsg, (0.0, 0.0, 0.0, 0.0), ids, 0, False, tuple(notes))

    try:
        bounds = unary_union(geoms).bounds
    except GEOSException as exc:
        log.warning("union failed, falling back to per-geometry bounds: %s", exc)
        xs = [b for g in geoms for b in (g.bounds[0], g.bounds[2])]
        ys = [b for g in geoms for b in (g.bounds[1], g.bounds[3])]
        bounds = (min(xs), min(ys), max(xs), max(ys))

    verified, problem = _frame_plausible(declared_epsg, bounds)
    if problem:
        notes.append(problem)

    return ChunkMeta(declared_epsg, tuple(round(v, 7) for v in bounds),
                     ids, len(geoms), verified, tuple(notes))

The degenerate extent for a geometry-free chunk is a deliberate choice over None. A null extent forces every consumer to branch, and one of them will forget; a zero-area extent at the origin simply never intersects a real query region, which fails safe. The notes field carries the reason so the chunk is not mistaken for a coding error.

The union-failure fallback exists because unary_union is the one call here that can fail on geometry that individually parsed fine — self-intersecting rings that survive wkt.loads and confound the overlay. Taking the envelope of individual bounds gives the same answer for well-behaved input and a slightly loose answer for the pathological case, which is the right direction to be wrong in.

Validation & Testing

def test_extent_contains_every_geometry():
    meta = build_chunk_meta(LITERALS, IDS, 4326)
    for lit in LITERALS:
        g = wkt.loads(lit)
        assert g.bounds[0] >= meta.bbox[0] - 1e-6
        assert g.bounds[2] <= meta.bbox[2] + 1e-6


def test_geometryless_chunk_gets_a_degenerate_extent():
    meta = build_chunk_meta([], ["f-1"], 4326)
    assert meta.geometry_count == 0
    assert meta.bbox == (0.0, 0.0, 0.0, 0.0)
    assert "no parseable geometry" in " ".join(meta.notes)


def test_implausible_frame_is_flagged_not_rejected():
    # British coordinates declared as a Japanese frame: usable record, honest flag.
    meta = build_chunk_meta(["POINT(-3.19 55.95)"], ["f-2"], 6668)
    assert meta.frame_verified is False and meta.notes

The first test is a containment property and should be run over the whole corpus during an index build, not just over a fixture; it is cheap and it is the assertion that catches an off-by-one in extent rounding. Rounding to seven decimal places is roughly a centimetre, comfortably below any real query tolerance, and it keeps the metadata compact — but rounding inward would break containment, which is why the rounding happens after the bounds are taken and is tested for.

Gotchas & Edge Cases

Antimeridian-crossing extents. A chunk holding features either side of 180 degrees longitude produces an extent spanning almost the whole world, and every bounding-box filter then matches it. Detect the case — a longitude span above 180 degrees — and store two extents or a flag, rather than one meaningless rectangle.

Rounding that shrinks the box. Truncating rather than rounding, or rounding both corners the same way, can pull the maximum corner inside a geometry’s true bound. Round the minimum down and the maximum up if you round at all; the containment test above is what catches this.

Frames that are projected. The plausibility check short-circuits for projected frames because comparing metres against a degree-based area of use is meaningless. Transform a corner into geographic coordinates first if you want the check to apply, and be explicit that an unverified frame is not a wrong one.

Identifiers that are not stable. Feature identifiers assigned by the chunker — index positions, hashes of coordinate text — change whenever the document is re-exported, which breaks every citation made against the previous build. Prefer identifiers from the source, and where none exist, derive one from a stable attribute rather than from position.

Metadata that outgrows the chunk. A chunk holding two hundred features accumulates two hundred identifiers, which can rival the text in size and inflate storage across a large corpus. Cap the stored list and record the total count alongside it, so citation still works for small chunks and the count remains honest for large ones.

Frequently Asked Questions

Related

• Up to the parent topic: Chunk-Boundary Strategies for Spatial Corpora
• Splitting Polygon-Heavy Documents Without Severing Geometries
• Concept: Coordinate Reference System Normalization
• Related topic: Spatial Metadata and Catalog Indexing