Chunk-Boundary Strategies for Spatial Corpora

Geometry-rich documents break naive chunkers. A fixed-size splitter that cuts a corpus every 800 tokens will sever a polygon mid-coordinate, orphan a…

Geometry-rich documents break naive chunkers. A fixed-size splitter that cuts a corpus every 800 tokens will sever a polygon mid-coordinate, orphan a feature from the CRS header that gives its numbers meaning, or divorce a geometry from the attributes that describe it. Chunk-boundary strategy for spatial corpora is the discipline of placing splits where they preserve meaning — never inside a geometry, never between a feature and its context.

This area belongs to Geospatial RAG Pipelines and targets a specific retrieval failure: a chunk that contains half a WKT polygon or a coordinate block with no reference frame retrieves and embeds as noise, then poisons downstream reasoning with truncated geometry. The strategies here keep each feature atomic and CRS-anchored, so retrieval returns spans a model can actually parse. They lean on geometry tokenization strategies for how geometry becomes tokens and on context-window optimization for maps for how much of it a chunk can afford to carry.

Geometry-aware chunk boundary placement A spatial document is scanned for geometry spans, split only at safe boundaries between whole features, each chunk is stamped with the CRS header, guarded against a token budget, and emitted for indexing. Detectgeometry spans Split at safeboundaries Stamp CRSheader Tokenbudget guard Emitchunk Whole features, CRS-anchored, within budget Fallback: simplify oversized feature · flag · isolate in its own chunk

Foundational Principles

A geometry is indivisible. A WKT POLYGON(...) or a GeoJSON Feature is the atomic unit; a split may fall between features but never inside one. A half-polygon is not a smaller polygon — it is malformed text that embeds as garbage and can crash a parser downstream.

CRS travels with every chunk. Coordinates without a reference frame are meaningless numbers. Each emitted chunk carries the document’s CRS header in its metadata so a retrieved fragment is self-describing, consistent with the anchoring enforced by CRS normalization.

The token budget is a hard ceiling with a defined escape. Chunks must fit the embedding model’s window, but the budget can never justify severing a feature. When a single feature exceeds the budget, the strategy simplifies or isolates it — it does not cut it.

Step-by-Step Implementation Pipeline

1. Detect geometry spans before choosing any boundary

The first pass locates the byte ranges occupied by WKT and GeoJSON geometries so later steps treat them as no-cut zones. Detection must be conservative: a false negative severs a geometry, so ambiguous spans are treated as geometry. The token accounting here feeds the budget logic described in context-window optimization for maps.

import re
from dataclasses import dataclass

# Matches WKT primitives and GeoJSON geometry objects at a coarse span level.
_WKT = re.compile(r"\b(?:POINT|LINESTRING|POLYGON|MULTIPOLYGON|GEOMETRYCOLLECTION)\s*\([^;]*?\)",
                  re.IGNORECASE | re.DOTALL)
_GEOJSON = re.compile(r'\{[^{}]*"type"\s*:\s*"(?:Polygon|MultiPolygon|Feature)"[^{}]*\}', re.DOTALL)

@dataclass(frozen=True)
class Span:
    start: int
    end: int

def detect_geometry_spans(text: str) -> list[Span]:
    """Return sorted, merged no-cut spans; on regex failure, protect the whole doc."""
    try:
        spans = [Span(m.start(), m.end()) for m in _WKT.finditer(text)]
        spans += [Span(m.start(), m.end()) for m in _GEOJSON.finditer(text)]
    except Exception as exc:                       # pathological input, catastrophic backtracking
        return [Span(0, len(text))]                # conservative: forbid all interior cuts
    spans.sort(key=lambda s: s.start)
    merged: list[Span] = []
    for s in spans:
        if merged and s.start <= merged[-1].end:
            merged[-1] = Span(merged[-1].start, max(merged[-1].end, s.end))
        else:
            merged.append(s)
    return merged

2. Place boundaries only at safe offsets

Candidate split points are the paragraph breaks that fall outside every geometry span. The splitter walks the document accumulating a token count and cuts at the last safe offset before the budget is reached. The mechanics of keeping each feature whole are developed fully in splitting polygon-heavy documents without severing geometries.

def is_safe_offset(offset: int, spans: list[Span]) -> bool:
    """A cut is safe only if it lies strictly outside every geometry span."""
    return not any(s.start < offset < s.end for s in spans)

def safe_boundaries(text: str, spans: list[Span]) -> list[int]:
    candidates = [m.start() for m in re.finditer(r"\n\s*\n", text)]
    safe = [c for c in candidates if is_safe_offset(c, spans)]
    if not safe:                                   # no natural break survived
        safe = [len(text)]                         # emit the document whole
    return safe

3. Stamp CRS metadata and guard the budget

Each chunk is emitted with the source CRS attached and its token count verified. When a chunk containing a single indivisible feature still exceeds the budget, the guard routes it to the simplify-or-isolate fallback rather than forcing a cut.

def emit_chunk(body: str, crs: str, count_tokens, budget: int) -> dict:
    tokens = count_tokens(body)
    over_budget = tokens > budget
    return {
        "text": body.strip(),
        "crs": crs,                                # self-describing chunk
        "token_count": tokens,
        "needs_fallback": over_budget,             # triggers simplify/isolate
    }

Failure Modes & Root Causes

Severed geometry. A boundary lands inside a coordinate list, emitting a truncated ring. Root cause: chunking on token count without a no-cut map. Mitigation: the span detection in step 1 gated by is_safe_offset.

Orphaned CRS. The header declaring the reference frame lands in chunk one; the geometries land in chunk five with no frame. Root cause: treating the CRS header as ordinary prose. Mitigation: stamp CRS onto every chunk in step 3.

Attribute divorce. A feature’s geometry and its attribute table split into different chunks, so retrieval returns coordinates with no description. Root cause: boundaries that ignore feature grouping. Mitigation: extend geometry spans to include the adjacent attribute block before choosing a boundary.

Budget-forced cut. An oversized feature tempts a mid-geometry split to satisfy the window. Root cause: treating the budget as absolute. Mitigation: the simplify-or-isolate fallback, never a cut.

Production Validation Protocols

  1. Parse every geometry per chunk. In CI, load each emitted chunk’s geometries with a WKT/GeoJSON reader; any parse failure fails the build — it proves a severed feature escaped.
  2. CRS presence assertion. Assert every chunk carries a non-empty crs field; a missing frame is a hard error.
  3. Budget-with-escape check. Assert token_count <= budget OR needs_fallback is True; no chunk may quietly exceed the window without flagging.
  4. Round-trip feature count. Sum features across all chunks and assert it equals the source count — no feature dropped, none duplicated by overlap.
  5. Reconstruction test. Concatenating chunks in order (minus overlap) must reproduce the source geometry set byte-for-byte.