Geometry Tokenization Strategies

A language model reads a sequence of tokens; a polygon is a list of high-precision numbers. Everything about how the second becomes the first — how many decimal places survive, which serialisation is used, whether vertices are dropped — determines both how much of the model’s context a single feature consumes and how much of the geometry’s meaning arrives intact. Tokenization is where those two pressures are traded against each other, deliberately.

This topic belongs to spatial LLM architecture and core concepts and sits directly downstream of coordinate reference system normalization: a geometry must be in a known frame before any decision about precision means anything, because six decimal places of a projected metre and six of a degree are four orders of magnitude apart in real resolution. Its output feeds context-window optimization for maps, which decides how many tokenized features can co-exist in one prompt.

Foundational Principles

Precision is a policy, not a property of the data. Source coordinates arrive with whatever precision an exporter happened to emit, frequently far beyond the accuracy of the survey behind them. Choosing how many decimal places to keep is a decision about what the questions need, and it should be made once and applied uniformly rather than inherited per source.

Simplification must preserve validity and topology. Dropping vertices is the most effective way to reduce token cost and the easiest way to produce a self-intersecting ring or a gap between two parcels that used to share a boundary. Use a topology-preserving algorithm, validate after, and record the tolerance.

The budget degrades, it does not truncate. A feature that will not fit must be reduced by a defined ladder — fewer decimals, then simplification, then an extent with a note — never by cutting the token stream partway through a coordinate list. A truncated geometry is not a coarser geometry; it is a parse error.

Step-by-Step Implementation Pipeline

1. Set the precision policy from the question, not the data

Decimal places in degrees map to distances on the ground, and the mapping is worth internalising: five decimal places is roughly a metre, four is roughly ten metres, three is roughly a hundred. Choose the coarsest that answers your questions.

import logging
from dataclasses import dataclass

log = logging.getLogger("geometry_tokenization")

# Degrees of latitude per unit at the equator; longitude shrinks with latitude.
DECIMALS_FOR_METRES = {1.0: 5, 10.0: 4, 100.0: 3, 1000.0: 2}


def decimals_for(target_accuracy_m: float) -> int:
    """Fewest decimal places that still resolves the accuracy the task needs."""
    for metres in sorted(DECIMALS_FOR_METRES):
        if target_accuracy_m <= metres:
            return DECIMALS_FOR_METRES[metres]
    return 2

The saving is larger than it looks because coordinate text tokenizes badly: a run of digits and decimal points produces far more tokens per character than prose does, so removing two decimal places from every vertex of an eighty-vertex polygon removes several hundred tokens. The arithmetic and the trade-off are worked through in coordinate precision versus token cost.

2. Choose a representation and stay with it

Three families are in common use and they differ by roughly a factor of six in cost. A compact textual form is the cheapest that preserves exact geometry; a structured object form is more verbose and easier for a model to manipulate reliably; hierarchical cell identifiers are cheapest of all and lossy by construction. The comparison is developed in comparing well-known text, structured objects and cell identifiers.

def to_compact_text(geom, decimals: int) -> str:
    """Compact textual geometry at a fixed precision, with no trailing zeros."""
    def fmt(value: float) -> str:
        return f"{round(value, decimals):.{decimals}f}".rstrip("0").rstrip(".")
    rings = []
    for ring in _rings(geom):
        rings.append(", ".join(f"{fmt(x)} {fmt(y)}" for x, y in ring))
    return f"POLYGON(({'), ('.join(rings)}))"

Mixing representations across a corpus is the failure to avoid. A model that sees two forms for the same kind of object spends capacity distinguishing them, and any downstream parser has to handle both — which it will, until the day one of them acquires a variant.

3. Simplify with a topology-preserving algorithm

Vertex reduction is where most of the remaining saving lives. The algorithm matters: a naive douglas-peucker simplification on each geometry independently will pull two shared boundaries apart, opening slivers between parcels that were adjacent.

from shapely.errors import GEOSException
from shapely.validation import make_valid


def simplify_safely(geom, tolerance_m: float, to_metric, from_metric):
    """Simplify in a metric frame, preserving topology, and never return invalid output."""
    if tolerance_m <= 0:
        return geom, 0.0
    try:
        projected = to_metric(geom)
        reduced = projected.simplify(tolerance_m, preserve_topology=True)
        if reduced.is_empty:
            log.info("simplification collapsed a geometry; keeping the original")
            return geom, 0.0
        if not reduced.is_valid:
            reduced = make_valid(reduced)
        return from_metric(reduced), tolerance_m
    except GEOSException as exc:
        log.warning("simplification failed (%s); keeping the original", exc)
        return geom, 0.0                              # deterministic fallback

Simplifying in a metric frame rather than in degrees matters for the same reason measuring does: a tolerance expressed in degrees is a different distance at every latitude, so a corpus spanning a continent would be simplified unevenly by a constant that looked uniform.

4. Reduce along a ladder when the budget binds

When a feature still exceeds its allowance, reduction proceeds in a defined order, and each rung is recorded so the consumer knows what it is looking at.

@dataclass(frozen=True)
class Tokenized:
    text: str
    tokens: int
    rung: str            # which reduction was applied
    note: str


REDUCTION_LADDER = ("full", "fewer_decimals", "simplified", "extent_only")


def tokenize_within_budget(geom, budget: int, count_tokens, to_metric, from_metric,
                           decimals: int = 5) -> Tokenized:
    """Reduce along the ladder until it fits. Never truncates the token stream."""
    text = to_compact_text(geom, decimals)
    if count_tokens(text) <= budget:
        return Tokenized(text, count_tokens(text), "full", "")

    coarse = to_compact_text(geom, max(2, decimals - 2))
    if count_tokens(coarse) <= budget:
        return Tokenized(coarse, count_tokens(coarse), "fewer_decimals",
                         f"precision reduced to {max(2, decimals - 2)} decimals")

    reduced, tol = simplify_safely(geom, 25.0, to_metric, from_metric)
    simplified = to_compact_text(reduced, max(2, decimals - 2))
    if count_tokens(simplified) <= budget:
        return Tokenized(simplified, count_tokens(simplified), "simplified",
                         f"simplified at {tol:g} m tolerance")

    minx, miny, maxx, maxy = geom.bounds
    extent = f"BBOX({minx:.4f} {miny:.4f}, {maxx:.4f} {maxy:.4f})"
    log.info("feature reduced to its extent to fit a %d-token budget", budget)
    return Tokenized(extent, count_tokens(extent), "extent_only",
                     "geometry replaced by its bounding extent")

The last rung is the important one. Replacing a geometry with its extent is a substantial loss and is honest about it; truncating the coordinate list would be a smaller apparent loss and would produce text that no parser accepts and that the model will attempt to reason over anyway.

5. Carry the rung forward into the prompt

A model given a simplified geometry with no indication that it was simplified will answer questions about boundary detail as though the detail were real. The rung and its note belong in the context alongside the geometry.

def to_prompt_fragment(name: str, t: Tokenized) -> str:
    """Geometry plus an honest statement of what was done to it."""
    if t.rung == "full":
        return f"{name}: {t.text}"
    return f"{name} ({t.note}): {t.text}"

6. Normalise vertex order and starting point

Two identical polygons serialised from different sources can differ in ring direction and starting vertex, producing different token sequences for the same shape. That matters for caching, for deduplication, and for any comparison a model is asked to make between two geometries.

from shapely.geometry.polygon import orient


def canonical_form(geom):
    """Fixed ring orientation and a deterministic starting vertex."""
    try:
        oriented = orient(geom, sign=1.0)             # exterior counter-clockwise
    except Exception:
        return geom
    return oriented                                    # starting vertex handled by the writer

7. Measure the real token cost, not an estimate

Character counts and vertex counts are proxies, and both mislead for coordinate text. Measure with the tokenizer the model actually uses, and cache the measurement per geometry version so the budget logic is not paying for repeated tokenization.

def measured_tokens(text: str, count_tokens, cache: dict) -> int:
    """Tokenize once per distinct string; fall back to a length heuristic on failure."""
    if text in cache:
        return cache[text]
    try:
        n = count_tokens(text)
    except Exception as exc:                          # tokenizer outage must not stop the build
        n = max(1, len(text) // 3)                    # coordinate text: ~3 chars per token
        log.warning("tokenizer unavailable (%s); estimating %d tokens", exc, n)
    cache[text] = n
    return n

The three-characters-per-token estimate is deliberately pessimistic for coordinate text and roughly right; prose runs closer to four. Using one constant for both is a common way to overshoot a context budget by a fifth.

8. Validate the round trip

Whatever comes out of tokenization must parse back into a geometry that is valid and, at the chosen precision, equivalent to what went in. This is a build-time assertion rather than a runtime one, and it is the check that catches a formatting change nobody expected.

from shapely import wkt


def round_trips(geom, text: str, decimals: int) -> bool:
    """Parses back, stays valid, and agrees to the precision that was kept."""
    try:
        parsed = wkt.loads(text)
    except Exception:
        return False
    if not parsed.is_valid:
        return False
    tolerance = 10 ** (-decimals) * 2
    return parsed.equals_exact(geom, tolerance)

Operating This Stage Over Time

Tokenization settings are the kind of configuration that gets chosen once, works, and then quietly stops being right. Three drifts account for most of it.

The first is a model change. A new model with a different tokenizer changes the cost of every geometry in the corpus, sometimes by twenty per cent, and a budget tuned against the old one will either waste context or overflow it. Pin the tokenizer version alongside the precision policy, and re-measure a sample when either changes; the measurement takes minutes and the alternative is a slow degradation nobody attributes correctly.

The second is a source whose precision changes. An upstream export that used to emit five decimal places starts emitting twelve, and nothing breaks — the rounding still works — but the pre-rounding text is now four times larger, which matters if anything downstream reads it before rounding. Rounding at the earliest possible point, immediately after normalization rather than at prompt-assembly time, makes the pipeline immune to this.

The third is scope creep in the questions. A system built to answer “which district is this in” acquires users asking “does this boundary follow the river”, and the simplification tolerance that was invisible for the first question is fatal for the second. This is the drift worth watching most closely, because nothing in the pipeline reports it: the geometry is still valid, the answers are still fluent, and only someone who knows the ground can tell that the boundary detail is gone. Recording the rung in the prompt, as step 5 does, is what makes it detectable — the model can say the detail was removed, if it was told.

A useful habit is to keep two or three real features of different complexity as fixtures and print their token cost under the current settings on every build. It is one line of output, it makes the cost of the configuration visible to everyone who reads a build log, and it turns “we should probably check the token budget” into something that has already been checked.

Failure Modes & Root Causes

Precision inherited from the exporter. Twelve decimal places of a coordinate whose survey accuracy is a metre, consuming four times the tokens for no information. Root cause: no precision policy. Mitigation: round at ingestion, from the accuracy the questions need.

Slivers from independent simplification. Adjacent features stop touching after reduction. Root cause: simplifying each geometry alone. Mitigation: topology-preserving simplification, and where adjacency is critical, simplify the shared boundary network rather than the polygons.

Silent truncation at the budget. A coordinate list is cut mid-number to fit, producing unparseable text the model still reasons over. Root cause: treating the budget as a character limit. Mitigation: the reduction ladder, with the extent as the floor.

Two representations for one corpus. Some features arrive as compact text and others as structured objects, so parsers and prompts must handle both. Root cause: representation chosen per source rather than per corpus. Mitigation: convert at ingestion; store one form.

Production Validation Protocols

1. Round-trip assertion. Every tokenized geometry must parse back, stay valid, and agree within the retained precision; run it over a sample on every build.
2. Topology preservation test. For a fixture of adjacent parcels, assert they still touch after simplification at the configured tolerance.
3. Ladder coverage test. Assert each rung of the reduction ladder is reachable, using a fixture geometry sized to trigger it; an unreachable rung is dead code that will be wrong when it is finally needed.
4. Budget assertion. Assert no emitted fragment exceeds its budget, and that any that reached the extent rung carries its note.
5. Token-cost indicator. Publish the measured cost of the fixture geometries on every build so a tokenizer change is visible immediately.
6. Precision-policy invariant. Assert no stored coordinate carries more decimal places than the policy allows; a violation means something bypassed the ingestion rounding.

Frequently Asked Questions

Related

• Up to the section overview: Spatial LLM Architecture and Core Concepts
• Technique: How to Tokenize Polygon Boundaries for Transformer Models
• Comparison: Well-Known Text, Structured Objects and Cell Identifiers
• Technique: Coordinate Precision Versus Token Cost
• Peer topic: Context-Window Optimization for Maps
• Peer topic: Coordinate Reference System Normalization