Spatial Embedding Models

An embedding is a claim that similar things end up near each other. For geometry-bearing corpora that claim needs interrogating, because “similar” has at least three meanings — about the same subject, about the same place, and about the same kind of place — and no single vector space captures all of them. This topic covers what to embed, what to leave as metadata, and how to measure whether a candidate model is actually better for your corpus rather than better on a leaderboard.

It belongs to spatial LLM architecture and core concepts and supplies the vectors that spatial vector store selection has to store and spatial context retrieval and reranking has to rank. Its most consequential output is a single number — dimensionality — which sets the memory bill for the whole retrieval system.

Foundational Principles

Do not embed coordinates. A vector space arranges things by learned similarity, and coordinate strings have no useful similarity structure: 51.5 and 51.6 are adjacent, 51.5 and -0.1 are not, and the model has no way to know that the pair means London. Position belongs in a spatial index where it can be filtered exactly.

Embed the name as written, and keep the lexical index too. Place names are the strings users type and the strings dense models blur. Keep them in the embedded text because the surrounding prose gives them meaning, and keep the lexical half described in hybrid spatial and keyword retrieval because only exact matching finds a rare toponym reliably.

Dimensionality is an infrastructure decision. Doubling it doubles the memory of every index and every replica, and the retrieval quality gain is usually small on corpora of ordinary size. Choose it against measured recall on your own data, as set out in choosing vector dimensionality for spatial retrieval.

Step-by-Step Implementation Pipeline

1. Decide what text represents each chunk

The embedded text is a construction, not simply the chunk body. Including the feature’s name, its type and a short parent context measurably improves retrieval for place-oriented questions; including the raw coordinates measurably harms it.

import logging
from dataclasses import dataclass

log = logging.getLogger("spatial_embeddings")


@dataclass(frozen=True)
class Chunk:
    body: str
    name: str | None
    feature_type: str | None
    parents: tuple[str, ...]        # containing administrative names
    epsg: int | None


def embedding_text(chunk: Chunk, max_chars: int = 4000) -> str:
    """Build the string that gets embedded. Coordinates deliberately excluded."""
    head = []
    if chunk.name:
        head.append(chunk.name)
    if chunk.feature_type:
        head.append(chunk.feature_type)
    if chunk.parents:
        head.append(", ".join(chunk.parents[:2]))     # nearest containers only
    prefix = " — ".join(head)
    text = f"{prefix}\n{chunk.body}" if prefix else chunk.body
    if len(text) > max_chars:
        log.info("embedding text truncated from %d to %d chars", len(text), max_chars)
        text = text[:max_chars]
    return text

Truncating rather than splitting is right here because the chunk has already been sized for retrieval; a chunk whose embedding text exceeds the model’s input is a chunking problem, and the log line is what surfaces it.

2. Choose candidates on the basis of your corpus, not a leaderboard

General-purpose benchmarks measure performance on general-purpose text. A corpus of survey reports, planning documents and technical assessments is not that, and the ranking frequently reorders. Shortlist on practical grounds — input length, dimensionality, licence, latency — and settle it by measurement.

@dataclass(frozen=True)
class Candidate:
    name: str
    dim: int
    max_input_tokens: int
    normalises: bool               # does it return unit vectors already?


def shortlist(candidates, corpus_p95_tokens: int, memory_budget_gib: float,
              n_chunks: int) -> list[Candidate]:
    """Filter on hard constraints before measuring anything."""
    out = []
    for c in candidates:
        if c.max_input_tokens < corpus_p95_tokens:
            log.info("%s rejected: input limit %d below corpus p95 %d",
                     c.name, c.max_input_tokens, corpus_p95_tokens)
            continue
        gib = n_chunks * c.dim * 4 / 1024 ** 3
        if gib > memory_budget_gib:
            log.info("%s rejected: %.1f GiB of raw vectors exceeds the %.1f GiB budget",
                     c.name, gib, memory_budget_gib)
            continue
        out.append(c)
    return out

3. Measure on place-oriented queries, not generic ones

The evaluation set for an embedding decision needs the queries this corpus actually receives, and those skew toward names, types and local descriptions. Measuring on generic paraphrase pairs will rank models by a property you are not buying.

def recall_at_k(model, queries, truth, encode_corpus, k: int = 10) -> float:
    """Recall over a labelled set of real place-oriented queries."""
    corpus_vecs = encode_corpus(model)
    hit = tot = 0
    for q, want in zip(queries, truth):
        try:
            got = set(nearest(model.encode(q), corpus_vecs, k))
        except Exception as exc:
            log.warning("encode failed for %r: %s — scoring as a miss", q, exc)
            got = set()
        hit += len(want & got)
        tot += len(want)
    return round(hit / tot, 4) if tot else 0.0

4. Measure under the filter, because that is the workload

Unfiltered recall is a property of the model; filtered recall is a property of the model and the corpus together, and it is what users experience. A model whose vectors cluster tightly by region can look excellent unfiltered and mediocre once a bounding box has removed everything outside one town.

def recall_under_region(model, queries, truth, encode_corpus, regions, k: int = 10) -> float:
    """Recall when the candidate set is restricted to the query's own region."""
    corpus_vecs = encode_corpus(model)
    hit = tot = 0
    for q, want, region in zip(queries, truth, regions):
        eligible = [i for i, meta in enumerate(corpus_meta) if meta["region"] == region]
        if not eligible:
            continue
        got = set(nearest_within(model.encode(q), corpus_vecs, eligible, k))
        hit += len(want & got)
        tot += len(want)
    return round(hit / tot, 4) if tot else 0.0

5. Normalise once, consistently

Cosine similarity assumes unit vectors. Some models return them and some do not, and a pipeline that normalises inconsistently produces scores that are not comparable between the two halves of its own corpus.

import numpy as np


def normalise(vectors: np.ndarray, already_unit: bool) -> np.ndarray:
    """Return unit vectors. Idempotent, and safe on zero vectors."""
    if already_unit:
        return vectors
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    zero = norms.squeeze() == 0
    if zero.any():
        log.warning("%d zero-length embedding(s); leaving them unnormalised", int(zero.sum()))
        norms[norms == 0] = 1.0
    return vectors / norms

6. Decide about fine-tuning by measuring the gap, not by ambition

Fine-tuning an embedding on domain text is a real improvement in some corpora and a maintenance burden in all of them. The decision should follow a measurement: how much of the retrieval failure is attributable to the embedding rather than to chunking, filtering or the lexical half.

def attribution(recall_dense: float, recall_lexical: float, recall_hybrid: float) -> str:
    """Where is the headroom? Fine-tune only when the dense half is the binding constraint."""
    if recall_hybrid - max(recall_dense, recall_lexical) < 0.02:
        return "fusion is not contributing — fix the weaker half before touching the model"
    if recall_dense < recall_lexical - 0.10:
        return "dense half is the constraint — a domain-tuned embedding may help"
    return "dense half is competitive — invest in chunking or filtering instead"

Most teams that reach for fine-tuning discover, on running something like this, that their retrieval failures come from chunk boundaries or a missing spatial filter. Those are cheaper to fix and do not create a model artefact that has to be retrained whenever the corpus shifts.

7. Pin the model and record it with the index

An embedding is only comparable to other embeddings from the same model and the same version. Storing the model identifier alongside the index is what lets a later query detect that it is comparing vectors from two generations.

def index_manifest(model_name: str, model_version: str, dim: int, normalised: bool) -> dict:
    return {"model": model_name, "version": model_version,
            "dim": dim, "normalised": normalised}


def assert_compatible(query_manifest: dict, index_manifest_: dict) -> None:
    for key in ("model", "version", "dim", "normalised"):
        if query_manifest[key] != index_manifest_[key]:
            raise ValueError(
                f"embedding mismatch on {key}: query {query_manifest[key]!r} "
                f"against index {index_manifest_[key]!r}")

8. Plan the migration before you need it

Changing an embedding means re-encoding the corpus and rebuilding the index, and doing that without downtime requires the plan to exist beforehand: build the new index alongside the old, verify recall on the labelled set, switch reads, then retire. A team that has not rehearsed this will treat the embedding as immutable, which is the real cost of not planning it.

def migration_ready(new_recall: float, old_recall: float, tolerance: float = 0.01) -> bool:
    """Do not switch on faith; switch on a measurement against the same labelled set."""
    if new_recall + tolerance < old_recall:
        log.warning("new embedding recall %.3f is below current %.3f", new_recall, old_recall)
        return False
    return True

Operating This Stage Over Time

Embedding choices age differently from most infrastructure: they get better around you. A model two generations old is not broken, and the pressure to migrate comes from a general sense that something newer exists rather than from a measurement. The labelled retrieval set is what turns that into a decision — run the new candidate against it, and migrate when the number moves by an amount that justifies the rebuild.

The corpus drifts too, and less visibly. A retrieval system that started on planning documents and has since absorbed field survey notes is being asked to embed a different register of text, and the model that measured best on the first corpus may not be best on the mixed one. Re-running the comparison annually, or whenever a substantial new source is added, costs an afternoon and occasionally changes the answer.

Watch for one specific failure that has no error: a partial re-encode. If half the corpus is embedded with one model version and half with another, every similarity comparison across that boundary is meaningless, and nothing raises. The manifest assertion in step 7 is the guard; it only works if it is checked at query time as well as at build time, because the mixed state arises during a migration rather than after it.

Finally, keep the dimensionality decision open. It is tempting to treat it as settled once the index exists, and it is one of the few parameters where a later reduction — through a model that supports shortened outputs, or through dimensionality reduction fitted on the corpus — can halve the memory bill for a measurable, and often acceptable, recall cost. Revisit it whenever the index outgrows its host, before reaching for a larger host.

Two habits make the rest of this cheaper. Keep the encoding step idempotent and content-addressed, so re-encoding a corpus skips everything whose embedding text has not changed — which turns a full rebuild from an overnight job into a short one for most changes. And keep the labelled retrieval set in the repository next to the encoder, so the comparison that justifies a model choice is something anyone can re-run rather than something that lived in one person’s notebook.

Failure Modes & Root Causes

Coordinates in the embedded text. Retrieval quality drops for no apparent reason after a chunking change that started including geometry. Root cause: numeric strings occupying vector space with no useful structure. Mitigation: build the embedding text explicitly, as in step 1, rather than embedding the raw chunk.

Mixed model versions in one index. Similarity scores that are meaningless across part of the corpus, with no error anywhere. Root cause: a partial re-encode. Mitigation: the manifest, asserted at query time.

Leaderboard selection. A model chosen on general benchmarks underperforms a smaller one on the actual corpus. Root cause: measuring the wrong thing. Mitigation: a labelled set of real queries, measured under the real filter.

Inconsistent normalisation. Cosine scores from two halves of the corpus are on different scales. Root cause: normalising conditionally on a per-batch flag. Mitigation: normalise once at write time, record it in the manifest.

Production Validation Protocols

1. Manifest assertion. Assert model, version, dimensionality and normalisation match between query and index on every request.
2. No-coordinate test. Assert the embedding text contains no coordinate-shaped token, using a fixture chunk full of geometry.
3. Filtered recall gate. Measure recall under a realistic region filter on every index build; gate on a regression greater than two points.
4. Zero-vector check. Assert no stored vector has zero length; a zero vector matches nothing and usually means an empty embedding text.
5. Dimensionality budget. Recompute the index memory model whenever the model changes, and compare it against the host before the rebuild rather than after.
6. Migration rehearsal. Rebuild the index with the current model on a schedule, to keep the migration path exercised even when the model has not changed.

The zero-vector check is the cheapest of these and catches the most confusing failure. An empty embedding text — produced by a chunk whose body was whitespace, or by a truncation that removed everything — encodes to a vector that matches nothing and produces a chunk which is present in the index, counted in every statistic, and unreachable by any query.

Frequently Asked Questions

One closing note on expectations. For most geometry-bearing corpora a competent general-purpose embedding, a good chunking strategy and a working spatial filter will outperform a domain-tuned model bolted onto weak chunking, by a wide margin. The ordering of effort implied by that is unglamorous and reliable: fix the boundaries, add the filter, close the lexical gap, and only then ask whether the vectors are the constraint.

Related

• Up to the section overview: Spatial LLM Architecture and Core Concepts
• Technique: Benchmarking Spatial Embedding Models for Vector GIS
• Technique: Choosing Vector Dimensionality for Spatial Retrieval
• Related topic: Spatial Vector Store Selection
• Related topic: Hybrid Spatial and Keyword Retrieval
• Peer topic: Geometry Tokenization Strategies