Benchmarking Spatial Embedding Models for Vector GIS

Published embedding benchmarks measure general text and rank models accordingly. A corpus of survey reports, planning documents and site assessments is not general text, and the ranking frequently reorders on it — usually in favour of a smaller model. This guide runs the comparison that actually decides the choice, as the measurement half of spatial embedding models.

When to Use This Approach

Benchmark when choosing a model, when a candidate replacement appears, and when the corpus acquires a substantially new kind of document. Not otherwise — the comparison costs a day and its answer is stable between those events.

The inversion in that table is the point. The one number that is always available is the one least likely to decide the outcome, and the numbers that decide it have to be produced locally.

Implementation

The harness encodes the corpus once per candidate, runs the evaluation set with and without a region filter, and reports both alongside cost.

import logging
import time
from dataclasses import dataclass
from typing import Callable, Sequence

log = logging.getLogger("embedding_benchmark")


@dataclass(frozen=True)
class Candidate:
    name: str
    dim: int
    encode: Callable[[Sequence[str]], object]     # batch encode
    max_input_tokens: int


@dataclass(frozen=True)
class Result:
    name: str
    dim: int
    recall_unfiltered: float
    recall_filtered: float
    recall_rare_names: float
    encode_docs_per_s: float
    notes: tuple[str, ...]


def evaluate(candidate: Candidate, corpus: Sequence[dict], queries: Sequence[dict],
             search, k: int = 10) -> Result:
    """Encode the corpus once, then measure three recall figures and throughput."""
    notes: list[str] = []
    texts = [c["embedding_text"] for c in corpus]

    started = time.monotonic()
    try:
        vectors = candidate.encode(texts)
    except Exception as exc:                        # a candidate that cannot encode is a result
        log.warning("%s failed to encode the corpus: %s", candidate.name, exc)
        return Result(candidate.name, candidate.dim, 0.0, 0.0, 0.0, 0.0,
                      (f"encode failed: {exc}",))
    throughput = len(texts) / max(1e-6, time.monotonic() - started)

    def recall(subset, region_filter) -> float:
        hit = tot = 0
        for q in subset:
            want = set(q["relevant"])
            if not want:
                continue
            try:
                got = set(search(candidate.encode([q["text"]])[0], vectors, k, region_filter))
            except Exception as exc:
                notes.append(f"search failed on {q['id']}: {exc}")
                got = set()
            hit += len(want & got)
            tot += len(want)
        return round(hit / tot, 4) if tot else 0.0

    rare = [q for q in queries if q.get("has_rare_name")]
    return Result(
        candidate.name, candidate.dim,
        recall(queries, None),
        recall(queries, "region"),
        recall(rare, "region") if rare else 0.0,
        round(throughput, 1),
        tuple(notes[:5]),
    )

Encoding the corpus once per candidate rather than per query is what makes the harness affordable; the query encoding inside the recall loop is the small cost. Recording throughput at the same time is nearly free and answers the rebuild-time question that will be asked immediately afterwards.

Separating the rare-name recall is the measurement most likely to change a decision. If a candidate’s overall recall is competitive but its rare-name recall is poor, the lexical half of a hybrid system is carrying those queries — which is fine, and it means the dense half is contributing less than the headline number suggests.

def summarise(results: Sequence[Result], target_gap: float = 0.02) -> str:
    """Report the comparison in the terms that decide it."""
    if not results:
        return "no candidates evaluated"
    best_filtered = max(results, key=lambda r: r.recall_filtered)
    close = [r for r in results
             if best_filtered.recall_filtered - r.recall_filtered <= target_gap]
    cheapest = min(close, key=lambda r: (r.dim, -r.encode_docs_per_s))
    if cheapest.name != best_filtered.name:
        return (f"{cheapest.name} at dim {cheapest.dim} is within {target_gap:.02f} "
                f"of {best_filtered.name} on filtered recall, at lower cost")
    return f"{best_filtered.name} leads on filtered recall at dim {best_filtered.dim}"

Validation & Testing

def test_encode_failure_is_a_result_not_an_abort():
    class Broken(Candidate):
        pass
    broken = Candidate("broken", 384, lambda _t: (_ for _ in ()).throw(RuntimeError("x")), 512)
    r = evaluate(broken, CORPUS, QUERIES, search)
    assert r.recall_filtered == 0.0 and r.notes


def test_filtered_and_unfiltered_are_reported_separately():
    r = evaluate(CANDIDATE, CORPUS, QUERIES, search)
    assert r.recall_unfiltered != r.recall_filtered or len(QUERIES) < 5


def test_summary_prefers_a_cheaper_candidate_within_the_gap():
    results = [Result("big", 1536, 0.93, 0.90, 0.7, 40.0, ()),
               Result("small", 384, 0.91, 0.89, 0.7, 180.0, ())]
    assert "small" in summarise(results, target_gap=0.02)


def test_rare_name_recall_is_measured_on_a_subset():
    r = evaluate(CANDIDATE, CORPUS, [q for q in QUERIES if q.get("has_rare_name")], search)
    assert 0.0 <= r.recall_rare_names <= 1.0

The third test encodes the decision rule rather than a measurement, which is the point of having a summary function at all. Left to a table of numbers, a comparison is settled by whoever reads it first; a rule that prefers the cheaper candidate within a stated gap makes the trade explicit and reviewable.

Record the harness output as a file in the repository rather than as a message in a channel. The comparison will be re-litigated — when a new model appears, when someone new joins, when recall drops — and a stored table with its conditions attached settles the question in minutes where a remembered conclusion restarts it.

Gotchas & Edge Cases

Evaluating on queries written by the person choosing the model. They will unconsciously favour the phrasing the current system handles. Draw queries from real traffic, or from the corpus by someone who has not seen the candidates.

Corpus sample too small to cluster. A few thousand chunks drawn at random are more uniformly distributed than the real corpus, which flatters every candidate. Sample by region and keep everything from those regions.

Chunk text differing between candidates. If one candidate is fed the chunk body and another the constructed embedding text, the comparison measures the construction. Build the text once and reuse it.

Input truncation going unnoticed. A candidate with a shorter input limit silently truncates long chunks and loses their tails. Check the corpus percentile against the limit before encoding, and record it as a note rather than discovering it as poor recall.

Throughput measured on a warm cache. The first candidate encodes cold and the rest encode warm, which flatters everything after the first. Randomise the order or discard a warm-up batch.

Query encoding excluded from the throughput figure. Corpus encoding dominates a rebuild and query encoding dominates the request path, and a candidate can be fast at one and slow at the other. Record both if latency matters to you.

Normalisation differing between candidates. Some models return unit vectors and some do not, and comparing cosine similarity across the two without normalising measures the vector lengths. Normalise consistently before searching.

Frequently Asked Questions

Related

• Up to the parent topic: Spatial Embedding Models
• Choosing Vector Dimensionality for Spatial Retrieval
• Related topic: Hybrid Spatial and Keyword Retrieval
• Related technique: Indexing Spatial Embeddings with HNSW and Metadata Filters