Cost and Latency Budgets for Spatial Agents

Spatial agents are expensive in a way text agents are not. A single question can trigger a gazetteer lookup, a catalog search, a filtered vector query, several geometry operations and a raster read, each with its own latency and each contributing tokens to a context that has to be re-sent on every turn. Without a budget the agent will spend whatever the slowest path costs, and the first sign is a bill or a timeout rather than a design decision.

This topic belongs to geospatial prompt engineering and tool routing and is the constraint the routing decisions elsewhere in it operate under. It shares its deadline mechanics with fallback routing for geospatial queries, and its token accounting with context-window optimization for maps.

Foundational Principles

The budget is a constraint on the plan, not a limit on the run. Checking cost as it accrues catches an overrun after the money is spent. Estimating a plan before executing it lets the agent choose a cheaper plan, which is the only intervention that actually saves anything.

Tokens and time are separate budgets with separate failure modes. A plan can be cheap and slow, or fast and expensive, and collapsing them into one number hides which constraint bound. Track both, report both, and let the routing decision see both.

Cached results are the largest available saving. Within one conversation the same place resolves, the same region filters and the same geometry loads repeatedly, and none of it changes. A cache keyed correctly removes more cost than any prompt optimisation, and it makes answers consistent between turns as a side effect.

Step-by-Step Implementation Pipeline

1. Give the request an explicit budget

Both numbers come from the product rather than from the implementation: what a user will wait for, and what a turn may cost. Deriving them from what the current pipeline happens to spend produces a description rather than a budget.

import logging
from dataclasses import dataclass

log = logging.getLogger("agent_budget")


@dataclass(frozen=True)
class Budget:
    tokens: int
    seconds: float
    label: str = "turn"

    def scaled(self, fraction: float, label: str) -> "Budget":
        if not 0 < fraction <= 1:
            raise ValueError("fraction must be in (0, 1]")
        return Budget(int(self.tokens * fraction), self.seconds * fraction, label)


@dataclass
class Spend:
    tokens: int = 0
    seconds: float = 0.0

    def add(self, tokens: int, seconds: float) -> None:
        self.tokens += max(0, tokens)
        self.seconds += max(0.0, seconds)

    def within(self, budget: Budget) -> bool:
        return self.tokens <= budget.tokens and self.seconds <= budget.seconds

2. Estimate a plan before running it

Every tool has a typical cost in both currencies, measured from production rather than guessed. Summing them over a proposed plan gives an estimate accurate enough to reject the expensive plans, which is all it needs to do.

from typing import Mapping, Sequence

TOOL_COST = {                                   # measured, refreshed periodically
    "resolve_place":     {"tokens": 120,  "seconds": 0.15},
    "catalog_search":    {"tokens": 400,  "seconds": 0.30},
    "vector_query":      {"tokens": 900,  "seconds": 0.45},
    "geometry_op":       {"tokens": 250,  "seconds": 0.60},
    "raster_summary":    {"tokens": 180,  "seconds": 1.20},
}


def estimate(plan: Sequence[str], costs: Mapping[str, dict] = TOOL_COST) -> Spend:
    """Sum the known cost of a plan. Unknown steps are charged a pessimistic default."""
    spend = Spend()
    for step in plan:
        cost = costs.get(step)
        if cost is None:
            log.info("no cost model for %r — charging the pessimistic default", step)
            cost = {"tokens": 800, "seconds": 1.0}
        spend.add(cost["tokens"], cost["seconds"])
    return spend

Charging an unknown step the pessimistic default rather than zero is what keeps a new tool from silently escaping the budget. A step with no cost model is exactly the step most likely to be expensive, because nobody has measured it yet.

3. Reject or trim a plan that does not fit

An over-budget plan is not an error; it is a plan that needs to be smaller. The agent should be told which constraint bound and by how much, so the next proposal is a reduction rather than a repeat.

@dataclass(frozen=True)
class PlanVerdict:
    admissible: bool
    reason: str
    over_tokens: int
    over_seconds: float


def admit(plan: Sequence[str], budget: Budget) -> PlanVerdict:
    """Decide whether a plan fits, and say precisely how it does not."""
    spend = estimate(plan)
    over_tokens = max(0, spend.tokens - budget.tokens)
    over_seconds = max(0.0, spend.seconds - budget.seconds)
    if not over_tokens and not over_seconds:
        return PlanVerdict(True, "", 0, 0.0)
    parts = []
    if over_tokens:
        parts.append(f"{over_tokens} tokens over")
    if over_seconds:
        parts.append(f"{over_seconds:.1f}s over")
    return PlanVerdict(False, "; ".join(parts), over_tokens, round(over_seconds, 2))

4. Make the cheap plan discoverable

Rejecting a plan is only useful if a cheaper one exists and the agent can find it. The practical mechanism is a set of substitutions — a coarser raster product, a cached region instead of a fresh query, a name lookup instead of a catalog search — offered alongside the rejection.

SUBSTITUTIONS = {
    "raster_summary": ("cached_raster_summary", "a stored statistic for this shape"),
    "catalog_search": ("known_collection", "the collection used on the previous turn"),
    "vector_query":   ("cached_vector_query", "the same region, already retrieved"),
}


def cheaper_plan(plan: Sequence[str], budget: Budget) -> tuple[list[str], list[str]]:
    """Substitute the most expensive steps until the plan fits, listing what changed."""
    trimmed, notes = list(plan), []
    for _ in range(len(plan)):
        if admit(trimmed, budget).admissible:
            break
        costly = max(range(len(trimmed)),
                     key=lambda i: TOOL_COST.get(trimmed[i], {"seconds": 1.0})["seconds"])
        replacement = SUBSTITUTIONS.get(trimmed[costly])
        if replacement is None:
            break                                 # nothing cheaper: the plan must shrink
        trimmed[costly] = replacement[0]
        notes.append(f"{plan[costly]} replaced by {replacement[1]}")
    return trimmed, notes

Recording the substitutions is what lets the answer carry its caveats. A plan that used a cached statistic instead of a fresh one has produced an answer with an age, and that age belongs in the sentence.

5. Cache the results that repeat

Within a conversation the same lookups recur constantly, and caching them is worth more than any other optimisation here. The keys matter: a place resolution keyed on the name and corpus, a region query keyed on the rounded extent, a raster statistic keyed on the shape and the product version.

def cache_key(kind: str, **parts) -> str:
    """Deterministic keys — rounded extents, sorted parts, version stamps included."""
    rendered = ",".join(f"{k}={parts[k]}" for k in sorted(parts))
    return f"{kind}:{rendered}"


def cached(kind: str, cache, compute, ttl_s: int, **parts):
    """Read through a cache that never fails a request when it is unavailable."""
    key = cache_key(kind, **parts)
    try:
        hit = cache.get(key)
        if hit is not None:
            return hit, True
    except Exception as exc:
        log.warning("cache read failed for %s: %s", key, exc)
    value = compute()
    try:
        cache.set(key, value, ttl=ttl_s)
    except Exception as exc:
        log.warning("cache write failed for %s: %s", key, exc)
    return value, False

The consistency benefit is as valuable as the cost saving. A cached place resolution guarantees that “the station” means the same station on turn four as on turn one, which users notice far more than they notice latency. The keying strategy and its pitfalls are developed in caching spatial tool results across agent turns.

6. Charge the context, not just the calls

The largest token cost in a multi-turn spatial conversation is usually the context being re-sent, and it grows with every tool result added to the history. A budget that counts only the tokens a tool returns will be wrong by the number of turns.

def turn_token_cost(history_tokens: int, new_tool_tokens: int,
                    generation_tokens: int) -> int:
    """Everything the provider will bill for this turn."""
    return history_tokens + new_tool_tokens + generation_tokens


def prune_history(history: list[dict], keep_tokens: int, count_tokens) -> list[dict]:
    """Keep the most recent and the load-bearing; summarise the rest."""
    kept, used = [], 0
    for entry in reversed(history):
        cost = count_tokens(entry["text"])
        if used + cost > keep_tokens and not entry.get("pinned"):
            continue
        kept.append(entry)
        used += cost
    return list(reversed(kept))

Pinning matters more than recency for spatial conversations. The resolved place, the chosen collection and the region are load-bearing for every subsequent turn, while an intermediate geometry result from three turns ago usually is not.

7. Estimate the plan’s cost in the prompt, not only in the code

An agent that cannot see the budget cannot plan within it. Putting the remaining budget and the tool cost table into the context — briefly — changes the plans a model proposes, and it is much cheaper than rejecting plans repeatedly. The estimation approach is set out in estimating token cost of a geoprocessing plan.

def budget_hint(budget: Budget, spend: Spend) -> str:
    """A short, honest statement of what is left, for the model's context."""
    return (f"Budget remaining: {max(0, budget.tokens - spend.tokens)} tokens, "
            f"{max(0.0, budget.seconds - spend.seconds):.1f}s. "
            "Prefer cached results and fewer tool calls when the budget is tight.")

8. Report the spend with the answer

The spend belongs in the response metadata, not only in a metric. It is what lets a caller decide whether to retry with a larger budget, and what lets an operator see which question shapes are expensive without instrumenting each one separately.

def to_response_meta(budget: Budget, spend: Spend, cache_hits: int,
                     substitutions: list[str]) -> dict:
    return {
        "tokens_used": spend.tokens,
        "seconds_used": round(spend.seconds, 2),
        "within_budget": spend.within(budget),
        "cache_hits": cache_hits,
        "substitutions": substitutions,
    }

Operating This Stage Over Time

Cost models go stale in one direction: everything gets more expensive as data grows. A vector query measured at 450 milliseconds against a two-million-chunk corpus will not hold at twenty million, and a plan admitted on the old figures will overrun without anything having changed in the agent. Refresh the cost table from observed latencies on a schedule, and treat a large drift as a signal to look at the tool rather than at the table.

The second drift is in the cache hit rate, and it moves for reasons that have nothing to do with caching. A change in how regions are computed will change the rounded extents used as keys, and a hit rate that falls from seventy per cent to five overnight is almost always a key-shape change rather than a traffic change. Track the rate per cache kind, not in aggregate, or the collapse of one cache disappears into the average of the others.

The third is budget creep, and it happens one incident at a time. Each individual increase is justified by a specific slow case, and the cumulative effect is a system that takes eight seconds and costs four times what it did. Review the budget as a whole periodically against what a user will actually wait for, rather than only when something times out.

Finally, be careful about optimising the wrong thing. Generation is the most visible cost and usually the smallest; the two larger ones — tool latency and re-sent context — are less obvious and far more responsive to effort. Measuring the split once, on real traffic, redirects most teams’ optimisation work immediately.

Failure Modes & Root Causes

The unbounded plan. An agent proposes eleven tool calls and executes all of them. Root cause: no admission step. Mitigation: estimate and admit before running, with substitutions offered on rejection.

The invisible context bill. Token cost grows every turn as tool results accumulate, and nobody notices until the conversation hits a window limit. Root cause: counting only new tokens. Mitigation: charge the history, prune with pinning.

The cache that never hits. Keys include a timestamp, an unrounded extent or an unsorted parameter list, so every request misses. Root cause: keys built from whatever was to hand. Mitigation: deterministic key construction, tested.

The stale cost table. Plans are admitted on figures measured a year ago against a corpus a tenth of the size. Root cause: a table treated as configuration rather than as a measurement. Mitigation: scheduled refresh from observed latencies, with a drift alert.

Production Validation Protocols

1. Admission assertion. Assert no plan runs without passing admission; a bypass path is the first thing to appear under deadline pressure.
2. Estimate accuracy check. Compare estimated against actual spend on a sample of turns and alert when the median error exceeds 30%; a cost model that is systematically wrong is worse than none.
3. Key determinism test. Assert that the same logical request produces the same cache key across processes, including after a restart.
4. Hit-rate indicator per kind. Publish cache hit rates separately for place, region and raster caches, and alert on a step change.
5. Budget-overrun rate. Track the share of turns that exceed budget despite admission; a rising rate means the cost model has drifted.
6. Pinned-context test. Assert the resolved place and chosen collection survive history pruning; losing them is what makes a long conversation start contradicting itself.

Of these six, the estimate-accuracy check is the one that keeps the whole mechanism honest. Admission is only as good as the numbers it admits against, and a cost model that has drifted produces a gate that passes everything — which looks exactly like a gate that is working, since nothing is being rejected.

The pinned-context test is the second worth building early. Losing the resolved place from a pruned history is the failure that makes a long conversation start contradicting itself, and it is invisible in every cost metric because the pruning did exactly what it was asked to do.

Frequently Asked Questions

Related

• Up to the section overview: Geospatial Prompt Engineering and Tool Routing
• Technique: Estimating Token Cost of a Geoprocessing Plan
• Technique: Caching Spatial Tool Results Across Agent Turns
• Peer topic: Multi-Step Spatial Agent Orchestration
• Related topic: Fallback Routing for Geospatial Queries
• Related topic: Context-Window Optimization for Maps