Implementing Fallback Routing for Failed Spatial Queries

Build the router that walks a degradation ladder within one deadline, classifies failures into retry, degrade and stop, and returns an answer that states what it lost.

Most fallback logic is written as nested exception handlers, and it accumulates: a retry here, a cached read there, a bare except that swallowed something once. This guide replaces that with a router built from a declared ladder, so the degradation path is data you can review rather than control flow you have to trace — the working implementation of fallback routing for geospatial queries.

When to Use This Approach

Use a declared ladder when there is more than one way to answer and they differ in cost or fidelity. A single-route operation needs a timeout and an error message, not a router.

Situation Ladder Bottom rung
Geometry query with a cache Exact, simplified, cached Refusal
Frame resolution Index lookup, local subset, default Flagged fallback
Raster statistic Full resolution, coarser product, none Refusal with the reason
Boundary membership Exact only Refusal — no degradation is acceptable
Descriptive lookup Exact, cached, stale-cached Answer with an age

The fourth row is the one to configure first. A ladder that degrades every intent equally will eventually answer a regulatory containment question from a simplified geometry, and that answer is a coin flip presented as a fact.

A declared ladder against nested exception handlersA ladder expressed as data is reviewable in one place, while the same behaviour spread through nested exception handlers has to be traced through the code to be understood.nested handlersorder implied by nestingclaims implied by which brancha bare except somewherereviewed by tracingdeclared ladderorder is a listeach rung states its claimone place to add a routereviewed by reading
Same behaviour, different reviewability. The nested form is not wrong; it is simply impossible to answer "what happens when the geometry engine is down" without reading every handler, which is why it drifts.

Implementation

The router takes a ladder, a deadline and a classifier, and returns an outcome that always says which rung produced it.

import logging
import time
from dataclasses import dataclass
from typing import Callable, Optional, Sequence

log = logging.getLogger("fallback_router")


class Transient(Exception):
    """Worth one retry on the same rung."""


class Degradable(Exception):
    """This rung cannot serve this input; move down."""


class Fatal(Exception):
    """Stop — neither retrying nor degrading can help."""


@dataclass(frozen=True)
class Rung:
    name: str
    run: Callable[[dict, float], object]      # (request, seconds_left) -> value
    claim: str                                 # what an answer from here may assert
    typical_s: float


@dataclass(frozen=True)
class Outcome:
    value: Optional[object]
    rung: str
    claim: str
    note: str
    elapsed_s: float
    attempts: int


def classify(exc: Exception) -> str:
    if isinstance(exc, Fatal):
        return "stop"
    if isinstance(exc, Transient):
        return "retry"
    if isinstance(exc, Degradable):
        return "degrade"
    log.warning("unclassified %s — treating as degradable", type(exc).__name__)
    return "degrade"


def route(request: dict, ladder: Sequence[Rung], budget_s: float) -> Outcome:
    """Walk the ladder inside one budget. Always returns an Outcome."""
    if not ladder:
        return Outcome(None, "misconfigured", "none", "no rungs configured", 0.0, 0)
    if budget_s <= 0:
        return Outcome(None, "misconfigured", "none", f"non-positive budget {budget_s}", 0.0, 0)

    started = time.monotonic()
    attempts = 0

    def left() -> float:
        return budget_s - (time.monotonic() - started)

    for rung in ladder:
        remaining = left()
        if remaining <= 0:
            log.info("budget spent before %s", rung.name)
            break
        if remaining < rung.typical_s * 0.5:
            log.info("skipping %s: %.2fs left, typically %.2fs",
                     rung.name, remaining, rung.typical_s)
            continue
        for attempt in (1, 2):
            attempts += 1
            try:
                value = rung.run(request, left())
                return Outcome(value, rung.name, rung.claim, "",
                               time.monotonic() - started, attempts)
            except Exception as exc:
                action = classify(exc)
                log.info("%s attempt %d failed (%s): %s", rung.name, attempt, action, exc)
                if action == "stop":
                    return Outcome(None, rung.name, "none", f"stopped: {exc}",
                                   time.monotonic() - started, attempts)
                if action == "degrade" or attempt == 2 or left() <= 0:
                    break

    return Outcome(None, "refusal", "none",
                   "no route could answer within the time available",
                   time.monotonic() - started, attempts)

The two misconfiguration guards look trivial and earn their place. An empty ladder is the natural consequence of trimming for an intent that has no exact rung configured, and without the guard it presents as a refusal — sending an investigation after missing data when the real problem is a missing configuration entry.

Trimming the ladder per intent is what enforces the non-degradable cases, and it belongs before routing rather than inside it.

NON_DEGRADABLE = {"contains", "regulatory", "boundary"}


def ladder_for(intent: str, ladder: Sequence[Rung]) -> Sequence[Rung]:
    """Trim rungs whose claim cannot support this intent."""
    if intent in NON_DEGRADABLE:
        trimmed = [r for r in ladder if r.claim == "exact"]
        if not trimmed:
            log.warning("intent %r admits no rung: no exact route configured", intent)
        return trimmed
    return ladder
Three failure classes and what each one does to the walkA transient failure retries once on the same rung, a degradable failure moves down immediately, and a fatal failure stops the walk entirely.transientdegradablefatala timeout, a blipone retry herethen move downbad input for this rungretrying is pointlessmove down at oncenot authorised, not validno rung can helpstop and say why
Classification is what stops a router wasting a budget. Retrying a malformed geometry three times on three rungs is nine identical failures and one exhausted deadline, and it is what an unclassified handler does by default.

Validation & Testing

def test_ladder_reaches_the_cache_when_engines_fail():
    ladder = [_failing("exact", Degradable), _failing("simplified", Degradable),
              _returning("cached", "cached-value", claim="cached")]
    out = route({}, ladder, budget_s=2.0)
    assert out.value == "cached-value" and out.claim == "cached"


def test_fatal_stops_the_walk():
    ladder = [_failing("exact", Fatal), _returning("cached", "should-not-run")]
    out = route({}, ladder, budget_s=2.0)
    assert out.value is None and "stopped" in out.note


def test_transient_retries_once_then_degrades():
    ladder = [_failing("exact", Transient), _returning("cached", "v")]
    out = route({}, ladder, budget_s=2.0)
    assert out.value == "v" and out.attempts == 3          # two on the first rung, one on the second


def test_non_degradable_intent_refuses_rather_than_degrading():
    ladder = ladder_for("contains", [_failing("exact", Degradable),
                                     _returning("cached", "v", claim="cached")])
    out = route({}, ladder, budget_s=2.0)
    assert out.value is None and out.rung == "refusal"


def test_empty_ladder_reports_misconfiguration():
    out = route({}, [], budget_s=1.0)
    assert out.rung == "misconfigured"

The last two are the tests that matter over time. The non-degradable case is a policy that will be quietly eroded by anyone adding a convenient cached rung, and the misconfiguration case is the one that turns a silent refusal into a legible error.

Build the fixtures as small helpers that fail in a stated way rather than as mocks of real dependencies. The router’s behaviour depends only on the exception class it sees, so testing it against a real geometry engine tests the engine and leaves the interesting cases — fatal, transient, misconfigured — unexercised.

Gotchas & Edge Cases

A bare exception handler above the router. It converts every classified outcome back into an opaque failure. The router already returns rather than raising for expected conditions; anything it does raise is a bug worth surfacing.

Retry counts that multiply across rungs. Two attempts on each of four rungs is eight calls to systems that are probably all unhealthy at once. One retry, on the first rung only, is usually the right budget.

Claims that drift from behaviour. A rung labelled “simplified” whose tolerance was later increased still reports the same claim string. Assert the relationship — a simplified rung must report the tolerance it actually applied.

Fallback rungs that are never exercised. They only run during incidents, which is the worst time to discover a bug. Fail the rungs above them in continuous integration and assert both the value and the claim.

Rungs exercised only by incidentsThe top rung runs constantly and is well tested by traffic; the lower rungs run only when something is already broken, which is the worst time to discover a defect in them.exact — 93% of trafficsimplified — 6%cached — 1%, and only during incidentsFail the rungs above it in continuous integration, or it is untested code
Traffic tests the top rung and nothing else. The rungs that matter most when things go wrong are the ones with the least exposure to normal operation, which inverts the usual relationship between usage and confidence.

Ladders that grow. Every rung is a code path, an outcome to explain and a claim to keep honest. Three or four covers nearly every real degradation; eight is usually two ladders that should be separate.

The refusal treated as an error. A refusal is a successful outcome of the router — it means no route could answer honestly. Counting it as an error rate makes a well-behaved system look broken and hides the actual error rate underneath it.

Frequently Asked Questions

Where should the ladder be defined?

In configuration adjacent to the intent classification, so the two are read together. A ladder buried in the function that uses it is invisible to anyone asking what happens when a dependency fails, and that question is asked most often by people who are not in that file. Keeping it as a list of named rungs with claims also makes review possible for people who do not read the implementation.

Should the router emit metrics itself?

Yes, and the distribution across rungs is the single most useful signal it produces. A system serving ninety per cent from the top rung is healthy; the same system serving forty per cent from cache has a degradation that no error rate will show, because every request succeeded. Emit the rung name as a dimension and alert on a shift rather than on a threshold.

How should the claim reach the user?

As a sentence opener the answer layer is required to use, not as a field the model may ignore. "From a cached result about four hours old, the nearest depot is…" reads naturally and carries the caveat where the reader will see it. A structured field alongside the answer is easy to implement and reliably dropped by the time the text is composed.

What about partial results from a failed rung?

Discard them unless the rung was explicitly designed to return partials. A geometry query that timed out halfway has computed something, and that something is a subset of unknown shape — using it produces an answer that is confidently incomplete in a way nothing downstream can detect. If partial results are valuable, make them a rung of their own with a claim that says so.

Should the router know about the agent, or only about routes?

Only about routes. A router that inspects the question and decides which rungs apply has absorbed the intent classifier, and the two then change together for unrelated reasons. Keep the trimming outside — classify the intent, trim the ladder, pass the result in — so the router remains a small piece of machinery that can be tested with fixtures rather than with questions.