Retry and Circuit-Breaker Patterns for Spatial Services

Blindly retrying a failed spatial call is worse than not retrying at all when the failure is a self-intersecting polygon: the geometry is invalid on attempt one and still invalid on attempt five, so every retry burns a connection and hides the real error. This guide separates transient spatial faults from permanent ones and wraps them in a retry-with-jitter plus a circuit breaker that trips on the topology-failure rate. It belongs to error mapping for spatial API calls and governs the resilience layer between an agent and its geoprocessing backend.

The decisive move is classification. A statement timeout, a dropped socket, or a pool-exhaustion error is transient — the same input may succeed moments later, so backoff-and-retry is correct. A GEOS topology exception, an invalid WKT, or a CRS mismatch is permanent for that input — retrying only repeats the fault. Conflating the two produces retry storms that amplify load exactly when the service is already unhealthy.

When to Use This Approach

Use a retry decorator on any call that crosses a network or contends for a pooled resource. Add a circuit breaker when repeated failures signal a systemic problem — a wedged replica, a corrupt input batch from an upstream model — where continuing to call is actively harmful. Skip retries entirely for pure validation errors: surface them immediately so the caller can re-prompt or repair.

Error class	Example	Strategy
Transient	Statement timeout, connection reset, pool exhausted	Backoff + jitter, bounded attempts
Permanent	GEOS topology error, invalid WKT, CRS mismatch	Fail fast, no retry, return mapped error
Systemic	Sustained topology-failure rate, replica down	Open circuit, shed to fallback

The breaker specifically watches the rate of permanent topology failures, not just transient ones. A sudden spike of TopologyException across many inputs usually means an upstream generator started emitting garbage geometries; opening the circuit stops the flood and gives operators a clean signal. For turning the resulting errors into readable guidance, see mapping spatial API errors to user-friendly prompts.

Three of these four must not be retried. A retry policy that does not classify spends three attempts confirming a failure it had enough information to recognise on the first.

Implementation

The decorator below classifies exceptions, retries only transient ones with exponential backoff and full jitter, and feeds a circuit breaker whose window trips on either a global failure ratio or a topology-error surge. When the circuit is open, calls return a deterministic fallback without touching the backend.

import asyncio
import logging
import random
import time
from collections import deque
from functools import wraps

log = logging.getLogger("spatial_resilience")

# Substring markers that identify a permanent, non-retryable spatial fault.
_PERMANENT = ("topologyexception", "invalid wkt", "geos", "crs mismatch",
              "non-noded intersection", "ring self-intersection")


def classify(exc: Exception) -> str:
    msg = str(exc).lower()
    if any(tok in msg for tok in _PERMANENT):
        return "permanent"
    if isinstance(exc, (asyncio.TimeoutError, ConnectionError, OSError)):
        return "transient"
    # Default to transient for unknown DB errors but cap attempts tightly.
    return "transient"


class CircuitOpen(Exception):
    """Raised when the breaker is open and calls are shed."""


class Breaker:
    def __init__(self, window: int = 50, fail_ratio: float = 0.5,
                 topo_trip: int = 10, cooldown_s: float = 15.0):
        self._events: deque[tuple[float, str]] = deque(maxlen=window)
        self._fail_ratio = fail_ratio
        self._topo_trip = topo_trip
        self._cooldown_s = cooldown_s
        self._opened_at: float | None = None

    def allow(self) -> bool:
        if self._opened_at is None:
            return True
        if time.monotonic() - self._opened_at >= self._cooldown_s:
            self._opened_at = None      # half-open: let one probe through
            self._events.clear()
            return True
        return False

    def record(self, outcome: str) -> None:
        self._events.append((time.monotonic(), outcome))
        fails = sum(1 for _, o in self._events if o != "ok")
        topo = sum(1 for _, o in self._events if o == "permanent")
        if self._events.maxlen and len(self._events) >= self._events.maxlen:
            if fails / len(self._events) >= self._fail_ratio or topo >= self._topo_trip:
                self._opened_at = time.monotonic()
                log.error("circuit opened: fails=%d topo=%d", fails, topo)


def resilient(breaker: Breaker, *, attempts: int = 4, base: float = 0.2,
              cap: float = 4.0, fallback=None):
    def decorate(fn):
        @wraps(fn)
        async def wrapper(*args, **kwargs):
            if not breaker.allow():
                log.warning("circuit open; shedding %s", fn.__name__)
                return fallback
            last: Exception | None = None
            for attempt in range(attempts):
                try:
                    result = await fn(*args, **kwargs)
                    breaker.record("ok")
                    return result
                except Exception as exc:  # noqa: BLE001 - classified below
                    kind = classify(exc)
                    breaker.record(kind)
                    last = exc
                    if kind == "permanent":
                        log.info("permanent spatial fault; no retry: %s", exc)
                        return fallback
                    sleep = min(cap, base * 2 ** attempt) * random.random()
                    log.warning("transient (%d/%d), backoff %.2fs: %s",
                                attempt + 1, attempts, sleep, exc)
                    await asyncio.sleep(sleep)
            log.error("exhausted retries: %s", last)
            return fallback
        return wrapper
    return decorate


_breaker = Breaker()


@resilient(_breaker, fallback=[])
async def nearby_hydrants(pool, wkt: str, radius_m: float):
    async with pool.acquire() as conn:
        # && bbox pre-filter uses the GiST index before the exact ST_DWithin test.
        return await conn.fetch(
            """
            SELECT h.id
            FROM hydrants h
            WHERE h.geom && ST_Expand(ST_GeomFromText($1, 3857), $2)
              AND ST_DWithin(h.geom, ST_GeomFromText($1, 3857), $2)
            """,
            wkt, radius_m,
        )

Full jitter (base * 2**attempt * random()) is deliberate: it spreads retries across the backoff window so a fleet of clients does not re-collide in synchronized waves. The fallback ([]) lets the caller degrade to a cached or empty answer, consistent with implementing fallback routing for failed spatial queries.

Jitter is not a refinement. Without it, a dependency that comes back up receives every waiting client simultaneously and goes down again, which is indistinguishable from it never having recovered.

Validation & Testing

Permanent faults are not retried. Stub fn to raise Exception("TopologyException: side location conflict") and assert it is called exactly once and returns the fallback — no backoff sleeps occur.
Transient faults back off then succeed. Have the stub fail with asyncio.TimeoutError twice then return a value; assert three total calls and that recorded sleeps are non-decreasing in expectation across attempts.
Breaker opens on topology surge. Feed the breaker topo_trip permanent outcomes within one window and assert allow() returns False, then advance a monotonic clock past cooldown_s and assert it half-opens and permits a single probe.

Gotchas & Edge Cases

String matching is brittle. Classifying by message substring breaks across driver and GEOS versions. Prefer matching on SQLSTATE/exception subclasses where available and treat the substring list as a fallback heuristic, logging any unclassified error for review.
Retrying non-idempotent writes. Backoff on an INSERT that partially committed can duplicate rows. Only decorate idempotent reads or writes guarded by ON CONFLICT; never blind-retry a bare mutation.
Half-open thundering herd. When the breaker half-opens, allowing many probes at once can immediately re-trip it. Admit exactly one probe (as above) and reopen on its failure before letting general traffic resume.
Global breaker hides per-shard health. One breaker across many databases trips on the busiest shard and starves the healthy ones. Key a breaker per backend endpoint so isolation is local.

The half-open state is what makes recovery cheap. Without it the circuit either stays open too long or reopens under a flood of simultaneous probes, and both look like the dependency never came back.

Operating This Step Over Time

A circuit that has never opened is telling you its threshold is too high, and one that opens weekly is telling you about a dependency. Both are findings, and neither is visible unless openings are counted and reviewed — a circuit is one of the few controls whose non-firing is as informative as its firing.

Cool-off periods drift toward the long end after every incident, because a longer cool-off makes a recurring problem quieter. The cost is that a brief blip now takes the full cool-off to recover from, and users experience that as the outage rather than the blip. Reviewing the cool-off against observed recovery times, rather than against how the last incident felt, keeps it proportionate.

Watch for retries that survive a classification change. An error that was transient becomes permanent when a service is retired, and a retry policy that still treats it as worth attempting spends three attempts on every affected request indefinitely. Reviewing recovery outcomes — how often each class actually recovered after a retry — is the check that catches it.

Frequently Asked Questions

How many retries are worth attempting?

One, in almost every interactive case. A second retry succeeds where the first failed only when the failure was very brief, and it costs a full attempt's latency out of a budget that has already lost one. In batch contexts where nobody is waiting, two or three with exponential backoff are reasonable — the constraint there is total throughput rather than a user's patience.

Should the circuit be per dependency or per operation?

Per dependency, usually, because that is the thing that fails. A per-operation circuit will keep sending traffic to a dead service through the operations that have not yet accumulated failures, and a per-dependency one stops all of it at once. The exception is a dependency where one operation is genuinely much more fragile than the others, which is worth its own circuit rather than dragging the healthy operations down with it.

What should a request see when the circuit is open?

The same thing it would see from a capability failure — a degradation to the next route, or a refusal naming the unavailable dependency. An open circuit is not an error to be reported to the user as such; it is a routing fact, and the user's experience should be the fallback rather than a message about internal state.

Does a circuit breaker replace a timeout?

No — the timeout bounds one request and the circuit bounds the aggregate. A dependency that is slow rather than down will pass every timeout individually while consuming the entire budget of every request that touches it, and only the circuit stops that. Set both, and count a timeout as a failure for the circuit's purposes.