Cache-Backed Idempotency Checks for Geospatial Webhooks

A distributed cache storing a hash of each spatial payload’s normalised geometry provides sub-millisecond duplicate detection across any number of webhook worker replicas, preventing double-execution of expensive spatial joins, tile regenerations, and database mutations.

This topic is part of Idempotency & Spatial Deduplication, the broader discipline of ensuring geospatial event pipelines process every webhook exactly once regardless of upstream retry behaviour.

Prerequisites

Before implementing this pattern, confirm your stack meets the following baseline. Check off each item as you verify it:

Python 3.10+ — required for structural pattern matching in geometry type dispatch
redis-py 5.0+ — the async client (redis.asyncio) is needed for FastAPI / aiohttp integration
shapely 2.0+ — used for geometry normalisation and ring-orientation enforcement
pyproj 3.6+ — required if incoming payloads may arrive in a CRS other than EPSG:4326
pydantic 2.x — strict schema validation before any cache interaction
Redis 6.2+ (or Valkey / KeyDB) — SET NX EX atomicity and AOF/RDB persistence required
Webhook receiver capable of sub-100 ms responses under concurrent load

Architecture Overview

The pipeline enforces a strict order: normalise first, hash second, look up third. No spatial computation occurs until the cache confirms the event is new.

Layer breakdown:

Ingestion — Pydantic validates schema and rejects 400 Bad Request before any cache touch.
Spatial normalisation — coordinates rounded to a fixed decimal precision, exterior rings enforced counter-clockwise, non-deterministic metadata stripped.
Key derivation — SHA-256 over normalised geometry plus a stable business identifier, prefixed with a schema version.
Cache check — Redis SET NX EX atomically claims the key; a hit short-circuits the response; a miss proceeds to step five.
Spatial processing — the real work: spatial joins, tile invalidation via Tile Update Event Pipelines, database mutations, and downstream dispatch.

Step-by-Step Implementation

Step 1 — Validate the Incoming Payload

Reject invalid GeoJSON before touching the idempotency layer. Early rejection prevents cache pollution from malformed events and keeps error-rate metrics clean.

from pydantic import BaseModel, field_validator, model_validator
from typing import Any, Literal


class GeometryModel(BaseModel):
    type: Literal["Point", "LineString", "Polygon", "MultiPolygon",
                  "MultiLineString", "MultiPoint", "GeometryCollection"]
    coordinates: Any  # validated structurally below

    @field_validator("coordinates")
    @classmethod
    def coords_not_empty(cls, v: Any) -> Any:
        if not v:
            raise ValueError("coordinates must not be empty")
        return v


class SpatialWebhookPayload(BaseModel):
    event_id: str
    device_id: str
    # CRS defaults to EPSG:4326 (WGS 84) per RFC 7946
    crs: str = "EPSG:4326"
    geometry: GeometryModel
    event_type: str

    @model_validator(mode="after")
    def reject_unsupported_crs(self) -> "SpatialWebhookPayload":
        supported = {"EPSG:4326", "CRS84"}
        if self.crs not in supported:
            raise ValueError(
                f"CRS {self.crs!r} not supported; normalise to EPSG:4326 upstream"
            )
        return self

Return 422 Unprocessable Entity (FastAPI default) or 400 Bad Request on Pydantic validation errors. Do not propagate the error into the idempotency layer.

Step 2 — Normalise the Spatial Payload

Raw sensor data carries floating-point representation drift. Two payloads encoding the same physical boundary may differ at the 12th decimal place. Normalisation collapses that drift to a stable byte representation.

import json
from shapely.geometry import shape, mapping
from shapely.validation import make_valid


def normalise_geometry(raw_geometry: dict, precision: int = 7) -> dict:
    """
    Round all coordinates to `precision` decimal places (~1.1 cm at the equator
    for precision=7) and enforce CCW exterior ring orientation.
    Returns a canonical __geo_interface__ dict suitable for deterministic hashing.
    """
    geom = shape(raw_geometry)

    # Repair self-intersections introduced by coordinate rounding
    if not geom.is_valid:
        geom = make_valid(geom)

    def _round_coords(coords: list) -> list:
        if isinstance(coords[0], (int, float)):
            return [round(c, precision) for c in coords]
        return [_round_coords(ring) for ring in coords]

    raw = mapping(geom)
    rounded = {**raw, "coordinates": _round_coords(list(raw["coordinates"]))}

    # Sort keys for deterministic JSON serialisation
    return json.dumps(rounded, sort_keys=True)

Precision 7 gives ~1.1 cm resolution at the equator (EPSG:4326), which is sufficient for drone telemetry and parcel boundary deduplication. For sub-centimetre sensor grids, use precision 8 (~1.1 mm). The approach to normalising coordinates across mixed-CRS payloads is covered in depth in CRS Normalisation Strategies.

Step 3 — Derive a Deterministic Idempotency Key

The key must encode both the geometry and a stable business identifier so that the same geometry arriving from two different sensors does not collapse into a single key.

import hashlib


def derive_idempotency_key(payload: SpatialWebhookPayload) -> str:
    """
    Produces a versioned, collision-resistant key of the form:
      idem:v1:<sha256-hex>

    The version prefix (v1) allows rolling key-schema changes without
    collisions during migration windows. See the Event Key Generation
    notes below for multi-polygon identifier strategies.
    """
    normalised_geom = normalise_geometry(payload.geometry.model_dump())
    composite = f"{payload.device_id}:{payload.event_type}:{normalised_geom}"
    digest = hashlib.sha256(composite.encode("utf-8")).hexdigest()
    return f"idem:v1:{digest}"

Prefix with idem:v1: so you can scan and audit idempotency keys in Redis independently of other key namespaces. When the normalisation algorithm changes (e.g., you upgrade the precision from 7 to 8), increment to v2 and run both versions in parallel during the migration window.

Detailed strategies for handling multi-polygon edge cases, provider-specific business identifiers, and key versioning are covered in Event Key Generation for Spatial Data.

Step 4 — Atomic Cache Check and Claim

SET NX EX is the only safe primitive here. A naive GET followed by SET creates a race condition where two concurrent duplicate payloads both see a miss, both proceed, and the spatial computation runs twice.

import redis.asyncio as aioredis
from fastapi import FastAPI, Request, Response
import logging

logger = logging.getLogger(__name__)
app = FastAPI()


async def check_and_claim(
    client: aioredis.Redis,
    key: str,
    ttl_seconds: int = 259_200,  # 72 hours: covers a 48-hour upstream retry window
) -> bool:
    """
    Atomically claim the key. Returns True if this worker is the first to
    see this event (key was absent). Returns False if the event is a duplicate
    (key was already present).
    """
    acquired = await client.set(key, "processing", nx=True, ex=ttl_seconds)
    return bool(acquired)


async def release_on_failure(client: aioredis.Redis, key: str) -> None:
    """Delete the key so legitimate retries can proceed after a processing error."""
    await client.delete(key)
    logger.warning("idempotency key released after processing failure: %s", key)


@app.post("/webhook/spatial")
async def receive_spatial_webhook(request: Request) -> Response:
    body = await request.json()

    try:
        payload = SpatialWebhookPayload.model_validate(body)
    except Exception as exc:
        return Response(content=str(exc), status_code=400)

    key = derive_idempotency_key(payload)
    redis_client: aioredis.Redis = request.app.state.redis

    is_new = await check_and_claim(redis_client, key)
    if not is_new:
        logger.info("duplicate spatial event short-circuited: %s", key)
        return Response(status_code=200, content="duplicate")

    try:
        await process_spatial_event(payload)
        return Response(status_code=202, content="accepted")
    except Exception:
        await release_on_failure(redis_client, key)
        raise

The ttl_seconds=259_200 default covers a 72-hour window. Adjust downward for IoT streams with 15-minute retry windows to avoid Redis memory bloat.

Step 5 — Spatial Validation and Error Handling

Geometry that passes JSON schema validation can still be topologically invalid. Validate before any spatial indexing or database write.

from shapely.geometry import shape
from shapely.validation import explain_validity


def validate_topology(raw_geometry: dict) -> None:
    """
    Raise ValueError with a human-readable explanation if the geometry
    is topologically invalid (self-intersecting rings, duplicate vertices, etc.).
    Call this after normalisation, before any spatial join or DB write.
    """
    geom = shape(raw_geometry)
    if not geom.is_valid:
        reason = explain_validity(geom)
        raise ValueError(f"Invalid geometry topology: {reason}")
    if geom.is_empty:
        raise ValueError("Geometry is empty after normalisation")


async def process_spatial_event(payload: SpatialWebhookPayload) -> None:
    raw_geom = payload.geometry.model_dump()
    validate_topology(raw_geom)

    # Safe to proceed: geometry is valid and this event is confirmed new
    normalised = normalise_geometry(raw_geom)
    # ... spatial joins, tile invalidation, DB mutations

Topology failures after normalisation are rare but real — they commonly arise from self-intersecting polygon rings in sensor exports or from aggressive coordinate rounding on near-degenerate geometries. Treat them as 422 errors and log the geometry digest for debugging.

Retry, Backoff, and Delivery Guarantees

Cache-backed idempotency shifts the delivery guarantee from at-least-once to effectively-once, but only when combined with sensible retry configuration on the consumer side.

import asyncio
import random


async def dispatch_with_backoff(
    client: aioredis.Redis,
    payload: SpatialWebhookPayload,
    max_attempts: int = 5,
    base_delay: float = 0.5,
) -> None:
    """
    Retry the full pipeline (including idempotency check) with exponential
    backoff and full jitter. Because the cache key is released on failure,
    legitimate retries will re-acquire the key and re-attempt processing.
    Duplicate retries caused by the upstream provider will be filtered by
    the existing cache entry from the first successful claim.
    """
    for attempt in range(1, max_attempts + 1):
        try:
            key = derive_idempotency_key(payload)
            is_new = await check_and_claim(client, key)
            if not is_new:
                return  # upstream duplicate — already processed
            await process_spatial_event(payload)
            return
        except Exception as exc:
            if attempt == max_attempts:
                raise
            # Full jitter: randomise within [0, base * 2^attempt]
            delay = random.uniform(0, base_delay * (2 ** attempt))
            logger.warning(
                "Attempt %d/%d failed (%s); retrying in %.2fs",
                attempt, max_attempts, exc, delay,
            )
            await asyncio.sleep(delay)

At-least-once vs. exactly-once tradeoffs:

Guarantee	Mechanism	Risk for spatial workloads
At-least-once	Retry on any failure, no deduplication	Duplicate spatial joins corrupt analytics; tile cache regenerated twice per event
Effectively-once	Cache-backed idempotency (`SET NX EX`)	Cache eviction or partition can let one duplicate through per eviction event
Exactly-once	Distributed transaction + two-phase commit	High latency; rarely justified for spatial ingestion pipelines

For most geospatial webhook pipelines, effectively-once with a database-level UNIQUE constraint fallback is the right tradeoff. True exactly-once requires a distributed transaction coordinator and adds significant latency to every event.

The at-least-once delivery model and its interaction with spatial state are explored in detail in Implementing At-Least-Once Delivery for GIS Webhooks.

Fallback When the Cache Is Unavailable

A Redis connection failure must not silently drop valid events. Implement a two-tier fallback: the distributed cache is the fast path, and a database UNIQUE constraint is the durable safety net that catches duplicates whenever the cache is unreachable.

from contextlib import asynccontextmanager
import asyncpg  # or your preferred async Postgres driver


async def idempotency_check_with_fallback(
    redis_client: aioredis.Redis,
    pg_pool: asyncpg.Pool,
    payload: SpatialWebhookPayload,
    ttl_seconds: int = 259_200,
) -> bool:
    """
    Primary: Redis SET NX EX.
    Fallback: PostgreSQL unique constraint on (device_id, event_digest).
    Returns True if this call should proceed with processing.
    """
    key = derive_idempotency_key(payload)

    try:
        return await check_and_claim(redis_client, key, ttl_seconds)
    except (aioredis.ConnectionError, aioredis.TimeoutError) as exc:
        logger.warning("Redis unavailable, falling back to DB idempotency: %s", exc)

    # Fallback: insert-or-ignore into the DB idempotency table
    digest = key.split(":")[-1]  # strip the version prefix
    try:
        await pg_pool.execute(
            """
            INSERT INTO spatial_event_log (device_id, event_digest, received_at)
            VALUES ($1, $2, NOW())
            ON CONFLICT (device_id, event_digest) DO NOTHING
            """,
            payload.device_id,
            digest,
        )
        return True  # optimistic: proceed if insert succeeded
    except asyncpg.UniqueViolationError:
        return False  # genuine duplicate caught by DB constraint

This two-tier approach is consistent with the conflict resolution patterns described in Conflict Resolution Strategies.

For near-duplicate events that pass exact-match deduplication (e.g., two sensor readings of the same boundary with minor calibration drift), the next line of defence is Spatial Overlap Deduplication, which uses geometric similarity scoring rather than hash equality.

Verification

Confirm the pipeline end-to-end with a pytest integration test against a real (or containerised) Redis instance:

import pytest
import redis.asyncio as aioredis
import asyncio


@pytest.fixture
async def redis_client():
    client = aioredis.from_url("redis://localhost:6379/15")  # test DB
    yield client
    await client.flushdb()  # clean up after each test
    await client.aclose()


SAMPLE_PAYLOAD = {
    "event_id": "evt-001",
    "device_id": "drone-42",
    "crs": "EPSG:4326",
    "event_type": "boundary_update",
    "geometry": {
        "type": "Polygon",
        "coordinates": [
            [[-0.1276, 51.5074], [-0.1277, 51.5075],
             [-0.1275, 51.5075], [-0.1276, 51.5074]]
        ],
    },
}


@pytest.mark.asyncio
async def test_first_event_is_claimed(redis_client):
    payload = SpatialWebhookPayload.model_validate(SAMPLE_PAYLOAD)
    key = derive_idempotency_key(payload)
    assert await check_and_claim(redis_client, key, ttl_seconds=60) is True


@pytest.mark.asyncio
async def test_duplicate_event_is_rejected(redis_client):
    payload = SpatialWebhookPayload.model_validate(SAMPLE_PAYLOAD)
    key = derive_idempotency_key(payload)
    await check_and_claim(redis_client, key, ttl_seconds=60)  # first claim
    assert await check_and_claim(redis_client, key, ttl_seconds=60) is False


@pytest.mark.asyncio
async def test_concurrent_duplicates_only_one_claimed(redis_client):
    payload = SpatialWebhookPayload.model_validate(SAMPLE_PAYLOAD)
    key = derive_idempotency_key(payload)

    results = await asyncio.gather(
        check_and_claim(redis_client, key, ttl_seconds=60),
        check_and_claim(redis_client, key, ttl_seconds=60),
        check_and_claim(redis_client, key, ttl_seconds=60),
    )
    assert results.count(True) == 1, "Exactly one concurrent claim should succeed"


@pytest.mark.asyncio
async def test_floating_point_drift_same_key(redis_client):
    """Payload with minor coordinate drift must produce the same cache key."""
    payload_a = dict(SAMPLE_PAYLOAD)
    payload_b = dict(SAMPLE_PAYLOAD)
    payload_b["geometry"] = {
        "type": "Polygon",
        "coordinates": [
            [[-0.12760001, 51.50740001], [-0.12770001, 51.50750001],
             [-0.12750001, 51.50750001], [-0.12760001, 51.50740001]]
        ],
    }
    key_a = derive_idempotency_key(SpatialWebhookPayload.model_validate(payload_a))
    key_b = derive_idempotency_key(SpatialWebhookPayload.model_validate(payload_b))
    assert key_a == key_b, "Sub-precision drift must not produce different keys"

Run with pytest -v --asyncio-mode=auto. The final test confirms the normalisation step is actually collapsing floating-point drift — without it, the last assertion will fail and you will have a silent deduplication gap in production.

For Redis configuration detail — including eviction policies, AOF persistence settings, and memory footprint estimates at scale — see Using Redis to Cache Spatial Webhook Signatures.

Troubleshooting

Symptom	Likely spatial cause	Fix
Duplicate events reaching the spatial processing step	Floating-point drift producing different keys for same geometry	Verify `normalise_geometry` rounds to consistent precision; add the drift test above
Cache miss rate spike after provider update	Provider changed coordinate precision or serialisation order	Re-run the normalisation audit; check whether `sort_keys=True` is applied before hashing
Redis memory growing faster than expected	TTL set longer than upstream retry window; full payloads accidentally cached	Store only the key + `"processing"` status string; lower TTL to match actual retry SLA
`make_valid` silently discards polygon vertices	Near-degenerate geometry collapses after rounding	Log the geometry digest before and after `make_valid`; alert on area-change above threshold
Fallback DB constraint never fires	Postgres `ON CONFLICT` clause targets wrong columns	Confirm the UNIQUE index covers `(device_id, event_digest)` exactly as the INSERT writes
Concurrent workers each claim the key	Using `GET` + `SET` instead of `SET NX EX`	Replace all two-step checks with a single atomic `SET NX EX` — never split the check and set
Key disappears before processing completes	TTL too short for slow spatial join workloads	Extend TTL or use a `KEEPTTL` refresh on processing start

FAQ

Why does floating-point drift break exact-match cache lookups for spatial webhooks?

GPS and sensor hardware encode coordinates as IEEE 754 doubles. Serialising, deserialising, and re-serialising those values through different JSON parsers or projection libraries can shift the least-significant bits even when the physical location is unchanged. Normalising to a fixed decimal precision before hashing eliminates this drift and ensures identical real-world positions produce identical cache keys.

What TTL should I set on idempotency keys for geospatial webhooks?

Match your TTL to the upstream provider’s maximum retry window plus a 50% safety buffer. If your webhook provider retries for up to 48 hours, set a 72-hour TTL. For IoT telemetry streams that retry for minutes, a 15-minute TTL is sufficient and avoids unnecessarily consuming Redis memory. Do not use PERSIST (no-TTL) keys: a deployment bug or schema migration can leave zombie keys that permanently block legitimate event replay.

Can I use an in-process LRU cache instead of Redis for idempotency?

Only in single-process deployments. The moment you run multiple webhook worker replicas, each process has its own in-process cache and duplicates flow through undetected. Redis or another distributed cache is required for horizontal scale. For multi-region deployments, consider Redis Cluster with consistent hashing to avoid hot-key concentration during regional webhook spikes.

What happens if the Redis connection drops mid-request?

Implement a fallback to a lightweight database UNIQUE constraint as shown above. Log every cache bypass as a warning metric so you can distinguish planned degradation from infrastructure failure. Never silently drop events — prefer processing a duplicate once over losing a real event. The at-least-once guarantee is the safer floor; the exactly-once optimisation is layered on top via cache.

Idempotency & Spatial Deduplication — the parent section covering the full deduplication strategy for geospatial pipelines
Event Key Generation for Spatial Data — deterministic key derivation strategies, versioning, and multi-polygon identifier design
Using Redis to Cache Spatial Webhook Signatures — Redis configuration, eviction policies, and persistence settings for idempotency workloads
Spatial Overlap Deduplication — geometric similarity scoring for near-duplicate events that pass exact-match filters
CRS Normalisation Strategies — handling mixed coordinate reference systems before spatial hashing

Cache-Backed Idempotency Checks for Geospatial Webhooks

Explore this section

Using Redis to Cache Spatial Webhook Signatures