Geometry Validation Pipelines

A geometry validation pipeline is the gatekeeping stage that rejects, repairs, or normalizes every incoming spatial payload before it reaches your broker — so malformed coordinates, broken topology, and CRS drift never corrupt downstream analytics or routing.

This page is part of Spatial Payload Routing & Parsing, the section covering how to route, parse, and transform geospatial webhook payloads at production scale. Validation sits immediately after deserialization and before transformation, storage, or dispatch, so it can be scaled independently of the heavier downstream stages.

Prerequisites

Before wiring up a validation pipeline, verify the following are in place:

Anchor your rules to external standards: the RFC 7946 GeoJSON specification defines coordinate ordering and ring closure, and the OGC Simple Features model defines what “valid” means for each geometry type.

Pipeline Architecture

The diagram below shows the four-stage path from a raw webhook delivery to a validated geometry published on the broker. Each stage either passes the payload forward, hands it to the repair branch, or isolates it for review — so a single bad payload can never stall the stream.

Geometry Validation Pipeline Four numbered stages laid out left to right: Schema and Type, Coordinate Bounds, Topology and Repair, and CRS Alignment, connected by arrows toward a validated publish. The Schema stage rejects structural errors with HTTP 400; the Topology stage branches to a repair queue or, when repair fails, to a dead-letter queue. A lower panel compares the three error tiers: recoverable, structural, and fatal. Stage 1 Schema & Type Pydantic v2 required fields Stage 2 Coordinate Bounds ring length no NaN / Inf Stage 3 Topology & Repair is_valid make_valid() Stage 4 CRS Align → EPSG:4326 round precision Publish to broker Structural error → HTTP 400 Repair ok → repair queue Repair fails → DLQ Error Categorization Tiers Recoverable make_valid() resolves it Route to repair queue, flag repaired e.g. bowtie polygon Structural Schema mismatch, missing fields Reject with 400 + failed_field Producer self-corrects Fatal NaN / Inf coords, unsupported CRS Archive to DLQ with full context Manual triage

The pipeline has four numbered stages:

  1. Schema and type — enforce required fields and a known geometry type with Pydantic before any GEOS work runs.
  2. Coordinate bounds — verify ring lengths and reject NaN, Infinity, or out-of-range coordinates.
  3. Topology and repair — run is_valid, attempt make_valid(), and branch failures to repair or the dead-letter queue.
  4. CRS alignment — normalize to a canonical projection (EPSG:4326) and apply precision rounding before publishing.

Step-by-Step Implementation

Step 1 — Schema and Type Enforcement

Validation begins at the JSON boundary. Reject payloads missing type or coordinates, or carrying an unsupported geometry type, before spending CPU on GEOS. This runs synchronously inside the request so producers get an immediate, actionable error.

python 
from pydantic import BaseModel, field_validator
from typing import Literal, Union

ALLOWED_TYPES = {
    "Point", "LineString", "Polygon",
    "MultiPoint", "MultiLineString", "MultiPolygon",
}

class GeometryPayload(BaseModel):
    feature_id: str
    type: Literal[
        "Point", "LineString", "Polygon",
        "MultiPoint", "MultiLineString", "MultiPolygon",
    ]
    coordinates: Union[list, list[list], list[list[list]]]
    crs_epsg: int = 4326  # CRS reported by the provider; EPSG:4326 default

    @field_validator("coordinates", mode="before")
    @classmethod
    def ensure_list(cls, v):
        if not isinstance(v, list) or not v:
            raise ValueError("coordinates must be a non-empty JSON array")
        return v

If validation fails, return a structured error with a clear error_code and failed_field so producers can self-correct without platform-team intervention.

Step 2 — Coordinate Sequence and Bounds Validation

A passing schema does not mean the numbers are usable. A LineString needs at least two distinct points; a Polygon exterior ring needs at least four coordinates with the first and last matching. Scan for NaN, Infinity, and coordinates outside valid WGS84 bounds.

python 
import math

def validate_coordinate_bounds(coords: list) -> bool:
    """Flatten nested coordinate arrays and bounds-check every (lon, lat) pair."""
    flat: list[float] = []

    def flatten(node):
        for item in node:
            flatten(item) if isinstance(item, list) else flat.append(item)

    flatten(coords)
    if len(flat) % 2 != 0:
        return False  # odd count means a malformed pair

    for i in range(0, len(flat), 2):
        lon, lat = flat[i], flat[i + 1]
        if math.isnan(lon) or math.isinf(lon) or not (-180.0 <= lon <= 180.0):
            return False
        if math.isnan(lat) or math.isinf(lat) or not (-90.0 <= lat <= 90.0):
            return False
    return True

Bounds checks assume the payload is already in EPSG:4326. When the provider reports a projected CRS, run this check after Step 4, against the bounds of that CRS instead.

Step 3 — Topology Integrity and Automatic Repair

Coordinate bounds say nothing about self-intersections, unclosed rings, duplicate consecutive vertices, or wrong ring orientation. Shapely’s is_valid is a fast boolean predicate; make_valid() resolves common violations such as bowtie polygons. Reject only when repair changes the geometry type or yields an empty result.

python 
from shapely.geometry import shape, mapping
from shapely.validation import make_valid, explain_validity

def validate_topology(geom_dict: dict) -> tuple[bool, dict, bool]:
    """
    Returns (ok, geometry_dict, was_repaired).
    ok=False means the geometry could not be repaired and must be dead-lettered.
    """
    geom = shape(geom_dict)
    if geom.is_valid:
        return True, geom_dict, False

    reason = explain_validity(geom)  # e.g. "Self-intersection[12.3 45.6]"
    repaired = make_valid(geom)

    # Reject repairs that collapse the geometry or change its dimensionality
    if repaired.is_empty or repaired.geom_type != geom.geom_type:
        return False, {"reason": reason}, False

    return True, mapping(repaired), True

Flagging was_repaired lets downstream consumers audit which geometries were silently altered — important for regulatory and cadastral data where a repaired boundary is not the same as the one a surveyor submitted.

Step 4 — CRS Alignment and Precision Control

Raw payloads arrive in arbitrary coordinate reference systems. Normalize to a single canonical projection before publishing; consistent projection is the topic of CRS Normalization Strategies, and skipping it is the leading cause of false-positive overlaps and broken spatial joins. Cache transformers, since per-request Transformer construction triggers an expensive PROJ database lookup.

python 
from pyproj import Transformer
from shapely.geometry import shape, mapping
from shapely.ops import transform as shapely_transform

_transformer_cache: dict[tuple[int, int], Transformer] = {}

def _get_transformer(source_epsg: int, target_epsg: int = 4326) -> Transformer:
    key = (source_epsg, target_epsg)
    if key not in _transformer_cache:
        # always_xy=True keeps (lon, lat) order across PROJ 6+ axis conventions
        _transformer_cache[key] = Transformer.from_crs(
            source_epsg, target_epsg, always_xy=True
        )
    return _transformer_cache[key]

def normalize_to_wgs84(geom_dict: dict, source_epsg: int, precision: int = 7) -> dict:
    """Reproject to EPSG:4326, then round AFTER transform to avoid topology drift."""
    geom = shape(geom_dict)
    if source_epsg != 4326:
        geom = shapely_transform(_get_transformer(source_epsg).transform, geom)

    def _round(x, y, z=None):
        # ~1 cm precision at the equator with 7 decimal places
        return (round(x, precision), round(y, precision))

    return mapping(shapely_transform(_round, geom))

Round precision only after transformation — rounding first can move a vertex enough to introduce a self-intersection that Stage 3 would have caught but Stage 4 reintroduces.

Step 5 — Error Categorization and Routing

Once a stage fails, route the payload by tier so retries, producer feedback, and manual triage stay separate. Valid geometries proceed to the broker.

python 
from enum import Enum

class ErrorTier(str, Enum):
    RECOVERABLE = "recoverable"  # make_valid() fixed it; publish with repaired=True
    STRUCTURAL = "structural"    # schema / missing field; reject with HTTP 400
    FATAL = "fatal"              # NaN, unsupported CRS, unrepairable; archive to DLQ

async def route_result(broker, tier: ErrorTier | None, payload: dict) -> dict:
    if tier is None:
        await broker.publish("validated-geometries", payload)
        return {"status": "valid"}
    if tier is ErrorTier.RECOVERABLE:
        await broker.publish("repaired-geometries", {**payload, "repaired": True})
        return {"status": "repaired"}
    if tier is ErrorTier.STRUCTURAL:
        return {"status": "rejected", "tier": tier.value}  # caller returns HTTP 400
    await broker.publish("geometry-dlq", payload)  # FATAL: keep full context
    return {"status": "dead_lettered"}

Teams standardizing on binary serialization can map validated GeoJSON straight to Protocol Buffers to cut payload size and parse cost — see GeoJSON to Protobuf Mapping. For latency-critical routing, converting WKT to Protobuf for low-latency routing shows how to attach validation metadata as Protobuf extensions while stripping redundant JSON syntax.

Step 6 — Async Handler Integration

The FastAPI endpoint wires all stages together. Cheap checks run inline; the GEOS-heavy stages run in a thread pool so they never block the event loop, and a result cache short-circuits immutable geometries.

python 
import asyncio
import hashlib
import json
from fastapi import FastAPI, HTTPException, Request
from pydantic import ValidationError
from redis.asyncio import Redis

app = FastAPI()
redis_client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)

@app.post("/webhooks/geometry")
async def validate_geometry(request: Request, broker=...):
    raw = await request.json()

    # Stage 1: schema (sync, fast) — structural failures become HTTP 400
    try:
        payload = GeometryPayload(**raw)
    except ValidationError as exc:
        raise HTTPException(status_code=400, detail=exc.errors())

    geom_dict = {"type": payload.type, "coordinates": payload.coordinates}

    # Cache hit: identical normalized geometry already validated
    cache_key = "geomval:" + hashlib.sha256(
        json.dumps(geom_dict, sort_keys=True).encode()
    ).hexdigest()
    if await redis_client.exists(cache_key):
        return {"status": "valid", "cached": True}

    # Stages 2-4 are CPU-bound: run off the event loop
    def _heavy() -> tuple[ErrorTier | None, dict]:
        if not validate_coordinate_bounds(payload.coordinates):
            return ErrorTier.FATAL, {**raw, "reason": "out_of_bounds_or_nan"}
        ok, fixed, repaired = validate_topology(geom_dict)
        if not ok:
            return ErrorTier.FATAL, {**raw, **fixed}
        normalized = normalize_to_wgs84(fixed, payload.crs_epsg)
        tier = ErrorTier.RECOVERABLE if repaired else None
        return tier, {"feature_id": payload.feature_id, "geometry": normalized}

    tier, out = await asyncio.to_thread(_heavy)
    result = await route_result(broker, tier, out)

    if result["status"] in ("valid", "repaired"):
        await redis_client.set(cache_key, "1", ex=86400)  # 24h for static geometries
    return result

Spatial Validation and Error Handling

The most damaging failures are the ones that pass is_valid but are semantically wrong. Three guards catch them:

  • Type allowlist before parsing. shape() will happily build a GeometryCollection you never intended to support. Constrain the type in Pydantic so an unexpected type is a structural rejection, not a downstream surprise.
  • make_valid() on every operand before any predicate. Self-touching rings make intersects() and union() raise TopologyException. Repairing first keeps the resolution path free of defensive try/except blocks.
  • Bounds re-check after reprojection. A transform can push a coordinate just outside valid range due to floating-point drift; assert bounds again on the normalized geometry before publishing.
python 
from shapely.validation import explain_validity
from shapely.geometry import shape

def diagnose(geom_dict: dict) -> str | None:
    """Return a human-readable reason a geometry is unusable, or None if fine."""
    geom = shape(geom_dict)
    if geom.is_empty:
        return "empty geometry"
    if not geom.is_valid:
        return explain_validity(geom)  # pinpoints self-intersection coordinates
    if not validate_coordinate_bounds(geom_dict["coordinates"]):
        return "coordinates outside WGS84 bounds after normalization"
    return None

When make_valid() cannot produce a usable geometry — rare, usually coordinate overflow — log the raw payload and route to the dead-letter queue rather than returning HTTP 500.

Retry, Backoff, and Delivery Guarantees

Validation is CPU-bound, so the failure you must plan for is not network loss but broker backpressure: validation workers can produce faster than the broker accepts. When publishing fails, retry with exponential backoff and full jitter, and shed load at the receiver rather than buffering unbounded in-flight tasks.

python 
import asyncio
import random

async def publish_with_backoff(
    broker,
    topic: str,
    payload: dict,
    max_attempts: int = 5,
    base_delay_ms: int = 50,
) -> bool:
    """At-least-once publish with full-jitter backoff. Returns False if exhausted."""
    for attempt in range(max_attempts):
        try:
            await broker.publish(topic, payload)
            return True
        except (ConnectionError, TimeoutError):
            ceiling_ms = (2 ** attempt) * base_delay_ms
            await asyncio.sleep(random.uniform(0, ceiling_ms) / 1000.0)
    return False

At-least-once vs exactly-once: most providers deliver at-least-once, so the same geometry can arrive twice. The validation result cache (Step 6) plus the upstream idempotency key turn that into effectively-once for identical payloads. If publish_with_backoff exhausts its budget, return HTTP 503 with a Retry-After header — pushing redelivery back to the provider — instead of accumulating tasks. When in-flight work exceeds a bounded ceiling, return HTTP 429 at the receiver to apply backpressure cleanly.

Verification

This test suite confirms each stage in isolation without a broker or network, using two deliberately broken geometries.

python 
import math
import pytest
from shapely.geometry import Polygon, mapping

# A self-intersecting "bowtie" polygon — invalid but repairable
BOWTIE = {"type": "Polygon",
          "coordinates": [[[0, 0], [1, 1], [1, 0], [0, 1], [0, 0]]]}
# A valid square in EPSG:4326
SQUARE = mapping(Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]))

def test_bounds_rejects_nan():
    from your_module import validate_coordinate_bounds
    assert validate_coordinate_bounds([[0.0, 0.0], [float("nan"), 1.0]]) is False

def test_bounds_rejects_out_of_range():
    from your_module import validate_coordinate_bounds
    assert validate_coordinate_bounds([[200.0, 0.0]]) is False  # lon > 180

def test_topology_repairs_bowtie():
    from your_module import validate_topology
    ok, fixed, repaired = validate_topology(BOWTIE)
    assert ok is True and repaired is True

def test_valid_square_is_untouched():
    from your_module import validate_topology
    ok, fixed, repaired = validate_topology(SQUARE)
    assert ok is True and repaired is False

def test_crs_normalization_rounds_precision():
    from your_module import normalize_to_wgs84
    # EPSG:3857 metres near the equator -> degrees
    out = normalize_to_wgs84(
        {"type": "Point", "coordinates": [111319.49, 0.0]}, source_epsg=3857
    )
    assert math.isclose(out["coordinates"][0], 1.0, abs_tol=1e-4)

Run with pytest -v — all five assertions pass offline.

Troubleshooting

Symptom Likely Spatial Cause Fix
TopologyException on union() / intersection() downstream Geometry passed is_valid but a consumer skipped make_valid() Repair in Stage 3 and publish only the repaired geometry; never the raw input
Validation passes but spatial joins return zero rows CRS never normalized — coordinates are in metres (EPSG:3857), not degrees Enforce Stage 4 normalization to EPSG:4326 before publishing
make_valid() collapses a Polygon to a LineString Degenerate ring (zero area) or all-collinear vertices Reject as fatal when geom_type changes; route to the DLQ for inspection
Self-intersection reappears after normalization Precision rounding applied before reprojection moved a vertex Round AFTER the transform, not before, as in normalize_to_wgs84
Event loop stalls under load, p99 latency spikes GEOS topology checks running inline on the async handler Offload Stages 2-4 with asyncio.to_thread or a worker pool
DLQ fills with the same feature_id repeatedly A producer is emitting coordinates with NaN or in an unsupported CRS Bounds-check in Stage 2 and return the failed_field so the producer self-corrects

FAQ

Should validation run in the handler or in a worker?

Run cheap schema and bounds checks synchronously inside the request so malformed payloads get an immediate HTTP 400 and never reach the broker. Offload GEOS-backed topology checks and CRS transformation to a thread pool (asyncio.to_thread) only when geometries are large or throughput is high — those operations are CPU-bound and will block the event loop if run inline.

When should I auto-repair with make_valid versus rejecting?

Auto-repair self-intersections, bowtie polygons, and unclosed rings when make_valid() returns the same geometry type and the area change is within tolerance — and flag those as repaired so consumers can audit them. Reject (route to the dead-letter queue) when make_valid() returns an empty geometry, changes the geometry type, or when coordinates contain NaN or Infinity, since those indicate upstream corruption that repair would only mask.

Why does is_valid pass but a join still fails?

is_valid only checks OGC Simple Features topology — it does not check the coordinate reference system. A geometry can be valid in EPSG:3857 yet break a join against EPSG:4326 data because the coordinates are in metres, not degrees. Always normalize CRS to a single canonical projection before any join, and assert coordinates fall within the expected bounds for that CRS.

How do I avoid re-validating the same static geometry?

Cache validation results in Redis keyed by a deterministic hash of the normalized coordinate array (json.dumps with sort_keys=True, then SHA-256). Immutable geometries such as administrative boundaries hit the cache and skip the GEOS computation entirely, which removes the dominant cost from p99 latency during traffic spikes.

Operational Considerations

Instrument every stage. Track validation_duration_ms per stage (schema, bounds, topology, CRS), validation_pass_rate, repair_success_rate, dlq_ingestion_count by tier, and geometry_size_bytes distribution. The geometry_repair_count metric in particular is a leading indicator of upstream provider quality and feeds directly into the conflict-handling metrics discussed in Conflict Resolution Strategies.

Scale on CPU, not connections. Because GEOS work dominates, deploy validation workers behind a horizontal autoscaler keyed on CPU utilization or broker queue depth rather than request rate.

Validate before serialization, not after. Running validation upstream of binary encoding means the repaired, normalized geometry is what gets serialized — so the integrity guarantee is preserved end to end into GeoJSON to Protobuf Mapping and any downstream broker.

Keep heavy parsing off the hot path. For payloads with very large geometries, defer parsing to a dedicated worker as covered in Async Processing for Heavy Geometries so the validation receiver stays responsive under burst load.

Explore this section

Converting WKT to Protobuf for Low-Latency Routing

Read more