Architecture¶

How oex turns an ISO3 country code into ready-to-use vector datasets.

Data flow¶

flowchart TD
    CLI["oex-cli osm npl --config hot.yaml"]:::cli

    subgraph MIDDLE[" "]
        direction TD
        subgraph INPUTS["Inputs"]
            direction TD
            BND["Boundary<br/>geoBoundaries ADM0<br/>or custom GeoJSON"]:::input

            subgraph OSM_FLOW["OSM"]
                direction TD
                PBF["Geofabrik PBF<br/>or Planet PBF (osmium clip)"]:::input
                QOSM["QuackOSM"]:::input
                OSM_PQ["GeoParquet cache"]:::input
                PBF --> QOSM --> OSM_PQ
            end

            OVT["Overture<br/>S3 GeoParquet via httpfs"]:::input
        end

        subgraph LOOP["Per-category loop (DuckDB)"]
            direction TD
            SQL["SQL SELECT<br/>bbox + boundary clip"]:::step
            PCODE["Pcode tagging<br/>fieldmaps.io + H3 join<br/>adm0 to adm4"]:::step
            TRANSLIT["Transliteration<br/>name to name_latin"]:::step
            ISO["ISO3 language columns<br/>name_hi, name_ar, name_ne"]:::step
            REPORT["Export report<br/>count, null %, top values"]:::step
            FMT["Format writers<br/>gpkg, shp, geojson, kml"]:::step
            SQL --> PCODE
            PCODE --> TRANSLIT
            TRANSLIT --> ISO
            ISO --> REPORT
            REPORT --> FMT
            SQL ~~~ ISO
            PCODE ~~~ REPORT
            TRANSLIT ~~~ FMT
        end

        INPUTS --> LOOP
    end

    subgraph OUTPUT["Output"]
        direction LR
        ZIP["Zip bundle<br/>README + metadata.json"]:::out
        DEST["HDX / S3"]:::out
        ZIP --> DEST
    end

    CLI --> MIDDLE
    MIDDLE --> OUTPUT

    classDef cli fill:#f5f5f5,stroke:#475569,color:#111,stroke-width:1px
    classDef input fill:#eef2ff,stroke:#6366f1,color:#111,stroke-width:1px
    classDef step fill:#ecfdf5,stroke:#10b981,color:#111,stroke-width:1px
    classDef out fill:#fff7ed,stroke:#f97316,color:#111,stroke-width:1px

1. Boundary - resolves the country polygon from geoBoundaries gbOpen ADM0 (or a custom GeoJSON) and derives a bounding box for all spatial queries.
2. Data - OSM: downloads the per-country PBF from Geofabrik (or clips from a local planet PBF) and converts it to GeoParquet once via QuackOSM. Overture: reads parquet directly from S3 over DuckDB httpfs, no download.
3. Export loop - for each category, DuckDB clips the cached parquet to the country boundary, applies column and tag filters, optionally tags every feature with administrative pcodes (adm1-adm4), transliterates names to Latin script, and writes the requested output formats.
4. Bundle - each format is zipped with a README and optional metadata JSON, then optionally uploaded to HDX or S3.

Technical pipeline¶

Query engine¶

DuckDB runs embedded in the Python process and reads GeoParquet files via memory-mapped columnar scans. Per-category exports are DuckDB SQL SELECT statements with a bbox clip and ST_Intersects boundary filter, so a feature crossing the boundary is kept whole rather than dropped.

OSM path¶

The default OSM engine downloads the per-country PBF from Geofabrik. For countries Geofabrik does not publish, oex falls back to a local planet PBF:

1. The country boundary polygon is simplified by simplify_meters, expanded by the configured buffer_meters, and simplified again, all in an azimuthal equidistant projection centred on the boundary so the distance is metres on the ground at that latitude.
2. osmium extract --strategy=complete_ways clips the planet PBF to that polygon, producing a country-sized PBF.
3. QuackOSM converts the PBF to GeoParquet with tag filtering applied at parse time. The result is written to a local cache and reused on subsequent runs.
4. Per-category queries run as DuckDB SELECT statements over the cached parquet.

Overture path¶

Overture Maps publishes release parquet at s3://overturemaps-us-west-2/release/<release>/theme=.../type=.../. DuckDB's httpfs extension reads these files with parallel HTTP range requests, applying the bbox filter at the parquet page level.

Pcode tagging¶

Once per country run, the admin polygons of each level are filtered to the country, simplified, and written to parquet files in the run's output folder that every layer then reads. Admin pcodes are assigned in three stages:

1. Cover - each admin polygon (adm1-adm4) is filled with H3 hexagonal cells at resolution 7 (~5.16 km² per cell). MULTIPOLYGON geometries are first decomposed into their constituent parts before coverage.
2. Index - each feature's centroid is converted to the H3 cell ID at the same resolution. The centroid is used for attribution: a building, road segment, school, or POI is assigned to the admin area it sits inside.
3. Join - the centroid cell ID is joined against the admin cell lookup on integer equality. Features whose centroid falls on a shared cell boundary are resolved with a ST_Contains point-in-polygon check.

Performance¶

Full HOT 12-category schema for Brazil on a 20 GB Docker container, single worker, 4 CPU cores. DuckDB memory limit: 60% of container RAM (~12 GB).

Peak memory: ~5.7 GB across the full run (measured during pcode tagging of the 11.2 M building category).

Pcode tagging cost is largely independent of feature count: Brazil's admin tessellation produces ~1.54 M H3 cells per admin level and all four levels are built per category, accounting for ~3-4 min of each category's runtime.