Skip to content

API Reference

oex

oex: country-scale OSM and Overture vector exports.

__version__ module-attribute 

__version__ = '0.4.15'

__all__ module-attribute 

__all__ = [
    "BoundaryConfig",
    "CategoryConfig",
    "DuckdbConfig",
    "ExportResult",
    "Exporter",
    "HdxConfig",
    "OsmSourceConfig",
    "OutputConfig",
    "OvertureSourceConfig",
    "ParallelConfig",
    "PcodesSourceConfig",
    "RootConfig",
    "__version__",
]

BoundaryConfig dataclass

CategoryConfig dataclass

DuckdbConfig dataclass

HdxConfig dataclass

OsmSourceConfig dataclass

OutputConfig dataclass

OvertureSourceConfig dataclass

ParallelConfig dataclass

PcodesSourceConfig dataclass

RootConfig dataclass

Exporter

ExportResult dataclass

boundary

Country boundary resolution: user-supplied geom or geoBoundaries ADM0.

cli

Typer CLI for oex.

cmd_overture 

cmd_overture(
    iso3_or_yaml: str | None = typer.Argument(
        None,
        help="ISO3 like NPL, or name of a YAML in ./configs/ (prefer --iso3)",
    ),
    theme: str | None = typer.Argument(
        None,
        help="Optional theme override (e.g. buildings)",
    ),
    configs_dir: Path | None = typer.Option(
        None,
        "--configs-dir",
        help="Run every YAML in this directory",
    ),
    config: Path | None = typer.Option(
        None,
        "--config",
        "-c",
        help="Explicit config YAML path",
    ),
    iso3: str | None = typer.Option(
        None,
        "--iso3",
        help="ISO3 country code (e.g. NPL, COD). Overrides the positional argument and YAML.",
    ),
    dataset_name: str | None = typer.Option(
        None,
        "--dataset-name",
        help="Free-form area label used as the {country} substitution in hdx.title_template. Set this to fix pycountry inversions (e.g. DRC) or for sub-national exports.",
    ),
    output_dir: Path | None = typer.Option(
        None, "--output-dir", "-o"
    ),
    hdx_push: bool | None = typer.Option(
        None, "--hdx-push/--no-hdx-push"
    ),
    hdx_purge: bool | None = typer.Option(
        None,
        "--hdx-purge/--no-hdx-purge",
        help="Destructive: delete every existing resource on the dataset before upload.",
    ),
    hdx_combine: bool | None = typer.Option(
        None,
        "--hdx-combine/--no-hdx-combine",
        help="Publish every category onto one HDX dataset instead of one per category.",
    ),
    pmtiles: bool | None = typer.Option(
        None,
        "--pmtiles/--no-pmtiles",
        help="Generate PMTiles. With --hdx-combine, all layers merge into one tileset.",
    ),
    s3: bool | None = typer.Option(
        None,
        "--s3/--no-s3",
        help="Upload artifacts to S3. Overrides output.s3.enabled.",
    ),
    resume: bool | None = typer.Option(
        None,
        "--resume/--no-resume",
        help="Skip categories already built and uploaded according to the state file. Default: enabled (configurable via output.resume).",
    ),
) -> None

Export Overture data.

cmd_osm 

cmd_osm(
    iso3_or_yaml: str | None = typer.Argument(
        None,
        help="ISO3 like NPL, or name of a YAML in ./configs/ (prefer --iso3)",
    ),
    theme: str | None = typer.Argument(
        None,
        help="Optional theme override (e.g. buildings)",
    ),
    configs_dir: Path | None = typer.Option(
        None, "--configs-dir"
    ),
    config: Path | None = typer.Option(
        None, "--config", "-c"
    ),
    iso3: str | None = typer.Option(
        None,
        "--iso3",
        help="ISO3 country code (e.g. NPL, COD). Overrides the positional argument and YAML.",
    ),
    dataset_name: str | None = typer.Option(
        None,
        "--dataset-name",
        help="Free-form area label used as the {country} substitution in hdx.title_template. Set this to fix pycountry inversions (e.g. DRC) or for sub-national exports.",
    ),
    output_dir: Path | None = typer.Option(
        None, "--output-dir", "-o"
    ),
    hdx_push: bool | None = typer.Option(
        None, "--hdx-push/--no-hdx-push"
    ),
    hdx_purge: bool | None = typer.Option(
        None,
        "--hdx-purge/--no-hdx-purge",
        help="Destructive: delete every existing resource on the dataset before upload.",
    ),
    engine: str | None = typer.Option(
        None,
        "--engine",
        help="OSM engine: 'geofabrik' (default), 'planet', or 'postpass'/'rawdata' for live OSM.",
    ),
    download_if_missing: bool | None = typer.Option(
        None,
        "--download-if-missing/--no-download-if-missing",
        help="When the planet path is missing, download the ~87 GB planet PBF before running. Overrides source.osm.auto_download_planet.",
    ),
    resume: bool | None = typer.Option(
        None,
        "--resume/--no-resume",
        help="Skip categories already built and uploaded according to the state file. Default: enabled (configurable via output.resume).",
    ),
    hdx_combine: bool | None = typer.Option(
        None,
        "--hdx-combine/--no-hdx-combine",
        help="Publish every category onto one HDX dataset instead of one per category.",
    ),
    pmtiles: bool | None = typer.Option(
        None,
        "--pmtiles/--no-pmtiles",
        help="Generate PMTiles. With --hdx-combine, all layers merge into one tileset.",
    ),
    s3: bool | None = typer.Option(
        None,
        "--s3/--no-s3",
        help="Upload artifacts to S3. Overrides output.s3.enabled.",
    ),
) -> None

Export OSM data via the configured engine.

cmd_file 

cmd_file(
    configs_dir: Path | None = typer.Option(
        None, "--configs-dir"
    ),
    config: Path | None = typer.Option(
        None, "--config", "-c"
    ),
    iso3: str | None = typer.Option(
        None,
        "--iso3",
        help="ISO3 country code; required when pcode tagging is enabled.",
    ),
    dataset_name: str | None = typer.Option(
        None, "--dataset-name"
    ),
    output_dir: Path | None = typer.Option(
        None, "--output-dir", "-o"
    ),
    hdx_push: bool | None = typer.Option(
        None, "--hdx-push/--no-hdx-push"
    ),
    hdx_combine: bool | None = typer.Option(
        None, "--hdx-combine/--no-hdx-combine"
    ),
    pmtiles: bool | None = typer.Option(
        None, "--pmtiles/--no-pmtiles"
    ),
    s3: bool | None = typer.Option(None, "--s3/--no-s3"),
) -> None

Export categories read from spatial data files.

Each category names a file and maps its columns, and the export then follows the same path as any other source: clipped to the boundary, pcode tagged, written to every configured format, and published with the metadata the config carries.

cmd_all 

cmd_all(
    iso3_or_yaml: str | None = typer.Argument(
        None,
        help="ISO3 like NPL, or name of a YAML in ./configs/ (prefer --iso3)",
    ),
    theme: str | None = typer.Argument(
        None,
        help="Optional theme override (e.g. buildings)",
    ),
    configs_dir: Path | None = typer.Option(
        None, "--configs-dir"
    ),
    config: Path | None = typer.Option(
        None, "--config", "-c"
    ),
    iso3: str | None = typer.Option(
        None,
        "--iso3",
        help="ISO3 country code (e.g. NPL, COD). Overrides the positional argument and YAML.",
    ),
    dataset_name: str | None = typer.Option(
        None,
        "--dataset-name",
        help="Free-form area label used as the {country} substitution in hdx.title_template. Set this to fix pycountry inversions (e.g. DRC) or for sub-national exports.",
    ),
    output_dir: Path | None = typer.Option(
        None, "--output-dir", "-o"
    ),
    hdx_push: bool | None = typer.Option(
        None, "--hdx-push/--no-hdx-push"
    ),
    hdx_purge: bool | None = typer.Option(
        None,
        "--hdx-purge/--no-hdx-purge",
        help="Destructive: delete every existing resource on the dataset before upload.",
    ),
    engine: str | None = typer.Option(
        None,
        "--engine",
        help="OSM engine override for the OSM half of the run. Ignored by the Overture run.",
    ),
    download_if_missing: bool | None = typer.Option(
        None,
        "--download-if-missing/--no-download-if-missing",
        help="When the planet path is missing, download the ~87 GB planet PBF before running. Overrides source.osm.auto_download_planet.",
    ),
    resume: bool | None = typer.Option(
        None,
        "--resume/--no-resume",
        help="Skip categories already built and uploaded according to the state file. Default: enabled (configurable via output.resume).",
    ),
    hdx_combine: bool | None = typer.Option(
        None,
        "--hdx-combine/--no-hdx-combine",
        help="Publish every category onto one HDX dataset instead of one per category.",
    ),
    pmtiles: bool | None = typer.Option(
        None,
        "--pmtiles/--no-pmtiles",
        help="Generate PMTiles. With --hdx-combine, all layers merge into one tileset.",
    ),
    s3: bool | None = typer.Option(
        None,
        "--s3/--no-s3",
        help="Upload artifacts to S3. Overrides output.s3.enabled.",
    ),
) -> None

Export every source the config enables, in order (OSM then Overture).

Each enabled source runs into the same config, so with hdx.combine the second source accumulates onto the dataset the first published.

cmd_metadata 

cmd_metadata(
    configs_dir: Path | None = typer.Option(
        None,
        "--configs-dir",
        help="Run every YAML in this directory",
    ),
    config: Path | None = typer.Option(
        None,
        "--config",
        "-c",
        help="Explicit config YAML path",
    ),
    iso3: str | None = typer.Option(
        None,
        "--iso3",
        help="ISO3 country code (e.g. NPL, COD)",
    ),
    check: bool = typer.Option(
        False,
        "--check",
        help="Report what would change and write nothing.",
    ),
    prune: bool = typer.Option(
        False,
        "--prune",
        help="Destructive: delete resources whose category is no longer in the config.",
    ),
) -> None

Push config text to HDX without re-running the export.

Rewrites the dataset title, description, caveats, source and tags, and every resource's name and description, from the config. Data files are untouched, so a wording change takes seconds instead of a full export.

cmd_osm_build_cache 

cmd_osm_build_cache(
    config: Path | None = typer.Option(
        None,
        "--config",
        "-c",
        help="Config providing source.osm settings",
    ),
) -> None

Download the planet OSM PBF to <cache_dir>/_pbf/planet-latest.osm.pbf.

Per-country extraction from that PBF happens lazily in oex-cli osm <ISO3> when source.osm.engine is planet (or geofabrik with planet_fallback: true).

config

Typed configuration loading.

ConfigError

Bases: ValueError

Raised when a configuration is malformed.

apply_overrides 

apply_overrides(
    cfg: RootConfig, overrides: dict[str, Any]
) -> RootConfig

Apply a dict of dotted overrides to an already-loaded config.

load_config 

load_config(
    user_config: str | PathLike[str] | None = None,
    overrides: list[str] | None = None,
) -> RootConfig

Build a RootConfig. categories precedence: defaults < categories_file < inline categories:.

select_categories 

select_categories(
    cfg: RootConfig, theme: str | None
) -> RootConfig

Restrict the config to a single category whose slugified name matches theme.

loader

Layered YAML config: bundled defaults < user YAML < dotlist overrides.

ConfigError

Bases: ValueError

Raised when a configuration is malformed.

load_config 
load_config(
    user_config: str | PathLike[str] | None = None,
    overrides: list[str] | None = None,
) -> RootConfig

Build a RootConfig. categories precedence: defaults < categories_file < inline categories:.

apply_overrides 
apply_overrides(
    cfg: RootConfig, overrides: dict[str, Any]
) -> RootConfig

Apply a dict of dotted overrides to an already-loaded config.

select_categories 
select_categories(
    cfg: RootConfig, theme: str | None
) -> RootConfig

Restrict the config to a single category whose slugified name matches theme.

schema

Typed run configuration.

CombinedHdx dataclass

Metadata for the one dataset that hdx.combine publishes every category onto.

Mirrors a category's hdx: block, so dataset-level metadata is described the same way per-layer metadata is. Every value is optional: an empty one falls back to what oex derives from the categories and the source that built them.

MapAssetsConfig dataclass

Where the report map gets its basemap and its JavaScript.

The published page fetches these at view time, so they are configurable: pin a different version, or serve them from your own host when a CDN is unreachable.

FileSourceConfig dataclass

Defaults for categories read from spatial files.

CategoryFile dataclass

A category read from a user-supplied spatial file rather than a mapped source.

license_label 
license_label(value: str) -> str

Human-readable licence name, falling back to whatever the config set.

dataset_identity 
dataset_identity(cfg: RootConfig) -> str

Path and filename id for a dataset: iso3 when set, else the S3 folder id.

Sub-national exports (Tasking Manager projects) have no country code; they identify by project id through output.s3.folder.

defaults

Bundled default YAML configuration.

duckdb_session

exporter

Per-category export loop, shared by Overture and OSM sources.

BuiltCategory dataclass

A processed category plus the artifacts phase-B combined publishing needs.

Exporter

hdx_publisher

HDX dataset and resource publication. Imports hdx-python-api lazily.

HdxPublisher

refresh_metadata 
refresh_metadata(
    cfg: RootConfig,
    category: CategoryConfig,
    ctx: PublishContext,
) -> str | None

Restate an existing dataset's snapshot date when a category exports nothing.

A category with no features publishes no resources, so without this its dataset keeps the date of the last run that did find some and drifts past the update frequency it advertises. The source data was still read at ctx.snapshot_date, so recording that is accurate. Returns None when nothing is published yet, since an empty category is no reason to create a resourceless dataset.

update_metadata 
update_metadata(
    cfg: RootConfig,
    *,
    dry_run: bool = False,
    prune: bool = False,
) -> tuple[str, int, int]

Rewrite dataset and resource text from the config, leaving files and ids alone.

Returns the dataset name and how many resources changed.

geoparquet_resource_payload 

geoparquet_resource_payload(
    category: CategoryConfig, url: str, source_name: str
) -> dict

Resource fields for a layer GeoParquet, which is published unzipped at its staged URL.

locale

Resolve a country's OSM name:<lang> tags from its ISO3 code via babel.

primary_osm_language 

primary_osm_language(iso3: str) -> str | None

First non-English official language for the country, or None.

local_osm_languages 

local_osm_languages(iso3: str) -> list[str]

Up to three non-English official languages for the country.

Babel sometimes lists English first for multilingual countries (Sudan, Philippines), so English is dropped: name_en is already covered by the schema's static select.

logging_setup

Idempotent root logger setup.

metadata

Per-dataset metadata report (feature counts, geom types, bbox, column stats).

naming

Names derived from config values, shared so the exporter and the publisher agree.

slugify 

slugify(value: str) -> str

Config text to a filename and HDX-safe token.

osm

OSM source: planet PBF download + quackosm conversion + per-country query.

OsmRunner

Bases: SourceRunner

category_filter

Translate oex CategoryConfig.osm.filter into quackosm filters and SQL.

Two consumers: - planet engine prep: build a single union OsmTagsFilter from N categories for the one-pass quackosm call. - planet engine query_for: build the per-category SQL WHERE predicate that picks just one category's features from the unified country.parquet.

union_tag_filter 
union_tag_filter(
    categories: Iterable[CategoryConfig],
) -> OsmTagsFilter

Merge N category osm.filter blocks into one quackosm OsmTagsFilter.

Rules: - Any True wins for a key (any-value match). - list+list -> sorted union; list+str -> list with str added; str+str -> list of both.

category_where_predicate 
category_where_predicate(category: CategoryConfig) -> str

SQL WHERE clause matching this category's osm.filter on tags MAP.

Returns a parenthesised expression suitable for AND'ing into a larger WHERE. Empty filter -> "TRUE" (matches all).

country_parquet

The country.parquet contract every OSM engine writes.

quackosm produces feature_id, tags and geometry for the geofabrik and planet engines; the live engines assemble the same three columns so the exporter, the category selects and the published schema stay identical whichever engine ran.

LiveSnapshot dataclass

What a live engine returns: the parquet it wrote and when the data was current.

write_country_parquet 
write_country_parquet(
    rows: list[Row], out_path: Path, source: str
) -> None

Write (feature_id, tags_json, wkt) rows to the contract, raising if it drifts.

engines

The OSM source engines, named once so config validation and dispatch agree.

errors

Errors shared by the OSM source engines.

OsmEngineUnavailableError

Bases: RuntimeError

An engine's upstream could not be reached or refused the request.

Distinct from the RuntimeErrors an engine raises for bad data, so a fallback covers an outage without masking a defect in what came back.

extract

osmium-tool subprocess wrappers for polygon-based PBF extraction.

The planet engine uses osmium-tool's extract command to clip a country PBF out of a planet PBF using a 5km-buffered admin polygon. We shell out because pyosmium does not expose extract --strategy=complete_ways and reimplementing the multi-pass strategy in Python is out of scope.

OsmiumNotInstalledError

Bases: RuntimeError

osmium binary not found on PATH.

OsmiumExtractError

Bases: RuntimeError

osmium extract exited non-zero.

osmium_polygon_extract 
osmium_polygon_extract(
    pbf_path: Path,
    polygon_geojson: dict[str, Any],
    out_pbf: Path,
    *,
    strategy: ExtractStrategy = "complete_ways",
) -> None

Clip pbf_path to polygon_geojson, write to out_pbf.

Polygon vertex count is engineered away by osmium's banded algorithm, so we pass the full-precision boundary (no simplification needed).

fetch_planet

OSM PBF download with HTTP Range resume and optional md5 verification.

geofabrik

Geofabrik country-PBF URL lookup via the public index-v1.json.

GeofabrikLookupError

Bases: LookupError

Raised when the index does not contain a country-level extract.

GeofabrikUnavailableError

Bases: GeofabrikLookupError

Geofabrik does not publish a country-level PBF for this ISO3.

Distinct subclass so callers can catch precisely (e.g. for planet_fallback) without swallowing other Geofabrik errors like network failures.

postpass

Postpass source engine: live OSM through Geofabrik's PostGIS SQL API.

Postpass mirrors the planet in PostGIS and refreshes every five minutes, so a small area comes back far fresher than a daily country PBF. The fetch writes the same country.parquet contract quackosm produces: feature_id, tags, geometry.

hstore_predicate 
hstore_predicate(tag_filter: OsmTagsFilter) -> str

SQL predicate over the hstore tags column matching an oex tag filter.

Mirrors category_where_predicate, which targets a DuckDB MAP instead.

build_sql 
build_sql(
    table: str, predicate: str, boundary_geojson: str
) -> str

One table's query: geometry travels as WKT so tags stays a flat JSON string.

fetch_country_parquet 
fetch_country_parquet(
    *,
    boundary_geojson: str,
    tag_filter: OsmTagsFilter,
    out_path: Path,
    endpoint: str = DEFAULT_ENDPOINT,
    timeout: int,
) -> LiveSnapshot

Query Postpass for everything matching the filter and write country.parquet.

rawdata

HOT Raw Data API source engine: live OSM through an asynchronous export job.

The API tracks OSM continuously, so a small area comes back far fresher than a daily country PBF. A snapshot is submitted, polled until it completes, then downloaded.

build_filter 
build_filter(tag_filter: OsmTagsFilter) -> dict

Tag filter in the API's join_or shape; an empty value list means any value.

fetch_country_parquet 
fetch_country_parquet(
    *,
    boundary_geojson: str,
    tag_filter: OsmTagsFilter,
    out_path: Path,
    endpoint: str = DEFAULT_ENDPOINT,
    timeout: int,
) -> LiveSnapshot

Run one snapshot job and write its result to country.parquet.

runner

OSM source runner.

Two engines, one unified pipeline. Both produce a single country.parquet per (iso3, snapshot) by running quackosm once with the union of all category tag filters and keep_all_tags=True. Per-category extraction is a tag-predicate WHERE at query time, no per-category PBF reparse.

  • geofabrik: download per-country PBF from Geofabrik, then build the country parquet. Cache: <cache>/geofabrik/<iso3>/<snapshot>/country-<fingerprint>.parquet.

  • planet: clip a country PBF out of a local planet PBF via osmium-tool, then build the country parquet. Cache: <cache>/planet/<iso3>/<snapshot>/country-<fingerprint>.parquet.

The fingerprint covers the boundary and the category tag filters, so changing either builds a fresh parquet instead of silently reusing one that was clipped to a different area or filtered to different tags.

OsmRunner

Bases: SourceRunner

overture

Overture Maps source: query the public S3 release bucket via DuckDB.

resolve_release 

resolve_release(
    release: str, *, bucket: str = "overturemaps-us-west-2"
) -> str

Return a concrete release version, resolving "latest" via S3 listing.

runner

Overture source runner: DuckDB httpfs read from s3://overturemaps-us-west-2.

resolve_release 
resolve_release(
    release: str, *, bucket: str = "overturemaps-us-west-2"
) -> str

Return a concrete release version, resolving "latest" via S3 listing.

palette

Layer colours, shared by the config default and the report pages.

pcodes

P-code tagging via fieldmaps.io edge-matched humanitarian admin polygons.

cache

Fetch and cache fieldmaps.io edge-matched admin parquets.

tagger

Pcode tagging via H3 integer hash join at resolution 7. Boundary residuals (~1-5% of features whose centroid H3 cell isn't owned by any admin) are resolved by either a 1-ring H3 neighbour hash lookup (default, memory-bounded) or a GEOS ST_Contains spatial join (precise but can OOM on large countries).

parse_boundary_resolution 
parse_boundary_resolution(value: str) -> BoundaryResolution

Validate and narrow a config string to BoundaryResolution. Fails loud on typos.

preflight

Pre-run sanity checks. Fail loud before doing any expensive work.

PreflightError

Bases: RuntimeError

A required precondition is not satisfied.

check_writable_paths 

check_writable_paths(cfg: RootConfig) -> None

Verify every directory the run needs to write to is writable.

Catches read-only filesystems and permission errors before downloading PBFs or running quackosm. Tests by creating, writing, then deleting a tiny temp file in each candidate path.

report

Multi-source HTML report rendered from per-source metadata.json payloads.

render_report 

render_report(
    sources: dict[str, SourceMetadata],
    tilesets: dict[str, tuple[str, str]] | None = None,
    boundary_bbox: tuple[float, float, float, float]
    | None = None,
    layer_label: str | None = None,
    palette: list[str] | None = None,
    map_assets: MapAssetsConfig | None = None,
    *,
    boundary_geojson: str | None = None,
) -> str

Per-category data quality report.

tilesets maps a source name to its (pmtiles_url, layer_name). A per-category dataset carries one tileset per source, so each becomes its own map entry. layer_label names the category, so the legend reads "Buildings (Overture)" rather than just "Overture".

html

Multi-source HTML report renderer.

render_report 
render_report(
    sources: dict[str, SourceMetadata],
    tilesets: dict[str, tuple[str, str]] | None = None,
    boundary_bbox: tuple[float, float, float, float]
    | None = None,
    layer_label: str | None = None,
    palette: list[str] | None = None,
    map_assets: MapAssetsConfig | None = None,
    *,
    boundary_geojson: str | None = None,
) -> str

Per-category data quality report.

tilesets maps a source name to its (pmtiles_url, layer_name). A per-category dataset carries one tileset per source, so each becomes its own map entry. layer_label names the category, so the legend reads "Buildings (Overture)" rather than just "Overture".

panel_sections 
panel_sections(source: SourceMetadata) -> str

Metadata sections for one source, without the tab wrapper (reused by the landing page).

landing

Combined-dataset landing page: a layers-and-data-quality table, then a map.

Rendered as the HDX custom viz when several categories are published onto one dataset. The table carries one row per layer (source colour swatch, feature count, named coverage, geometry mix, top categories); the interactive map comes last and is coloured to match the swatches. The map is shown only when a combined PMTiles resource exists.

render_landing 
render_landing(
    *,
    title: str,
    subtitle: str,
    panels: list[CategoryPanel],
    pmtiles_url: str | None,
    pmtiles_layer: str | None,
    boundary_bbox: tuple[float, float, float, float]
    | None = None,
    boundary_geojson: str | None = None,
    palette: list[str] | None = None,
    map_assets: MapAssetsConfig | None = None,
) -> str

Render the combined landing page. panels must be in the intended display order.

map_block

Shared PMTiles map for the report pages.

Both report pages draw the same map. They differ only in how their tiles are packaged: the combined landing page has one merged tileset whose features carry category and source, while a per-category dataset has one tileset per source. A MapEntry names its own tileset, so both shapes render through this one path.

MapEntry dataclass

One toggleable, coloured layer on the map.

render_map 
render_map(
    entries: list[MapEntry],
    boundary_bbox: tuple[float, float, float, float] | None,
    assets: MapAssetsConfig | None = None,
    *,
    boundary_geojson: str | None = None,
) -> str

Render the map section: heading, canvas, legend with a checkbox per entry.

quality

One definition of a layer's data quality, shared by both report pages.

The per-category report and the combined landing page describe the same layers, so they measure them the same way: how many attribute columns carry data, and how much of the layer is named.

s3

Upload artifacts to S3 and return a public URL for HDX linking.

artifact_key 

artifact_key(
    prefix: str,
    iso3: str,
    category_slug: str,
    filename: str,
    *,
    folder: str = "",
    nest_by_category: bool = True,
    geometry: str = "",
) -> str

S3 key for a category artifact. folder overrides the iso3 segment (empty falls back to iso3); a false nest_by_category flattens the path. geometry adds a points/lines/polygons segment under the category.

build_layer_key 

build_layer_key(
    prefix: str,
    iso3: str,
    source: str,
    slug: str,
    folder: str = "",
) -> str

Stable, globbable key for a per-layer GeoParquet so runs accumulate across sources.

folder overrides the iso3 segment, so two configs covering the same country stage into separate areas instead of overwriting each other.

list_layer_urls 

list_layer_urls(
    cfg: S3Config,
    iso3: str,
    *,
    exclude_source: str | None = None,
) -> list[tuple[str, str, str]]

List staged per-layer GeoParquets as (source, slug, url), optionally skipping one source.

sources

Per-source query builders.

A source knows how to expose a parquet read expression and a metadata block for a given category. The shared exporter then does the bbox/clip/select/ write/zip steps in a uniform way.

CategorySkippedError

Bases: RuntimeError

Raised by a source when a category is not applicable to it.

SourceRunner

Bases: ABC

peek_snapshot_label 
peek_snapshot_label(cfg: RootConfig) -> str | None

Best-effort snapshot label without doing network work or running prepare.

Returns the label the runner would adopt in prepare(), or None if it can't be determined cheaply. Used by the exporter to short-circuit the run when every category is already uploaded for that label.

base

Abstract source interface.

CategorySkippedError

Bases: RuntimeError

Raised by a source when a category is not applicable to it.

SourceRunner

Bases: ABC

peek_snapshot_label 
peek_snapshot_label(cfg: RootConfig) -> str | None

Best-effort snapshot label without doing network work or running prepare.

Returns the label the runner would adopt in prepare(), or None if it can't be determined cheaply. Used by the exporter to short-circuit the run when every category is already uploaded for that label.

file

File source: a user-supplied spatial file read through DuckDB's ST_Read.

Anything GDAL opens works (shapefile, GeoPackage, GeoJSON, FlatGeobuf), from a local path, an https URL or s3://. Coordinates are reprojected to OGC:CRS84 when the file declares a different CRS, because oex clips and filters in lon/lat.

FileSourceError

Bases: RuntimeError

The configured file cannot be used as a source.

resolve_layer 
resolve_layer(available: list[str], configured: str) -> str

Which layer to read. ST_Read silently takes the first, which would drop the rest.

read_expr 
read_expr(
    path: str, source_crs: str, layer: str = ""
) -> str

Subquery that opens the file and exposes exactly one geometry column in lon/lat.

The rest of the pipeline renames geometry to geom itself, so the file's own geometry column is dropped here to avoid a second one reaching the writer.

declared_crs 
declared_crs(
    conn: DuckDBPyConnection, path: str
) -> str | None

AUTH:CODE the file declares, or None when it carries no CRS at all.

resolve_crs 
resolve_crs(declared: str | None, configured: str) -> str

The CRS to transform from.

A file with no CRS is ambiguous, and assuming lon/lat would silently place the data in the wrong part of the world, so the config has to say.

geometry_expr 
geometry_expr(source_crs: str) -> str

Geometry in OGC:CRS84, transforming only when the source is something else.

always_xy is required: ST_Transform otherwise honours the authority's axis order, while ST_Read always hands back x/y, so a latitude-first CRS such as EPSG:4258 comes out with its coordinates swapped and EPSG:3035 lands in the wrong country.

file_timestamp 
file_timestamp(path: str) -> datetime

When the data was last written, which becomes the HDX time period.

labels

How each source is named in output, so the four places that show it agree.

sql

SELECT/WHERE clause builders + materialise(); shared by both sources.

build_where_clause 

build_where_clause(
    boundary: Boundary,
    where_conditions: list[str],
    bbox_cols: str,
) -> str

Combine bbox prune + boundary intersect + caller-supplied conditions.

bbox_cols="bbox" uses an upstream bbox struct (Overture); "geom" derives the bbox from the geometry column (OSM cache).

state

Per-(country, source) resume state, atomic-write JSON.

A run keeps a single .state.json per (output_dir, iso3, source) recording, for each category, when the local build finished and when the HDX upload completed. With output.resume enabled the exporter consults this to skip already-finished work after a partial run, and HDX rate-limit storms become recoverable without rebuilding zips.

State is keyed by category slug. A snapshot label mismatch (different PBF) is treated as a miss so a fresh snapshot always rebuilds.

StateStore

Read/write the per-(iso3, source) resume state JSON, atomically.

system

Thread count and memory limit defaults, read from the cgroup before psutil.

psutil reports the machine, so under a container or unit limit it oversizes the run.

total_memory_gb 

total_memory_gb() -> float

Memory this process may use, in GB. OEX_MEMORY_GB overrides the detection.

cpu_count 

cpu_count() -> int

CPUs this process may use, honouring cgroup quota and scheduler affinity.

adaptive_parallel_resources 

adaptive_parallel_resources() -> tuple[int, int]

Compute (parallel_workers, memory_gb_per_worker) scaled to available RAM.

Always returns 1 worker. DuckDB's intra-query pipeline engine parallelises every operation (joins, scans, aggregations) across all CPU cores within one session. Concurrent sessions split the RAM budget with zero cross-session coordination and OOM-kill each other on large countries (BRA, IND, CHN).

Memory: 60% of available RAM, DuckDB's recommended safe fraction for a single session. Leaves headroom for GDAL write allocations, string heaps, and spatial index structures that bypass the buffer manager.

translit

DuckDB-side transliteration to Latin via unidecode.

engine

Add Latin display columns to a materialised table via unidecode.

writers

GIS format writers over materialised DuckDB tables.

geometry_labels 

geometry_labels(
    conn: DuckDBPyConnection, table_name: str
) -> dict[str, list[str]]

Geometry labels present in a table, mapped to the source types they cover.

write_pmtiles 

write_pmtiles(
    conn: DuckDBPyConnection,
    table_name: str,
    out_path: Path,
    *,
    min_zoom: int,
    max_zoom: int,
) -> Path

Write a single-layer PMTiles archive via GDAL. The layer name is the file stem.

write_geoparquet 

write_geoparquet(
    conn: DuckDBPyConnection,
    table_name: str,
    out_path: Path,
) -> Path

Write the full table as GeoParquet. DuckDB emits GeoParquet metadata for the GEOMETRY column, so the file round-trips as GEOMETRY and reads back for tiling.

build_combined_pmtiles 

build_combined_pmtiles(
    conn: DuckDBPyConnection,
    layers: list[TileLayer],
    out_path: Path,
    *,
    min_zoom: int,
    max_zoom: int,
) -> Path

Merge per-layer GeoParquets (in the given order) into one PMTiles layer.

category and source are injected per layer so the single tileset stays styleable by both. Each layer's parquet only needs a geom column; name is carried when present.

zip_bundle

Per-format zip bundles with README, config snapshot, and optional metadata.